The UK's got it right - government-collected data access for free

January 21st, 2010 3 comments

"A new website, data.gov.uk, will offer reams of public sector data, ranging from traffic statistics to crime figures, for private or commercial use.

The target is to kickstart a new wave of services that find novel ways to make use of the information."

Awesome. I had an idea like this for infoporn.org a while back - generic data being available for consumption, but I really don't have any origin feeds that aren't already exposed. I'd just be re-syndicating them. But this would be awesome to have, just think of all the mashups you could create depending on what data is exposed.

Ref: http://news.bbc.co.uk/2/hi/technology/8470797.stm

Categories: Consumerism, Development

You are an idiot.

January 18th, 2010 No comments

FUD FUD FUD FUD

So there's more annoyance polluting the blogopshere, and it's about Facebook's privacy practices again.

They keep your data, even if you delete it. Imagine that!

Anyone who thinks that their data is truly deleted because they click "delete" on anything needs to wake up. YOU own your own privacy. If you don't want it to stick around then DON'T SHARE IT. Period.

I'm going to drop some more knowledge for you, and this one is for free. Expect whatever you say, wherever you say it, to be there forever. True deletion should be considered a BONUS, not an EXPECTATION.

FUD FUD FUD FUD

Refs:

Categories: Consumerism

SPNEGO For nginx - a start, at least

January 17th, 2010 4 comments

I've been posting on the nginx mailing lists for a while that I've had a developer working on SPNEGO (Kerberos/GSSAPI/etc.) module for authentication for nginx. I never produced any code though, because I was constantly waiting for a bit more of a matured module before giving it out.

For now though, it might just be best to throw out there what I have, explain my findings, and let the community start testing it, hacking it, improving it, etc.

Download
ngx_http_auth_sso_module-0.3.zip

Opens for the developer still

  • He said he'd like to remove the dependency on the bundled spnegohelp library (apparently it's not needed or it can be filled with a system package)
  • Needs some more documentation

My questions, comments

  • Should it be called "mod_auth_sso" or something like "mod_auth_gssapi" - I believe Apache's equivalent has "gssapi" in the title somewhere. It was hard to determine which was the most up to date version - mod_auth_kerb, modgssapache, etc.
  • I cannot verify this still using my corporate domain setup. I did have to make a minor tweak in the source to change the principal name it was using (see below) and I still got access denied. I have no clue if the module is to blame, the machine's setup with the domain, or what.

A possible required tweak... around line 474 or so in ngx_http_auth_sso_module.c

- host_name = r->headers_in.host->value;
+ host_name = alcf->realm;

How to compile - yes, it's a bit messy, mainly due to the spnegohelp library dependency 🙂

wget http://sysoev.ru/nginx/nginx-0.8.31.tar.gz
tar xvfz nginx-0.8.31.tar.gz
cd nginx-0.8.31
./configure --conf-path=/etc/nginx/nginx.conf --prefix=/usr --user=www-data --group=www-data --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-http_stub_status_module --with-http_gzip_static_module --without-mail_pop3_module --without-mail_smtp_module --without-mail_imap_module --with-http_flv_module --with-http_ssl_module --with-http_dav_module --with-http_realip_module --with-http_xslt_module --with-debug --add-module=/usr/src/build/ngx_http_auth_sso_module-current --with-ld-opt="-L/usr/src/build/ngx_http_auth_sso_module-current/spnegohelp"
cp -f /usr/src/build/ngx_http_auth_sso_module-0.3/spnegohelp/libspnegohelp.so /usr/lib64/libspnegohelp.so
... make, make install, whatever ...

I copied the libspnegohelp.so into the system so nginx can use it without any special runtime LD_LIBRARY_PATH crap.

How to configure (again, I could not validate this 100%)

location /test {
   auth_gss on;
   auth_gss_realm YOUR.KERBEROS.REALM;
   auth_gss_keytab /etc/krb5.keytab;
   auth_gss_service_name YOURMACHINENAMEIBELIEVE;
}

Conclusion
Your mileage will definitely vary but hopefully some people with more experience with Kerberos, C, nginx modules, or anything else helpful can pick it apart. I will post updates as I get them and I do want to post this on github or something... unless someone else wants to take ownership over it who will actually actively maintain/keep it up to date with nginx internals and maintain the github/whatever repository for it.

This was self-funded personally with no company sponsorship or anything, part of the terms of the project were also to keep it BSD licensed. Feel free to do whatever you want with it. If you want to send me any money to reimburse, I'll never say no to that - paypal AT mike2k.com. Or, pay a developer who can ensure it works end-to-end, if there is something missing.

Sadly, it does require the machine to be on the domain, I was hoping I could get away with it not having to be on the domain. I confirmed with Sam Hartman, who was a chief technologist at the MIT Kerberos Consortium - you can't really get more knowledgeable than that. I would have hired Sam for the project but it would have been too expensive. However if a company is willing to take this initial coding and have Sam add his magic to it, he knows C and was able to give me a real quick estimate and check how nginx modules work. He could possibly do it for cheaper now since the initial work might be done, and he would definitely be able to confirm all the logic is intact.

Thanks to YoctoPetaBorg at RentACoder.com for the initial work (and for hopefully soon finishing up the last bits of this :p) - it will be exciting to be able to use this at work and be able to have a fully functional module out there that people find useful.

Categories: nginx

The future of open source SQL databases (as I see it)

January 9th, 2010 1 comment

With the whole MySQL/Oracle issue going on, I find myself looking into the future and how I see it. As far as I'm concerned, MySQL will start to lose it's popularity as the landscape changes. As far as I am concerned, there will be two key players in the MySQL replacement market, those being Drizzle and MariaDB.

I am not just saying Drizzle just because I help out with the project in various ways, however, that should be a good sign that I believe in it if I am willing to put any effort into it. With people behind it like Brian Aker, Eric Day, Monty Taylor, Stewart Smith, Jay Pipes, you've got a coding powerhouse that could solve the cancer issue if it was up to software development to fix it. These guys work around the clock and have been refactoring and re-examining everything inside of MySQL. What's going to be left ideally is a superfast microkernel that supports plugins for everything - leveraging the best options out there for replication, messaging, storage engines, etc. Growing apart from the monolithic huge distribution model that MySQL currently follows.

The second key player is MariaDB. Another fork off of MySQL, led by Monty Widenius himself and with other MySQL key players behind it, there is no doubt it will continue Monty's legacy as being able to spin success out of a tiny little open source product. I believe it will stay more traditional in-line with MySQL, but will provide more advanced functionality and scalability as it is developed further.

I won't get into other options like PostgreSQL as I don't follow the rest of the community there much.

Also, we'll see more NoSQL (did we ever bottom out on a better term for that?) options. CouchDB and MongoDB (both of which from a 50,000 foot view look identical from a usage model) and options like Cassandra will also become important and your data needs will become the decision maker for going with a SQL or a NoSQL database. Both of which offer advantages. However, I see Drizzle as making huge strides in leveling the playing field (or attempting to) with it's replication work to make it as scalable as NoSQL databases seem to be with their ability to scale out and replicate changes easily (which to me are their main selling point right now...)

Anyway, this is from a user perspective, not a developer perspective, and from what I've seen from #drizzle on freenode, a few SQL and open source conferences, blog talk and my own gut feelings.

I should make a note that I still use MySQL and will probably continue for some time. Neither Drizzle nor MariaDB are production-friendly yet. However, I believe 2010 should see the first "production capable" release of Drizzle (not sure of MariaDB.)

It is an exciting time though as we're starting to be presented with more options by the day, in fact there are so many various NoSQL databases now, key/value stores, and even a few more SQL databases that it's too hard to keep track of them anymore. There's a lot of code being written and with this whole Oracle possibly inheriting MySQL depending on the EU's judgement, it could ultimately help usher in some of these smaller projects into the spotlight quicker depending on what Oracle does with MySQL...

Categories: Drizzle, Software

Jérôme Loyet is a saint!

December 15th, 2009 No comments

I'd like to announce that Jérôme Loyet stepped up this weekend and hacked up the first round of code to get dynamic process management going. Antony committed it (see below) and it's on its way to being part of PHP core as well. So two major events in only a week and a half or so!

See http://news.php.net/php.internals/46414.

Good job Jérôme. Where were you a couple months ago, buddy? 🙂

Categories: PHP, PHP-FPM

PHP-FPM brought in to PHP core - interesting surprise

December 5th, 2009 No comments

Read it here: http://news.php.net/php.internals/46277.

First off, big thanks to Antony Dovgal. I've exchanged words with him in the past about PHP-FPM (and actually other PHP things) but was completely unaware he was working on this.

So, we've got a blessing but also an interesting dilemma on our hands. We've got a wishlist and some bugs to work out. I have a feeling if Antony updated some of the CGI internals it may have resolved some of those bugs. Not sure. I'm trying to get some specifics now - what version of PHP-FPM he brought in, how the community can still support it and how difficult it may be to submit patching, if he thinks a separate management daemon makes more sense than keeping it glued inside of the SAPI (it seems out of place to me for a SAPI to require a proprietary configuration file and daemon .pid, log, etc. files...)

Hopefully I can get ahold of him soon and discuss some of this. I was already mid-discussion with another PHP core developer about how they think the best approach would be to get PHP-FPM aligned with PHP core (they leaned more towards a separate SAPI too.)

My main goal is to make it easy as it can be for PHP-FPM to become an official package or included with PHP so that people who use PHP from repositories on their favorite distributions and such can enjoy the benefits of PHP-FPM without patches or separate downloads. If the management portion does split off, I fully intend on making sure it is aligned properly and is as simple as "apt-get install php5-fpm" or something of that nature. Still easily installed and everything.

Anyway, we'll see how things go. This caught me off guard and now I have to figure out at what point we're at now with development. Jérôme Loyet has expressed interested in trying to convert the configuration file to nginx style - something Andrei had told me he had wanted to do. The XML throws some people off, thinking it's an actual XML parsed document with XML include support and such... also if done right, this will allow PHP-FPM's configuration to support includes, and who knows, maybe variables some day. But for now it would be a lot cleaner to read, and it seems the majority of PHP-FPM users are nginx users already anyway 🙂

Categories: PHP, PHP-FPM

WordPress 2.8.x - hiding those unwanted dashboard items

November 18th, 2009 No comments

There's a completely new way now to hide dashboard items, and I finally think I found the easiest way to do it, after having to poke around looking for the right hooks. This basically applies the same set of "screen options" to every user regardless if they've customized it or not. It leaves only the "Right Now" and "Recent Drafts" on the dashboard, but feel free to customize this array by checking the name that you want to hide and adding it to the array.

Also you can easily add more options such as column layout and such by customizing it on a user account, going into the database and looking for the 'metaboxhidden_dashboard' and related keys in the usermeta table and force-applying those (which is essentially what I did here)

Just thought I'd share this little tidbit of information. It can be put in its own plugin or inside of an existing one, as long as the filter gets triggered you're good to go. I've got it working properly in 2.8.6 currently without an issue.

function hide_dashboard_stuff() {
   $hide = array(
      0 => 'dashboard_recent_comments',
      1 => 'dashboard_incoming_links',
      2 => 'dashboard_plugins',
      3 => 'dashboard_quick_press',
      4 => 'dashboard_primary',
      5 => 'dashboard_secondary',
   );
   return $hide;
}

add_filter('get_user_option_metaboxhidden_dashboard', 'hide_dashboard_stuff', 1);
Categories: WordPress

Transifex + Django + nginx on Ubuntu

November 1st, 2009 No comments

You can probably also use this for just Django + nginx on Ubuntu too, it should work the same for any Django-based package, just change the Transifex-specific stuff (like /site_media and such.)

Dependencies:

  • python-django package from Ubuntu (currently Jaunty's 1.0.2-1ubuntu0.1)
  • Transifex installed to /home/transifex/web/transifex, from source
  • Transifex setup to answer over fastcgi on port 8080
  • Tested on nginx 0.7.x and 0.8.x compiled from source

/etc/nginx/nginx.conf:

server {
   listen 80;
   server_name foo.com;
   include /etc/nginx/confs/defaults.conf;
   location / {
      fastcgi_pass 127.0.0.1:8080;
      include /etc/nginx/confs/django.conf;
   }      
   location /site_media {
      root /home/transifex/web/transifex;
      include /etc/nginx/confs/expires.conf;
   }
   location /media {         
      root /usr/share/python-support/python-django/django/contrib;
      include /etc/nginx/confs/expires.conf;
   }
}

/etc/nginx/confs/defaults.conf:

location ~ /\.ht {
   deny all;
}

location ~ \.svn {
   deny all;
}

log_not_found off;
server_name_in_redirect off;

/etc/nginx/confs/django.conf:

# http://www.rkblog.rk.edu.pl/w/p/django-nginx/
# http://code.djangoproject.com/wiki/ServerArrangements
fastcgi_param PATH_INFO $fastcgi_script_name;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param SERVER_NAME $server_name;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_pass_header Authorization;
fastcgi_intercept_errors off;

/etc/nginx/confs/expires.conf:

location ~* \.(jpg|jpeg|gif|css|png|js|ico|html)$ {
   expires max;
   access_log off;
}
Categories: nginx

My recipe for ZFS at home

September 28th, 2009 4 comments

After spending a lot of time Googling around, reading forums, blogs and bug reports, etc. I was able to come up with the most amount of storage in the quietest case I could find, that should be compatible with Solaris/OpenSolaris. I've found a recipe that works and I'd like to share it.

My main goal was something with as many drive bays as possible and as quiet as possible. That was it. I think I've been able to do that pretty successfully with this configuration. So much so I just finished building my second one.

Of course, you can piece this together almost 100% from Newegg (which I what I did originally) for a little bit more money, but possibly save on shipping - and of course you'd also be ordering from the best online computer retailer.

Anyway I've gone through and found what I believe to be are the cheapest places to find these parts. Together it totals $1,390.27 and gives you 16 drive bays for data in a pretty quiet configuration. The loudest components are the fans on the 5-in-3 SATA Chassis units - if you can swap those out with something else, you'll have less drives available but could make it almost silent then.

Then all you need is to buy the drives... I used 1.5TB Seagate drives. Their price point is excellent right now, typically running around $105 at Fry's consistently.

I set them up in two 8 disk RAIDZ2 pools, for double redundancy in both vdevs, and combined them into one large pool.

Performance seems to be good, have no complaints yet. It's been reliable as hell too. All off a single power supply!

Compatibility - I am running Solaris 10u7 on one, because I don't need any of the OpenSolaris/SXCE/etc. features, I'd rather have something a bit more "stable" - one of the reasons is mentioned below.

The other one is running snv_104 (SXCE) - I wish I would have tried Solaris 10 first on that one. Oh well. I did try upgrading it to snv_110 or thereabouts at one point and it got stuck at boot time. I am not sure if that bug has been fixed yet or not. So be aware, regression issues can break compatibility. Although it shouldn't! *cough*

Hope this helps some people out!

Categories: Toys

I believe I am in love with Mila Kunis

September 4th, 2009 5 comments

Every single thing I see with her in it warms my heart. How smoking was she in Forgetting Sarah Marshall? Now I'm watching some Family Guy documentary and she just looks amazing in every scene, and some of these are not all "made up" either.

Mila, I am formally proposing to you. Forget all those Hollywood guys and your current marital status. I'll take good care of you. You'll most likely be the breadwinner though.

Love,
Mike

Categories: Uncategorized