The Life and Times of Michael Shadle

You know, Netflix is neat. Not only is their website pretty simple and clean, the company itself seems to "do right" by its customers too. Today is the second time they've made me feel warm and fuzzy being a customer. The first time was when they decided to change their rate plans to be cheaper and automatically started charging me less. A lot of companies drop the price on their monthly rates and you have to re-signup or call and complain, etc...

Dear Michael,

Great news! We're lowering the price of your 3 DVDs out at-a-time plan to $16.99 a month plus applicable taxes. Now you can enjoy Netflix for less!

You don't need to do a thing - except pay less. Your membership will automatically move to the lower price and be reflected in your Membership Terms and Details. The lower price will take effect beginning with your statement on or after July 23, 2007.

Membership Terms and Details: http://www.netflix.com/Terms

Your $16.99 plan not only gives you 3 DVDs out at-a-time but you can also watch 17 hours of movies and TV episodes instantly on your PC each month - for no additional charge.

Check it out: http://www.netflix.com/WatchNow

Enjoy!
Your friends at Netflix

Now that is cool. But then they've given me another reason to praise them. They're reporting disc shipment issues, and being nice about it, they've just decided to issue a credit since we're technically losing a few days of service. They didn't have to do that, but they did.

I just received this email today:

We're Sorry DVD Shipments Are Delayed

Dear Michael,

Our shipping system is unexpectedly down. We received a DVD back from you and should have shipped you a DVD, but we likely have not. Our goal is to ship DVDs as soon as possible, and we will keep you posted on the status of your DVD shipments.

We are sorry for any inconvenience this has caused. If your DVD shipment is delayed, we will be issuing a credit to your account in the next few days. You don't need to do anything. The credit will be automatically applied to your next billing statement.

Again, we apologize for the delay and thank you for your understanding. If you need further assistance, please call us at 1-888-638-3549.

-The Netflix Team

I think I keep my machines pretty well up to date. I'm fixing up my girlfriend's computer right now, and I finally got all the updates applied except apparently one got missed. Okay, simple enough right?

Step #1: Install .NET Framework 1.0, click "Check for updates" - what? there's another update?

Step #2: Install .NET Framework 1.1 Service Pack 1, click "Check for updates" - what? ANOTHER one?

Step #3: Install .NET Framework 1.1 Service Pack 1 security update, click "Check for updates" ...

Really? Why couldn't we have a pre-packaged full install? Why do I have to run this "check for updates" over and over (and why does it take so goddamn long nowadays?)

Okay, looks like that really was the last update and this machine is now 100% up to date. At least for the moment...

At first I thought Lighttpd was the cat's meow. After talking with Andrei (who maintains PHP-FPM) I thought - if he got PHP-FPM right, he must know his stuff. He recommended I try nginx - and boy am I glad I did.

Igor is an interesting character to me. He is very matter-of-fact, he has no problem issuing patches almost instantly to enhance his product, and he also has no problem being short with people when rejecting an idea or informing them they're wrong. To me it seems like sometimes people who maintain projects try to be more politically correct, but from what I've seen, Igor seems to be extremely technical by heart, and does not really stop to smell the flowers (at least on the mailing list...)

Igor gets mad props for creating nginx - quite possibly the most efficient web/proxy server on earth. For a bit I was using it to proxy 4+ million web requests a day (small php, html, graphics and even larger file and video downloads) through a single server, doing gzip as well - and it handled it all without using more than 14 megs of physical RAM. Nginx is such an engineering feat that I've actually started contemplating how to re-do my architecture since it is totally viable now to have a single frontend server proxying all the dynamic requests to dedicated FastCGI servers. No need really to be running multiple nginx instances anymore... one handles everything!

Anyway, back to Igor - my new goal in life is not only to help promote PHP-FPM but now nginx as well. There's only a couple minor things I wish nginx would do a bit better, but otherwise, it is my web solution and possibly even my proxy solution to replace LVS. Who would have thought a userland daemon could be so efficient? Even though Igor maintains nginx almost 100% by himself, releases come out often, he can produce patches to fix bugs or add features within a couple hours, and he usually replies to emails within the same day on the mailing list. I haven't seen that level of support from any other open source project, much less commercial products. "Want a patch? Wait for our next release in six months!"

Igor's website is at http://sysoev.ru/en/, however it doesn't really have much info on it - and you'll probably be looking for nginx at http://nginx.net/ anyway. Thanks Igor 🙂

Note: this has been outdated now. I have an updated post with the latest and Igor-approved method here.

When I was switching over to nginx, I found a handful of random and overkill config examples to make it work. I thought I had found the simplest solution already, but Igor (the creator of nginx) actually gave me an even "better" solution in nginx.

1st example (seems like overkill, can't figure out a reason why people are splitting up the rewrite rules - I guess because they assume "wp-anything" is all static?):

if (!-e $request_filename) {
   rewrite ^([_0-9a-zA-Z-]+)?(/wp-.*) $2  break;
   rewrite ^([_0-9a-zA-Z-]+)?(/.*\.php)$ $2 last;
   rewrite ^ /index.php last;
}

2nd example (this was as good as I thought it could be, but I was wrong):

if (!-e $request_filename) {
   rewrite ^(.+)$ /wordpress/index.php?q=$1 last;
}

3rd example (this is the best, according to Igor):

error_page 404 = /wordpress/index.php?q=$uri;

(Ref: http://article.gmane.org/gmane.comp.web.nginx.english/4739)

In the past, I've always been able to disable IPV6 in Linux with a simple line in any /etc/modprobe.d file:

alias net-pf-10 off

It appears though it has no effect anymore. I had to change it to this line:

blacklist net-pf-10

How annoying.

For the most part, open source has been good to me. I can run my business on free software that is for the most part better than its commercial counterparts. I'm able to use often-updated and relatively bug-free software in all facets of my life. However, there are a couple things that have been bugging me - one of them has been SATA Port Multipler (PMP, not to be confused with PM - Power Management!) support. 1TB drives hit an all time low last week (at least in my ads) and it got me motivated to check in to the status of PMP support once again. Previously there weren't a lot of people using it and not a lot of feedback or information. Today, I have a bunch of good information, but still not the magic combination, which for me, includes ZFS (as it is the only filesystem out there still to offer all the good checksumming, self-healing, etc.) This post isn't going to talk about why ZFS rocks. Most people already know why. I want a filesystem that has been built for integrity. This is for home archiving, and I'll be damned if bit rot is going to ruin my memories.

My goal was to find an OS that has ZFS support, which currently is between Solaris (or OpenSolaris, Nexenta, etc.) and FreeBSD 7 (which has "experimental" but usable ZFS support) - I'd prefer Linux as it has the fastest rate of bug fixes and driver support and I am already using Linux on one of my home machines, but ZFS doesn't run natively on it (FUSE is not native enough.)

The following below are the current status I am aware of taken from direct emails exchanged with Adriaan de Groot (who said he might port PMP support to FreeBSD 7) and Tejun Heo (who maintains the PMP driver libata-pmp for Linux) and Google searches, mailing list and forum postings as of today.

Windows

I'm not even going to bother here. Most drivers for the controllers are coded for Windows first, and then if we're lucky an open source Linux/etc. version is put out. So Windows support is probably 100% - but you're stuck to using NTFS or something (blech.)

FreeBSD

According to Adriaan:

"Don't bother, as port multipliers are not supported. I had some PMP support hacked in at some point, but without NCQ there's not much (performance) point. It *might* be that Soren has added NCQ / PMP in the meantime, but I'm not aware of it."

Also:

"PMP requires NCQ to be useful; otherwise you end up queuing all the requests for the different disks and performance goes down a fair bit - but like I said, it's *possible* and if all you're after is big storage, it will work. Anyway, it's not the cards' fault: there just isn't any NCQ in the (FreeBSD) kernel."

Tejun's comment:

Although NCQ and PMP supports don't really have to go together. Non-NCQ PMP should work as good as direct non-NCQ. Nothing more to lose there because it's attached via PMP. Maybe they were talking about pushing multiple commands to a port. If the driver can't do that, PMP won't work too nice.

As of now according to Adriaan, there is no PMP support in FreeBSD (except possibly some highly experimental patches.)

Solaris (and variants)

From Adriaan:

"I run OSOL ... but OSOL also does not support PMP"

Additional links:

• http://www.genunix.org/wiki/index.php/OpenSolaris_Storage_Developer_Wish_List
• http://www.mail-archive.com/[email protected]/msg13192.html
• http://www.opensolaris.org/jive/thread.jspa?threadID=49507&tstart=45
• http://www.opensolaris.org/jive/thread.jspa?threadID=54481&tstart=45

OS X

Possibly? I did not look into this. It would be a shame if some were supported with NCQ and not given back to FreeBSD... this guy claims to have what sounds like two 4:1 multipliers on his Mac:

http://www.mail-archive.com/[email protected]/msg13749.html

Linux

(There is a site that tries to keep up to date with SATA support in the Linux kernel. This is probably the best place to look. As of writing, it still appears up to date. http://linux-ata.org/driver-status.html)

According to Tejun, who was extremely helpful answering my barrage of questions gracefully, it does look like Linux has some PMP options (now if only we had ZFS or BTRFS...!) with the following notes for each chipset:

Controller chipsets: SiI3124, SiI3132 - both will work. His comments:

"3124/3132 controllers are good"

"Performance-wise, all the PMPs on the market should be okay but 3124/3132s have certain limitations on PCI bus side and can't deliver full bandwidth concurrently from all drives attached through PMP. I don't remember the exact numbers but it maxes out after three drives or so."

"I have second gen chips from SIMG and they behave really good."

Note: SiI3124-1 is only 1.5Gbps. SiI3124-2 supports 3Gbps.

Enclosure chipsets: SiI3726, SiI4726, Marvell 88SM4140  - all should work great. His comments:

"3726/4726 PMPs are the first gen PMPs and both are a bit quirky"

"Marvell port multipliers behave really well."

"That said, 3726/4726's work okay too. The only problem is that they always need their first slot occupied to operate correctly. As long as you occupy the first slot, there shouldn't be much problem."

Update: "There have been reports of 3726/4726 PMPs having trouble with 3Gbps link speed under certain configurations. I still don't know what's the exact cause.  It doesn't seem common but if you can maybe trying out with one drive before ordering the whole array is a good idea."

Note: the Marvell is only a 4:1 multiplier, the SiI ones are 5:1.

So the big winner here for the time being is Linux - looks like I will be purchasing a SiI3124-2-based controller (probably this one, which claims it is a 3124A, a successor of the 3124-2) and a SiI3726 or SiI4726 based PMP enclosure. I am trying to determine which one will be the quietest right now. I don't believe that hardware RAID-5 is supported on any PMP, only RAID 0, 1, and 10. It gets confusing because most advertise RAID but it leverages some software RAID stuff. So I will probably using mdadm for RAID5 at home.

Hopefully this is a useful (and correct) summary. I invite anyone with updated information to leave a comment, and I will correct it. Especially if you have experience with eSATA PMP enclosures and have anything to add! Not many people have ventured out into these waters yet, but it looks pretty neat - off a single PCI-X or PCI-e card you could chain 20 drives all with decent bandwidth and only have one cable for each set of five drives...

For a long time I've been using Lighttpd as an alternative to Apache, Ubuntu, MySQL and PHP.

It looks like I might be mixing it up soon, swapping out some and adding a couple extras. Some of these I've been able to mess around with, others I am excited but have not yet had the chance...

PHP-FPM

PHP-FPM is just a patch for PHP, and a well-received one. I consider it important enough to showcase here. It matures the FastCGI SAPI and adds a couple of performance enhancements, graceful reloading, Apache-like process management for adaptive process spawning, and a much needed suexec-ish capability which will save me many headaches. It will hopefully be merged into PHP core when Andrei considers it "feature complete" and changes the license to be compatible. I can't wait. Right now it can still be used, and I've installed it without an issue on all my boxes - some serving up over a million PHP requests per day under different uid/gids. (Note: use Google Translate for the Russian URLs on the site above, the docs have not been translated to English yet. Google does a good job on it and it reads just fine.)

CouchDB

Throwing out all the concepts of structured databases and building a new system from the ground up with interoperability and scalability in mind as a data store? You've got me sold. It seems a lot of people are nervous about scaling MySQL (with good reason) and CouchDB might be a good alternative. Using RESTful URLs for everything and JSON as a lightweight (compared to SOAP/XML/etc.) transport language, it seems like we'll have plenty of options and usage models. I think I heard on a presentation as well that it will support files of any size, which potentially means it could be used not only as a possible RDBMS replacement (even though it says it isn't, I'm sure plenty of apps could use it), but also as a distributed document storage system (which it might already be considered.) Added bonuses: designed for high traffic, supports disconnected computing, self-healing replication, optimistic locking... I can't wait to play with this.

MogileFS

A distributed document storage system. I've thought about trying this out in the past, but I was mainly looking for a drop-in replacement for standard file storage. MogileFS may have some wacky method to do that via FUSE someday, but in the meantime, it can be leveraged for all application-based file storage, which I'd say is 95%+ of the files I deal with. Just like CouchDB, it leverages standards for communication (WebDAV for actual file access) and simple text-based socket communication which can be used from any programming language that supports a socket. Currently I am successfully running it with nginx serving up the WebDAV portion as opposed to the standard Perl-based webserver. It was too easy. I plan on trying to leverage this on xMike for all of the image uploads and other user assets, most definately. I like how it doesn't require any special low level support - it simply spreads files over N number of hosts and uses a tracker to determine which host(s) have which file(s) - and includes replication management so a broken node does not mean a broken file.

nginx

Powering a handful of extremely busy sites, nginx is the tiny webserver you may or may not have heard of. Every account of it I have read has done nothing but rave about it; I'm in the process of converting all the servers I manage over to it. With complaints about memory leaks in Lighttpd and Apache being bloated, I think it's prime time for nginx to get more attention as a viable option. It's still "beta" but what isn't nowadays? It's been running for over 2.5 years in production on the main site it was developed for and I'm sure many others. The configuration file syntax is extremely simple. It has a couple neat little additions, like the built-into-memory 1x1 transparent gif support (for all those webbugs and spacer images) so you no longer have to host it yourself and it serves it directly from memory. While that's a little bit off the basic needs of a webserver, that seems extremely useful as someone who has had to deal with those for years. Anyway, don't let the old "Mother Russia" style logo on the English wiki scare you. It's worth a shot, and could even replace Pound, other reverse proxies and Layer 7 capable load balancing solutions. I'm sure someone might even be able to write a replacement for Squid and Varnish using it too, by enhancing the proxy module to save local cached copies of the content...

I'm sure I might be missing a couple. It's getting late. I'd add memcached to the list, but I'm already using it. It's no longer "taking my interest" as I've been able to fully integrate it now 🙂

Interesting to note that all these products (especially MogileFS and CouchDB) are capable of being distributed and allow for transparent node failures and were designed to be ran on unreliable/commodity hardware. nginx basically does the same as well, since it is a web node (and I run the FastCGI processes on them too) and can scale horizontally already. It does kind of make me wonder though if I am slowly becoming deprecated by fully automated cloud computing-based solutions (like RightScale, 3Tera, Mosso, Elastra and even DIY Scalr)

A co-worker and I were discussing the lack of languages in Google's AppEngine program. I'd want PHP, he'd probably prefer .NET... there seems to be a lot of "please add XYZ!" in the bug tracker for it.

Anyway, I have a sneaking suspicion that this is a covert operation for Google to provide it's preferred architecture, authentication, etc. so people create the next generation of massively scalable sites (and/or convert their existing ones or create clones of existing ones) that run on their platform... and then Google can swoop in and buy them. The total headache and cost of bringing it in house is essentially nil, since there are no user conversions, data conversions or anything else to deal with. Just slap Google on it, and viola.

I could be wrong, and I suppose we will see as Google starts addressing these requests. Perhaps they'll be fine with -any- language and still want people to make apps they can easily consume. I guess people will report out as they begin developing and growing their sites and Google approaches them (or not...)

That's right. Someone's actually got it going. I just discovered this within the last hour, and I am already trying to figure out how I can accelerate it's growth and adoption.

http://php-fpm.anight.org/

I've already pledged some money and anything else I can help with. Obviously getting an English version up would be great. I have to send a big thanks out there to tony2001 from the PHP project for pointing out his friend's project.

Hopefully in the next week or two I can help fast track this into an English version and show some configuration examples. Famous last words right? I barely update this already... well, I assure you fans, while I may not update my blog that often, I am still alive and kickin and always trying to knock down technical hurdles. I just never get back to the site to share with the world...

I believe this "product" (if you may) will help optimize PHP/FastCGI resources and possibly let some people run setuid/setgid processes and lessen the impact of exploited PHP scripts, not to mention for FastCGI-based hosting it makes life so much simpler. In fact, this has some added optimizations which might accelerate normal FastCGI behavior in PHP too. There's only one feature missing from the list (adaptive process spawning) that I am looking for, otherwise, from what the features show, it's got everything I am looking for and more. Let's get more attention on this project and make it a priority to get into the PHP core...

I've had two requests now for MySQLi updates, so here ya go. It's pretty simple to do it yourself - just look for any mysql_ string, replace it to mysqli_ (where applicable) and in some instances, change the order of the parameters.

Below is the version of wp-includes/wp-db.php for production WP 2.3.3

1. Download the file here
2. Copy it over the existing wp-includes/wp-db.php file
3. Enjoy!

Below is a version that works against the latest in SVN trunk (right now revision 7152) using the new db.php override feature (apparently still not in 2.3.x)

1. Download the file here
2. Copy it to wp-content/db.php (important - this is a hard-coded location)
3. Enjoy!

Let me know if there are any issues, but in my quick little sandbox I made it worked fine...