The Life and Times of Michael Shadle

It's no secret I'm a fan of nginx. Probably no secret that I'm a fan of front controller pattern design as well.

Why? I find it to be the most universal and easy to support. I create things from scratch using it, and the most popular software out there uses it - Drupal, Joomla, WordPress - and frameworks including the Zend Framework.

A lot of people cook up crazy nginx configurations for this stuff, when all it really takes for at least a standard setup is a single line of configuration code. However, I just found out something that was sitting in front of me this entire time.

BAD: I used to advise people to use this:

try_files $uri $uri/ /index.php?page=$uri$is_args$args;

BETTER: When actually, even simpler, you can use this:

try_files $uri $uri/ /index.php?page=$request_uri;

So my whole blog post here is slightly wrong. If you use $request_uri, you don't have to remember the $args. It's already there.

Note: different packages have different parameters. CMS Made Simple uses "page", Drupal uses "q", WordPress uses "q" - some software, like WordPress, doesn't technically need anything, they can fall back by reading PATH_INFO or REQUEST_URI, and not a parameter.

In which case, all you need is:

try_files $uri $uri/ /index.php;

Don't care about directories? Drop the "$uri/" too. Save some stat calls - if you don't need em

I've told this story countless times, but I've never publicly documented it. This happened over a year ago, but I feel obligated to share it. nginx is the main reason for the success and deserves bragging rights.

We had an upcoming product release, and another team was handling all the communication, content, etc. I was simply just the "managed server" administrator type person. Helped setup nginx, PHP-FPM, configure packages on the server, stuff like that.

The other team managed the web applications and such. I had no clue what we would be in for that day.

There is a phrase, "too many eggs in one basket" - well, this was our first system and we were using it for many things, it hadn't gotten to the point where we needed to split it up or worry about scaling. Until this day, of course.

To me, the planning could have been a bit better, with the ISO image we were providing from the origin being staged on other mirrors, using torrents to offload traffic to P2P, etc. However, that wasn't done proactively, only reactively.

The story

Launch day occurs. I get an email about someone unable to SSH to a specific IP. I check it out - the IP is unreachable, but the server is up. I submit a ticket to the provider, and they let me know why - apparently the server had triggered their DDoS mitigation system once it hit something like ~500mbit; it was automatically flagged as probably being under attack, and the IP was blackholed.

Once we informed them this was all legit, they instantly worked to put us onto a switch that was not DDoS protected, and we resumed taking traffic. This was all done within 15 or 20 minutes, if I recall. I've never seen anything so smoothly handled - no IP changes, completely transparent to us.

I believe the pocket of no traffic (seen in the graph below) was when we were moved off their normal network monitoring and into a Cisco Guard setup. We were definately caught off guard; like I said, I knew we'd get some traffic, but not filling the entire gigabit port. Not a lot of providers would be so flexible about this and handle it with such grace. There are some reports of it being slow, and that is literally because the server itself has too much going on. PHP is trying to handle all the Drupal traffic, and during the night, the disk I/O was at 100% for a long period of time. Oh yeah - since this was the origin, the servers mirroring us had to start pulling stuff from us too

Luckily we're billed by the gigabyte there, not by 95th like most places, or this would be one heck of a hosting bill. We wound up able to reroute the bandwidth fast enough to not even be charged ANY overage for it!

All in all, without nginx in the mix, I doubt this server would have been able to take the pounding. There was no reverse proxy cache, no Varnish, no Squid, nothing of that nature. I am not sure Drupal was even setup to use any memory caching, and I don't believe memcached was available. There were a LOT of options to reduce load - the main one was just cutting down the bandwidth usage to open up space in the pipe, which was eventually done by removing the ISO file off the origin server, and pushing it to a couple mirror sites. Things calmed down then.

However, it attests to how amazinly efficient nginx is - the entire experience wound up taking only 60 something megabytes of RAM for nginx.

Want the details? See below.

The hardware

• Intel Xeon 3220 (single CPU, quad-core)
• 4GB RAM
• single 7200RPM 250GB SATA disk, no RAID, no LVM, nothing
• Gigabit networking to the rack, with a 10GbE uplink

The software - to the best of my memory (remember, all on a single box!)

• nginx (probably 0.7.x at that point)
  • proxying PHP requests to PHP-FPM with PHP 5.2.x (using the patch)
    • Drupal 6.x - as far as I know, no advanced caching, no memcached, *possibly* APC
  • proxying certain hosts for CGI requests to Apache 2.x (not sure if it was 2.2.x or 2.0.x)
    • mailman - this was a public mail list server
    • Bugzilla
  • This server was also a yum repo for the project, serving through nginx
• PHP-FPM - for Drupal, possibly a couple other apps
• Postfix - the best MTA out there
  • I believe amavisd/clamav/etc. was involved for mail scanning
  • Integration with mailman, of course
• MySQL - 5.0.x, using MyISAM tables by default, I don't believe things were converted to InnoDB
• rsync - mirrors were pulling using the rsync protocol

The provider

• SoftLayer - they just rock. Not a paid placement.

The stats

nginx memory usage
During some of that time... only needed 60 megs of physical RAM. 240 megs including virtual. At 2200+ concurrent connections... eat that, Apache.

root     13023  0.0  0.0  39684  2144 ?        Ss   03:56   0:00 nginx: master process /usr/sbin/nginx
www-data 13024  2.0  0.3  50148 14464 ?        D    03:56   9:30 nginx: worker process
www-data 13025  1.1  0.3  51052 15256 ?        D    03:56   5:38 nginx: worker process
www-data 13026  1.3  0.3  50760 15076 ?        D    03:56   6:13 nginx: worker process
www-data 13027  1.3  0.3  50584 14900 ?        D    03:56   6:22 nginx: worker process

nginx status (taken at some random point)

Active connections: 2258
server accepts handled requests
711389 711389 1483197
Reading: 2 Writing: 2040 Waiting: 216

Bandwidth (taken from the provider's switch)

Exceeded Bits Out: 1001.9 M (Threshold: 500 M)
Auto Manage Method: FCR_BLOCK
Auto Manage Result: SUCCESSFUL

Exceeded Bits Out: 868.1 M (Threshold: 500 M)
Auto Manage Method: FCR_BLOCK
Auto Manage Result: SUCCESSFUL

To give you an idea of the magnitude of growth, this is the amount of gigabytes the server pushes on a normal day:

• 2009-05-18 155.01 GB
• 2009-05-17 127.48 GB
• 2009-05-16 104.21 GB
• 2009-05-15 152.42 GB
• 2009-05-14 160.12 GB
• 2009-05-13 148.6 GB

On launch day and the spillover into the next day:

• 2009-05-19 2036.37 GB
• 2009-05-20 2481.87 GB

The pretty pictures

Click for larger versions!

Hourly traffic graph

(Note: I said 600M, apparently their threshhold from their router says 500M)

Weekly traffic graph

The takeaway

Normally I would never think a server could get slammed with so much while it is having to service so much. Perhaps if it was JUST a PHP/MySQL server, or JUST a static file server, but no - we had two webservers, a mailing list manager, Drupal (which is not the most optimized PHP software), etc. The server remained responsive enough to service requests, on purely commodity hardware.

1. Generate the CSR:

   openssl genrsa 2048 > yourhost.com.key
   openssl req -new -key yourhost.com.key > yourhost.com.csr
   

2. Enter in whatever you want - you NEED the "Common Name" everything else is not really required for it to work.

   Country Name (2 letter code) [AU]:US
   State or Province Name (full name) [Some-State]:.
   Locality Name (eg, city) []:.
   Organization Name (eg, company) [Internet Widgits Pty Ltd]:Something Here
   Organizational Unit Name (eg, section) []:.
   Common Name (eg, YOUR name) []:yourhost.com
   Email Address []:.
   
   Please enter the following 'extra' attributes
   to be sent with your certificate request
   A challenge password []:
   An optional company name []:
   

3. Paste the CSR into Go Daddy, get back the .crt file
4. Combine the cert + Go Daddy chain:

   cat yourhost.com.crt gd_bundle.crt > yourhost.com.pem

5. Lastly, in nginx.conf:

   ssl_certificate /etc/nginx/certs/yourhost.com.pem;
   ssl_certificate_key /etc/nginx/certs/yourhost.com.key;
   

Additionally I have these SSL tweaks which seems to maintain a better SSL experience, passes McAfee Secure's SSL checks, etc.:

ssl on;
ssl_protocols SSLv3 TLSv1;
ssl_ciphers ALL:-ADH:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP;
ssl_session_cache shared:SSL:10m;

Upstart is becoming the de-facto standard for everything in Ubuntu, and I do enjoy it's process management and re-spawning capabilities.

(Actually, before PHP-FPM I used to use upstart jobs to su - $user php-cgi -b $port )

These are VERY simple but effective scripts, and would actually be beefed up to be more intelligent (chaining nginx to start after PHP-FPM for example. However, if you do not need PHP, then that's a useless chain. So I kept it simple. I suppose you could add a short delay to start nginx then...)

Note: make sure PHP-FPM is set to daemonized = yes.

Second note: this works for PHP 5.2.x w/ the PHP-FPM patch, on Ubuntu Lucid Lynx (10.04) - anything else YMMV. I am not using PHP-FPM w/ PHP 5.3 yet since I have no environments that I know will support 5.3 code. When I finally get one, I will look for the same opportunity.

/etc/init/nginx.conf:

description "nginx"

start on (net-device-up and local-filesystems)
stop on runlevel [016]

expect fork
respawn
exec /usr/sbin/nginx

/etc/init/php-fpm.conf:

description "PHP FastCGI Process Manager"

start on (net-device-up and local-filesystems)
stop on runlevel [016]

expect fork
respawn
exec /usr/local/bin/php-cgi --fpm --fpm-config /etc/php-fpm.conf

Once you've done this you can remove all the /etc/init.d/php-fpm, /etc/init.d/nginx and /etc/rc?.d/[K,S]??php-fpm and /etc/rc?.d/[K,S]??nginx symlinks and files. This takes care of all of that.

Feel free to comment and leave better tips, tricks and ideas!

I've been posting on the nginx mailing lists for a while that I've had a developer working on SPNEGO (Kerberos/GSSAPI/etc.) module for authentication for nginx. I never produced any code though, because I was constantly waiting for a bit more of a matured module before giving it out.

For now though, it might just be best to throw out there what I have, explain my findings, and let the community start testing it, hacking it, improving it, etc.

Download
http://michaelshadle.com/media/ngx_http_auth_sso_module-0.3.zip

Opens for the developer still

• He said he'd like to remove the dependency on the bundled spnegohelp library (apparently it's not needed or it can be filled with a system package)
• Needs some more documentation

My questions, comments

• Should it be called "mod_auth_sso" or something like "mod_auth_gssapi" - I believe Apache's equivalent has "gssapi" in the title somewhere. It was hard to determine which was the most up to date version - mod_auth_kerb, modgssapache, etc.
• I cannot verify this still using my corporate domain setup. I did have to make a minor tweak in the source to change the principal name it was using (see below) and I still got access denied. I have no clue if the module is to blame, the machine's setup with the domain, or what.

A possible required tweak... around line 474 or so in ngx_http_auth_sso_module.c

- host_name = r->headers_in.host->value;
+ host_name = alcf->realm;

How to compile - yes, it's a bit messy, mainly due to the spnegohelp library dependency

wget http://sysoev.ru/nginx/nginx-0.8.31.tar.gz
tar xvfz nginx-0.8.31.tar.gz
cd nginx-0.8.31
./configure --conf-path=/etc/nginx/nginx.conf --prefix=/usr --user=www-data --group=www-data --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-http_stub_status_module --with-http_gzip_static_module --without-mail_pop3_module --without-mail_smtp_module --without-mail_imap_module --with-http_flv_module --with-http_ssl_module --with-http_dav_module --with-http_realip_module --with-http_xslt_module --with-debug --add-module=/usr/src/build/ngx_http_auth_sso_module-current --with-ld-opt="-L/usr/src/build/ngx_http_auth_sso_module-current/spnegohelp"
cp -f /usr/src/build/ngx_http_auth_sso_module-0.3/spnegohelp/libspnegohelp.so /usr/lib64/libspnegohelp.so
... make, make install, whatever ...

I copied the libspnegohelp.so into the system so nginx can use it without any special runtime LD_LIBRARY_PATH crap.

How to configure (again, I could not validate this 100%)

location /test {
   auth_gss on;
   auth_gss_realm YOUR.KERBEROS.REALM;
   auth_gss_keytab /etc/krb5.keytab;
   auth_gss_service_name YOURMACHINENAMEIBELIEVE;
}

Conclusion
Your mileage will definitely vary but hopefully some people with more experience with Kerberos, C, nginx modules, or anything else helpful can pick it apart. I will post updates as I get them and I do want to post this on github or something... unless someone else wants to take ownership over it who will actually actively maintain/keep it up to date with nginx internals and maintain the github/whatever repository for it.

This was self-funded personally with no company sponsorship or anything, part of the terms of the project were also to keep it BSD licensed. Feel free to do whatever you want with it. If you want to send me any money to reimburse, I'll never say no to that - paypal AT mike2k.com. Or, pay a developer who can ensure it works end-to-end, if there is something missing.

Sadly, it does require the machine to be on the domain, I was hoping I could get away with it not having to be on the domain. I confirmed with Sam Hartman, who was a chief technologist at the MIT Kerberos Consortium - you can't really get more knowledgeable than that. I would have hired Sam for the project but it would have been too expensive. However if a company is willing to take this initial coding and have Sam add his magic to it, he knows C and was able to give me a real quick estimate and check how nginx modules work. He could possibly do it for cheaper now since the initial work might be done, and he would definitely be able to confirm all the logic is intact.

Thanks to YoctoPetaBorg at RentACoder.com for the initial work (and for hopefully soon finishing up the last bits of this :p) - it will be exciting to be able to use this at work and be able to have a fully functional module out there that people find useful.

You can probably also use this for just Django + nginx on Ubuntu too, it should work the same for any Django-based package, just change the Transifex-specific stuff (like /site_media and such.)

Dependencies:

• python-django package from Ubuntu (currently Jaunty's 1.0.2-1ubuntu0.1)
• Transifex installed to /home/transifex/web/transifex, from source
• Transifex setup to answer over fastcgi on port 8080
• Tested on nginx 0.7.x and 0.8.x compiled from source

/etc/nginx/nginx.conf:

server {
   listen 80;
   server_name foo.com;
   include /etc/nginx/confs/defaults.conf;
   location / {
      fastcgi_pass 127.0.0.1:8080;
      include /etc/nginx/confs/django.conf;
   }
   location /site_media {
      root /home/transifex/web/transifex;
      include /etc/nginx/confs/expires.conf;
   }
   location /media {
      root /usr/share/python-support/python-django/django/contrib;
      include /etc/nginx/confs/expires.conf;
   }
}

/etc/nginx/confs/defaults.conf:

location ~ /\.ht {
   deny all;
}

location ~ \.svn {
   deny all;
}

log_not_found off;
server_name_in_redirect off;

/etc/nginx/confs/django.conf:

# http://www.rkblog.rk.edu.pl/w/p/django-nginx/
# http://code.djangoproject.com/wiki/ServerArrangements
fastcgi_param PATH_INFO $fastcgi_script_name;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param SERVER_NAME $server_name;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_pass_header Authorization;
fastcgi_intercept_errors off;

/etc/nginx/confs/expires.conf:

location ~* \.(jpg|jpeg|gif|css|png|js|ico|html)$ {
   expires max;
   access_log off;
}

August 2010 UPDATE: Read here for new information!

I just realized when trying to use the try_files shortcut that either the behavior has changed or the code I've used it for has never had to take advantage of it, but what happens when you do something like this:

try_files $uri $uri/ /foo/index.php?u=$uri;

Is that the query string is stripped out and the only thing PHP sees is the $_GET['u']

to fix this, it's easy enough:

try_files $uri $uri/ /foo/index.php?u=$uri&$args;

I never noticed this behavior in WordPress (I've been using try_files with it for a while without noticable issue) but it was an issue with Drupal (which uses the Front-Controller Pattern just like WP does) and custom code we've written.

UPDATE: depending on what custom WordPress plugins you may be using that rely on the $_GET params, you should add this. This should be the final recipe from what I can tell.

I haven't asked Igor if the behavior has changed slightly with it, but the changelogs have not mentioned any behavior changes, from what I can tell.

Anyway if you notice your pagination or other GET params are being dropped, you might want to take a quick look at that

OLD:

error_page 404 = /wordpress/index.php?q=$request_uri;

or:

if (!-e $request_filename) {
   rewrite ^/(.*) /wordpress/index.php?uri=$request_uri last;
}

NEW:

try_files $uri $uri/ /wordpress/index.php?q=$uri&$args;

Why make life more complicated if you don't need to?

Also thanks to Igor's patch you can have multiple of these, i.e.:

location /wordpress {
   try_files $uri $uri/ /wordpress/index.php?q=$uri&$args;
}

location /anotherapp {
   try_files $uri $uri/ /anotherapp/controllerfile.php?q=$uri&$args;
}

etc.

This is good, as Drupal, WordPress and many other packages (including anything I develop anymore) use the Front Controller pattern of development, which makes deploying applications a lot simpler and helps support a consistent framework across the entire application.

NOTE: Updated on 1/5/10 to include the $args

Self-explanatory. nginx surpasses Lighttpd and moves up to #4 according to Netcraft. Woohoo!

http://survey.netcraft.com/Reports/200812/

Typically, updates on the open source packages work without a hitch. However, my upgrade last weekend on my servers from Ubuntu Hardy to Intrepid wound up creating a couple major headaches, and at the same time, I noticed a handful of other snafoos happening to open source packages I use daily.

This wound up in server instability, client annoyance, and 20-30 hours solid of trial-and-error compiling, testing, debugging, etc. Even right now, if I forget to hold back the libgpac-dev package from being updated, all videos being converted lose their sound due to MP4Box crashing.

• NFS broke - on only *one* of six identically configured machines. For the time being, running 2.6.24-16-server has been my only workaround. Trying all three kernels in the Ubuntu repository (2.6.27-7-server, 2.6.27-10-server, and 2.6.28-2-server) all suffered from this issue. Sadly, could not provide much debugging or relevant information to anyone, the machine is in production and I've got nothing in logs to go off of. Five of the six servers are running the newer kernels without an issue, which is what is odd.
• ffmpeg broke resampling of a common audio type - oddly enough, this can be fixed by issuing two separate ffmpeg commands. There is a patch on the mailing list, but still not implemented into SVN.
• MP4Box no longer works on Ubuntu, due to an issue with libgpac-dev - still no fix available, have to force using an older libgpac-dev version.
• nginx releases 0.7.25, a new feature in 0.7.25 had a bug, 0.7.26 comes out - luckily, Igor's fix was quick and effective, before I even had a chance to run 0.7.25.
• PHP 5.2.7 comes out with a somewhat critical "security" issue (magic quotes was broken), has to release 5.2.8 shortly after - I have to wait for Suhosin and PHP-FPM patches before I can adopt a new version of PHP; otherwise, this may have caused some headache for me.
• MySQL 5.1 comes out with crashing bugs - I stick to using MySQL from the standard Ubuntu repositories, which usually has a decent lag while seeking "stability" - thank God for that.