The Life and Times of Michael Shadle

When examining the PHP-FPM configuration, I realized that I only tweak a few key pieces per pool, so I decided to share my approach to minimize redundancy and keep things simple (more files, but simpler to manage)

Because I still have to maintain PHP 5.2.x for clients, I have decided to try building everything in self-contained in an /opt/php53 directory. So consider that my $prefix, and change appropriately.

I have the following setup, and it seems to work great so far.

/opt/php53/etc/php-fpm.conf:

[global]
log_level = notice
error_log = /opt/php53/var/log/php-fpm.log
pid = /opt/php53/var/run/php-fpm.pid
emergency_restart_threshold = 10
emergency_restart_interval = 1m
process_control_timeout = 5s
daemonize = yes

; pools
include = /opt/php53/etc/fpm.d/pools/*.conf

One file per pool, for example, a pool named "mike" -

/opt/php53/etc/fpm.d/pools/mike.conf:

[mike]
listen = 127.0.0.1:11000
user = mike
group = mike
request_slowlog_timeout = 5s
slowlog = /opt/php53/var/log/slowlog-mike.log
pm.max_children = 5
pm.start_servers = 3
pm.min_spare_servers = 2
pm.max_spare_servers = 4
pm.max_requests = 500
include = /opt/php53/etc/fpm.d/common.conf

Common elements for each pool (if these could be inherited globally, which they MIGHT be, I could just toss them in the main php-fpm.conf. Perhaps a feature request. Will post on the mailing list...)

Remember that rlimit_files needs to be something set in your sysctl.conf or on the system level or you'll get that RLIMIT_NOFILE warning. Also, depending on how you want to limit resources per pool/client, you may want to tweak things, such as request_terminate_timeout.

/opt/php53/etc/fpm.d/common.conf:

listen.backlog = -1
listen.allowed_clients = 127.0.0.1
pm = dynamic
pm.status_path = /status
ping.path = /ping
ping.response = pong
request_terminate_timeout = 120s
rlimit_files = 131072
rlimit_core = unlimited
catch_workers_output = yes
env[HOSTNAME] = $HOSTNAME
env[PATH] = /bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/sbin:/usr/sbin:/opt/php53/bin:/opt/php53/sbin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp

As always, YMMV.

Someone finally published an article that caught my eye about this, so I thought I'd add my two cents.

Summary in my own words:

Never pay more than $5/foot for an HDMI cable at big box places like Best Buy, etc.

The best places to go are small shops around town - those are roughly $1/foot. There's a handful of places around me that offer plenty of options - length, color, etc. Obviously the length-to-cost ratio is different based on the length (I see some that are less than $1/foot, and some that are more than $1/foot, but all pretty close.)

When looking at an HDMI cable all you need to see is "High Speed" or 10.2Gbps on the packaging. Don't look for "Hz", "4K" or anything else. Look for either "10.2 Gbps" or "High Speed" and then find the cheapest cable for the length you need.

As it says, quality of manufacture can be an issue - but like any product, that is always a possibility. However, most cables should have absolutely no issue.

This goes for Ethernet cables too - you shouldn't have to pay more than $1/foot anymore. Even for cat6. I'm looking at a local place with 50 foot cat6 for $15. 75 foot cat5e for $11. Don't get ripped off!

It's no secret I'm a fan of nginx. Probably no secret that I'm a fan of front controller pattern design as well.

Why? I find it to be the most universal and easy to support. I create things from scratch using it, and the most popular software out there uses it - Drupal, Joomla, WordPress - and frameworks including the Zend Framework.

A lot of people cook up crazy nginx configurations for this stuff, when all it really takes for at least a standard setup is a single line of configuration code. However, I just found out something that was sitting in front of me this entire time.

BAD: I used to advise people to use this:

try_files $uri $uri/ /index.php?page=$uri$is_args$args;

BETTER: When actually, even simpler, you can use this:

try_files $uri $uri/ /index.php?page=$request_uri;

So my whole blog post here is slightly wrong. If you use $request_uri, you don't have to remember the $args. It's already there. 🙂

Note: different packages have different parameters. CMS Made Simple uses "page", Drupal uses "q", WordPress uses "q" - some software, like WordPress, doesn't technically need anything, they can fall back by reading PATH_INFO or REQUEST_URI, and not a parameter.

In which case, all you need is:

try_files $uri $uri/ /index.php;

Don't care about directories? Drop the "$uri/" too. Save some stat calls - if you don't need em 🙂

I've told this story countless times, but I've never publicly documented it. This happened over a year ago, but I feel obligated to share it. nginx is the main reason for the success and deserves bragging rights.

We had an upcoming product release, and another team was handling all the communication, content, etc. I was simply just the "managed server" administrator type person. Helped setup nginx, PHP-FPM, configure packages on the server, stuff like that.

The other team managed the web applications and such. I had no clue what we would be in for that day.

There is a phrase, "too many eggs in one basket" - well, this was our first system and we were using it for many things, it hadn't gotten to the point where we needed to split it up or worry about scaling. Until this day, of course.

To me, the planning could have been a bit better, with the ISO image we were providing from the origin being staged on other mirrors, using torrents to offload traffic to P2P, etc. However, that wasn't done proactively, only reactively.

The story

Launch day occurs. I get an email about someone unable to SSH to a specific IP. I check it out - the IP is unreachable, but the server is up. I submit a ticket to the provider, and they let me know why - apparently the server had triggered their DDoS mitigation system once it hit something like ~500mbit; it was automatically flagged as probably being under attack, and the IP was blackholed.

Once we informed them this was all legit, they instantly worked to put us onto a switch that was not DDoS protected, and we resumed taking traffic. This was all done within 15 or 20 minutes, if I recall. I've never seen anything so smoothly handled - no IP changes, completely transparent to us.

I believe the pocket of no traffic (seen in the graph below) was when we were moved off their normal network monitoring and into a Cisco Guard setup. We were definately caught off guard; like I said, I knew we'd get some traffic, but not filling the entire gigabit port. Not a lot of providers would be so flexible about this and handle it with such grace. There are some reports of it being slow, and that is literally because the server itself has too much going on. PHP is trying to handle all the Drupal traffic, and during the night, the disk I/O was at 100% for a long period of time. Oh yeah - since this was the origin, the servers mirroring us had to start pulling stuff from us too 🙂

Luckily we're billed by the gigabyte there, not by 95th like most places, or this would be one heck of a hosting bill. We wound up able to reroute the bandwidth fast enough to not even be charged ANY overage for it!

All in all, without nginx in the mix, I doubt this server would have been able to take the pounding. There was no reverse proxy cache, no Varnish, no Squid, nothing of that nature. I am not sure Drupal was even setup to use any memory caching, and I don't believe memcached was available. There were a LOT of options to reduce load - the main one was just cutting down the bandwidth usage to open up space in the pipe, which was eventually done by removing the ISO file off the origin server, and pushing it to a couple mirror sites. Things calmed down then.

However, it attests to how amazinly efficient nginx is - the entire experience wound up taking only 60 something megabytes of RAM for nginx.

Want the details? See below.

The hardware

• Intel Xeon 3220 (single CPU, quad-core)
• 4GB RAM
• single 7200RPM 250GB SATA disk, no RAID, no LVM, nothing
• Gigabit networking to the rack, with a 10GbE uplink

The software - to the best of my memory (remember, all on a single box!)

• nginx (probably 0.7.x at that point)
  • proxying PHP requests to PHP-FPM with PHP 5.2.x (using the patch)
    • Drupal 6.x - as far as I know, no advanced caching, no memcached, *possibly* APC
  • proxying certain hosts for CGI requests to Apache 2.x (not sure if it was 2.2.x or 2.0.x)
    • mailman - this was a public mail list server
    • Bugzilla
  • This server was also a yum repo for the project, serving through nginx
• PHP-FPM - for Drupal, possibly a couple other apps
• Postfix - the best MTA out there 🙂
  • I believe amavisd/clamav/etc. was involved for mail scanning
  • Integration with mailman, of course
• MySQL - 5.0.x, using MyISAM tables by default, I don't believe things were converted to InnoDB
• rsync - mirrors were pulling using the rsync protocol

The provider

• SoftLayer - they just rock. Not a paid placement. 🙂

The stats

nginx memory usage
During some of that time... only needed 60 megs of physical RAM. 240 megs including virtual. At 2200+ concurrent connections... eat that, Apache.

root     13023  0.0  0.0  39684  2144 ?        Ss   03:56   0:00 nginx: master process /usr/sbin/nginx
www-data 13024  2.0  0.3  50148 14464 ?        D    03:56   9:30 nginx: worker process
www-data 13025  1.1  0.3  51052 15256 ?        D    03:56   5:38 nginx: worker process
www-data 13026  1.3  0.3  50760 15076 ?        D    03:56   6:13 nginx: worker process
www-data 13027  1.3  0.3  50584 14900 ?        D    03:56   6:22 nginx: worker process

nginx status (taken at some random point)

Active connections: 2258
server accepts handled requests
711389 711389 1483197
Reading: 2 Writing: 2040 Waiting: 216

Bandwidth (taken from the provider's switch)

Exceeded Bits Out: 1001.9 M (Threshold: 500 M)
Auto Manage Method: FCR_BLOCK
Auto Manage Result: SUCCESSFUL

Exceeded Bits Out: 868.1 M (Threshold: 500 M)
Auto Manage Method: FCR_BLOCK
Auto Manage Result: SUCCESSFUL

To give you an idea of the magnitude of growth, this is the amount of gigabytes the server pushes on a normal day:

• 2009-05-18 155.01 GB
• 2009-05-17 127.48 GB
• 2009-05-16 104.21 GB
• 2009-05-15 152.42 GB
• 2009-05-14 160.12 GB
• 2009-05-13 148.6 GB

On launch day and the spillover into the next day:

• 2009-05-19 2036.37 GB
• 2009-05-20 2481.87 GB

The pretty pictures

Click for larger versions!

Hourly traffic graph

(Note: I said 600M, apparently their threshhold from their router says 500M)

Weekly traffic graph

The takeaway

Normally I would never think a server could get slammed with so much while it is having to service so much. Perhaps if it was JUST a PHP/MySQL server, or JUST a static file server, but no - we had two webservers, a mailing list manager, Drupal (which is not the most optimized PHP software), etc. The server remained responsive enough to service requests, on purely commodity hardware.

The world is getting smaller.

• AT&T's got a Femtocell called a "MicroCell"
• PayPal is an example of a "micropayment" service
• Microformats are becoming increasingly popular to add more metadata into websites, mainly for richer machine processing
• MicroATX is one of many small form factors. However, not as small as Nano-ITX, Mini-ITX
• Twitter is the world's most popular "Microblogging" service
• Need to clean something fragile? Microfiber cloths are typically used to clean luxury cars, computer parts, screens, etc.
• Those small tweaks to squeeze out a little bit more performance? Micro-optimizations can be useful or can be a pain. It's up to you to decide what is worth it or not.

In this world of larger cars, larger boats, larger cruise ships, larger meals, just remember this - a lot of things are getting smaller. Typically technology... but I threw in another couple terms I seem to say often.

Update:

I forgot, microexpressions, now being made a household name thanks to the TV show "Lie to Me" (highly recommended show, btw.)

I've seen this before, and it was mentioned earlier today at OSCON, but I never knew if it was a browser behavior or a standard. Looks like I got it with some help from IRC.

Say you have a foreign host and you don't want to have to figure out if you're on http:// or https:// and call their assets appropriately so you don't get a mixed-mode warning. You can actually use a syntax that is defined in RFC 3986, specifically section 4.2:

A relative reference that begins with two slash characters is termed a network-path reference; such references are rarely used. A relative reference that begins with a single slash character is termed an absolute-path reference. A relative reference that does not begin with a slash character is termed a relative-path reference.

Which means you can do this:

<img src="//foo.com/bar.jpg" />

and your browser will request http://foo.com/bar.jpg or https://foo.com/bar.jpg, depending on what scheme your browser is currently on.

I was hesitant at first to consider it "okay" but as it is published in the RFC and Chromium's fixed bugs relating to it, it does appear to be a properly supported method that could save you a few keystrokes. Let me know if it doesn't work for you! Be sure to give browser/OS information and conditions to reproduce.

Oh yeah, and the other host needs to be on https as well, of course. I shouldn't really have to say that, though 🙂

1. Generate the CSR:

   openssl genrsa 2048 > yourhost.com.key
   openssl req -new -key yourhost.com.key > yourhost.com.csr
   

2. Enter in whatever you want - you NEED the "Common Name" everything else is not really required for it to work.

   Country Name (2 letter code) [AU]:US
   State or Province Name (full name) [Some-State]:.
   Locality Name (eg, city) []:.
   Organization Name (eg, company) [Internet Widgits Pty Ltd]:Something Here
   Organizational Unit Name (eg, section) []:.
   Common Name (eg, YOUR name) []:yourhost.com
   Email Address []:.
   
   Please enter the following 'extra' attributes
   to be sent with your certificate request
   A challenge password []:
   An optional company name []:
   

3. Paste the CSR into Go Daddy, get back the .crt file
4. Combine the cert + Go Daddy chain:

   cat yourhost.com.crt gd_bundle.crt > yourhost.com.pem

5. Lastly, in nginx.conf:

   ssl_certificate /etc/nginx/certs/yourhost.com.pem;
   ssl_certificate_key /etc/nginx/certs/yourhost.com.key;
   

Additionally I have these SSL tweaks which seems to maintain a better SSL experience, passes McAfee Secure's SSL checks, etc.:

ssl on;
ssl_protocols SSLv3 TLSv1;
ssl_ciphers ALL:-ADH:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP;
ssl_session_cache shared:SSL:10m;

Might not be the most advanced, but hey, it works. You just need to alter the mac address and the display for each machine. I'm running this on Ubuntu 10.04 (Lucid) and it seems to work great.

/etc/init/my-kvm.conf:

description     "my-kvm"

start on (net-device-up
   and local-filesystems)
stop on runlevel [016]

respawn
exec /usr/bin/kvm -hda /root/virtual-machines/my-kvm.bin -no-acpi -m 128 -net nic,macaddr=DE:AD:BE:EF:18:12 -net tap -vnc :0

Enjoy.

Officially in the 5.3.3RC1 distribution. Sweet! From the NEWS file:

17 Jun 2010, PHP 5.3.3 RC1
...
- Added FastCGI Process Manager (FPM) SAPI. (Tony)
...

and on an unrelated note:

- Added support for JSON_NUMERIC_CHECK option in json_encode() that converts numeric strings to integers. (Ilia)

Shouldn't this be called JSON_NUMERIC_CONVERT? or JSON_FORCE_INTEGER? It's not just a "check" - guess it's too late now? 🙂

WordPress 3.0...
WordPress 3.0 came out today. Tonight I'll probably upgrade this site and see how well it works. I'm going to check it in to Subversion first so I can roll back if needed.

Some key changes I wanted to talk about...

• One thing that was highlighted is the option to use custom header images - which can easily be done right now. I did it well over a year ago in a theme. With post meta you can always load metadata about a post and use it in the theme, so this update seems a bit specific to me, since themes were already customizable. Why build a feature that is so specific? Same with background images/colors...
• Custom menus/menu editor - this could get cool, the menu editor is the more exciting piece as it will allow a visual way to manage the taxonomy. Not sure how it will mix in with tags and categories though, guess that's "I'll see it when I upgrade."
• MU merge - finally, I can run multiple installs off the same WP install, hopefully, without wp-config.php hacks. How exactly it works I will have to find out.
• Custom post types - now all of a sudden you can make any sort of object with custom attributes, which opens the door to things such as the item below.
• WP e-Commerce says they're going to change from using all their extra tables to using core WP schema. That's awesome.

A couple bones to pick...

• It's not a rewrite. It's still a blogging tool that is being extended further to be a full-featured that can handle "anything" - however the tables are still named "posts" even though now you can create an arbitrary type of item. I'd like to see it renamed and normalized.
• All the plugins and themes and such are procedural code, but some inner workings such as the DB layer are OO. That seems amateur to me, and unnecessary.

I'd love to see WP get rewritten. It has a LOT of overhead and includes built in that need calling and a lot of other cruft that I stumble across. Go back to the drawing board with building a list of every feature it has, and look at it from a longer term perspective. It's great to see something keep growing, but when it comes down to it, it is still a fork of b2, which was made for blogging, not for anything and everything.

It's got the right idea with extensibility and such, but to me the core has a lot of code - and lots of code means more complicated execution paths, more "I'll just add this in instead of refactor this old code," more cruft. I'm quite sure I could get as much extensibility out of a fresh rewrite with less than half the code under the hood. Things like text styling for example should be moved to a plugin (I disable all the wptexturize filters for example... throw those in an include and make it enabled by default instead!)

Of course, WordPress does have millions of users so it has a proven track record. I can't complain that much, I do use it myself. For blogging, it's the best tool out there. For other things, it typically leverages plugins which may or may not have decent UIs or APIs to interact with. That's where it shows signs of weakness. It also isn't as strict as Drupal when it comes to code conventions either, which would greatly increase the usability of a lot of plugins.

Today AT&T reported a bunch of data plan changes, perfectly aligned with screwing over people when iPhone OS 4 comes out.

OS 4 will give us tethering ability with AT&T. Of course, that comes with an additional cost now. If you want to "officially" tether with AT&T, you'll have to switch to one of their two data plans, and THEN buy tethering on top of that. Instead of your $30 for unlimited, you'll be paying $45 for only two gigs of bandwidth, vs. $30 currently for unlimited. Or $35 for a measily 250 megs.

They claim that 98% of its users fall within those tiers. Sounds like they're giving 98% of us a way to save a few bucks, right? Wrong. What people are overlooking is the fact that those statistics don't include the fact that you will be TETHERING. As in, your computer will be transferring data, and you're already accustomed to a much heavier data consumption through that medium. Think of how those little widgets on your laptop grab information off the net every so often, or a single Youtube video could easily be 20+ megs. It does add up, and ultimately anyone who wants to tether with AT&T should use it sparsely - otherwise, they'll be paying $10/gig for your overages.

I'll give AT&T some credit - their marketing geniuses and social engineers are gaming this system well - using the upcoming iPhone and OS for service type changes, some of which require contracts, some don't - helping them limit the impact on their network in creative ways while making you think you're getting a better deal. I especially like the fact once you drop out of an unlimited contract you can't go back, and they're trying to sway you out of it by not changing your expiration date if you drop down, and "allowing" you to switch back and forth between the two plans. Oh, and new iPad users? If you don't have a plan by the time OS 4 launches, you'll be subject to these new limits.

Let's face it - AT&T's network can't handle the iPhone. It's a blame game that ultimately both could be doing better at. I'm sure the iPhone could change it's tower-hogging style behavior, and I'm sure AT&T could be deploying all these upgrades they keep telling us about faster.

For me, if I do decide to go with the next iPhone, I won't be buying it with a contract. I'm going to stay a free agent, and not get locked in to a new agreement, and not buy their joke for insurance (monthly fee and still the cost of buying a refurbished unit for the deductable?!) - and I am getting closer every day to switching over to maybe the Evo 4G on Sprint - a hybrid solution with a limited 4g network around the area. Sprint and Verizon seem to have a much more reliable network, and since this device is still primarily a phone, voice calls do count for something. Sprint's at least would afford me faster WiMAX when I could get it, and openly promote the device's ability to share the connection with up to 8? devices.

I haven't investigated the fees, but when I can't go a single day without a dropped call, and I try to find landlines to fall back on where I can instead of my phone, something has got to change.