The Life and Times of Michael Shadle

I dislike wikis. At work we were using them. Now we're changing from wikis to an article management system. The idea on paper sounded great, but I'm realizing the pitfalls about the implementation.

There are some traits that wikis have that I like. There's also plenty of bad traits. Also, I'm thinking of MediaWiki when I say this stuff.

Pros:

• Interlinking and updating
• Anyone can edit pages by design
• Templates and categorization can be done by anyone, inline
• Every page has a discussion option

Cons:

• Lots of overhead
• Semi-proprietary syntax (= makes h1, == makes h2, * makes one level lists, ** makes it a two level list, etc.)
• Inconsistent syntax - some HTML works, some bbcode-ish wiki stuff works, some __HEREDOC__ stuff works
• Forces everything to be CamelCase
• No attachments per page, only global
• By default, wikis have no page-level security, since their mantra is "anyone can edit"

So now that I've said that - I have to say that the ideal approach would be to take the existing CMS approach but add in a few wiki features. Specifically the interlinking. We already have a comment feature on every page. Attachment management is per-page too.

Late night half dazed thoughts:

Link syntax couldn't be something like foo:bar - since javascript:foo would conflict. Perhaps something like [foo:bar] would be a good idea? For links to file attachments, [file:fileID] or [article:slug-title]. This would make life simpler for tracking what files are orphans, can display information about the file in-line in the documents (file type, size, etc.)

One thing most places forget to mention, at least clearly, is the need to update the LU tools before you run the upgrade.

This assumes you have the DVD .iso already on the system, and you are mounting it to a directory (in this example, /export/loop)

First update the LU tools off the new disc. This process wasn't really included in a lot of blogs I read:

mount -F hsfs /path/to/sol-nv-b101-x86-dvd.iso /export/loop
cd /export/loop/Solaris_11/Tools/Installers
./liveupgrade20

*Now* you can run all the LU commands:

lucreate -p rpool -n snv_101
luupgrade -u -n snv_101 -s /export/loop
luactivate snv_101
init 6
shutdown -i6 -g0 -y  (if init 6 doesn't reboot - maybe I was impatient)

P.S. Don't put the .iso on your root pool or it will be included in the filesystem forever and you won't be able to get that space back. Thanks Tim 🙂

I comment on a lot of random sites and blogs. I usually write down the URL in a text file and hope that someday I'll remember to return to see if there's a reply.

I believe a service exists - I know that "page change notification" services exist, but I'd rather be able to subscribe to an RSS feed of reply notifications - i.e. I post on a blog, I dump the URL into a textbox, and it monitors the site every so often for changes (probably the specific post's comment feed is the only thing that is worthwhile) - even better, someday maybe even automatically adding the URL when the browser notices I've commented.

Anyone got a site out there that does this yet? I've had this on my mental radar for a while. Of course, that doesn't really do anything if I can never get around to finishing anything.

In my mind, Drizzle is shaping up to be the next hottest thing. As I can't help the project directly with code (C/C++ isn't my forte, no matter how hard I try), I'm trying to help in other ways. One of the ways is helping Drizzle establish a solid web presence and providing them with anything else I can give in support.

First off, I hunted for a proper domain name - drizzle.org. It wasn't "in use" per-se, it was parked, and I approached the owner. After some negotiation I was able to get it down to a number that didn't give the team much heartburn. Prior to announcing this publicly which could interfere with the transaction, I fronted the money myself and I am now working on getting the money back from donations. A lot has already been pledged by #drizzle on freenode. If you'd like to contribute, please feel free to send any amount via Paypal to "paypal AT mike2k.com"

I will keep the domain in my possession safely until the Drizzle Foundation is created (I believe it is in the works) or one of the core team has a safe haven for it. In the meantime, if it's desired we can start developing/working on a website, etc.

The goal is to raise roughly $1000 USD to cover the domain + Escrow costs. I've already said I would contribute a chunk of that. Please include in the PayPal description your full name/company/whatever identifying information you'd like and if you'd like it recorded, and I will record it and if the Drizzle guys wish, we can post your info on the [not established yet] website as a Drizzle supporter.

Note that this is not tax deductable or anything as there isn't an official foundation yet, and I am not sure the foundation will be a proper 501(c) anyway. This is purely an announcement that the domain has been purchased and a chance for the community to donate (or otherwise reimburse me, which will help since next month is property tax month :))

I just changed the table prefix from "wp_" to "wp_en_us_" to support multiple installs in the same database, one for each locale. This seemed pretty straightforward - just rename the table names and change the table prefix in wp-config.php, right?

Wrong.

The WP code is littered with references to $table_prefix - including columns in the "usermeta" and "options" table. Key columns such as "wp_capabilities" in the "options" table need to be renamed appropriately, or your user roles are totally broken. In "usermeta" you need to make sure the "wp_user_level" and "wp_capabilities" meta_keys are updated too.

Sadly, after spending a couple hours var_dump() and exit()'ing throughout the code I realized the issue. I am not exactly sure the prefix of some of these keys needs to be aligned with the $table_prefix - maybe WPMU uses that or something. But anyway, what an annoying headache.

I am not sure but I think -all- "wp_" keys might need this same treatment. I've only really cared about the user roles and such since I had a lot of users trying to figure out why they can't post anymore. At least that's fixed.

Note: I don't know if this is clearly documented anywhere. If it is, I should be embarrased. However, I still want to voice a complaint that this isn't very straightforward. Maybe put a comment in wp-config.php?

Today I received an email to an OLD email alias I had used in the late 90's for one of those browser bars that pays you to surf (Spedia) - that address has obviously been resold to various people and companies, but this is pretty nuts. The marketeers behind the McCain/Palin campaign have actually purchased a list that includes that email alias, which means they're not really sending out emails to so-called "targeted" folks. The email alias I had registered has no context related to it; they're just sending out emails to anyone on a list they've purchased.

Pretty weak. I checked the headers too. This is legitimately from his campaign email marketing!

Delivered-To: [email protected]
Received: by 10.142.104.5 with SMTP id b5cs104203wfc;
        Wed, 17 Sep 2008 17:02:21 -0700 (PDT)
Received: by 10.150.97.19 with SMTP id u19mr247318ybb.24.1221696140048;
        Wed, 17 Sep 2008 17:02:20 -0700 (PDT)
Return-Path: <[email protected]>
Received: from sm1.johnmccain.com (sm1.johnmccain.com [64.203.105.81])
        by mx.google.com with ESMTP id 6si22813870ywn.0.2008.09.17.17.02.19;
        Wed, 17 Sep 2008 17:02:20 -0700 (PDT)
Received-SPF: pass (google.com: domain of [email protected] designates 64.203.105.81 as permitted sender) client-ip=64.203.105.81;
Authentication-Results: mx.google.com; spf=pass (google.com: domain of [email protected] designates 64.203.105.81 as permitted sender) [email protected]
Received: from unknown (unknown [192.168.201.181])
	by sm1.johnmccain.com (Postfix) with QMQP id 4A6E3490F6
	for <[email protected]>; Wed, 17 Sep 2008 20:02:19 -0400 (EDT)
Errors-To: <[email protected]>
X-Bounce-Track: <[email protected]>
From: "McCain Palin 2008" <[email protected]>
To: <[email protected]>
Subject: Get Your Oregon Absentee Ballot
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_5A3_E4DD8AB6.47E75371"
Message-Id: <[email protected]>
Date: Wed, 17 Sep 2008 20:02:19 -0400 (EDT)

The text of the email?

Your vote in this election is more critical than ever as the polls show John McCain and Governor Palin in a virtual tie. Voting early by absentee ballot or in person is the best way to get your vote counted now and avoid the long lines on Election Day.

... more stuff hoping to rope in absentee votes for their party ...

P.S. If you are not yet registered to vote, please register right away by following this link. [link to johnmccain.com]

...

Please visit this page if you want to remove yourself from the email list. [link to johnmccain.com]

...

Paid for by McCain-Palin 2008

It even has a proper SPF record. So yes, they have officially sent me unsolicited email from an email list sold to them... However, they use Postfix - I'll give them props for that.

Problem statement: HTTP sucks for file uploads.

You know it. I know it. The problems?

• No resuming
• POST multipart/form-data bloats the size of the file due to encoding
• Slow connections typically time out on large files
• Any server resets or any other network "burps" on the path from client to server effectively kills the upload
• People have had moderate success by tuning their webserver and PHP to accept large POSTs, and in general, it works - but not for everyone and it suffers from everything previously noted.

What would the ideal file upload experience support?

• It should resume after manual pause, a server reset, a timeout, or any other error.
• It should allow for multiple files being uploaded at once.
• It should work transparently over HTTP - which means proxies will support it like any normal web request, it can be done over HTTPS (SSL), it will reuse cookies and standard HTTP authentication methods.
• (Ideally!) the browser would handle this itself without requiring Java, Flash, or any other applets.

With all this in mind, I somehow stumbled across the idea (roughly posted here) based on the time-tested learnings from Usenet and NZB files, and BitTorrent. The main idea? Splitting the file up into manageable segments. There's also some other logic too, but that's the main idea.

Why do I claim this is the final solution?

• It can reuse the same HTTP/HTTPS connection, so proxies and HTTP authentication can be honored.
• It doesn't care what speed of your connection - due to the small size of the files, it's easier to get them to the server and each piece can be confirmed one step at a time. No more having to start from the beginning due to a failure or timeout.
• It will support multiple files at once. The server could (although we might not implement it this way) support multi-threaded uploading of the same file, too, just like BitTorrent or Usenet downloading - upload multiple pieces at the same time and assemble them in the end. We're trying to make a decision whether or not we want to do that right now. The fundamental difference is an implementation detail on the server end.
• It allows for any client that can split a file up, hash it, encode it and upload it via POST
• It will still require an applet, since browsers have no support for anything but standard file upload semantics (Although this would be a neat thing to get into a specification)

What's required, how does it work?

As of right now, this is what I have down (it has changed already since the PHP post):

1. The client contacts the server to begin the transaction. It supplies the following information:
   • Action = "begin"
   • Final filesize
   • Final filename
   • Final file hash (SHA256 or MD5, still haven't determined which one)
   • A list of all the segments - their segment id, byte size, hash (again SHA256 or MD5) - XML or JSON or something
2. Server sends back "server ready, here's your $transaction_id"
3. Client starts sending the file, one segment at a time, with the following information:
   • Action = "process"
   • Transaction ID = $transaction_id
   • Segment ID = $segment_id
   • Content = base64 or uuencoded segment (for safe transit)
4. Server replies back "segment received, transaction id $transaction_id, segment id $segment_id, checksum $checksum"
5. Client compares the checksum for $segment_id, if it matches, move on to the next segment. If not, retransmit.
6. When the client is done sending all the segments, client sends message to the server:
   • Action = "finish"
   • Transaction ID = $transaction_id
7. Server assembles all the segments (if they're separate) and sends to the client:
   • Transaction ID = $transaction_id
   • Checksum = $checksum of the final file
8. Client compares the checksum to it's own checksum. If it matches, client sends message to server:
   • Action = "complete"
   • Transaction ID = $transaction_id

Viola, done. I think the "protocol" transmits some extra information that isn't needed; so some of this might need to be cleaned up. This is the initial idea though. Props to Newzbin for inventing NZB files which was a big influence in this concept.

I'm somewhat rushing this post out, hopefully it solicits some feedback. I'm going to be revising this and working with a Java developer to work on a client written in Java. Hopefully someday we'll get one with less overhead. I'll post PHP code as I write it too to handle the server portion of it.

For a while I was looking into and hoping to go the eSATA route. Immature chipsets and lack of OS support have somewhat kept that idea frozen.

I want to use ZFS for a filesystem. Or a filesystem -like- ZFS. Currently there are no others out there like it. There is Btrfs, and there is another one I thought picking up steam (although I can't remember it now for the life of me) - both of those however aren't stable yet. ZFS is still not as stable as I wish on FreeBSD. It won't run natively on Linux, and I don't think it's very stable either. The only true way to get a stable filesystem like ZFS is to in fact run ZFS on Solaris.

I was not excited to try Solaris. It used to be a joke to call it "Slowaris" - I remember the old days of using random UNIX shells and hating Solaris boxes because I couldn't run hardly anything or compile anything. However, that's changed somewhat now. I took the plunge and installed SXCE (Nevada build 94) since Solaris 10u5 did not support the new CIFS implementation. So far, I've learned a little bit here and there about Solaris system administration and I've been using ZFS to create some snapshots, filesystems, etc. It is so easy even my mom could handle it. Not to mention Solaris has some pretty neat tools like the Solaris Fault Manager, which I have crontabbed to run every 30 minutes and email me if -any- hardware/faults get reported. So I have this great box sitting there running the best filesystem possible integrity-wise, and it is also damn quiet. It's not a small form factor which I would have liked, though.

So I begin looking into trying to get a small form factor ZFS box. I might be able to, if I want to hack up a Shuttle style case (see Udat at Mashie Design - there are mini-itx motherboards now with 6 SATA ports onboard which would allow for a 5 drive RAIDZ1 + maybe use a Compact Flash card for a boot drive. However, that requires case modding and can only fit 5 drives. I'm not sure I really want to try all that.

Instead, even Mashie himself has admitted to moving into larger form factors for storage boxes (I believe he's using a CM Stacker nowadays) - and from a space perspective, it probably does make the most sense.

Currently I'm exploring going with a full-size case that could hold 15 drives, or a mid-size case that could hold 10 drives (not including optical + boot)

I think I may have found a winner, for the mid-size option. Lian-li has a self-proclaimed "silent" chassis that has 9 bays (which means 6 bays for 2x5-in-3 modules) + optical + 1 boot disk in the spare 5.25" bay. Roughly 8-9TB usable in a mid-size case that would be about as silent as it can get. It even has a front door on it. Actually, there's a second place one - this one is much more extensible, but has no door on it, which I think would help shield any noise coming from these 5-in-3 modules. See here. Cooler Master also has a case like that too - again, no door on the front. I wish I had local access to all of these cases to try each of them out. Right now I have to order them online, and then pay possible restocking fees, and at least the cost of shipping the product back. I'm tired of that back and forth game. I've had to do it too many times in the past.

Lian-li also has a full-size chassis that already includes 10 internal drive bays, + 5x 5.25" front bays. Those extra bays could be used for more drives too. So many options... I'm trying to determine the amount of space I want to use in my office and how large and bulky I want these machines to be. Ideally I would like as little CPUs as possible - no need to have full-out operating systems installed and having to manage all that. I'm just tired of all my equipment making noise, getting hot, failing, and data corrupting due to failure or bit-rot. It's time to upgrade and streamline.

I've pretty much examined every chassis at Lian-li, Coolermaster and Antec. I have an Antec P182 right now. It's great and quiet, but does not support as many drives as I want for these next boxes. Stay tuned as I pull the trigger and build another storage box soon. Perhaps I'll share some pictures and specifications with my existing storage box I just built, which is for "off site" daily snapshots of my hosting infrastructure and some other servers I administer.

These next machines will be for my own personal use... and now I've gained some good knowledge on what to expect. It's been a while since I've built a normal-sized machine, as I've been a Shuttle XPC user for years now 🙂

Note: this has been outdated once again. Now it can be simplified with one simple directive shown here.

There's been a minor tweak required in my original WordPress+nginx rewrite rule post.

server {
   listen 80;
   server_name michaelshadle.com;
   index index.php;
   root /home/mike/web/michaelshadle.com/;
   include /etc/nginx/defaults.conf;
   include /etc/nginx/expires.conf;
   error_page 404 = /wordpress/index.php?q=$request_uri;
   location ^~ /wordpress/wp-admin {
      auth_basic "wordpress";
      auth_basic_user_file /home/mike/web/michaelshadle.com/.htpasswd;
      location ~ \.php$ {
         fastcgi_pass 127.0.0.1:11000;
      }
   }
   location ~ \.php$ {
      fastcgi_pass 127.0.0.1:11000;
   }
}

server {
   listen 80;
   server_name michaelshadle.com;
   index index.php;
   root /home/mike/web/michaelshadle.com/;
   include /etc/nginx/defaults.conf;
   include /etc/nginx/expires.conf;
   error_page 404 = /wordpress/index.php?q=$request_uri;
   location ~ \.php$ {
      fastcgi_pass 127.0.0.1:11000;
   }
}

Let me know if it doesn't work! Or better yet, let the nginx mailing list know 🙂

I don't know how or when, but I wound up getting indexed with index.php in some of my URLs.

For some reason, WordPress hasn't decided that they should parse and remove that. So for the interim, I've decided to finally throw in a couple quick lines of code to do the trick. Throw it in any plugin you want or make a standalone file, it works fine with my 2.6.1 so far.

add_action('init', 'chop_index');

function chop_index() {
   if(preg_match('/index\.php$/', $_SERVER['REQUEST_URI'])) {
        $url = preg_replace('/index\.php$/', '/', $_SERVER['REQUEST_URI']);
        $url = preg_replace('/\/\/$/', '/', $url);
        header("Location: http://".$_SERVER['HTTP_HOST'].$url, true, 301);
        exit();
   }
}