The Life and Times of Michael Shadle

Blog | Projects | Netflix

Development


« Previous Entries

WordPress on Drizzle - Beaten To the Punch
Wed, 17 Jun 2009 15:52:28 -0700

Looks like Jeff Waugh actually beat me to it.

Haven't seen the code... but he's done it and sounds like he's done a somewhat thorough job.

Sadly I learned this from Brian's presentation on Drizzle at OS Bridge. Doh.

Posted in Drizzle, WordPress | No Comments »


Who Ever Said Open Source Software Was Perfect?
Sun, 14 Dec 2008 23:58:15 -0800

Typically, updates on the open source packages work without a hitch. However, my upgrade last weekend on my servers from Ubuntu Hardy to Intrepid wound up creating a couple major headaches, and at the same time, I noticed a handful of other snafoos happening to open source packages I use daily.

This wound up in server instability, client annoyance, and 20-30 hours solid of trial-and-error compiling, testing, debugging, etc. Even right now, if I forget to hold back the libgpac-dev package from being updated, all videos being converted lose their sound due to MP4Box crashing.

Posted in PHP, Software, nginx | No Comments »


Updates On the HTTP File Upload Front, Part 2
Wed, 03 Dec 2008 04:58:06 -0800

Continuing from http://michaelshadle.com/2008/11/26/updates-on-the-http-file-upload-front/ ...

I've been doing some research and more hacking. Code should find its way out there sometime soon. Here's my notes since the last installment of the "As The File Upload World Turns"

The only thing missing is a better attempt to see if Gears will retry the upload on a failure. I believe it is possible when dealing with a worker pool but this is -very- basic XHR usage at the moment. Perhaps since it is JavaScript-based we can add in our own re-transmission code. That's the next piece I'm going to mess around with.

Stay tuned for the results... (and code, most likely!)

Posted in PHP | No Comments »


Quick Code Snippet: Normalizing a URI (For Friendly URLs, etc.)
Wed, 03 Dec 2008 03:45:47 -0800

When you enter the realm of "friendly URLs" "slugs" "nice names" or whatever else you call them, it can make everything a lot better looking. However, if done incorrectly, you can get some duplicate indexed pages and the like. I couldn't sleep and wanted to try approaching this again a different way, and every URI I've thrown at it comes out how I want it.

Why does this matter? Typically, without rewrites and using normal webserver, directory and file semantics, a request for "/foo" should make the webserver bounce you to "/foo/" - but when dealing with rewritten URLs, there is no enforcement of this behavior. A lot of the time (at least with the stuff I'm currently dealing with) the same page shows up with "/foo" or "/foo/" and both are considered unique to a search engine. It's duplication of data which violates the normalization devil in me! Even worse, certain apps might not even process the request the same. "/foo" could load one page, and "/foo/" could load another, or an error. That's worse; when people send URLs out, sometimes they take artistic license with what they look like. This is to thwart all that and force search engines, users, etc. to all view the same URL. I chose to enforce the URL structure ending with "/" as I think it helps establish that "final" signoff of the non-query string portion of the URI.

You could go about this different ways; however, due to the way I have my nginx rewrites done, I can't rely on $_SERVER['QUERY_STRING'] which is how I had originally written it - and I wondered why I was getting some weird behavior. Now I realize I need the function to handle any string that is passed to it, and then I can make this behave appropriately.

Below is the function and some sample code to run it. There's probably a few opcodes that could be saved in trade for a little bit of memory by calculating the length of the URI, position of the "?" if there is one, etc. However, I hate defining variables so much that get used only once (a major pet peeve is when people create a new variable for absolutely no reason, this one would at least save a couple CPU cycles...) - so this is my least-amount-of-code-possible version. Enjoy.

# some example URIs
$uris = array(
"/bar/",
"/bar",
"/bar/ee",
"/bar/index.html",
"/bar/index.php",
"/bar/index.php?fds",
"/bar/index.php?f=bar&fbahd=3",
"/bar/index.php?http://www.foo.com",
"/bar/index.php/bark",
"/bar/index.php!meow|fJG)*#)$*J:g",
"?somehow",
""
);

foreach($uris as $uri) {
echo $uri." => ".normalize_uri($uri)."\n";
}

function normalize_uri($uri) {
# if there is query string, we want to chop it off and put it aside
if(strstr($uri, '?')) {
$query = substr($uri, strpos($uri, '?'), strlen($uri));
$uri = substr($uri, 0, strpos($uri, '?'));
}

# scrub any index.* stuff off the end
$uri = preg_replace("/index.(\S{0,3})$/", '', $uri);

# if it doesn't end with a '/', then add one
if(substr($uri, strlen($uri)-1, strlen($uri)) != '/') {
$uri .= '/';
}

# finally, put the query string back on
if(!empty($query)) {
$uri .= $query;
}

return $uri;
}

You'd tie this in with something like a:

header('Location: http://'.$_SERVER['HTTP_HOST'].$uri, true, 301); exit();

To make sure that it is redirecting with a 301 (search engine friendly) header. (Don't hardcode the scheme - https or http, depending on what your site uses.) Something like this should work:

if(isset($_SERVER['HTTPS']) && strtolower($_SERVER['HTTPS']) == 'on') {
$scheme = 'https://';
} else {
$scheme = 'http://';
}

This is how I would throw it all together:

if(isset($_SERVER['REQUEST_URI']) && substr($_SERVER['REQUEST_URI'], strlen($_SERVER['REQUEST_URI'])-1, strlen($_SERVER['REQUEST_URI'])) != '/') {
if(isset($_SERVER['HTTPS']) && strtolower($_SERVER['HTTPS']) == 'on') {
$url = 'https://';
} else {
$url = 'http://';
}
# fill in $_SERVER['REQUEST_URI'] here with whatever is holding the original URI
$url .= $_SERVER['HTTP_HOST'].normalize_uri($_SERVER['REQUEST_URI']);
header('Location: '.$url, true, 301); exit();
}

Now, it is 3:30am and I am trying to compose this inside of WordPress, but I believe that will work.

I apologize - for some reason the indentation is not showing up... remind me to add in that neat code sample plugin soon.

Posted in PHP | 1 Comment »


Updates On the HTTP File Upload Front
Wed, 26 Nov 2008 01:10:05 -0800

Right now I'm happy. Why's that? Because I have a neat Google Gears-based file uploader that uses hardly any code and no special server requirements or code requirements. It is *almost* everything I've been trying to accomplish with my behind-the-scenes type efforts to scare up some development... and this was accomplished so quickly it was scary.

Big props to Raghava @ work. He did the legwork of putting together the base code that makes Gears work properly and enough for me to play around with. Between the two of us I think we'll be able to conquer the rest of it.

Okay, so what exactly was I hoping for?

Right now what we've got pretty much meets most of the needs. The biggest gap here is making sure it attempts re-transmission. Without that, this is still very cool, but that will be one of the major benefits this exercise could offer.

Perhaps soon I will post some code. I'll want to ask Raghava if he cares first. Maybe some JavaScript/Gears gurus could even clean it up or add more functionality.

Posted in PHP | No Comments »


Disabling Autosave in WordPress (Properly)
Sun, 23 Nov 2008 16:46:36 -0800

This is for 2.6.3 - not sure what other versions it will work on. Originally I got this from http://www.untwistedvortex.com/2008/06/27/adjust-wordpress-autosave-or-disable-it-completely/, however, it was still giving me a JavaScript error.

Note that this worked fine in FF 2.x, and IE6 for English; but for some reason it broke in Chinese and only in IE6. It kept telling me it was failing to update a post, even though it was a new post. I traced it down finally to autosave.js changing the value of the action to "editpost" instead of "post" - I am not exactly sure why the behavior is different in different browsers, but at this point I don't care. The autosave feature is not necessary for the blog site I am supporting for work.

At first I tried a wp-config.php attempt, from http://sheldon.lendrum.co.nz/disabling-wordpress-auto-save-and-revision-saving_227/04/ - but that must not work in 2.6.x - it still appeared to be autosaving. The next Google result was a plugin fashion that dequeued the autosave.js script - which was the culprit, but it wasn't quite complete (at least in my 2.6.3 install) - so here is the complete fix (no JavaScript errors that I can tell)

First, you need to make a file called "autosave.js" and throw it in wp-content/plugins, the content of which is:

function autosave() {
        return true;
}

Next, throw this in an existing plugin or make a standalone file (I have a file with a bunch of random WP "fixes" so I just threw it in there):

function disable_autosave() {
wp_deregister_script('autosave');
wp_enqueue_script('wphacks', '/wp-content/plugins/autosave.js', array(), '20081123');
}
add_action('wp_print_scripts', 'disable_autosave');

A big thanks to the guy at Untwisted Vortex for getting me on the right path. I'm sure there might be a couple other ways to do this, but this works just fine, and maps with the primary goal of not editing the core code of the package.

Posted in WordPress | No Comments »


Changing the WordPress $table_prefix - a word of caution
Thu, 02 Oct 2008 01:20:30 -0700

I just changed the table prefix from "wp_" to "wp_en_us_" to support multiple installs in the same database, one for each locale. This seemed pretty straightforward - just rename the table names and change the table prefix in wp-config.php, right?

Wrong.

The WP code is littered with references to $table_prefix - including columns in the "usermeta" and "options" table. Key columns such as "wp_capabilities" in the "options" table need to be renamed appropriately, or your user roles are totally broken. In "usermeta" you need to make sure the "wp_user_level" and "wp_capabilities" meta_keys are updated too.

Sadly, after spending a couple hours var_dump() and exit()'ing throughout the code I realized the issue. I am not exactly sure the prefix of some of these keys needs to be aligned with the $table_prefix - maybe WPMU uses that or something. But anyway, what an annoying headache.

I am not sure but I think -all- "wp_" keys might need this same treatment. I've only really cared about the user roles and such since I had a lot of users trying to figure out why they can't post anymore. At least that's fixed.

Note: I don't know if this is clearly documented anywhere. If it is, I should be embarrased. However, I still want to voice a complaint that this isn't very straightforward. Maybe put a comment in wp-config.php?

Posted in WordPress | 1 Comment »


Large File Uploads Over HTTP - The Final Solution (I Think)
Tue, 26 Aug 2008 20:18:11 -0700

Problem statement: HTTP sucks for file uploads.

You know it. I know it. The problems?

What would the ideal file upload experience support?

With all this in mind, I somehow stumbled across the idea (roughly posted here) based on the time-tested learnings from Usenet and NZB files, and BitTorrent. The main idea? Splitting the file up into manageable segments. There's also some other logic too, but that's the main idea.

Why do I claim this is the final solution?

What's required, how does it work?

As of right now, this is what I have down (it has changed already since the PHP post):

  1. The client contacts the server to begin the transaction. It supplies the following information:
    • Action = "begin"
    • Final filesize
    • Final filename
    • Final file hash (SHA256 or MD5, still haven't determined which one)
    • A list of all the segments - their segment id, byte size, hash (again SHA256 or MD5) - XML or JSON or something
  2. Server sends back "server ready, here's your $transaction_id"
  3. Client starts sending the file, one segment at a time, with the following information:
    • Action = "process"
    • Transaction ID = $transaction_id
    • Segment ID = $segment_id
    • Content = base64 or uuencoded segment (for safe transit)
  4. Server replies back "segment received, transaction id $transaction_id, segment id $segment_id, checksum $checksum"
  5. Client compares the checksum for $segment_id, if it matches, move on to the next segment. If not, retransmit.
  6. When the client is done sending all the segments, client sends message to the server:
    • Action = "finish"
    • Transaction ID = $transaction_id
  7. Server assembles all the segments (if they're separate) and sends to the client:
    • Transaction ID = $transaction_id
    • Checksum = $checksum of the final file
  8. Client compares the checksum to it's own checksum. If it matches, client sends message to server:
    • Action = "complete"
    • Transaction ID = $transaction_id

Viola, done. I think the "protocol" transmits some extra information that isn't needed; so some of this might need to be cleaned up. This is the initial idea though. Props to Newzbin for inventing NZB files which was a big influence in this concept.

I'm somewhat rushing this post out, hopefully it solicits some feedback. I'm going to be revising this and working with a Java developer to work on a client written in Java. Hopefully someday we'll get one with less overhead. I'll post PHP code as I write it too to handle the server portion of it.

Posted in Development | 3 Comments »


nginx + WordPress - Redux
Sun, 17 Aug 2008 01:52:48 -0700

Note: this has been outdated once again. Now it can be simplified with one simple directive shown here.


There's been a minor tweak required in my original WordPress+nginx rewrite rule post.

I think we've finally got it right now. I've been exchanging emails ad nauseum with Igor and I believe the behavior now works properly (part of the exchange was regarding PHP+basic http auth, there were some bugs around the regexps/nested location blocks or something)

Anyway, here is a perfect working example, running this very website:

server {
   listen 80;
   server_name michaelshadle.com;
   index index.php;
   root /home/mike/web/michaelshadle.com/;
   include /etc/nginx/defaults.conf;
   include /etc/nginx/expires.conf;
   error_page 404 = /wordpress/index.php?q=$request_uri;
   location ^~ /wordpress/wp-admin {
      auth_basic "wordpress";
      auth_basic_user_file /home/mike/web/michaelshadle.com/.htpasswd;
      location ~ \.php$ {
         fastcgi_pass 127.0.0.1:11000;
      }
   }
   location ~ \.php$ {
      fastcgi_pass 127.0.0.1:11000;
   }
}

Likewise, you can also omit the basic HTTP authentication if you don't think you need it:

server {
   listen 80;
   server_name michaelshadle.com;
   index index.php;
   root /home/mike/web/michaelshadle.com/;
   include /etc/nginx/defaults.conf;
   include /etc/nginx/expires.conf;
   error_page 404 = /wordpress/index.php?q=$request_uri;
   location ~ \.php$ {
      fastcgi_pass 127.0.0.1:11000;
   }
}

Note: this does require a patched version of 0.7.10 - I assume he will put these changes into 0.7.11. Some of the changes required include the basic HTTP auth stuff. Also, when using error_page 404, it generated logfile noise in the error log. That was fixed as of 0.7.9 or 0.7.10, so no longer will you receive script-handled 404's in your error log.

This should be the last and final need to run WordPress properly under nginx. This has the approval of Igor, the creator - you cannot get better than that. (Note: this is WordPress 2.6.1, but I have not seen any reason the rewrite rule would be different since even before 2.x)

Let me know if it doesn't work! Or better yet, let the nginx mailing list know :)

Posted in WordPress, nginx | 1 Comment »


Simple WordPress Hack - Redirect the index.php!
Sat, 16 Aug 2008 01:05:42 -0700

I don't know how or when, but I wound up getting indexed with index.php in some of my URLs.

For some reason, WordPress hasn't decided that they should parse and remove that. So for the interim, I've decided to finally throw in a couple quick lines of code to do the trick. Throw it in any plugin you want or make a standalone file, it works fine with my 2.6.1 so far.

add_action('init', 'chop_index');

function chop_index() {
   if(preg_match('/index\.php$/', $_SERVER['REQUEST_URI'])) {
        $url = preg_replace('/index\.php$/', '/', $_SERVER['REQUEST_URI']);
        $url = preg_replace('/\/\/$/', '/', $url);
        header("Location: http://".$_SERVER['HTTP_HOST'].$url, true, 301); exit();
   }
}

Posted in WordPress | No Comments »


« Previous Entries

Entries (RSS) and Comments (RSS). 13 queries. 0.282 seconds.