Blog Pile

Digging Out from a Blog Crash


cc licensed flickr photo shared by foreversouls

My blog crashed this afternoon.

I was not driving, but with some lucky bits of intervention and guessing, I was able to roll it over and get it back on the road. This was after spending a good chunk on Sunday on a side blog that had gotten hacked (By the way, Donncha’s Exploit Scanner was a crucial key to cleaning up that one, tho it needs a bump to WP 2.9.1).

But today’s episode of “Blog and Order” was a different game. I thought I’d share some of the things I did- but keep in mind that when something like this happens, there is rarely one single recipe; and like most, I try all kinds of things, though the whole box of tricks at the wall and see what sticks.

So I went to my own blog today to look something up (whatever that was, I now forgot), and the page took a long time of just whiteness, and then I got the dreaded HTTP Server Error 500 – which tells you nothing more than some script on your site is borked.

I looked at my resources on Dreamhost (I run a private server, meaning I can restart the server- that did not help– as well as allocated more RAM to the server — at more $$ over month). I could see that the CPU usage had begun climbing well above normal levels around 3:00 AM; but even throwing more RAM at the site, it still would not load pages.

Next I check other sites hosted on the same domain. They are fine, even one that is WordPress. Then I try a non WordPress URL on my site. That also loads fine. This tells me the problem is in WordPress.

I then download the server error log. The last entries are fill of “index.php: premature end of headers” — again this tells me nothing specific except that somewhere in the chain of PHP scripts WordPress loads when it builds you a page, something borked.

By the way- another Mac tip… access or error logs can be huge files, hard to open in a text editor (BBEdit was having issues with my 180 Mb access.log file); but if you open the file on the Console utility, it manages nicely, and even offers a nice filter option when you are trying to isolate a file name or a rogue IP address.

Maybe all of my billions of fans are frantically pressing reload? I let the server sit a while. Then I decide to take it offline- you can do this easily (I do this when doing a manual update by ftp because you have to remove key files in the process–) I have an alternate index.php file I use as a sign to hang on the door (you can see what it says). Everything in WordPress is routed through this single entry point, so some basic HTML:

I thought that might alleviate the stress, and let all the fans go take a nap.


cc licensed flickr photo shared by Robbie Howell

Nope.

Next I got desperate- I contacted Dreamhost support. I got a response pretty quickly:

Hi Alan, I’m very sorry that you’ve been plagued by these errors!

I took a look through your Apache error logs, and I’ve tracked down the root cause. It seems that you’ve been hitting the default FCGI concurrent process limit of 20:

[Mon Jan 11 13:52:00 2010] [notice] mod_fcgid: too much
/home/cogdog/cogdogblog.com/index.php process(current:20, max:20), skip
the spawn request
[Mon Jan 11 13:52:01 2010] [notice] mod_fcgid: too much
/home/cogdog/cogdogblog.com/index.php process(current:20, max:20), skip
the spawn request

I’ve bumped this up to 40 concurrent processes, and have re-configured your Apache services accordingly. You should no longer be experiencing issues stemming from this, but please let me know if you have any other questions or concerns. I’d be happy to help!

I was real hopeful after that, but still no dice.

Sometimes it is a bad plugin. I cant disable them w/o being able to log in, but I tried to disable them by renaming the /wp-content/plugins folder (I can do this via Fetch, my Mac OS X ftp app) and then creating a blank one.

No change (I found out why later, because it was a plugin issue after all).

I then decided to let it go a while (sometimes this actually works).

But it was still broken.


cc licensed flickr photo shared by boxchain

I began to think maybe it was weird Firefox caching or just resource hogginess, so I opted to try my site in Safari. It’s status at the bottom gives a different sense of the progress as it lists the count of the different web pieces that load, like “completed 1 of 11 items”, “completed 2 of 11 items” — well it was getting hung on “completed 7 of 11 items”. (yikes, on my main page it loads 208! Lots of little graphics tid bits)

And then looking at the Safari Activity window I saw the problem, anarchy.js had failed to load/connect/do something (I forget the exact message, cant reproduce it). This is a part of a media plugin I used, and because it was failing in an epic fashion, the blog crashed to a halt.

So I went to my ftp site and deleted the plugin.

So it is gone, right?

Nope, the next reload (SHIFT-reload) repeated the problem. And then I realized that my wp-super-cache plugin was working, so it meant even if I deleted the Anarchy plugin, its broken bits were still cached.

And I could not log in to my dashboard to do anything.

To disable Anarchy, I needed as well to disable WP-Super-Cache.

So I then reached for the giant wrench, lifting the hood of the blog, and going into the database, with phpMyAdmin.

I managed to disable these two plugins by editing a gnarly row in the database. Do not try this at home. Well, maybe you should.

The key is in the wp_options table. This is a long list of variables and values your blog uses. You have to know the name of the option, and in this case, because I had done detective work on the hacked blog sunday, I knew to search on the option_name of active_plugins.

This gives a one row result, which you can then edit via the pencil icon.

Now this looks really scary:

This is a long string of text that represents the data of a PHP array (another list of data), in this case a list of the plugins that were activated and some other info about them.

What this looks like is nothing intelligble at first (or maybe second)

but then I note some patterns- each starts with i:x;s:yy followed by the name of the plugin in quotes, and ends in a semi-colon. So first I delete (I am working with a copy of this text in a desktop editor) the two entries with the plugins I want to disable.

Am I done? Not sure. I look more closely at the string, and remember from doing my PHP var_dump debugging that the bit in front a:17 indicates that this is an array of 17 items (I do a search on “i:” it returns 17, so this fills right; which means I have to change this to a:15.

Finis?

Nope.

I look again at the patterns, and notice that all of the numbers after i: are incrementing from 0, so they are array counters. So when I remove the two entries, I need to re-number the others:

This looks right (I save a copy of the original string just in case). I replace this entry in my database, cross my thumbs, through some salt over my shoulder….


cc licensed flickr photo shared by VOD Cars

It runs! My blog is back!

OMG, I cannot believe that worked.

I go to my WP dashboard and look at my plugins, sure enough, Anarchy is gone, and WP-Super-Cache is disabled. I re-enable it (cause it helps the server load so much).

Now I don’t really know what happened to Anarchy, as it has worked for years, and I use it on other blogs. But it was borking, and when I removed it from my stalled blog, the engine lights came on.

Now, no wonder I got so much else done today!

Profile Picture for Alan Levine aka CogDog
An early 90s builder of the web and blogging Alan Levine barks at CogDogBlog.com on web storytelling (#ds106 #4life), photography, bending WordPress, and serendipity in the infinite internet river. He thinks it's weird to write about himself in the third person.

Comments

  1. Console! I hadn’t thought to load log files from a remote server in it. Duh. I use it all the time for local logs, but never thought to download apache logs from other servers and use it to mine them. Great tip! Thanks! (and glad you’re back on the road)

  2. Well, I hope I never actually have to do this myself (fingers crossed) but as always thanks for blogging your process, it makes a lot of sense, and it is good to know where to manually disable plugins in the wp_options table, as that is a not-totally-unheard-of necessity.

  3. Dude that is just nuts! Like Scott I hope I never have to do this myself but I truly do appreciate your debriefing your encounter with us so that we can learn from your experience. And I have to say I love the way you wrapped your experience up in the car wreck story.

    Keep on truckin’!

  4. Funny, I’ve been dealing with some problems on a few WP sites lately that pointed me towards a lot of the same issues.

    The first was when a plugin on UMW Blogs (which I was the only one using) borked our DTLT site. After some searching, Jim and I found some advice about active_plugins. (Link here: http://perishablepress.com/press/2008/02/18/quickly-disable-or-enable-all-wordpress-plugins-via-the-database/). In our case, we disabled everything by deleting the entire contents of active_plugins and then I re-enabled them one by one once I was back in (In our case, we weren’t sure which plugin was causing the problem.)

    In another case I had some seriously screwing permissions after migrating a site from WPMU to a single WP install (don’t ask). After doing the same database hunting, I found the field that assigns capabilities to WP roles. I ran across the same serialized data that you mention, and that led me to this link: http://www.functions-online.com/unserialize.html. Basically, that decodes the serialized data — makes it easier if you need to do edits. As I discovered, one small typo can wreck even more havoc.

    Anyway, I thought I’d drop those resources here in case anyone comes across your post and is looking for a few additional tips.

    Thanks for blogging this! It’s always useful to see how people work through these kinds of issues.

    1. Thanks Martha- I think nuking the entire active_plugins row and re-setting the plugins is easier and a more sensible approach! I also meant to thank you foir tweeting that functions-online site- it is very very handy! It saves me having to write test scripts to do things like process date / unix time conversions, as well as deciphering some of the spoam stuff using the base64_decode.

Leave a Reply

Your email address will not be published. Required fields are marked *