cc licensed flickr photo shared by foreversouls

My blog crashed this afternoon.

I was not driving, but with some lucky bits of intervention and guessing, I was able to roll it over and get it back on the road. This was after spending a good chunk on Sunday on a side blog that had gotten hacked (By the way, Donncha’s Exploit Scanner was a crucial key to cleaning up that one, tho it needs a bump to WP 2.9.1).

But today’s episode of “Blog and Order” was a different game. I thought I’d share some of the things I did- but keep in mind that when something like this happens, there is rarely one single recipe; and like most, I try all kinds of things, though the whole box of tricks at the wall and see what sticks.

So I went to my own blog today to look something up (whatever that was, I now forgot), and the page took a long time of just whiteness, and then I got the dreaded HTTP Server Error 500 – which tells you nothing more than some script on your site is borked.

I looked at my resources on Dreamhost (I run a private server, meaning I can restart the server- that did not help– as well as allocated more RAM to the server — at more $$ over month). I could see that the CPU usage had begun climbing well above normal levels around 3:00 AM; but even throwing more RAM at the site, it still would not load pages.

Next I check other sites hosted on the same domain. They are fine, even one that is WordPress. Then I try a non WordPress URL on my site. That also loads fine. This tells me the problem is in WordPress.

I then download the server error log. The last entries are fill of “index.php: premature end of headers” — again this tells me nothing specific except that somewhere in the chain of PHP scripts WordPress loads when it builds you a page, something borked.

By the way- another Mac tip… access or error logs can be huge files, hard to open in a text editor (BBEdit was having issues with my 180 Mb access.log file); but if you open the file on the Console utility, it manages nicely, and even offers a nice filter option when you are trying to isolate a file name or a rogue IP address.

Maybe all of my billions of fans are frantically pressing reload? I let the server sit a while. Then I decide to take it offline- you can do this easily (I do this when doing a manual update by ftp because you have to remove key files in the process–) I have an alternate index.php file I use as a sign to hang on the door (you can see what it says). Everything in WordPress is routed through this single entry point, so some basic HTML:

!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
        "http://www.w3.org/TR/1999/REC-html401-19991224/loose.dtd">








	
Please Stand By
	
Sorry. The dog is not feeling well and until we can get the blog to a vet, the dog shall be sleeping. Please resume your regular web clicking.

I thought that might alleviate the stress, and let all the fans go take a nap.

cc licensed flickr photo shared by Robbie Howell

Nope.

Next I got desperate- I contacted Dreamhost support. I got a response pretty quickly:

Hi Alan, I’m very sorry that you’ve been plagued by these errors!

I took a look through your Apache error logs, and I’ve tracked down the root cause. It seems that you’ve been hitting the default FCGI concurrent process limit of 20:

[Mon Jan 11 13:52:00 2010] [notice] mod_fcgid: too much
/home/cogdog/cogdogblog.com/index.php process(current:20, max:20), skip
the spawn request
[Mon Jan 11 13:52:01 2010] [notice] mod_fcgid: too much
/home/cogdog/cogdogblog.com/index.php process(current:20, max:20), skip
the spawn request

I’ve bumped this up to 40 concurrent processes, and have re-configured your Apache services accordingly. You should no longer be experiencing issues stemming from this, but please let me know if you have any other questions or concerns. I’d be happy to help!

I was real hopeful after that, but still no dice.

Sometimes it is a bad plugin. I cant disable them w/o being able to log in, but I tried to disable them by renaming the /wp-content/plugins folder (I can do this via Fetch, my Mac OS X ftp app) and then creating a blank one.

No change (I found out why later, because it was a plugin issue after all).

I then decided to let it go a while (sometimes this actually works).

But it was still broken.

cc licensed flickr photo shared by boxchain

I began to think maybe it was weird Firefox caching or just resource hogginess, so I opted to try my site in Safari. It’s status at the bottom gives a different sense of the progress as it lists the count of the different web pieces that load, like “completed 1 of 11 items”, “completed 2 of 11 items” — well it was getting hung on “completed 7 of 11 items”. (yikes, on my main page it loads 208! Lots of little graphics tid bits)

And then looking at the Safari Activity window I saw the problem, anarchy.js had failed to load/connect/do something (I forget the exact message, cant reproduce it). This is a part of a media plugin I used, and because it was failing in an epic fashion, the blog crashed to a halt.

So I went to my ftp site and deleted the plugin.

So it is gone, right?

Nope, the next reload (SHIFT-reload) repeated the problem. And then I realized that my wp-super-cache plugin was working, so it meant even if I deleted the Anarchy plugin, its broken bits were still cached.

And I could not log in to my dashboard to do anything.

To disable Anarchy, I needed as well to disable WP-Super-Cache.

So I then reached for the giant wrench, lifting the hood of the blog, and going into the database, with phpMyAdmin.

I managed to disable these two plugins by editing a gnarly row in the database. Do not try this at home. Well, maybe you should.

The key is in the wp_options table. This is a long list of variables and values your blog uses. You have to know the name of the option, and in this case, because I had done detective work on the hacked blog sunday, I knew to search on the option_name of active_plugins.

This gives a one row result, which you can then edit via the pencil icon.

Now this looks really scary:

This is a long string of text that represents the data of a PHP array (another list of data), in this case a list of the plugins that were activated and some other info about them.

What this looks like is nothing intelligble at first (or maybe second)

a:17:{i:0;s:19:"akismet/akismet.php";i:1;s:38:"anarchy_media/
anarchy_media_player.php";i:2;s:35:"exploit-scanner/exploit-scanner
.php";i:3;s:38:"php-execution-plugin/php_execution.php";i:4;s:29:"
plugins-list/plugins-list.php";i:5;s:53:"preserve-code-formatting/
preserve-code-formatting.php";i:6;s:15:"stats/stats.php";i:7;s:25:"
subscribe-to-comments.php";i:8;s:44:"syntaxhighlighter-plus/
syntaxhighlighter.php";i:9;s:18:"theme-switcher.php";i:10;s:49:"
twitter-friendly-links/twitter-friendly-links.php";i:11;s:40:"
twitter-tools/twitter-tools-hashtags.php";i:12;s:31:"twitter-tools/
twitter-tools.php";i:13;s:29:"wp-db-backup/wp-db-backup.php";i:14;s
:33:"wp-security-scan/securityscan.php";i:15;s:27:"wp-super-cache/
wp-cache.php";i:16;s:19:"wptouch/wptouch.php";}

but then I note some patterns- each starts with i:x;s:yy followed by the name of the plugin in quotes, and ends in a semi-colon. So first I delete (I am working with a copy of this text in a desktop editor) the two entries with the plugins I want to disable.

i:1;s:38:"anarchy_media/anarchy_media_player.php";
i:15;s:27:"wp-super-cache/wp-cache.php";

Am I done? Not sure. I look more closely at the string, and remember from doing my PHP var_dump debugging that the bit in front a:17 indicates that this is an array of 17 items (I do a search on “i:” it returns 17, so this fills right; which means I have to change this to a:15.

Finis?

Nope.

I look again at the patterns, and notice that all of the numbers after i: are incrementing from 0, so they are array counters. So when I remove the two entries, I need to re-number the others:

a:15:{i:0;s:19:"akismet/akismet.php";i:1;s:35:"exploit-scanner/
exploit-scanner.php";i:2;s:38:"php-execution-plugin/php_execution.
php";i:3;s:29:"plugins-list/plugins-list.php";i:4;s:53:"preserve-
code-formatting/preserve-code-formatting.php";i:5;s:15:"stats/stats
.php";i:6;s:25:"subscribe-to-comments.php";i:7;s:44:"
syntaxhighlighter-plus/syntaxhighlighter.php";i:8;s:18:"theme-
switcher.php";i:9;s:49:"twitter-friendly-links/twitter-friendly-
links.php";i:10;s:40:"twitter-tools/twitter-tools-hashtags.php";i:
11;s:31:"twitter-tools/twitter-tools.php";i:12;s:29:"wp-db-backup/
wp-db-backup.php";i:13;s:33:"wp-security-scan/securityscan.php";i:
14;s:19:"wptouch/wptouch.php";}

This looks right (I save a copy of the original string just in case). I replace this entry in my database, cross my thumbs, through some salt over my shoulder….

cc licensed flickr photo shared by VOD Cars

It runs! My blog is back!

OMG, I cannot believe that worked.

I go to my WP dashboard and look at my plugins, sure enough, Anarchy is gone, and WP-Super-Cache is disabled. I re-enable it (cause it helps the server load so much).

Now I don’t really know what happened to Anarchy, as it has worked for years, and I use it on other blogs. But it was borking, and when I removed it from my stalled blog, the engine lights came on.

Now, no wonder I got so much else done today!

 Share this barking on social media

If this kind of stuff has value, please support me by tossing a one time PayPal kibble or monthly on Patreon