cogdogblog

Note: CogDogBlog has a new WordPress powered home at http://cogdogblog.com/. All entries from this version have been moved there, so as a guide dog service try finding this article in its new home by title search.

The Upside of Being Hacked... Well, There is None

Last weekend I discovered the web server that hosts our Ocotillo Blogs+Wikis+boards had been compromised and some nefarious person had been able to change the root password for the server. As I was 90 miles from the machine, I had no way to seize back my root account, no way to shut it down remotely. The only recourse was to contact our security office, admit that I had missed some critical updates, and have them take the server port off the net.

So it has been shut down for 5+ days. I am not much of a server admin, and being short staffed (my office is a tech staff of 1.. me, and I lost the part-time programmer who had done our server admin tasks) I've been under the thumb of our IT department. On my return to the office Tuesday, after restarting the server and taking back my root account, I contacted the head of our security department by a voicemail message, first thing in the morning.

Nothing happened.

I left an email message at noon to ask for some assistance and to find out when we could bring the server back.

Nothing happened.

I called again at 5, left another message.

This time he called back. He was very evasive on whether they could bring the server up, not knowing what was done to it, but then surprised me by saying he would talk to his director about moving the servers up to their area and taking over the admin chores. I was encouraged and told we would talk about it the next morning.

In the meantime, I proposed another solution-- we already have another server up there, and looking at the needs, I proposed consolidating what runs on 3 servers to one. I had most of the stuff I would need, including some dumps from the database prior to the hack attack. However, a few files (mostly the directory of wiki content files) I need to get from the hacked server, and I had no way to connect to it. I spent 2 hours trying to get the sever to recognize a USB Zip drive. No luck.

The next day came. I heard nothing. Another voicemail and email left, and one more to his Director.

By the time I finished up lunch today, I figure it was time for the next level of communication-- hunt them down in their offices. I found one of them, got a "Oh I forgot to send you an email". He actually came down, showed me how to mount my zip drive, and then this afternoon, over the course of a few more hours, I was able to rebuild the MovableType blogs form the database, reload the discussion board software and populate that from the database. As always MovableType presents a challenge for the requirement of the DBI / DBD per modules to use mySQL.

I have installed MT on 4 or 5 different servers, and the experience of installing perl modules amaze me as some sort of black magic alchemy. On this machine, cpan was not even a command line option- I had to download it and run from an obscure command line. Then, apparently I lacked a bunch of libraries and had to use cpan to install pieces of itself. I was able to get DBI to install cleanly within cpan (the first time that happened), but the DBD:mysql module install crapped out "cannot make, make failed blah blah", but like another recent server, I was able to get it installed outside of cpan via the 5 command line options.

I should be able by mmid day tomorrow to have everything back in place (in addition to all passwords being changed).

I have yet to hear back from the first person I spoke to in our IT department; 4 days later. They leave me little option to bark up the management chain, again.

But this is all just my whining of 3 days of productivity mostly lost to farting around with a server.

And why did this happen? Because someone out there on another continent (ahem, we do have some coordinates) felt like they were entitled to bust into our box. I have a very hard time understanding this behavior- sure it must be some sort of ego boost to crash the gates of a corporate server, to bust into the government, to sneak around the walls of Microsoft, but to pick on a .edu site that is operated on a technical shoestring by someone intent in building tools and content for sharing-- well there is nothing lower in phylogeny of net scum to describe this action, worse than blog spam roaches. It just reeks and discourages me. Maybe I should just retire and go into shoe sales.

No, there is absolutely no upside of being hacked, and I was fortunate-- no content was munged, deleted. But still, my time, energy, and optimism are scuffed up.