Please comment fellow IT professionals!

I have a situation that I need help from the community on!
I am a 20+ year IT professional that dives into all aspects of IT as well as all OS’s

My employer is not in a technology driven field. It’s just me in IT and I do everything IT related.
Here is a summary of what we have. I have a 2003 server with Terminal Services enabled that the users use to run a proprietary ERP system that is made with Microsoft Access. This computer is running on a Dell Poweredge 1850 – with 2 hard drives in a NON RAID format, just a data drive as well as an OS drive.
I have another server here that is the same machine, a Dell Poweredge 1850 – also with no RAID configuration. This system runs IIS services and Microsoft SQL that communicates with the first server mentioned and takes the data and dynamically updates their website. Both servers are in an Active Directory structure. I have 2 other servers, one being Linux, for a collaboration system, and the other is being made into a new ERP system with new software.
Here’s what happened.

Last Thursday (May 3rd, 2012) an employee with a fairly new laptop with about 6 months of custom data on it, approached me and said that his computer just suddenly froze, and when he restarts, it just goes to a blinking cursor. So I take the machine and turn it on, and notice right away that the Hard Drive is not showing up during POST. I take the laptop apart and pull the drive, and place it on my SATA to USB adapter and try firing it up, the drive doesn’t even spin up. I ask about backups and there are none that have been made. So I tell him I will come up with a plan for tomorrow and end the day. This laptop had all of our forms and documents over the last 6 months, that are very crucial to the company.

Friday morning, I come in and everyone is telling me that the whole system is down, the usual information, and nothing more, so I go to look at the first server I mentioned (the TS and ERP machine) server and its stuck on the RAID configuration after BIOS and the messages across the screen are the drives spun up, but one is missing. I comfirmed the drive not available to be the OS drive of the server ( C: ) and run it to a server shop one building down and ask them if they have any USCSI available to hook the drive up and see if it fails, to quickly locate the issue.
They confirm spin-up but not recognizable. So…. What’s on the list for today???? 2 dead drives on 2 critical devices. The only thing that exists for help on the server is a VMware image I had made several months back.

The data for the ERP system is stored on the other hard drive, and it appeared to be functional, I decide since the backups for the server consisted of only data, I need to rebuild the OS and get the program running and synchronized. Here’s a list of the things done:
1. Got new hard drive into server
2. Installed 2003 server
3. Found and updated all driver’s to get internet access
4. Ran all the SP’s and updates
5. Installed MS Office – note that I have no product keys other than a Virtual Machine snapshot I had of the server from months back.
6. Installed and configured Terminal Services into Application Mode
7. Installed IIS and configured
8. Recreated the Active Directory structure and established the Domain.
9. Copied over the Access .accdb file and began to attempt to run it, with nothing but missing DLL errors
10. Hand copied and registered DLL after DLL until it finally got to a login screen
11. Access indicates the database is corrupt, finally pinpoint it down to an update in Microsoft Access.
12. Recreated ODBC connections to counter OBDC errors
13. Went to start Internet explorer to download a print driver, and the shortcut to IE was not associated??????

This list is obviously a list of the larger tasks that had to be done, we all know that in between all this, there are thousands of teeny tasks that need to be done to accommodate this, such as trying a USB hub and realizing it doesn’t work with the USB devices, or trying to run Office and realizing that it does not have SP1 or the best, running a configuration change and walking away, only to come back 20 minutes later to a prompt that says “Please insert 2003 Server CD 1”

While all this is going on, I had the laptop that was dead to deal with, after trying several things to get it going such as a tap to the spindle area I realized I had to do a last resort. The drive was taken to Geek Squad, and told it was unrecoverable, but at a cost of $500 to $2,000 they could have professional data recovery. I opened the drive and exposed it briefly to air, but got in and spun it by hand, you could hear the whine of seized bearings, but it did spin up! I was able to get chunks of data off the drive, hand spinning and closing it up as needed until I ultimately got all the data!! I began to rebuilt the OS for the laptop and started setting up ALL the things like email, network printers, mapped drives, etc… and putting data back into the retrospective directory.

I worked on all this, while maintaining IT stability in the office 6 hours Friday, nothing during the weekend, 8 hours Monday and 8 Hours Tuesday

Back to the server, I had been informed of AV alerts a day ago and briefly confirmed a possible heuristic threat that wanted to get sent to the AV company for research. When I clicked on Internet Explorer with the lack of association icon, of course nothing happened. I then followed the path and directory etc of where it was pointing to and saw a file called mso.sss and right away, knew I had found something that may explain my server not being able to shutdown without ending several instances of RUNDLL32.EXE. I went back to the machine that had the possible threat indication earlier, and searched well known virus locations, and bingo, hidden in a folder in the APPDATA directory was mso.sss. so now I realize I am dealing with a worm that is spreading thru the network. I google the file to find only 4 or 5 links to information, and soon learn that its new, rare and severe. So now I am dealing with recreating a server environment, and fighting a worm that is spreading thru the network.

The point of my email is this, ever since Monday morning, I am getting a lot of heat and asked as to why this is not fixed??? Management is well aware that I did not do anything over the weekend, but I feel like my expertise is being criticized because 7 hours into this, I am being asked constantly why this is not operational as well as a precise ETA of when it will be fully operational. Due to the lack of 100 percent knowledge on the 3rd party ERP system and that it is dynamically updating numbers on another server and everything else that is going on, I feel like they should be more proud of the status of everything. I understand the frustrations of downed systems, I am NO stranger to that, but with me being the only one in a company of around 4 servers and 12 computers, I think I am on track or even ahead of the game. I really believe that they have no idea what entails in IT and I am looking for comments to share with them from the community, what do you think?? Comments?

Might I add that a better backup system with a disaster recovery plan other than just copied data to removable drives, and a VM environment to allow replications and snapshots has been proposed weeks after I started, but shot down due to cost.

Thank you…….
Matt

 

Looks like you are doing nicely under the circumstances.
Give yourself a pat on the back for what you have accomplished so far.
Spun the drive by hand? Outstanding.
Have you managed to find out how the vermin was able to invade your system?

 

Hi Matt,

I totally agree with you man, it's all about the backup system. I've been doing IT for over 13 years and the first question I ask when doing consulting work is about what type of backup is in place. It's sad that at your company this got shot down due to cost. I mean, look at the state they're in now - all their systems down with no solid ETA to carry on with the business - now there's COST.

-Alex

 

Thats way i all ways keep a back up HD on my desktop There are dumb people that like to go threw sites that they ant suppose to on the job and that is what can F up the computers.I would give your self a Pat on the Back some times it can be hard to back up a system that size.You might want to give them servers a once over with Virus scaners and malware scans as well you never no what these people download.Tell them to take a chill pill it will get done when its done your only one person your not freaking superman.

 

you know due to frustration, I left out alot of details such as, I don't have a key to the building, so I could NOT work on this over the weekend. Our IT budget is $0, unless we beg and beg, as well as they have been presented MANY items to address the lack of disaster recovery planning etc, just to shoot it all down - to expensive... its one of those issues where you tell them that there's issues, and when it finally happens, they don't understand whats taking so long!



Thanks

 

MCgendraft,

I never understood why this is the last thing on the minds of business execs. :confused

Data is the bread and butter to the business!

I personally like online and offsite backups. You know what - tell them this: If you came in tomorrow and found your office burnt to the ground - how would you proceed? Getting new office space and computers isn't the issue here. Without the data - nothing else matters.

-Alex

 

For the cost of a external terabyte drive, I personally would of paid just to have my servers backed up (CYA). Just for my piece of mind, but maybe the execs will learn something from this.
I have keys and codes to the building so I can do what has to be done.
You're taking heat, I keep reminding them they didn't pay for any redundancy!
I'd also be looking for a new job, unless the reason they can't afford a backup is because they pay you megabucks... then, oh well.