I woke up about 0730 this morning to my phone being alerted that my webserver was offline. right after that several of my web hosting clients also began texting and messaging me about the issue. Once i got up I was able to see that not only was my server offline but the entire datacenter that houses my server. I knew now I had a major issue on my hands as nobody that had websites with me could be seen. This also took my site offline and all of the services it provides were offline. My provider tweeted about they were having na issue with their upstream so this was a much larger problem than just the datacenter itself.
I already had one appointment with a client whose phone system is malfunctioning and another client who now has a teslacrypt infection that i was supposed to get put back online this morning. Since i could not do anything about the datacenter outage I went to restoring the teslacrypt struck client. I got them fixed and then began working on standing up a backup Virtual Private Server to get my own customers back online.
I headed over to 1and1.com whom I have used more than a few times for various outsourced needs. They have a vps product that is well priced and easy to setup. they promised rapid startup so i got the XXXl vps package and hit setup. More than 20 minutes later the vps is still not online. I wait on hold for more than half an hour and hang up. I went to cancel the account and that failed. Now I am more than 2 hours into my own outage and my primary backup has now also failed. I then start searching online for another backup vps.
I hit up linode.com and found one of their vps offerings to be suitable as a lifeboat. i order the vps and literally within 30 seconds the server is created. Another minute to boot and i am off and rolling. i finally was able to get my control panel installed and then I was able to start restoring backups from my offsite location since the datacenter was still unreachable. I got the vps back online and my customers were glad to at least have their presences restored.
Despite my best efforts ETC Maryland lost two web hosting clients due to the delays in restoration of service. I know both owners personally and they were very sad to go but I understood they needed their sites online. I gave them a copy of all of their files and backups on flash drives and helped them with moving to their new hosting providers.
I now have to find a way to be able to better react to is this happens again. Datacenters are supposed to have systems in place to prevent a total outage but nothing is 100% safe. It turns out the datacenters provider in Ashburn virginia had a core router failure that took down several customers in their demarc room. What made it hard to figure out is all indications showed the demarc as being ok. After multiple conferences the demarc physically changed out the router in question and all circuits came back online.
In the future if there is a website outage i will post on my twitter feed @etc_md and also here on this Facebook page. This is one time there would have been nothing I would have done directly to prevent the outage but I now know where i will be heading if there is another catastrophic failure again to stand up a vps in a hurry..Linode.com..:)