I am so pleased with the new site.  The site launch looked to be going smoothly until a software update took my servers ability to provide hosting took me offline.  What an embarrassment.  I can hear the questions now.  Didn’t you test the updates?  What was your plan for recovery…etc etc etc.  I actually have a second server dedicated to testing system updates on.  It has the same setup as the server that provides my hosting services.  I actually installed the update on the testing server not once..not twice..but three times just to make sure.  How i tested was to install the update and let it roll for 24 hours.  I then restored the system image from before the update and re-installed.  I did this three times for a total of 4 days of testing.  When no errors presented themselves I proceeded to install the update on my main server here.  Everything went smoothly.  Sometime within a hour or two of installation something went wrong and the server went offline.  I was asleep during that time.  Once I woke up I knew something was amiss.  I first starting trying to troubleshoot and also engaged my software vendor via their support line.  Once it was clear a fix was not going to be quick I made a system image for troubleshooting….nuked the server and rebuilt from backups.  Once I had decided to nuke the server I was up and running within an hour.  My vendor still doesn’t know what went wrong and is investigating.  Adding to the fun is now my server is up and running WITH the update and no errors.  I think this was a case of an upgrade gone bad due to a bug.  I had recently upgraded from one version of my hosting software to the newest build.  I think this upgrade went boom. Reinstalling from scratch and then restoring backups fixed it. Thank you for your understanding.

 

Skip to content