| Upstream Provider Causes Unplanned Server Outages |
|
Written by GVPS Staff, November 30th, 2007
|
|
On Friday 30 November 2007, an unexpected carryover from yesterday’s rack transition incident caused a major outage for all of our servers. A neighboring rack blew a power coupling while two of the three redundant fiber switches were being relocated. This damage to the power coupling also damaged the remaining fiber switch. The switch was patched as quickly as possible. Ironically, it was the case that the relocation of the fiber switches was being done in an effort to put them on their own coupling to eliminate any chance of what actually happened.
While our servers actually remained on and up, there was no route to your hosts between 8:36AM and 10:34AM PST. The situation was resolved at 10:34AM PST today, and a more permanent solution will be in place by 12:01AM tomorrow, Saturday 01 December. Due to this permanent solution, there will be another downtime which is expected to last less than 5 minutes, and no more than 15 on the outside, at approximately just before midnight tonight.
You may have noticed that your servers have, in fact, reported a downtime. We used the debacles of late to go ahead and migrate the servers to their new and better homes since there was no route to your hosts from the public internet anyway. This allowed us to be more efficient within the parameters of the outage and prevented us from having to reschedule the downtime to relocate the machines later.
This issue is an isolated incident, and was the result of our demand of our upstream provider to make sure that an incident like this never actually happens. If you have any questions or wish to discuss further any of the issues, please do not hesitate to open a support ticket by either calling us directly or via email.
|
|
Read more...
|