Service Outage 07 Aug 2011

Posted on 08/08/2011 by

1


This Sunday there was a major outage at the Amazon EC2 hosting site in Dublin, Ireland, from about 7pm to 12pm BST  (read more here : http://www.irishtimes.com/newspaper/breaking/2011/0808/breaking20.html).

This is where the Kasabi API servers are hosted, and we had some service downtime as a result (though the website at beta.kasabi.com was unaffected).

Apparently, lightning hit a local power distribution site, causing the power supplies to Amazon to fail. Worse, the backup power provision was also affected, so large chunks of the hosted services went down. Amazon restored the power, but also faced problems with restarting a large number of servers that were running in that zone.

In the meantime, we set up a new server to handle the Kasabi API traffic, and routed the api.kasabi.com domain to it. The changes took a while to progagate to the loadbalancers and the name servers, but meant that the APIs were functioning again by 12pm BST on Sunday. We gave updates on Twitter to pass on updates (http://twitter.com/#!/teamkasabi).

At the time of writing (8th Aug 2011, 12:30 BST) full services are still being restored at the Amazon data centre, but we’re hoping that the disruption to the Kasabi service is over.

We were already in the process of re-evaluating our infrastructure, with a view to increasing the support levels and resilience of our servers – this event will hasten that review..!

Please contact us on the usual channels if you have any questions about this incident, or any other aspects of Kasabi.

Posted in: Announcement, Issues