At approx 16:45 the Coreix Data Center suffered a power loss, UPS took over the load for short while, however the Generators did not come online in time and it took till 7:35 for power to be restored. There are details posted on the coreix status page at http://status.coreix.net/
While power did get restored to the entire DC, only 1 of the machines came back online. All the rest have needed some level of manual intervention! I'm working with the support people ( who are a really good and effective bunch of guys ) to get the other machines online.
Services affected are:
- RPMForge svn repo
- RPMForge master mirror
- RPMForge mailing lists
- Karan.org Build services
- Karan.org testing services
- CentOS.org ipv6 test / qa setups
- CentOS.org Package and Automated testing development machine
Once all services are restored, I'll update this blog post with details. And apologies for this completely unplanned and avoidable outage.
Update: as of 23:45 26th Nov, all services are now restored.
Comment feed for this post