We have a major incident on SBG2. The fire declared in the building. Firefighters were immediately on the scene but could not control the fire in SBG2. The whole site has been isolated which impacts all services in SGB1-4. We recommend to activate your Disaster Recovery Plan.
2:42 AM · Mar 10, 2021
Update 10am. (I’m there). We finished to shutdown the UPS in SBG3. Now they are off. We are looking to enter into SBG3 and check the servers. The goal is to create a plan to restart , at least SBG3/SBG4, maybe SBG1. To do so, we need to check the network rooms too.
Update 1pm Plan for the next 1-2 weeks: 1) rebuilding 20KV for SBG3 2) rebuilding 240V in SBG1/SBG4 3) verifying DWDM/routers/switchs in the network room A (SBG1). checking the fibers Paris/Frankfurt 4) rebuilding the network room B (in SBG5). checking fibers Paris/Frankfurt
Update 4pm We plan to restart SBG1+SBG4+the network by Monday March,15 and SBG3 by Friday March,19. In RBX+GRA we have the stock of new servers, pcc, pci ready to be delivered for all the impacted customers. Of course for free. We will add 10K servers in the next 3-4 weeks.
Many thanks for all the empathy messages you sent us today ! I want to thank the teams who have been working all night/day. It’s been the worst day for the last 22y and there is no word strong enough to say how sorry I feel today. We keep working hard to restart SBG1/3/4 asap!
Update March,12 11am 1/4 This afternoon, we will send the email to each customer with his specific situation and the options. In any case, we recommand to restart the service in our others DC (RBX/GRA) where we are adding the additional resources. Free months will be applied asap
Update March,12 11am 3/4 New network room (B) + 20KV for SBG3 are assembled and will be on the road this night, on the side tomorrow. Then, we start pluging them. It will take over the generators SBG3 by Fri. We verify 20KV + 240V in SBG1/SBG4 to take over the generators by Tue
Update March,13 / 3:30am Last emails are in the process to be sent to all customers, with the current information, about the state of the primaire data and the backup (if the service was subscribed) for Baremetal, Public Cloud (pCI pCS pCA K8S..), Hosted Private Cloud (pCC),NAS..
Update March, 13 / 3pm We are creating a page for - the services hosted in SBG - the service backups (FTP Backup, VPS Automated Backup, VPS Snapshot SSD/Cloud, Instance Backup/Snapshot, Volume Snapshot/Backup) - the internal backup - the state of the backups - next steps / ETA
Update March 14, 1pm 1/3 Still working on restarting of the A optical network in SBG1. We hope to have the network A UP today. It will allow us to redeploy the internal tools locally. Reconstruction of the B optical network will take 2 days. Lot of fibers have to connected.
Update March 14, 1pm 3/3 60 tech work on the site. Still lot of to do, but it’s going faster that expected. We keep the same ETA and maybe we will be faster. A webpage with the final state of: - primary data - internal backup (not contractual) - service backup is in progress.
Update March,15 0:30am Here, the URL with - the state of the each service in each DC - the state of our Internal Backup (no contractual) - the state of the Service Backup (if cust took the service) EN: ovhcloud.com/en/lp/status-se… FR: ovhcloud.com/fr/lp/status-se…