We have a major incident on SBG2. The fire declared in the building. Firefighters were immediately on the scene but could not control the fire in SBG2. The whole site has been isolated which impacts all services in SGB1-4. We recommend to activate your Disaster Recovery Plan.

2:42 AM · Mar 10, 2021

412
2,359
4,946
Update 5:20pm. Everybody is safe. Fire destroyed SBG2. A part of SBG1 is destroyed. Firefighters are protecting SBG3. no impact SBG4.
135
477
1,774
Update 7:20am Fire is over. Firefighters continue to cool the buildings with the water. We don’t have the access to the site. That is why SBG1, SBG3, SBG4 won’t be restarted today.
109
346
1,270
Update 10am. (I’m there). We finished to shutdown the UPS in SBG3. Now they are off. We are looking to enter into SBG3 and check the servers. The goal is to create a plan to restart , at least SBG3/SBG4, maybe SBG1. To do so, we need to check the network rooms too.
104
224
1,096
Update 11:20am All servers in SBG3 are okey. They are off, but not impacted. We create a plan how to restart them and connect to the network. no ETA. Now, we will verify SBG1.
111
218
1,278
Update 11:40am The network room in SBG1 is okey. 4 rooms destroyed. 8 rooms are okey.
98
164
998
Update 1pm Plan for the next 1-2 weeks: 1) rebuilding 20KV for SBG3 2) rebuilding 240V in SBG1/SBG4 3) verifying DWDM/routers/switchs in the network room A (SBG1). checking the fibers Paris/Frankfurt 4) rebuilding the network room B (in SBG5). checking fibers Paris/Frankfurt
279
251
1,213
Update 4pm We plan to restart SBG1+SBG4+the network by Monday March,15 and SBG3 by Friday March,19. In RBX+GRA we have the stock of new servers, pcc, pci ready to be delivered for all the impacted customers. Of course for free. We will add 10K servers in the next 3-4 weeks.
312
365
1,436
Update 9:30pm The teams are working to fix the issue on APIv6, Manager, Support etc. restoring the emails services hosted in SBG2.
24
52
477
Many thanks for all the empathy messages you sent us today ! I want to thank the teams who have been working all night/day. It’s been the worst day for the last 22y and there is no word strong enough to say how sorry I feel today. We keep working hard to restart SBG1/3/4 asap!
297
314
3,238
Update 3pm 1/3 From yesterday, we had the meetings with Police, DREAL, Experts and Insurance. Also, we started to clean up the site. The goal is to have the full and secured access to SBG1, SBG3 and SBG4.
9
26
225
2/3 The optical network from Network Room A (SBG1) to Paris + Frankfurt works. Working on the power SBG1 option based on the generators. The SBG3’ generators work. Good option to use if the network is UP before we finished to rebuild SBG3’s 20KV.
25
34
289
3/3 In 1H I will post a short (8m) video with more informations / more details.
51
35
396
Update 11 mars 16h40 C'est trop lent de vous communiquer tous ces détails via 280 caractères de Twitter. Voici un concentré de 8min qui résume la situation à ce jour. (English version is coming) ovh.com/fr/images/sbg/Octave…
218
376
1,211
Update 11 mars 16h40 It's too slow to give you all the information with just 280 chars. Here, my video with 8min of information we have today. ovh.com/fr/images/sbg/Octave…
82
131
566
Update March 12, 8am
Michel Paulin (Directeur Général d’OVH Cloud): Suite à l'incendie, "les data ne sont pas forcément perdues mais non-accessibles" 🎙 @chrisjaku, Sandra Gandoin, @OVHcloud_FR
26
24
128
Update March,12 11am 1/4 This afternoon, we will send the email to each customer with his specific situation and the options. In any case, we recommand to restart the service in our others DC (RBX/GRA) where we are adding the additional resources. Free months will be applied asap
49
34
165
Update March,12 11am 2/4 We are working with the insurance’s experts to repower SBG1/3/4 today from the generators. The goal is to verify, room by room, equipment by equipment, that infra works. We don’t expect to restart the servers before next week.
9
17
113
Update March,12 11am 3/4 New network room (B) + 20KV for SBG3 are assembled and will be on the road this night, on the side tomorrow. Then, we start pluging them. It will take over the generators SBG3 by Fri. We verify 20KV + 240V in SBG1/SBG4 to take over the generators by Tue
7
16
139
Update March,12 11am 4/4 We will start repowering SBG1/SBG4 by the end of the next week, Fri, 19. It will take 2 days to have all servers UP. We will start repowering SBG3 by the mid of the next week, Wed, 17, it will take 6-8 days to have all servers UP.
53
40
194
State of Backup Service (Free or Paid) for the SBG customers: 1/5 - FTP Backup in SBG (Free/Paid) for VPS and Baremetal : the datas are in RBX. You have full access.
27
22
106
State of Backup Service (Free or Paid) for the SBG customers: 2/5 - pCS in SBG: We hope to restore the pCS cluster in SBG next week. The servers are in SBG1 + SBG3. We may have some bad news, this is why the final status has to be confirmed in the next days.
12
14
85
State of Backup Service (Free or Paid) for the SBG customers: 3/5 - Paid Backup VPS & PCI: 80% of the data are on pCS in SBG. please read the previous tweet about the state of pCS in SBG. 20% of the data was on pCA which was in SBG2.
19
15
90
State of Backup Service (Free or Paid) for the SBG customers: 4/5 - Free/Paid Backup pCC in SBG1 was hosted in an separated room of SBG1. Both rooms are destroyed. - Free/Paid Backup pCC in SBG3: all datas seems to be safe.
30
20
114
State of Backup Service (Free or Paid) for the SBG customers: 5/5 We recommand to restart the service in our other DC (RBX or GRA). Free months will be applied asap.
123
22
146
Update March,13 / 3:30am Last emails are in the process to be sent to all customers, with the current information, about the state of the primaire data and the backup (if the service was subscribed) for Baremetal, Public Cloud (pCI pCS pCA K8S..), Hosted Private Cloud (pCC),NAS..
106
18
184
Update March, 13 / 3pm We are creating a page for - the services hosted in SBG - the service backups (FTP Backup, VPS Automated Backup, VPS Snapshot SSD/Cloud, Instance Backup/Snapshot, Volume Snapshot/Backup) - the internal backup - the state of the backups - next steps / ETA
91
27
198
Update March 14, 1pm 1/3 Still working on restarting of the A optical network in SBG1. We hope to have the network A UP today. It will allow us to redeploy the internal tools locally. Reconstruction of the B optical network will take 2 days. Lot of fibers have to connected.
6
17
116
Update March 14, 1pm 2/3 240V in SBG1 is ready to be tested. Tomorrow 20KV for SBG1 will be UP. The 20KV is SBG3 is in progress. 2 days. Watercooling SBG1/3/4 checked and protected. Cleaning and drying are in progress.
6
12
149
Update March 14, 1pm 3/3 60 tech work on the site. Still lot of to do, but it’s going faster that expected. We keep the same ETA and maybe we will be faster. A webpage with the final state of: - primary data - internal backup (not contractual) - service backup is in progress.
44
32
252
Update March,15 0:30am Here, the URL with - the state of the each service in each DC - the state of our Internal Backup (no contractual) - the state of the Service Backup (if cust took the service) EN: ovhcloud.com/en/lp/status-se… FR: ovhcloud.com/fr/lp/status-se…
77
49
207