Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OVH CEO Octave Klaba speaking about the incident [video]

ovh.com

Post reply on HN
Print view View original

71–80 of 143 posts

Re: OVH CEO Octave Klaba speaking about the incident [video]

#71

Post by gruez »

bigyikes wrote: post #42
notJim wrote: post #27

Not to be rude, but it's really wild to me that even after all this time during the pandemic, the CEO doesn't have a headset he can use so that the audio is intelligible. It's gotta be one of the highest ROI investments you can possibly make at this point.

This also frustrates me so much with all the newly-live-streamed events this year. So many companies are spending so much money putting together virtual conferences, but can’t be bothered to ship their speakers a decent mic or webcam. Heck, Apple’s $20 headphones would make a huge difference. Instead we get audio that sounds like it was recorded in my shower.

The audio was mostly fine, it was just randomly crapping out for a few seconds at a time.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#72

Post by throw0101a »

deftnerd wrote: post #23

Are there industry options or methods of wiring to allow for a UPS room separate from the actual rooms the racks are stored in? It's almost tradition to have a rack with UPS's in the bottom and then the rest of the space filled with servers or drive arrays. We wouldn't ever think of putting a tiny backup generator in the bottom of every rack, so why do we put a battery storage system there? Also, with the advances in…

> Are there industry options or methods of wiring to allow for a UPS room separate from the actual rooms the racks are stored in?

Yes: longer cables.

See Figure 1 in the Schneider-APC white paper, where they have "Electrical Space", "Mechanical Space" (HVAC), and IT Space:

* https://download.schneider-electric.com/files?p_File_Name=VA...

Power is generated hundreds of kilometres from where it is used, so having your UPS room a few dozen metres from your actual DC room isn't a big deal. I-squared-R losses aren't going to be that huge.

Europe uses 400Y/230 for nominal low-voltage distribution (see Table 1 in above), so stringing some 400V extra copper to the PDUs, which then have 230V at the plugs, isn't a big deal.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#73

Post by LinuxBender »

mikepurvis wrote: post #34
deftnerd wrote: post #23

Are there industry options or methods of wiring to allow for a UPS room separate from the actual rooms the racks are stored in? It's almost tradition to have a rack with UPS's in the bottom and then the rest of the space filled with servers or drive arrays. We wouldn't ever think of putting a tiny backup generator in the bottom of every rack, so why do we put a battery storage system there? Also, with the advances in…

For the extreme opposite of that, Google famously trolled everyone 10 years ago by announcing that every one of their servers had its own in-chassis 12V battery: https://www.cnet.com/news/google-uncloaks-once-secret-server...

And Facebook had small UPS/ATS units at the end of each row. Not sure if they still do that today it was like that when I walked through their datacenter. They did that for the purpose of power efficiency. They lost far less power by having many smaller units.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#74

Post by hinkley »

deftnerd wrote: post #23

Are there industry options or methods of wiring to allow for a UPS room separate from the actual rooms the racks are stored in? It's almost tradition to have a rack with UPS's in the bottom and then the rest of the space filled with servers or drive arrays. We wouldn't ever think of putting a tiny backup generator in the bottom of every rack, so why do we put a battery storage system there? Also, with the advances in…

The farther your UPS is away from the server the fewer causes of power loss you can prevent.

I've seen people use UPSes to allow them to rearrange wiring. I've seen them fail by relying on the UPSes as well, of course.

If you wire the entire room with 2-3 separate electrical systems all powered off of separate remote UPSes, you can do whatever you want, but it's harder to change your mind or build out incrementally if you do.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#75

Post by fanf2 »

verytrivial wrote: post #68
mgbmtl wrote: post #19

Earlier quoted context omitted.

Other random bits: SBG-2 was an older generation datacenter, had ventilation issues? They have 4 other datacenters who have a similar design. Others, including SBG-3, have newer designs. They're building 2500 servers per week. For the offline buildings that are not destroyed, they have to rebuild the electrical distribution and network. It was not clear if they are also moving servers physically.

I read his description and literal hand waving as saying that for efficiency reasons, in 2011 they were sort of build like how you'd build a camp fire: convective airflow drawn through and out the top.

Right, this is the OVH “tower” data center design. Here’s a google maps link to the OVH Strasbourg site https://goo.gl/maps/rQjir8byKNwMDZrm9 with (from south to north) SBG1 (a collection of shipping containers), SBG2 (a figure 8 shape), and SBG3 (a box).

The idea of the tower design is to take ambient air in through the outside walls, through I think just one row of servers, then into a central void where convection pulls the air out.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#76

Post by dvfjsdhgfv »

jonas21 wrote: post #39
ev1 wrote: post #18

Earlier quoted context omitted.

This is an apples to oranges comparison. OVH largely sells bare metal; their public cloud wasn't really impacted. If you are using AWS, Google, or Azure, ran a single (or multiple machines) inside a single AZ with no backups and opted out of snapshots, you would face the exact same situation. I can definitely say I see people complaining about how everything they have is down on AWS when us-east-1 goes down periodica…

On the other hand, just about every month, there's a story on HN saying why are you wasting your money on AWS when OVH is so much cheaper (for example https://news.ycombinator.com/item?id=24966028 ). And well, I guess this is one of the reasons.

No, it doesn't work like this. I have several bare-metal (with Heztner, I use OVH for DNS), it's been over 10 years already. I know that if I only rent one machine in one location, I'm asking for trouble. Based on my experience, I would say that every 2-6 years something dies in a server. A disk, a controller, a fan, you name it. It's rare to have servers running for longer than 7 years without any issues, and they're outdated by that time anyway so they need to be migrated to a new machine.

So, as a bare minimum, you rent at least two different machines at two different locations for each project and make offsite backups. It's still way less expensive than AWS.

If I don't need a powerful server and just need to spin some instances for testing or small projects, I use Hetzner Cloud, it's ridiculously cheap.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#77

Post by raphaelj »

tpmx wrote: post #24

I think the key issue here is that there wasn't a functioning fire suppresssion system in place. Second question: is such a system required for this kind of operation? Maybe?

I’m pretty confident they have efficient fire suppression systems.

They are hosting at least 400,000 servers. They for sure have multiple servers taking fire every single day, and yet it’s the first time it ends up in a catastrophic fire.

The fire suppression system either catastrophically failed, or something out of design happened with one of their inverters, as suggested in the video.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#78

Post by bcrl »

walshemj wrote: post #55
deftnerd wrote: post #23

Are there industry options or methods of wiring to allow for a UPS room separate from the actual rooms the racks are stored in? It's almost tradition to have a rack with UPS's in the bottom and then the rest of the space filled with servers or drive arrays. We wouldn't ever think of putting a tiny backup generator in the bottom of every rack, so why do we put a battery storage system there? Also, with the advances in…

Depends for a DC / Telco your normally feeding 48v DC from the UPS which is normally separate. The lack of fire suppression is also very worrying.

s/UPS/batteries/. DC makes uninterruptible power much more simpler than an AC UPS.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#79

Post by walshemj »

bcrl wrote: post #78
walshemj wrote: post #55

Earlier quoted context omitted.

Depends for a DC / Telco your normally feeding 48v DC from the UPS which is normally separate. The lack of fire suppression is also very worrying.

s/UPS/batteries/. DC makes uninterruptible power much more simpler than an AC UPS.

Oh yes I sort of skipped over that part :-)

Re: OVH CEO Octave Klaba speaking about the incident [video]

#80

Post by paulfurtado »

raphaelj wrote: post #77
tpmx wrote: post #24

I think the key issue here is that there wasn't a functioning fire suppresssion system in place. Second question: is such a system required for this kind of operation? Maybe?

I’m pretty confident they have efficient fire suppression systems. They are hosting at least 400,000 servers. They for sure have multiple servers taking fire every single day, and yet it’s the first time it ends up in a catastrophic fire. The fire suppression system either catastrophically failed, or something out of design happened with one of their inverters, as suggested in the video.

Do servers really catch on fire nearly daily at this scale?
Post reply on HN