Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ceph: A Journey to 1 TiB/s

ceph.io

Post reply on HN
Print view View original

161–170 of 216 posts

Re: Ceph: A Journey to 1 TiB/s

#161

Post by Snow_Falls »

nijave wrote: post #154
matheusmoreira wrote: post #88

Earlier quoted context omitted.

> Also consider how (or if) you are going to back up your hoard of data. I actually emailed backblaze years ago about their supposedly unlimited consumer backup plan. Asked them if they would really allow me to dump into their systems dozens of terabytes of encrypted undeduplicable data. They responded that yes, they would. Still didn't believe them, these corporations never really mean it when they say unlimited. Pl…

> these corporations never really mean it when they say unlimited. Plus they had no Linux software Afaik they rely on the latter to mitigate the risk of the former.

Considering the fact that most data heavy servers are llnux, that would be a pretty clever way of staying true to their word.

Re: Ceph: A Journey to 1 TiB/s

#162

Post by amadio »

theyinwhy wrote: post #129
amadio wrote: post #112

Nice article! We've also recently reached the mark of 1TB/s at CERN, but with EOS ( https://cern.ch/eos ), not ceph: https://www.home.cern/news/news/computing/exabyte-disk-stora... Our EOS clusters have a lot more nodes, however, and use mostly HDDs. CERN also uses ceph extensively.

Great! What's your take on ceph? Is the idea to migrate to EOS long term?

EOS and ceph have different use cases at CERN. EOS holds physics data and user data in CERNBox, while ceph is used for a lot of the rest (e.g. storage for VMs, and other applications). So both will continue to be used as they are now. CERN has over 100PB on ceph.

Re: Ceph: A Journey to 1 TiB/s

#163

Post by plagiarist »

qingcharles wrote: post #102
chx wrote: post #92

There was a point in history when the total amount of digital data stored worldwide reached 1TiB for the first time. It is extremely likely this day was within the last sixty years. And here we are moving that amount of data every second on the servers of a fairly random entity. We not talking of a nation state or a supranatural research effort.

That reminds me of a calculation I did which showed that my desktop PC would be more powerful than all of the computers on the planet combined in like 1978 :D

My phone has more computation than anything I would have imagined owning, and I sometimes turn on the screen just to use as a quick flashlight.

Re: Ceph: A Journey to 1 TiB/s

#164

Post by matheusmoreira »

deadbunny wrote: post #101
matheusmoreira wrote: post #72

Earlier quoted context omitted.

I just want to hoard data. I hate having to delete stuff to make space. Things disappear from the web every day. I should hold onto them. My requirements for a storage solution are: > Single root file system > Storage device failure tolerance > Gradual expansion capability The problem with every storage solution I've ever seen is the lack of gradual expandability. I'm not a corporation, I'm just a guy. I don't have t…

ZFS using mirrors is extremely easy to expand. Need more space and you have small drives? Replace the drives in a mirror one by one with bigger ones. Need more space and already have huge drives? Just add another vdev mirror. And the added benefit of not living in fear of drive failure while resilvering as it is much faster with mirrors than raidX. Sure the density isn't great as you're essentially running at 50% or…

> Replace the drives in a mirror one by one with bigger ones.

That's exactly what I meant by "just as bad as RAID". Expanding an existing array is analogous to every single drive in the array failing and getting replaced with higher capacity drives.

When a drive fails, the array is in a degraded state. Additional drive failures put the entire system in danger of data loss. The rebuilding process generates enormous I/O loads on all the disks. Not only does it take an insane amount of time, according to my calculations the probability of read errors happening during the error recovery process is about 3%. Such expansion operations have a real chance of destroying the entire array.

Re: Ceph: A Journey to 1 TiB/s

#165

Post by matheusmoreira »

nijave wrote: post #155
matheusmoreira wrote: post #72

Earlier quoted context omitted.

I just want to hoard data. I hate having to delete stuff to make space. Things disappear from the web every day. I should hold onto them. My requirements for a storage solution are: > Single root file system > Storage device failure tolerance > Gradual expansion capability The problem with every storage solution I've ever seen is the lack of gradual expandability. I'm not a corporation, I'm just a guy. I don't have t…

Not sure what the multidisk consensus is for btrfs now-a-days but adding/removing devices is trivial, you can do "offline" dedupe, and you can rebalance data if you change the disk config. As an added bonus it's also in-tree so you don't have to worry about kernel updates breaking things I think you can also potentially do btrfs+LVM and let LVM manage multi device. Not sure what performance looks like there, though

That's all great but btrfs parity striping is still unusable. How many more decades will it take?

Re: Ceph: A Journey to 1 TiB/s

#167

Post by PiratesScorn »

brobinson wrote: post #114

I'm curious what the performance difference would be on a modern kernel.

For context, I’ve been leading the work on this cluster client-side (not the engineer that discovered the IOMMU fix) with Clyso.

There was no significant difference when testing between the latest HWE on Ubuntu 20.04 and kernel 6.2 on Ubuntu 22.04. In both cases we ran into the same IOMMU behaviour. Our tooling is all very much catered around Ubuntu so testing newer kernels with other distros just wasn’t feasible in the timescale we had to get this built. The plan was Awesome to see this on HN, we’re a pretty under-the-radar operation so there’s not much more I can say but proud to have worked on this!

Re: Ceph: A Journey to 1 TiB/s

#168

Post by louwrentius »

justinclift wrote: post #150
louwrentius wrote: post #149

I wrote an intro to Ceph[0] for those who are new to Ceph. It featured in a Jeff Geerling video briefly recently :-) [0]: Understanding Ceph: open-source scalable storage https://louwrentius.com/understanding-ceph-open-source-scala...

Has anything important changed since 2018, when you wrote that? :)

Conceptually not as far as I know.

Re: Ceph: A Journey to 1 TiB/s

#169

Post by PiratesScorn »

up2isomorphism wrote: post #166

This is an insanely expensive cluster built to show a benchmark. 68 node cluster serving only 15TB storage in total.

The purpose of the benchmarking was to validate the design of the cluster and to identify any issues before going into production, so it achieved exactly that objective. Without doing this work a lot of performance would have been left on the table before the cluster could even get out the door.

As per the blog, the cluster is now in a 6+2 EC configuration for production which gives ~7PiB usable. Expensive yes, but well worth it if this is the scale and performance required.

Re: Ceph: A Journey to 1 TiB/s

#170

Post by loeg »

matheusmoreira wrote: post #82
loeg wrote: post #6

Earlier quoted context omitted.

Why would you bother with a distributed filesystem when you don't have to?

I'm indifferent towards the distributed nature thing. What I want is ceph's ability to pool any combination of drives of any make, model and capacity into organized redundant fault tolerant storage, and its ability to add arbitrary drives to that pool at any point in the system's lifetime. RAID-like solutions require identical drives and can't be easily expanded.

ZFS and BtrFS have some capability for this.
Post reply on HN