Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ceph: A Journey to 1 TiB/s

ceph.io

Post reply on HN
Print view View original

191–200 of 216 posts

Re: Ceph: A Journey to 1 TiB/s

#191

Post by deadbunny »

matheusmoreira wrote: post #164
deadbunny wrote: post #101

Earlier quoted context omitted.

ZFS using mirrors is extremely easy to expand. Need more space and you have small drives? Replace the drives in a mirror one by one with bigger ones. Need more space and already have huge drives? Just add another vdev mirror. And the added benefit of not living in fear of drive failure while resilvering as it is much faster with mirrors than raidX. Sure the density isn't great as you're essentially running at 50% or…

> Replace the drives in a mirror one by one with bigger ones. That's exactly what I meant by "just as bad as RAID". Expanding an existing array is analogous to every single drive in the array failing and getting replaced with higher capacity drives. When a drive fails, the array is in a degraded state. Additional drive failures put the entire system in danger of data loss. The rebuilding process generates enormous I/…

That's not the case it mirrored vdevs. There is no degredatuon of the array with a failed drive in a mirrored vdev, it continues humming along perfectly fine.

Also resilvers are not as intensive when rebuilding a mirror as you are just copying from one disk in the vdev to the other, not all X other drives and recalculating parity at the same time. This means less reads across the entire array and much much quicker resilver times, thus less window for drive failure.

But don't just take my word for it. This is a blog post that go much into much more detail https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs...

Re: Ceph: A Journey to 1 TiB/s

#192

Post by yencabulator »

amluto wrote: post #67
Palomides wrote: post #60

Earlier quoted context omitted.

here's a weird calculation: this cluster does something vaguely like 0.8 gigabits per second per watt (1 terabyte/s * 8 bits per byte * 1024 gb per tb / 34 nodes / 300 watts a new mac mini (super efficient arm system) runs around 10 watts in interactive usage and can do 10 gigabits per second network, so maybe 1 gigabit per second per watt of data so OP's cluster, back of the envelope, is basically the same bits per…

I think the Mac Mini has massively more compute than needed for this kind of work. It also has a power supply, and computer power supplies are generally not amazing at low output. I’m imagining something quite specialized. Use a low frequency CPU with either vector units or even DMA engines optimized for the specific workloads needed, or go all out and arrange for data to be DMAed directly between the disk and the NI…

> or go all out and arrange for data to be DMAed directly between the disk and the NIC.

Ceph OSDs do a lot more work than you're imagining.

Re: Ceph: A Journey to 1 TiB/s

#193

Post by yencabulator »

louwrentius wrote: post #146
samcat116 wrote: post #94

Earlier quoted context omitted.

Latency is quite poor, I wouldn't recommend running high performance database loads there.

From my dated experience, Ceph is absolutely amazing but latency is indeed a relative weak spot. Everything has a trade-off and for Ceph you get a ton of capability but latency is such a trade-off. Databases - depending on requirements - may be better off on regular NVMe and not on Ceph.

It's pretty unfair to compare latency of a local NVMe SSD to over-the-network 3x replicated storage. "It's faster if I do less."

[Disclaimer: ex-Inktank employee]

Re: Ceph: A Journey to 1 TiB/s

#194

Post by wmf »

peter_d_sherman wrote: post #123
lmz wrote: post #98

Earlier quoted context omitted.

NFS root docs here https://www.kernel.org/doc/Documentation/filesystems/nfs/nfs...

NFS is an excellent point! NFS (now that I think about it!) -- brings up two additional software engineering considerations: 1) Distributed file system protocol . 2) Software that implements that distributed (or at least remote/network) file system -- via that file system protocol . NFS is both. That's not a bad thing(!) -- but ideally from a software engineering "separation of concerns" perspective, this future soft…

Are you familiar with the kernel virtual filesystem (VFS) layer?

Re: Ceph: A Journey to 1 TiB/s

#195

Post by pl4nty »

amluto wrote: post #30

I wish someone would try to scale the nodes down. The system described here is ~300W/node for 10 disks/node, so 30W or so per disk. That’s a fair amount of overhead, and it also requires quite a lot of storage to get any redundancy at all. I bet some engineering effort could divide the whole thing by 10. Build a tiny SBC with 4 PCIe lanes for NVMe, 2x10GbE (as two SFP+ sockets), and a just-fast-enough ARM or RISC-V C…

Nvidia's SODIMM compute module interface can prove this concept already. I have two 7W ARM Turing RK1s arriving soon, each with PCIe 3x4 at 4GB/s, and the Turing Pi 2 cluster board can fit four in an ITX form factor. I'm expecting over 3Gbps per watt at a total cost of 820USD

PCIe lanes are the bottleneck so far - even my $90 2TB SSDs are rated at 7GB/s on PCIe 4x4. So I don't think SBCs are the optimal solution yet. Looks like Ampere's Altra line can do PCIe 4x128 at 40W so a 1U blade with 100G networking could be interesting. I've seen lots of bugs and missing optimisations with ARM though, even in a homelab, so this kind of solution might not be ready for datacenters yet

Re: Ceph: A Journey to 1 TiB/s

#196

Post by matheusmoreira »

deadbunny wrote: post #191
matheusmoreira wrote: post #164

Earlier quoted context omitted.

> Replace the drives in a mirror one by one with bigger ones. That's exactly what I meant by "just as bad as RAID". Expanding an existing array is analogous to every single drive in the array failing and getting replaced with higher capacity drives. When a drive fails, the array is in a degraded state. Additional drive failures put the entire system in danger of data loss. The rebuilding process generates enormous I/…

That's not the case it mirrored vdevs. There is no degredatuon of the array with a failed drive in a mirrored vdev, it continues humming along perfectly fine. Also resilvers are not as intensive when rebuilding a mirror as you are just copying from one disk in the vdev to the other, not all X other drives and recalculating parity at the same time. This means less reads across the entire array and much much quicker re…

I see. That addresses my concerns, and it's starting to make a lot of sense. I'm gonna study this in depth, starting with the post you linked. Thank you.

Re: Ceph: A Journey to 1 TiB/s

#197

Post by matheusmoreira »

chromatin wrote: post #79
matheusmoreira wrote: post #72

Earlier quoted context omitted.

I just want to hoard data. I hate having to delete stuff to make space. Things disappear from the web every day. I should hold onto them. My requirements for a storage solution are: > Single root file system > Storage device failure tolerance > Gradual expansion capability The problem with every storage solution I've ever seen is the lack of gradual expandability. I'm not a corporation, I'm just a guy. I don't have t…

> ZFS is nearly perfect but when it comes to expanding capacity it's just as bad as RAID. if you don't mind the overhead of a "pool of mirrors" approach [1], then it is easy to expand storage by adding pairs of disks! This is how my home NAS is configured. [1] https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs...

Looks good. I don't mind the overhead. That seems to be much more resilient compared to RAID5/6 and its ZFS equivalents and addresses all the concerns I outlined in this comment:

https://news.ycombinator.com/item?id=39068493

I'm still somewhat alarmed by the possibility of the surviving mirror drive failing during resilver and destroying the pool... Are there any failure chance calculators for this pool of mirrors topology? No doubt it's much lower than the RAID5/6 setups but still.

Is this topology usable in btrfs without the famous reliability issues? How good is ZFS support is on Linux? I'm a Linux guy so I'd really like to keep using Linux if possible. Maybe Linux LVM/mdadm?

Re: Ceph: A Journey to 1 TiB/s

#198

Post by juiiiced »

markhpc wrote: post #156
mobilemidget wrote: post #152

Cool benchmark, and interesting, however it would have read a lot better if abbreviations are explained at first usage. Not everybody is familiar with all terminology used in the post. Nonetheless congrats with results.

Thanks (truly) for the feedback! I'll try to remember for future articles. It's easy to forget how much jargon we use after being in the field for so long.

Yea great post! Any post recommendations to clear up the acronyms?

Re: Ceph: A Journey to 1 TiB/s

#199

Post by speedgoose »

BlackLotus89 wrote: post #190
speedgoose wrote: post #42

Earlier quoted context omitted.

Also garage. https://garagehq.deuxfleurs.fr/

Garage seems to only to duplication https://garagehq.deuxfleurs.fr/documentation/design/goals/ > Storage optimizations: erasure coding or any other coding technique both increase the difficulty of placing data and synchronizing; we limit ourselves to duplication. This is probably a nogo for most use cases where you work with large datasets....

Yes I mentioned it in the context of homelabs, it’s not an alternative to Ceph or MinIO for large installations.

But the simplicity is appreciated. You can recover all the data using rsync.

Re: Ceph: A Journey to 1 TiB/s

#200

Post by brobinson »

PiratesScorn wrote: post #167
brobinson wrote: post #114

I'm curious what the performance difference would be on a modern kernel.

For context, I’ve been leading the work on this cluster client-side (not the engineer that discovered the IOMMU fix) with Clyso. There was no significant difference when testing between the latest HWE on Ubuntu 20.04 and kernel 6.2 on Ubuntu 22.04. In both cases we ran into the same IOMMU behaviour. Our tooling is all very much catered around Ubuntu so testing newer kernels with other distros just wasn’t feasible in…

Hey, thanks for the response! There have been a lot of across the board improvements in the kernel in the last four years so I'm surprised there's not a noticeable performance improvements in 6.2 (although I also consider 6.2 old at this point).
Post reply on HN