Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ceph: A Journey to 1 TiB/s

ceph.io

Post reply on HN
Print view View original

141–150 of 216 posts

Re: Ceph: A Journey to 1 TiB/s

#141

Post by camkego »

rglullis wrote: post #33
nullwarp wrote: post #5

Earlier quoted context omitted.

There still are. As someone who has done both production and homelab deployments: unless you are specifically just looking for experience with it and just setting up a demo - don't bother. When it works, it works great - when it goes wrong it's a huge headache. Edit: As just an edit, if distributed storage is just something you are interested in there are much better options for a homelab setup: - seaweedfs has been…

> glusterfs is still fine as long as you know what you are going into. Does that include storage volumes for databases? I was using glusterFS as a way to scale my swarm cluster horizontally and I am reasonably sure that it corrupted one database to the point I lost more than a few hours of data. I was quite satisfied with the setup until I hit that. I know that I am considered crazy for sticking with Docker Swarm unt…

Docker Swarm seems to be underrated, from a simplicity and reliability perspective, IMHO.

Re: Ceph: A Journey to 1 TiB/s

#142

Post by eurekin »

victorhooi wrote: post #24
matheusmoreira wrote: post #4

Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.

I have some experience with Ceph, both for work, and with homelab-y stuff. First, bear in mind that Ceph is a distributed storage system - so the idea is that you will have multiple nodes. For learning, you can definitely virtualise it all on a single box - but you'll have a better time with discrete physical machines. Also, Ceph does prefer physical access to disks (similar to ZFS). And you do need decent networking…

NUC with 10gbit eth - can you recommend any?

Re: Ceph: A Journey to 1 TiB/s

#143

Post by eurekin »

progval wrote: post #136
eurekin wrote: post #135

Earlier quoted context omitted.

Looks so good, wish for a > 1gbit version, since HDDs alone can saturate that

Did you look at their H3? It's pricier but it has two 2.5Gbits ports (along with a NVMe slot and an Intel CPU)

I have one and love it! It bravely holds together my intranet dev services :)

For a ceph node would still consider a version with 10gbit eth

Re: Ceph: A Journey to 1 TiB/s

#144

Post by louwrentius »

patrakov wrote: post #130
antongribok wrote: post #122

Earlier quoted context omitted.

Not sure about putting DBs on CephFS directly, but Ceph RBD can definitely run RDBMS workloads. You need to pay attention to the kind of hardware you use, but you can definitely get Ceph down to 0.5-0.6 ms latency on block workloads doing single thread, single queue, sync 4K writes. Source, I run Ceph at work doing pretty much this.

It is important to specify which kind of latency percentile this is. Checking on a customer's cluster (made from 336 SATA SSDs in 15 servers, so not the best one in the world): 50th percentile = 1.75 ms 90th percentile = 3.15 ms 99th percentile = 9.54 ms That's with 700 MB/s of reads and 200 MB/s of writes, or approximately 7000 reads IOPS and 9000 writes IOPS.

These numbers may be good enough for your use case but from what’s possible with SSDs these numbers aren’t great. Please note, I mean well. Still a cool setup.

I’d like to see much more latency consistency and 99th even sub ms. Might want to set a latency target with fio and see what kind of load is possible until 99 hits 1ms.

However, I can say all of this but it’s all about context and depending on workload your figures may be totally fine.

Re: Ceph: A Journey to 1 TiB/s

#145

Post by narism »

somat wrote: post #109
amluto wrote: post #30

I wish someone would try to scale the nodes down. The system described here is ~300W/node for 10 disks/node, so 30W or so per disk. That’s a fair amount of overhead, and it also requires quite a lot of storage to get any redundancy at all. I bet some engineering effort could divide the whole thing by 10. Build a tiny SBC with 4 PCIe lanes for NVMe, 2x10GbE (as two SFP+ sockets), and a just-fast-enough ARM or RISC-V C…

I have always wanted to set up a ceph system with one drive per node. The ideal form factor would be a drive with a couple network interfaces built in. western digital had a press release about an experiment they did that was exactly this, but it never ended up with drive you could buy. The hardkernel HC2 SOC was a nearly ideal form factor for this, and I still have a stack of them laying around that I bought to make…

Sounds like the Seagate Kinetic HDD: https://www.seagate.com/www-content/product-content/hdd-fam/...

Re: Ceph: A Journey to 1 TiB/s

#146

Post by louwrentius »

samcat116 wrote: post #94
one_buggy_boi wrote: post #93

Is modern Ceph appropriate for transactional database storage, how is the IO latency? I'd like to move to a cheaper cfs that can compete with systems like Oracle's clustered file system or DBs backed by something like Veritas. Veritas supports multi-petabyte DBs and I haven't seen much outside of it or ocfs that similarly scales with acceptable latency

Latency is quite poor, I wouldn't recommend running high performance database loads there.

From my dated experience, Ceph is absolutely amazing but latency is indeed a relative weak spot.

Everything has a trade-off and for Ceph you get a ton of capability but latency is such a trade-off. Databases - depending on requirements - may be better off on regular NVMe and not on Ceph.

Re: Ceph: A Journey to 1 TiB/s

#147

Post by louwrentius »

rafaelturk wrote: post #99

I'm playing a lot with MicroCeph. Its aopinionated low TOS, friendly setup of Ceph. Looking forward additional comments. Planning to use it in production and replace lots of NAS servers.

I think Ceph can be fine for NAS use cases, but be wary of latency and do some benchmarking. You may need more nodes/osds than you think to reach latency and throughput targets.

Re: Ceph: A Journey to 1 TiB/s

#148

Post by sekh60 »

matheusmoreira wrote: post #72
nullwarp wrote: post #5

Earlier quoted context omitted.

There still are. As someone who has done both production and homelab deployments: unless you are specifically just looking for experience with it and just setting up a demo - don't bother. When it works, it works great - when it goes wrong it's a huge headache. Edit: As just an edit, if distributed storage is just something you are interested in there are much better options for a homelab setup: - seaweedfs has been…

I just want to hoard data. I hate having to delete stuff to make space. Things disappear from the web every day. I should hold onto them. My requirements for a storage solution are: > Single root file system > Storage device failure tolerance > Gradual expansion capability The problem with every storage solution I've ever seen is the lack of gradual expandability. I'm not a corporation, I'm just a guy. I don't have t…

I've run Ceph at home since the jewel release. I migrated to it after running FreeNAS.

I use it for RBD volumes for my OpenStack cluster and for CephFS. With a total raw capacity of around 350TiB. Around 14 of that is nvme storage for RBD and CephFS metadata. The rest is rust. This is spread across 5 nodes.

I currently am only buying 20TB exos drives for rust. SMR and I think HSMR are both no goes for Ceph as are non enterprise SSDs, so storage is expensive. Ibdinhave a mix of disks though as the cluster has grown organically. So I have a few 6TB WD Reds in there, before their SMR shift.

My networks for OpenStack, Ceph and Ceph backend are all 10Gbps. With the flash storage when repairing I get about 8GiB/s a second. With rust it is around 270MiB/s. The bottle neck I think is due to 3 of the nodes running on first gen xeon-d boards, the the few Reds do slow things down too. The 4th node runs an AMD Rome CPU, and the newest an AMD Genoa cpu. So I am looking at about 5k CAD a node before disks. I colocate the MDS, OSDs and MONs, with 64GiB of ram each. Each node gets 6 rust, and 2 nvme drives.

Complexity is pretty simple. I deployed the initial iteration by hand, and then when cephadmin was released i converted it daemon by daemon smoothly. I find on the mailing lists and Reddit most of the people encountering problems deploy it via Proxmox and don't really understand Ceph because of it.

Re: Ceph: A Journey to 1 TiB/s

#150

Post by justinclift »

louwrentius wrote: post #149

I wrote an intro to Ceph[0] for those who are new to Ceph. It featured in a Jeff Geerling video briefly recently :-) [0]: Understanding Ceph: open-source scalable storage https://louwrentius.com/understanding-ceph-open-source-scala...

Has anything important changed since 2018, when you wrote that? :)
Post reply on HN