Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ceph: A Journey to 1 TiB/s

ceph.io

Post reply on HN
Print view View original

151–160 of 216 posts

Re: Ceph: A Journey to 1 TiB/s

#151

Post by justinclift »

eurekin wrote: post #142
victorhooi wrote: post #24

Earlier quoted context omitted.

I have some experience with Ceph, both for work, and with homelab-y stuff. First, bear in mind that Ceph is a distributed storage system - so the idea is that you will have multiple nodes. For learning, you can definitely virtualise it all on a single box - but you'll have a better time with discrete physical machines. Also, Ceph does prefer physical access to disks (similar to ZFS). And you do need decent networking…

NUC with 10gbit eth - can you recommend any?

If you want something cheap, you could go with Lenovo M720q's:

https://www.servethehome.com/lenovo-thinkcentre-m720q-tinymi...

They have a PCIe slot and can take 8th/9th gen intel cpus (6 core, etc). That PCIe slot should let you throw in a decent network card (eg 10GbE, 25GbE, etc).

Re: Ceph: A Journey to 1 TiB/s

#153

Post by chx »

fiddlerwoaroof wrote: post #106
chx wrote: post #92

There was a point in history when the total amount of digital data stored worldwide reached 1TiB for the first time. It is extremely likely this day was within the last sixty years. And here we are moving that amount of data every second on the servers of a fairly random entity. We not talking of a nation state or a supranatural research effort.

It’s at least 20ish years ago: I remember an old sysadmin talking about managing petabytes before 2003

I raised this to retro se and https://retrocomputing.stackexchange.com/a/28322/3722 notes a TiB of digital data likely was reached in the 1930s with punch cards.

Re: Ceph: A Journey to 1 TiB/s

#154

Post by nijave »

matheusmoreira wrote: post #88
bityard wrote: post #86

Earlier quoted context omitted.

On a single host, you could do this with LVM. Add a pair of disks, make them a RAID 1, create a physical volume on them, then a volume group, then a logical volume with XFS on top. To expand, you add a pair of disks, RAID 1 them, and add them to the LVM. It's a little stupid, but it would work. If multiple nodes are not off the table, also look into seaweedfs. Also consider how (or if) you are going to back up your h…

> Also consider how (or if) you are going to back up your hoard of data. I actually emailed backblaze years ago about their supposedly unlimited consumer backup plan. Asked them if they would really allow me to dump into their systems dozens of terabytes of encrypted undeduplicable data. They responded that yes, they would. Still didn't believe them, these corporations never really mean it when they say unlimited. Pl…

> these corporations never really mean it when they say unlimited. Plus they had no Linux software

Afaik they rely on the latter to mitigate the risk of the former.

Re: Ceph: A Journey to 1 TiB/s

#155

Post by nijave »

matheusmoreira wrote: post #72
nullwarp wrote: post #5

Earlier quoted context omitted.

There still are. As someone who has done both production and homelab deployments: unless you are specifically just looking for experience with it and just setting up a demo - don't bother. When it works, it works great - when it goes wrong it's a huge headache. Edit: As just an edit, if distributed storage is just something you are interested in there are much better options for a homelab setup: - seaweedfs has been…

I just want to hoard data. I hate having to delete stuff to make space. Things disappear from the web every day. I should hold onto them. My requirements for a storage solution are: > Single root file system > Storage device failure tolerance > Gradual expansion capability The problem with every storage solution I've ever seen is the lack of gradual expandability. I'm not a corporation, I'm just a guy. I don't have t…

Not sure what the multidisk consensus is for btrfs now-a-days but adding/removing devices is trivial, you can do "offline" dedupe, and you can rebalance data if you change the disk config.

As an added bonus it's also in-tree so you don't have to worry about kernel updates breaking things

I think you can also potentially do btrfs+LVM and let LVM manage multi device. Not sure what performance looks like there, though

Re: Ceph: A Journey to 1 TiB/s

#156

Post by markhpc »

mobilemidget wrote: post #152

Cool benchmark, and interesting, however it would have read a lot better if abbreviations are explained at first usage. Not everybody is familiar with all terminology used in the post. Nonetheless congrats with results.

Thanks (truly) for the feedback! I'll try to remember for future articles. It's easy to forget how much jargon we use after being in the field for so long.

Re: Ceph: A Journey to 1 TiB/s

#157

Post by sgarland »

sixdonuts wrote: post #111
sgarland wrote: post #61

Earlier quoted context omitted.

Yes. I first tried it with Rook, and that was a disaster, so I shifted to Longhorn. That has had its own share of problems, and is quite slow. Finally, I let Proxmox manage Ceph for me, and it’s been a dream. So far I haven’t migrated my K8s workloads to it, but I’ve used it for RDBMS storage (DBs in VMs), and it works flawlessly. I don’t have an incredibly great setup, either: 3x Dell R620s (Ivy Bridge-era Xeons), a…

Just get a 25G switch and MM fiber. 25G switches are cheaper, use less power and can work with 10 and 25G SFPs.

The main blocker (other than needing to buy new NICs, since everything I have already came with quad 1/1/10/10) is I'm heavily invested into the Ubiquiti ecosystem, and since they killed off the USW-Leaf (and the even more brief UDC-Leaf), they don't have anything that fits the bill.

I'm not entirely opposed to getting a Mikrotik or something and it just being the oddball out, but it's nice to have everything centrally managed.

EDIT: They do have the PRO-Aggregation, but there are only 4x 25G ports. Technically it _would_ meet my needs for Ceph, and Ceph only.

Re: Ceph: A Journey to 1 TiB/s

#158

Post by markhpc »

mrb wrote: post #40

I wanted to see how 1 TiB/s compares to the actual theoretical limits of the hardware. So here is what I found: The cluster has 68 nodes, each a Dell PowerEdge R6615 ( https://www.delltechnologies.com/asset/en-us/products/server... ). The R6615 configuration they run is the one with 10 U.2 drive bays. The U.2 link carries data over 4 PCIe gen4 lanes. Each PCIe lane is capable of 16 Gbit/s. The lanes have negligible ~…

I wanted to chime in and mention that we've never seen any issues with IOMMU before in Ceph. We have a previous generation of the same 1U chassis from Dell with AMD Rome processors in the upstream ceph lab and they don't suffer from the same issue despite performing similarly at the same scale (~30 OSDs). The customer did say they've seen this in the past in their data center. I'm hoping we can work with AMD to figure out what's going on.

I did some work last summer kind of duct taping the OSD's existing threading model (double buffering the hand-off between async msgr and worker threads, adaptive thread wakeup, etc). I could achieve significant performance / efficiency gains under load, but at the expense of increased low-load latency (Ceph by default is very aggressive about waking up threads when new IO arrives for a given shard).

One of the other core developers and I discussed it and we both came to the conclusion that it probably makes sense to do a more thorough rewrite of the threading code.

Re: Ceph: A Journey to 1 TiB/s

#159

Post by Snow_Falls »

matheusmoreira wrote: post #72
nullwarp wrote: post #5

Earlier quoted context omitted.

There still are. As someone who has done both production and homelab deployments: unless you are specifically just looking for experience with it and just setting up a demo - don't bother. When it works, it works great - when it goes wrong it's a huge headache. Edit: As just an edit, if distributed storage is just something you are interested in there are much better options for a homelab setup: - seaweedfs has been…

I just want to hoard data. I hate having to delete stuff to make space. Things disappear from the web every day. I should hold onto them. My requirements for a storage solution are: > Single root file system > Storage device failure tolerance > Gradual expansion capability The problem with every storage solution I've ever seen is the lack of gradual expandability. I'm not a corporation, I'm just a guy. I don't have t…

If you're willing to use mirror vdevs, expansions can be done two drives at a time.Also, depending on how often your data changes, you should check out snapraid. Doesn't have all the features of ZFS but its perfect for stuff that rarely changes (media or, in your case, archiving).

Also unionfs or similar can let you merge zfs and snapraid into one unified filesystem so you can place important data in zfs and unchanging archive data in snapraid.

Re: Ceph: A Journey to 1 TiB/s

#160

Post by Snow_Falls »

chromatin wrote: post #79
matheusmoreira wrote: post #72

Earlier quoted context omitted.

I just want to hoard data. I hate having to delete stuff to make space. Things disappear from the web every day. I should hold onto them. My requirements for a storage solution are: > Single root file system > Storage device failure tolerance > Gradual expansion capability The problem with every storage solution I've ever seen is the lack of gradual expandability. I'm not a corporation, I'm just a guy. I don't have t…

> ZFS is nearly perfect but when it comes to expanding capacity it's just as bad as RAID. if you don't mind the overhead of a "pool of mirrors" approach [1], then it is easy to expand storage by adding pairs of disks! This is how my home NAS is configured. [1] https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs...

50% storage efficiency is a tough pill to swallow, but drives are pretty big and the ability to expand as you go means it can be cheaper in the long run to just buy the larger, new drives coming out than pay upfront for a bunch of drives in a raidz config.
Post reply on HN