Brilliant, but I wonder now that we know how filesystems work if we could redesign ceph to do the right thing. For instance a lot of work was made to schedule the important writes at the right time. Perhaps they could have handled these latency issues explicitly.
File systems unfit as distributed storage back ends: 10 years of Ceph
31–37 of 37 posts
Re: File systems unfit as distributed storage back ends: 10 years of Ceph
#32Was it Ceph they were using at CERN (ATLAS Project, at least?) they were using some kind of file system federation.
Re: File systems unfit as distributed storage back ends: 10 years of Ceph
#33Ceph is awesome, even years ago it was a great Technology. We from croit.io do provide a free software to manage Ceph with ease.
How does it compare to Rook? We use rook to manage ceph in our on prem Kubernetes cluster and it is excellent
Re: File systems unfit as distributed storage back ends: 10 years of Ceph
#34This all makes me grateful that I use sqlite3 instead of FS for storage, even for fairly trivial projects.
Could you expand a little on how you're doing that? I've been thinking about transitioning entirely to sqlite for all my data.
The GP may seem like sarcasm to some ... sqlite is an overlooked, novel, and faster way (up to 35%!) to store things than the filesystem [0].
You can use something like libsqlfs [1] for POSIX file heuristics with sqlite as the backing store.
One HA single primary/multi-master solution to use sqlite may be drbd.
[0] https://www.sqlite.org/fasterthanfs.html [1] https://github.com/guardianproject/libsqlfs
Re: File systems unfit as distributed storage back ends: 10 years of Ceph
#35Yes it really cannot emphasized enough that the legacy filesystem system interface with it's too-simple 1970s origin and then far, far, far too complex decades of duck tape is a disasterous albatros. C.f. What linus is saying in https://news.ycombinator.com/item?id=21673372 except turn it around. When an interface has devolved into two sides hating and Postel's-law-enabling each other ad infinitum, and a statement li…
Do you have an opinion on whether filesystem in Windows is a comparable mess?
I don't, but I imagine it is no better. The VMS NT people had good intentions but both ecosystems are smothered in backwards compatibility and Postel's law issues.
Re: File systems unfit as distributed storage back ends: 10 years of Ceph
#36Was it Ceph they were using at CERN (ATLAS Project, at least?) they were using some kind of file system federation.
On one of the bigger experiments we use quite a few things:
- AFS as a federated posix file system for user home directories. My impression is that a distributed posix filesystem is... well... hard, for basically the reasons listed in the link. We're actually trying to phase it out, starting by reducing the size of the federation by cutting off access outside the CERN network.
- A few in-house developments like xrootd [1] (basically CERN's version of an object store) and EOS (a posix file system built on top), to store data. These projects have their roots in a time when CERN was at the forefront of "big data" and it made sense to develop an in-house project. These days there are a number of alternatives and my impression is that the reasons for continuing the projects are mostly historical.
- For read-only data we have cvmfs [2], a FUSE module which is synced to some other file system a few times a day. Making it read-only simplifies the metadata handling considerably: it's actually quite nice for a CERN project.
- Some people have started using Ceph for more experimental things, but in general these "industry" projects are only starting to replace the home-grown ones.
Re: File systems unfit as distributed storage back ends: 10 years of Ceph
#37This all makes me grateful that I use sqlite3 instead of FS for storage, even for fairly trivial projects.
Could you expand a little on how you're doing that? I've been thinking about transitioning entirely to sqlite for all my data.
There's no hard formula. It's basically forming a habit to never use file I/O and use SQL queries instead.
Bonus points for the `lsm1` extension of sqlite3 which allows you to use it as a key-value store, which I used with mixed success (if I could remember the key names that seemed the most logical thing in the world last week, lol).
There's nothing to it, really. sqlite3 is a very mature software and save for a mechanical failure of your storage drive, the odds of it losing your data are practically zero.
For even more bonus points, encrypt your sqlite3 storage. That way you can freely distribute it on Git hosting services.