Live data from Hacker News

Modern storage is plenty fast, but the APIs are bad

itnext.io

121–130 of 168 posts

Re: Modern storage is plenty fast, but the APIs are bad

#121
What I hate about Unix filesystems: the fact that you can't take a drive, put it in another computer and have permissions (user/group-ids) working instantly. Same for sharing over nfs.

Of course, people have tried to solve this, but I think not well enough. It's a huge amount of technical debt right there in the systems we use every day.

Re: Modern storage is plenty fast, but the APIs are bad

#122

What I hate about Unix filesystems: the fact that you can't take a drive, put it in another computer and have permissions (user/group-ids) working instantly. Same for sharing over nfs. Of course, people have tried to solve this, but I think not well enough. It's a huge amount of technical debt right there in the systems we use every day.

Because permissions is an OS-enforced feature, not the filesystem feature. If you have access to the drive, permissions are meaningless.

Re: Modern storage is plenty fast, but the APIs are bad

#123

What I hate about Unix filesystems: the fact that you can't take a drive, put it in another computer and have permissions (user/group-ids) working instantly. Same for sharing over nfs. Of course, people have tried to solve this, but I think not well enough. It's a huge amount of technical debt right there in the systems we use every day.

Because permissions is an OS-enforced feature, not the filesystem feature. If you have access to the drive, permissions are meaningless.

Permissions (and ownership info) can be useful even if you have complete access to a filesystem.

By the way, assume you have root permission. How would you replace a single file in a random tar-file, without changing any of the permissions/userids/groupids inside the tar-file? You can't untar it because the users inside the tar file don't correspond with the ones on your system. So, you'll have to use special tools, which is (only) one demonstration of the inadequacy of the permissions mechanism of our filesystems.

Re: Modern storage is plenty fast, but the APIs are bad

#124

Earlier quoted context omitted.

please ask my several NVMe devices to take notice ! actual performance under Linux OS is far less than that, here

I just tested my laptop with the Ubuntu benchmark tool on the partition editor. 3.5GB/s read on 100MB chunks.

> Ubuntu benchmark tool on the partition editor

Known as "GNOME Disks".

https://en.wikipedia.org/wiki/GNOME_Disks

Re: Modern storage is plenty fast, but the APIs are bad

#125
post #64
post #36

Earlier quoted context omitted.

> I expect the price of NVMe drives to drop over the next few years until they're cheap enough that the majority of computers are running NVMe drives. Price no longer has anything to do with it. PC OEMs are simply not shipping SATA SSDs any more, and major drive vendors have started to discontinue their client (OEM) SATA SSD product lines. We're just waiting for the SATA-based PC install base to be retired.

I have 8 SATA SSDs in my workstation; are there motherboards that could run a similar NVMe setup?

With PCIe lane bifurcation, you won’t even need a PCIe switch on your expansion card. I have 10 Samsung 980 Pro PCIe SSDs in my AMD ThreadRipper PRO/WX machine (2 in motherboard M.2 slots, and 2 x “expansion cards” that hold 4 SSDs each). Had to configure PCIe bifurcation in BIOS, so lanes connected to a PCIe x16 card will be treated like 4 x PCIe x4 instead.

So far the best aggregate results with io_uring 10.5 M 4K IOPS and 66.5 GB/s with large reads...

Re: Modern storage is plenty fast, but the APIs are bad

#126
post #57

Earlier quoted context omitted.

Yeah how many people are running apps on servers served at all or even partially by NVMe SSDs? Where I work for our on prem stuff it's basically all network storage.

This is starting to change a bit because of things like the DPUs that companies are making. Basically it's an intelligent PCI-e network bridge that lets you emulate/share PCI-e devices on the host while the actual hardware (NVMe storage, GPU, etc.) is located elsewhere. This lets you reconfigure the host in software without having to physically change the hardware in the servers itself. It also lets you change the wa…

The linked article states that DPUs behave like a PCIe device implementing the NVME protocol but instead of directly connected storage it can forward requests over network (fabric) via NVMeoF.

This doesn't look like a generic PCIe over network/fabric bridge. Did I misunderstand you or did I fail to locate that information in the linked article?

Re: Modern storage is plenty fast, but the APIs are bad

#127

Earlier quoted context omitted.

I think a lot of it depends what the machines are used for. I'm not actually the IT department, but I believe in my org, we started out with a SAN-backed high availability cluster, because the immediate initial need was getting basic infrastructure (wiki, source control, etc) off of a dedicated machine that was a single point of failure. But then down the road a different set of hosts were brought online that had fas…

I’m laughing a little bit because an old place I used to work had a similar setup. The SAN/NAS/whatever it was was pretty slow, provisioning VMs was slow, and as much as we argued that we didn’t need redundancy for a lot of our VMs (they were semi-disposable), the IT department refused to give us a fast non-redundant machine. And then one day the SAN blew up. Some kind of highly unlikely situation where more disks fa…

Lol. Hope may not be a strategy, but you need a strategy if you don't want to really learn that lession the hard way: if it hurts do it more often.

Re: Modern storage is plenty fast, but the APIs are bad

#128

Earlier quoted context omitted.

This is starting to change a bit because of things like the DPUs that companies are making. Basically it's an intelligent PCI-e network bridge that lets you emulate/share PCI-e devices on the host while the actual hardware (NVMe storage, GPU, etc.) is located elsewhere. This lets you reconfigure the host in software without having to physically change the hardware in the servers itself. It also lets you change the wa…

The linked article states that DPUs behave like a PCIe device implementing the NVME protocol but instead of directly connected storage it can forward requests over network (fabric) via NVMeoF. This doesn't look like a generic PCIe over network/fabric bridge. Did I misunderstand you or did I fail to locate that information in the linked article?

It's specifically NVMe over network, not PCIe over network.

Re: Modern storage is plenty fast, but the APIs are bad

#129
Because I am a "glue" programmer, and I realize that all storage options suck, I've decided to wait on any infrastructure choices for now, and just use the filesystem as a key-value store when developing my projects.

When I need indexing, I use SQLite, but I limit myself to very basic subset of SQL that would work in any of Oracle, Maria, Microsoft stores without changes.

Re: Modern storage is plenty fast, but the APIs are bad

#130
post #96
post #88

Earlier quoted context omitted.

is this with threadripper boards?

3 M.2 slots is common even on AMD's mainstream X570 and B550 platforms. I don't know if any of those motherboards also bundle riser cards for further M.2 PCIe SSDs, but they do support PCIe bifurcation so you can run your GPU at PCIe 4.0 x8 and use the second x16/x8 slot to run two more SSDs in a passive riser purchased separately.

The b550 Aorus Master I just got is unique in its bifurcation. It maps the CPU's gen 4 PCI-E to the main 16x slot and one M.2 4x, or drops the 16x slot down to 8x and then has three gen 4 M.2 4x. The other two PCI-E slots are gen 3 through the b550 chipset.

I chose this board for its absurdly overkill CPU voltage regulation. The pci-e configuration seems like a pretty good compromise, though.

Post reply on HN