Live data from Hacker News

Writing a file system from scratch in Rust

blog.carlosgaldino.com

21–30 of 63 posts

Re: Writing a file system from scratch in Rust

#21

frankly, It never helped on my resume but I enjoyed writing. Frankly, if I saw this code on a resume, I'd keep looking. scanf("%s",buffer); // Debug :: printf("hash : %ld\n",hash("cdRoot")); switch(hash(buffer)) { //command : "ls" case 5863588:

Everyone starts somewhere, and everyone has written far worse at some point.

Saying terribly unconstructive and disheartening things like that makes everyone, yourself included, feel bad. Do better.

Re: Writing a file system from scratch in Rust

#22

Earlier quoted context omitted.

> It would be nice if the intro had a brief explanation of why a disk needs to be divided into blocks. One reason is that HDDs simply don't have a byte-wise resolution, so there's little point talking to HDDs in sub-sector units. Sectors are usually 512 bytes to 4k. A second reason is being able to simply address the drive. Using 32b indices, if you index bytewise you're limited to 4GB which was available in the earl…

> HDDs simply don't have a byte-wise resolution Sufficient explanation for code. Now why is it that disks lack byte-wise resolution?

I believe for performance and error correction features for spinning hard drives. Check out the tables and graphic in the overview section here[1]. Any write to a sector requires updating the checksum, which means the whole sector has to be read before the drive can complete the write. Since the sector is probably in the kernel disk cache already, the whole sector can be flushed to disk at once and the drive can compute the checksum on the fly (instead of flushing 1 byte, making the drive do a read of the sector, recompute the checksum, and then do a write).

1: https://en.wikipedia.org/wiki/Advanced_Format

Re: Writing a file system from scratch in Rust

#23

Earlier quoted context omitted.

> It would be nice if the intro had a brief explanation of why a disk needs to be divided into blocks. One reason is that HDDs simply don't have a byte-wise resolution, so there's little point talking to HDDs in sub-sector units. Sectors are usually 512 bytes to 4k. A second reason is being able to simply address the drive. Using 32b indices, if you index bytewise you're limited to 4GB which was available in the earl…

> HDDs simply don't have a byte-wise resolution Sufficient explanation for code. Now why is it that disks lack byte-wise resolution?

Try designing the format keeing the required features in mind and you might seen why they have.

The hardware has to be able to read and write data from an arbitrary location and do some kind of checksum for error detection and they have to be able to efficiently transfer this data to the cpu.

Re: Writing a file system from scratch in Rust

#24
Always fun to see this type of work. I notice the usage of OsString, and it made me wonder: does the way an OS encodes it’s strings potentially make this FS non-portable between OSes? If I want to mount a drive formatted with this FS, would the OsString be potentially non-portable?

There was a lot of discussion in the past around TFS https://github.com/redox-os/tfs, my understanding is that effort has kinda lost steam.

Re: Writing a file system from scratch in Rust

#25
I recently wrote a very simple and naive filesystem in rust for a toy OS I'm building and it was quite an interesting thing to do: https://github.com/vinc/moros/blob/master/doc/filesystem.md

Then I implemented a little FUSE driver in Python to read the disk image from the host system and it was wonderful to mount it the first time and see the files! https://github.com/vinc/moros-fuse

Re: Writing a file system from scratch in Rust

#26

Always fun to see this type of work. I notice the usage of OsString, and it made me wonder: does the way an OS encodes it’s strings potentially make this FS non-portable between OSes? If I want to mount a drive formatted with this FS, would the OsString be potentially non-portable? There was a lot of discussion in the past around TFS https://github.com/redox-os/tfs , my understanding is that effort has kinda lost ste…

This is really cool, I wish someone would fund it.

Re: Writing a file system from scratch in Rust

#27
post #19

Earlier quoted context omitted.

> HDDs simply don't have a byte-wise resolution Sufficient explanation for code. Now why is it that disks lack byte-wise resolution?

.. And does it hold true for ram disks?

Yes it would, but you would be working in cache lines rather than disk blocks.

When working with RAM your PC will transfer a cache line of memory from RAM to your CPU caches. Again this is a feature of hardware limitations and trading off granularity against speed.

Your CPU operates many time faster than RAM so it makes sense to request multiple bytes at once, then your RAM can access those bytes, and line them up on it’s output pins ready for your CPU to come back and read them into cache. This give you better bandwidth. (It’s a little more complicated than this because the bytes are transferred in serial, rather than in parallel, but digging into the details here is tad tricky).

On the granularity point, the more granular your memory access pattern is, the more bits you need to address that memory. In RAM that either means more physical pins and motherboard traces, or more time to communicate that address. It also means more transistors are needed to hold that address in memory while you’re looking it up. All of those things mean more money.

And before we start looking at memory map units, that let you do magical things like map a 64 bit address space into only 16GB of physical RAM. Again the greater the granularity, the more transistors your MMU needs to store the mapping, and thus more cost.

So really the reason we don’t have bit level addressing granularity is ultimately cost. There’s no reason you could build a machine that did that, it would just cost a fortune and wouldn’t provide any practical benefits. So engineers made trade off to build something more useful.

Re: Writing a file system from scratch in Rust

#28

Earlier quoted context omitted.

> It would be nice if the intro had a brief explanation of why a disk needs to be divided into blocks. One reason is that HDDs simply don't have a byte-wise resolution, so there's little point talking to HDDs in sub-sector units. Sectors are usually 512 bytes to 4k. A second reason is being able to simply address the drive. Using 32b indices, if you index bytewise you're limited to 4GB which was available in the earl…

> HDDs simply don't have a byte-wise resolution Sufficient explanation for code. Now why is it that disks lack byte-wise resolution?

Cost, more resolution means more transistors, address lines, and protocol overhead (your memory address take up space too).

You could build a HDD with byte-wise resolution, which did a whole load of clever things internally to handle error correction (modern HDDs have far more complex error correction that just block wise checksums, they’re a piece of technological magic and analogue electronics).

But sure a device would cost far more, and offer no real benefits. Especially when you consider that reading a single byte in isolation is a very rare task, and it takes so long (compared to accessing a CPU cache) to find the data and transfer it that you might as well grab a whole load of bytes at the same time.

Byte-wise resolution in a HDD is like ordering rice by the grain in individual envelopes. Sure you could do it, but why the hell would you?

Re: Writing a file system from scratch in Rust

#29

Shameless plug. I did similar in my OS course. But, in C. Github: https://github.com/immortal3/EbFS Warning: Terribly written. many hacks.

Soooo... how does it work?

I'm not asking about the structure or how it's organized. I mean... is the filesystem in a file or... how?

Background: I mostly do embedded stuff so at a glance I would have expected low level primitives (like, HW interactions, registers and stuff) but I see none. So maybe, my expectation, when tacking a problem, of interacting with the HW directly, does not stand in modern environments.

Even better, but unrelated question... how the heck does a x86 OS request data from the HDD?

Re: Writing a file system from scratch in Rust

#30
post #12

Earlier quoted context omitted.

A very accessible (though dated) intro to filesystems is Practical File System Design, by Dominic Giampaolo. PDF link: http://www.nobius.org/practical-file-system-design.pdf

Is he the guy who did the BeOS filesystem?

Yes. Later went to Apple and did Spotlight.

https://en.wikipedia.org/wiki/Dominic_Giampaolo

Post reply on HN