Shameless plug. I did similar in my OS course. But, in C.
Github: https://github.com/immortal3/EbFS
Warning: Terribly written. many hacks.
Writing a file system from scratch in Rust
11–20 of 63 posts
Re: Writing a file system from scratch in Rust
#12I have read down to the implementation section, but for my money, this is the best way to describe the high level function and behavior of a filesystem that I have ever seen.
A very accessible (though dated) intro to filesystems is Practical File System Design, by Dominic Giampaolo. PDF link: http://www.nobius.org/practical-file-system-design.pdf
Is he the guy who did the BeOS filesystem?
Re: Writing a file system from scratch in Rust
#13It would be nice if the intro had a brief explanation of why a disk needs to be divided into blocks. Otherwise, I really enjoyed this read from the perspective of a lay person.
The disk / inodes need to know where to start looking for a file's contents, like the address in memory for RAM. Or like the mail: We subdivide by city, then ZIP, then street, then address. So the inode says "The data for my file starts at block 72 and is 3 blocks long" (or something like that). The disk then goes there, and reads blocks 72,73,74. Each block is 4KiB large often, so if you have a 10KiB file, you still…
This doesn't explain why blocks are valuable as you could use byte addressing. The reason why blocks are valuable is for similar reasons to why memory is divided into pages. (Which is all I am going to write, as I don't have the time to answer this well today. But "you need to know where things are on disk" isn't an answer.)
Re: Writing a file system from scratch in Rust
#14It would be nice if the intro had a brief explanation of why a disk needs to be divided into blocks. Otherwise, I really enjoyed this read from the perspective of a lay person.
> It would be nice if the intro had a brief explanation of why a disk needs to be divided into blocks.
One reason is that HDDs simply don't have a byte-wise resolution, so there's little point talking to HDDs in sub-sector units. Sectors are usually 512 bytes to 4k.
A second reason is being able to simply address the drive. Using 32b indices, if you index bytewise you're limited to 4GB which was available in the early 90s. With 512 bytes blocks, you get an addressing capacity of 2TB, and with 4k blocks (the AF format), you get 16TB. In fact I remember 'round the late 90s / early aught we'd reformat our drives using higher-size blocks because the base couldn't see the entire thing.
Re: Writing a file system from scratch in Rust
#15Earlier quoted context omitted.
The disk / inodes need to know where to start looking for a file's contents, like the address in memory for RAM. Or like the mail: We subdivide by city, then ZIP, then street, then address. So the inode says "The data for my file starts at block 72 and is 3 blocks long" (or something like that). The disk then goes there, and reads blocks 72,73,74. Each block is 4KiB large often, so if you have a 10KiB file, you still…
This doesn't explain why blocks are valuable as you could use byte addressing. The reason why blocks are valuable is for similar reasons to why memory is divided into pages. (Which is all I am going to write, as I don't have the time to answer this well today. But "you need to know where things are on disk" isn't an answer.)
This isn’t the reason about block addressing at all since it existed before paging was a thing. Addressing storage content in blocks is closely aligned with how direct access storage is organized in “sectors”, usually 512-byte blocks. Disks don’t have byte access interfaces, so it doesn’t make sense to use one.
It also lets you to address much larger data structures with limited sized variables. With block addressing, you can access terabytes of storage with a 32-bit integer.
Re: Writing a file system from scratch in Rust
#16It would be nice if the intro had a brief explanation of why a disk needs to be divided into blocks. Otherwise, I really enjoyed this read from the perspective of a lay person.
> It would be nice if the intro had a brief explanation of why a disk needs to be divided into blocks. One reason is that HDDs simply don't have a byte-wise resolution, so there's little point talking to HDDs in sub-sector units. Sectors are usually 512 bytes to 4k. A second reason is being able to simply address the drive. Using 32b indices, if you index bytewise you're limited to 4GB which was available in the earl…
> HDDs simply don't have a byte-wise resolution
Sufficient explanation for code. Now why is it that disks lack byte-wise resolution?
Re: Writing a file system from scratch in Rust
#17It would be nice if the intro had a brief explanation of why a disk needs to be divided into blocks. Otherwise, I really enjoyed this read from the perspective of a lay person.
A drive (be it HDD or SSD) is a block device, all drives are divided into blocks/sectors since they are a rather lossy medium. Data written to a bit may not necessarily be read accurately and so the solution is to have an Error-Correcting-Code that will allow to recover from some number of badly read bits (above that and you get a medium error from the drive). Since an ECC for a byte is a bit excessive the drives use blocks of 512 bytes or 4096 bytes, one reason for the move from 512 to 4096 is that the ECC becomes more efficient.
HDDs also have small "gaps" that have headers to locate the sectors (in the distant past you could do a low-level format to correct these gaps as well).
SSDs do not have these gaps but they do need the ECC.
Re: Writing a file system from scratch in Rust
#18frankly, It never helped on my resume but I enjoyed writing.
Frankly, if I saw this code on a resume, I'd keep looking.
scanf("%s",buffer);
// Debug :: printf("hash : %ld\n",hash("cdRoot"));
switch(hash(buffer))
{
//command : "ls"
case 5863588:Re: Writing a file system from scratch in Rust
#19Earlier quoted context omitted.
> It would be nice if the intro had a brief explanation of why a disk needs to be divided into blocks. One reason is that HDDs simply don't have a byte-wise resolution, so there's little point talking to HDDs in sub-sector units. Sectors are usually 512 bytes to 4k. A second reason is being able to simply address the drive. Using 32b indices, if you index bytewise you're limited to 4GB which was available in the earl…
> HDDs simply don't have a byte-wise resolution Sufficient explanation for code. Now why is it that disks lack byte-wise resolution?
.. And does it hold true for ram disks?
Re: Writing a file system from scratch in Rust
#20My dream is to add enough type parameters so in-memory collections can also work as (not horribly tuned!) on-disk datastructures.
It's a nice ambitious goal which can really drive language and library design.