Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing a file system from scratch in Rust

blog.carlosgaldino.com

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Writing a file system from scratch in Rust

#31

Post by mcpherrinm »

RealityVoid wrote: post #29
Immortal333 wrote: post #11

Shameless plug. I did similar in my OS course. But, in C. Github: https://github.com/immortal3/EbFS Warning: Terribly written. many hacks.

Soooo... how does it work? I'm not asking about the structure or how it's organized. I mean... is the filesystem in a file or... how? Background: I mostly do embedded stuff so at a glance I would have expected low level primitives (like, HW interactions, registers and stuff) but I see none. So maybe, my expectation, when tacking a problem, of interacting with the HW directly, does not stand in modern environments. Ev…

You'd presumably have some "block device" abstraction between your filesystem and your device driver. Don't want to re-implement a FS for each type of hardware. On a Linux system, you can read, eg, /dev/sda1 from userspace, which is what it looks like this filesystem probably does.

As for how you actually request data from the hard drive: There's older ATA interfaces, and BIOS routines from them, which I suspect is what most hobbyist OSes would use.

A more modern interface is AHCI. The OSDev wiki has an overview, where you can see how the registers work: https://wiki.osdev.org/AHCI

Re: Writing a file system from scratch in Rust

#32

Post by pkaye »

RealityVoid wrote: post #29
Immortal333 wrote: post #11

Shameless plug. I did similar in my OS course. But, in C. Github: https://github.com/immortal3/EbFS Warning: Terribly written. many hacks.

Soooo... how does it work? I'm not asking about the structure or how it's organized. I mean... is the filesystem in a file or... how? Background: I mostly do embedded stuff so at a glance I would have expected low level primitives (like, HW interactions, registers and stuff) but I see none. So maybe, my expectation, when tacking a problem, of interacting with the HW directly, does not stand in modern environments. Ev…

Looks like a filesystem in a file.

Re: Writing a file system from scratch in Rust

#33

Post by pkaye »

jagged-chisel wrote: post #16
masklinn wrote: post #14

Earlier quoted context omitted.

> It would be nice if the intro had a brief explanation of why a disk needs to be divided into blocks. One reason is that HDDs simply don't have a byte-wise resolution, so there's little point talking to HDDs in sub-sector units. Sectors are usually 512 bytes to 4k. A second reason is being able to simply address the drive. Using 32b indices, if you index bytewise you're limited to 4GB which was available in the earl…

> HDDs simply don't have a byte-wise resolution Sufficient explanation for code. Now why is it that disks lack byte-wise resolution?

Because of the sector based error correction. I remember some earlier 8-bit disks has 128 byte sectors. Some newer flash SSDs have 1k-4k physical sector size for ecc coding efficiency so in those cases they will internally do a read-modify-write to handle a 512 byte logical sector size.

Re: Writing a file system from scratch in Rust

#34

Post by monadic2 »

masklinn wrote: post #14
dm319 wrote: post #7

It would be nice if the intro had a brief explanation of why a disk needs to be divided into blocks. Otherwise, I really enjoyed this read from the perspective of a lay person.

> It would be nice if the intro had a brief explanation of why a disk needs to be divided into blocks. One reason is that HDDs simply don't have a byte-wise resolution, so there's little point talking to HDDs in sub-sector units. Sectors are usually 512 bytes to 4k. A second reason is being able to simply address the drive. Using 32b indices, if you index bytewise you're limited to 4GB which was available in the earl…

Well, one benefit would be a relaxation of constraints on filesystems. This might be worth performance loss.

Re: Writing a file system from scratch in Rust

#35

Post by keithnz »

RealityVoid wrote: post #29
Immortal333 wrote: post #11

Shameless plug. I did similar in my OS course. But, in C. Github: https://github.com/immortal3/EbFS Warning: Terribly written. many hacks.

Soooo... how does it work? I'm not asking about the structure or how it's organized. I mean... is the filesystem in a file or... how? Background: I mostly do embedded stuff so at a glance I would have expected low level primitives (like, HW interactions, registers and stuff) but I see none. So maybe, my expectation, when tacking a problem, of interacting with the HW directly, does not stand in modern environments. Ev…

as an aside, for our embedded system we use https://github.com/ARMmbed/littlefs for our flash file system, it has a bit of a description on its design and its copy on write system so that it can handle random power loss. Be nice to see some of these kinds of libraries done in Nim or Rust.

Re: Writing a file system from scratch in Rust

#36

Post by cesarb »

jagged-chisel wrote: post #16
masklinn wrote: post #14

Earlier quoted context omitted.

> It would be nice if the intro had a brief explanation of why a disk needs to be divided into blocks. One reason is that HDDs simply don't have a byte-wise resolution, so there's little point talking to HDDs in sub-sector units. Sectors are usually 512 bytes to 4k. A second reason is being able to simply address the drive. Using 32b indices, if you index bytewise you're limited to 4GB which was available in the earl…

> HDDs simply don't have a byte-wise resolution Sufficient explanation for code. Now why is it that disks lack byte-wise resolution?

> Now why is it that disks lack byte-wise resolution?

Overhead.

Each disk sector is not 512 bytes (or 4096 bytes in modern disks), it's a bit more. It needs at least the sector number (to detect when the head is flying over the correct sector), a CRC or error-correcting code, a gap of several hundred bits before the next sector (so writing to one sector won't overwrite the next one), and a preamble with an alternating pattern (to synchronize the analog to digital conversion). All this overhead would be a huge waste if the addressable unit were a single byte.

Re: Writing a file system from scratch in Rust

#37

Post by ravenstine »

Is there any advantage in writing a custom file system for a niche purpose? It seems like most file systems are just different variations of managing where/when files are written simultaneously. Could a file system written specifically for something like PostgreSQL cut out the middle-man and increase performance?

Re: Writing a file system from scratch in Rust

#38

Post by tene »

ravenstine wrote: post #37

Is there any advantage in writing a custom file system for a niche purpose? It seems like most file systems are just different variations of managing where/when files are written simultaneously. Could a file system written specifically for something like PostgreSQL cut out the middle-man and increase performance?

You may be interested in a paper written by the Ceph team: "File Systems Unfit as Distributed Storage Backends: Lessons from 10 Years of Ceph Evolution"

https://www.pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf

There are definitely some significant benefits you can get from managing your own storage, rather than using a filesystem.

Re: Writing a file system from scratch in Rust

#39

Post by [deleted] »

ravenstine wrote: post #37

Is there any advantage in writing a custom file system for a niche purpose? It seems like most file systems are just different variations of managing where/when files are written simultaneously. Could a file system written specifically for something like PostgreSQL cut out the middle-man and increase performance?

[deleted]

Re: Writing a file system from scratch in Rust

#40

Post by topspin »

ravenstine wrote: post #37

Is there any advantage in writing a custom file system for a niche purpose? It seems like most file systems are just different variations of managing where/when files are written simultaneously. Could a file system written specifically for something like PostgreSQL cut out the middle-man and increase performance?

Yes. Oracle has done this (ASM) to eliminate overhead, implement fault tolerance and provide a storage management interface based on SQL, for example.

I once made a 'file system' to mount cpio archives (read-only) in an embedded system. Cpio is an extremely simple format to generate and edit (in code) and mounting it directly was very effective.

Post reply on HN