Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing a file system from scratch in Rust

blog.carlosgaldino.com

Post reply on HN
Print view View original

51–60 of 63 posts

Re: Writing a file system from scratch in Rust

#51

Post by yencabulator »

jagged-chisel wrote: post #16
masklinn wrote: post #14

Earlier quoted context omitted.

> It would be nice if the intro had a brief explanation of why a disk needs to be divided into blocks. One reason is that HDDs simply don't have a byte-wise resolution, so there's little point talking to HDDs in sub-sector units. Sectors are usually 512 bytes to 4k. A second reason is being able to simply address the drive. Using 32b indices, if you index bytewise you're limited to 4GB which was available in the earl…

> HDDs simply don't have a byte-wise resolution Sufficient explanation for code. Now why is it that disks lack byte-wise resolution?

For what it's worth, this is slowly starting to change. https://venturebeat.com/2019/09/20/optane-persistent-memory-...

tl;dr This new tech might do to SSDs what SSDs did to hard drives. Might.

Re: Writing a file system from scratch in Rust

#53

Post by sedatk »

saurik wrote: post #50
sedatk wrote: post #15

Earlier quoted context omitted.

This isn’t the reason about block addressing at all since it existed before paging was a thing. Addressing storage content in blocks is closely aligned with how direct access storage is organized in “sectors”, usually 512-byte blocks. Disks don’t have byte access interfaces, so it doesn’t make sense to use one. It also lets you to address much larger data structures with limited sized variables. With block addressing…

(I am awake now, so I can provide a more useful answer to this question.) > This isn’t the reason about block addressing at all since it existed before paging was a thing. I didn't say "blocks exist because pages exist", I said "blocks exist for similar reasons to why pages exist". How you thought that required temporal causality is beyond me. :/ > Addressing storage content in blocks is closely aligned with how dire…

Yeah, I read your answer as if disk block addressing was directly related to paging. My bad.

Re: Writing a file system from scratch in Rust

#54

Post by brandmeyer »

RealityVoid wrote: post #29
Immortal333 wrote: post #11

Shameless plug. I did similar in my OS course. But, in C. Github: https://github.com/immortal3/EbFS Warning: Terribly written. many hacks.

Soooo... how does it work? I'm not asking about the structure or how it's organized. I mean... is the filesystem in a file or... how? Background: I mostly do embedded stuff so at a glance I would have expected low level primitives (like, HW interactions, registers and stuff) but I see none. So maybe, my expectation, when tacking a problem, of interacting with the HW directly, does not stand in modern environments. Ev…

> how the heck does a x86 OS request data from the HDD?

Entirely too short summary: Use PCI to discover the various devices attached to the CPU. One or more of them are AHCI or NVMe devices. The AHCI and NVMe standards each describe sets of memory-mapped configuration registers and DMA engines. Eventually, you get to a point where you can describe linked lists of transactions to be executed that are semantically similar to preadv, pwritev, and so on.

There's tons of info on osdev.org, such as https://wiki.osdev.org/AHCI

Re: Writing a file system from scratch in Rust

#55

Post by saagarjha »

peterkelly wrote: post #30
vondur wrote: post #12

Earlier quoted context omitted.

Is he the guy who did the BeOS filesystem?

Yes. Later went to Apple and did Spotlight. https://en.wikipedia.org/wiki/Dominic_Giampaolo

And APFS: https://developer.apple.com/videos/play/wwdc2016/701/

Re: Writing a file system from scratch in Rust

#56

Post by rrdharan »

RealityVoid wrote: post #29
Immortal333 wrote: post #11

Shameless plug. I did similar in my OS course. But, in C. Github: https://github.com/immortal3/EbFS Warning: Terribly written. many hacks.

Soooo... how does it work? I'm not asking about the structure or how it's organized. I mean... is the filesystem in a file or... how? Background: I mostly do embedded stuff so at a glance I would have expected low level primitives (like, HW interactions, registers and stuff) but I see none. So maybe, my expectation, when tacking a problem, of interacting with the HW directly, does not stand in modern environments. Ev…

https://en.m.wikipedia.org/wiki/INT_13H

Re: Writing a file system from scratch in Rust

#57

Post by anitil »

topspin wrote: post #40
ravenstine wrote: post #37

Is there any advantage in writing a custom file system for a niche purpose? It seems like most file systems are just different variations of managing where/when files are written simultaneously. Could a file system written specifically for something like PostgreSQL cut out the middle-man and increase performance?

Yes. Oracle has done this (ASM) to eliminate overhead, implement fault tolerance and provide a storage management interface based on SQL, for example. I once made a 'file system' to mount cpio archives (read-only) in an embedded system. Cpio is an extremely simple format to generate and edit (in code) and mounting it directly was very effective.

Wow this comment just made me fall down a rabbit hole. I've only just surfaced. The Kaitai project actually comes with some pre-defined bindings for cpio which meant I was up and running very quickly.

https://formats.kaitai.io/cpio_old_le/index.html

Re: Writing a file system from scratch in Rust

#59

Post by topspin »

anitil wrote: post #57
topspin wrote: post #40

Earlier quoted context omitted.

Yes. Oracle has done this (ASM) to eliminate overhead, implement fault tolerance and provide a storage management interface based on SQL, for example. I once made a 'file system' to mount cpio archives (read-only) in an embedded system. Cpio is an extremely simple format to generate and edit (in code) and mounting it directly was very effective.

Wow this comment just made me fall down a rabbit hole. I've only just surfaced. The Kaitai project actually comes with some pre-defined bindings for cpio which meant I was up and running very quickly. https://formats.kaitai.io/cpio_old_le/index.html

Yes, cpio is ancient and simple and very easy to work with. The Kaitai project didn't exist at the time; I used the C structs documented in man pages.

Re: Writing a file system from scratch in Rust

#60

Post by mlindner »

blackrock wrote: post #46
smt88 wrote: post #43

Earlier quoted context omitted.

> How soon until someone builds an Operating System developed in Rust? Redox[1] has been around for almost as long as Rust has. I first heard about it 4-5 years ago. They had an interesting competition a while back challenging people to figure out how to crash it. 1. https://www.redox-os.org/

Yeah, I heard about this project. But there’s a graveyard of dead OS projects out there. What’s the progress and potential of Redox?

Redox is being worked on continuously. They just added support for gdb.
Post reply on HN