Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Interview with Andreas Kling of Serenity OS (2022)

corecursive.com

Post reply on HN
Print view View original

31–40 of 181 posts

Re: Interview with Andreas Kling of Serenity OS (2022)

#31

Post by saagarjha »

josephg wrote: post #2

I'm increasingly tempted to try my hand at making a toy OS - or at least a kernel. There's a few things I desperately want an operating system to do: 1. Have either database-like transactions or (at a minimum) write barriers for file operations. The current filesystem semantics are terrible for databases and other applications which need to not corrupt their data after a crash. fsync() is too heavy, because you can't…

One problem with 2. is that subscribing to changes isn't always good enough, since if you go to act on the changes things may update out from under you regardless. Like, if you react to filesystem events, if a new file is created you might have it get deleted before you even register that it was added and do whatever you wanted with it.

Re: Interview with Andreas Kling of Serenity OS (2022)

#32

Post by alberth »

https://www.redox-os.org/

Redox is a super interesting project.

Not because it’s Rust based.

But because rarely do people start from scratch to build a new OS.

(eg, OpenBSD forked NetBSD, Dragonly forked FreeBSD, Ubuntu from Debian, etc)

—-

https://fuchsia.dev/

Would be another of interest.

Re: Interview with Andreas Kling of Serenity OS (2022)

#33

Post by crimsontech »

mysterydip wrote: post #25
torginus wrote: post #16

I just wonder how much of the truly exceptional engineering in today's software was done by people on drugs (yes, 'ADHD' medication counts). I fully expect it to be a significant chunk, but I hope it's not the majority.

> 'ADHD' I'm confused by the use of quotes. Do you think ADHD is a made-up thing?

I took it to mean people using it when they don’t have ADHD.

Re: Interview with Andreas Kling of Serenity OS (2022)

#34

Post by josephg »

shiroiuma wrote: post #15
josephg wrote: post #13

Earlier quoted context omitted.

Lots - maybe most - applications need to essentially run their own mini database to store user data. I obviously don't want a partition per application, or per word document. Abstracting away your hard disk is literally the job of your filesystem.

I don't mean that some little SQLite DB for your web browser's storage and settings needs its own partition; that's obviously going too far. I mean for really huge databases, where an entire server is dedicated to running that DB and it has high performance needs.

Fair enough. But I am including all of those little databases.

I agree with you - you could hack around this for a big database by putting it on a dedicated partition (even though you shouldn't need to). But I'm also thinking about applications with little databases that need to live inside the filesystem. They shouldn't need to embed sqlite in order to survive a crash.

A different syscall API would allow you to write a small, fast, efficient database engines (like redis or mysql) without needing all of the complex, performance gobbling tricks that these databases currently have.

A better filesystem API would help databases everywhere.

Re: Interview with Andreas Kling of Serenity OS (2022)

#36

Post by mysterydip »

crimsontech wrote: post #33
mysterydip wrote: post #25

Earlier quoted context omitted.

> 'ADHD' I'm confused by the use of quotes. Do you think ADHD is a made-up thing?

I took it to mean people using it when they don’t have ADHD.

Ah that makes sense, thanks!

Re: Interview with Andreas Kling of Serenity OS (2022)

#37

Post by 0x0203 »

sph wrote: post #6
josephg wrote: post #2

I'm increasingly tempted to try my hand at making a toy OS - or at least a kernel. There's a few things I desperately want an operating system to do: 1. Have either database-like transactions or (at a minimum) write barriers for file operations. The current filesystem semantics are terrible for databases and other applications which need to not corrupt their data after a crash. fsync() is too heavy, because you can't…

Do it. Ignore whoever says it's too complicated. You probably won't supplant Linux with your toy OS for your own research, but for a software engineer, writing a OS is the closest thing we have to the exhilarating feeling of seeing your code affect real hardware. You haven't lived until a bug of your own causes a triple-fault exception that causes a literal system shutdown. I started this career writing my own tiny O…

> You haven't lived until a bug of your own causes a triple-fault exception that causes a literal system shutdown.

On the OS I work on, I deliberately added a triple fault trigger as a fallback method of rebooting the system if the "normal" methods fail, which I have seen before.

To anyone looking to play with writing their own OS, definitely just do it. It's a lot of fun, and there's almost no limit to how much you can learn. With QEMU and bochs, and VirtualBox, etc... it's not that hard getting started. If you want to play around without starting from scratch, but find Linux source too complicated, give OpenBSD a look. Not that Linux is that hard, but it takes some getting used to how they do things. I find OpenBSD considerably more approachable.

Re: Interview with Andreas Kling of Serenity OS (2022)

#38

Post by rkagerer »

shiroiuma wrote: post #7
actionfromafar wrote: post #4

Earlier quoted context omitted.

Some of the issues can be improved "hackily" by for instance giving a database software exclusive access to two discrete disks - one for the db file, one for the journal.

Why is this "hacky"? I've long wondered why databases need to be on filesystems, other than simply convenience. If you think about it, a filesystem is, itself, basically a database (not a relational one of course). So it seems like you could improve performance a lot by eliminating the filesystem layer and going straight to direct disk access. Also, how this would be done would probably change depending on whether yo…

I'd like to go the other way, and have a filesystem that leverages a proper database for its metadata.

I can search millions of records in a split second in SQL. But searching records on my hard disk takes enormously more time. The solutions out there for speeding it up (like Windows Indexing service) rely on asynchronous indexing that's patched on top of the filesystem instead of integrated realtime within it, and are subsequently brittle (in my experience), slow down your computer at inconvenient times, and too often aren't instantly up date.

Being able to have stored procedures execute when changes trigger would be another interesting aspect that could take the place of event filters and file watchers (especially if critical portions could be configured to happen sequentially & atomically, akin to IRQ handlers).

I feel like databases have gotten lots of love over the decades and are super optimized (and by this point are pretty dang reliable). Filesystems feel like they lost their time in the spotlight and have floundered with less innovation.

Hardware that calculates CRC's of blocks in realtime would also be awesome (imagine what it would do for sync). The ironic thing is your HDD already does this, there's just no way to tap into that information at the software / filesystem driver level.

Re: Interview with Andreas Kling of Serenity OS (2022)

#39

Post by throw10920 »

josephg wrote: post #2

I'm increasingly tempted to try my hand at making a toy OS - or at least a kernel. There's a few things I desperately want an operating system to do: 1. Have either database-like transactions or (at a minimum) write barriers for file operations. The current filesystem semantics are terrible for databases and other applications which need to not corrupt their data after a crash. fsync() is too heavy, because you can't…

> It would also be interesting to try making all userspace programs be wasm bundles and run them in ring0. But I suspect that will run slower on modern hardware compared to using native binaries and context switching.

If you have all of your programs reside in the same virtual memory space (which is the implication with running them in ring 0), then you get a unique performance advantage in that you don't have to invalidate the TLB. I have no idea how much the amortized performance impact of virtual memory is, unfortunately, so the gains might be trivial, but who knows.

Re: Interview with Andreas Kling of Serenity OS (2022)

#40

Post by jasode »

rkagerer wrote: post #38
shiroiuma wrote: post #7

Earlier quoted context omitted.

Why is this "hacky"? I've long wondered why databases need to be on filesystems, other than simply convenience. If you think about it, a filesystem is, itself, basically a database (not a relational one of course). So it seems like you could improve performance a lot by eliminating the filesystem layer and going straight to direct disk access. Also, how this would be done would probably change depending on whether yo…

I'd like to go the other way, and have a filesystem that leverages a proper database for its metadata. I can search millions of records in a split second in SQL. But searching records on my hard disk takes enormously more time. The solutions out there for speeding it up (like Windows Indexing service) rely on asynchronous indexing that's patched on top of the filesystem instead of integrated realtime within it, and a…

>I'd like to go the other way, and have a filesystem that leverages a proper database for its metadata.

Microsoft made an attempt back in 2003 at creating a "smarter" file system with rich metadata based on a real database engine (MS SQL Server) ... but eventually abandoned the project: https://en.wikipedia.org/wiki/WinFS

WinFS wasn't necessarily going to completely replace NTFS but Microsoft did have ambitious plans for its integration into everything.

It's interesting that other platforms like Unix/Linux/Apple that were designing "next generation" file systems like ZFS/ext4/APFS after the failed WinFS project had a chance to try adding their own WinFS smart metadata features to but they didn't.

The later fragmentation of user data being spread around in the cloud such as AWS objects, Apple iCloud, Backblaze backups, etc instead of every file saved on home computers makes a "universal rich metadata index stored in a database engine" to be a cross-platform standard by all operating systems less realistic today than back in 2003. Therefore, the core os file systems remain a "dumb blob of bytes" with a separate layer of metadata index db in vendor-specific schemes like Apple Spotlight db, Windows Search in ".edb" file, etc.

Post reply on HN