Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a 2x faster lexer, then discovered I/O was the real bottleneck

modulovalue.com

Post reply on HN
Print view View original

81–90 of 96 posts

Re: I built a 2x faster lexer, then discovered I/O was the real bottleneck

#81

Post by dwattttt »

paulddraper wrote: post #47
marginalia_nu wrote: post #2

Zip with no compression is a nice contender for a container format that shouldn't be slept on. It effectively reduces the I/O, while unlike TAR, allowing direct random to the files without "extracting" them or seeking through the entire file, this is possible even via mmap, over HTTP range queries, etc. You can still get the compression benefits by serving files with Content-Encoding: gzip or whatever. Though it has…

> I wouldn't expect this to be something that transfers between machines Maybe non-UNIX machines I suppose. But I 100% need executable files to be executable.

This seems like something that shouldn't be the container formats responsibility. You can record arbitrary metadata and put it in a file in the container, so it's trivial to layer on top.

On the other hand, tie the container structure to your OS metadata structure, and your (hopefully good) container format is now stuck with portability issues between other OSes that don't have the same metadata layout, as well as your own OS in the past & future.

Re: I built a 2x faster lexer, then discovered I/O was the real bottleneck

#82

Post by LorenPechtel »

torginus wrote: post #68

I started programming on DOS - I remember how amazing was that you basically almost talked to hardware directly, there was very little restriction on what you could do, and the OS (which imo was much more akin to a set of libraries) provided very little abstraction for you. Then I moved to Windows, and Linux. Each had its own idiosyncrasies, like how everything is a file on Linux, and you're supposed to write program…

I'd love to see this.

Bring back the "segmented" memory architecture. It was not evil because of segments, but because of segment size. If any segment can be any size the bad aspects fall away.

File handles aren't needed anymore. You open a file, you get back a selector rather than an ID. You reference memory from that selector, the system silently swaps pages in as needed.

You could probably do the same thing with directories but I haven't thought about it.

Re: I built a 2x faster lexer, then discovered I/O was the real bottleneck

#83

Post by Sparkyte »

I/O has been the bottleneck for many things especially databases.

So as someone who has seen a long spread of technological advancements over the years I can confidently tell you that chips have far surpassed any peripheral components.

Kind of that scenario where compute has to be fast enough anyway to support I/O. So really it always has to be faster, but I am saying that it has exceeded those expectations.

Re: I built a 2x faster lexer, then discovered I/O was the real bottleneck

#84

Post by zahlman »

charcircuit wrote: post #78
zahlman wrote: post #76

Earlier quoted context omitted.

Yes, that's clear. I'm just not aware of people actually doing that, or having done it back in the era when Java was more dominant.

The bigger issue is that glibc doesn't support loading libraries from zip archives where bionic's linker ddoes. So on platforms where glibc is used you wouldn't see it being done.

Again, I was talking about Java (not C). Good to know, though.

Re: I built a 2x faster lexer, then discovered I/O was the real bottleneck

#85

Post by torginus »

LorenPechtel wrote: post #82
torginus wrote: post #68

I started programming on DOS - I remember how amazing was that you basically almost talked to hardware directly, there was very little restriction on what you could do, and the OS (which imo was much more akin to a set of libraries) provided very little abstraction for you. Then I moved to Windows, and Linux. Each had its own idiosyncrasies, like how everything is a file on Linux, and you're supposed to write program…

I'd love to see this. Bring back the "segmented" memory architecture. It was not evil because of segments, but because of segment size. If any segment can be any size the bad aspects fall away. File handles aren't needed anymore. You open a file, you get back a selector rather than an ID. You reference memory from that selector, the system silently swaps pages in as needed. You could probably do the same thing with d…

The idea as I stated it is super half-baked but

> You could probably do the same thing with directories but I haven't thought about it.

For example in the FAT filesystem, a directory is just a file with a special flag set in its file descriptor and inside said file there is just a list of file descriptors. Not sure if something so simple would a good idea, but it certainly works and has worked IRL.

Re: I built a 2x faster lexer, then discovered I/O was the real bottleneck

#86

Post by [deleted] »

ralferoo wrote: post #44
rcxdude wrote: post #32

Earlier quoted context omitted.

git updates timestamps in part by necessity of compatibility with build systems. If it applied the timestamp of when the file was last modified on checkout then most build systems would break if you checked out an older commit.

git blame is more useful than the file timestamp in any case.

[deleted]

Re: I built a 2x faster lexer, then discovered I/O was the real bottleneck

#87

Post by account42 »

marginalia_nu wrote: post #2

Zip with no compression is a nice contender for a container format that shouldn't be slept on. It effectively reduces the I/O, while unlike TAR, allowing direct random to the files without "extracting" them or seeking through the entire file, this is possible even via mmap, over HTTP range queries, etc. You can still get the compression benefits by serving files with Content-Encoding: gzip or whatever. Though it has…

One problem with the zip format is that metadata is stored both in the central directory and also before each file data - that creates ambiguity when the metadata differs which different programs/libraries don't handle consistently.

Re: I built a 2x faster lexer, then discovered I/O was the real bottleneck

#88

Post by paulddraper »

dwattttt wrote: post #81
paulddraper wrote: post #47

Earlier quoted context omitted.

> I wouldn't expect this to be something that transfers between machines Maybe non-UNIX machines I suppose. But I 100% need executable files to be executable.

This seems like something that shouldn't be the container formats responsibility. You can record arbitrary metadata and put it in a file in the container, so it's trivial to layer on top. On the other hand, tie the container structure to your OS metadata structure, and your (hopefully good) container format is now stuck with portability issues between other OSes that don't have the same metadata layout, as well as yo…

What is a container then?

Just an id,blob format?

The purpose of tar (or competitors) is to serialize files and their metadata.

Re: I built a 2x faster lexer, then discovered I/O was the real bottleneck

#89

Post by renox »

torginus wrote: post #68

I started programming on DOS - I remember how amazing was that you basically almost talked to hardware directly, there was very little restriction on what you could do, and the OS (which imo was much more akin to a set of libraries) provided very little abstraction for you. Then I moved to Windows, and Linux. Each had its own idiosyncrasies, like how everything is a file on Linux, and you're supposed to write program…

> Eventually both Windows and Linux programs moved to a model where the OS just gave you the window as a drawing surface, and you were supposed to fill it.

If you follow this model, how do you solve the accessibility issue?

Re: I built a 2x faster lexer, then discovered I/O was the real bottleneck

#90

Post by pram »

Joker_vD wrote: post #54
paulddraper wrote: post #47

Earlier quoted context omitted.

> I wouldn't expect this to be something that transfers between machines Maybe non-UNIX machines I suppose. But I 100% need executable files to be executable.

Honestly, sometimes I just want to mark all files on a Linux system as executable and see what would even break and why. Seriously, why is there a whole bit for something that's essentially an 'read permission, but you can also directly execute it from the shell'?

From the days when UNIX was primarily multiuser/timeshare. You can prevent users from running wacky stuff with the umask.
Post reply on HN