Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a 2x faster lexer, then discovered I/O was the real bottleneck

modulovalue.com

Post reply on HN
Print view View original

91–96 of 96 posts

Re: I built a 2x faster lexer, then discovered I/O was the real bottleneck

#91

Post by Joker_vD »

pram wrote: post #90
Joker_vD wrote: post #54

Earlier quoted context omitted.

Honestly, sometimes I just want to mark all files on a Linux system as executable and see what would even break and why. Seriously, why is there a whole bit for something that's essentially an 'read permission, but you can also directly execute it from the shell'?

From the days when UNIX was primarily multiuser/timeshare. You can prevent users from running wacky stuff with the umask.

No you can't. If a user can read something, it can execute it. The only thing where it matters is setuid applications where the setuid bit allows the user to run an application as someone else. But it's already a separate permission bit, and frankly, the whole setuid idea turned out to be quite a high-maintenance design in the end, with lots of additional features heaped on top of it to help mitigate the worst vulnerabilities.

Re: I built a 2x faster lexer, then discovered I/O was the real bottleneck

#92

Post by thaumasiotes »

nikanj wrote: post #74
thaumasiotes wrote: post #69

Earlier quoted context omitted.

> It effectively reduces the I/O, while unlike TAR, allowing direct random to the files without "extracting" them or seeking through the entire file How do you access a particular file without seeking through the entire file? You can't know where anything is without first seeking through the whole file.

At the end of the ZIP file, there's a central directory of all files contained in that archive. Read the last block, seek to the block containing the file you want to access, done

> At the end of the ZIP file, there's a central directory of all files contained in that archive.

Where does that begin?

> Read the last block

You mean the last 4KB chunk defined by the file system, or what? The comment can be up to 64KB long.

Re: I built a 2x faster lexer, then discovered I/O was the real bottleneck

#93

Post by dwattttt »

paulddraper wrote: post #88
dwattttt wrote: post #81

Earlier quoted context omitted.

This seems like something that shouldn't be the container formats responsibility. You can record arbitrary metadata and put it in a file in the container, so it's trivial to layer on top. On the other hand, tie the container structure to your OS metadata structure, and your (hopefully good) container format is now stuck with portability issues between other OSes that don't have the same metadata layout, as well as yo…

What is a container then? Just an id,blob format? The purpose of tar (or competitors) is to serialize files and their metadata.

Tar is not the pinnacle of "containers"; it has age and ubiquity, and that's about it at this point.

Tar's purpose was to serialise files and metadata in 1979, accounting for tape foibles such as fixed or variable data block size.

Re: I built a 2x faster lexer, then discovered I/O was the real bottleneck

#94

Post by Dylan16807 »

1718627440 wrote: post #8
marginalia_nu wrote: post #2

Zip with no compression is a nice contender for a container format that shouldn't be slept on. It effectively reduces the I/O, while unlike TAR, allowing direct random to the files without "extracting" them or seeking through the entire file, this is possible even via mmap, over HTTP range queries, etc. You can still get the compression benefits by serving files with Content-Encoding: gzip or whatever. Though it has…

Isn't this what is already common in the Python community? > I don't want to unpack an archive and have to scrutinize it for files with o+rxst permissions, or have their creation date be anything other than when I unpacked them. I'm the opposite, when I pack and unpack something, I want the files to be identical including attributes. Why should I throw away all the timestamps, just because the file were temporarily i…

> I'm the opposite, when I pack and unpack something, I want the files to be identical including attributes. Why should I throw away all the timestamps, just because the file were temporarily in an archive?

I would expect modified dates to stay the same, and other dates to change similar to copying a directory. I think this is the normal experience with zip?

For creation dates, Linux usually doesn't even track those at all. There's partial support on BTRFS and ZFS, and on ext4 there nowhere to store it at all.

Re: I built a 2x faster lexer, then discovered I/O was the real bottleneck

#95

Post by Dylan16807 »

thaumasiotes wrote: post #92
nikanj wrote: post #74

Earlier quoted context omitted.

At the end of the ZIP file, there's a central directory of all files contained in that archive. Read the last block, seek to the block containing the file you want to access, done

> At the end of the ZIP file, there's a central directory of all files contained in that archive. Where does that begin? > Read the last block You mean the last 4KB chunk defined by the file system, or what? The comment can be up to 64KB long.

> You mean the last 4KB chunk defined by the file system, or what? The comment can be up to 64KB long.

Okay, the last 65KB.

Are you nitpicking now that you learned about the directory, or did you know about it before your first comment and pretended not to for some reason?

Re: I built a 2x faster lexer, then discovered I/O was the real bottleneck

#96

Post by nh2 »

pwg wrote: post #45
nh2 wrote: post #41

Earlier quoted context omitted.

There is some confusion here. ZIP retains timestamps. This makes sense because timestamps are a global concept. Consider them a attribute dependent on only the file in ZIP, similar to the file's name. Owners and permissions are dependent also on the computer the files are stored on. User "john" might have a different user ID on another computer, or not exist there at all, or be a different John. So there isn't one ob…

> ZIP retains timestamps. It does, but unless the 'zip' archive creator being used makes use of the extensions for high resolution timestamps, the basic ZIP format retains only old MSDOS style timestamps (rounded to the closed two seconds). So one may lose some precision in ones timestamps when passing files through a zip archive.

That's correct. I think it is not hard to use the high resolution timestamps, but still they do not have the same precision as a UNIX tv_nsec value, which can be annoying if you want to preserve the _exact_ time that common Linux filesystems can store.
Post reply on HN