Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

35% Faster Than The Filesystem (2017)

sqlite.org

Post reply on HN
Print view View original

151–160 of 166 posts

Re: 35% Faster Than The Filesystem (2017)

#151

Post by Const-me »

kukx wrote: post #121
Const-me wrote: post #73

Earlier quoted context omitted.

Windows kernel uses all free RAM as a file cache. Low priority one i.e. when an app wants to allocate RAM, the file system cached pages are evicted. Their Win7 test machine has 4GB RAM, Win10 16GB RAM. I think I know what’s going on. They don’t publish absolute numbers, they only publish result compared to SQLight on the same machine. Windows has approximately fixed overhead when opening files. Most of that overhead…

Nitpick, but file permissions in Linux take normally 12 bits. You probably did not count SUID, SGID and sticky bit.

I’ve only counted these famous RWX bits, settable by chmod. You’re right, there’s more security metadata involved in Linux.

Still, Windows permissions are way more complicated, especially when AD domains are involved. These access control lists have arbitrary count of entries each. There’s another list, SACL, for audit. Access permissions are inherited through file system with non-trivial inheritance rules, e.g. allow + deny = deny. User permissions are inherited through group membership, and security groups can be nested i.e. include other groups. Security descriptors may include custom permissions unknown to MS. Any thread can run as any user in the forest through client impersonation, many network servers do. The kernel tries to apply effective permissions instantly, as soon as anything changes (but not for every read/write, that would be too slow, it only does access checks when opening files).

Combined, these features make access checks way more expensive in Windows, by orders of magnitude.

Re: 35% Faster Than The Filesystem (2017)

#152

Post by chii »

realshowbiz wrote: post #137
otterley wrote: post #124

Earlier quoted context omitted.

The interviewer didn't like the answer because it didn't answer the implicit question, which is how well you can write your own software. We know that you can load a table into SQLite and have it do the heavy lifting for you. But we're often not looking for software _users_ as much as we're looking for _engineers_. When we interview people, the answers we want often aren't real-world answers because we're trying to g…

Yeah, although imagine if carpenters were interviewed/judged based on the quality by which they could create hammers and drills from scratch. But it was a good thing overall. I showed me signs that the company has an elevated degree of not invented here culture, and that they aren’t very clear when requesting/scoping work. https://en.m.wikipedia.org/wiki/Not_invented_here

If software engineers are carpenters, the hammers and drills are the programming languages, editors, and IDEs. And sqllite is an off the shelf plank.

Re: 35% Faster Than The Filesystem (2017)

#153

Post by Spooky23 »

Zenst wrote: post #75
Spooky23 wrote: post #54

Earlier quoted context omitted.

I ran a large at the time Informix implementation in the late 90s/early 2000s. Raw disk vs file system (Solaris UFS) performance was 35% faster on our transaction systems.

As did I upon AIX and NCR towers. Did you find you had to offset the start block of the raw partition? As with AIX, even though the OS saw it as raw, it would stamp over the first 8k (iirc) with volume control data and with that, could easily corrupt a raw partition if you didn't factor in an offset.

I don’t recall the details, but we used Veritas Volume Manager for managing the raw disk, and I’m pretty sure vxvm either hid that or had a driver that didn’t do it.

Our restore run book was scripted and tight, so we may have been doing it ourselves, but I didn’t write the code or at least didn’t break it! Fun times.

Re: 35% Faster Than The Filesystem (2017)

#154

Post by liability »

esmi wrote: post #136
otterley wrote: post #124

Earlier quoted context omitted.

The interviewer didn't like the answer because it didn't answer the implicit question, which is how well you can write your own software. We know that you can load a table into SQLite and have it do the heavy lifting for you. But we're often not looking for software _users_ as much as we're looking for _engineers_. When we interview people, the answers we want often aren't real-world answers because we're trying to g…

If that’s the case then the restrictions should’ve been given in the problem statement and the interviewer should’ve been able to answer why the candidate’s answer was rejected.

Consider it a test of your ability to infer the obvious (a valuable skill.)

Re: 35% Faster Than The Filesystem (2017)

#155

Post by aitchnyu »

liability wrote: post #46
pedrocr wrote: post #10

In the past I've had huge speedups by moving simple single-table databases that had grown a bit (e.g., time series data) from sqlite to postgres. Insert performance is also quite bad forcing you to write applications with extra caching layers to be able to do a bunch of inserts at once. Sqlite is great for many applications but it's speed is somewhat oversold. I really wish the postgres engine was embeddable into app…

I once tried out some RSS aggregator/reader that required me to install and set up postgres for no good reason other than it's what the developer was accustomed to. Huge pain in the ass, particularly since simply using sqlite would have been perfectly adaquate (does a RSS reader really need concurrent writes? Really? ) For any software meant to be used on a non-technie's desktop, anything other than an embedded 'zero…

Amarok, the kitchen sink music player had an option of connecting to a Mysql server back in 2007. IIRC it used a slower embedded DB out of the box.

Re: 35% Faster Than The Filesystem (2017)

#156

Post by fanf2 »

kukx wrote: post #121
Const-me wrote: post #73

Earlier quoted context omitted.

Windows kernel uses all free RAM as a file cache. Low priority one i.e. when an app wants to allocate RAM, the file system cached pages are evicted. Their Win7 test machine has 4GB RAM, Win10 16GB RAM. I think I know what’s going on. They don’t publish absolute numbers, they only publish result compared to SQLight on the same machine. Windows has approximately fixed overhead when opening files. Most of that overhead…

Nitpick, but file permissions in Linux take normally 12 bits. You probably did not count SUID, SGID and sticky bit.

Or the ACLs or extended attributes

https://linux.die.net/man/1/chattr

https://linux.die.net/man/1/setfacl

Re: 35% Faster Than The Filesystem (2017)

#157

Post by mkl »

Fnoord wrote: post #107
dymk wrote: post #71

Earlier quoted context omitted.

Here you go: https://github.com/guardianproject/libsqlfs

Thank you, I was reading the comments exactly to figure out if there was a FUSE sqlfs module. TL;DR this URL describes that, yes, there is. Why is this relevant? Because this way, you keep the flexibilities of a filesystem and, as a consequence of it being able to function like a filesystem, you keep the "power of Unix/Linux" ie. all your nix [1] tools keep working. [1] Calling it Nix confuses with NixOS...

So call it *nix? It takes two asterisks two turn on italics between them.

Re: 35% Faster Than The Filesystem (2017)

#158

Post by [deleted] »

Fnoord wrote: post #107
dymk wrote: post #71

Earlier quoted context omitted.

Here you go: https://github.com/guardianproject/libsqlfs

Thank you, I was reading the comments exactly to figure out if there was a FUSE sqlfs module. TL;DR this URL describes that, yes, there is. Why is this relevant? Because this way, you keep the flexibilities of a filesystem and, as a consequence of it being able to function like a filesystem, you keep the "power of Unix/Linux" ie. all your nix [1] tools keep working. [1] Calling it Nix confuses with NixOS...

[deleted]

Re: 35% Faster Than The Filesystem (2017)

#159

Post by Fnoord »

mkl wrote: post #157
Fnoord wrote: post #107

Earlier quoted context omitted.

Thank you, I was reading the comments exactly to figure out if there was a FUSE sqlfs module. TL;DR this URL describes that, yes, there is. Why is this relevant? Because this way, you keep the flexibilities of a filesystem and, as a consequence of it being able to function like a filesystem, you keep the "power of Unix/Linux" ie. all your nix [1] tools keep working. [1] Calling it Nix confuses with NixOS...

So call it *nix? It takes two asterisks two turn on italics between them.

Cheers. Yeah, I guess it triggered cause I used two asterisks in my text since originally I used the term twice.

Re: 35% Faster Than The Filesystem (2017)

#160

Post by ferzul »

Fnoord wrote: post #107
dymk wrote: post #71

Earlier quoted context omitted.

Here you go: https://github.com/guardianproject/libsqlfs

Thank you, I was reading the comments exactly to figure out if there was a FUSE sqlfs module. TL;DR this URL describes that, yes, there is. Why is this relevant? Because this way, you keep the flexibilities of a filesystem and, as a consequence of it being able to function like a filesystem, you keep the "power of Unix/Linux" ie. all your nix [1] tools keep working. [1] Calling it Nix confuses with NixOS...

well, it was *nix before nix (the package manager) existed. So I think Nix, as good as it is, stuffed up with that name.
Post reply on HN