35% Faster Than The Filesystem (2017) • Hacker News Board

Re: 35% Faster Than The Filesystem (2017)

#21

Post by quickquestion42 » Sun, Aug 18, 2019, 1:55 PM UTC

pedrocr wrote: post #10

In the past I've had huge speedups by moving simple single-table databases that had grown a bit (e.g., time series data) from sqlite to postgres. Insert performance is also quite bad forcing you to write applications with extra caching layers to be able to do a bunch of inserts at once. Sqlite is great for many applications but it's speed is somewhat oversold. I really wish the postgres engine was embeddable into app…

EDIT: Deleted a controversial comment about the often praised SQLite test suite not actually being public and about locking behaviour in SQLite. References are [1] and [2].

[1] https://www.sqlite.org/th3.html

[2] https://beets.io/blog/sqlite-nightmare.html

Re: 35% Faster Than The Filesystem (2017)

#22

Post by perspective1 » Sun, Aug 18, 2019, 1:58 PM UTC

zaroth wrote: post #6

For small- to mid-sized projects, I’ve always realized huge gains in simplicity by haves “Files” tables to store various assets. It means instances in a web-farm can pull the files down when they initialize easily, it means files are automatically versioned, it provides an obvious place to put the files when they are being uploaded on the Admin panel. It means all the files are getting backed up as part of the databa…

My (naive?) view is that Nginx caches static files by default after the first hit so the inconvenience isn't worth it in my web apps. Would I really see an improvement switching to a SQLite-esque file system?

Re: 35% Faster Than The Filesystem (2017)

#23

Post by pjc50 » Sun, Aug 18, 2019, 1:59 PM UTC

Tepix wrote: post #11

What if you create something that is optimized for performance without offering the full SQL language? Just a container to store small files with padding to avoid the expensive open/close calls.

Read-only versions of this are extremely common for games; I first encountered it with 90s DOOM WAD files.

Re: 35% Faster Than The Filesystem (2017)

#24

Post by mirekrusin » Sun, Aug 18, 2019, 2:07 PM UTC

OJFord wrote: post #15

Next week on Show HN: `sqlitefs`, the SQLite-backed FUSE filesystem?

It would be interesting to see how it performs ie. on nodejs projects with node_modules.sqlite3 - for practical reasons it would be nice to mount it via fuse to node_modules - if needed.

Re: 35% Faster Than The Filesystem (2017)

#25

Post by herf » Sun, Aug 18, 2019, 2:08 PM UTC

Wonder what the number is with "noatime"? Every open is a write on a normal filesystem, but you can turn it off.

Re: 35% Faster Than The Filesystem (2017)

#26

Post by wolf550e » Sun, Aug 18, 2019, 2:12 PM UTC

herf wrote: post #25

Wonder what the number is with "noatime"? Every open is a write on a normal filesystem, but you can turn it off.

The default (relatime) is to update atime only once every 24h, so it's negligible for many workloads.

Re: 35% Faster Than The Filesystem (2017)

#27

Post by typon » Sun, Aug 18, 2019, 2:13 PM UTC

Tepix wrote: post #11

What if you create something that is optimized for performance without offering the full SQL language? Just a container to store small files with padding to avoid the expensive open/close calls.

Write API abstractions that offer baked and templated Sql queries for your specific use case. Flexible enough for future growth, easy enough for every day normal use without inventing a whole new system.

Re: 35% Faster Than The Filesystem (2017)

#28

Post by Nevermindmine » Sun, Aug 18, 2019, 2:13 PM UTC

zaroth wrote: post #6

For small- to mid-sized projects, I’ve always realized huge gains in simplicity by haves “Files” tables to store various assets. It means instances in a web-farm can pull the files down when they initialize easily, it means files are automatically versioned, it provides an obvious place to put the files when they are being uploaded on the Admin panel. It means all the files are getting backed up as part of the databa…

What kind of versioning are your referring to? I know some databases has it but not sure if I understand it.

Personally I use S3 or similar for this. At the number of files I'm working with storing it in a relational database would be crazy expensive.

Re: 35% Faster Than The Filesystem (2017)

#29

Post by wolf550e » Sun, Aug 18, 2019, 2:13 PM UTC

Beware opening sqlite files from untrusted sources.

https://research.checkpoint.com/select-code_execution-from-u...

Re: 35% Faster Than The Filesystem (2017)

#30

Post by falcolas » Sun, Aug 18, 2019, 2:16 PM UTC

zaroth wrote: post #6

For small- to mid-sized projects, I’ve always realized huge gains in simplicity by haves “Files” tables to store various assets. It means instances in a web-farm can pull the files down when they initialize easily, it means files are automatically versioned, it provides an obvious place to put the files when they are being uploaded on the Admin panel. It means all the files are getting backed up as part of the databa…

Personal opinion: Those things (static assets) belong in a container with Nginx serving them (and that’s not even counting SAAS options like S3). Easy to integrate with CDNs, easy to update, easy to roll out and back. And you get to take advantage of layers to keep the network and disk usage minimal.