What if you create something that is optimized for performance without offering the full SQL language? Just a container to store small files with padding to avoid the expensive open/close calls.
35% Faster Than The Filesystem (2017)
11–20 of 166 posts
Re: 35% Faster Than The Filesystem (2017)
#12This might be the answer on how to make file-heavy tools like npm/yarn work adequately on windows...
Re: 35% Faster Than The Filesystem (2017)
#13In the past I've had huge speedups by moving simple single-table databases that had grown a bit (e.g., time series data) from sqlite to postgres. Insert performance is also quite bad forcing you to write applications with extra caching layers to be able to do a bunch of inserts at once. Sqlite is great for many applications but it's speed is somewhat oversold. I really wish the postgres engine was embeddable into app…
Obviously if it was embeddable it wouldn’t be as fast and be subject to the same design limitations as SQLite.
Re: 35% Faster Than The Filesystem (2017)
#14For small- to mid-sized projects, I’ve always realized huge gains in simplicity by haves “Files” tables to store various assets. It means instances in a web-farm can pull the files down when they initialize easily, it means files are automatically versioned, it provides an obvious place to put the files when they are being uploaded on the Admin panel. It means all the files are getting backed up as part of the databa…
>Really the only downside I ever found — at the scale I was operating at — was that it bloats the database backups.
Easily solved by having two separate databases. One for dynamic content, one for static files, which is probably good practice regardless.
Re: 35% Faster Than The Filesystem (2017)
#15Next week on Show HN: `sqlitefs`, the SQLite-backed FUSE filesystem?
Re: 35% Faster Than The Filesystem (2017)
#16While an impressive feat on SQLite’s end, I sincerely hope no engineers see this and take it at face value. The filesystem is slower because it is more flexible, and I would imagine that once you have concurrent reads and writes to your thumbnails, a filesystem is to prefer.
If you want to support concurrent reads and writes, you probably should pick a particular database engine and understand its concurrency semantics, instead of assuming that whatever filesystem your software lands on will have the semantics you expect.
Re: 35% Faster Than The Filesystem (2017)
#17This shouldn't be especially surprising - in fact I would expect this to be the case, all other things being equal, for two reasons: firstly, open() followed by read() involves two kernel-userspace context switches, while SQLite operating on an already open file only needs to call read(), and secondly SQLite is more aggressive about making forwards-incompatible changes than most popular filesystems, giving it more latitude to optimize (ZFS is an exception, but with most filesystems you expect a 10-year old computer to read a disk created yesterday).
Re: 35% Faster Than The Filesystem (2017)
#18As someone who has spent time consulting for different kinds of "we do stuff on the internet" companies, I can confidently say that this is a premature optimization for 99.999% of the companies/projects. The companies simply don't have enough IO traffic to need it.
Is it cool? Sure. Is it sexy? Maybe. Is it needed? Nope.
Do boring stuff. Use files. If you send your small files over the internet via web and you need to squeeze additional performance, do boring stuff again: use Varnish.
Re: 35% Faster Than The Filesystem (2017)
#19What if you create something that is optimized for performance without offering the full SQL language? Just a container to store small files with padding to avoid the expensive open/close calls.
Look at Google’s LevelDB and Facebook’s RocksDB.
Re: 35% Faster Than The Filesystem (2017)
#20Next week on Show HN: `sqlitefs`, the SQLite-backed FUSE filesystem?
Having it on top of FUSE would make it significantly slower than an in-kernel filesystem.