Live data from Hacker News

How bloom filters made SQLite 10x faster

avi.im

121–127 of 127 posts

Re: How bloom filters made SQLite 10x faster

#121
post #119

Earlier quoted context omitted.

Any idea how I get unbroken?

You could contact them but honestly that probably wouldn't work very well. Things that might help: switch your settings from old reddit to new reddit, use new reddit to load a few pages, and then switch back. Also try using "newest" reddit at https://sh.reddit.com .

Hey that brought it down to about 650ms. Thank you! That's a noticeable improvement.

Re: How bloom filters made SQLite 10x faster

#122
post #112

The article states that order of join matters because then nest loops differently. But we still go through entire loops everywhere. Where do those numbers in the example come from? If we have 1000, 20 and 200 elements in 3 loops, algorithmically, it does not matter in which order you iterate. Complexity is always 1000×20×200. What am I missing?

You don't go through entire loops everywhere because if there isn't a match in the first two tables, you don't have to check the match with the third table. It's better to check A x C before A x B if you know that A x C has less matching rows, because the final loop will be shorter.

Ah, I see the numbers in the example are the numbers of matched rows, not a total number of rows... Make sense. I do not work with databases, did not know that you should pay attention to the order here.

Re: How bloom filters made SQLite 10x faster

#123
post #38

Thanks to its simplicity for development and hosting SQLite has become our first choice for new Apps. We use a number of different ways to workaround its single concurrent writer limitation [1]. Whilst we're currently using Litestream for replication, we're currently evaluating switching to SQLite's native rsync [2]. [1] https://servicestack.net/posts/scalable-sqlite [2] https://www.sqlite.org/rsync.html

Is switching to SQLite really making hosting your web apps less of a headache? Most hosting providers make spinning up your standard client-server RDBMSs (MySQL, Postgres) a breeze.

It probably depends!

If you’re using one of the hosting providers or managed services that just give you a DB to use and you don’t have to think about administering it, it’s a pretty relaxed experience.

Even if you self-host, running a container with the DB of your choice usually isn’t that much more difficult than just having an SQLite DB for your app, at that point it’s probably more about the features which each solution provides.

Personally, I have some stuff running on MariaDB, PostgreSQL and SQLite and I’ve almost never had any problems with either. Then again, I’ve never gotten to a scale with personal stuff where the specifics of that choice would matter much.

Re: How bloom filters made SQLite 10x faster

#124

Earlier quoted context omitted.

UUIDs are very wasteful [1]. For most use cases you can replace them with much shorter strings and still have very low chances of collisions [2] [1] https://henvic.dev/posts/uuid/ [2] https://alex7kom.github.io/nano-nanoid-cc/

Call me crazy, but I'm simply splitting my UUID into the higher and lower bits and indexing off that. IE CREATE TABLE foo( id_ms UNSIGNED BIG INT NOT NULL, id_ls UNSIGNED BIG INT NOT NULL, PRIMARY KEY (id_ms, id_ls) ) WITHOUT ROWID; That works well with UUIDv7 and is just storing 128bits rather than a full string. In most languages it's pretty trivial to turn 2 longs into a UUID and vice versa.

Is there any advantage to this approach over Postgres native uuid support which should store the same number of bits?

Re: How bloom filters made SQLite 10x faster

#125
Steve Gibson (grc.com) did a really great job of explaining how cascading bloom filters work in order to efficiently achieve fast certificate revocation lookups in Firefox. It's definitely worth checking out the episode

https://twit.tv/posts/tech/cascading-bloom-filters-revolutio...

Re: How bloom filters made SQLite 10x faster

#126

Earlier quoted context omitted.

Call me crazy, but I'm simply splitting my UUID into the higher and lower bits and indexing off that. IE CREATE TABLE foo( id_ms UNSIGNED BIG INT NOT NULL, id_ls UNSIGNED BIG INT NOT NULL, PRIMARY KEY (id_ms, id_ls) ) WITHOUT ROWID; That works well with UUIDv7 and is just storing 128bits rather than a full string. In most languages it's pretty trivial to turn 2 longs into a UUID and vice versa.

Is there any advantage to this approach over Postgres native uuid support which should store the same number of bits?

No. This approach is strictly for DBS like sqlite without uuid or 128bit integer support.

Re: How bloom filters made SQLite 10x faster

#127
post #96

Earlier quoted context omitted.

> but this where you get all of your performance caveats. You mean the one where it locks on write? It’s totally fine, if you wrote any cross process code yourself it’s probably going to do similar locking.

Yeah, but you're locking the whole file and if you try to open it while it's locked, sleep-polling for it to be unlocked. It's safe, but it's a minimum viable product - as they say, sqlite is a replacement for fopen, not for a fully featured database. Client/server systems have better locking.

I mostly agree. If you need an RDMS get a real one that has all the features and granular locking.

The only point I’m disagreeing about is the blanket statement “your SQLite website won’t work if it has concurrent writers”. It will.

Post reply on HN