35% Faster Than The Filesystem (2017)

Post by krainboltgreene » Sun, Aug 18, 2019, 9:49 PM UTC

otterley wrote: post #124

realshowbiz wrote: post #122

I remember a job interview where the take home project was to parse a log file and search/sort by various top attributes as fast as possible. I thought about writing something from scratch, but I knew that whatever I could optimize in an hour or two would be nothing compared to more robust tools that already existed. So I used sqlite to ingest the log once, and query it in various ways to produce the requested report…

The interviewer didn't like the answer because it didn't answer the implicit question, which is how well you can write your own software. We know that you can load a table into SQLite and have it do the heavy lifting for you. But we're often not looking for software _users_ as much as we're looking for _engineers_. When we interview people, the answers we want often aren't real-world answers because we're trying to g…

Post by ysleepy » Sun, Aug 18, 2019, 10:09 PM UTC

wolf550e wrote: post #26

herf wrote: post #25

Wonder what the number is with "noatime"? Every open is a write on a normal filesystem, but you can turn it off.

The default (relatime) is to update atime only once every 24h, so it's negligible for many workloads.

Post by thomascgalvin » Sun, Aug 18, 2019, 10:31 PM UTC

krainboltgreene wrote: post #141

otterley wrote: post #124

Earlier quoted context omitted.

The interviewer didn't like the answer because it didn't answer the implicit question, which is how well you can write your own software. We know that you can load a table into SQLite and have it do the heavy lifting for you. But we're often not looking for software _users_ as much as we're looking for _engineers_. When we interview people, the answers we want often aren't real-world answers because we're trying to g…

You deserve the programmers you're looking for.

Post by muststopmyths » Sun, Aug 18, 2019, 10:39 PM UTC

SQLite wrote: post #113

muststopmyths wrote: post #104

stat to get filesize is a pathlogically bad pattern on Windows. If I remove the call to fileSize() in kvtest and instead replace it with 16k (since the max size in the post is 12k), I can significantly improve the time measured on Windows 10. Before: C:\temp>kvtest run test1.dir --count 100k --blob-api --count 100000 --max-id 1000 --asc Total elapsed time: 9.495 Microseconds per BLOB read: 94.950 Content read rate: 1…

Thanks for this. I did not realize that fileSize() was a performance issue on Windows. I'll attempt to update kvtest.c to deal with that, rerun the tests, and update the page, as I have opportunity.

Post by carapace » Sun, Aug 18, 2019, 11:06 PM UTC

SiVal wrote: post #118

c-smile wrote: post #94

Slightly off the topic. In one of web applications (CRM alike) I was experimenting with SQLite-per-user approach. In that application user related data was clustered on per-user basis. So instead of single-db-for-all-users I had one-db-per-user. Performance gain was significant - around 25% for most of requests. I suspect that this is due the fact that index trees were significantly shorter for each particular user,…

This is exactly the topic that made me read this thread. I have been wondering whether one sqlite file per customer was a good option for an online service where each customer needed only his own, private data but wanted to work on it from his desktop, laptop, or phone, with the option of occasionally downloading the file for additional backup. I just don't know enough about multi-tenant database approaches to know i…