Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why sqlite3 temp files were renamed 'etilqs_*' (2006)

github.com

Post reply on HN
Print view View original

11–20 of 146 posts

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#11

Post by endisneigh »

ShadowBanThis01 wrote: post #7
endisneigh wrote: post #4

Somewhat related - I’m very very curious to hear a detailed account of someone who uses SQLite for a production app with high traffic. For the embedded use case I think it’s a slam dunk, but there are many interesting use cases for server side but they all seem to be toyish. The locking behavior of SQLite is somewhat problematic unless you use WAL and even then not perfect

Wasn't there an article on here within the last week saying how SQLite is "perfect for the 'edge'?"

That was more of the vein of toy example/hypothetically

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#12

Post by endisneigh »

nickstinemates wrote: post #6
endisneigh wrote: post #4

Somewhat related - I’m very very curious to hear a detailed account of someone who uses SQLite for a production app with high traffic. For the embedded use case I think it’s a slam dunk, but there are many interesting use cases for server side but they all seem to be toyish. The locking behavior of SQLite is somewhat problematic unless you use WAL and even then not perfect

Fly[1] uses it 1: https://fly.io/blog/introducing-litefs/

From what I can tell I haven’t seen anything on their end customers using it much

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#13

Post by sltkr »

ShadowBanThis01 wrote: post #8

Love it. Too bad there aren't enough Mac users to prompt a similar backlash against Macs littering every computer they visit on the network with .DS_Store and other turds.

This isn't remotely comparable. Those .DS_Store files are created in arbitrary directories by the Apple file manager or something. The SQLite temp files are created in the OS-specific temporary directory (e.g. C:/Users/username/AppData/Local/Temp or whatever on Windows) which is specifically intended for that purpose. SQLite isn't doing anything wrong; that's where it's supposed to store temporary data that doesn't fit into memory.

The problem is that virus scanners sometimes misclassify those temp files as belonging to malware apps, or sometimes they might be written by real malware apps, but even in the latter case, that only happens because the malware uses sqlite as a library. The malware isn't written by the SQLite authors, so complaining to them is pointless.

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#14

Post by sedatk »

pronoiac wrote: post #5

Looks like the line numbers were lost: https://github.com/mackyle/sqlite/blob/18cf47156abe94255ae14... It's because McAfee started using SQLite, angry users would stumble upon the files, do a minimum of searching or thinking, and be furious at SQLite developers.

I wonder why users were angry about some files in their temp folder. Did McAfee fail to cleanup those files, or were they too big?

Edit: more information here: https://www2.sqlite.org/cvstrac/wiki?p=McafeeProblem Apparently, McAfee kept those files locked when it was using them, so the files couldn't be deleted and people got angry that they couldn't clean them up. Sounds like a loud minority to me.

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#15

Post by sedatk »

ShadowBanThis01 wrote: post #8

Love it. Too bad there aren't enough Mac users to prompt a similar backlash against Macs littering every computer they visit on the network with .DS_Store and other turds.

.DS_Store files are totally out of control: https://twitter.com/rdohms/status/1667554818603245570

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#16

Post by eyegor »

endisneigh wrote: post #4

Somewhat related - I’m very very curious to hear a detailed account of someone who uses SQLite for a production app with high traffic. For the embedded use case I think it’s a slam dunk, but there are many interesting use cases for server side but they all seem to be toyish. The locking behavior of SQLite is somewhat problematic unless you use WAL and even then not perfect

Yes, the sqlite defaults are quite terrible out of the box. I'm not sure why they never changed them, it will start choking at 5k inserts where other dbs can do 100x that (and so will sqlite in wal and a few other settings). Getting it to perform well in high traffic scenarios would be a lot of effort. I struggle to get it to be vaguely performant in embedded use cases and often roll my own poor man's version unless I really care about data integrity (which is rare).

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#17

Post by MatthiasPortzel »

endisneigh wrote: post #4

Somewhat related - I’m very very curious to hear a detailed account of someone who uses SQLite for a production app with high traffic. For the embedded use case I think it’s a slam dunk, but there are many interesting use cases for server side but they all seem to be toyish. The locking behavior of SQLite is somewhat problematic unless you use WAL and even then not perfect

Apps aren’t divided into “high-traffic” and “toys.” There are plenty of use cases where you have a low-write server in a production environment, and SQLite would work fine there.

If you need high write volume, then yes, the locking behavior means SQLite is not a good fit.

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#18

Post by banana_giraffe »

sedatk wrote: post #14
pronoiac wrote: post #5

Looks like the line numbers were lost: https://github.com/mackyle/sqlite/blob/18cf47156abe94255ae14... It's because McAfee started using SQLite, angry users would stumble upon the files, do a minimum of searching or thinking, and be furious at SQLite developers.

I wonder why users were angry about some files in their temp folder. Did McAfee fail to cleanup those files, or were they too big? Edit: more information here: https://www2.sqlite.org/cvstrac/wiki?p=McafeeProblem Apparently, McAfee kept those files locked when it was using them, so the files couldn't be deleted and people got angry that they couldn't clean them up. Sounds like a loud minority to me.

There's some level of power user that will try to fix things, not understand what's going on, and yell at the world when they break things further.

I used to have a popular freely available 3rd party DLL that was included in lots of software packages. Because I was silly, it had my email address in the metadata that'd show up if you clicked "Properties" in Explorer. I'd get plenty of emails from random people asking for and sometimes _demanding_ help with software I've never heard of. I'm sure if I had an easy to find website with my name and a forum on it, it'd be full of such angry comments.

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#19

Post by giantrobot »

sedatk wrote: post #14
pronoiac wrote: post #5

Looks like the line numbers were lost: https://github.com/mackyle/sqlite/blob/18cf47156abe94255ae14... It's because McAfee started using SQLite, angry users would stumble upon the files, do a minimum of searching or thinking, and be furious at SQLite developers.

I wonder why users were angry about some files in their temp folder. Did McAfee fail to cleanup those files, or were they too big? Edit: more information here: https://www2.sqlite.org/cvstrac/wiki?p=McafeeProblem Apparently, McAfee kept those files locked when it was using them, so the files couldn't be deleted and people got angry that they couldn't clean them up. Sounds like a loud minority to me.

> Sounds like a loud minority to me.

Imagine for a moment the cross section of Windows users that 1) have McAfee running 2) poke around the Windows temp folders and/or run some sort of "cleaner" to "optimize" their system and 3) don't recognize the name "sqlite". I can imagine the exact type of user, peak Dunning-Kruger and utterly insufferable. Even just seeing their e-mails before deleting them would be maddening. I think most people on HN can vividly imagine the sort or remember back when they were that insufferable twat.

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#20

Post by bob1029 »

endisneigh wrote: post #4

Somewhat related - I’m very very curious to hear a detailed account of someone who uses SQLite for a production app with high traffic. For the embedded use case I think it’s a slam dunk, but there are many interesting use cases for server side but they all seem to be toyish. The locking behavior of SQLite is somewhat problematic unless you use WAL and even then not perfect

> there are many interesting use cases for server side but they all seem to be toyish.

> The locking behavior of SQLite is somewhat problematic unless you use WAL and even then not perfect

SQLite with WAL and synchronous configured appropriately will insert a row in ~50uS on NVMe hardware. This is completely serialized throughput (i.e. the highest isolation level available in SQL Server, et. al.). At scale, this can be reasonably described as "inserting a billion rows per day".

I have yet to witness a database engine with latency that can touch SQLite. For some applications, this is a go/no-go difference. Why eschew the majesty of SQL because you can't wait for a network hop? Bring that engine in process to help solve your tricky data problems. I'd much rather write SQL than LINQ if I have to join more than 2 tables.

We've been exclusively using SQLite in production for a long time. Big, multi-user databases. Looking at migrating to SQL Server Hyperscale, but not because SQLite is slow or causing any technical troubles. We want to consolidate our installs into one physical place so we can keep a better eye on them as we grow.

Fun fact: SQL Server "Hyperscale" is capped at 100MB/s on its transaction log. I have written to SQLite databases at rates far exceeding this.

Post reply on HN