Live data from Hacker News

Why sqlite3 temp files were renamed 'etilqs_*' (2006)

github.com

41–50 of 146 posts

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#41
I often search for weird files in my %userprofile% (there are a lot random ones) just out of curiosity, despite I know they're not malicious.

It doesn't help that if you Google any filenames, or even any semi-obscure file extensions, there would always be plenty of blogspam articles saying they're "possible virus". And oftentimes, there is no legit article to say what they really are even if you try, if they're from some relatively less popular software.

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#42

Earlier quoted context omitted.

Did you even open the link? Human readability was not a goal, in fact quite the opposite.

I meant use the name of the program that embeds SQLite, for example, McAfee, Google Chrome etc. This way the user could easily understand which program has created the files.

Why spend time and effort on all of that when applications can just configure it themselves if they want to?

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#43
post #38
post #14

Earlier quoted context omitted.

I wonder why users were angry about some files in their temp folder. Did McAfee fail to cleanup those files, or were they too big? Edit: more information here: https://www2.sqlite.org/cvstrac/wiki?p=McafeeProblem Apparently, McAfee kept those files locked when it was using them, so the files couldn't be deleted and people got angry that they couldn't clean them up. Sounds like a loud minority to me.

In Linux, these files are unlinked, so they are invisible in the filesystem. It is legal to unlink an active file descriptor, but continue reads/writes to it. I think that lsof can still see these temporary files; I'm not sure how I first noticed it. Windows implements a POSIX kernel layer, so perhaps this functionality could be coaxed out of it.

lsof shows “open file handles” which can include deleted files. Can be useful when you have a disk showing tons of usage but you can’t find any big files.

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#44
post #37

Earlier quoted context omitted.

Is your application a networked one?

It is a web app that has exclusive ownership over its SQLite databases. Exactly one SQLiteConnection instance per for the lifetime of the application.

Interesting. Any more info or link?

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#45

Somewhat related - I’m very very curious to hear a detailed account of someone who uses SQLite for a production app with high traffic. For the embedded use case I think it’s a slam dunk, but there are many interesting use cases for server side but they all seem to be toyish. The locking behavior of SQLite is somewhat problematic unless you use WAL and even then not perfect

Apps aren’t divided into “high-traffic” and “toys.” There are plenty of use cases where you have a low-write server in a production environment, and SQLite would work fine there. If you need high write volume, then yes, the locking behavior means SQLite is not a good fit.

SQLite can easily hit 15k INSERTs per minute or more (setting processor affinity to a single core helps drive the max rate up). However, if a process begins a transaction and then stalls, it halts all dml.

I think performance can be good, as long as a competent schema design is in place. Allowing ad-hoc queries from less trusted users will surely tank performance.

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#46
post #38
post #14

Earlier quoted context omitted.

I wonder why users were angry about some files in their temp folder. Did McAfee fail to cleanup those files, or were they too big? Edit: more information here: https://www2.sqlite.org/cvstrac/wiki?p=McafeeProblem Apparently, McAfee kept those files locked when it was using them, so the files couldn't be deleted and people got angry that they couldn't clean them up. Sounds like a loud minority to me.

In Linux, these files are unlinked, so they are invisible in the filesystem. It is legal to unlink an active file descriptor, but continue reads/writes to it. I think that lsof can still see these temporary files; I'm not sure how I first noticed it. Windows implements a POSIX kernel layer, so perhaps this functionality could be coaxed out of it.

Apparently, recent Windows uses POSIX semantics by default: https://news.ycombinator.com/item?id=23745019

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#47

This is a bad choice, I wouldn't understand that it needs to be read backwards. Why not use process's executable name instead?

Did you even open the link? Human readability was not a goal, in fact quite the opposite.

Its a horrible link. It points to a specific commit...Okay, so it must have something to do with the commit? Nope, the commit is putting a limit on the number of symlinks to resolve for whatever reason. Then you also notice that the link is simply a link to the file, not a specific line (something like https://github.com/mackyle/sqlite/blob/18cf47156abe94255ae14...), so you are expected to read the whole C file to figure out why.

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#48
post #13

Earlier quoted context omitted.

This isn't remotely comparable. Those .DS_Store files are created in arbitrary directories by the Apple file manager or something. The SQLite temp files are created in the OS-specific temporary directory (e.g. C:/Users/username/AppData/Local/Temp or whatever on Windows) which is specifically intended for that purpose. SQLite isn't doing anything wrong; that's where it's supposed to store temporary data that doesn't f…

> Those .DS_Store files are created in arbitrary directories by the Apple file manager or something. .DS_Store files come from Apple's file systems containing a separate data and resource fork. APFS can natively store the contents, but for foreign file systems/network shares, a .DS_Store file is created to store those attributes.

I don't have OCD but the casing of .DS_Store annoys me a lot for some reason. I have turned on option to display dot files in finder and I see this god awful name everywhere. It could be .DSStore, .ds_store, .DS Store or even .DS_STORE, current one is the worst.

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#49
post #38
post #14

Earlier quoted context omitted.

I wonder why users were angry about some files in their temp folder. Did McAfee fail to cleanup those files, or were they too big? Edit: more information here: https://www2.sqlite.org/cvstrac/wiki?p=McafeeProblem Apparently, McAfee kept those files locked when it was using them, so the files couldn't be deleted and people got angry that they couldn't clean them up. Sounds like a loud minority to me.

In Linux, these files are unlinked, so they are invisible in the filesystem. It is legal to unlink an active file descriptor, but continue reads/writes to it. I think that lsof can still see these temporary files; I'm not sure how I first noticed it. Windows implements a POSIX kernel layer, so perhaps this functionality could be coaxed out of it.

If a file is opened with `FILE_SHARE_DELETE` then it can be deleted while held open. This flag has existed since forever. It's just that unlike Linux this isn't the default, and no developer would think of setting it. I think the only common software I saw using it when I still used Windows was the media player mpv.

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#50
post #16

Somewhat related - I’m very very curious to hear a detailed account of someone who uses SQLite for a production app with high traffic. For the embedded use case I think it’s a slam dunk, but there are many interesting use cases for server side but they all seem to be toyish. The locking behavior of SQLite is somewhat problematic unless you use WAL and even then not perfect

Yes, the sqlite defaults are quite terrible out of the box. I'm not sure why they never changed them, it will start choking at 5k inserts where other dbs can do 100x that (and so will sqlite in wal and a few other settings). Getting it to perform well in high traffic scenarios would be a lot of effort. I struggle to get it to be vaguely performant in embedded use cases and often roll my own poor man's version unless…

Isn't it more rare to not care about data integrity? Being unsure what state your data is at any point in time does not seem like a safe scenario.
Post reply on HN