Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why sqlite3 temp files were renamed 'etilqs_*' (2006)

github.com

Post reply on HN
Print view View original

51–60 of 146 posts

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#51

Post by bennettnate5 »

arp242 wrote: post #23
pronoiac wrote: post #5

Looks like the line numbers were lost: https://github.com/mackyle/sqlite/blob/18cf47156abe94255ae14... It's because McAfee started using SQLite, angry users would stumble upon the files, do a minimum of searching or thinking, and be furious at SQLite developers.

Daniel Stenberg has also gotten some ... interesting ... emails because people found the curl license somewhere and assumed he was responsible for $stuff. https://daniel.haxx.se/blog/2016/01/11/tales-from-my-inbox-p... https://daniel.haxx.se/blog/2016/01/19/subject-urgent-warnin... https://daniel.haxx.se/blog/2018/02/16/why-is-your-email-in-... https://daniel.haxx.se/blog/2021/02/19/i-will-slaughter-you/

I've always wondered if Daniel gets a particular amount of email of this sort because his address includes "haxx"--for someone without tech savvy, it's not all that much of a jump to assume an email that includes something sounding like hacking is the email of the hacker that has attacked them.

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#53

Post by belinder »

bob1029 wrote: post #37
endisneigh wrote: post #36

Earlier quoted context omitted.

Is your application a networked one?

It is a web app that has exclusive ownership over its SQLite databases. Exactly one SQLiteConnection instance per for the lifetime of the application.

Does that mean you have to bring the whole app down if you need to manually insert something in sql?

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#54

Post by hinkley »

chasil wrote: post #38
sedatk wrote: post #14

Earlier quoted context omitted.

I wonder why users were angry about some files in their temp folder. Did McAfee fail to cleanup those files, or were they too big? Edit: more information here: https://www2.sqlite.org/cvstrac/wiki?p=McafeeProblem Apparently, McAfee kept those files locked when it was using them, so the files couldn't be deleted and people got angry that they couldn't clean them up. Sounds like a loud minority to me.

In Linux, these files are unlinked, so they are invisible in the filesystem. It is legal to unlink an active file descriptor, but continue reads/writes to it. I think that lsof can still see these temporary files; I'm not sure how I first noticed it. Windows implements a POSIX kernel layer, so perhaps this functionality could be coaxed out of it.

I knew two guys long long ago that used to circumvent the per user disk quota on shared machines, which was tiny and not conducive to power users. One found a deep dark corner of the file system, and used it to hold binaries the rest of us used. I think his idea was that if it wasn’t for personal gain it was easier to answer difficult questions, which at some point came up and he got a pass, since it would take more disk space if we had private copies.

The other was keeping file handles open to deleted files, as you describe. I don’t recall how this worked, but I suspect it involved uncompressing data into a file descriptor, then reading it back. I guess as long as his terminal window was open it was more stable than tmp (he may have also been using screen).

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#55

Post by rootw0rm »

giantrobot wrote: post #19
sedatk wrote: post #14

Earlier quoted context omitted.

I wonder why users were angry about some files in their temp folder. Did McAfee fail to cleanup those files, or were they too big? Edit: more information here: https://www2.sqlite.org/cvstrac/wiki?p=McafeeProblem Apparently, McAfee kept those files locked when it was using them, so the files couldn't be deleted and people got angry that they couldn't clean them up. Sounds like a loud minority to me.

> Sounds like a loud minority to me. Imagine for a moment the cross section of Windows users that 1) have McAfee running 2) poke around the Windows temp folders and/or run some sort of "cleaner" to "optimize" their system and 3) don't recognize the name "sqlite". I can imagine the exact type of user, peak Dunning-Kruger and utterly insufferable. Even just seeing their e-mails before deleting them would be maddening.…

I was that person once. I was 12 and couldn't understand why MS-DOS disk maintenance tools shouldn't be used on PC-DOS because obviously repair tools can only fix things

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#56

Post by pavlov »

bennettnate5 wrote: post #51
arp242 wrote: post #23

Earlier quoted context omitted.

Daniel Stenberg has also gotten some ... interesting ... emails because people found the curl license somewhere and assumed he was responsible for $stuff. https://daniel.haxx.se/blog/2016/01/11/tales-from-my-inbox-p... https://daniel.haxx.se/blog/2016/01/19/subject-urgent-warnin... https://daniel.haxx.se/blog/2018/02/16/why-is-your-email-in-... https://daniel.haxx.se/blog/2021/02/19/i-will-slaughter-you/

I've always wondered if Daniel gets a particular amount of email of this sort because his address includes "haxx"--for someone without tech savvy, it's not all that much of a jump to assume an email that includes something sounding like hacking is the email of the hacker that has attacked them.

Yes, that seems to be the likely explanation at least for the second linked email which closes with:

”Please see attached screen dump for a screen shot of your contact info included in Spotify (or what more likely seems to be a hacked Spotify developed illegally by a third party).”

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#57

Post by aseipp »

endisneigh wrote: post #4

Somewhat related - I’m very very curious to hear a detailed account of someone who uses SQLite for a production app with high traffic. For the embedded use case I think it’s a slam dunk, but there are many interesting use cases for server side but they all seem to be toyish. The locking behavior of SQLite is somewhat problematic unless you use WAL and even then not perfect

High-volume single-writer + WAL mode is perfectly doable for a lot of applications, but you have to keep the concurrency model in mind. I think of it more like a really advanced data structure library than a "database" in that sense.

An old product I worked on (which is still around) used (still uses?) SQLite for storing filesystem blocks that came from the paired backup software. A single database only contained the blocks backed up from a single disk (based on its disk UUID.) So, this is a perfect scenario since there's only one writer at a time because there's only one device and one backup on that device at a time. We could write blocks to SQLite fast enough to saturate the IOPS of the underlying device and filesystem containing the database. This was over 10 years ago. Very practical and durable and way better than anything we could do in house.

Today, you can definitely do GB/s on NVMe class hardware with the right pragma settings and the right workload. So for certain classes of multi-tenant solutions I think it's not so bad, you can just naturally have a single writer to a single data store (or can otherwise linearize writes) and the raw performance will be excellent and more than enough.

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#58

Post by throwaway09223 »

I have a related story:

Around the year 2000 I was working operations in the NOC for WebTV (then owned by Microsoft). For those who don't know, WebTV was a little set-top box with a modem which would dial up on demand and provide a very basic web/chat/email experience on the TV. The box would call a 1800 number to figure out its own phone number, then re-dial on a local toll-free number with a local sub-contracted ISP.

One of the services we had would periodically send a UDP datagram out to online clients to let them know they had new email. The settop box would then light up a little indicator light.

Of course, sometimes the client would hang up. The IP might get allocated to a PC dialup user. And sometimes, that PC dialup user might be running a firewall that was popular back then, called BLACK ICE DEFENDER.

BLACK ICE DEFENDER had all these (not so) cool features, the kind that semi-technical people love. For example, it would log ATTACKS. What are ATTACKS? Unrecognized traffic, of course.

Sometimes the little UDP datagram for our "you have mail" service would be delivered to a PC user running BLACK ICE DEFENDER, which would register it as an ATTACK. It would then ever so helpfully look up the ARIN contact information to see who sent the errant datagram -- which had the NOC phone number. It would then tell the user "THIS ENTITY IS HACKING YOU" and imply that contacting them would be productive. Yes, you could pick up a phone and call the Microsoft NOC. Back then, the internet was a smaller place.

My job was to check the NOC voicemail, which was reliably filled with very angry people. Often they would threaten that they've reported us to the FBI or somesuch, or that it confirmed some conspiracy theory or another. We played the good ones on speakerphone for entertainment.

Good times. Doesn't happen anymore.

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#59

Post by giantrobot »

hinkley wrote: post #54
chasil wrote: post #38

Earlier quoted context omitted.

In Linux, these files are unlinked, so they are invisible in the filesystem. It is legal to unlink an active file descriptor, but continue reads/writes to it. I think that lsof can still see these temporary files; I'm not sure how I first noticed it. Windows implements a POSIX kernel layer, so perhaps this functionality could be coaxed out of it.

I knew two guys long long ago that used to circumvent the per user disk quota on shared machines, which was tiny and not conducive to power users. One found a deep dark corner of the file system, and used it to hold binaries the rest of us used. I think his idea was that if it wasn’t for personal gain it was easier to answer difficult questions, which at some point came up and he got a pass, since it would take more…

> I don’t recall how this worked

Unlinking files reduces their reference count. Once the reference count reaches zero the file is considered deleted and the space can be reclaimed. Every open file descriptor and hard link increases a file's reference count. So if you've got a file descriptor open on a file in a background task (daemon, nohup, screen, etc) and unlink it with rm or something the file's reference count will decrement but not go to zero. Only when that program closes the file descriptor will the reference count go to zero and the file actually be deleted.

Re: Why sqlite3 temp files were renamed 'etilqs_*' (2006)

#60

Post by somat »

chasil wrote: post #38
sedatk wrote: post #14

Earlier quoted context omitted.

I wonder why users were angry about some files in their temp folder. Did McAfee fail to cleanup those files, or were they too big? Edit: more information here: https://www2.sqlite.org/cvstrac/wiki?p=McafeeProblem Apparently, McAfee kept those files locked when it was using them, so the files couldn't be deleted and people got angry that they couldn't clean them up. Sounds like a loud minority to me.

In Linux, these files are unlinked, so they are invisible in the filesystem. It is legal to unlink an active file descriptor, but continue reads/writes to it. I think that lsof can still see these temporary files; I'm not sure how I first noticed it. Windows implements a POSIX kernel layer, so perhaps this functionality could be coaxed out of it.

When youtube was flash based that is how flash would save it's cache file for the stream. to save a video you would go looking for what file descriptors were in use and yank them out of I believe /dev/fd and back into the filesystem. On windows the file was visible but locked behind the normal windows file locking shenanigans. To copy locked files on windows I used a program called hobocopy that utilized volume shadow tricks to work around the locks.
Post reply on HN