Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Is a Library of Congress Recommended Storage Format

sqlite.org

Post reply on HN
Print view View original

201–205 of 205 posts

Re: SQLite Is a Library of Congress Recommended Storage Format

#201

Post by 1718627440 »

electroly wrote: post #143
giza182 wrote: post #47

Earlier quoted context omitted.

Perhaps a dumb question, but how do you get data into it if you’re not doing writes

I have a system that builds SQLite databases and uploads them to S3. Once they're in S3, they are never changed. The program that builds the databases only does writes, and the program that queries the databases only does reads. It uses a VFS to query the database in-place with HTTP range requests. This is indeed not an optimal setup. A more careful design from first principles would not require seeking around the fi…

If it's that small, why not compile it directly into the application? The compiler might even be able to optimize a good chunk of data away, or optimize the code based on the data.

Re: SQLite Is a Library of Congress Recommended Storage Format

#202

Post by 1718627440 »

LoganDark wrote: post #133
lenwood wrote: post #127

Just yesterday it occurred to me that it had been a while since I last saw an SQLite post at the top of HN. I really like the simplicity and speed of SQLite, I've used in both personal and professional projects. For day-to-day work I still end up in Excel, not because I like it more (I don't), but because its ubiquity makes it the lowest friction way to share & explore datasets with less technical stakeholders and ex…

I've always been irked by how SQLite relies on text parsing to work. Why is it that I have to write queries in text rather than expressing them in programmatic logic? I have never used a relational database because of this, because I hate them, but they can be more performant than pure structured data, but I hate SQL and the entire idea of SQL and I don't want to write it or learn it or use a system that relies on it…

> Why is it that I have to write queries in text rather than expressing them in programmatic logic?

Do you only use graphical programming languages or how is your "programmatic logic" typically expressed?

Re: SQLite Is a Library of Congress Recommended Storage Format

#203

Post by moralestapia »

alexpotato wrote: post #2

I have always loved SQLite. I have also heard that some firms ban its use. Why? Because it makes it SO easy to set up a database for your app that you end up with a super critical component of your application that looks exactly like a file. A file that can have any extension. And that file can be copied around to other servers. Even if there is PII in that file. Multiply this times the number of applications in your…

>DevOps and DBA teams would prefer that the database be a big, heavy iron thing that is very obviously a database server. And when you connect to it, that's also very obvious etc etc.

If I was their CTO and they told me this, and it is not a joke, I'd fire them on the spot.

Re: SQLite Is a Library of Congress Recommended Storage Format

#204

Post by electroly »

1718627440 wrote: post #201
electroly wrote: post #143

Earlier quoted context omitted.

I have a system that builds SQLite databases and uploads them to S3. Once they're in S3, they are never changed. The program that builds the databases only does writes, and the program that queries the databases only does reads. It uses a VFS to query the database in-place with HTTP range requests. This is indeed not an optimal setup. A more careful design from first principles would not require seeking around the fi…

If it's that small, why not compile it directly into the application? The compiler might even be able to optimize a good chunk of data away, or optimize the code based on the data.

Nothing small about it. This is terabytes of data and new files are added daily with new data. That's why we had to move it into S3. It was costing too much to store in SQL Server.

I only reach for SQLite when it's big. Otherwise, I'd prefer to just use the existing SQL Server infrastructure, or .csv.gz files in S3. Internally, I use the term "bulk data" for data sets that are stored in S3 because they were too big for SQL Server.

Re: SQLite Is a Library of Congress Recommended Storage Format

#205

Post by nashashmi »

StilesCrisis wrote: post #113
nashashmi wrote: post #93

Earlier quoted context omitted.

Taking a minute to appreciate the level of long term thinking required for storing data, to plan for 300-500 years into the future, to be able to withstand all kinds of innovations, and survive basic obsolescence. What is the longest surviving paper medium?

Dead sea scrolls come to mind, for some values of "survive." And the Book of Kells is in good shape.

Longest surviving paper document (as opposed to parchment) https://en.wikipedia.org/wiki/Missal_of_Silos?wprov=sfti1
Post reply on HN