Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We’re happy with SQLite and not urgently interested in a fancier DBMS (2016)

beets.io

Post reply on HN
Print view View original

111–120 of 125 posts

Re: We’re happy with SQLite and not urgently interested in a fancier DBMS (2016)

#111

Post by wenc »

AutoEngineer wrote: post #101
diogofranco wrote: post #68

Earlier quoted context omitted.

> Why aren't we replacing CSV with it, especially for big data applications? Big data applications tend to use other structured binary formats like parquet and avro, which any big data tool can typically parse.

Speaking for an Engineer unfortunately often data is simple CSV. Parsing this data robustly and visualizing it without scripting is still something that hasn't been solved. If you have any idea how to do it, I'd be more than glad to hear about it.

You can visualize CSVs easily without scripting: use Tableau or some other BI tool [1] (not free though).

There are free alternatives, but many require programming.

[1] https://www.absentdata.com/tableau-alternatives/

Re: We’re happy with SQLite and not urgently interested in a fancier DBMS (2016)

#112

Post by [deleted] »

merb wrote: post #110
jstimpfle wrote: post #49

Earlier quoted context omitted.

> 0h to manage backups ("cp") Are you aware that's unsafe? To make a safe backup use sqlite3's ".dump" command (or filesystem snapshotting, but I've had bad experiences with that, at least on btrfs).

> > > 0h to manage backups ("cp") > > 0h to manage backups ("cp") not if you are on btrfs and use a snapshot (same for xfs or even lvm)

[deleted]

Re: We’re happy with SQLite and not urgently interested in a fancier DBMS (2016)

#113

Post by ufmace »

I am on the train that the idea of using SQLite for a webapp, even one where SQLite can handle the traffic just fine, isn't nearly as good of an idea as some would have you think. Nevertheless, the idea of switching a local end-user application off of SQLite in favor of a client-server RDBMS is bonkers IMHO. You would be making everything about the application dramatically more complex for no real benefit. This blog post is considerably nicer than the idea deserved.

Re: We’re happy with SQLite and not urgently interested in a fancier DBMS (2016)

#114

Post by laumars »

wenc wrote: post #60
Someone1234 wrote: post #47

Earlier quoted context omitted.

CSV's biggest competitor is XML. But both XML and SQLite have the same issue: They give you just enough rope to hang yourself with. While SQLite is a fantastic micro-database engine and a file format, it isn't a very good universal B2B format because there are too many features you'd have to support to interoperate. I'd argue CSV's biggest strength is that it is easy to parse correctly, because the format is so simpl…

> I'd argue CSV's biggest strength is that it is easy to parse correctly Not sure I resonate with that. I work with large CSVs of varying provenance every day (I work in big data) and there's always some CSV edge case that stymies my analysis pipeline. Timestamp parsing is extremely hard if it's not ISO-8601, as well as handling of unicode encoding, missing data, type inference, European usage of , as a decimal point…

sqlite wouldn't solve any of those issues because it's dynamically typed and what you're asking for is something which is strictly typed. In that regard you're better off with a data format like YAML.

I would also disagree that Excel does a good job with CSV. Well, maybe opening files but certainly not saving them as it rewrites values in weird ways. Back when I used to have to handle CSVs regularly (which admittedly was a while ago now) I found OpenOffice Calc to be far superior in terms of CSV support. I'd assume the same is true for LibreOffice Calc but honestly I've not done a huge amount with CSV since.

Re: We’re happy with SQLite and not urgently interested in a fancier DBMS (2016)

#115

Post by libx »

BjoernKW wrote: post #71
Someone1234 wrote: post #47

Earlier quoted context omitted.

CSV's biggest competitor is XML. But both XML and SQLite have the same issue: They give you just enough rope to hang yourself with. While SQLite is a fantastic micro-database engine and a file format, it isn't a very good universal B2B format because there are too many features you'd have to support to interoperate. I'd argue CSV's biggest strength is that it is easy to parse correctly, because the format is so simpl…

> that it is easy to parse correctly While that's not exactly correct - CSV can be quite hard to parse if you want to cover each and every variation and edge case - for all practical intents and purposes that statement is true in my opinion. CSV is something of a lowest common denominator, a compromise between well-defined, structured data and portability / accessibility.

CSV could be much easier to parse if people use the ASCII codes meant to be used as field and line separators, thus avoiding having problems with commas.

See https://ronaldduncan.wordpress.com/2009/10/31/text-file-form...

Re: We’re happy with SQLite and not urgently interested in a fancier DBMS (2016)

#116

Post by trumped »

batmansmk wrote: post #30

We use SQLite in an app with about 1,000 active users. It took us: - 0h to manage backups ("cp"), - 0h to manage seeds and tests fixtures ("cp"), - 0h to configure and secure (void), - 0h to write the deployment scripts (void), - 0h monitoring/watchdog jobs (void), - 1h to rsync for failover ("rsync") My last projects always spent at least a good 100h to do all of this the right way. Then, if it is not good enough, w…

I've had locking issues with an sqlite db while being the only user (only 2 processes were accessing it), I guess I'm doing something majorly wrong...

Re: We’re happy with SQLite and not urgently interested in a fancier DBMS (2016)

#117

Post by candiodari »

jstimpfle wrote: post #87
ufo wrote: post #81

Earlier quoted context omitted.

I'm not very familiar with the details here. What are the problems with cp?

cp does not make atomic snapshots. It copies by reading (usually sequentially) chunk by chunk from the source file, and writing these chunks to the destination file. This takes time. If the database has writes at the time of backup, the backup might be invalid (it contains some old parts and some new parts). (Unless you use e.g. the --reflink option of GNU cp, in which case it makes atomic snapshots on filesystems th…

Isn’t one of the more important points of POSIX that reading sequentially results in an atomic copy ? As long as you keep the file handle open and the fs supports it.

Re: We’re happy with SQLite and not urgently interested in a fancier DBMS (2016)

#118

Post by petre »

jimktrains2 wrote: post #105
Sohcahtoa82 wrote: post #83

Earlier quoted context omitted.

> I'm just not sold on it as a multi-process, multi-user database. That's fine. It was never designed to be a multi-process, multi-user database. Even the authors don't pretend it works in that use case: https://www.sqlite.org/whentouse.html

But that's my point. People using sqlite as the store for a web application seems like the wrong tool for the job.

Why, if you just read from it? It's single writer multiple reader.

Re: We’re happy with SQLite and not urgently interested in a fancier DBMS (2016)

#119

Post by dang »

marknadal wrote: post #50

I agree wholeheartedly with this article, despite being somebody who has written a "fancy" popular database system ( https://github.com/amark/gun ). SQLite is good enough for pretty much everything, just use it!!!

You've posted countless comments like this to promote your database. Often you include a lot of low-substance, barely relevant verbiage as packing, if not camouflage, for the promotion. That doesn't help.

I don't say it lightly, but you're basically spamming HN, as users have been complaining for a long time (e.g. [1], [2]). Even when you don't appear to be doing it, you're doing it [3]. It's past time for this to stop; please stop.

1. https://news.ycombinator.com/item?id=16893429

2. https://news.ycombinator.com/item?id=16677493

3. https://news.ycombinator.com/item?id=16749503

Re: We’re happy with SQLite and not urgently interested in a fancier DBMS (2016)

#120

Post by kimi »

kozhevnikov wrote: post #5
kimi wrote: post #4

Related note: in the last year I started using SQLite with my Clojure scripts instead of using a remote database, and I'm storing everything in git. Works like a charm for single-user tasks.

Do you store the binary .sqlite in Git or source .sql and build the .sqlite?

I store the binary; it's small. But of course you could use a filter, e.g. https://github.com/DataWraith/gitfilters
Post reply on HN