Live data from Hacker News

We’re happy with SQLite and not urgently interested in a fancier DBMS (2016)

beets.io

101–110 of 125 posts

Re: We’re happy with SQLite and not urgently interested in a fancier DBMS (2016)

#101
post #35

This is just a thought: SQLite is a really good file format. Why aren't we replacing CSV with it, especially for big data applications? CSV can be difficult and ambiguous to parse correctly (because there's no real standard) and isn't extremely performant. The only thing it has going for it is its universality. SQLite is lightweight, structured, supports indexing for performance and is extremely easy to use.

> Why aren't we replacing CSV with it, especially for big data applications? Big data applications tend to use other structured binary formats like parquet and avro, which any big data tool can typically parse.

Speaking for an Engineer unfortunately often data is simple CSV. Parsing this data robustly and visualizing it without scripting is still something that hasn't been solved.

If you have any idea how to do it, I'd be more than glad to hear about it.

Re: We’re happy with SQLite and not urgently interested in a fancier DBMS (2016)

#102
post #92
post #39

Earlier quoted context omitted.

I think 10m is far too low to be the total number of albums. But it would be impossible to actually calculate that figure given the number of producers that have released content outside the scope of any particular authority (eg NIN has released stuff to download only from their site. Aphex Twin uploaded a load of stuff to SoundCloud. Etc). But we digress. My music collection is massive. It's got 30 years of singles…

Yeah. To be clear, I think even a 150GB sqlite database would perform just fine for beets' purposes.

Even as a heavy PostgreSQL fan and promoter I don't see why you would choose anything BUT sqlite for single-user applications unless you needed something that only other databases provide (richer procedural language support or other extensions that the fairly basic but still adequate featureset SQLite provides) - size of the dataset is only one factor to consider when choosing the storage layer for your application, and there's a lot of valid solutions for handling 1TB or less of data easily.

Re: We’re happy with SQLite and not urgently interested in a fancier DBMS (2016)

#103

Earlier quoted context omitted.

> To allow others to impl the data store they want and to avoid having to write blog posts like these. But why? For the longest time MySQL didn't support window function or CTEs. What do you do if you're using those? (I'm not sure of SQLite's window support, but I'm pretty sure it has CTEs.) If you're using postgres, should you not use pg_trgm (trigram indexer to use an index for regex searches on text), postgis (GIS…

I think your talking about SQL abstraction and I'm talking about persistence abstraction. I would never consider sharing SQL across vendors.

The way you persist has consequences to what types of features you'll build, and it'll end up being the least common denominator, not allowing you to use any advanced or efficient features of your store.

Re: We’re happy with SQLite and not urgently interested in a fancier DBMS (2016)

#105

Earlier quoted context omitted.

Do you have a write-heavy workload? How do you handle contention? Do you have multiple servers? How do you handle real-time filesystem sync? If you only have one server, how do you handle its inevitable failure? Do you loose any data collected between last backup and failure? (I'm not saying SQLite isn't good, because it's f-ing amazing. I'm just not sold on it as a multi-process, multi-user database.) (If you have a…

> I'm just not sold on it as a multi-process, multi-user database. That's fine. It was never designed to be a multi-process, multi-user database. Even the authors don't pretend it works in that use case: https://www.sqlite.org/whentouse.html

But that's my point. People using sqlite as the store for a web application seems like the wrong tool for the job.

Re: We’re happy with SQLite and not urgently interested in a fancier DBMS (2016)

#106
post #60

Earlier quoted context omitted.

> I'd argue CSV's biggest strength is that it is easy to parse correctly Not sure I resonate with that. I work with large CSVs of varying provenance every day (I work in big data) and there's always some CSV edge case that stymies my analysis pipeline. Timestamp parsing is extremely hard if it's not ISO-8601, as well as handling of unicode encoding, missing data, type inference, European usage of , as a decimal point…

> Excel's CSV parsing isn't the greatest, but does a surprisingly decent job considering how ill-defined CSV is. No, it really doesn't. Save this as a CSV, open it in Excel, hit save, and then review the raw CSV: "1000000000012345", "Hello", "2000000000067890", "World", Here's what Excel (O365) does to it for me: 1E+15, Hello, 2E+15, World, That data is now permanently lost. And this corruption occurs for almost all…

1. I think in the first instance, Excel performs type-inference and coerces it into a numeric type and writes a truncated precision version of numeric fields in CSV, which is an abominable sin. There are workarounds for this [1], but agree this is terrible behavior due to CSV's lack of types.

Arguably, there should have been an explicit Excel switch which forces all CSV data to be parsed as raw strings.

I've seen problems with Excel CSVs that are worse than that: I have serial numbers that have leading 0's that have semantic meaning, like 000002324122323. Most CSV parsers cannot tell that this isn't a numeric type and so handle it wrongly. So I resort to [1].

2. Big data is of course somewhat ambiguous nomenclature as well, but is nowadays typically understood to mean the Hadoop ecosystem or similar. Data is typically ingested into a distributed file system (HDFS, S3, etc.) in formats such as Parquet, Avro, JSON and often CSV. A schema-on-read database like Hive sits on top of this layer and presents a SQL interface to the user. Tools like Apache Spark provide programmatic transformations that operate on the data on the large.

CSV is often promoted as a format for storing structured data due to ease of ingestion and inspection (all you need is a text editor for troubleshooting). However, you pay a performance penalty every time an analytic query is run because CSV doesn't support indexes, predicate pushdowns, compression, etc. and records can sometimes be uninterpretable under certain schemas (in which case they are simply excluded or dropped).

Sometimes having one too many commas in a row can completely mess things up (all the columns get shifted in a record) -- I found this out the hard way when Spark exported malformed CSVs with un-escaped commas.

[1] http://support.pitneybowes.com/SearchArticles/VFP06_Knowledg...

Re: We’re happy with SQLite and not urgently interested in a fancier DBMS (2016)

#108
post #35

This is just a thought: SQLite is a really good file format. Why aren't we replacing CSV with it, especially for big data applications? CSV can be difficult and ambiguous to parse correctly (because there's no real standard) and isn't extremely performant. The only thing it has going for it is its universality. SQLite is lightweight, structured, supports indexing for performance and is extremely easy to use.

Can you open a SQLite database in Excel, without programming skills?

There's your reason.

Re: We’re happy with SQLite and not urgently interested in a fancier DBMS (2016)

#109

Earlier quoted context omitted.

I'd actually argue that CSV's biggest competitor specifically is xlsx. Which is kind of XML...

I'm talking about business to business transactions, as in automated transactions on VANs and similar data interchanges. The industry doesn't really use XLSX or XLS because they're unreliable to parse without Microsoft's Office libraries which require Windows licenses, and the two formats have a huge amount of overhead compared to XML formats and CSV (which costs money on VANs). This is a space where plain text files…

> The industry doesn't really use XLSX or XLS because they're unreliable to parse without Microsoft's Office libraries

XLS yes, however XLSX is just easy to parse if you document only contains simple types. of course with embedded mathml/wordml it gets a little bit wierd.

Re: We’re happy with SQLite and not urgently interested in a fancier DBMS (2016)

#110

We use SQLite in an app with about 1,000 active users. It took us: - 0h to manage backups ("cp"), - 0h to manage seeds and tests fixtures ("cp"), - 0h to configure and secure (void), - 0h to write the deployment scripts (void), - 0h monitoring/watchdog jobs (void), - 1h to rsync for failover ("rsync") My last projects always spent at least a good 100h to do all of this the right way. Then, if it is not good enough, w…

> 0h to manage backups ("cp") Are you aware that's unsafe? To make a safe backup use sqlite3's ".dump" command (or filesystem snapshotting, but I've had bad experiences with that, at least on btrfs).

> > > 0h to manage backups ("cp")

> > 0h to manage backups ("cp")

not if you are on btrfs and use a snapshot (same for xfs or even lvm)

Post reply on HN