31–40 of 125 posts
Earlier quoted context omitted.
Congratulation, you just rewrote the base for an orm.
More of a DAO than an ORM.
Earlier quoted context omitted.
Not only desktop, but most mobile platforms as well. Beeing coded in a single C file makes it portable as hell. Their documentation is also a prime example, with great insight.
Not only desktop and mobile, but it can be used as the storage format for larger scale distribued systems. For example it's the on disk format for FoundationDB. >Beeing coded in a single C file makes it portable as hell. It's not coded in a single C file...
Abstract the persistence layer, document the contract, build a conformance suite, and write your one impl that uses sqlite. I know sometimes there seems to be an impedance mismatch between callers and databases, but often it's not that bad. And no, don't use an ORM or existing DB abstraction to do this. Sure it takes work, but pluggable storage is nice on some of these types of things.
I know this is kind of a snarky response, but seriously... why? This would take developer resources away from real feature work, introduce boatloads of complexity, add more points of failure, make testing and validation harder and make it harder to reason about the entire system end-to-end. Not that that isn't sometimes necessary, but you need a compelling reason, not just jumping into it because it sounds like a good idea.
Programmers have this strange compulsion to design swappable interfaces even when they're totally unnecessary and more often than not just break everything while– ironically– only one implementation is ever used.
YAGNI.
We use SQLite in an app with about 1,000 active users. It took us: - 0h to manage backups ("cp"), - 0h to manage seeds and tests fixtures ("cp"), - 0h to configure and secure (void), - 0h to write the deployment scripts (void), - 0h monitoring/watchdog jobs (void), - 1h to rsync for failover ("rsync") My last projects always spent at least a good 100h to do all of this the right way. Then, if it is not good enough, w…
Do you have multiple servers? How do you handle real-time filesystem sync? If you only have one server, how do you handle its inevitable failure?
Do you loose any data collected between last backup and failure?
(I'm not saying SQLite isn't good, because it's f-ing amazing. I'm just not sold on it as a multi-process, multi-user database.)
(If you have a filesystem capable of atomic snapshots, then you can simply snapshot any database and treat the backup as a single file.)
CSV can be difficult and ambiguous to parse correctly (because there's no real standard) and isn't extremely performant. The only thing it has going for it is its universality.
SQLite is lightweight, structured, supports indexing for performance and is extremely easy to use.
SQLite is a fantastic database for desktop applications. https://www.sqlite.org/appfileformat.html is a great essay on its benefits for this kind of use-case.
Not only desktop, but most mobile platforms as well. Beeing coded in a single C file makes it portable as hell. Their documentation is also a prime example, with great insight.
This is just a thought: SQLite is a really good file format. Why aren't we replacing CSV with it, especially for big data applications? CSV can be difficult and ambiguous to parse correctly (because there's no real standard) and isn't extremely performant. The only thing it has going for it is its universality. SQLite is lightweight, structured, supports indexing for performance and is extremely easy to use.
I'm inclined to say we are. R, for instance, includes an embedded sqlite, and it seems to be used a lot.
Related link: https://sqlite.org/appfileformat.html
Abstract the persistence layer, document the contract, build a conformance suite, and write your one impl that uses sqlite. I know sometimes there seems to be an impedance mismatch between callers and databases, but often it's not that bad. And no, don't use an ORM or existing DB abstraction to do this. Sure it takes work, but pluggable storage is nice on some of these types of things.
Why? I know this is kind of a snarky response, but seriously... why? This would take developer resources away from real feature work, introduce boatloads of complexity, add more points of failure, make testing and validation harder and make it harder to reason about the entire system end-to-end. Not that that isn't sometimes necessary, but you need a compelling reason, not just jumping into it because it sounds like…
To allow others to impl the data store they want and to avoid having to write blog posts like these.
It's not that much real work to quickly pull out an iface on a stable system. Assuming it would be just mostly pass-through to your main impl anyways and this is a common approach in refactorization. You don't need some huge system, just drop it back a layer and abstract it.
Often, what you end up learning about your system when you do this minimal work (again, nothing big) is that you screwed up and buried SQL and string concatenation and a bunch of other hard-to-review pieces all throughout your app because you keep a stranglehold on terms like YAGNI when separation of concerns has real value. Sometimes you also end up learning that your tests don't cover what you thought they did. Nobody wants bloat, but this adds very little and helps devs clamoring for alternative stores.
> The idea is that a more complicated DBMS should be faster, especially for huge music libraries. How "huge" are they talking about? I built a tool that imports Apache logs into sqlite for quick analysis and that easily handles several million records.
Just a ballpark: my beets DB is 15 MB for ~1000 albums (well, folders). This post[0] claims 10M albums ever released. So maybe 150 GB upper bound? Assuming my db has been vacuumed recently (~~probably not true~~ Edit: nope, still 15M after vacuum) and ignoring that indexes will scale slightly non-linearly. [0]: https://www.quora.com/How-many-music-albums-are-available-in...
My music collection is massive. It's got 30 years of singles and albums in there. And several years of DJing too. Plus a massive amount of DJ sets (like several hundred gig of DJ sets alone). Yet XBMC / Kodi handles it fine "despite" being sqlite backed. So does Subsonic and that just uses some Java equivalent. And as I've said already, I can throw several million records of Apache logs into sqlite without any issue.
The performance of sqlite is actually really good. It's super fast. Which is why the author can make those boasts on its website. Plus the compactness of the db (ie one file produced) and the ease to create a db makes it a perfect choice for desktop and mobile applications.
I think some people read that WordPress runs on MySQL or spend 10 minutes building their first CMS in PostgreSQL and then think the world and their mother should be using this miraculous new RDBMS they've just discovered themselves. (I'm probably being too harsh there)
This is just a thought: SQLite is a really good file format. Why aren't we replacing CSV with it, especially for big data applications? CSV can be difficult and ambiguous to parse correctly (because there's no real standard) and isn't extremely performant. The only thing it has going for it is its universality. SQLite is lightweight, structured, supports indexing for performance and is extremely easy to use.