Earlier quoted context omitted.
> The kernel can map that data in dynamically much faster than a userland process can. Not necessarily. The kernel's mmap implementation has quite a strong bias towards certains kinds of access patterns; deviate from them and it can become slower than read(2). We tried using mmap(2) for audio file i/o in Ardour, and it got notably less bandwidth than just using read(2).
I'm curious if you tried different madvise strategies and if any of them worked better than others?
Do you even need a database?
271–280 of 311 posts
Re: Do you even need a database?
#272For anything that starts as a side project, SQLite covers it. The moment you need to ask this question the answer is usually "not yet." Ship first, migrate when it actually hurts
Most software is stateful and needs to persist state across restarts, so I would argue that one needs at least SQLite.
On SQLite being safe default: in practice it means supporting multiple databases, say SQLite and Postgres, this is more complicated that supporting just Postgres. As soon as a project leaves localhost and enters cloud development you need talk to a database over network, which warrants MySQL or Postgres.
Which is more complicated: supporting a docker container with mysql or Postgres for local development OR supporting multiple databases in the project?
Of course, the answer could be “it depends”, I but I would not call SQLite a default choice. It would be if you are writing desktop or mobile app, but for anything like a web app it’s a questionable choice.
Re: Do you even need a database?
#273Earlier quoted context omitted.
SQLite has become my new go-to when starting any project that needs a DB. The performance is very fast, and if anything is ever successful enough to outgrow SQLite, it wouldn't be that hard to switch it out for Postgres. Not having to maintain/backup/manage a separate database server is cheaper and easier.
Backups are super-simple as well. I'm also a convert.
Re: Do you even need a database?
#274* Not always super easy
Re: Do you even need a database?
#275Earlier quoted context omitted.
Backups are super-simple as well. I'm also a convert.
why sqlite over postgres?
Re: Do you even need a database?
#276Re: Do you even need a database?
#277Earlier quoted context omitted.
Good points, but Postgres has all those, along with much better local testing story, easier and more reliable CDC, better UDFs (in Python, Go etc.), a huge ecosystem of extensions for eg. GIS data, no licencing issues ever, API compatability with DuckDB, Doris and other DBs, and (this is the big one) is not Oracle.
Unless I’ve missed something, Postgres doesn’t have automatic index creation, nor does it have JSON introspection to automatically convert it to a normalized schema (which is insane; I love it). It also doesn’t do any kind of sharding on its own, though of course forks like Citus exist. It definitely doesn’t do RAC / Exadata (not sure which part this falls under), where multiple nodes are connected and use RDMA to tr…
Index creation https://stackoverflow.com/questions/23876479/will-postgresql...
JSON->DB schema https://jsonschema2db.readthedocs.io/en/latest/index.html
Pg shared disk failover is similar but RAC is quite unique, you’re not going to use though with a rented cluster?
https://www.postgresql.org/docs/current/different-replicatio...
Personally for me any technical advantages don’t outweigh the business side, YMMV :)
Re: Do you even need a database?
#278Earlier quoted context omitted.
Backups are super-simple as well. I'm also a convert.
why sqlite over postgres?
Re: Do you even need a database?
#2791. You're paying very little for 1000x the features.
2. The chances your application will keep doing "pure ID lookups" forever are zero. The moment you need to query the data in any other way or have more than one writer you're going to have to throw all this nonsense code into the trash.
3. Do you need the 1.7x speedup? No, of course you don't. It's just optimizing for the sake of optimizing.
I'd have just used sqlite to begin with and not benchmarked anything. No custom code, no need to waste any time, great performance, amazing flexibility — all with minimum effort.
Re: Do you even need a database?
#280Earlier quoted context omitted.
Unless I’ve missed something, Postgres doesn’t have automatic index creation, nor does it have JSON introspection to automatically convert it to a normalized schema (which is insane; I love it). It also doesn’t do any kind of sharding on its own, though of course forks like Citus exist. It definitely doesn’t do RAC / Exadata (not sure which part this falls under), where multiple nodes are connected and use RDMA to tr…
I think that’s the beauty of PG here, you can find solutions to most of this: Index creation https://stackoverflow.com/questions/23876479/will-postgresql... JSON->DB schema https://jsonschema2db.readthedocs.io/en/latest/index.html Pg shared disk failover is similar but RAC is quite unique, you’re not going to use though with a rented cluster? https://www.postgresql.org/docs/current/different-replicatio... Personally…
> Index creation https://stackoverflow.com/questions/23876479/will-postgresql...
I was ambiguous. That's an answer telling how to create indexes manually, and saying that you get an index for primary keys and unique constraints automatically. Sure, all databases do that. Oracle can create arbitrary indexes for any relation in the background without it being requested, if it notices that common queries would benefit from them.
Forgetting to create indexes is one of the most common issues people face when writing database apps because the performance will be fine on your laptop, or when the feature is new, and then it slows down when you scale up. Or worse you deploy to prod and the site tanks because a new query that "works fine on my machine" is dog slow when there's real world amounts of data involved. Oracle will just fix it for you, Postgres will require a manual diagnosis and intervention. So this isn't the same capability.
> JSON->DB schema https://jsonschema2db.readthedocs.io/en/latest/index.html
Again I didn't provide enough detail, sorry.
What that utility is doing is quite different. For one, it assumes you start with a schema already. Oracle can infer a schema from a collection of documents even if you don't have one by figuring out which fields are often repeated, which values are unique, etc.
For another, what you get after running that utility is relational tables that you have to then access relationally via normal SQL. What JSON duality views give you is something that still has the original document layout and access mode - you GET/PUT whole documents - and behind the scenes that's mapped to a schema and then through to the underlying SQL that would be required to update the tables that the DB generated for you. So you get the performance of normalized relations but you don't have to change your code.
The nice thing about this is it lets developers focus on application features and semantics in the early stages of a startup by just reshaping their JSON documents at will, whilst someone else focuses on improving performance and data rigor fully asynchronously. The app doesn't know how the data is stored, it just sees documents, and the database allows a smooth transition from one data model to another.
I don't think Postgres has anything like this. If it does it'll be in the form of an obscure extension that cloud vendors won't let you use, because they don't want to/can't support every possible Postgres extension out there.