Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite the only database you will ever need in most cases (2021)

unixsheikh.com

Post reply on HN
Print view View original

151–160 of 378 posts

Re: SQLite the only database you will ever need in most cases (2021)

#151

Post by berdon »

JohnBooty wrote: post #134
bogwog wrote: post #86

Earlier quoted context omitted.

> I'm an application developer and I do not want to become a release engineer. I resent even having to learn Docker. :-) Sorry, but that's a terrible attitude to have. You don't need to be a full-blown sysadmin to know how to do basic deployments, and learning these things will make you a better developer.

You don't need to be a full-blown sysadmin to know how to do basic deployments, and learning these things will make you a better developer. Yes. True. However, it is also true that we have a limited amount of time per week and a limited number of weeks on Earth. Time spent learning sysadmin-y stuff is less time spent mastering developer-y stuff. Think about what it means to be a "full stack" engineer in 2023: - Unix…

Fwiw, I’ve used all of those in my career, extensively here and there. I’ve written many thousands of lines of bash scripts. Even written my own php-for-bash-script style code tags that support arbitrary shells. I’ve written my own log based distributed kv store, gone down the YouTube trail of writing my own db, gone through the angular and react iterations, deep dove in docker, docker compose, k8s, crds and custom handlers, and handled infrastructure via infra as code and with a control plane.

I also have a wife and a 2.5 year old.

It’s certainly possible if you don’t stop yourself.

Re: SQLite the only database you will ever need in most cases (2021)

#152

Post by chillfox »

kcb wrote: post #129
chillfox wrote: post #76

Earlier quoted context omitted.

Most of the time people separate the app and database into two different VMs that the infrastructure team then runs on the same box. edit: This is done not because of any considered technical reasons, but because that's how one learned to deploy apps.

That would never fly in production for any sort of enterprise application with uptime requirements.

And yet it has been done just about everywhere I have worked in the past, The justification is usually that VMWare will move the machines to another host if it goes down.

Re: SQLite the only database you will ever need in most cases (2021)

#154

Post by nl »

kragen wrote: post #17
adamkf wrote: post #10

>The only time you need to consider a client-server setup is: Where you have multiple physical machines accessing the same database server over a network. In this setup you have a shared database between multiple clients. This caveat covers "most cases". If there's only a single machine, then any data stored is not durable. Additionally, to my knowledge SQLite doesn't have a solution for durability other than asynchr…

people have been providing acid transaction semantics on single machines for 50 years do you think ims/db ran on a cluster the d in acid stands for durability you're talking about pitr, which is what mysql semi-sync provides (and afaik you are correct that sqlite doesn't offer pitr)

The OPs point is that the single process ACID semantics of SQLite don't provide a durability guarantee that includes replication.

Other databases have a commit level that makes sure logs have been shipped.

For me this is an edge case in just about everything except financial transactions (the performance penalty of distributed transactions is pretty high!) but it is correct to note it.

Re: SQLite the only database you will ever need in most cases (2021)

#155

Post by chillfox »

crazygringo wrote: post #104
chillfox wrote: post #76

Earlier quoted context omitted.

Most of the time people separate the app and database into two different VMs that the infrastructure team then runs on the same box. edit: This is done not because of any considered technical reasons, but because that's how one learned to deploy apps.

It's because the very first step in scaling will usually be separate machines for webserver and database. And it costs almost nothing to write it that way from the start, but it's a pain to separate them out later. I'd call that a considered technical reason.

And so it is, but I have never gotten that as an answer when asking.

edit: I feel like I should be more specific here. I would only call it a considered technical reason if it was actually considered. The fact that it is possible to come up with good reasons is not relevant if no thought went into it at the time of design/development.

Re: SQLite the only database you will ever need in most cases (2021)

#156

Post by joshspankit »

I’ve argued this before and I’ll argue it here now:

Modern computers are fast enough that in many cases “the only database you will ever need” can be files on the filesystem. For example “1 row = 1 file”.

It brings additional benefits as well: for low-write applications you can use git to get a history (+transactions if you store them in the log), backups are super easy, replication is trivial. For higher-write applications it gets more complex but you can still plan and implement most of the traditional DB scaling techniques (and even implement them one at a time as you go grow).

Computers are “stupid fast” now that we’ve gotten off platters.

Re: SQLite the only database you will ever need in most cases (2021)

#158

Post by sanderjd »

danielmarkbruce wrote: post #116
sanderjd wrote: post #102

Earlier quoted context omitted.

> It's not a good option for what you appear to want. Ok but the article you're replying in the comments to says "SQLite is all you need for nearly everything", and what the comment you're replying to is describing is, to use their very apt word choice, entirely ordinary . So how do we square this circle of somebody being told both "SQLite is all you need for everything" and "it is not a good option for your totally…

Third paragraph of the article says: "In contrast to many other database management systems, SQLite is not a client-server database engine, but you actually very rarely need that. If your application software runs on the same physical machine as the database, which is what most small to medium sized web applications does, then you probably only need SQLite." That's how we square it. It's right there in the article.

I've realized that this whole thing comes down to the word "most" doing too much duty here. I don't think it's true that most applications just run a single db-and-application node. I've never worked on such an application. You and the author do seem to think this is true. It would be difficult for either of us to support our intuition empirically, so this is where the divide comes from.

Re: SQLite the only database you will ever need in most cases (2021)

#159

Post by EGreg »

joshspankit wrote: post #156

I’ve argued this before and I’ll argue it here now: Modern computers are fast enough that in many cases “the only database you will ever need” can be files on the filesystem. For example “1 row = 1 file”. It brings additional benefits as well: for low-write applications you can use git to get a history (+transactions if you store them in the log), backups are super easy, replication is trivial. For higher-write appli…

Transactions, really? How do you lock rows, how do you have relations, how do you do joins? In fact, ext3 can only handle about 50,000 files in one directory. So you'll have to split up your "primary key" into letters like abc/def/foo like we do

Re: SQLite the only database you will ever need in most cases (2021)

#160

Post by mjmsmith »

berdon wrote: post #151
JohnBooty wrote: post #134

Earlier quoted context omitted.

You don't need to be a full-blown sysadmin to know how to do basic deployments, and learning these things will make you a better developer. Yes. True. However, it is also true that we have a limited amount of time per week and a limited number of weeks on Earth. Time spent learning sysadmin-y stuff is less time spent mastering developer-y stuff. Think about what it means to be a "full stack" engineer in 2023: - Unix…

Fwiw, I’ve used all of those in my career, extensively here and there. I’ve written many thousands of lines of bash scripts. Even written my own php-for-bash-script style code tags that support arbitrary shells. I’ve written my own log based distributed kv store, gone down the YouTube trail of writing my own db, gone through the angular and react iterations, deep dove in docker, docker compose, k8s, crds and custom h…

Maybe he wants to spend more of his time with his wife and floating point child.
Post reply on HN