Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite the only database you will ever need in most cases (2021)

unixsheikh.com

Post reply on HN
Print view View original

311–320 of 378 posts

Re: SQLite the only database you will ever need in most cases (2021)

#311

Post by bravura »

ilyt wrote: post #307
bravura wrote: post #124

Earlier quoted context omitted.

I used slackware linux. I wrote assembly code when I was a teenager. I use 'perl -i -pe' constantly. I still don't want to learn a bunch of arcane flags from a bunch of tools with a million gotchas. Sorry. I'd rather focus on the code craft that gives me joy these days. Sorry, I detest this attitude of: "Well I was hazed so why aren't you hazed too?" Did I mention that Russ Cox and I used to write C code with pen and…

I mean, you can just put it into automation and never have to repeat same linux cmdline dance to do the same thing again. There are plenty of configuration management tools to do so. Hell, I outright forgot some skills because I encoded whatever I will ever need from a given tool in Puppet manifest and didn't had a need to touch the internals again. Docker, k8s and friends are essentially just level above the "just C…

I didn't spend six years on a PhD to be called Mr. Important.

I'm joking.

Okay, bringing this conversation back to a normal tone: I have specific needs and want to get specific things done. I agree everyone should learn basic unix. I also think people should get as low to the metal as possible so they understand performance.

But can you acknowledge that there is a near infinite number of things people should learn, and that the priority of that list might be different for other people? That for some, and their technical goals, they might not have the time to prioritize multi-server web app dev when they're focusing on figuring out how to get GPUs to do DDP correctly? I never learned k8s because I all my web apps have super low usage. e.g. data annotation frameworks. I don't know JS and I guess maybe I'd love to, but I'd also like to wait until the field stops moving so correctly so that I could just pick a great framework and not have to relearn everything constantly. HTMX is cool and gives me joy.

Can you acknowledge that building tooling that allow devs to focus on what gives them joy is actually a nice thing? We have enough problems in the work we like that we end up spending endless hours debugging. If someone refuses to learn what they need to solve their main technical pursuits, yeah that's a moral failing, but nonetheless there's a finite surface area we can cover and I prefer to focus on learning the things that come up in the line of duty.

p.s. what are you working on? Maybe we can play together. I am not being snarky. I'm collaborative and maybe you're cool too. :)

Re: SQLite the only database you will ever need in most cases (2021)

#312

Post by lenkite »

irskep wrote: post #29

This sentiment pops up regularly on HN, and I've seen at least one article per month for the past few months, but the trouble is, none of them seem to help you actually deploy it. They assume you're comfortable spinning up public web servers. If you want to use a PaaS to deploy an app, because you don't want to spend your time learning to be a sysadmin, then all the tutorials are going to put you on the Postgres path…

Also "as long as your web application can run on the same machine as the database, which it can in 99% of the time"

Where did 99% of the time come from ?

Re: SQLite the only database you will ever need in most cases (2021)

#313

Post by nikeee »

srcreigh wrote: post #279
nikeee wrote: post #179

I really like SQLite and I use it a lot. There are only two things that are missing to make it near-perfect: - A type for Instants / time handling - being strict with types. No inserts of ints into a string column

SQLite added strict mode recently, check it out. Agreed time handling is sub par - could use builtin date formatting from epoch to ISO which would fix all the problems IMO.

I know about STRICT tables [0], but they still follow the quirky coercion rules. The reasoning seems to be that other DBMs have a similar behaviour. However, I want _errors_ if I insert '123' into an INT column, so it's easier to find problems in my code.

[0]: https://www.sqlite.org/stricttables.html

Re: SQLite the only database you will ever need in most cases (2021)

#314

Post by christophilus »

deathclassic wrote: post #11

I like sqlite as much as the next guy but it's built-in datatypes are limited. Things like arrays, UUIDs, geometry stuff, JSON, etc. Sure you can store more advanced stuff as blobs or text but then you have to mess around with deserializing it in the host language and you lose the ability to query it directly in the db engine.

Lack of int64 is also a problem for some applications.

Re: SQLite the only database you will ever need in most cases (2021)

#315

Post by christophilus »

0xbadcafebee wrote: post #168

I agree. Most uses of databases definitely don't need to grow larger than, say, a single filesystem, or a single application, or a single host, or a single network, or a single geographical region, or a single customer, or a single organization, or a single global network of customers in organizations in regions on networks on hosts on applications on filesystems. There could not be any features of any other database…

I don't think this characterization of SQLite advocates is fair. I've written pet database systems from scratch, studied CS theory at university--including databases, worked for big, famous tech companies.

Looking back on my long career, I can say that literally every application I've ever worked on-- with one exception-- could have been run on a single modern machine, probably with SQLite as a backing store. I'm not sure, but there's a pretty good case that such a design would have improved things a fair bit.

The network was never our problem. The hardware turned out to not be the cause of any of our outages. Outages were always caused by a software bug or a misconfiguration of our complex applications. These misconfigurations were in large part caused by avoidable complexity. The complexity was often due to us trying to solve for scale and reliability problems that we didn't really have.

You can easily host multiple customers or organizations in any database. That's not something that SQLite or SQL Server or Postgres helps or hinders.

Very few of us ever work on a Google or Facebook scale-- even though we think we do.

Anyway, in my career, the one exception I can think of was a big-data application that really did need many servers just to store the actual petabytes of data. Even there, though, we could probably have partitioned the data across SQLite instead of Elastic and probably done alright.

All of that said, I run Postgres in production like a sane person, because I don't want to have to deal with managing a database. Render manages it for me so that I can focus on reading Hacker News. So, this entire rant is moot.

Re: SQLite the only database you will ever need in most cases (2021)

#316

Post by ilyt »

bravura wrote: post #311
ilyt wrote: post #307

Earlier quoted context omitted.

I mean, you can just put it into automation and never have to repeat same linux cmdline dance to do the same thing again. There are plenty of configuration management tools to do so. Hell, I outright forgot some skills because I encoded whatever I will ever need from a given tool in Puppet manifest and didn't had a need to touch the internals again. Docker, k8s and friends are essentially just level above the "just C…

I didn't spend six years on a PhD to be called Mr. Important. I'm joking. Okay, bringing this conversation back to a normal tone: I have specific needs and want to get specific things done. I agree everyone should learn basic unix. I also think people should get as low to the metal as possible so they understand performance. But can you acknowledge that there is a near infinite number of things people should learn, a…

>I don't know JS and I guess maybe I'd love to, but I'd also like to wait until the field stops moving so correctly so that I could just pick a great framework and not have to relearn everything constantly.

/js rant start

I think when that happens it won't be because JS stopped moving, it will be because people will just go "fuck it" and replace it wholly with + WASM.

At least that's how I managed how to not learn JS properly for years. Web frontend work is just drudgery, and I only did it few times when we decided trying to explain a frontend developer ops stuff to make admin page out of is more complex that stitching some shitty code together.

And every single time it was miserable experience and I think only thing that I made and is still in supported framework is fucking jQuery, because every other one seems to shit on backward compatibility and decide to just make new framework. Then call it same name but just increase the number after the name.

/js rant end

> Can you acknowledge that building tooling that allow devs to focus on what gives them joy is actually a nice thing?

I did just say that yes we have those tools and it is fine that dev might not care about internals, I just want to highlight that the whole "I don't care what's underneath as long as it works" might eventually bite, so doing the boring part of understanding something about underlying system might save a lot of effort going forward.

Especially if that bit extra effort allows for overall simpler architecture, as easier debugging usually comes with it. "Just look in logs" or see what process is doing in system directly using installed tools is infinitely easier than fucking with docker/k8s CLI, sidecars and other methods to observe the app in container.

Recent example: A bunch of devs in company we provide infrastructure for got starry eyed for k8s and we just recently had to re-explain that no, the "just give us big POSIX filesystem mounted everywhere" on dozens of nodes isn't gonna work well, and the fact you can tell k8s "give me storage with ReadWriteMany" won't just work. That was after they tried to make shared block device and run XFS on it, which fell apart for obvious reasons

That after same company had year+ long migration from "a big GFS2 volume that was slow as fuck because GFS2 is slow as fuck when you have dozens of nodes for it" to plain S3.

One meeting later and they figured out they don't even need shared storage in the first place, but nobody researched that before and their solution "worked" on test 1 node k8s cluster (for obvious reasons). While a bit of research would have saved the whole ordeal

So less communicative company might just go "right on, we will set up CephFS for you and go ahead" then get more maintenance and the inevitable "this clustered filesystem doesn't work EXACTLY like my XFS partition on ubuntu" problems. It would probably be better for billable hours tho...

More low-scale example: My VPS just have few systemd services for few apps I put there. unattended-upgrades and it is near-zero maintenance for years. Granted, I knew how to do it because that's my day job but I have 100x less complex setup on my VPS. Maybe I should write blog about it...

> p.s. what are you working on? Maybe we can play together. I am not being snarky. I'm collaborative and maybe you're cool too. :)

Well, pandemic/nearby war kinda fucked up every project I started

I was working on midi2sid chip eurorack module. Which was delayed for year+ just waiting to get parts. I think they might be available again, maybe I should resurrect it. It did ran in rust and actually played notes, I just ran out of pins on the MCU I used so I had nice big board with bigger MCU and some niceties added, made it ready to be assembled... right as the chipaggedon started.

I was also working on car datalogger, even got prototype working, except recent economic fuckery put any track days on hold. The plan was to finally get a house, the inflation put any sensible mortgage out of question for at least nearby future so all the fun money are going to the savings now.

The other plan was to replace car's radio with something newer and integrate that with it (via dumb "just send rPi HDMI image thru the infotainment screen" probably), but yeah, can't really do that without garage in the first place...

I experimented with home automation a bit and had some ideas (involving embedded Rust, it is surprisingly palatable) but, well, again, the plans involved having a home, and not really that much possible to do in rented apartment

In meantime like 30 projects that scratch itch nobody else but me have and were not touched after scratching part started to work fine.

I did decide to implement the CPU from The Art of Computer Programming. But it looked annoyingly weird so I just made Go Z80 "emulator". Then rewrote it in Rust as exercise (both in ImGui as frontend). Then realized that getting the other chips than Z80 "right" is a ton of work and.... not that fun work to boot so I called it learning experience for Rust and ImGui and left it alone. It did run enough assembly to do something tho. No longer runs after Rust version upgrade tho. So much for backward compat...

But currently nothing really interesting. I have those cycles, where if work is interesting I spend a most of free time in brainless/semi-brainless fun bracket, and if work is boring I do the interesting hobby stuff, and, well, recent company buyout and some other changes made sure I'm having all the "fun" at work I can take so I haven't been touching hobbies much.

Re: SQLite the only database you will ever need in most cases (2021)

#317

Post by capableweb »

srcreigh wrote: post #300
bagels wrote: post #292

Earlier quoted context omitted.

Zero downtime upgrades, hardware fault, aws decides that specific instance needs to die. It also doesn't let you cheat statelessness very easily, so it's easier to scale horizontally.

Fair enough I guess. I don’t think you need two servers to do zero downtime upgrades. And the other issues are, imo, beyond the 0.99 uptime threshold that most services realistically have when you add in breakage due to upgrades. I like your statelessness point. I suppose in your view it’s better to have the concentrated stateful core with stateless servers as opposed to just one stateful instance. Two instances mean…

> I don’t think you need two servers to do zero downtime upgrades

Absolutely not and I can't understand why I keep hearing this argument. Doing zero downtime upgrades on a single server have been simple since basically forever, run another process on another port, change config, restart front balancer gracefully and there you go.

Re: SQLite the only database you will ever need in most cases (2021)

#318

Post by bravura »

sanderjd wrote: post #167
pastor_bob wrote: post #121

Earlier quoted context omitted.

Distributed databases are rarely needed. I wouldn't call it an ordinary use case. The article isn't claiming Elon Musk can run Twitter off SQlite

It is honestly difficult for me to imagine what kind of applications people are working on that have these - to me - very lax reliability constraints. But we're just disagreeing over what "most" applications are like, based on our experiences, without any empirical data to say either way, so :shrug:.

Off-the-top of my head.

* Hobby apps designed for yourself and your friends.

* Hobby apps that you hope will become products but probably won't achieve traction.

* Apps where you're trying a new stack or framework.

* Annotation apps for academia.

Basically, every single app written by people who aren't doing internet facing web dev as their core engineering function.

Also intranet apps as other commenter mentioned.

Re: SQLite the only database you will ever need in most cases (2021)

#319

Post by ilyt »

michaelteter wrote: post #309
ilyt wrote: post #305

Earlier quoted context omitted.

entirely depends what you're deploying. Go app which is one blob and some static file ? Sure. Java app that just needs JVM in system ? Go ahead, .service systemd file with some limits is all you need. Ruby on Rails ? Put that radioactive shit in container, else your server will need to have a bunch of -dev packages just to compile gems and make it running (or alternatively you'd have to compile the gems on same envir…

> Ruby on Rails I've been deploying Rails since v 2, and never with Docker. The only time I ever had trouble was setting up Rails dev on M1 Mac when M1 was new. In Linux production, it just worked; and pulling/building gems was less a drag than what people regularly experience in the NPM world. To be fair, I don't use a ton of gems because I generally don't like additional dependencies unless the value is really ther…

I mean, it isn't a big deal, just few packages need to be installed, just compared to "put a blob on server, run" it's more complex. It's all fine if you use any kind of configuration management as it is just write once in manifest and done.

But if you just install it directly via package manager, leave alone and need to say reinstall server or something, the knowledge is lost.

I remember having a lot of ping-pong between ruby devs where they, well, didn't note down what system libs and dev package they needed, just had them installed locally at some point and forgot about, then surprise when app isn't compiling on server.

Re: SQLite the only database you will ever need in most cases (2021)

#320

Post by bravura »

azurelake wrote: post #276
irskep wrote: post #63

Earlier quoted context omitted.

I think it's more that PaaS vendors aren't interested in first-class SQLite support when they can sell overpriced managed Postgres instead. Sure, it doesn't scale the same way, so it's hard to move upmarket and sell to Enterprise, but it's a shame that there's no one-click solution like there is for a managed database.

Let's say the following product existed: 1. You can only run one instance of your app. 2. There is a small window of downtime each deploy. 3. Your app has access to 10 GB of storage. The storage is persisted across deploys and can be used for sqlite. 4. Your sqlite data is automatically backed up and can be restored / downloaded as needed. How much per month would you pay for that product?

$20 per month assuming:

* Someone who knows Django but has zero devops skills can deploy my app with a few simple commands, add a fix, and demo it to me. Crucial: They must ask me zero questions.

* The backups happen to a non-you service, I one-click auth my google drive and/or dropbox.

* There are instructions on how to stand-up the web app on another service if you shut down. Those instructions might require two hours of my time, but should be complete.

Post reply on HN