Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite the only database you will ever need in most cases (2021)

unixsheikh.com

Post reply on HN
Print view View original

251–260 of 378 posts

Re: SQLite the only database you will ever need in most cases (2021)

#251

Post by canadiantim »

nikita wrote: post #232
hamandcheese wrote: post #191

Earlier quoted context omitted.

Neon only recently entered public preview and I am unable to find any pricing information.

Launching on mar 15th

Congrats on the upcoming launch! I'm definitely very strongly considering using Neon in my upcoming project. Mind if I ask a couple questions?

Would you be able to comment on if Neon is a good fit for having one postgres database per user and how well that would scale? E.g. what if millions of users?

Also with the managed service, is there help with applying migrations or helping manage migrations for such a multitude of postgres databases?

Lastly, is there any way of querying across neon databases, e.g. aggregating data from many user's databases? I saw you partnered with Hasura a bit. Hasura would be a great fit here for aggregating and "federating" each user's postgres database, but with their most recent price change I'm scared of integrating too much with them. Wondering how you think one can best query e.g. a million user's individual postgres databases?

Thank you and feel free to ignore I know it's kinda asking alot

Re: SQLite the only database you will ever need in most cases (2021)

#252

Post by fukawi2 »

adamkf wrote: post #10

>The only time you need to consider a client-server setup is: Where you have multiple physical machines accessing the same database server over a network. In this setup you have a shared database between multiple clients. This caveat covers "most cases". If there's only a single machine, then any data stored is not durable. Additionally, to my knowledge SQLite doesn't have a solution for durability other than asynchr…

Our primary product is backed by sqlite, using BedrockDB to make it client/server and multinode. 10mil users, 2+ TB sqlite database, 6 database nodes.

Re: SQLite the only database you will ever need in most cases (2021)

#253

Post by chillfox »

maxthegeek1 wrote: post #235
chillfox wrote: post #141

Earlier quoted context omitted.

Most things don't need zero downtime deployment. Now it's nice to have and might be an interesting technical challenge to solve, but it's usually not strictly needed. edit: when I worked as a sysadmin we would have to schedule outages for updating apps that were designed for zero downtime anyways because those were the processes of the organisation.

What? I've never worked for a company that would tolerate downtime during deployments. Downtime is ok for personal projects, but not for most business applications.

Most business have daily downtime where the entire business is closed as in not business hours. Being sensitive to downtime is more common for companies that has some kind of online service as their primary product, but most companies are not online companies, or even global companies with offices across all the time zones.

Even for online business some downtime might be acceptable or even preferred. I used to work for a company that made most of it's money through a website and it would take the site offline for 8 hours a few weeks before important events to run a full scale load test on the production infrastructure.

Another place I worked we had to schedule downtime even when updating applications that were designed for zero downtime.

I have never worked a place that didn't allow for downtime or even heard of such a place other than the big tech companies.

Re: SQLite the only database you will ever need in most cases (2021)

#254

Post by srcreigh »

benhoyt wrote: post #250
srcreigh wrote: post #244

Earlier quoted context omitted.

I don’t get this. Go can be compiled to a single binary containing SQLite. Scp the executable and run it. This will work on any Linux vm without any setup…

That's what I do now. But there's a bunch more setup: - Install and configure Caddy to terminate the SSL. Caddy is great, but still stuff to think about and 20 lines of config to figure out. - Configure systemd to run Caddy and my Go server. Not rocket science, but required me figuring out systemd for the first time and the appropriate 25-line config file for each server. - Scripts to upgrade Caddy when a new version…

Fair enough, but those things are unrelated to SQLite. Any app which uses a Postgres or MySQL PaaS would need to overcome the same hurdles.

It'd be nice if those things were easier, but the SQLite part of it cannot be any simpler or easier than it already is.

Backing up SQLite can be done with a 2 line shell script

    ssh vm 'sqlite3 my_database.db ".backup my_database.db.bak"'
    scp vm:my_database.db.bak .

Re: SQLite the only database you will ever need in most cases (2021)

#255

Post by kragen »

jpgvm wrote: post #190
kragen wrote: post #165

Earlier quoted context omitted.

in 01973 the s/360 did not have hot-swappable cpus or memory even in the 01990s i don't think ibm had such an offering, though tandem did (but it couldn't run ims)

it didn't run ims but it ran nonstop SQL instead which was a rdbms designed for their redundant hw architecture

right, i didn't mean to imply it didn't support acid

Re: SQLite the only database you will ever need in most cases (2021)

#256

Post by hajile »

SnowHill9902 wrote: post #188

The only database you’ll need 60% of the time every time.

Everyone thinks they have big data and 99.99% of the time, their entire DB could be served from a single 100GB SQLite file on a single SSD on a random dev's laptop with better performance than the expensive AWS deployment they've rented. They think they have super-concurrent stuff and locks are important, but modern machines are so fast that data can be added faster than their users can input data.

People don't realize just how big 1MB is if it's not multimedia. If you had 100,000 users consistently writing 1kb to your DB every single day, it would still take a 10,000 days or over 27 YEARS to fill a 1TB harddrive.

Most companies have 1,000 users per day updating mostly existing data or adding small snippets here and there. That 1TB drive would die of old age long before your average company could come close to filling even 10% of the capacity.

Re: SQLite the only database you will ever need in most cases (2021)

#257

Post by chillfox »

sanderjd wrote: post #163
chillfox wrote: post #141

Earlier quoted context omitted.

Most things don't need zero downtime deployment. Now it's nice to have and might be an interesting technical challenge to solve, but it's usually not strictly needed. edit: when I worked as a sysadmin we would have to schedule outages for updating apps that were designed for zero downtime anyways because those were the processes of the organisation.

Yeah I've realized this is just a disagreement over the word "most" which won't really be resolvable empirically. It doesn't fit my experience that most applications are single node with lax uptime and failover requirements.

That's fair.

Re: SQLite the only database you will ever need in most cases (2021)

#258

Post by sph »

JohnBooty wrote: post #225
berdon wrote: post #177

Earlier quoted context omitted.

No whiteboard, music video, or a published novel yet. But, come now, his point was he’d never worked with a full stack developer and that they don’t exist.

As the sibling comment notes, yes -- my point was that the 2023 version of a "full stack developer" is distressingly shallow in most of the individual skills. I'll use myself as an example. I am a "full stack developer." I have fairly deep backend knowledge. But my React skills basically amount to the ability to make small JSX tweaks, and my sysadmin-y skills are also at roughly the same level. Conversely, the folks…

No offense and not here to toot my own horn, but just because you are better on one side of the stack doesn't mean that true full stack developers are unicorns.

Like others upthread, I am one of them, and probably because I've been paid to do this job for 16+ years.

Honestly my challenge is making potential employers understand that yes, I know my way around Elixir or React, as I do around C or Rust, as I do around sysadmin (not only DevOps) or DBA work, as I do around low-level system code. I'm no world expert at any of them, but most companies need a person that can wear many hats when things go crazy and specialisation is for insects anyway.

Sadly, generalist engineers like me have lost against the trend of "full stack developer" to have been stolen from us by recruiters and less skilled devs to now mean "can use Express and React and maybe deploy on Heroku".

Re: SQLite the only database you will ever need in most cases (2021)

#260

Post by srcreigh »

bagels wrote: post #81
adamkf wrote: post #10

>The only time you need to consider a client-server setup is: Where you have multiple physical machines accessing the same database server over a network. In this setup you have a shared database between multiple clients. This caveat covers "most cases". If there's only a single machine, then any data stored is not durable. Additionally, to my knowledge SQLite doesn't have a solution for durability other than asynchr…

My standard for any serious service is at least minimal redundancy for improved availability during failures. At least two webservers.

Does this practically improve the situation? The odds of two servers breaking at the same time for the same reasons seems very high. I actually can't think of a single example where the secondary sever would keep running.

Regression via a code or dependency update? Full disk? DNS is down? Too much load? All of these would bring down both servers in quick succession.

I guess something like a "once every 2 days" race condition could buy you some time if you had a 2nd server. But that's not a common error

Post reply on HN