SQLite the only database you will ever need in most cases (2021) • Hacker News Board

Re: SQLite the only database you will ever need in most cases (2021)

#331

Post by JohnBooty » Thu, Feb 16, 2023, 2:15 PM UTC

sph wrote: post #258

JohnBooty wrote: post #225

Earlier quoted context omitted.

As the sibling comment notes, yes -- my point was that the 2023 version of a "full stack developer" is distressingly shallow in most of the individual skills. I'll use myself as an example. I am a "full stack developer." I have fairly deep backend knowledge. But my React skills basically amount to the ability to make small JSX tweaks, and my sysadmin-y skills are also at roughly the same level. Conversely, the folks…

No offense and not here to toot my own horn, but just because you are better on one side of the stack doesn't mean that true full stack developers are unicorns. Like others upthread, I am one of them, and probably because I've been paid to do this job for 16+ years. Honestly my challenge is making potential employers understand that yes, I know my way around Elixir or React, as I do around C or Rust, as I do around s…

    I know my way around Elixir or React, as I 
    do around C or Rust, as I do around sysadmin 
    (not only DevOps) or DBA work, as I do around 
    low-level system code

To what level do you know these tools?

This is a challenging discussion because the idea of "knowing" or "being good at" a tool is so nebulous.

As I mentioned elsewhere I'm using a definition that is essentially, "the level of skill a solid engineer would acquire after working full-time with a technology for 1-3 years." Not world-class expertise necessarily, but enough time to surpass basic literacy and achieve a real fluency. To encounter edge cases and pitfalls and develop well-supported opinions about best practices.

If you have that level of fluency in all those tools, great! Sounds like you're pretty awesome. In my experience that level of mastery of the entire stack at once is exceedingly rare.

Re: SQLite the only database you will ever need in most cases (2021)

#332

Post by canadiantim » Thu, Feb 16, 2023, 2:38 PM UTC

azurelake wrote: post #290

irskep wrote: post #283

Earlier quoted context omitted.

Interesting question. I think you could probably get those features by deploying Piku or Dokku for like $4/mo on a VPS, and the equivalent managed Postgres on Render would be like $20/mo (16GB SSD), so something like $6-10 seems like the right range to me. Maybe not a booming business, but a nice margin percentage and worth paying for good UX. And the cheap version doesn't need to be the only version if people want m…

Let's say the price was 12. If this idea is compelling enough for you at that price point to venmo me an advance for your first month, I have some ideas to explore that could actually make this viable. FWIW, I've worked professionally on a PaaS and a managed database offering.

If you make it so I can have one SQLite database per user, I’d be a customer too

Re: SQLite the only database you will ever need in most cases (2021)

#333

Post by bogwog » Thu, Feb 16, 2023, 2:53 PM UTC

irskep wrote: post #148

JohnBooty wrote: post #134

Earlier quoted context omitted.

You don't need to be a full-blown sysadmin to know how to do basic deployments, and learning these things will make you a better developer. Yes. True. However, it is also true that we have a limited amount of time per week and a limited number of weeks on Earth. Time spent learning sysadmin-y stuff is less time spent mastering developer-y stuff. Think about what it means to be a "full stack" engineer in 2023: - Unix…

I really appreciate you taking the time to respond with these details. I do go deep on application-level tech (CSS, HTML, JS with and without frameworks, iOS dev, etc) and it's always disheartening for people to claim that choosing not to focus on specific technologies makes you a worse human being. The other thing that's going completely over people's heads here is that I made this repo with the intent of helping ot…

> and it's always disheartening for people to claim that choosing not to focus on specific technologies makes you a worse human being.

Hey, I'm the guy who wrote that original response to you. I apologize if that's how you interpreted my comment, but that's not what I meant by it. I wasn't even trying to say that you're a bad engineer (and certainly not a bad human being), I just wanted to say that I disagree with the attitude of that comment. I'm sure if I said it to you in person, you would have picked up on the non-hostile intent and narrow scope of my criticism, but the internet tends to twist things in very negative ways.

Nobody can know everything, and not knowing something doesn't make someone a bad engineer. Software development in particular is more about learning than about knowing, which is why when I see someone saying that they don't need to learn something because , it bothers me. In your comment, you said that you don't want to be a release engineer, but IMO that's a poor excuse since "release engineer" isn't even a role that exists at many places.

And for the record, I didn't even look at the project you linked to. So if you also thought I was criticizing it, then rest assured that I'm much too lazy for that. When I was still learning, online tutorials and resources like yours were extremely valuable to me, so I know how helpful they can be to people even if they're not "perfect" by some snobby asshole's definition...so keep at it!

Re: SQLite the only database you will ever need in most cases (2021)

#334

Post by azurelake » Thu, Feb 16, 2023, 3:08 PM UTC

canadiantim wrote: post #332

azurelake wrote: post #290

Earlier quoted context omitted.

Let's say the price was 12. If this idea is compelling enough for you at that price point to venmo me an advance for your first month, I have some ideas to explore that could actually make this viable. FWIW, I've worked professionally on a PaaS and a managed database offering.

If you make it so I can have one SQLite database per user, I’d be a customer too

When you say per user, you mean per user of your app?

Re: SQLite the only database you will ever need in most cases (2021)

#335

Post by 0xbadcafebee » Thu, Feb 16, 2023, 3:29 PM UTC

srcreigh wrote: post #269

0xbadcafebee wrote: post #168

I agree. Most uses of databases definitely don't need to grow larger than, say, a single filesystem, or a single application, or a single host, or a single network, or a single geographical region, or a single customer, or a single organization, or a single global network of customers in organizations in regions on networks on hosts on applications on filesystems. There could not be any features of any other database…

I bet most non-FAANG programmers have indeed never worked on an application which could not be built on a single host with SQLite. And SQLite is in fact more fully featured in some ways than some client-server DBs (when will Postgres add support, even via a plugin, for primary indexes?) I agree with you that it is important to realize where a client-server DB may be needed, but the it really almost never is.

For perspective, same conversation, but different subject:

"A bicycle is all the transportation you will ever need in most cases.

I bet most non-bicycle drivers have indeed never been anywhere which couldn't have been gotten to by bike. And the bike is in fact more fully featured in some ways than some cars (when will cars shrink down? not require fuel? not pollute? cause less harm?)

I agree it's important to realize when a car may be needed, but in reality it almost never is."

Even if all of that is true... I still want to use a car, for good reasons. I may not need it, and it's more complex to use and maintain. But I get enough value from it that it's worth it.

Re: SQLite the only database you will ever need in most cases (2021)

#336

Post by acuozzo » Thu, Feb 16, 2023, 3:32 PM UTC

di456 wrote: post #239

acuozzo wrote: post #68

Earlier quoted context omitted.

FWIW, storage is cheap, but caches are not.

It's cheap if the cache is also on SQLite. Might not even need a separate db for some use cases. This runs on python and works with several backends including SQLite. https://requests-cache.readthedocs.io/en/stable/

I meant hardware caches like L1, L2, and L3 on the CPU.

SQLite is used in some HPC work.

ISO-8601 datetime strings can easily wreck your L1 cache. Instead of filling an eighth of the cache with a 64-bit value, you wind up filling almost half with a 27-character-long string.

Re: SQLite the only database you will ever need in most cases (2021)

#337

Post by erulabs » Thu, Feb 16, 2023, 3:38 PM UTC

srcreigh wrote: post #275

erulabs wrote: post #45

See I’ve been using Vitess on Kubernetes for even personal projects and I gotta say I love that I can run, for 10 bucks a month on Linode, the same tools that I know by experience I can scale to a multi-billion dollar valuation worth of customers. Heck I even run it in development on my laptop thanks to Skaffold. Sure it’s all insane overkill - but I use Linux for the same reasons - I want one API that I can use ever…

You also run a company which offers kubernetes self hosting as a business, which is quite the bias. Anyways, the question isn't whether client-server databases scale, it's whether SQLite doesn't scale. What were the database size, resource constraints, architecture for the multi-billion $ company? How did this scale over tiem? Do you have any reason to believe that a SQLite architecture couldn't scale to support the…

If you’re exceptionally careful and skilled about writing smart queries, SQLite can for sure scale to the point you can afford to resign anything you want. The idea behind vitess and sharded MySQL (and for that matter kubernetes too) is that you can move fast and not make a terrible mess of things. I can split off one poorly designed table with Vitess - with SQLite that would be an application redesign.

But in general I compare the intended scale to successful startups I’ve been at - where several TB of data are being read by hundreds of thousands of users per second. Typically this requires a fleet of the largest instance types most cloud providers offer.

Anyways - you’re not wrong and neither is the author - but if I’m going to choose one tool - I’d rather it work for all intentions and be a bit more complex than the other way around.

Re: SQLite the only database you will ever need in most cases (2021)

#338

Post by cldellow » Thu, Feb 16, 2023, 4:04 PM UTC

azurelake wrote: post #276

irskep wrote: post #63

Earlier quoted context omitted.

I think it's more that PaaS vendors aren't interested in first-class SQLite support when they can sell overpriced managed Postgres instead. Sure, it doesn't scale the same way, so it's hard to move upmarket and sell to Enterprise, but it's a shame that there's no one-click solution like there is for a managed database.

Let's say the following product existed: 1. You can only run one instance of your app. 2. There is a small window of downtime each deploy. 3. Your app has access to 10 GB of storage. The storage is persisted across deploys and can be used for sqlite. 4. Your sqlite data is automatically backed up and can be restored / downloaded as needed. How much per month would you pay for that product?

I'd probably be willing to pay ~$10/mo. i.e. a 100% markup on a low-end $5 VPS box.

...but in addition to bravura's requirements, I'd add: I'd want SSL termination, ideally with the option of me bringing a custom domain.

Re: SQLite the only database you will ever need in most cases (2021)

#339

Post by srcreigh » Thu, Feb 16, 2023, 4:19 PM UTC

erulabs wrote: post #337

srcreigh wrote: post #275

Earlier quoted context omitted.

You also run a company which offers kubernetes self hosting as a business, which is quite the bias. Anyways, the question isn't whether client-server databases scale, it's whether SQLite doesn't scale. What were the database size, resource constraints, architecture for the multi-billion $ company? How did this scale over tiem? Do you have any reason to believe that a SQLite architecture couldn't scale to support the…

If you’re exceptionally careful and skilled about writing smart queries, SQLite can for sure scale to the point you can afford to resign anything you want. The idea behind vitess and sharded MySQL (and for that matter kubernetes too) is that you can move fast and not make a terrible mess of things. I can split off one poorly designed table with Vitess - with SQLite that would be an application redesign. But in genera…

Google reportedly receives 99k search queries per second - which company actually handles 6 digit QPS reads?

Re: SQLite the only database you will ever need in most cases (2021)

#340

Post by napsterbr » Thu, Feb 16, 2023, 4:20 PM UTC

RedShift1 wrote: post #302

napsterbr wrote: post #170

Earlier quoted context omitted.

SQLite is a great but complex software. In order to properly use it, you do need to read the documentations, guides and resources available at the official website. If you don't, you will shoot yourself in the foot. In that sense, I have the impression SQLite is different from other database software. You can usually get by with Postgres or MySQL (after they are set up) without looking at their docs. I spent several…

If you are blindly doing multi-threaded connections without understanding the implications, you also risk corrupting the database entirely. This is false? SQLite has always been able to handle multiple processes and threads, reading and writing to the same database?

> SQLite has always been able to handle multiple processes and threads

This is true for the vast majority of cases, although there's at least one documented scenario where using modern SQLite coupled with an old threading implementation (the one that predates NPTL in Linux) may lead to database corruption.

I guess that's why I said "blindly": as a way to incentivize OP to look this up. It's not that the SQLite database is fragile, but rather that it expects to be used in a certain way, and if you don't, you risk corrupting it.