Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Is Serverless

sqlite.org

Post reply on HN
Print view View original

341–350 of 453 posts

Re: SQLite Is Serverless

#341

Post by hinkley »

at_a_remove wrote: post #287
hinkley wrote: post #248

Earlier quoted context omitted.

Do you generate the file from scratch every time or do you modify the previous one as new data arrives?

It is a bit of a hybrid. In the early stages of development of whatever the ETL process is, I keep the database and just empty it out each time. As I got more of a sense of what I needed, I started DROPing my TABLEs more often and remaking them. Eventually I would make the whole database from scratch once I was along the way and had most everything fleshed out.

Ok. So each export is a full dump, not a delta on a previous one.

Do you anticipate hitting a wall at some point where the total time becomes a problem?

Re: SQLite Is Serverless

#342

Post by siscia »

erezsh wrote: post #325
siscia wrote: post #309

Earlier quoted context omitted.

I was thinking something based on size or performance. 1 req for second, free. 50 req for second, 5€. Unmetered, 30€. Maybe not counting for seconds but for hour or day so to accommodate burst. Another option would be on features. But that will requires time.

If you have full support for SQL, it means some queries can run for a few seconds, possibly even minutes (even sqlite supports some form of bfs). So will you just time-out those requests? That would make a lot of possible uses suddenly impossible. I might consider the unmetered option, if the performance gain is worth the 6x cost of other providers. For web and games, maybe your existing model could work, but then al…

Hummm, what you mean by data science?

Like running analysis in a Jupiter notebook?

Indeed I am not sure this case is a good fit...

Re: SQLite Is Serverless

#343

Post by bane »

hinkley wrote: post #239
bane wrote: post #149

I love SQLite, but people approach it from a classic RDBMS angle which confuses them. Here's the deal: SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. That's it. Stop overthinking it. Can you write a microservice that stores its data in a big JSON file that you've built some code around to read/write to? Yes. It's just a file, but you have to build all the read/w…

I think you’re stuck in 90/10 rule territory here. But even so, SQLite was 220,000 lines the last time they measured, which was five years ago. You can do a lot of functionality in 22kloc, even ignoring the other 90%, which you shouldn’t Lodash, for instance, is much smaller than 22k lines, and it “just” manipulates objects and lists. If you downplay others like this, I wonder how you feel about your own work. Have y…

I'm not downplaying anybody's work. Many people approach SQLite as something in the RDBMS territory. It's not. Almost all of the confusion I've ever seen related to SQLite comes from starting from that basis. If one simply thinks of it as an alternative to fopen() then it makes very simple and intuitive sense.

The people in this thread seem to be very resistant to this simple clarity of thought, but whatever, they can stay confused and keep coming up with feature comparisons of SQLite vs Redshift vs Elasticsearch or some such.

If one were to draw a spectrum:

   file-format::DBMS
SQLite is the x on this line and .txt files are about the only thing that any further left on it.

Re: SQLite Is Serverless

#344

Post by [deleted] »

nicholaides wrote: post #185

I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…

[deleted]

Re: SQLite Is Serverless

#345

Post by at_a_remove »

hinkley wrote: post #341
at_a_remove wrote: post #287

Earlier quoted context omitted.

It is a bit of a hybrid. In the early stages of development of whatever the ETL process is, I keep the database and just empty it out each time. As I got more of a sense of what I needed, I started DROPing my TABLEs more often and remaking them. Eventually I would make the whole database from scratch once I was along the way and had most everything fleshed out.

Ok. So each export is a full dump, not a delta on a previous one. Do you anticipate hitting a wall at some point where the total time becomes a problem?

Well, it depends on the process. Some were full dumps, some were deltas pushed up to the final database, sometimes both (this product in particular had a load from file capability that you were supposed to use but some edge cases that were not well-addressed).

No, the time never grew significantly.

For one of the analysis projects, just one step of the analysis was quite time consuming but it would have been that way no matter what. SQLite allowed me to let it grind away overnight (or even over a weekend) on a workstation without tormenting production servers.

Re: SQLite Is Serverless

#346

Post by aidenn0 »

cle wrote: post #329
jiofih wrote: post #322

Earlier quoted context omitted.

This such an air-headed comment to make. You must realize that page describing how SQLite is “serverless” has been up probably longer than your entire adult life. It is important in its context, they are not trying to “redefine” (lol) the term.

It doesn’t matter. It is not the generally accepted definition of serverless. The meaning of a word is based on how it is generally used, and almost nobody means this when they use the word “serverless”. At this point trying to use the word in this way just creates a bunch of unnecessary confusion. Call it something else so we can move on to more important (and clear) discussions. (Also there are a lot of assumptions…

The page was written before that usage of the term serverless existed, and anyone reading it would have known exactly how it was meant. The "neo- vs classical-" note was added in 2018 because the distinction was being made.

Is your argument that they should replace "serverless" with "runs without a server?" That seems like a strange position to me.

Re: SQLite Is Serverless

#347

Post by forgotmypw »

F_J_H wrote: post #288
nicholaides wrote: post #185

I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…

I've wondered about this too, but have not gotten around to trying it yet. We get a gnarly csv log file back from our sensors in the field, which is really a "flattened" relational data model. What I mean by that is a file with "sets" of records of various lengths, all stacked on top of each other. So, if you open it in Excel, (which many users do), the first set of 50 rows may be 10 columns wide, the next 100 rows w…

Yes, it is great for that.

You only have to read the CSV file once, and after that you have a nice set of tables you can query any which way you want.

I use SQLite as an intermediate step between text files and static HTML, for example.

Re: SQLite Is Serverless

#348

Post by ForHackernews »

cle wrote: post #329
jiofih wrote: post #322

Earlier quoted context omitted.

This such an air-headed comment to make. You must realize that page describing how SQLite is “serverless” has been up probably longer than your entire adult life. It is important in its context, they are not trying to “redefine” (lol) the term.

It doesn’t matter. It is not the generally accepted definition of serverless. The meaning of a word is based on how it is generally used, and almost nobody means this when they use the word “serverless”. At this point trying to use the word in this way just creates a bunch of unnecessary confusion. Call it something else so we can move on to more important (and clear) discussions. (Also there are a lot of assumptions…

I think hosting providers marketing their managed servers as "serverless" is what has created confusion.

Imagine if Uber called itself a "carless" taxi.

Re: SQLite Is Serverless

#349

Post by ForHackernews »

cle wrote: post #336
coldtea wrote: post #328

Earlier quoted context omitted.

> Everyone knows what "serverless" means at this point, and it's not an embedded DB. Serverless is a marketing term at this point (as it was when it started). This post brings some welcome definitions and expands it to something that has many of the same attributes but wasn't appreciated as such.

Serverless refers to platforms with specific and well-understood properties, such as managed servers, autoscaling, usage-based pricing, etc. What parts of SQLite have these properties?

SQLite has specific, well-understood pricing and scaling properties.

Re: SQLite Is Serverless

#350

Post by zzo38computer »

nicholaides wrote: post #185

I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…

I don't know what is "ETL" meaning here, although SQLite does include a JSON extension to read/write JSON data too, so you can use SQL and JSON together if necessary.
Post reply on HN