Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Is Serverless

sqlite.org

Post reply on HN
Print view View original

371–380 of 453 posts

Re: SQLite Is Serverless

#371

Post by catalogia »

manigandham wrote: post #366
catalogia wrote: post #364

Earlier quoted context omitted.

> Nobody used it before. The archive.org links already provided to you prove that you're flat wrong about that. SQLite used the term many years before it was a webshit buzzword. They used it in the intuitive straight forward english sense; somebody who has no hat is hat less , somebody who has no home is home less , and a database system that has no server process is server less . In 2007 when it was written, nobody…

Just because a single page on the internet used it does not mean the term was used in the industry. It also doesn't matter if it technically makes sense, although it's a stretch (because there is a server process, you just share it). It's a marketing term, and a poor one at that. That's why SQLite even describes itself as a embedded, in-process database without client/server architecture. Because that's the common ja…

Nobody was saying it was a buzzword in the industry back then (actually...[1]). It was being used as a normal English word, not some misleading marketing drivel like the new use of the term. Somebody who has no shirt is shirtless. An MRE heater that doesn't catch fire is flameless. Somebody who isn't witty is witless. Somebody who doesn't have a clue is clueless.

Are you starting to notice the pattern here? "less" is a suffix that can be applied to nearly any noun to describe the trait of lacking that noun. When this is done, the meaning is clear to native English speakers who've never heard that combination before. When this article was written in 2007, the meaning was clear. SQLite didn't muddy anything, didn't redefine anything.

[1] The term 'serverless' was in fact in use in the decade and a half prior to 2007: https://books.google.com/ngrams/graph?content=serverless&yea... I don't know if the term was being used primarily in a technical context, but I have little doubt the term was being used to describe something that did not have a server. In fact I'm just about certain that spike was the term being used in a tech context. I've found you an example of the term being used in 1995 (DOI 10.1145/224057.224066):

> A serverless network file system distributes storage, cache, and control over cooperating workstations. This approach contrasts with traditional file systems such as Netware, NFS, [...] where a central server machine provides all file system services.

This is not precisely the same way that SQLite has used the term. Rather, they're using the term in an intuitive natural way that fits their context, just as SQLite used it in 2007.

Here is an example from 2007 (DOI: 10.1109/TNET.2006.886289):

> Abstract—We explore exploits possible for cheating in real-time, multiplayer games for both client-server and serverless architectures.

Here we have the term "serverless" clearly being contrasted with "client-server", which seems very similar to the way in which that SQLite document used it.

Re: SQLite Is Serverless

#372

Post by wpietri »

adamcharnock wrote: post #365
wpietri wrote: post #148

Earlier quoted context omitted.

I'm sure people will give you the orthodox answer, so let me give you mine: when other things become more important than performance, ease of development, and code clarity. Many years ago I happened to meet the creator of Prevayler, an open-source persistence framework that provided ACID guarantees and was thousands of times faster than a database as long as your data fit in RAM. I tried it out for a project and we l…

Prevayler sounds a lot like Redis, unless I’m missing something. I’m a big fan of Redis, but I was bitten many years ago when I tried to use it as a replacement for an RDBMS. There were two reasons for this: 1) lack of development libraries and operational tools, and 2) lack of data integrity checks. 1 has changed these days, but 2 is still very much the case (and rightly so, IMHO). Perhaps Prevayler had this? But ye…

Not quite. Redis is still an external database. Prevayler was much simpler, just a library.

With Prevayler, all data is kept in RAM, reachable from one root object, which Prevayler holds. All changes to the data model must be expressed as command objects. Each object is handed to Prevayler, which serializes it to a log and then executes it. Once in a while, you can snapshot the data out and start a new log. If there's a crash, you just load the latest snapshot and replay the log.

You got exactly as much data integrity as you wrote into your objects and your commands. Which without much work could be quite a bit, because you get a lot of data integrity by not writing things. E.g., if a kind of object should never be deleted, you just don't write any deletion code. If, say, account balances should never be changed directly, but only as part of properly structured credits and debits, then you just write your code like that.

Most databases, Redis I think included, are made for arbitrary operations on data. Developers add integrity and security later, hopefully. And that is often duplicative of the code base, so that one ends up having integrity checks both in the code and in the database.

I should say that made a lot of sense for the era databases came out of. Databases were a huge step forward in the 1970s and 1980s. My dad was a developer in that era and it was a big relief not to have to get a bunch of programmers to all follow the same conventions for exactly which record was stored in exactly which spot on their precious and expensive disk drives. Not having to know the minutia of the hardware let a lot of people just get in there and build business reports. But if we were starting fresh today, I don't think we would do anything like a SQL database. Redis was definitely a step away from that era, and I look forward to many more.

Re: SQLite Is Serverless

#373

Post by chadcmulligan »

vdfs wrote: post #204
bob1029 wrote: post #119

Earlier quoted context omitted.

Multiple processes is an instant anti-pattern for a single SQLite database. I would stop and reconsider your approach before trying to build this solution using it. The way I see it there are 3 options: 1) Implement another process which will have exclusive ownership of the shared SQLite database, and then use some IPC scheme to delegate database operations from multiple processes. 2) Give each process its own copy o…

It's ironic how using a severless database require your app to be the server and do top-level access management to the database

Yes, I thought the same - so you have to write a server. What about locking? roll your own I suppose. Multi user - roll your own? What about hot backups? Rollbacks?

I always conclude these things are advocated by people who have no experience in large multi user systems. The same as the NoSQL movement. They'll eventually build a database server. They build a system using the cool thing which works fine when they test it on their single user system. Go live - aagghh what's happening, why are all these people trying to access my data simultaneously? and so on.

One I'll always remember was when XML was the next big thing - they decided to store the raw XML in a database. It was a commercial product, and we were interfacing to it from our system. Once we found out this we started asking questions - no no it works fine we were told, laughing at us old database guys. Went live couldn't handle 5 TPS - what a surprise, it never worked as far as I'm aware. There is this continuous circle of databases are bad, no no do this you don't need to do this, no things have changed - what do you database guys know. Its entertaining to watch if nothing else, my advice - learn SQL, and some database tuning, its not that hard, at least compared to writing your own database engine.

Re: SQLite Is Serverless

#374

Post by wpietri »

notduncansmith wrote: post #197
wpietri wrote: post #148

Earlier quoted context omitted.

I'm sure people will give you the orthodox answer, so let me give you mine: when other things become more important than performance, ease of development, and code clarity. Many years ago I happened to meet the creator of Prevayler, an open-source persistence framework that provided ACID guarantees and was thousands of times faster than a database as long as your data fit in RAM. I tried it out for a project and we l…

This is exactly the style in which I build applications these days. Most of the time I just describe it as “hella caching” but it is a different paradigm from treating the database as the primary state engine. The speed and simplicity of working on in-memory structures are great. When you outgrow a single server’s RAM capacity, you can use Kafka or another durable message queue as your application’s WAL and shard you…

Have you written up the systems you've built? I'd love to read more about the practical details. Feel free to email me or DM me on Twitter if that's better.

Re: SQLite Is Serverless

#375

Post by chadcmulligan »

bob1029 wrote: post #241
vdfs wrote: post #204

Earlier quoted context omitted.

It's ironic how using a severless database require your app to be the server and do top-level access management to the database

It may be ironic but there are benefits to rolling your own. Having your persistence layer talk in terms of your business models instead of raw SQL could be seen as a benefit in some contexts. The process which is the "server" can be written to allow for relaxed consistency based on specific business operation being performed (e.g. no need for transactions around log entries). This could be used to leverage substanti…

you can just turn off these things if you so desire - in reality the overhead is a lot less than you'd think. You'd be way better off spending the time you'd use to write half a database to optimising your application and just use a database. Or maybe you don't really need a database.

Re: SQLite Is Serverless

#376

Post by imtringued »

bane wrote: post #351
kgwgk wrote: post #263

Earlier quoted context omitted.

> SQLite is a file format with a nice API that uses SQL as the paradigm for reading/writing to the file. is a collection of bits with a nice interface that uses a query language as the paradigm for reading/writing data.

No, an RDBMS is a piece of software for managing databases in the relational model, access to the databases (such as users and permissions) and provides services such as a server, connection pooling, and so on. SQLite provides almost no RDBMS features. This isn't just semantics. A car is not an engine. A fork is not a kitchen. A SQLite file is not a DBMS.

https://en.wikipedia.org/wiki/Database#Database_management_s...

>Connolly and Begg define database management system (DBMS) as a "software system that enables users to define, create, maintain and control access to the database".[24]

>The functionality provided by a DBMS can vary enormously. The core functionality is the storage, retrieval and update of data. Codd proposed the following functions and services a fully-fledged general purpose DBMS should provide:[25]

[x] Data storage, retrieval and update

[x] User accessible catalog or data dictionary describing the metadata

[x] Support for transactions and concurrency

[x] Facilities for recovering the database should it become damaged

[ ] Support for authorization of access and update of data

[ ] Access support from remote locations

[x] Enforcing constraints to ensure data in the database abides by certain rules

Under this definition SQLite - the library - clearly is an RDBMS that leaves out some common features that do not make sense within its niche but is otherwise fully functional and under this definition the files that SQLite manages are the database, not just merely a file format.

Re: SQLite Is Serverless

#377

Post by tootie »

swiley wrote: post #292
buckminster wrote: post #99

Earlier quoted context omitted.

This was a standard feature of flat file databases in the early 90s. There were many products. They often had an ODBC driver, which provided a SQL front end. dBase .dbf files were often used for storage. The arcane file locking in Windows is intended for exactly this kind of application. Apart from quality (!), SQLite's main advantage over these products is broad platform support. And continued existence.

Unix like operating systems also allow locking files for on disk databases (like mbox files) it’s just not what every application does by default.

Used to love dbm files. 1980s serverless NoSQL. They're still totally usable although we have LevelDB nowadays too.

Re: SQLite Is Serverless

#378

Post by imtringued »

bane wrote: post #337
travbrack wrote: post #305

Earlier quoted context omitted.

But SQLite has rdbms features. I remember being able to show tables and do SQL queries in SQLite DBs.

Those are relational database features, not RDBMS features. The SQLite file format specifies a way to organize, store and retrieve fairly arbitrary data using a relational database model. The library knows how to handle SQL to describe the work being done. The SQL is optional, one can, with the specification, read/write SQLite files in many other ways. There are almost no RDBMS features in SQLite. There are many many…

You're making some basic assumptions that do not make sense.

The database can be physically stored in any arbitrary format. One can build an RDBMS that stores all its data in tar or JSON files. No matter how inefficient as long as software exists that manages the database.

>Are .tar files RDBMSs?

This question doesn't make sense because you are asking if databases can be management systems which is obviously false by definition.

If the 5 steps you have described are implemented in software then that software would be considered an RDBMS and the .tar file clearly would be a database. There is no confusion.

>However, you need to build a server, user access controls, connection pooling, import/export tools, partitioning, clustering, etc. before you start to arrive at an RDBMS that uses this engine.

Those features are not necessary for a piece of software to be called RDBMS but most industry standard RDBMS do indeed support these features and SQLite clearly is an RDBMS that pursues a certain niche that only makes sense in certain situations.

Re: SQLite Is Serverless

#379

Post by imtringued »

bane wrote: post #343
hinkley wrote: post #239

Earlier quoted context omitted.

I think you’re stuck in 90/10 rule territory here. But even so, SQLite was 220,000 lines the last time they measured, which was five years ago. You can do a lot of functionality in 22kloc, even ignoring the other 90%, which you shouldn’t Lodash, for instance, is much smaller than 22k lines, and it “just” manipulates objects and lists. If you downplay others like this, I wonder how you feel about your own work. Have y…

I'm not downplaying anybody's work. Many people approach SQLite as something in the RDBMS territory. It's not. Almost all of the confusion I've ever seen related to SQLite comes from starting from that basis. If one simply thinks of it as an alternative to fopen() then it makes very simple and intuitive sense. The people in this thread seem to be very resistant to this simple clarity of thought, but whatever, they ca…

There is no such spectrum. SQLite is a piece of software that implements some but not all commonly expected RDBMS features. Software is not a file format but software may be written with the expectation that a given file follows the requirements of a certain file format and the software may be written in a way that it produces files that follow the file format specification. Since SQLite - the software - is an RDBMS the files it produces can be considered to be databases.

Re: SQLite Is Serverless

#380

Post by elyobo »

elyobo wrote: post #363
nicholaides wrote: post #185

I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…

We do something like this; one of the outputs of the data pipeline is an sqlite file that's deployed nightly along with code to App Engine. The sqlite stuff is all read only, read/write data for the app is stored in firestore instead. We initially used json but ran in to memory issues; sqlite is more memory efficient and being able to use SQL instead of the wild SQL-esque is both faster and more reliable.

"wild SQL-esque" should have been "wild SQL-esque thing I wrote to query the JSON"
Post reply on HN