Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Is Serverless

sqlite.org

Post reply on HN
Print view View original

451–453 of 453 posts

Re: SQLite Is Serverless

#451

Post by tenken »

chrisweekly wrote: post #231
at_a_remove wrote: post #191

Earlier quoted context omitted.

Using SQLite in my ETL processes is something I have done for over a decade. It's just so convenient and, at the end, I have this file that can be examined and queried to see where something might have gone wrong. All of my "temporary" tables are right there for me to look at. It is wonderful!

Yes! Along these lines I heartily recommend `lnav` ^1, a fantastic, lightweight, scriptable CLI mini-ETL tool w embedded sqlite engine, ideally suited for working with moderately-sized data sets (ie, millions of rows not billions) ... so useful! 1. https://lnav.org

I have used it to inspect say the history of a users' requests on a load-balanced server. I like to permanently store the results of the logfile excerpt to a DB table for posterity and future reporting.

Figuring out how to enter "sql" mode in lnav, generate a logfile table, and then persist it from an in-memory sqlite db to a saved-to-disk sqlite db .... was frustratingly annoying.

It boils down to:

    :create-logline-table custom_log
    ;ATTACH DATABASE `test02.db` AS bkup;
    ;create table bkup.custom_log as select * from custom_log;
    ;detach database bkup;
if i recall you cannot call sqlite commands ".backup" or similar in lnavs sql mode. So lnavs interjection into the sqlite command processing is annoying (I'm actually very familiar with sqlite).

Re: SQLite Is Serverless

#452

Post by tracker1 »

PeterCorless wrote: post #434
tracker1 wrote: post #279

Earlier quoted context omitted.

Not the author, or knowing of all the technical details... simplistic replication structure and redundancy/failover without an expensive or more complex RDBMS solution while still self-hosting the service. There are still a lot of instances where you cannot use a cloud provider for your app or database. To be honest, I'd probably lean more towards a nosql database that has in the box, relatively easy replication stra…

Well, ScyllaDB is free and open source, so that should help the budget. (Though we do have an enterprise version, base price is FREE!)

I'd probably reach for Scylla wherever I would consider Cassandra (or similar), though the similar bigtable/columnstore-like services in most cloud hosts is generally still going to be easier.

The performance over C* is surprisingly good.

Re: SQLite Is Serverless

#453

Post by yencabulator »

nicholaides wrote: post #185

I think a good under-appreciated use case for SQLite is as a build artifact of ETL processes/build processes/data pipelines. Seems like lot of people's default, understandably, is to use JSON as the output and intermediate results, but if you use SQLite, you'd have all the benefits of SQL (indexes, joins, grouping, ordering, querying logic, and random access) and many of the benefits of JSON files (SQLite DBs are jus…

Beware opening SQLite files you didn't create: https://research.checkpoint.com/select-code_execution-from-u...
Post reply on HN