Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serverless SQLite

sql.lspgn.workers.dev

Post reply on HN
Print view View original

51–60 of 167 posts

Re: Serverless SQLite

#51

Post by mozey »

ramraj07 wrote: post #17
scary-size wrote: post #12

Nice, you can actually do something a bit more flexible using SQLite on AWS Lambda. "Just" mount an elastic file system (EFS) onto your functions. I wrote about it here: https://franz.hamburg/writing/shared-storage-for-lambda-func...

Did I read it right you get 1 second responses? Why even bother with lambdas? A t3 nano reserved instance costs 2 bucks a month or something! Surely most folks don't anticipate 1000x scaling in minutes? Coupled to elastic beanstalk you can get reaonable scaling as well!

Might be useful for services that aren't continually used? For example, month or year end processes. Not convinced I would use SQLite for a service like that though, seeing as AWS has serverless RDS

Re: Serverless SQLite

#52

Post by flower-giraffe »

jsf01 wrote: post #4

What would this be best used for?

Cloudflare workers run at the edge along with a simple but fast KV store.

We use KV and workers to serve static data (json) to browser based code.

This could be used to filter, sort or aggregate that data before sending the client.

Perhaps for inventory / product lookup on a e-commerce site.

Re: Serverless SQLite

#53

Post by sradman »

edge-sql [1] allows arbitrary SQLite queries to be executed over an immutable external data set. The demo uses a Forex data set stored in Workers KV.

Client issued arbitrary queries is one of the use cases for GraphQL and publishing immutable data sets on the web is the main use case for Simon Wilson’s Datasette [2].

In-memory SQLite compiled to WASM works in the browser and Node.js too. In the future, we can expect proper ACID operations on any WASM runtime that supports fsync in WASI [3], a POSIX-like API.

[1] https://github.com/lspgn/edge-sql

[2] https://datasette.io/

[3] https://wasi.dev/

Re: Serverless SQLite

#54

Post by senko »

Animats wrote: post #23
ithkuil wrote: post #16

Earlier quoted context omitted.

Indeed it's quite stupid. Any alternative catchy name for the concept?

"Shared hosting".

I would agree and here's why: when compared to PHP scripts of old, the concept is the same. You have some bit of executable code that gets called when a specific URL is hit.

Re: Serverless SQLite

#55

Post by tomashubelbauer »

kuter wrote: post #35

I was thinking of making it possible for SQLite to be used with static pages. My idea is to modify SQLite to use ajax with the HTTP Range header to fetch B+ pages from the server as they are needed. SQLite already has a VFS (virtual file system) so this shouldn't be too hard. I am not sure how fast it would be and it would waste a lot of bandwidth. That's why I haven't made it yet. This would only be useful for using…

Hm, interesting. I made this, if you want to take a look: https://github.com/TomasHubelbauer/sqlite-javascript. You can test it using the Demo link in the readme and then use the prefilled value in the Load from URL prompt. Based on this I find your idea to be very doable. I didn't do it yet, but while working on this I had async fetching of pages using Range requests on my mind. Shame I didn't get to it when I was working on this project, could be easy to put together a demo. Similar to this I made https://github.com/TomasHubelbauer/fatcow-icons which parses a big ZIP file piece-wise on demand. As you scroll, new parts of the archive are fetched and icons extracted so you don't need to download the whole thing.

Re: Serverless SQLite

#57

Post by Mutjake »

ithkuil wrote: post #16
throw_m239339 wrote: post #7

Earlier quoted context omitted.

> SQLite was already serverless. Yes but now it runs on someone else server! Wait a minute, this makes no sense... 'serverless' is perhaps the stupidest marketing buzzword developers have come up with.

Indeed it's quite stupid. Any alternative catchy name for the concept?

Dunno about the catchy, but Runtime as a Service could fit the *aaS pattern.

Re: Serverless SQLite

#58

Post by scary-size »

ramraj07 wrote: post #17
scary-size wrote: post #12

Nice, you can actually do something a bit more flexible using SQLite on AWS Lambda. "Just" mount an elastic file system (EFS) onto your functions. I wrote about it here: https://franz.hamburg/writing/shared-storage-for-lambda-func...

Did I read it right you get 1 second responses? Why even bother with lambdas? A t3 nano reserved instance costs 2 bucks a month or something! Surely most folks don't anticipate 1000x scaling in minutes? Coupled to elastic beanstalk you can get reaonable scaling as well!

This was just a proof-of-concept using the smallest lambdas available. I definitely would opt for a EC2 instance. Though lambda would be enough for something like a configuration service.

Re: Serverless SQLite

#59

Post by quietbritishjim »

kuter wrote: post #50
ec109685 wrote: post #45

Earlier quoted context omitted.

That could be a lot of round trips.

Yes. Maybe you could increase the sizes of B+ pages ? The only reason that databases use B+ trees rather than a red-black tree or avl trees is because of the overhead of reading data from the hard drive. This would be a interesting hack.

You can increase the page size [1]. That will increase the size of the B tree nodes [2] (and other pages too but that's probably what you want):

> The upper bound on [the number of keys on an interior b-tree page] is as many keys as will fit on the page.

I think a tricky part of this idea would be the locking. Usually SQLite relies on the locking of the underlying filesystem. You could add your own mechanism that causes a lock to be assigned to a single client connection, but what if it never unlocks? (On a single machine you can tell if the client process has crashed.) You could add a timeout but what if the client process then does respond?

[1] https://www.sqlite.org/pragma.html#pragma_page_size

[2] https://www.sqlite.org/fileformat2.html#b_tree_pages

Re: Serverless SQLite

#60

Post by emmanueloga_ »

I was thinking a bit about an "on-edge DB" recently, perhaps something as simple as some way to persist and sync IndexedDB data between user sessions. Not usable for everything but probably great for a small CMS or a blog.

For running a write-enabled DB on a CF worker a major problem is that the only storage option that I could find has really low write limits [0], 1000 writes a day for the free option. "Durable Objects" is a beta API, perhaps its transactional-storage-api [1] has better limits?

--

0: https://developers.cloudflare.com/workers/platform/limits#kv...

1: https://developers.cloudflare.com/workers/runtime-apis/durab...

Post reply on HN