Live data from Hacker News

Hosting SQLite Databases on GitHub Pages

phiresky.netlify.app

51–60 of 82 posts

Re: Hosting SQLite Databases on GitHub Pages

#51
This is a great example of how as technology changes, it changes use cases, which can prompt a revisiting of what was once considered a good idea. You'll often see the pendulum of consensus swing in one direction, and then swing back to the exact opposite direction less than a decade later.

2010s saw REST-conforming APIs with json in the body largely as an (appropriate) reaction to what came before, and also in accordance with changes around what browsers were able to do, and thus how much of web apps moved from the backend to the front.

But then, that brought even more momentum where web apps started doing /even more/. There was a time when downloading a few megabytes per page, generating an SVG chart or drawing an image, interacting to live user interaction was all unthinkable. But interactive charting is now de facto. So now we need ways to access ranges and pieces of bulk data. And it looks a lot more like block storage access than REST.

---

These are core database ideas: you maintain a fast and easy to access local cache of key bits of data (called a bufferpool, stored in memory, in e.g. mysql). In this local cache you keep information on how to access the remaining bulk of the data (called an index). You minimize dipping into "remote" storage that takes 10-100x time to access.

Database people refer to the "memory wall" as a big gap in the cache hierarchy(CPU registers, L1-L3, main memory, disk / network) where the second you dip beyond it, latency tanks (Cue the "latency numbers every programmer should know" chart). And so you have to treat this specially and build your query plan to work around it. As storage techniques changed (e.g. SSDs, then NVMEs and 3d x-point etc), databases research shifted to adapt techniques to leverage new tools.

In this new case, the "wall" is just before the WAN internet, instead of being before the disk subsystem.

---

This new environment might call for a new database (and application) architectural style where executing large and complex code quickly at the client side is no problem at all in an era of 8 core CPUs, emscripten, and javascript JITs. So the query engine can move to the client, the main indexes can be loaded and cached within the app, and the function of the backend is suddenly reduced to simply storing and fetching blocks of data, something "static" file hosts can do no problem.

The fundamental idea is: where do I keep my data stored, where do I keep my business logic, and where do I handle presentation. The answer is what varies. Variations on this thought:

We've already had products that completely remove the "query engine" from the "storage" and provides it as a separate service, e.g. Presto / Athena where you set it up to use anything from flat files to RDBMSs as "data stores" across which it can do fairly complicated query plans, joins, predicate pushdown, etc. Slightly differently, Snowflake is an example of a database that's architected around storing main data in large, cheap cloud storage like s3, no need to copy and keep entire files to the ec2 node, only the block ranges you know you need. Yet another example of leveraging the boundary between the execution and the data.

People have already questioned the wisdom of having a mostly dumb CRUD backend layer with minimal business logic between the web client and the database. The answer is because databases just suck at catering to this niche, but nothing vastly more complicated than that. They certainly could do granular auth, serde, validation, vastly better performance isolation, HTTP instead of a special protocol, javascript client, etc etc. Some tried.

Stored procedures are also generally considered bad (bad tooling, bad performance characteristics and isolation, large element of surprise), but they needn't be. They're vastly better in some products that are generally inaccessible to or unpopular with large chunks of the public. But they're a half baked attempt to keep business logic and data close together. And some companies had decided at a certain time that their faults were not greater than their benefits, and had large portions of critical applications written in this way not too long ago.

---

Part of advancing as an engineer is to be able to weigh the cost of when it's appropriate to sometimes free yourself from the yoke of "best practices" and "how it's done". You might recognize that something about what you're trying to do is different, or times and conditions have changed since a thing was decided.

And also to know when it's not appropriate: existing, excellent tooling probably works okay for many use cases, and the cost of invention is unnecessary.

We see this often when companies and products that are pushing boundaries or up against certain limitations might do something that seems silly or goes against the grain of what's obviously good. That's okay: they're not you, and you're not them, and we all have our own reasons, and that's the point.

Re: Hosting SQLite Databases on GitHub Pages

#53
post #23

I can’t fully put my finger on why exactly, but I feel that this is a transformative idea. What’s to stop me from emulating a private SQLite DB for every user of a web app, and use that instead of GraphQL?

Nothing stopping you doing that right now with localStorage or IndexedDB. The issue is the browser cannot be trusted to keep that data, or at least these APIs aren't designed for long-term persistent storage. If we could solve this problem, we could go a long way towards some level of decentralisation. On the other hand, which is more secure? Your service or the user's machine. So there's a lot to consider.

Wouldn't such a model limit the user to just one device? Usually there's no sync of localStorage across devices.

Re: Hosting SQLite Databases on GitHub Pages

#54

Earlier quoted context omitted.

Android is used more here in India. Personally I haven't even touched an Apple device till date because it's just hard to find except in Apple store or if you have very rich friends. So I had no idea. > They are all Safari skins That's crazy.

There might be good reasons to do it from Apple’s point of view. For example, if Apple allowed third-party browser engines, any vendor could offer a browser with a vulnerability. That browser could be abused to install apps through that vulnerability, including malicious ones. I’m not implying that it’s ok for Apple to act like that. My point is that it’s consistent with their security model for iOS.

The real reason is that they want you to get your apps from the app store and that's it. They can't get paid when people install PWAs or sideload open source apps. Same reason for killing the headphone jack. Same reason for having a different shape magsafe for every generation of MacBook. To keep the suckers spending.

Re: Hosting SQLite Databases on GitHub Pages

#55

Earlier quoted context omitted.

There might be good reasons to do it from Apple’s point of view. For example, if Apple allowed third-party browser engines, any vendor could offer a browser with a vulnerability. That browser could be abused to install apps through that vulnerability, including malicious ones. I’m not implying that it’s ok for Apple to act like that. My point is that it’s consistent with their security model for iOS.

The real reason is that they want you to get your apps from the app store and that's it. They can't get paid when people install PWAs or sideload open source apps. Same reason for killing the headphone jack. Same reason for having a different shape magsafe for every generation of MacBook. To keep the suckers spending.

MagSafe has had only two variants while it existed (2006–2017). That’s more than a decade.

Both MacBook Pros I’ve owned at that time each came with its power supply included, and lasted me seven years each.

I consider myself frugal, not a sucker.

Re: Hosting SQLite Databases on GitHub Pages

#56

Earlier quoted context omitted.

There’s a market for both devices, and justifiably so. Why not celebrate the fact that people have a choice?

separate from this, one of those devices is slowing the web from fully transitioning back to webapps.

You mean Safari and how it restricts local storage for privacy reasons? Or have I missed something else?

Re: Hosting SQLite Databases on GitHub Pages

#57

Earlier quoted context omitted.

It was deprecated because it was difficult to write a standard spec for the existing SQLite code (but a key value system is much easier to specify as there is no SQL language).

Yeah but a key value system lacks all the really good things about SQL. And locally with sqlite you don't really have to worry about latency, so you should be able to just get atomicity and consistency on the thread. This shouldn't be a lot to ask from an embedded web DB. I think as with other standards bickering, ten years from now something (Canvas API) will come out that more or less replicates the technology that…

> Yeah but a key value system lacks all the really good things about SQL.

IndexedDB lacks SQL support natively, but there is nothing about key value systems in general that preclude the use of SQL. CockroachDB comes to mind as a key value store that depends on SQL. SQL support could be built atop IndexedDB by a willing developer. JsStore is in that vein, although the approach is a little bit different.

Re: Hosting SQLite Databases on GitHub Pages

#58

I get this error on Firefox 90.0.2 on Debian 10. It works in chrome though. [error: RuntimeError: abort(Error: Couldn't load https://phiresky.netlify.app/world-development-indicators-sq... . Status: 0). Build with -s ASSERTIONS=1 for more info.] Other than that, is pretty awesome and exactly what I was hoping for.

Looks like Netlify changed something since I wrote this article regarding what headers they send. Detecting support for Range-requests is kinda tricky and relies on heuristics [1]. Not sure why it still works in Chrome though. You can go to this version of my blog, it should work there: https://phiresky.github.io/blog/2021/hosting-sqlite-database... Maybe the link could be updated? Except for the DOM demo, since thos…

Don't rely on browser implementation details and a hope that they won't break in the future. Add a small supplementary file to your published data which has a known pattern at a fixed offset, then make a request for that offset and check the response.

Re: Hosting SQLite Databases on GitHub Pages

#59

Huh, this is a funny one. I had this idea a long time ago when doing some napkin design of a "static wiki". Problem was the querying didn't fit how software optimizes content delivery, so millions of people requesting from a single database would most likely be difficult to accomplish in a performant manner. Secondarily writing to said database would of course be impossible because locking, and you'd need a server an…

> People with older phones (which is many) wouldn't be able to use it at all.

Caniuse.com does not agree with this assertion.

Re: Hosting SQLite Databases on GitHub Pages

#60

This is a great example of how as technology changes, it changes use cases, which can prompt a revisiting of what was once considered a good idea. You'll often see the pendulum of consensus swing in one direction, and then swing back to the exact opposite direction less than a decade later. 2010s saw REST-conforming APIs with json in the body largely as an (appropriate) reaction to what came before, and also in accor…

I would think that if we know the SQL-queries we need we could pre-perform them and store the results into simple indexed tables. The web-app would then need to just ask for the data at a given index-value. No SQL needed on the browser. Could this work? Pre-executing SQL.
Post reply on HN