Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we sped up Notion in the browser with WASM SQLite

notion.so

Post reply on HN
Print view View original

31–40 of 107 posts

Re: How we sped up Notion in the browser with WASM SQLite

#31

Post by bastawhiz »

bearjaws wrote: post #10

Overall I think using SQLite locally to offload database work is incredibly powerful. Given most laptops have an SSD now a days, you can scan an entire 30-100MiB SQLite db in miliseconds. Meanwhile in AWS you would pay $27k a month to have the same IOPS as a Lenovo Thinkpad X1. I just got done with a side project using WASM SQLite as well, it's incredibly powerful, even supports full text search. My project "cluttr"…

> Meanwhile in AWS you would pay $27k a month to have the same IOPS as a Lenovo Thinkpad X1.

This is kind of an unfair comparison. Essentially nobody needs a million iops for their database. Even an extremely busy database doesn't need to scan all of the data it holds (or at least, if it does, you're using it very wrong—that's why we have indexes).

A fast disk is possible on a laptop because it's a tiny hop to RAM. And it's desirable because you probably have nowhere near 2TB of RAM handy, so you need it to be fast.

In the cloud you can get 2TB of ram for $11k/mo (4x r6g.16xlarge). Not that you need anything like that to run your database. Most of that data is never being queried.

It's also the case that a laptop workload is very different than a server workload. If I run a steam game, I want it open fast. My laptop isn't crunching numbers on all the bytes at that moment. When I run a table scan on a Postgres table, processing needs to happen on every single tuple. A million iops isn't useful if your CPU immediately becomes the bottleneck. A Thinkpad would simply never match the response times of a server with a tenth of the iops under load (if the workload required scanning huge amounts of data).

So yes, the iops are more expensive, but that's really not a metric that anyone in the target market is hurting over.

Re: How we sped up Notion in the browser with WASM SQLite

#32

Post by dbalatero »

NelsonMinar wrote: post #29

Why is IndexedDB so bad that you have to load a second database in WASM instead? I've used the API successfully but never measured performance.

It's buggy depending on browser implementation, Firefox will corrupt DBs, it's a herculean task to debug things with users, it blocks the main thread for reads or writes (maybe wasm stuff does too).

Re: How we sped up Notion in the browser with WASM SQLite

#33

Post by stackskipton »

bastawhiz wrote: post #31
bearjaws wrote: post #10

Overall I think using SQLite locally to offload database work is incredibly powerful. Given most laptops have an SSD now a days, you can scan an entire 30-100MiB SQLite db in miliseconds. Meanwhile in AWS you would pay $27k a month to have the same IOPS as a Lenovo Thinkpad X1. I just got done with a side project using WASM SQLite as well, it's incredibly powerful, even supports full text search. My project "cluttr"…

> Meanwhile in AWS you would pay $27k a month to have the same IOPS as a Lenovo Thinkpad X1. This is kind of an unfair comparison. Essentially nobody needs a million iops for their database. Even an extremely busy database doesn't need to scan all of the data it holds (or at least, if it does, you're using it very wrong—that's why we have indexes). A fast disk is possible on a laptop because it's a tiny hop to RAM. A…

I'd also point out about AWS cost, which is pretty high, getting IOPS is easy. Getting Redundant IOPS is hard part.

Re: How we sped up Notion in the browser with WASM SQLite

#34

Post by threetonesun »

ilrwbwrkhv wrote: post #24
nailer wrote: post #23

Earlier quoted context omitted.

They’re a very un-user focused company. They used the | cursor for any horizontally scrolling content for years, making a kanban board with non default states is nightmarishly complex, and there’s a billion odd UI slownesses and niggles. But they’re recently added AI.

I think their real innovation was the community and getting "cosy productivity" during Covid going. They are more of a social phenomenon than a technical one.

Notion is a great example of my hatred of the current "upgrade or die" delivery method of software. Notion 1.0, when it was mostly text with a unique way to view tabular data, was fast and enjoyable to use. Once it shifted to competing with Atlassian et al, it acquired all the bloat those products already have.

Re: How we sped up Notion in the browser with WASM SQLite

#35

Post by bearjaws »

bastawhiz wrote: post #31
bearjaws wrote: post #10

Overall I think using SQLite locally to offload database work is incredibly powerful. Given most laptops have an SSD now a days, you can scan an entire 30-100MiB SQLite db in miliseconds. Meanwhile in AWS you would pay $27k a month to have the same IOPS as a Lenovo Thinkpad X1. I just got done with a side project using WASM SQLite as well, it's incredibly powerful, even supports full text search. My project "cluttr"…

> Meanwhile in AWS you would pay $27k a month to have the same IOPS as a Lenovo Thinkpad X1. This is kind of an unfair comparison. Essentially nobody needs a million iops for their database. Even an extremely busy database doesn't need to scan all of the data it holds (or at least, if it does, you're using it very wrong—that's why we have indexes). A fast disk is possible on a laptop because it's a tiny hop to RAM. A…

"Nobody needs IOPS until they need IOPS"

I've had to scrub a multi terabyte database of PII before moving to a staging environment, it hurts. With modern data architectures, you may write the same data 4-5 times in its life cycle, staging data, data warehouses, marketing, PowerBI, Looker, etc...

Especially reporting solutions, where they may aggregate massive amounts of data and write it to temp tables.

It will require IOPS, and you will pay handsomely for it.

Re: How we sped up Notion in the browser with WASM SQLite

#37

Post by ggregoire »

> Using SQLite improved page navigation times by 20 percent in all modern browsers.

Couldn't you cache the data in LocalStorage and get similar speed improvements?

I mean, how to run SQLite in the browser is a great topic and surely has a lot of very good use cases. But attributing the performance improvements to it sounds a bit misleading, when you had no client caching strategy before and you do now. Obviously loading data from a local cache is faster than loading data from a remote server.

Comparing WASM SQLite vs. other client caching solutions (LS, IndexDB, etc) could be interesting. In terms of perf, complexity, compatibility, storage capacity, query language, etc.

Re: How we sped up Notion in the browser with WASM SQLite

#38

Post by jauntywundrkind »

AlexErrant wrote: post #6

> OPFS doesn’t come with graceful handling of concurrency out of the box. Developers should be aware of this and design around it. There's a multiple readers and writers proposal [0]. It's been "position: positive" by Firefox [1], implemented in Chrome [2], and ignored by Webkit [3] (of course). 0: https://github.com/whatwg/fs/blob/main/proposals/MultipleReadersWriters.md 1: https://github.com/mozilla/standards-posit…

It sucks that even the newest make-wadm-sqlite-fast FileAccessHandles are still an intermediated virtual file system hosted by the browser. I don't get why we have three different file system APIs on the web and none that just use files.

Meanwhile WASI has their own server-side file system APIs but neither WASI nor the browser side seem to have any effort to get on the same page. It'll be 2035 before component-model ends up being usable on the web, at this rate, before modules are modular.

Re: How we sped up Notion in the browser with WASM SQLite

#39

Post by jamil7 »

remram wrote: post #36

Notion takes 15s to load to an empty page. Then another 5 to dismiss the popup about new AI features and the like. I'm glad they are making their app faster, in the meantime I (browser user) have cancelled my team's subscription and will be using something else.

I was about to say, does Notion actually feel any faster to anyone? I stopped using it years ago but the last place I contracted at was invested in it heavily and the thing was crazy slow.

Re: How we sped up Notion in the browser with WASM SQLite

#40

Post by jauntywundrkind »

dbalatero wrote: post #32
NelsonMinar wrote: post #29

Why is IndexedDB so bad that you have to load a second database in WASM instead? I've used the API successfully but never measured performance.

It's buggy depending on browser implementation, Firefox will corrupt DBs, it's a herculean task to debug things with users, it blocks the main thread for reads or writes (maybe wasm stuff does too).

Wow this is so embarrassing as hell. Apparently even using a WebWorker can still cause the UI to block when talking to indexeddb. https://nolanlawson.com/2015/09/29/indexeddb-websql-localsto... https://issues.chromium.org/issues/41204713#comment28

It's crazy how much effort goes into adding new things to the web but stuff like this is almost 10 years old.

Post reply on HN