Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lobste.rs is now running on SQLite

lobste.rs

Post reply on HN
Print view View original

131–140 of 209 posts

Re: Lobste.rs is now running on SQLite

#131

Post by abetusk »

kgeist wrote: post #130

They use WAL in SQLite. If I continuously perform reads/writes so that they overlap with no gaps, I can make their VM go down because SQLite will not have time to initiate a checkpoint to trim the WAL file. SQLite waits for a time window without any active reads/writes before starting a WAL checkpoint. If there isn't one, the WAL will grow indefinitely, eating up all the disk space on the VM. It's in SQLite's documen…

Could you go into more detail about this issue and provide some links to the documentation?

I'm absolutely no expert, I'm just reading about it now, but from the SQLite WAL documentation [0]:

> 7. There is the extra operation of checkpointing which, though automatic by default, is still something that application developers need to be mindful of.

and from their "checkpointing" documentation [1]:

> By default, SQLite does a checkpoint automatically when the WAL file reaches a threshold size of 1000 pages.

I'm only skimming but I see no mention of waiting for an idle time window.

Have you been successful in DoS'ing lobste.rs?

[0] https://sqlite.org/wal.html

[1] https://sqlite.org/wal.html#ckpt

Re: Lobste.rs is now running on SQLite

#132

Post by alfiedotwtf »

eterm wrote: post #43
homebrewer wrote: post #6

It's been fairly unstable recently, pages sometimes render for several seconds which I've never seen under MariaDB. Used to be instantaneous, always. Sometimes (maybe 5% or less) the request won't render at all, and you get a browser error page. Today they ran into this bug, lost a bunch of voting data, and went into read-only mode for several hours: https://github.com/rails/rails/pull/57128 I wonder how much of this…

SQLite definitely seems like a poor choice for dealing with many concurrent requests. Maybe it's improved since I last used it, but to my knowledge SQLite essentially forces all writes to be serialised, at risk of data corruption otherwise. There are tricks for improving the performance such as WALs, but that is merely a performance boost rather than genuine concurrency with things like row-level locks that you might…

Depends on the architecture..

They could have one SQLite instance per user and then have a single sweeper that goes through all last writes and then replicates it to the main instance - eventual consistency fanned out across files

Re: Lobste.rs is now running on SQLite

#133

Post by alfiedotwtf »

zinodaur wrote: post #111
evanelias wrote: post #16

Some of the cited reasons for moving off MariaDB [1] seem misguided, in my opinion. Especially the part about "K1 are very enterprise-focused, so the database is likely to focus its work on features that are not relevant to us. There's increased risk they drop the free/open source version we use" K1 acquired the commercial entity behind MariaDB Enterprise, but that's separate from the non-profit MariaDB Foundation. A…

As a big user of MySQL - moving off of it does seem like a good idea, if you can. The longer you wait, the harder it will be to flee

Any reason besides trust me bro?

Re: Lobste.rs is now running on SQLite

#134

Post by evanelias »

zinodaur wrote: post #111
evanelias wrote: post #16

Some of the cited reasons for moving off MariaDB [1] seem misguided, in my opinion. Especially the part about "K1 are very enterprise-focused, so the database is likely to focus its work on features that are not relevant to us. There's increased risk they drop the free/open source version we use" K1 acquired the commercial entity behind MariaDB Enterprise, but that's separate from the non-profit MariaDB Foundation. A…

As a big user of MySQL - moving off of it does seem like a good idea, if you can. The longer you wait, the harder it will be to flee

It's a large ecosystem with multiple server vendors (Oracle MySQL, MariaDB, Percona, VillageSQL) that are relatively easy to migrate between, compared to moving to a completely different DBMS ecosystem.

Sometimes there are valid technical motivations to move away completely, but a nonsensical fear of a massively-used GPL database somehow becoming closed-source (with no surviving forks or continuity) is not one of them in my book!

Re: Lobste.rs is now running on SQLite

#135

Post by binkHN »

arjie wrote: post #97
Aurornis wrote: post #25

Earlier quoted context omitted.

Lobste.rs is very anti-AI. They force any LLM topic to be tagged with “vibecoding” even when the majority of LLM posts are not about vibecoding. AI posts are usually the most commented on. Linus Torvalds’ comments saying that LLMs are actually useful is still on the front page, tagged as vibecoding, and has a lot of comments from people mostly disagreeing with him. Lobste.rs is more of a monoculture than Hacker News.…

I originally went to lobste.rs because I wanted to avoid repetitive discussion and get more technical stuff. In time, though, lobste.rs got fairly boring since it's always the same stuff over and over again. I found an alternative means[0] of filtering out boring things here on HN and it's enjoyable to use again. 0: deleting comments of people who aren't interesting from my view and highlighting ones who often say th…

lobste.rs, quite literally, allows you to filter the entire site by tags, so you only see what you want to see. I wish more sites did this.

Re: Lobste.rs is now running on SQLite

#136

Post by kgeist »

abetusk wrote: post #131
kgeist wrote: post #130

They use WAL in SQLite. If I continuously perform reads/writes so that they overlap with no gaps, I can make their VM go down because SQLite will not have time to initiate a checkpoint to trim the WAL file. SQLite waits for a time window without any active reads/writes before starting a WAL checkpoint. If there isn't one, the WAL will grow indefinitely, eating up all the disk space on the VM. It's in SQLite's documen…

Could you go into more detail about this issue and provide some links to the documentation? I'm absolutely no expert, I'm just reading about it now, but from the SQLite WAL documentation [0]: > 7. There is the extra operation of checkpointing which, though automatic by default, is still something that application developers need to be mindful of. and from their "checkpointing" documentation [1]: > By default, SQLite…

On the page you linked:

>However, if a database has many concurrent overlapping readers and there is always at least one active reader, then no checkpoints will be able to complete and hence the WAL file will grow without bound.

>This scenario can be avoided by ensuring that there are "reader gaps": times when no processes are reading from the database and that checkpoints are attempted during those times.

Dunno, maybe Rails has a built-in workaround for this.

My workaround was to run a separate thread that monitored the WAL size on disk every second. If it went above the target size of 8 MB, my framework would enter "slow down" mode, where all reads and writes were artificially delayed by calling "sleep()", starting at 16 ms and gradually increasing the sleep time based on a few heuristics.

This allowed the application to have short gaps with no reads or writes, so the checkpointer could actually proceed.

Re: Lobste.rs is now running on SQLite

#137

Post by badatnames »

kgeist wrote: post #130

They use WAL in SQLite. If I continuously perform reads/writes so that they overlap with no gaps, I can make their VM go down because SQLite will not have time to initiate a checkpoint to trim the WAL file. SQLite waits for a time window without any active reads/writes before starting a WAL checkpoint. If there isn't one, the WAL will grow indefinitely, eating up all the disk space on the VM. It's in SQLite's documen…

[flagged]

Re: Lobste.rs is now running on SQLite

#138

Post by chrismarlow9 »

kgeist wrote: post #136
abetusk wrote: post #131

Earlier quoted context omitted.

Could you go into more detail about this issue and provide some links to the documentation? I'm absolutely no expert, I'm just reading about it now, but from the SQLite WAL documentation [0]: > 7. There is the extra operation of checkpointing which, though automatic by default, is still something that application developers need to be mindful of. and from their "checkpointing" documentation [1]: > By default, SQLite…

On the page you linked: >However, if a database has many concurrent overlapping readers and there is always at least one active reader, then no checkpoints will be able to complete and hence the WAL file will grow without bound. >This scenario can be avoided by ensuring that there are "reader gaps": times when no processes are reading from the database and that checkpoints are attempted during those times. Dunno, may…

https://github.com/openclaw/openclaw/issues/72774

Seems to indicate a collection cron with a manual query to truncate can fix it. This is fun little things I like to learn. SRE Easter eggs.

Re: Lobste.rs is now running on SQLite

#139

Post by kgeist »

badatnames wrote: post #137
kgeist wrote: post #130

They use WAL in SQLite. If I continuously perform reads/writes so that they overlap with no gaps, I can make their VM go down because SQLite will not have time to initiate a checkpoint to trim the WAL file. SQLite waits for a time window without any active reads/writes before starting a WAL checkpoint. If there isn't one, the WAL will grow indefinitely, eating up all the disk space on the VM. It's in SQLite's documen…

[flagged]

I reproduced it in Sqlite with short-lived reads/writes though. Other DBMSes seem to not have this issue (IIRC MySQL will block a write if WAL falls behind)

Re: Lobste.rs is now running on SQLite

#140

Post by inigyou »

kgeist wrote: post #136
abetusk wrote: post #131

Earlier quoted context omitted.

Could you go into more detail about this issue and provide some links to the documentation? I'm absolutely no expert, I'm just reading about it now, but from the SQLite WAL documentation [0]: > 7. There is the extra operation of checkpointing which, though automatic by default, is still something that application developers need to be mindful of. and from their "checkpointing" documentation [1]: > By default, SQLite…

On the page you linked: >However, if a database has many concurrent overlapping readers and there is always at least one active reader, then no checkpoints will be able to complete and hence the WAL file will grow without bound. >This scenario can be avoided by ensuring that there are "reader gaps": times when no processes are reading from the database and that checkpoints are attempted during those times. Dunno, may…

Why not grab a rwlock so that no readers or writers can proceed and all existing ones are drained, and then force a checkpoint? What's with the exponentially increasing sleep?
Post reply on HN