Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lobste.rs is now running on SQLite

lobste.rs

Post reply on HN
Print view View original

141–150 of 209 posts

Re: Lobste.rs is now running on SQLite

#141

Post by inigyou »

wild_egg wrote: post #118
eterm wrote: post #43

Earlier quoted context omitted.

SQLite definitely seems like a poor choice for dealing with many concurrent requests. Maybe it's improved since I last used it, but to my knowledge SQLite essentially forces all writes to be serialised, at risk of data corruption otherwise. There are tricks for improving the performance such as WALs, but that is merely a performance boost rather than genuine concurrency with things like row-level locks that you might…

> SQLite definitely seems like a poor choice for dealing with many concurrent requests Can you qualify "many"? SQLite easily handles 100k+ writes per second and it's not hard to have app layer code serialize and batch writes to take advantage of that speed. Concurrent writes require a ton of overhead and your system and code can be quite a lot faster and simpler if you just skip the idea of them altogether.

SQLite serialises all writes - only one can proceed at a time and others must wait until that one is finished

Re: Lobste.rs is now running on SQLite

#142

Post by inigyou »

evanelias wrote: post #66
saila wrote: post #59

Earlier quoted context omitted.

The GPLv2 doesn't prevent a company from distributing binaries and making the source available on request while development is done behind closed doors. This wouldn't really be "open source" as we typically understand it and might be cause to exclude it from some Linux distros, for example. I'm not saying that K1 would do this (I have no idea who they are or what they do), but I also think it's a reasonable risk to c…

I don't see how that would be a realistic risk? Any MariaDB user (including Lobste.rs) could request the source, and per the GPL it must be supplied, so this isn't an effective strategy for any entity to go closed-source. And in any case, https://github.com/MariaDB/ links to mariadb.org which is run by the MariaDB Foundation, not the commercial enterprise owned by K1 .

Even with that in mind why switch to sqlite rather than PostgreSQL, which is roughly equivalent to MariaDB?

Re: Lobste.rs is now running on SQLite

#143

Post by christophilus »

gnulinux wrote: post #20
Kuyawa wrote: post #9

No intention to antagonize but... why not postgre? Even my side projects run on postgresql with no overhead at all

> run on postgresql with no overhead at all Citation needed. I love postgres as much as the next person, but it does have more overhead than Sqlite which is in an in-process db linked through compiled C code, it doesn't run as a separate server. Very significant difference that when you use Sqlite db, there is typically no db process other than your application logic, unless you implement the server yourself. If you…

If you run it on your one app server, it’s not much more work than SQLite. SQLite requires some special care when you’re dealing with multiple processes, and backups are about the same amount of effort.

Re: Lobste.rs is now running on SQLite

#144

Post by rsync »

sqlite3_rsync exists:

https://sqlite.org/rsync.html

... and is installed and available:

  # sqlite3_rsync -vv ./test.db user@rsync.net:test.db
"Ordinary rsync does not understand SQLite transactions. Rsync will make a copy of ORIGIN into REPLICA, however the copy might not be consistent. Parts of the copy might be from one transaction, while other parts might be from a different transaction. The database copy might be corrupt."

Re: Lobste.rs is now running on SQLite

#145

Post by inigyou »

cdrini wrote: post #90
qsera wrote: post #73

Earlier quoted context omitted.

I think it is better to have some elitist communities, than to drag everyone down to some least common denomination... I think that was one of the big mistakes/oversight that came with the Internet..

I think some are fine, but if I had to choose one I'd choose open access, so I think it's the more important of the two. And in fact it's because the web is open, "no permission needed" by default that allows elitist groups to exist on the web. So I don't think I see it as a mistake. The alternative would be... You can only create a website if you're invited to the internet by someone who already has access? You have…

That's actually how it already works, but there are large commercial access providers known as "ISPs". Before there were ISPs, you literally could only get on the internet with a direct connection to someone on the internet, usually a large university.

Re: Lobste.rs is now running on SQLite

#146

Post by inigyou »

pudgywalsh wrote: post #80
tverbeure wrote: post #57

Earlier quoted context omitted.

I wish lobste.rs would be a tiny bit easier to get access to. Some of my blog posts have been discussed there and every time I try to get access, I can’t get past their IRC stuff. As in: I just don’t understand how IRC works…

You should overlay a banner on your blog when they send you an HTTP Referer explaining their nonsense. Maybe then they'll get the message. Then again, maybe they like the ability to critique your posts without you being able to rebut their opinions. It's an effective tactic in online echo chambers.

Or redirect to a picture of a testicle in an eggcup.

Re: Lobste.rs is now running on SQLite

#147

Post by inigyou »

OutOfHere wrote: post #56
rickcarlino wrote: post #36

Earlier quoted context omitted.

Invitation-only is the future of online communities. Lobsters still has a pulse because you need to know someone to get in and it’s a bad look if you invite annoying people.

That's nonsense wrt invitation-only being the future. I can equally say that free speech is the future. For comparison, Reddit does very well without being invitation only.

Reddit is literally the dead internet right now.

It's also not open access. You have to run an invisible tightrope so its algorithms won't automatically shadowban you and then you have to run a barely visible tightrope to get enough karma to not have your posts automatically removed from every subreddit.

Re: Lobste.rs is now running on SQLite

#149

Post by abetusk »

kgeist wrote: post #136
abetusk wrote: post #131

Earlier quoted context omitted.

Could you go into more detail about this issue and provide some links to the documentation? I'm absolutely no expert, I'm just reading about it now, but from the SQLite WAL documentation [0]: > 7. There is the extra operation of checkpointing which, though automatic by default, is still something that application developers need to be mindful of. and from their "checkpointing" documentation [1]: > By default, SQLite…

On the page you linked: >However, if a database has many concurrent overlapping readers and there is always at least one active reader, then no checkpoints will be able to complete and hence the WAL file will grow without bound. >This scenario can be avoided by ensuring that there are "reader gaps": times when no processes are reading from the database and that checkpoints are attempted during those times. Dunno, may…

Thanks, this is really embedded in the documentation and is not at all even hinted at in the above documentation.

This seems like a huge DoS security hole.

I encourage you to write a blog post about this. It seems especially relevant.

Re: Lobste.rs is now running on SQLite

#150

Post by [deleted] »

inigyou wrote: post #140
kgeist wrote: post #136

Earlier quoted context omitted.

On the page you linked: >However, if a database has many concurrent overlapping readers and there is always at least one active reader, then no checkpoints will be able to complete and hence the WAL file will grow without bound. >This scenario can be avoided by ensuring that there are "reader gaps": times when no processes are reading from the database and that checkpoints are attempted during those times. Dunno, may…

Why not grab a rwlock so that no readers or writers can proceed and all existing ones are drained, and then force a checkpoint? What's with the exponentially increasing sleep?

[deleted]
Post reply on HN