This article reminded me of this site: http://howfuckedismydatabase.com/
I don't understand the goal of this site. :|
Database of Databases
31–40 of 65 posts
Re: Database of Databases
#32Earlier quoted context omitted.
Should be fine for reads. Sqlite.org is dynamic, pulling from sqlite data for ~20% of the pages, and it does fine with HN piling on. The single threaded would be an issue for writes, but I don't see why they would be doing writes for a page view. See https://www.sqlite.org/whentouse.html
Well it clearly wasn't :) Site is down already
There's several layers. It's Python/Django. I don't think we know what the issue really is. They could be, for example, logging visits to Sqlite. Or other issues unrelated to Sqlite.
Re: Database of Databases
#33But does it include itself?
Re: Database of Databases
#34They're using SQLite which is single threaded and single user by design. So this website will not be able to service that much traffic.
SQLite isn't single threaded - and it can support multiple readers very well.
The limitation with SQLite is that it doesn't support concurrent writes well - it needs to take a lock on the entire database to perform a write.
Writes are crazy fast (a few ms) so this often isn't a problem - but it does mean you wouldn't want to use it to build a site that has many people writing at once, like Hacker News for example.
For a site that has low (or no) writes, SQLite works really well even at a much larger scale - 100s of requests a second.
Re: Database of Databases
#35Archived page (5days ago):
https://web.archive.org/web/20200911140514/https://dbdb.io/
Re: Database of Databases
#36Re: Database of Databases
#37The amount of databases I'd like to play with and explore greatly outweighs the amount of time or reasons I have to explore databases.
Re: Database of Databases
#38They're using SQLite which is single threaded and single user by design. So this website will not be able to service that much traffic.
There is zero reason a carefully engineered application utilizing SQLite cannot completely saturate the IO capabilities of the host it resides on.
Going even further, there are no competing technologies (i.e. hosted SQL solutions) which, when running in single node/instance mode, are competitive with the performance of well-tuned SQLite.
PRAGMA journal_mode=WAL makes all the difference in the universe.
Re: Database of Databases
#39Earlier quoted context omitted.
Their repo shows the "DummyCache" (no caching) as the default for the Python/Django setup: https://github.com/cmu-db/dbdb.io/blob/master/dbdb/settings.... I wonder if that's how it is in production.
For a mostly static site you would want to cache "over" Django, probably in a caching proxy or CDN. Of course there are a lot of details, such as many CDNs will always go to the origin on a edge miss instead of locating another copy in the CDN. Of course running a caching nginx on the same box as the Django is probably way more performant than caching "under" Django
It's intermittently working now. Shows the server as Apache 2.4.18 / Ubuntu. Apache can page cache, but I assume they aren't using it. I don't see any typical CDN headers either.
Also, somewhat odd, they are bounding the browser-side cache to 10 minutes:
Date: Wed, 16 Sep 2020 18:19:13 GMT
Expires: Wed, 16 Sep 2020 18:29:14 GMT
Re: Database of Databases
#40Any good learning resources for implementing your own DBMS, and DB internals? (not DB theory, not SQL, not intro).
Yes, the Postgresql source and associated docs. It's a well-structured code base that is easy enough to dive into, if you don't try to understand the whole thing at once. I'm slowly reading it in bits and pieces.
Here's a Github mirror so you can browse around: https://github.com/postgres/postgres
Note that the offical repo is at git.postgresql.org.