41–50 of 66 posts
Sure. HN itself has no database, only the filesystem (or rather the filesystem is the DB).
Note that they are (or did? or might be?) moving to a DB format in the end, but for at least the first several years of Startup/Hacker News, it was S-expressions written to files.
Yes, it's totally possible to write a blog, forum, or pretty much any "Web 2.0" application without using a database. You just need to be very, very careful with how you design your filesystem layout and file format. As for the file format, it would be best to stick with standard formats like JSON, XML, or YML. Or the standard serialization method for your favorite language, such as pickle() for Python and serialize(…
As far as SQLite is concerned, a Raspberry Pi is a very
powerful machine. There's no reason why you sholdn't be
able to enjoy all the benefits of flat files together
with all the benefits of a relational database.
By the time you're done reinventing all the things a storage engine like SQLite or one of the NoSQL storage engines would give you, you'd be hard-pressed to do it more efficiently than they do.
So I wondered if it is possible to run a forum type
community - something along the lines of [elgg][1] or
[Friendica][2] - but without the MySQL or other type of
database backend?
A simple read-only query to SQLite or a NoSQL database is fast - generally just a few milliseconds and often less than one millisecond. The odds of you doing it much faster yourself are low.
Rendering pages is the slow part. Because each forum page you render is going to involve multiple queries to your storage engine (whether you go with SQL or roll your own) and a ton of string manipulation/concatenation.
So what I'd do is....
- Use SQLite for my storage engine. The 256 or 512MB of RAM on a RaspberryPi is plenty for SQLite. - Cache rendered pages (and/or page fragments) to disk, rather than writing my own storage engine.
I'd use a lazy caching/prerendering strategy. Suppose a discussion thread has 50 pages. One of your mods deletes a post on page 1. Now all 50 pages need to be re-rendered. You have two choices. You can either re-render all 50 pages right away, or you can simply delete all 50 pages right away and re-render & re-cache them as needed, when a user actually requests one of them. I'd do the latter.
I've used this strategy myself. It was a very common paradigm back in the 90s and early 2000s when web servers commonly had hardware specs (700mhz, 256/512MB RAM) that was quite similar to what a RaspberryPi has today. The hardest part of this strategy is getting cache invalidation correct. Every time your code writes to the database, it has to also be aware of which cached pages/fragments it needs to blow away.
(You know the old joke: "There are only two hard things in Computer Science: cache invalidation and naming things.")
For a forum:
1) You'll need a server with an API for submitting posts. You can use the filesystem as the DB, if you'd like.
2) This server will update the static website (i.e. regenerate it, and push it to GitHub Pages) on a regular interval (which could be as small as one minute).
So it's definitely doable, and offers huge benefits. For one, static pages can be served very efficiently. If you use GitHub pages, you can leverage its global Content Delivery Network (CDN) for free. And you are guaranteed a nearly zero downtime for free by GitHub.
The only "drawback" is the insignificant one minute delay before a new post goes "live". Shoot me an email if you'd like to collaborate on a static forum project!
So I wondered if it is possible to run a forum type community - something along the lines of [elgg][1] or [Friendica][2] - but without the MySQL or other type of database backend? You're attempting to optimize the wrong part of the architecture, I think. A simple read-only query to SQLite or a NoSQL database is fast - generally just a few milliseconds and often less than one millisecond. The odds of you doing it much…
https://en.wikipedia.org/wiki/Greymatter_%28software%29
When you'd add a comment to a blog post on Greymatter, it would write that content to it's own flatfile database, then trigger a rendering update which would read the content, transform it with a template, and produce a new .html file, overwriting the old one. Bam! Dynamic site based on (mostly) static pages.
I don't see why a forum or social platform couldn't be built in a similar manner. You won't be able to do much in the way of per-user authorization, but as long as all your content is supposed to be viewable to everyone, you should be OK.
One problem with Greymatter was that if a thread or site grew large, sometimes generating new files from templates would take so long that your connection to the CGI would time out before they were finished. Another problem was that two updates to the site posted at about the same time could stomp on each other during the rendering phase.
'Cause hey, flatfiles are hard. That's why I wouldn't forgo using a real database in favor of flatfiles. If you don't want to run MySQL or Postgres on an RPI, then use SQLite. There is, after all, a reason that these products exist. Let SQLite or MySQL deal with the flatfiles for your instead of re-inventing yet another wheel.
I also wouldn't update the .html files the moment someone posted. I'd have the scripts just commit the post to the database, and then regularly re-render the files from a cron script. This avoids having your rendering stage stomp on each other, provided that the rendering time is less than your rendering interval.
It may have been YaBB (http://sourceforge.net/projects/yabb/) (http://www.yabbforum.com) or some predecessor with a long-forgotten name. The earliest commit on that sourceforge site was 2003, but the screenshots look similar to what I remember.
https://www.dokuwiki.org/plugin:discussion
Though I think you could get the messages in a flat file (heck, BBSs did that since the 70s) though having a small db or some sort of link lists for user access, message indexing and such would probably add to the speed of the system.