Live data from Hacker News

Ask HN: Is it possible to have forum type community running without a database?

news.ycombinator.com

21–30 of 66 posts

Re: Ask HN: Is it possible to have forum type community running without a database?

#23
Yes, it's totally possible to write a blog, forum, or pretty much any "Web 2.0" application without using a database. You just need to be very, very careful with how you design your filesystem layout and file format.

As for the file format, it would be best to stick with standard formats like JSON, XML, or YML. Or the standard serialization method for your favorite language, such as pickle() for Python and serialize() for PHP. Try not to invent a brand-new format, it's going to be error-prone and generally slower than using a standard format.

A more difficult task is to maintain an index of some sort, separate from all the individual files, so that you won't have to read and parse every single file in order to generate the forum listing. Whenever a reply is posted, you'll need to bump a thread to the top of the listing. Think about how you can implement this without modifying or renaming several files at a time.

And then you'll need to ask yourself how you're going to prevent inconsistencies in your data over the next few years. What if you decide to add a new field to the JSON schema and all the old files don't have that field? What if you delete a thread but an error occurs halfway through and some of the individual posts still remain? You'll need to write logic to handle such edge cases as well.

SQLite solves a lot of these problems while offering the same sort of performance, if not better, on resource-constrained environments. As far as SQLite is concerned, a Raspberry Pi is a very powerful machine. There's no reason why you sholdn't be able to enjoy all the benefits of flat files together with all the benefits of a relational database.

Re: Ask HN: Is it possible to have forum type community running without a database?

#24
post #3

You can run a community of sorts, but the feature list will be sparse. The only viable system sans-database that comes to mind is plain-text. I'm a member of a forum that had an old system in the beginning that used plain text "database" for a very long time. Performance was fairly reasonable at 5000 visitors per day, if the daily hit counter was to be trusted. They since moved it to SQLite, which is very reliable in…

Thanks, I appreciate the nice answer but it's amazing how often newbies get laughed at for being curious! Glad it hasn't happened today :-) So on to your answer, I'm going to be running this community at way less than 5000 visitors per day! So from what I understand, the site was run using PHP and every time somebody made a new entry a new file was created on disk with a reference to that file in the main index page.…

There are lots of samples for PHP, but you don't need to study that. As d33n suggests, there are examples that already use flat files as a database and you can browse Github for code examples.

Our forum was setup so that one thread = one HTML file. Any new replies to the thread were appended to the bottom of the file right before the footer. The added benefit of this is that it was very quick to read new posts as there was no processing taking place after the post is created.

Ex: If a new thread is being created, it will generate an ID, say 140585637400000000. Now this gets translated into a path : /140/585/637/400/000000.html and a file is created there.

The body of the new thread is added to the HTML file. Any subsequent replies are added to that same file right below the previous one.

So when a visitor requests example.com/topic/140585637400000000, the script takes the last part, turns into a file path, adds a template - which has the reply form with that ID - and sends it to the user. No additional processing needed. When a new reply is made, the script builds the path again and adds it to the bottom of the same HTML file. And so on...

I think your biggest hurdle is the initial planning. Try to carefully plan this out as much as you can, but obviously you won't know what future circumstances will bring. If you build it in such a way so that the storage mechanism doesn't need to change much or at all, you should be most of the way there.

Edit: Maybe I can convince the admin to publish some of the source. You're not the only one to be interested in something similar and I think there's a real demand for lo-fi community software.

Re: Ask HN: Is it possible to have forum type community running without a database?

#25

One of the main points of static sites is that they don't update terribly often. Forums, on the other hand, update constantly. You are wanting something with good caching (server side and client side), and thats about it. Otherwise, you are going to be regenerating the pages constantly on every post/update.

"good caching" needs to be "stores in cache forever, unless an update is required, at which point, only the content that actually changes is updated in cache" to be as good as static files. In my experience, caching is never that good. What's the problem with writing to the fs on every POST? The number of GETs will still outweigh the number of POSTs significantly.

Wouldn't Russian Doll caching tick that box?

Re: Ask HN: Is it possible to have forum type community running without a database?

#26

One of the main points of static sites is that they don't update terribly often. Forums, on the other hand, update constantly. You are wanting something with good caching (server side and client side), and thats about it. Otherwise, you are going to be regenerating the pages constantly on every post/update.

"good caching" needs to be "stores in cache forever, unless an update is required, at which point, only the content that actually changes is updated in cache" to be as good as static files. In my experience, caching is never that good. What's the problem with writing to the fs on every POST? The number of GETs will still outweigh the number of POSTs significantly.

It's my understanding that forums like HN and Reddit don't regenerate the front page for every post or view; rather, it is regenerated every second or two and cached in RAM.

Of course, it's unlikely you'd run a forum as big as HN or Reddit on a Raspberry Pi.

Re: Ask HN: Is it possible to have forum type community running without a database?

#27
I hate it when people change the question so they can give the answer they want to give, but I'm going to do it anyway, as I think its a relevant data point; I think you're dismissing the ability to run a database too quickly.

I have done almost exactly what you're talking about with a raspberry pi, using Ruby on Rails (3.0.x era) and sqlite. the Rasppi (model B) had enough horsepower to run in development mode on webrick and handle ~5 rps with page rendering and database calls, and that was in develop! when running in production, rails does less reloading of resources, so should be even more efficient.

Re: Ask HN: Is it possible to have forum type community running without a database?

#28
I sure do think it is possible. I am not sure how soon you will actually run into problems, but I have been using Dokuwiki, which uses text files rather than a database for all storage purposes. Although it is not in itself a good example for a community-centric software, it shows what can be done, the contents are versioned, there are access control lists and so on.

If you can build a wiki based on files, why would you not be able to do so with some kind of forum community? Maybe Dokuwiki is already a good solution for you.

I feel that the others readers reactions show that you could elaborate your motivation some more. A database can be a quick and useful solution and solves many problems, such as distributed access, central management and administration, security and backup.

http://en.wikipedia.org/wiki/DokuWiki

Re: Ask HN: Is it possible to have forum type community running without a database?

#29
post #20

You're probably overthinking this. While the Pi is limited in processing power and I/O, if it's really a small community it will run a forum just fine. Here's what to look out for: 1) Make sure you run NginX, it's fast and memory-efficient. 2) Install PHP-FPM and have NginX connect to it using a socket file . Keep the number of FPM workers small. 2a) Zend Opcache comes standard with PHP now, but there are scenarios w…

"While the Pi is limited in processing power and I/O"

This line is the key to the question. Simply roll back time until the specs of the Pi would be a "decent" or at least "cheap" webserver for that time, the kind of server you'd run a small forum upon at that time. OR if not decent for bare metal hardware, decent for a virtualized image on a bigger server.

Its not like 2014 is the first year in human history with webservers or web based forums.

So people were running forums on tiny virtualized servers with about those specs on linode perhaps just a couple years ago. Perhaps even today either on very tiny linodes or competitors.

The only real problem is the software. So use old stuff open to individual and class of vulnerabilities that were fixed 5 years ago, or take an obese beast of a modern forum and give it a liposuction?

Look into puppet and run the master somewhere else and automate it enough that you can go from a bare metal Pi to a live host in ten minutes or so. You'll be doing it again sooner or later at a time not of your choosing, so may as well get it right now.

Also, implement a backup strategy and automate its restoral process (and integrate with above)

Re: Ask HN: Is it possible to have forum type community running without a database?

#30
I'm not sure if this counts as a database, but you can try out things like Firebase or Parse, which allow you to fetch data from a cloud server, and render it on your server. So your application will be just a dumb static client containing templates and application logic in HTML/CSS/JS, while the data would be stored on the firebase servers, directly fetched from there by your visitors on page-load, making your work pretty easy.
Post reply on HN