Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Smaller is better – The rise, fall, and rise of flat file software

wilcosky.com

Post reply on HN
Print view View original

121–130 of 152 posts

Re: Smaller is better – The rise, fall, and rise of flat file software

#121

Post by forgotmypw17 »

andrewstuart2 wrote: post #60
OJFord wrote: post #50

Earlier quoted context omitted.

For me it's about proprietary vs. readable - unless it's obviously necessary for performance I think I'd still prefer 'flat file' to sqlite, but basically I just want something that isn't an incomprehensible blob, that I can use with other tools etc. Bonus points for some open standard, but some kind of understandable format even if proprietary is miles ahead of proprietary and cryptic. I wish Fusion360 for example h…

Perhaps it was deliberate, but your last paragraph (plaintext vs db) really just reinforces your point of standards. There's nothing particularly special about plaintext aside from its total ubiquity. Everything today uses and understands ASCII/UTF-8 and we've used that, plus some handy human conventions like limited character count between newline characters, to build up tools like git to help manage the changes in…

> There's nothing particularly special about plaintext aside from its total ubiquity.

But ubiquity is extremely special and useful and essential if you want to maintain ownership of your data.

Re: Smaller is better – The rise, fall, and rise of flat file software

#122

Post by ravi-delia »

bombela wrote: post #43
ReactiveJelly wrote: post #34

Earlier quoted context omitted.

If you're running Postgres in Docker, you can just stop the container and tar up the data volume and start it somewhere else (This also works without Docker of course, but Docker makes me feel safer when I do stuff like this)

Why does Docker makes you feel safer here? Is it because you feel more confident there is no database process still running?

If you're an ignorant fool such as myself, your knowledge of a database's internals is sufficient only to inspire fear. With Docker I know that whatever black magic the database uses is contained within a black box I can copy wholesale. Anything important must be in there, since I didn't give it anything else!

Re: Smaller is better – The rise, fall, and rise of flat file software

#123

Post by WalterGR »

Danborg wrote: post #114
WalterGR wrote: post #106

Earlier quoted context omitted.

> Using flat files only solves half the problem, in order to truly be able to host anywhere While that may be a problem for some, I don't think it's the problem for the purposes of this article and discussion. Different situations have different requirements.

Still, it is a consideration, and it all factors in. Copying your WonderCMS files to another host, great… oops, it doesn’t have PHP mbstring extension, sorry, won’t run. (And if you’re trying to use “free” hosting, you probably don’t have privs to install the missing requirements.)

What I mean is that being able to host anywhere is a constraint you're adding, not one that's in the original article.

If you have zero control over your environment and no ability to inspect it, then I agree that static files are probably the way to go.

Re: Smaller is better – The rise, fall, and rise of flat file software

#124

Post by lmm »

DeathArrow wrote: post #53
GuB-42 wrote: post #30

Maybe it is just a reflection on the state of filesystems vs databases. There is no fundamental difference between a database and a flat file, it is all bytes on a disk/memory in the end. So it is mostly a question of balancing the roles of the hardware, OS, and application software. For example, if the reason you are using a database is that it does a particularly good job at limiting disk IO, then it may not be nec…

A database can be relational, it can make some guarantees, it can pass Acid. With a file you are responsible for performance, reliability, integrity and security of the data. Do you believe that you can do a better job than those tens of guys who contributed to a DBMS? Go ahead. Do you not need reliability, integrity, security, performance? Go ahead. So use a file to store data instead of database but know the trade-…

ACID comes with costs, those guys may be smart but they're not magicians. As a rule of thumb, if you never handle a transaction failure or deadlock with something that is not retrying or failing, you shouldn't be using an ACID-based system since you're not getting any benefit from it.

And if you e.g. have multiple dataflow paths with different priorities then you also probably shouldn't be using a DBMS, because they're too much of a closed world. Using database technology as a library would be a better approach; otherwise sooner or later you'll find you need to customize the internals of your DBMS, and with most extant products that's somewhere between hard and impossible.

Re: Smaller is better – The rise, fall, and rise of flat file software

#125

Post by lmm »

Gigachad wrote: post #109
DeathArrow wrote: post #51

A file system can be considered a document database with the key being the file name.

If you used it somewhat like how you use redis , you’d probably run out of inodes and other performance issues. Directories aren’t designed to have millions of files in them.

Depends on the filesystem. ReiserFS used to be popular for mail servers or especially news servers, precisely because it was great at handling millions of small files.

Re: Smaller is better – The rise, fall, and rise of flat file software

#126

Post by pinneycolton »

mgkimsal wrote: post #47
config_yml wrote: post #23

I remember when I got started with webdev around 2002, a lot was built on flat files. And so did I, because that’s how my “internet mentor” did it: guestbooks, bulletin boards, mailing lists, shoutboxes, CMSs. All in flat files, except I used php and not Perl like him.

I started with php and msql back in ... 1996. Then in 98 took a job and much was perl. This was a web agency, and there was one client running Oracle on a sparc (IIRC), and some folks were tested out 'asp' and using MSSQL 6.5 (again IIRC - 7 wasn't out until later). But for those of us using perl... we were forced in to flat files and/or dbm files for our data storage needs. I tried to ask for msql or mysql, but were…

Same. Around then, I requested MySQL and was told no for the same reasons you were. So I implemented my database as flat files and proceeded to use up all the inodes a few weeks later on a large call center server used for interviewing and data collection.

I received a panicked call from the admin that turned down my request. He wasn’t happy with me but understood the role he played in creating an expensive disaster.

Plenty of learning to go around in that experience :)

Re: Smaller is better – The rise, fall, and rise of flat file software

#127

Post by notriddle »

funstuff007 wrote: post #119
OJFord wrote: post #50

Earlier quoted context omitted.

For me it's about proprietary vs. readable - unless it's obviously necessary for performance I think I'd still prefer 'flat file' to sqlite, but basically I just want something that isn't an incomprehensible blob, that I can use with other tools etc. Bonus points for some open standard, but some kind of understandable format even if proprietary is miles ahead of proprietary and cryptic. I wish Fusion360 for example h…

> main reason I say I'd still prefer plaintext to db Is there a non-proprietary somewhat sane way to do version control for spreadsheets (Excel and LibreOffice)?

Tell LibreOffice to save the file as Flat XML.

Re: Smaller is better – The rise, fall, and rise of flat file software

#128

Post by jpgleeson »

hemmert wrote: post #79

I can‘t recommend Kirby enough: https://www.getkirby.com I built many of my (project) websites with it, https://www.humangodinterfaces.com https://www.perspectives-in-play.com https://www.fabianhemmert.com https://www.escape-team.com it‘s blazingly fast (performance-wise and in terms of building your site), super easy to customize and I never missed my database.

+1 to this. I made some websites for hire in my earlier days and really enjoyed using Kirby. It was simple to setup, simple to host, and then simple to teach the clients how to update the content themselves. I just checked there and one of them is still going strong nearly 10 years later.

Re: Smaller is better – The rise, fall, and rise of flat file software

#129

Post by rrrrrrrrrrrryan »

djrobstep wrote: post #120
rrrrrrrrrrrryan wrote: post #49

Earlier quoted context omitted.

> There is no fundamental difference between a database and a flat file, it is all bytes on a disk/memory in the end. A database has a whole engine, which I'd argue is a pretty fundamental difference. I think I get what you were trying to say, though: there's no fundamental difference between database tables and flat files, and there's no fundamental difference between a database and a filesystem.

How is a filesystem not a "whole engine"?

A filesystem is, but a flat file alone is not. That was the only point of my comment - I was just being pedantic.

Re: Smaller is better – The rise, fall, and rise of flat file software

#130

Post by eternityforest »

Lately I've become a pretty big believer in flat files.

I don't see much value at all in small software for the sake of being small, but files are something really special.

You can SyncThing them. You can version control them. You can browse and delete them.

And best of all, they're not really flat anymore. A file is a typed object that tools know how to work with(As long as you do the sane thing and use a common file type).

Databases are often appropriate, but in those cases, I think sqlite is the better choice more often than not, unless you've truly got a large number of users all doing things at once.

Post reply on HN