Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Smaller is better – The rise, fall, and rise of flat file software

wilcosky.com

Post reply on HN
Print view View original

101–110 of 152 posts

Re: Smaller is better – The rise, fall, and rise of flat file software

#101

Post by zokier »

tarr11 wrote: post #31
GuB-42 wrote: post #30

Maybe it is just a reflection on the state of filesystems vs databases. There is no fundamental difference between a database and a flat file, it is all bytes on a disk/memory in the end. So it is mostly a question of balancing the roles of the hardware, OS, and application software. For example, if the reason you are using a database is that it does a particularly good job at limiting disk IO, then it may not be nec…

Some databases also have the ability to be instantly portable, like a sqlite file. You can tar up your flat files as well, but that is a separate process and can be time consuming for larger directories. Perhaps there is a use case for a file system backed by portable database like sqlite, such that you can just mount the volume and send it around without pain, but can prefer to use normal command line utils (ls, rm,…

> You can tar up your flat files as well, but that is a separate process and can be time consuming for larger directories.

We have file formats like ODF which are just zip packages with regular files inside

Re: Smaller is better – The rise, fall, and rise of flat file software

#102

Post by chmod775 »

chme wrote: post #88
andrewstuart2 wrote: post #60

Earlier quoted context omitted.

Perhaps it was deliberate, but your last paragraph (plaintext vs db) really just reinforces your point of standards. There's nothing particularly special about plaintext aside from its total ubiquity. Everything today uses and understands ASCII/UTF-8 and we've used that, plus some handy human conventions like limited character count between newline characters, to build up tools like git to help manage the changes in…

> There's nothing preventing us from doing the same for other formats if they become commonly supported or expected. Much like SQLite and some of the tools cropping up. True. I mean if it would be common to put source code and text into sqlite files, then most tools would probably also deal with it. However there is a case about simplicity. For instance writing a diff algorithm on plain text files is much simpler and…

> For instance writing a diff algorithm on plain text files is much simpler and easy to understand, than writing diff algorithms on text or binary structures.

I'd argue the opposite. Coming up with a good, human readable, diff on unstructured text within acceptable runtime is not that easy. LCS is NP-hard and the naive algorithm has unacceptable performance and memory footprint.

However diffing something structured, especially when it has unique keys like in most databases, is really easy. Structure gives you an easy way to compare data systematically and is handy to express differences.

Re: Smaller is better – The rise, fall, and rise of flat file software

#103

Post by randombits0 »

throwawayboise wrote: post #97
DeathArrow wrote: post #53

Earlier quoted context omitted.

A database can be relational, it can make some guarantees, it can pass Acid. With a file you are responsible for performance, reliability, integrity and security of the data. Do you believe that you can do a better job than those tens of guys who contributed to a DBMS? Go ahead. Do you not need reliability, integrity, security, performance? Go ahead. So use a file to store data instead of database but know the trade-…

Yeah, none of that may matter for a "brochure" type of website, or a blog where you are the only person updating it. As soon as you start processing forms and doing anything transactional based on that, you'll be quickly reinventing a lot of wheels if you aren't using a DBMS of some sort.

But only the wheels you need, when you need them. Don’t carry the overhead unless needed.

Re: Smaller is better – The rise, fall, and rise of flat file software

#104

Post by zinclozenge »

throwawayboise wrote: post #94
Xunjin wrote: post #61

Earlier quoted context omitted.

I'd love watch a uncle Roger programming-wise reviews.

Maybe I would as well. Who is Uncle Roger though?

He's a character played by a comedian. Here's the video that made him popular https://www.youtube.com/watch?v=53me-ICi_f8

Re: Smaller is better – The rise, fall, and rise of flat file software

#105

Post by JumpCrisscross »

throwawayboise wrote: post #97
DeathArrow wrote: post #53

Earlier quoted context omitted.

A database can be relational, it can make some guarantees, it can pass Acid. With a file you are responsible for performance, reliability, integrity and security of the data. Do you believe that you can do a better job than those tens of guys who contributed to a DBMS? Go ahead. Do you not need reliability, integrity, security, performance? Go ahead. So use a file to store data instead of database but know the trade-…

Yeah, none of that may matter for a "brochure" type of website, or a blog where you are the only person updating it. As soon as you start processing forms and doing anything transactional based on that, you'll be quickly reinventing a lot of wheels if you aren't using a DBMS of some sort.

> none of that may matter for a "brochure" type of website, or a blog where you are the only person updating it

Doesn't Hacker News run on flat files?

Re: Smaller is better – The rise, fall, and rise of flat file software

#106

Post by WalterGR »

Danborg wrote: post #26

Using flat files only solves half the problem, in order to truly be able to host anywhere, you also need to be able to ditch PHP and server side processing requirements. (Wonder CMS, highlighted in this article, requires PHP with a couple of mods.) That’s why I think static site generators are the more interesting code in this space. I’m a big fan of Gatsby, but there are several other great ones as well. This allows…

> Using flat files only solves half the problem, in order to truly be able to host anywhere

While that may be a problem for some, I don't think it's the problem for the purposes of this article and discussion.

Different situations have different requirements.

Re: Smaller is better – The rise, fall, and rise of flat file software

#109

Post by Gigachad »

DeathArrow wrote: post #51

A file system can be considered a document database with the key being the file name.

If you used it somewhat like how you use redis , you’d probably run out of inodes and other performance issues. Directories aren’t designed to have millions of files in them.

Re: Smaller is better – The rise, fall, and rise of flat file software

#110

Post by Gigachad »

EGreg wrote: post #89

Why don’t more people store all the uploaded files inside BLOBS in a relational database instead of a file system? What are the downsides? I see the biggest downside being that the webserver will call your PHP app which will then proxy the data through itself one time. Well, maybe you can use ReactPHP or Swoole and then it’s faster than Node.js even… Also you then have more programmable options, for example around ac…

You usually want as much of your database stored in memory as possible. Binary blobs would bloat out the db pretty quick.

Fuse for s3 might work but you risk the chance of a generic abstraction triggering a massive set of actions that cost a lot. Something like running find or checking the size of a directory could end up costing you thousands.

Post reply on HN