Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Smaller is better – The rise, fall, and rise of flat file software

wilcosky.com

Post reply on HN
Print view View original

91–100 of 152 posts

Re: Smaller is better – The rise, fall, and rise of flat file software

#91

Post by encryptluks2 »

zitterbewegung wrote: post #59
GuB-42 wrote: post #30

Maybe it is just a reflection on the state of filesystems vs databases. There is no fundamental difference between a database and a flat file, it is all bytes on a disk/memory in the end. So it is mostly a question of balancing the roles of the hardware, OS, and application software. For example, if the reason you are using a database is that it does a particularly good job at limiting disk IO, then it may not be nec…

I think that the direction is moving torward SQLite which is a flat file database that has the most installs from apps in general.

A flat file database? It is a single file. This means you need specialized tools to read it, and if you want to sync it a lot of cloud storage solutions will force you to resync every change. This is why a transparent file system db is great for integration into native storage. Databases don't really optimize for the file system layer. I am hopeful one-day we'll have an auto organizing flat file system that auto sorts and sections data in a predictable manner.

Re: Smaller is better – The rise, fall, and rise of flat file software

#92

Post by cortesoft »

pixl97 wrote: post #62
DeathArrow wrote: post #53

Earlier quoted context omitted.

A database can be relational, it can make some guarantees, it can pass Acid. With a file you are responsible for performance, reliability, integrity and security of the data. Do you believe that you can do a better job than those tens of guys who contributed to a DBMS? Go ahead. Do you not need reliability, integrity, security, performance? Go ahead. So use a file to store data instead of database but know the trade-…

Portability to other systems.

A SQL dump is incredibly portable.

Re: Smaller is better – The rise, fall, and rise of flat file software

#93

Post by chme »

stjohnswarts wrote: post #84
Blackthorn wrote: post #78

Earlier quoted context omitted.

> I wish Fusion360 for example had a sanely git-able on-disk format like: There are two standards for this common to CAD, called IGES and STEP. They are both plaintext, though their gittability is questionable.

They are structured text though so I don't think you can just do deltas like like with code. You'd have to treat them like binaries would you not? That's the way we always kept our XML files in git forced to be binaries.

I would really like to see smudge/clean like-filters for git which convert between git objects and work dir files in deal with structured data or binary files more efficiently. So that diffs are minimal and readable even with those formats.

Re: Smaller is better – The rise, fall, and rise of flat file software

#95

Post by cultofmetatron »

sqlite seems better here. text files need to be parsed and loaded. fsOpen has sifficient overhead to be noticable when you use it a lot. unless you're statically pregenerating the html, sqlite is a super fast format with good library support and will give you the fastest access to the disk and its still a flat file.

Re: Smaller is better – The rise, fall, and rise of flat file software

#96

Post by The_Colonel »

pixl97 wrote: post #62
DeathArrow wrote: post #53

Earlier quoted context omitted.

A database can be relational, it can make some guarantees, it can pass Acid. With a file you are responsible for performance, reliability, integrity and security of the data. Do you believe that you can do a better job than those tens of guys who contributed to a DBMS? Go ahead. Do you not need reliability, integrity, security, performance? Go ahead. So use a file to store data instead of database but know the trade-…

Portability to other systems.

Which platform do you miss, which SQLite does not support?

Re: Smaller is better – The rise, fall, and rise of flat file software

#97

Post by throwawayboise »

DeathArrow wrote: post #53
GuB-42 wrote: post #30

Maybe it is just a reflection on the state of filesystems vs databases. There is no fundamental difference between a database and a flat file, it is all bytes on a disk/memory in the end. So it is mostly a question of balancing the roles of the hardware, OS, and application software. For example, if the reason you are using a database is that it does a particularly good job at limiting disk IO, then it may not be nec…

A database can be relational, it can make some guarantees, it can pass Acid. With a file you are responsible for performance, reliability, integrity and security of the data. Do you believe that you can do a better job than those tens of guys who contributed to a DBMS? Go ahead. Do you not need reliability, integrity, security, performance? Go ahead. So use a file to store data instead of database but know the trade-…

Yeah, none of that may matter for a "brochure" type of website, or a blog where you are the only person updating it.

As soon as you start processing forms and doing anything transactional based on that, you'll be quickly reinventing a lot of wheels if you aren't using a DBMS of some sort.

Re: Smaller is better – The rise, fall, and rise of flat file software

#98

Post by The_Colonel »

juangacovas wrote: post #54

I think DokuWiki also deserves a mention for being pretty flat-file with a lot of features and extensions

I would argue that DokuWiki actually implements its own little custom database, with its metadata, metadata indexes, fulltext indexes etc., adding complexity and doing it probably in a shittier way than a "real database" would do.

Re: Smaller is better – The rise, fall, and rise of flat file software

#99

Post by Mehdi2277 »

mkleczek wrote: post #77
GuB-42 wrote: post #30

Maybe it is just a reflection on the state of filesystems vs databases. There is no fundamental difference between a database and a flat file, it is all bytes on a disk/memory in the end. So it is mostly a question of balancing the roles of the hardware, OS, and application software. For example, if the reason you are using a database is that it does a particularly good job at limiting disk IO, then it may not be nec…

This opinion is quite popular nowadays but IMHO it is caused by misunderstanding what a database and database management system is and what it is for. DBMS (and in particular) RDBMS is not a merely persistence medium for an application - DBMS main role is to _share_ data between applications.

I don’t understand this as distinction with file systems. Networked file systems exist and I have a lot of production code that shares data as files in either some networked file/blob system. Often databases even just store a reference to a file path for stuff like images/other large artifacts that are shared from some file system.

Re: Smaller is better – The rise, fall, and rise of flat file software

#100

Post by zokier »

stjohnswarts wrote: post #84
Blackthorn wrote: post #78

Earlier quoted context omitted.

> I wish Fusion360 for example had a sanely git-able on-disk format like: There are two standards for this common to CAD, called IGES and STEP. They are both plaintext, though their gittability is questionable.

They are structured text though so I don't think you can just do deltas like like with code. You'd have to treat them like binaries would you not? That's the way we always kept our XML files in git forced to be binaries.

Code is (generally) structured too?
Post reply on HN