uncle roger voice just use SQLite haiyaaaaa
This made me giggle
Smaller is better – The rise, fall, and rise of flat file software
41–50 of 152 posts
Re: Smaller is better – The rise, fall, and rise of flat file software
#42Earlier quoted context omitted.
Some databases also have the ability to be instantly portable, like a sqlite file. You can tar up your flat files as well, but that is a separate process and can be time consuming for larger directories. Perhaps there is a use case for a file system backed by portable database like sqlite, such that you can just mount the volume and send it around without pain, but can prefer to use normal command line utils (ls, rm,…
If you're running Postgres in Docker, you can just stop the container and tar up the data volume and start it somewhere else (This also works without Docker of course, but Docker makes me feel safer when I do stuff like this)
This procedure is officially documented (1).
Since we're talking about copying a folder, I'm not sure why you'd feel safer doing it via an additional layer of abstraction.
Re: Smaller is better – The rise, fall, and rise of flat file software
#43Earlier quoted context omitted.
Some databases also have the ability to be instantly portable, like a sqlite file. You can tar up your flat files as well, but that is a separate process and can be time consuming for larger directories. Perhaps there is a use case for a file system backed by portable database like sqlite, such that you can just mount the volume and send it around without pain, but can prefer to use normal command line utils (ls, rm,…
If you're running Postgres in Docker, you can just stop the container and tar up the data volume and start it somewhere else (This also works without Docker of course, but Docker makes me feel safer when I do stuff like this)
Why does Docker makes you feel safer here? Is it because you feel more confident there is no database process still running?
Re: Smaller is better – The rise, fall, and rise of flat file software
#44Is a flat file usually considered text, or can it be a binary format as well? Different websites give different answers.
It can be binary. In fact, the origin of flat files goes back to early data processing on mainframes. At that time there was no databases (as we know it today) yet. It was a collection of records stored on punched cards, and loaded on tape (later disks) for processing sequentially. Records were not just text; they were a mix of binary, text, and BCD encoding.
Re: Smaller is better – The rise, fall, and rise of flat file software
#45I remember when I got started with webdev around 2002, a lot was built on flat files. And so did I, because that’s how my “internet mentor” did it: guestbooks, bulletin boards, mailing lists, shoutboxes, CMSs. All in flat files, except I used php and not Perl like him.
I also did some Perl and that was super common. I also remember most 90s non-web software also using flat files. Even enterprise stuff, even when they had centralized storage in a file server. Sometimes they had databases but those were also accessed directly in the disk, sqlite-style. I remember seeing Clipper, FoxPro and MS Access being used in this manner. After Perl I graduated to doing web stuff with a Microsoft…
I remember scriptarchive, which I’ve browsed quite a lot, but it wasnt’t Matt. It was a german guy who had a similiar script collection, some of which he sold online. He reviewed and even tested my scripts and helped me along when I got stuck on incorrect cgi-bin file permissions or syntax errors and stuff like that when I was just learning things. Invaluable.
Re: Smaller is better – The rise, fall, and rise of flat file software
#46Maybe it is just a reflection on the state of filesystems vs databases. There is no fundamental difference between a database and a flat file, it is all bytes on a disk/memory in the end. So it is mostly a question of balancing the roles of the hardware, OS, and application software. For example, if the reason you are using a database is that it does a particularly good job at limiting disk IO, then it may not be nec…
Some databases also have the ability to be instantly portable, like a sqlite file. You can tar up your flat files as well, but that is a separate process and can be time consuming for larger directories. Perhaps there is a use case for a file system backed by portable database like sqlite, such that you can just mount the volume and send it around without pain, but can prefer to use normal command line utils (ls, rm,…
I believe hdf5 is the format that can serve that purpose.
Re: Smaller is better – The rise, fall, and rise of flat file software
#47I remember when I got started with webdev around 2002, a lot was built on flat files. And so did I, because that’s how my “internet mentor” did it: guestbooks, bulletin boards, mailing lists, shoutboxes, CMSs. All in flat files, except I used php and not Perl like him.
I started with php and msql back in ... 1996. Then in 98 took a job and much was perl. This was a web agency, and there was one client running Oracle on a sparc (IIRC), and some folks were tested out 'asp' and using MSSQL 6.5 (again IIRC - 7 wasn't out until later). But for those of us using perl... we were forced in to flat files and/or dbm files for our data storage needs.
I tried to ask for msql or mysql, but were told 'no', that they were 'too heavy' for what we needed, so the folks doing perl were pushed back to the dark ages. We also weren't allowed to use 'shtml' - basic server side processing for things like 'include' to include common footers. Whenever we needed to make a footer change we had to write multiple search/replace scripts across dozens of client sites. The new 'asp' folks lauded this over us for a while with "look how advanced asp and Microsoft is - MS really gets the web". I showed what I'd been doing with PHP for 2-3 years at that point, with includes and more. But "well, that's not really powerful enough". Kept moving the goalposts.
tldr - there were many options for more advanced stuff than flat files back in the earliest web days, but often people were hamstrung by short sighted tech decisions made by people who were not responsible for actually delivering the work. has much changed in the past 25+ years? ;)
Re: Smaller is better – The rise, fall, and rise of flat file software
#48Is sqlite a flat file disqualifier?
I don’t have a dog in the fight, but I would think any real db would be a disqualified in the spirit of flat files. But, SQLite honestly seems like the best of both worlds to me.
Re: Smaller is better – The rise, fall, and rise of flat file software
#49Maybe it is just a reflection on the state of filesystems vs databases. There is no fundamental difference between a database and a flat file, it is all bytes on a disk/memory in the end. So it is mostly a question of balancing the roles of the hardware, OS, and application software. For example, if the reason you are using a database is that it does a particularly good job at limiting disk IO, then it may not be nec…
> There is no fundamental difference between a database and a flat file, it is all bytes on a disk/memory in the end.
A database has a whole engine, which I'd argue is a pretty fundamental difference.
I think I get what you were trying to say, though: there's no fundamental difference between database tables and flat files, and there's no fundamental difference between a database and a filesystem.
Re: Smaller is better – The rise, fall, and rise of flat file software
#50Maybe it is just a reflection on the state of filesystems vs databases. There is no fundamental difference between a database and a flat file, it is all bytes on a disk/memory in the end. So it is mostly a question of balancing the roles of the hardware, OS, and application software. For example, if the reason you are using a database is that it does a particularly good job at limiting disk IO, then it may not be nec…
For me it's about proprietary vs. readable - unless it's obviously necessary for performance I think I'd still prefer 'flat file' to sqlite, but basically I just want something that isn't an incomprehensible blob, that I can use with other tools etc.
Bonus points for some open standard, but some kind of understandable format even if proprietary is miles ahead of proprietary and cryptic.
I wish Fusion360 for example had a sanely git-able on-disk format like:
line 0xdeadbeef (0,0) - (123,456)
line 0x01234567
constraint parallel 0xdeadbeef 0x01234567
(Actually I suppose git - and incremental backups - is the main reason I say I'd still prefer plaintext to db.)