Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I love databases

medium.com

Post reply on HN
Print view View original

131–140 of 172 posts

Re: Why I love databases

#131

Post by sklogic »

SamReidHughes wrote: post #129
sklogic wrote: post #104

Earlier quoted context omitted.

> It depends on the problem domain you are working in. Unix is a pretty general purpose thing. And yet, you won't find anything in it that needs a database, nothing at all among hundreds of applications, including some fairly complex ones, like CAD/CAE tools, IDEs, compilers, etc. > I can't think of an application I've done where this wouldn't have required complex serialization and parsing. Are you doing CRUD mostly…

> Parsing?!? In ORMs and DBMSes?!? Yes. Stuff stored in files has to be parsed when you read it out of files.

And how exactly RDBMS crap could help with parsing? Mind parsing a C++ source file with your RDBMS?

Re: Why I love databases

#132

Post by sklogic »

SamReidHughes wrote: post #82
sklogic wrote: post #53

Earlier quoted context omitted.

We are using text files. How many programs in a typical Unix installation need to communicate to a database? Next to none. How many are communicating via text files? Almost all of them.

How do you randomly access variable length fields in a text file and modify them and add new ones?

Yes, web is a niche thing. Get over it.

Re: Why I love databases

#133

Post by SamReidHughes »

sklogic wrote: post #131
SamReidHughes wrote: post #129

Earlier quoted context omitted.

> Parsing?!? In ORMs and DBMSes?!? Yes. Stuff stored in files has to be parsed when you read it out of files.

And how exactly RDBMS crap could help with parsing? Mind parsing a C++ source file with your RDBMS?

The grandparent was talking about the serialization and parsing of the data that you'd be storing in flat files, not parsing of C++ files. Work that you avoid by using a DBMS.

Re: Why I love databases

#134

Post by sklogic »

SamReidHughes wrote: post #133
sklogic wrote: post #131

Earlier quoted context omitted.

And how exactly RDBMS crap could help with parsing? Mind parsing a C++ source file with your RDBMS?

The grandparent was talking about the serialization and parsing of the data that you'd be storing in flat files, not parsing of C++ files. Work that you avoid by using a DBMS.

Parsing text files is not a problem at all. Never been a problem. And there are far too many benefits of having flat text streams vs. some inaccessible stupid binary ad hoc formats hidden inside some enterprisey RDBMS guts.

Re: Why I love databases

#135

Post by SamReidHughes »

sklogic wrote: post #102
XorNot wrote: post #79

Earlier quoted context omitted.

Try doing any remotely large data handling. I got midway through writing a direct disk CSV parser for some Python data analysis recently before I realized that this problem has been solved far better by SQLite. A database is all about not reinventing the wheel for every problem. You shouldn't hand roll your own crypto, and if can avoid it you should also try not to hand roll those things other people make are already…

> Try doing any remotely large data handling. I was doing grid computing stuff in experimental particle physics. Data as large as it gets. Flat tuple storage (initially on tapes ) was all we needed, and we already have a much better way of handling tuples than anything that DBMSes could ever offer. No databases whatsoever. Only streams, no random access.

What's an example of work you've done that used database systems that could have been better done using flat files?

Re: Why I love databases

#136

Post by sklogic »

SamReidHughes wrote: post #135
sklogic wrote: post #102

Earlier quoted context omitted.

> Try doing any remotely large data handling. I was doing grid computing stuff in experimental particle physics. Data as large as it gets. Flat tuple storage (initially on tapes ) was all we needed, and we already have a much better way of handling tuples than anything that DBMSes could ever offer. No databases whatsoever. Only streams, no random access.

What's an example of work you've done that used database systems that could have been better done using flat files?

That's the point - I did not need any database systems whatsoever for nearly everything I ever done. Besides hierarchical DBMSes for CADs - but that's a totally different story, that's another area where RDBMSes fail miserably. There is no use for RDMBSes outside of the CRUD niche (which includes web, enterprise applications, and that's pretty much it).

I can't see any use for RDMBSes in the embedded world. No use for them in most of the data analysis tasks - stream storage works much better here. No RDBMSes in the system level (OSes, compilers, linkers, etc.). No RDBMSes in the scientific computing.

Re: Why I love databases

#137

Post by SamReidHughes »

sklogic wrote: post #136
SamReidHughes wrote: post #135

Earlier quoted context omitted.

What's an example of work you've done that used database systems that could have been better done using flat files?

That's the point - I did not need any database systems whatsoever for nearly everything I ever done. Besides hierarchical DBMSes for CADs - but that's a totally different story, that's another area where RDBMSes fail miserably. There is no use for RDMBSes outside of the CRUD niche (which includes web, enterprise applications, and that's pretty much it). I can't see any use for RDMBSes in the embedded world. No use fo…

Why does iOS ship with SQLite then? I used it for an iPhone app that I wrote. It was more convenient than making my own flat file format.

> No RDBMSes in the system level (OSes, compilers, linkers, etc.).

OS X has them. Any OS with file search needs some kind of indexing technology.

As do mail clients, web browsers, and a bunch of other stuff that works with data.

It's a lot easier to add fast indexing after-the-fact when you didn't design up-front to use flat files.

Re: Why I love databases

#138

Post by eropple »

sklogic wrote: post #105
eropple wrote: post #72

Earlier quoted context omitted.

Maybe you can replace a document database with the file system (though reinventing indexes for reasonable performance--which in so many use cases becomes remarkably important past a few hundreds of thousands of records--will be fun for you), but modeling relational concepts on the file system is both hard and time-consuming. And I'm comfortable saying that most nontrivial business applications are fundamentally relat…

> And I'm comfortable saying that most nontrivial business applications are fundamentally relational. Now try to assess, how many applications are supposed to be "nontrivial business applications". It's quite a niche thing, and that's where all this DBMS stuff belongs to. Please do not impose this crap on the rest of the world, we don't need your CRUD.

Why are you so randomly hostile about this? Your tone and attitude are super shitty throughout this thread. Chill.

As for CRUD--I'm a platform and devops engineer. I'm not a web developer (except incidentally). But I do often write relational things--like, say, "I have this node, running this application, using this artifact from that data store"--that are best modeled by a relational data store. They're otherwise trivial applications, but reinventing the relational data store would take orders of magnitude longer than writing the application against SQLite or Postgres. How do you propose to model relations across the file system in a way that doesn't make it the tail wagging the dog?

While I'm thinking about it, I'm pretty comfortable going further than "nontrivial business applications" to "a majority of networked applications". The world is relational. Modeling it relationally makes a lot of sense for a lot of use cases.

Re: Why I love databases

#139

Post by sklogic »

SamReidHughes wrote: post #137
sklogic wrote: post #136

Earlier quoted context omitted.

That's the point - I did not need any database systems whatsoever for nearly everything I ever done. Besides hierarchical DBMSes for CADs - but that's a totally different story, that's another area where RDBMSes fail miserably. There is no use for RDMBSes outside of the CRUD niche (which includes web, enterprise applications, and that's pretty much it). I can't see any use for RDMBSes in the embedded world. No use fo…

Why does iOS ship with SQLite then? I used it for an iPhone app that I wrote. It was more convenient than making my own flat file format. > No RDBMSes in the system level (OSes, compilers, linkers, etc.). OS X has them. Any OS with file search needs some kind of indexing technology. As do mail clients, web browsers, and a bunch of other stuff that works with data. It's a lot easier to add fast indexing after-the-fact…

All the stuff you've mentioned is pretty niche, and, as someone mentioned somewhere above in this thread, mostly done this way by the cargo cultists who defiled the sacred Unix Way. Look at a typical Unix distribution, count how many packages depend on sqlite or berkeley db - a tiny proportion of a total number.

And, no, file search and all that fancy UI stuff has absolutely nothing to do with OSes. It's an application level.

Re: Why I love databases

#140

Post by sklogic »

eropple wrote: post #138
sklogic wrote: post #105

Earlier quoted context omitted.

> And I'm comfortable saying that most nontrivial business applications are fundamentally relational. Now try to assess, how many applications are supposed to be "nontrivial business applications". It's quite a niche thing, and that's where all this DBMS stuff belongs to. Please do not impose this crap on the rest of the world, we don't need your CRUD.

Why are you so randomly hostile about this? Your tone and attitude are super shitty throughout this thread. Chill. As for CRUD--I'm a platform and devops engineer. I'm not a web developer (except incidentally). But I do often write relational things--like, say, "I have this node, running this application, using this artifact from that data store"--that are best modeled by a relational data store . They're otherwise t…

No, it's RDBMS proponents who are hostile. Their attitude is "you will be assimilated, resistance is futile, we already won, so shut up and obey". This cannot and will not be tolerated.

I was pressured many times into giving up doing things the right way and adopting RDBMSes, and that was not pleasant at all. In some cases someone else who picked up that projects tried to move to RDBMSes (from hierarchical or document-oriented storages used before) and failed miserably.

So, again, I never came across anything that would need a relational storage. Not even when I was doing heavily distributed HPC (which is as "networked" as it gets). Yes, of course I did a lot of things that are relational naturally - but all of them were in Datalog, with no storage required.

Btw., wtf is "devops"? I've heard this a number of times recently, but nobody could explain the meaning.

Post reply on HN