Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things to know about databases

architecturenotes.co

Post reply on HN
Print view View original

161–170 of 247 posts

Re: Things to know about databases

#161

Post by SulphurSmell »

YetAnotherNick wrote: post #143
SulphurSmell wrote: post #61

This article is informative. I have found that databases in general tend to be less sexy than the front-end apps...especially with the recent cohort of devs. As an old bastard, I would pass on one thing: Realize that any reasonably used database will likely outlast the applications leveraging it. This is especially true the bigger it gets, and the longer it stays in production. That said, if you are influencing the d…

> I have found that databases in general tend to be less sexy than the front-end apps I don't know if there is a single soul who believes this. If you are designing a database, it is much more cooler than front end apps.

I think they are wonderful (from the Codd and Date days...) but mostly everyone else disagrees.

Re: Things to know about databases

#162

Post by SulphurSmell »

mmcnl wrote: post #130
SulphurSmell wrote: post #61

This article is informative. I have found that databases in general tend to be less sexy than the front-end apps...especially with the recent cohort of devs. As an old bastard, I would pass on one thing: Realize that any reasonably used database will likely outlast the applications leveraging it. This is especially true the bigger it gets, and the longer it stays in production. That said, if you are influencing the d…

"Show me your flowchart and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowchart; it'll be obvious." -- Fred Brooks, The Mythical Man Month (1975)

This is going on my wall. Thanks so much.

Re: Things to know about databases

#163

Post by qaq »

emerongi wrote: post #52
w0m wrote: post #37

Earlier quoted context omitted.

Is 'Not performance bound, and dot knowing the future shape of your data' a valid reason? Less overhead on initial rollout to just Toss it up there. > choosing something else is rarely the right decision I think this is a little bit of a 'We always did it this way' statement.

With Postgres, you can always just have a JSONB column for data whose shape you're unsure of. Personally, I'd rather start with Postgres and dump data into there and retain the powers of RDBMS for the future, rather than the other way around and end up finding out that I really would like to have features that come out of the box with relational databases. I think a valid reason for not choosing a relational database…

No longer an issue with things like Spanner, CockroachDB etc

Re: Things to know about databases

#164

Post by qaq »

VirusNewbie wrote: post #87
tiffanyh wrote: post #23

#1 thing you should know, RDBMS can solve pretty much every data storage/retrieval problem you have. If you're choosing something other than an RDBMS - you should rethink why. Because unless you're at massive scale (which still doesn't justify it), choosing something else is rarely the right decision.

It is very frustrating to work with engineers who don't understand the nuances of RDBMS and assume they can solve all the things. The small company I work for has 3B rows. We have a high write volume. Can you use an RDBMs database to solve this? Sure, but it would be a terrible waste of engineering effort.

? as opposed to buggy reimplementation of subset of RDBMS functionality on the Application side?

Re: Things to know about databases

#165

Post by pixelmonkey »

lysecret wrote: post #30
yla92 wrote: post #3

Great post. Also highly recommend Designing Data-Intensive Applications by Martin Kleppmann ( https://www.amazon.com/Designing-Data-Intensive-Applications... ). The sections on "Storage and Retrieval", "Replication", "Partitioning" and "Transactions" really opened up my eyes!

Absolutely loved the book. Can someone recommend similar books?

There is a quite-nice interactive browser dataviz here that shows you books similar to the themes, categories, and topics discussed in DDIA:

https://anvaka.github.io/greview/ddia/1/

Re: Things to know about databases

#166

Post by electricshampo1 »

hashmash wrote: post #139
jandrewrogers wrote: post #125

Earlier quoted context omitted.

You are understating the limitations of B+trees for real workloads. A common and growing problem is the lack of online indexing that scales, the particularly data model doesn't matter that much. Index construction throughput and scaling has been a serious problem at some pretty boring companies I've done work for. Use of B+trees in new database kernels has definitely diminished. I'm not counting the installed base of…

Can you provide an explicitly named "modern" database that doesn't use B+tree indexes, and what specifically does it use instead?

" Like many modern analytical engines [18, 20], Procella does not use the conventional BTree style secondary indexes, opting instead for light weight secondary structures such as zone maps, bitmaps, bloom filters, partition and sort keys [1]. The metadata server serves this information during query planning time. These secondary structures are collected partly from the file headers during file registration, by the registration server, and partly lazily at query evaluation time by the data server. Schemas, table to file mapping, stats, zone maps and other metadata are mostly stored in the metadata store (in Bigtable [11] and Spanner [12])."

https://storage.googleapis.com/pub-tools-public-publication-...

Re: Things to know about databases

#167

Post by zippergz »

SulphurSmell wrote: post #61

This article is informative. I have found that databases in general tend to be less sexy than the front-end apps...especially with the recent cohort of devs. As an old bastard, I would pass on one thing: Realize that any reasonably used database will likely outlast the applications leveraging it. This is especially true the bigger it gets, and the longer it stays in production. That said, if you are influencing the d…

This is one reason that ORMs which wish to own the database schema make me uncomfortable. How much fun is that schema going to be years down the road when that ORM is out of fashion, but you still need an app working with that data? Some are better than others at doing things in a sane way, of course.

Re: Things to know about databases

#168

Post by pfarrell »

robonerd wrote: post #114
brightball wrote: post #91

Earlier quoted context omitted.

I've honestly never understood why people have such a distaste for SQL. SQL and Linux/Unix have been the biggest constants of my entire programming career to this point (20ish years). I always know I can count on them.

I blame ORMs. ORMs are promoted by scare mongering novice developers away from learning SQL in the first place. I'm ashamed to say I fell for it for a few years. When I eventually learned SQL it was like a fog being lifted from my mind that I hadn't even noticed before.

The irony is, to effectively use an ORM, you need to be able to debug the SQL it generates when it’s not performing correctly (eg operating on a collection, row by row). I like to use ORM for row based, transactional operations in an OLTP. But usually look for the way to write straight SQL when doing OLAP style reporting stuff. Both of which happen in almost every business.

Re: Things to know about databases

#169

Post by hashmash »

electricshampo1 wrote: post #166
hashmash wrote: post #139

Earlier quoted context omitted.

Can you provide an explicitly named "modern" database that doesn't use B+tree indexes, and what specifically does it use instead?

" Like many modern analytical engines [18, 20], Procella does not use the conventional BTree style secondary indexes, opting instead for light weight secondary structures such as zone maps, bitmaps, bloom filters, partition and sort keys [1]. The metadata server serves this information during query planning time. These secondary structures are collected partly from the file headers during file registration, by the re…

> modern analytical engines

What about relational databases? Most are best suited for OLTP workloads.

Re: Things to know about databases

#170

Post by ithrow »

eurasiantiger wrote: post #120
brightball wrote: post #91

Earlier quoted context omitted.

I've honestly never understood why people have such a distaste for SQL. SQL and Linux/Unix have been the biggest constants of my entire programming career to this point (20ish years). I always know I can count on them.

Can you write an SQL query to return arbitrary JSON? Returning queried data in a nested form is a must nowadays.

Make more than a single query and nest on the application layer.
Post reply on HN