Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Readings in Database Systems, 5th Edition (2015)

redbook.io

Post reply on HN
Print view View original

21–30 of 34 posts

Re: Readings in Database Systems, 5th Edition (2015)

#21

Post by cbcoutinho »

throwaway_pdp09 wrote: post #16
jrumbut wrote: post #7

Earlier quoted context omitted.

I think it's a bit of a shame that the MapReduce concept got the shiny object treatment since I thought it was a nice pragmatic approach to a useful set of problems that are faced all the time and often addressed with ad-hoc programs that make a mess. People always looked down on those that used Hadoop or somesuch for Now many places are back in the world of one-off scripts, and I think something of value was lost (e…

> People always looked down on those that used Hadoop or somesuch for What 'structure'? Why is it so important that it makes it worthwhile firing up a large, complex framework? I'm beyond baffled.

The same 'structure' that makes it easy to onboard new co-workers because they've seen the same project 'structure' before in the past. In that sense, the bottleneck in an organization is getting people productive as fast as possible, even that means using a cleaver instead of a scalpel.

Re: Readings in Database Systems, 5th Edition (2015)

#22

Post by throwaway_pdp09 »

cbcoutinho wrote: post #21
throwaway_pdp09 wrote: post #16

Earlier quoted context omitted.

> People always looked down on those that used Hadoop or somesuch for What 'structure'? Why is it so important that it makes it worthwhile firing up a large, complex framework? I'm beyond baffled.

The same 'structure' that makes it easy to onboard new co-workers because they've seen the same project 'structure' before in the past. In that sense, the bottleneck in an organization is getting people productive as fast as possible, even that means using a cleaver instead of a scalpel.

If all they can use is a massive cleaver (big data tools), and have no experience with scalpels (small, sharp, cheap and fast data tools), IMO your company has a serious, fundamental and systemic problem (no, let's call it failure) towards employee experience, training and knowledge. Edit: and resource management.

Re: Readings in Database Systems, 5th Edition (2015)

#23

Post by exdsq »

pixelmonkey wrote: post #3

Michael Stonebraker has an interesting set of conclusions in his assessment of the MapReduce vendor market in 2015 from the "Dataflow" chapter here: "- Just because Google thinks something is a good idea does not mean you should adopt it. - Disbelieve all marketing spin, and figure out what benefit any given product actually has. This should be especially applied to performance claims. - The community of programmers…

> The community of programmers has a love affair with “the next shiny object”. This is likely to create “churn” in your organization, as the “half-life” of shiny objects may be quite short."

This is an interesting thought. A company uses shiny tech because programmers like using them for whatever reason. This attracts employees who want to use this tech too. The half-life for shiny tech is short and so these developers move on to shinier pastures. I wonder if this explains why people change jobs so often in tech? I’m sure I read the average tenure is much lower (~1.5years) compared to other industries.

Re: Readings in Database Systems, 5th Edition (2015)

#24

Post by dinosaurdynasty »

exdsq wrote: post #23
pixelmonkey wrote: post #3

Michael Stonebraker has an interesting set of conclusions in his assessment of the MapReduce vendor market in 2015 from the "Dataflow" chapter here: "- Just because Google thinks something is a good idea does not mean you should adopt it. - Disbelieve all marketing spin, and figure out what benefit any given product actually has. This should be especially applied to performance claims. - The community of programmers…

> The community of programmers has a love affair with “the next shiny object”. This is likely to create “churn” in your organization, as the “half-life” of shiny objects may be quite short." This is an interesting thought. A company uses shiny tech because programmers like using them for whatever reason. This attracts employees who want to use this tech too. The half-life for shiny tech is short and so these develope…

I'm pretty sure it's the raises people tend to get jumping companies compared to what they get if they stay at a company.

Re: Readings in Database Systems, 5th Edition (2015)

#25

Post by ses1984 »

o1lab wrote: post #18
jrumbut wrote: post #7

Earlier quoted context omitted.

I think it's a bit of a shame that the MapReduce concept got the shiny object treatment since I thought it was a nice pragmatic approach to a useful set of problems that are faced all the time and often addressed with ad-hoc programs that make a mess. People always looked down on those that used Hadoop or somesuch for Now many places are back in the world of one-off scripts, and I think something of value was lost (e…

>> (even if it was a little ridiculous to fire up a cluster for something Excel or SQLite could handle) I know above comment will be lost - but this is such a genuine truth.

I'm sorry but if your problem can be solved on excel then hire people who are good with excel, not people good with Hadoop.

Re: Readings in Database Systems, 5th Edition (2015)

#26

Post by sradman »

pixelmonkey wrote: post #3

Michael Stonebraker has an interesting set of conclusions in his assessment of the MapReduce vendor market in 2015 from the "Dataflow" chapter here: "- Just because Google thinks something is a good idea does not mean you should adopt it. - Disbelieve all marketing spin, and figure out what benefit any given product actually has. This should be especially applied to performance claims. - The community of programmers…

I reread DeWitt and Stonebraker’s (D&S) MapReduce criticism [1] and I still find it misguided 12 years later.

Map() is not equivalent to a SQL GROUP BY clause, it is equivalent to a user-defined Table Function that is used in a FROM clause. This mimics the Extract and Transform stages in a SQL ETL pipeline. The Extract is implied by the input format.

The Reduce() is very much equivalent to a user-defined Aggregate Function. D&S accurately criticize the sub-optimal materialization of intermediate data sets but they under appreciate the implicit input split and distributed sorting mechanism which dominated the Terasort benchmark at the time (a Jim Gray creation).

On-Premise commodity Hadoop clusters lost out to public Infrastructure-as-a-Service clusters. None of the five takedown categories turned out to be important. The tools have evolved and cloud-native data warehouses and ETL systems are now the best of both worlds.

[1] https://homes.cs.washington.edu/~billhowe/mapreduce_a_major_...

Re: Readings in Database Systems, 5th Edition (2015)

#27

Post by omginternets »

oftenwrong wrote: post #20
omginternets wrote: post #13

Relatedly: I’ve been trying to wrap my head around MVCC (I’d like to write my own implementation). Any recommendations for a thorough overview of the subject?

https://vladmihalcea.com/how-does-mvcc-multi-version-concurr... http://www.interdb.jp/pg/pgsql05.html

Thanks! The second link is really excellent (as is the first, but I've already read it).

I also found this paper in the refs, which seems really good. [0]

[0] https://drkp.net/papers/ssi-vldb12.pdf

Re: Readings in Database Systems, 5th Edition (2015)

#28

Post by MaysonL »

ses1984 wrote: post #25
o1lab wrote: post #18

Earlier quoted context omitted.

>> (even if it was a little ridiculous to fire up a cluster for something Excel or SQLite could handle) I know above comment will be lost - but this is such a genuine truth.

I'm sorry but if your problem can be solved on excel then hire people who are good with excel, not people good with Hadoop.

Things tend to evolve - a system that worked on excel yesterday may well be a dangerous un maintainable monster next week.

Re: Readings in Database Systems, 5th Edition (2015)

#29

Post by _jal »

throwaway_pdp09 wrote: post #22
cbcoutinho wrote: post #21

Earlier quoted context omitted.

The same 'structure' that makes it easy to onboard new co-workers because they've seen the same project 'structure' before in the past. In that sense, the bottleneck in an organization is getting people productive as fast as possible, even that means using a cleaver instead of a scalpel.

If all they can use is a massive cleaver (big data tools), and have no experience with scalpels (small, sharp, cheap and fast data tools), IMO your company has a serious, fundamental and systemic problem (no, let's call it failure ) towards employee experience, training and knowledge. Edit: and resource management.

Seems to be a sort of inverse of the massive spreadsheets that run supply chains on accretions of spaghetti-macros.

But, a tree chipper can serve as a paper shredder, and I imagine a lot of shops in certain markets saw it as a sort of prestige asset around 5-8 years back, when a bunch of companies started hiring data scientists for no apparent rational reason.

(Not bashing data scientists or data companies. Just remembering the fad that went around Bay Area companies a while ago.)

Re: Readings in Database Systems, 5th Edition (2015)

#30

Post by mrits »

sradman wrote: post #26
pixelmonkey wrote: post #3

Michael Stonebraker has an interesting set of conclusions in his assessment of the MapReduce vendor market in 2015 from the "Dataflow" chapter here: "- Just because Google thinks something is a good idea does not mean you should adopt it. - Disbelieve all marketing spin, and figure out what benefit any given product actually has. This should be especially applied to performance claims. - The community of programmers…

I reread DeWitt and Stonebraker’s (D&S) MapReduce criticism [1] and I still find it misguided 12 years later. Map() is not equivalent to a SQL GROUP BY clause, it is equivalent to a user-defined Table Function that is used in a FROM clause. This mimics the Extract and Transform stages in a SQL ETL pipeline. The Extract is implied by the input format. The Reduce() is very much equivalent to a user-defined Aggregate Fu…

"Map() is not equivalent to a SQL GROUP BY clause, it is equivalent to a user-defined Table Function that is used in a FROM clause."

No, the projection doesn't remove redundancy under most cases. There also isn't any reason you couldn't have UDF's in the GROUP BY clause. I've written implementations of both and I think the GROUP BY is an excellent comparison for understanding Map in MapReduce Systems.

Post reply on HN