Readings in Database Systems, 5th Edition (2015) • Hacker News Board

Re: Readings in Database Systems, 5th Edition (2015)

#31

Post by andrewflnr » Sun, Jun 14, 2020, 3:42 PM UTC

dinosaurdynasty wrote: post #24

exdsq wrote: post #23

Earlier quoted context omitted.

> The community of programmers has a love affair with “the next shiny object”. This is likely to create “churn” in your organization, as the “half-life” of shiny objects may be quite short." This is an interesting thought. A company uses shiny tech because programmers like using them for whatever reason. This attracts employees who want to use this tech too. The half-life for shiny tech is short and so these develope…

I'm pretty sure it's the raises people tend to get jumping companies compared to what they get if they stay at a company.

If anything I would expect the causality to run the other direction, i.e. resume driven development to make sure they can get a new job and therefore a raise.

Re: Readings in Database Systems, 5th Edition (2015)

#32

Post by ramraj07 » Sun, Jun 14, 2020, 3:55 PM UTC

sradman wrote: post #26

pixelmonkey wrote: post #3

Michael Stonebraker has an interesting set of conclusions in his assessment of the MapReduce vendor market in 2015 from the "Dataflow" chapter here: "- Just because Google thinks something is a good idea does not mean you should adopt it. - Disbelieve all marketing spin, and figure out what benefit any given product actually has. This should be especially applied to performance claims. - The community of programmers…

I reread DeWitt and Stonebraker’s (D&S) MapReduce criticism [1] and I still find it misguided 12 years later. Map() is not equivalent to a SQL GROUP BY clause, it is equivalent to a user-defined Table Function that is used in a FROM clause. This mimics the Extract and Transform stages in a SQL ETL pipeline. The Extract is implied by the input format. The Reduce() is very much equivalent to a user-defined Aggregate Fu…

When you say cloud-native data warehouses do you mean things like snowflake/redshift/big-query or something else? As part of an org making the transition from spark to these I can definitely agree that these tools are better suited for practical data engineering in the medium-big-data scale (anything not Google/Facebook)

Re: Readings in Database Systems, 5th Edition (2015)

#33

Post by sradman » Sun, Jun 14, 2020, 5:54 PM UTC

ramraj07 wrote: post #32

sradman wrote: post #26

Earlier quoted context omitted.

I reread DeWitt and Stonebraker’s (D&S) MapReduce criticism [1] and I still find it misguided 12 years later. Map() is not equivalent to a SQL GROUP BY clause, it is equivalent to a user-defined Table Function that is used in a FROM clause. This mimics the Extract and Transform stages in a SQL ETL pipeline. The Extract is implied by the input format. The Reduce() is very much equivalent to a user-defined Aggregate Fu…

When you say cloud-native data warehouses do you mean things like snowflake/redshift/big-query or something else? As part of an org making the transition from spark to these I can definitely agree that these tools are better suited for practical data engineering in the medium-big-data scale (anything not Google/Facebook)

I was thinking AWS Athena (Presto) for the data warehouse and AWS Glue (Spark) for ETL. Redshift has always had the feel of a Column Store Appliance that runs side-by-side with your other IaaS resources. There is nothing particularly cloud-native about it other than the way it is provisioned and managed in the AWS web Console. Amazon QuickSight seems like an excellent alternative to Enterprise BI pivot tables like Tableau, Excel, PowerPivot, Business Objects, and Cognos. Amazon seems to be ahead of the competition (again) when it comes to ETL/DW/BI-as-a-Service, at least in terms of price-per-performance.

I don't know anything about Snowflake. SQL makes BigQuery and Hive easier to program than MapReduce/Pig but I don't think of these technologies as data warehouses.

Column Stores (compressed bitmap indexes batch updated with an ETL-like process) make exceptional data warehouses. Row oriented data warehouses all feel like anachronisms now.

Re: Readings in Database Systems, 5th Edition (2015)

#34

Post by sradman » Sun, Jun 14, 2020, 6:04 PM UTC

mrits wrote: post #30

sradman wrote: post #26

Earlier quoted context omitted.

I reread DeWitt and Stonebraker’s (D&S) MapReduce criticism [1] and I still find it misguided 12 years later. Map() is not equivalent to a SQL GROUP BY clause, it is equivalent to a user-defined Table Function that is used in a FROM clause. This mimics the Extract and Transform stages in a SQL ETL pipeline. The Extract is implied by the input format. The Reduce() is very much equivalent to a user-defined Aggregate Fu…

"Map() is not equivalent to a SQL GROUP BY clause, it is equivalent to a user-defined Table Function that is used in a FROM clause." No, the projection doesn't remove redundancy under most cases. There also isn't any reason you couldn't have UDF's in the GROUP BY clause. I've written implementations of both and I think the GROUP BY is an excellent comparison for understanding Map in MapReduce Systems.

> the projection doesn't remove redundancy under most cases

Maybe I'm missing something; I don't understand why projections are part of this discussion. Maybe I should have been more precise. I was thinking about the type of Table UDF that Aster Data made popular around the time DeWitt and Stonebraker wrote their article (Jan 2008). These Table UDFs were written in languages like Java or C/C++ and generally accessed data external to the database engine. Aster Data marketing defined the functionality in terms of MapReduce.

The point I was trying to make was that the Map() part of MapReduce is equivalent to a distributed ETL pipeline. This remains one of the key use cases for Spark. The Reduce() part is no longer relevant in the new world of cheap and scalable column stores. DeWitt and Stonebraker's Teradata-like enterprise data warehouses suffered the same fate.