Can someone explain to me why it took 26 minutes for hadoop? How can 7 machines be that many times slower than a single machine?
Easy lesson from distributed systems, most of the time is spent in communication instead of actual work.
Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
31–40 of 169 posts
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#32Earlier quoted context omitted.
You should try Databricks, especially the new Photon engine powering Spark. In general more performant than Snowflake in SQL and a lot more flexible. (There are some cases in which Databricks would be slower but the perf is improving rapidly.)
Very surprised by this. Do you have a reference ?
Fyi, rxin is co-founder of databricks.
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#33Can someone explain to me why it took 26 minutes for hadoop? How can 7 machines be that many times slower than a single machine?
Easy lesson from distributed systems, most of the time is spent in communication instead of actual work.
26 minutes worth? That can't be right. Sending 250 MB (uncompressed) to 7 machines shouldn't take that long.
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#34Can someone explain to me why it took 26 minutes for hadoop? How can 7 machines be that many times slower than a single machine?
Easy lesson from distributed systems, most of the time is spent in communication instead of actual work.
This is what we've found. For a certain kind of simulation, we can process data on a single core at ~.5x line rate (I think this could be a lot faster if we took this problem seriously) so if we move to some arrangement where hosts have to download the data in order to process it, they can only use 2 cores each, and every 2 cores we want use requires another storage box to download the data from! Paying for hardware to increase "line rate" and sending zstd-compressed files over the wire might get us a factor of ~60 here, but even then we pretty rapidly run into issues where the data has to be placed on a bunch of distinct hosts in order to be useful, since 120 cores is not a lot of machines.
In the future we expect to have workloads that do less computation per piece of data, which makes the need to move our compute to our data much more acute.
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#35This should be tagged (2014). This article has made the rounds many times. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Outside of legacy systems, Hadoop isn't widely used anymore.
Aside from Apache Spark, what's replaced it and does it still face the same speed of access limitations compared to just zipping through giant CSVs with awk or whatever streaming APIs you write with your own preferred language?
Apache Flink is an alternative with "real" (not micro batching) streaming support.
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#36There is no point running something like Hadoop or Spark or Flink if your problem is not bottlenecked by the resources of of a single machine.
E.g. how would this solution work if the data is so large or the problem requires more context (looking at more than one line at the time) and therefore more memory?
Answer: it wouldn't .
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#37Earlier quoted context omitted.
I use Spark for a number of jobs for language-specific features still but I think within 2 years all custom code will be trivially invoked as native UDFs in SQL data warehouses (ie Snowflake, which has essentially solved big-data performance as a going concern). I just write SQL in Snowflake and it replaces 95% of what I would otherwise have done in custom MapReduce or Spark code.
SQL will always be faster than Hadoop and MapReduce. The main reason to use those other slower services is developer are not use to SQL or declarative programming, and insist on having the code in Procedural way.
That's completely backwards. Mapreduce-like approaches are how SQL datastores are implemented underneath; the absolute best case for SQL is to equal hand-tuned mapreduce-like performance, and often it will be slower (you're at the mercy of your query planner to pick the right indices, do joins in the right order, etc.). The main reason people use SQL is because they find it easier to express a query that way (which is completely legitimate - if your query planner is good enough most of the time, you've got better things to be doing than hand-tuning your query execution).
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#38This should be tagged (2014). This article has made the rounds many times. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Outside of legacy systems, Hadoop isn't widely used anymore.
> Outside of legacy systems, Hadoop isn't widely used anymore.
This is not true at all. Almost all the cloud providers have their own Hadoop distributions that is used a lot in many companies.
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#39This should be tagged (2014). This article has made the rounds many times. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Outside of legacy systems, Hadoop isn't widely used anymore.
Aside from Apache Spark, what's replaced it and does it still face the same speed of access limitations compared to just zipping through giant CSVs with awk or whatever streaming APIs you write with your own preferred language?
There are two kind of approaches that I see:
- skip streaming entirely and have near real time solutions using just storage + a query engine
- have streaming using message queues and lambda architecture
In both cases the goal is that your freshest data shows up on a dashboard.
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#40Can someone explain to me why it took 26 minutes for hadoop? How can 7 machines be that many times slower than a single machine?
> How can 7 machines be that many times slower than a single machine?
A mere 7 machines is almost certainly going to be slower than 1.
There was a recent post here recently titled "Latency Numbers Every Programmer Should Know": https://news.ycombinator.com/item?id=30546995
In this case the dataset was small enough to fix in my laptop's DRAM without straining it. If we assume the 7 machines are in the data centre, that means the two numbers to compare are the main memory reference (100ns) versus the Data Centre round trip time (500us). That's a factor of 5,000.
If your intuition told you those 7 machines are going to be faster, then you really should invest the time to internalise those numbers. The article is 100% correct - every programmer should know them.