Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
1–10 of 169 posts
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#2This should be tagged (2014). This article has made the rounds many times.
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
Outside of legacy systems, Hadoop isn't widely used anymore.
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#3This should be tagged (2014). This article has made the rounds many times. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Outside of legacy systems, Hadoop isn't widely used anymore.
Well, a lot of AWS and GCP infrastructure use it, but they hide it from you.
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#4This should be tagged (2014). This article has made the rounds many times. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Outside of legacy systems, Hadoop isn't widely used anymore.
Aside from Apache Spark, what's replaced it and does it still face the same speed of access limitations compared to just zipping through giant CSVs with awk or whatever streaming APIs you write with your own preferred language?
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#5Walking to the store is faster than catching a plane
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#6This should be tagged (2014). This article has made the rounds many times. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Outside of legacy systems, Hadoop isn't widely used anymore.
Aside from Apache Spark, what's replaced it and does it still face the same speed of access limitations compared to just zipping through giant CSVs with awk or whatever streaming APIs you write with your own preferred language?
I use Spark for a number of jobs for language-specific features still but I think within 2 years all custom code will be trivially invoked as native UDFs in SQL data warehouses (ie Snowflake, which has essentially solved big-data performance as a going concern).
I just write SQL in Snowflake and it replaces 95% of what I would otherwise have done in custom MapReduce or Spark code.
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#7Walking to the store is faster than catching a plane
But if you're going to keep adding destinations and going to them, an airport is a better idea.
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#8[deleted]
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#9Earlier quoted context omitted.
Aside from Apache Spark, what's replaced it and does it still face the same speed of access limitations compared to just zipping through giant CSVs with awk or whatever streaming APIs you write with your own preferred language?
I use Spark for a number of jobs for language-specific features still but I think within 2 years all custom code will be trivially invoked as native UDFs in SQL data warehouses (ie Snowflake, which has essentially solved big-data performance as a going concern). I just write SQL in Snowflake and it replaces 95% of what I would otherwise have done in custom MapReduce or Spark code.
But I can write SQL in Spark just fine. Can't I?
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#10Crontab for scheduling
+ BASH (or NodeJS/Python/Lua) scripts for map-reduce or whatever kind of computation you want to do
+ NFS v4.1 (something like AWS EFS) will do the job.
You get immense flexibility, however the team of developer who is maintaining this must be more skilful.