Would be interested if the final one-liner would be faster if updated to use the rust equivalent utils like https://github.com/ezrosent/frawk and https://github.com/sharkdp/fd
Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
21–30 of 169 posts
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#22If you don't have to do any complex sorting or grouping, yes a simple script works way better. It doesn't have the overhead of using the scheduler, or distributing the data into chunks on many servers. Also consider using sqlite, postgres, or your EDW if you have one. Tools like CSVkit and XSV are useful for preprocessing, exploration.
I've seen many ETL scripts written where a simple SQL statement would have been better. SQL queries tend to work after a few queries have been verified to be correct, ETL jobs in languages like java can dump mysterious stack traces referencing many frameworks breaking due to data issues, memory issues, or unhandled cases.
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#23This should be tagged (2014). This article has made the rounds many times. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Outside of legacy systems, Hadoop isn't widely used anymore.
Maybe we need a reminder every now and then that text-files are quite capable and except from being faster to parse in a lot of situation has the benefits of being "future safe" and easy to backup and compress as well.
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#24Crontab for scheduling + BASH (or NodeJS/Python/Lua) scripts for map-reduce or whatever kind of computation you want to do + NFS v4.1 (something like AWS EFS) will do the job. You get immense flexibility, however the team of developer who is maintaining this must be more skilful.
I hate the failure semantics of cron. Unreasonably hard to get right or to not run duplicate jobs. I’ve replaced it with systemd timers and never looked back.
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#25Earlier quoted context omitted.
Aside from Apache Spark, what's replaced it and does it still face the same speed of access limitations compared to just zipping through giant CSVs with awk or whatever streaming APIs you write with your own preferred language?
I use Spark for a number of jobs for language-specific features still but I think within 2 years all custom code will be trivially invoked as native UDFs in SQL data warehouses (ie Snowflake, which has essentially solved big-data performance as a going concern). I just write SQL in Snowflake and it replaces 95% of what I would otherwise have done in custom MapReduce or Spark code.
SQL will always be faster than Hadoop and MapReduce. The main reason to use those other slower services is developer are not use to SQL or declarative programming, and insist on having the code in Procedural way.
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#26If you don't have to do any complex sorting or grouping, yes a simple script works way better. It doesn't have the overhead of using the scheduler, or distributing the data into chunks on many servers. Also consider using sqlite, postgres, or your EDW if you have one. Tools like CSVkit and XSV are useful for preprocessing, exploration. I've seen many ETL scripts written where a simple SQL statement would have been be…
As the predictable discussion unfolds below (above probably?), using the best subset you can means your analytics workloads can potentially scale from sqlite to bigquery, going through spark or over to clickhouse.
SQL is everywhere and it is fast and kinda portable. I have done sqlite analytics jobs over 4-6GB databases on a desktop machine 10 years ago to generate really complex reports. It worked wonderfully and was the shortest time from raw data (xml, html, csvs) to useful results.
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#27Crontab for scheduling + BASH (or NodeJS/Python/Lua) scripts for map-reduce or whatever kind of computation you want to do + NFS v4.1 (something like AWS EFS) will do the job. You get immense flexibility, however the team of developer who is maintaining this must be more skilful.
NFS is such a pain in the ass for this use case. If the client crashes, everything’s corrupted. If the server crashes, everything’s corrupted. You can’t even atomically write 4k with O_APPEND. Why anyone would choose to build distributed systems your way is beyond me
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#28Earlier quoted context omitted.
But I can write SQL in Spark just fine. Can't I?
You can write some SQL in Spark, but 1) Why would you want to maintain your own Spark infrastructure? Spark on Kube is a huge improvement over YARN but you still have to deal with OOMEs, filled disks, Kube upgrades, pushing custom images to container registries, etc etc etc. 2) Snowflake is probably 10-50x as performant as Spark for data manipulation. I don't know what kind of unholy demonic incantations Snowflake is…
A lot of articles I read about snowflake involves data vault which is a massive turn off. And when their tech lead (Kent Graziano) is a prominent figure in the DV bullshit...
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#29Can someone explain to me why it took 26 minutes for hadoop? How can 7 machines be that many times slower than a single machine?
Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
#30I went to a Hadoop workshop in 2016 where the speaker was insistent Hadoop would replace traditional relational databases in the next 5 years. It’s been six and I think the death of relational databases has still been greatly exaggerated. https://twitter.com/donatj/status/740210538320273408
Having done OLAP and used other DB models like ISAM and xBase, I am a firm beliver that relational databases will keep being the horse worth betting on.
Whatever else keeps poping up against them are only usefull in special use cases, not needed for 90% of the common use cases, and even then, it isn't like relational database vendors are frozen in time without improving them.