No, developers over-engineer because setting up a 20-node Hadoop cluster is fun, whereas doing the same task in an hour in Excel means you have to move onto some other boring task. Generic doesn't mean robust either, I don't know where you got that from,the two concepts are entirely unrelated.
Generic -> robust. i... i dont know how to explain that. honestly i haven't thought about the necessity of explaining things like this. its... basic mathematics.
Generic != Robust.
It can quite easily be the opposite, they are certainly orthogonal concepts.
Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…
And I can't stand developers who overengineer things. We have a couple of them at my company and something that should take a few hours always take several weeks just because of all the reasons you mention. Most things don't need that kind of features and maintainability and if they do in the future we can just rewrite them from scratch. The overall expected return on investment is still better since we seldom need t…
Because in all too many companies, re-writing from scratch is a no-go, no matter how quickly and sloppily an initial solution was thrown together. I've worked on a prototype => production type project, where the throwaway was never thrown away. (the initial team made some mistakes, chief among them was building one prototype of a whole system, rather than one per major risk)
1.75 GB isn't enough data to justify a hadoop solution. That data size fits easily in memory, and without doubt on a single system. It depends on what you do with the data. If you are processing the data in 512KB chunks and each chunk takes a day to process (because expensive computation), you probably do want to spread the work over some cluster.
I think the article said that you don't need to use Hadoop for everything and that it might be much faster to just use command line tools on a single computer. Of course you might find a use case where the total computing time is massive and in that case a cluster is better. I still don't think many use cases have that problem. We are doing some simple statistics at work for much smaller data sizes and the computing…
Definitely. I was reacting to my parent poster, because size does not say everything. 1TB can be small, 1GB can be big - it depends on the amount of computation time that is necessary for whatever processing of the data you do.
To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…
Alternative, real life scenario: navigate through 6 months of daily MySQL dumps, assorted YAML files and Rails production.log, looking for some cross product between tables, requests and serialised entities, for analysis and/or data recovery (pinpoint or retrieval). zcat/cut/sed/grep/awk/perl crawled through it in a couple of minutes and required less than half an hour to craft a reliable enough implementation (inclu…
to be fair, you could have achieved all that with a simple python script. sometimes i feel python is the new bash.
To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…
Alternative, real life scenario: navigate through 6 months of daily MySQL dumps, assorted YAML files and Rails production.log, looking for some cross product between tables, requests and serialised entities, for analysis and/or data recovery (pinpoint or retrieval). zcat/cut/sed/grep/awk/perl crawled through it in a couple of minutes and required less than half an hour to craft a reliable enough implementation (inclu…
> Side note: I'm torn between leaving this place where nobody seems to understand the point of anything remotely like engineering or keeping this job where I'm obviously being extremely useful to our customers.
You should always aim at working with people who are smarter or better than you. Unless they have stack ranking.
Alternative, real life scenario: navigate through 6 months of daily MySQL dumps, assorted YAML files and Rails production.log, looking for some cross product between tables, requests and serialised entities, for analysis and/or data recovery (pinpoint or retrieval). zcat/cut/sed/grep/awk/perl crawled through it in a couple of minutes and required less than half an hour to craft a reliable enough implementation (inclu…
to be fair, you could have achieved all that with a simple python script. sometimes i feel python is the new bash.
Yeah. This is a great article. While I was reading it, I was thinking about what I would have done, and my answer was Python. Bash is just too easy to do wrong (see the recent Steam rm -rf bug), and I don't code in it often enough to know the pitfalls by heart.
I'd be interested to see another article about doing this job in Python and how its performance compares to this simple one-liner.
And I can't stand developers who overengineer things. We have a couple of them at my company and something that should take a few hours always take several weeks just because of all the reasons you mention. Most things don't need that kind of features and maintainability and if they do in the future we can just rewrite them from scratch. The overall expected return on investment is still better since we seldom need t…
Because in all too many companies, re-writing from scratch is a no-go, no matter how quickly and sloppily an initial solution was thrown together. I've worked on a prototype => production type project, where the throwaway was never thrown away. (the initial team made some mistakes, chief among them was building one prototype of a whole system, rather than one per major risk)
This is a systemic problem. Engineering is always subordinate to business. This simply should not be the case. We desperately need new business organization models.
And I can't stand developers who overengineer things. We have a couple of them at my company and something that should take a few hours always take several weeks just because of all the reasons you mention. Most things don't need that kind of features and maintainability and if they do in the future we can just rewrite them from scratch. The overall expected return on investment is still better since we seldom need t…
Quite the opposite, and, quite simple: engineers over-engineer thing in order to make things generic. and generic make solutions robust. that's basic science. Unless the problem and solution are well understood, your investment won't guarantee a return at all.
Generic, by default, does not in any way make things more robust.
We've gone from engineering solutions to meet specific problems to engineering solution frameworks that (supposedly) will solve the problem and allow for any unknowns. The problem is, no matter how hard the engineer tries, he can never anticipate the unknowns to the extent that the application framework can support all of them.
We should go back to solving the specific problem at hand. In both scenarios you get the customer who wants a feature that absolutely doesn't fit with the current application, therefore a rewrite is necessary. And with the specific solution, you don't have nearly the man hours wasted.
To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…
Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…
What kind of operational supportability do you need for a script that took a few hours to write and takes <5 minutes to run?
No, developers over-engineer because setting up a 20-node Hadoop cluster is fun, whereas doing the same task in an hour in Excel means you have to move onto some other boring task. Generic doesn't mean robust either, I don't know where you got that from,the two concepts are entirely unrelated.
Generic -> robust. i... i dont know how to explain that. honestly i haven't thought about the necessity of explaining things like this. its... basic mathematics.
I'm sorry, but if you cannot explain it, you simply do not understand it yourself.
That's harsh, I get it, and I'm truly sorry, but that's a basic fact.