141–150 of 178 posts
I'm genuinely curious - to analyse data effectively, is there a baseline of statistical understanding you need to have? If so, what is it?
Sorry, I could not get any further than when the sale pitch kicked in, so sorry when there was anything new after that.
Earlier quoted context omitted.
I thought it nailed a lot of dynamics at my last gig as well. Of course, I joined for the challenges of scaling and wrote an ETL framework in Rails (that was a mixed bag but very instructive) and then got bored after I realized how small our data really was. Then I left for Google and all my data went up by two prefixes. I do love reading an article that supports my contention that ETL is the Charlie Work [0] of soft…
I do love reading an article that supports my contention that ETL is the Charlie Work [0] of software engineering. Though I've never formally done work with the "ETL" label, what I've seen of it reminds me of the work I used to do for client years ago where I'd take some CSV file (or what have you) and turn it into something that their FoxBase-based accounting system could use. It was boring grunt work (or shall we s…
... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”. OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there's this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh…
Earlier quoted context omitted.
There was a HN story in which the author had the same "big data claim" (it was his intention to show there is very little big data in the real world, for most people out there) but instead he only needed to process couple GBs. He just used standard Unix commands and the performance was proven to be incredibly awesome. Here is an example for SQLite: https://news.ycombinator.com/item?id=9359568
I think you're talking about this: http://aadrake.com/command-line-tools-can-be-235x-faster-tha... I quite enjoyed it as well.
... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”. OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there's this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh…
This is so true. I do business intelligence at Amazon, and I've seen this play out millions of times over. The fetishization of big data ends up meaning that everybody thinks their problem needs big data. After 4 years in a role where I am expected to use big data clusters regularly, I've really only needed it twice. To be fair, in a complex environment with multiple data sources (databases, flat files, excel docs, s…
Earlier quoted context omitted.
My favorite is when you jump on a project and do a simple estimation of compute throughput for the highly complex distributed system, its something like hundreds of kilobytes per second. You could literally copy all the files to one computer and process faster than the web of broken parts. It becomes cancerous; to work around system slowness ever complex caching mechanisms and adhoc work is constructed. I think part…
Would you happen to know a good way to learn performance optimization? I'm working with datasets currently that I am trying to get to run faster, and I cannot tell if the limitation is on my hardware or due to ignorance on my part
The universal knowledge is learning to run well designed experiments, and this comes from practice. It's like how you would debug code without a debugger. There are profiling tools in some contexts that help you run these experiments, but at the highest level simply calculating the number of bytes that move through components divided by the amount of time it took is very enlightening.
It's valuable to have some rough familiarity of the limits of computer architecture. You can also do this experimentally; for example, you could test disk performance by timing how long it takes to copy a file much larger than RAM. You could try copying from /dev/zero to /dev/null to get a lower bound on RAM bandwidth. You can use netcat to see network throughput.
Bandwidth is only part of the picture; in some cases latency is important (such as servicing many tiny requests). Rough latency numbers are in [0, 1], but can also be learned experimentally.
Many popular primitives actually dont perform that great per node. For example something like a single MySQL or Spark node might not move than ~10MB/s per node; significantly lower than network bandwidth. You can actually use S3 to move data faster if it has a sequential access pattern :)
[0] http://static.googleusercontent.com/media/research.google.co...
Earlier quoted context omitted.
But you can't say that the doer always deserves the credit either. Sometimes the idea is the hard part. Similar for blame. It doesn't work to make generalizations. You have to make a judgment call every time, and usually the answer will be a complicated mixture.
"Ideas" alone are almost never worth anything. You have to do the work to back it up. Everyone I know has about a dozen ideas (you hear them all the time as someone who makes ideas real). What matters is the technical skill to make the idea go from a fantasy to a reality semi-reminiscent of the idealized fantastic version, whether that skill is in business, accounting, programming, marketing, or whatever.
The real pain is making a decision and expending resources on your challenging/risky idea. There's very little appetite for the responsibility and risk that come with big ideas (in a BigCo).
Got the ability to think up new ideas, sell them within an organisation, and get them executed (hello 'doer') in a way that provides value to that organisation? You're gold, and worth way more than the 'doer'.
Earlier quoted context omitted.
Fellow amazonian here. We switched from a massively distributed datastore (not to be named) to rodb for storage and found 10x improvement, not to mention eliminating cost and other head-aches; kind of expected since rodb is an embedded db...
What is rodb?