Earlier quoted context omitted.
How long does it take you to load 2 TB into SQLite? How long do queries take? I believe you, but I'm in disbelief that it could be close to as efficient as throwing into ram. I mean, an EMR cluster takes like 5 minutes to spin up. Where do I learn how to do this? I've tried loading a TiB (one table one index) into SQLite on disk before, and it took forever. Granted this was a couple years ago, but I must be doing som…
There was a little bit of unfair comparison. I didn't have to load the data over a busy network connection, and I didn't have to decrypt the data once it was loaded (I had the benefit of a firewall and the raw data on a USB3 hard drive). I think there was a conversion from CSV to parquet on the cluster as well. And the engineer who set up the cluster was multitasking so I'm sure there was some latency issues just fro…
Engineers Shouldn’t Write ETL
171–178 of 178 posts
Re: Engineers Shouldn’t Write ETL
#172Re: Engineers Shouldn’t Write ETL
#173Earlier quoted context omitted.
Read only database. It is a hand optimized/compressed database engine that is used for big "once a day" data sets.
I'm Googling and finding a few things called "RODB" that don't quite match your descriptions. What in particular is it? Or are you talking about just rolling your own format to dump your data into? I've done that, but I'd still appreciate if I could use something that someone else had put the thought into. (Something like "cdb" by Daniel J. Bernstein, but that's an old 32-bit library that's limited to 4 GB files.)
Re: Engineers Shouldn’t Write ETL
#174Earlier quoted context omitted.
I'm Googling and finding a few things called "RODB" that don't quite match your descriptions. What in particular is it? Or are you talking about just rolling your own format to dump your data into? I've done that, but I'd still appreciate if I could use something that someone else had put the thought into. (Something like "cdb" by Daniel J. Bernstein, but that's an old 32-bit library that's limited to 4 GB files.)
rodb is in-house, non-FOSS tech @ Amazon.
Re: Engineers Shouldn’t Write ETL
#175He hits upon a quite interesting division of labor. Where I've worked in finance, there's been "strategists" and there's been "developers". You can guess which one is seen as high prestige. The problem arises when someone gets into a position where they can think big thoughts without having to do any nitty gritty. Effectively, they end up jumping in right when the real producers have finished the actual work, and the…
In the horizontal setup you describe (layer of thinkers on top of layer of doers), credit hits a barrier at the thinkers. The gradient isn't propagated.
In the vertical setup in the article (layer of thinker-doers), of course the backprop will be good because it is only one layer thick. You gain proper incentives, proper treatment of data on the whole pipeline. And the engineers can also concentrate on a purely orthogonal thing: writing tools.
But you lose the benefit of having the layer being able to focus on one thing. The author acknowledges those efficiencies (his word). It is hard to find people with a wide set of skills. Although in this case it is balanced, as now the engineers have gained specialization.
But I digress. My point was: humans in orgs are bad at backprop. Why share the credit at all? Organisations can be seen as neural networks/graphs, and they can lack proper backprop.
I'd love to see the results of some pagerank-like backprop. Every employee gets one base point. Every week, he is asked: "who helped you the most in doing your job this week?". Sales would credit analysts who would credit engineers, etc. Or Sales would credit analysts-engineers who would credit tool-writers, etc. It could go both ways: engineers could credit sales or analysts for writings well thought-out problem descriptions.
Then you would run pagerank on it, and base every promotion, every salary increase on it. Information would flow well, and everybody has a clear direction (his gradient) of what he can do to shine.
Also, by injecting revenue at the sales layer in a certain period of time, you could identify who conctributed the most in an increase of revenue.
Also, I posit that managers have a tiny view of what happens in a firm. They only get to see a fraction of interactions, while the brunt of what matters happens in the long tail of one-to-one interactions. Should you chose to promote people with the highest PR, you would have a true result-based bottom-up org.
Re: Engineers Shouldn’t Write ETL
#176Earlier quoted context omitted.
> If you're in this field, and consider yourself an "engineer" but your math sucks, go read up on all you can about mathematics and statistics, just like you did back when you were learning about programming, operating systems and networking. This assumes availability of time. Obviously, given enough time, people could develop both top-tier engineering and DS skillsets! Of course, if lots of free time were common we'…
I don't know... Are you saying that someone like Donald Knuth had a lot of free time? Data problems are arguably the hardest problems out there and they require deep understanding of mathematics as well as computers, that's just the way it is. It does take time and effort and even may be a bit of talent as well, not everyone is cut out for it.
/u/dxbydt commented on this very well, so I'll only reinforce the point that not everyone is Donald Knuth. If the distinction between DS and Engineer is blurred in your specific instance, and you're capable of Knuth-ian levels of work in both, you are almost certainly underpaid and need to lead a team or start a company yourself, stat, since a top-tier combination of those skills is exceedingly rare.
Re: Engineers Shouldn’t Write ETL
#177Earlier quoted context omitted.
And I didn't say that the doer always deserves the credit. As an example, who deserves more credit for the success of Apple, Steve Wozniak or Steve Jobs? Wozniak created the Apple I and most of the Apple II. But clearly Jobs' ideas built the current company. However these cases are the exception, not the rule. As a rule ideas are cheap, implementations are hard. And success has more to do with iterating on the implem…
> However the problem is that actual credit for > success REALLY belongs with the > people who did the work. > And I didn't say that the doer always > deserves the credit. You kind of did.
Take my Steve Jobs example. Do you really think that he didn't work hard?
Re: Engineers Shouldn’t Write ETL
#178Earlier quoted context omitted.
It depends a lot on what you're doing, but 2 TB in general seems a lot. If you have to perform an out-of-disk sort, you probably need a distributed setup. The funny thing is that most setups that use EMR defeat the data locality principle and I think that's the speedup people experience when they run it on a single laptop for example. Reading the original Google paper helped me a lot in understanding this.
If you have to perform an out-of-disk sort, buy another hard disk and now it's not out-of-disk anymore. This will suffice for nearly any data set you would ever need to sort. A 4 TB drive costs about $120, and you'll spend way more than that on software development and extra computers if you do distributed computing when you don't need to.