Engineers Shouldn’t Write ETL • Hacker News Board

Re: Engineers Shouldn’t Write ETL

#91

Post by btilly » Fri, Mar 18, 2016, 7:26 PM UTC

andrewflnr wrote: post #78

btilly wrote: post #53

The thinker/doer problem goes way back. In most organizations the person who thinks of something gets the lion's share of the credit, and the person who implements it does the lion's share of the work. And if it turns out to be a bad idea, the thinker can always blame a bad implementation, thereby passing the lion's share of the blame to the doer. I've seen careers made and broken based on whether people got to play…

But you can't say that the doer always deserves the credit either. Sometimes the idea is the hard part. Similar for blame. It doesn't work to make generalizations. You have to make a judgment call every time, and usually the answer will be a complicated mixture.

And I didn't say that the doer always deserves the credit. As an example, who deserves more credit for the success of Apple, Steve Wozniak or Steve Jobs? Wozniak created the Apple I and most of the Apple II. But clearly Jobs' ideas built the current company.

However these cases are the exception, not the rule. As a rule ideas are cheap, implementations are hard. And success has more to do with iterating on the implementation than the starting ideas.

Re: Engineers Shouldn’t Write ETL

#92

Post by kelvin0 » Fri, Mar 18, 2016, 7:27 PM UTC

arethuza wrote: post #20

throwawy31816 wrote: post #2

This author seriously needs to expand all of his TLIs (three-letter initialisms) the first time he uses them, as any writer worth his or her salt would do. There are those who may be interested in what he has to say, but can't follow because of assuming abbreviations.

ETL, DBA, API - aren't they all incredibly standard acronyms?

Standard for you means that we live under rocks? Obviously if it's standard we must be clueless. Nice way of phrasing it.

Re: Engineers Shouldn’t Write ETL

#93

Post by quadstick » Fri, Mar 18, 2016, 7:30 PM UTC

ams6110 wrote: post #62

mikestew wrote: post #7

Earlier quoted context omitted.

Though I agree with you on expanding TLIs, if you have to have "ETL" defined for you, you probably won't get the "joke". And though this will come out more cynical than I intend, if you don't know the acronyms, then you probably won't be buying what Stitch Fix is selling. Filtering their funnel, maybe?

While possibly true, it's simply a courtesy to the reader to parenthetically define any acronym the first time it's used in a published piece of writing (of course this would not apply to internal emails, casual comments such as discussion forums here, etc.)

The original post may have been intended for a select audience that would be familiar with the context, so that author may be forgiven, but the person that submitted the post should have kept in mind the much wider audience here.

As a hardware engineer, ETL is a NRTL that competes with UL and CSA. Oh, excuse me, Thomas Edison's Electrical Testing Labs is a Nationally Recognized Testing Laboratory that competes with Underwriters Laboratories and the Canadian Standards Association.

Re: Engineers Shouldn’t Write ETL

#94

Post by im_down_w_otp » Fri, Mar 18, 2016, 7:31 PM UTC

mikestew wrote: post #5

... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”. OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there's this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh…

Indeed. Whenever I give a talk about "big data" I make sure to include the phrase, "I'm pretty sure most of the big data market exists on the hubris that developers want to desperately believe their problems are bigger than they really are."

Re: Engineers Shouldn’t Write ETL

#95

Post by cookiecaper » Fri, Mar 18, 2016, 7:33 PM UTC

andrewflnr wrote: post #78

btilly wrote: post #53

The thinker/doer problem goes way back. In most organizations the person who thinks of something gets the lion's share of the credit, and the person who implements it does the lion's share of the work. And if it turns out to be a bad idea, the thinker can always blame a bad implementation, thereby passing the lion's share of the blame to the doer. I've seen careers made and broken based on whether people got to play…

But you can't say that the doer always deserves the credit either. Sometimes the idea is the hard part. Similar for blame. It doesn't work to make generalizations. You have to make a judgment call every time, and usually the answer will be a complicated mixture.

"Ideas" alone are almost never worth anything. You have to do the work to back it up. Everyone I know has about a dozen ideas (you hear them all the time as someone who makes ideas real).

What matters is the technical skill to make the idea go from a fantasy to a reality semi-reminiscent of the idealized fantastic version, whether that skill is in business, accounting, programming, marketing, or whatever.

Re: Engineers Shouldn’t Write ETL

#96

Post by tyre » Fri, Mar 18, 2016, 7:42 PM UTC

strictfp wrote: post #19

MrFoof wrote: post #10

>You Probably Don’t Have Big Data "Big Data" is like sex in high school. Everyone talks about it but few people really have lots of it and some just don't have any.

The thing is that everybody has big data, it's only a question of how much of your data you save.

This is not true.

For many startups, even if they audit everything they won't have petabytes of data.

Re: Engineers Shouldn’t Write ETL

#97

Post by 7952 » Fri, Mar 18, 2016, 7:46 PM UTC

lomnakkus wrote: post #71

darksaints wrote: post #18

Earlier quoted context omitted.

This is so true. I do business intelligence at Amazon, and I've seen this play out millions of times over. The fetishization of big data ends up meaning that everybody thinks their problem needs big data. After 4 years in a role where I am expected to use big data clusters regularly, I've really only needed it twice. To be fair, in a complex environment with multiple data sources (databases, flat files, excel docs, s…

The really fun aspect of this for me, personally, is that I've been doing computers for ages (25 yrs or so?) now and 3TiB still intuitively feels like a massive amount of data even though I think I have something like 6TiB free space on my home server disks... which, in total, didn't even cost as much as a months' grocery shopping. Sometimes it really takes effort to rid yourself of these old intuitions that don't re…

I can remember going through this with a 10MB file about 15 year ago. It felt like a lot after growing up with floppy disks. But even a modest CPU could iterate over it quickly, I just didn't realise. I just assumed I would need to process it in a database!

Re: Engineers Shouldn’t Write ETL

#98

Post by tyre » Fri, Mar 18, 2016, 7:47 PM UTC

The article is about not over-engineering solutions to problems you do not have. If you don't have interesting problems that require world-class solutions, then don't hire as if you do.

> If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”.

And then comes this line:

>At Stitch Fix, we strive to be Best in the World at the algorithms and analytics we produce.

Without further justification, why does StitchFix, a subscription shopping service, need to be the "Best in the World" at algorithms and analytics? They have harder problems than Google or the Centers for Disease Control or NASA?

Unless they have justification for that, it seems a bit ironic given the article's ire for over-engineering.

Re: Engineers Shouldn’t Write ETL

#99

Post by hnal943 » Fri, Mar 18, 2016, 7:56 PM UTC

cookiecaper wrote: post #95

andrewflnr wrote: post #78

Earlier quoted context omitted.

But you can't say that the doer always deserves the credit either. Sometimes the idea is the hard part. Similar for blame. It doesn't work to make generalizations. You have to make a judgment call every time, and usually the answer will be a complicated mixture.

"Ideas" alone are almost never worth anything. You have to do the work to back it up. Everyone I know has about a dozen ideas (you hear them all the time as someone who makes ideas real). What matters is the technical skill to make the idea go from a fantasy to a reality semi-reminiscent of the idealized fantastic version, whether that skill is in business, accounting, programming, marketing, or whatever.

Eh, this is true of start up business ideas, but not necessarily true in an enterprise that has hired a bunch of doers. Vision is rare and important.

Re: Engineers Shouldn’t Write ETL

#100

Post by andrewprock » Fri, Mar 18, 2016, 7:59 PM UTC

btilly wrote: post #91

andrewflnr wrote: post #78

Earlier quoted context omitted.

But you can't say that the doer always deserves the credit either. Sometimes the idea is the hard part. Similar for blame. It doesn't work to make generalizations. You have to make a judgment call every time, and usually the answer will be a complicated mixture.

And I didn't say that the doer always deserves the credit. As an example, who deserves more credit for the success of Apple, Steve Wozniak or Steve Jobs? Wozniak created the Apple I and most of the Apple II. But clearly Jobs' ideas built the current company. However these cases are the exception, not the rule. As a rule ideas are cheap, implementations are hard. And success has more to do with iterating on the implem…

The Jobs/Wozniak example is a great one. I might suggest that Wozniak's execution created the company, and Jobs' vision transformed the company into what it is today.

But that transformation involved nearly destroying the company first, getting himself ousted. Then recreating MacOS as a UNIX platform for a market that did not want it, only to be finally integrated by the dying Apple as a Hail Mary play by both Jobs' and Apple.

Jobs' get a lot of credit for the vision, and even the execution. But it could have turned out lots of different ways. If Pixar had not been successful (in no small part because of Jobs dumping millions and millions of his own money into it), one could imagine NeXT not being bought by Apple.