21–30 of 178 posts
This author seriously needs to expand all of his TLIs (three-letter initialisms) the first time he uses them, as any writer worth his or her salt would do. There are those who may be interested in what he has to say, but can't follow because of assuming abbreviations.
ETL, DBA, API - aren't they all incredibly standard acronyms?
The idea that engineers should build lego blocks without knowing what they're going to be used for is questionable at best. A better idea imho is to have small crossfunctional teams where scientists and engineers work together to build only what they need with short iteration cycles. If everyone involved doesn't have at least a broad perspective on the end-to-end purpose of what they're working on, they're probably g…
... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”. OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there's this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh…
I thought it nailed a lot of dynamics at my last gig as well. Of course, I joined for the challenges of scaling and wrote an ETL framework in Rails (that was a mixed bag but very instructive) and then got bored after I realized how small our data really was. Then I left for Google and all my data went up by two prefixes. I do love reading an article that supports my contention that ETL is the Charlie Work [0] of soft…
Though I've never formally done work with the "ETL" label, what I've seen of it reminds me of the work I used to do for client years ago where I'd take some CSV file (or what have you) and turn it into something that their FoxBase-based accounting system could use. It was boring grunt work (or shall we say, "Charlie work"), but I billed it at my usual rate and it paid the mortgage. I would never, ever wish to make a career of it, however. (And if my assessment of what someone knee-deep in ETL does all day is completely off base, I apologize.)
This author seriously needs to expand all of his TLIs (three-letter initialisms) the first time he uses them, as any writer worth his or her salt would do. There are those who may be interested in what he has to say, but can't follow because of assuming abbreviations.
It's a pretty common acronym in the field, actually.
... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”. OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there's this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh…
Like all things in tech, it depends. I agree with you that most startups will never see data that will not fit in memory. Remember http://yourdatafitsinram.com ? Now, a data warehouse where you have a hoarde of Hadoop engineers slaving over 1000 node clusters is probably overkill unless you are a company like facebook or are in the data processing biz. However, some database management systems can be very complementa…
I think you should try it again. I felt the same way when it was in the 1.5 version, but post 2.0 it is really easy. And the documentation has gotten so much better. It really is as simple as `sudo apt-get install postgis` and then `CREATE EXTENSION postgis;" in psql.
... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”. OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there's this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh…
Time and again, the tools prove that it's not.
To recollection, I can count the number of publicly-known companies dealing with these datasets on two hands, if being generous.
>You Probably Don’t Have Big Data "Big Data" is like sex in high school. Everyone talks about it but few people really have lots of it and some just don't have any.
The problem arises when someone gets into a position where they can think big thoughts without having to do any nitty gritty. Effectively, they end up jumping in right when the real producers have finished the actual work, and then coming up with some polish that makes it look like they came up with some interesting result.
This is not actually a way to get work done. It's a way to play politics.
Worse yet, it's actually completely detrimental to getting things done. When you have things split up between thinkers and doers, what do the incentives look like? It's quite simple. I may order some analysis, and I may not fully understand the nuances. But whatever happens, as a thinker I'll have to have something grandiose to say, and I'll need to keep the doers busy. That way if I don't find a real conclusion, it's everyone's fault. If I do find something, it's thanks to me.
Where I worked the people with the big plans couldn't code their way out of a paper bag. Ask them what Big-O is, they draw a blank. Ask them how their trading strategy will actually send orders to the exchange, they draw a blank. But ask them something that sounds like strategy, and they will feed you plenty of unsubstantiated BS.
My new venture is coders all the way down. Strategists who can actually use git without asking what it is, understand that algorithmic complexity actually matters, and so on. Coders who understand what the market is.
> "Rather than try to emulate the structure of well-known companies (who made the transition from BI to DS), we need to innovate and evolve the model! No more trying to design faster horses…
A couple years ago, I moved to Stitch Fix for just that very reason. At Stitch Fix, we strive to be Best in the World at the algorithms and analytics we produce. We strive to lead the business with our output rather than to inform it."
I find this article rather peculiar. At the start, you'd be forgiven for thinking this was an article about a company looking to find a solution to a problem, but as the article progresses it's clearer that they're selling themselves as the solution to the problem they outlined.
In other words, they start off looking like a customer, but only to set up the premise required to sell the solution to the problem their company supposedly has/had. Turned me off from taking the product seriously.