Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a data team at a mid-stage startup

erikbern.com

Post reply on HN
Print view View original

41–50 of 91 posts

Re: Building a data team at a mid-stage startup

#41

Post by marcinzm »

IMTDb wrote: post #9

What would be the name of the position/profile of someone in charge of building the data warehousing architecture/ETL pipelines? I my view, they need make sure the warehouse model is a correct representation of the business and that it can be leveraged to answer basic or not-so-basic questions using SQL. They also need to promote it's usage internally by ensuring it is accessible and easy to use and guide other team…

You're mixing up two different tasks as I see it:

* Building/defining the data infrastructure

* Building/defining the schemas

In a traditional ETL infrastructure they are jumbled together but if you do ELT they are not. A data engineer can build the infrastructure but the transformations can be handled better by technical analysts. They're simply one view on the underlying data so the risk is minimal. Analysts query the data day in and day out so they know much better what they need than someone who doesn't.

Re: Building a data team at a mid-stage startup

#43

Post by marcinzm »

czep wrote: post #24

This is so eerily familiar I swear I've had many of these exact conversations word for word. The only way this doesn't turn into a complete nightmare of a cluster is if the exec team "gets it". If so, you just might stand a chance at building a data team that gels with the rest of the org. But if the exec team simply hired you for window-dressing, expect to be treated like a scapegoat and a punching bag. Any mistakes…

In my experience much of this is a question of trust, political capital and soft power. Find out the problems that the key players in the business are actually having that you can solve and then solve them. Find out what the key KPIs are for the business and make a plan to improve them and then have a plan to publicize that improvement. And make sure to hire a team that covers your weaknesses rather than exposes them. Don't fight people if you can help it, either they're as competent as you on average or you shouldn't have taken the job. Figure out how to help them and what they need to work more efficiently and then give it to them. Sure there's a ton of politics involved in all of that but that's management in general.

Re: Building a data team at a mid-stage startup

#44

Post by plaidfuji »

So many gems in this article…

> You notice a a lot of the code starts with very complicated preprocessing steps, where data has to be fetched from many different systems. There appears to be several scripts that have to be run manually in the right order to run some of these things.

> “We need to focus on delivering business value as quickly as possible”, you say, but you add that “we might get back to the machine learning stuff soon… let's see”.

So so relatable. But the key insight is a really really key insight.

> What I think makes most sense to push for is a centralization the reporting structure, but keeping the work management decentralized. Why? Primarily because it creates a much tighter feedback loop between data and decisions. If every question has to go through a central bottleneck, transaction costs will be high. On the other hand, you don't want to decentralize the management. Strong data people want to report into a manager who understands data, not into a business person.

I have the same role at a non-software company, and to me this is nothing short of a complete reimagining of IT. It’s not just, “make sure everyone’s computer works and help them install software,” it’s, “build a model of the business, determine what information flows and metrics are crucial to success, and build an IT and analysis infrastructure around that model.” The CIO will soon be better thought of as the Chief Optimization Officer.

Re: Building a data team at a mid-stage startup

#45

Post by ramraj07 »

Orou wrote: post #37
IMTDb wrote: post #9

What would be the name of the position/profile of someone in charge of building the data warehousing architecture/ETL pipelines? I my view, they need make sure the warehouse model is a correct representation of the business and that it can be leveraged to answer basic or not-so-basic questions using SQL. They also need to promote it's usage internally by ensuring it is accessible and easy to use and guide other team…

I would also vote for "data engineer" (it's my current job title). You very likely don't want a data scientist to be doing a data engineer's job (and they probably don't want to be doing it themselves!). While there are similarities, data engineering tends to be a lot closer to software development than data science. If you're advertising for a data scientist role, don't expect them to be happy if 80% of their job is…

The one issue is that the gamut of experience and ability in a data engineer (and the salaries) is extremely wide, far wider than I’ve seen for any other role. Hiring a good DE is so hard!

Re: Building a data team at a mid-stage startup

#46

Post by jabagonuts »

Really enjoyed this narrative, but what about the next phase? Going from mid-stage to mature startup?

> Note that you took on a lot of “tech debt” earlier when you started dumping the production database tables straight into the data warehouse.

How do you manage expectations when the year-long honeymoon is over, the business grows tremendously, and the centralized data warehouse reaches a breaking point?

Re: Building a data team at a mid-stage startup

#47

Post by neighbour »

jabagonuts wrote: post #46

Really enjoyed this narrative, but what about the next phase? Going from mid-stage to mature startup? > Note that you took on a lot of “tech debt” earlier when you started dumping the production database tables straight into the data warehouse. How do you manage expectations when the year-long honeymoon is over, the business grows tremendously, and the centralized data warehouse reaches a breaking point?

Also thought this. Let's hope the author has a SQL in the works as I am keen to hear more.

Re: Building a data team at a mid-stage startup

#48

Post by rickeydidio »

Orou wrote: post #37
IMTDb wrote: post #9

What would be the name of the position/profile of someone in charge of building the data warehousing architecture/ETL pipelines? I my view, they need make sure the warehouse model is a correct representation of the business and that it can be leveraged to answer basic or not-so-basic questions using SQL. They also need to promote it's usage internally by ensuring it is accessible and easy to use and guide other team…

I would also vote for "data engineer" (it's my current job title). You very likely don't want a data scientist to be doing a data engineer's job (and they probably don't want to be doing it themselves!). While there are similarities, data engineering tends to be a lot closer to software development than data science. If you're advertising for a data scientist role, don't expect them to be happy if 80% of their job is…

This is spot on. As someone who has been looking for a data analyst role, I’ve actually read quite a few DS reqs that were geared more towards infrastructure and ETL. Then the flip side with the DE reqs wanting NumPy and Pandas along with the infrastructure and ETL. Weird, right?

Re: Building a data team at a mid-stage startup

#49

Post by Artgor »

When I had started reading this article, I had thought that it would be a sad story about another startup failure. The blogpost turned out to be a fascinating story of the success. I really liked it.

But after I had finished reading it, I have realized that it is a sad story, if we look from the eyes of data scientists in the team. People were hired to do cool machine learning projects, but it turned out there is no infrastructure for them. After the new boss had arrived, they had to work as analysts for months. What is more sad - the new boss dangled a carrot before them several times, but each time the carrot disappeared.

Post reply on HN