Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a data team at a mid-stage startup

erikbern.com

Post reply on HN
Print view View original

21–30 of 91 posts

Re: Building a data team at a mid-stage startup

#21

Post by hobs »

IMTDb wrote: post #9

What would be the name of the position/profile of someone in charge of building the data warehousing architecture/ETL pipelines? I my view, they need make sure the warehouse model is a correct representation of the business and that it can be leveraged to answer basic or not-so-basic questions using SQL. They also need to promote it's usage internally by ensuring it is accessible and easy to use and guide other team…

I currently do that job as a Data Architect - kind of a mouthful lol but it covers the gamut of understanding the entire business as an abstract set of data flows, being responsible for the ingest and outflows of data, the level of quality in our overarching system, managing data engineers, developers, business folks all accessing said data, at the end of the day explaining what it all means to our clients and devs via standard modeling stuff and more targeted things as needed.

Re: Building a data team at a mid-stage startup

#22

Post by edmundsauto »

IMTDb wrote: post #9

What would be the name of the position/profile of someone in charge of building the data warehousing architecture/ETL pipelines? I my view, they need make sure the warehouse model is a correct representation of the business and that it can be leveraged to answer basic or not-so-basic questions using SQL. They also need to promote it's usage internally by ensuring it is accessible and easy to use and guide other team…

This is what data engineers do, although that is also used to describe data ops (maintaining clusters, running kafka, etc.)

Re: Building a data team at a mid-stage startup

#23

Post by edmundsauto »

hobs wrote: post #21
IMTDb wrote: post #9

What would be the name of the position/profile of someone in charge of building the data warehousing architecture/ETL pipelines? I my view, they need make sure the warehouse model is a correct representation of the business and that it can be leveraged to answer basic or not-so-basic questions using SQL. They also need to promote it's usage internally by ensuring it is accessible and easy to use and guide other team…

I currently do that job as a Data Architect - kind of a mouthful lol but it covers the gamut of understanding the entire business as an abstract set of data flows, being responsible for the ingest and outflows of data, the level of quality in our overarching system, managing data engineers, developers, business folks all accessing said data, at the end of the day explaining what it all means to our clients and devs v…

You mention that you manage data engineers. Where does your role not overlap w/ a data eng?

Re: Building a data team at a mid-stage startup

#24

Post by czep »

This is so eerily familiar I swear I've had many of these exact conversations word for word. The only way this doesn't turn into a complete nightmare of a cluster is if the exec team "gets it". If so, you just might stand a chance at building a data team that gels with the rest of the org.

But if the exec team simply hired you for window-dressing, expect to be treated like a scapegoat and a punching bag. Any mistakes will be your fault. Any wins will be to the credit of the business. The Director of Product will ask to "embed" dedicated DS headcount and you won't have any real power to shape the roadmap. If the exec team doesn't give you equal footingf with Product (or Marketing, Finance, and Eng for that matter) then this will rapidly become a soul-sucking job. However, if E-team does give you the authority to call Product's bullshit, and tell Finance to stuff it, and not take direction from Eng leads, then you actually might be able to accomplish something really cool.

Re: Building a data team at a mid-stage startup

#25

Post by ttz »

munk-a wrote: post #6
ttz wrote: post #5

> MBA types I chuckled. Then cried, because at least his MBA types can use SQL. My MBA types use Excel. OT: Good article. Like and agree with the push for centralizing data first, then building outwards so external teams can move towards self-service.

Building a good process into your company to receive a query, execute it against a read-only database, and shovel the results back to the user as a CSV file will pay dividends and is, honestly, pretty trivial in most cases.

Funnily enough, this is what I did, except I built an app where I write the queries as "pre-built" parameterized ones (sanitized, of course).

People still do a bunch of stuff in Excel, though, and every once in a while, it breaks, and I have to dig through the mess. Excel is great when it's just for yourself and you can manage it... it's a pain when others have to figure out someone else's.

Re: Building a data team at a mid-stage startup

#26

Post by hobs »

edmundsauto wrote: post #23
hobs wrote: post #21

Earlier quoted context omitted.

I currently do that job as a Data Architect - kind of a mouthful lol but it covers the gamut of understanding the entire business as an abstract set of data flows, being responsible for the ingest and outflows of data, the level of quality in our overarching system, managing data engineers, developers, business folks all accessing said data, at the end of the day explaining what it all means to our clients and devs v…

You mention that you manage data engineers. Where does your role not overlap w/ a data eng?

In our team its mostly a difference of business focus and the overarching responsibility - most data engineers I work with manage a major leg of the business and are responsible for their domain but I am responsible for all of them.

I certainly spend time coding (especially because again, small-medium startups cant afford anyone in the data space who isnt able to heave ho) but much of it is translating pretty vague stuff into market research/a proof of concept/an initial design of what will bring value to the business and scale alright and then often more people will throw in.

That being said you can call me whatever you want, as long as its not late for dinner :)

Re: Building a data team at a mid-stage startup

#29

Post by PragmaticPulp »

czep wrote: post #24

This is so eerily familiar I swear I've had many of these exact conversations word for word. The only way this doesn't turn into a complete nightmare of a cluster is if the exec team "gets it". If so, you just might stand a chance at building a data team that gels with the rest of the org. But if the exec team simply hired you for window-dressing, expect to be treated like a scapegoat and a punching bag. Any mistakes…

This applies to most specialties. Companies tend to have a few teams that lead the charge and expect everyone else to follow. Knowing which teams get the authority and which teams are along for the ride at a company is important for knowing what your job experience will look like.

> However, if E-team does give you the authority to call Product's bullshit, and tell Finance to stuff it, and not take direction from Eng leads

I know this was meant partially in jest, but if you reach the point where you're at odds with all of the teams and departments in the company you may get a lot done in the short term, but long term it's going to be difficult if you don't have some allies in each of those departments. Obviously no one should roll over and take orders from other departments, but some times it's necessary to do some give and take to build rapport. It's a balance, not a war.

Post reply on HN