Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a data team at a mid-stage startup

erikbern.com

Post reply on HN
Print view View original

11–20 of 91 posts

Re: Building a data team at a mid-stage startup

#11

Post by sischoel »

IMTDb wrote: post #9

What would be the name of the position/profile of someone in charge of building the data warehousing architecture/ETL pipelines? I my view, they need make sure the warehouse model is a correct representation of the business and that it can be leveraged to answer basic or not-so-basic questions using SQL. They also need to promote it's usage internally by ensuring it is accessible and easy to use and guide other team…

What about "data engineer"? There seem to be a lot of jobs for that title nowadays.

Re: Building a data team at a mid-stage startup

#12

Post by herodoturtle »

For the last 15 years I've been building (what I consider to be) accessible database solutions, for a bunch of different industries.

This sentence from the article resonated with me:

> You're starting to lay the most basic foundation of what is most critically needed: all the important data, in the same place, easily queryable.

Re: Building a data team at a mid-stage startup

#13

Post by sjg007 »

IMTDb wrote: post #9

What would be the name of the position/profile of someone in charge of building the data warehousing architecture/ETL pipelines? I my view, they need make sure the warehouse model is a correct representation of the business and that it can be leveraged to answer basic or not-so-basic questions using SQL. They also need to promote it's usage internally by ensuring it is accessible and easy to use and guide other team…

The bigger issue is adaptability.. can you migrate schemas preserving older clients, typically that’s by providing a decent middleware…. SQL views are one way, APIs are another etc…

All of that while improving performance.

Re: Building a data team at a mid-stage startup

#14

Post by teej »

IMTDb wrote: post #9

What would be the name of the position/profile of someone in charge of building the data warehousing architecture/ETL pipelines? I my view, they need make sure the warehouse model is a correct representation of the business and that it can be leveraged to answer basic or not-so-basic questions using SQL. They also need to promote it's usage internally by ensuring it is accessible and easy to use and guide other team…

A new role has arisen in the last few years that captures much of this responsibility - Analytics Engineer.

This article by Claire Carroll describes the role and motivation for it https://www.getdbt.com/what-is-analytics-engineering/

Re: Building a data team at a mid-stage startup

#16

Post by jaggederest »

munk-a wrote: post #6
ttz wrote: post #5

> MBA types I chuckled. Then cried, because at least his MBA types can use SQL. My MBA types use Excel. OT: Good article. Like and agree with the push for centralizing data first, then building outwards so external teams can move towards self-service.

Building a good process into your company to receive a query, execute it against a read-only database, and shovel the results back to the user as a CSV file will pay dividends and is, honestly, pretty trivial in most cases.

Blazer is my go-to for this kind of thing:

https://github.com/ankane/blazer

Pretty easy to set up and share queries, dashboards, whatever

Re: Building a data team at a mid-stage startup

#17

Post by tmp_anon_22 »

IMTDb wrote: post #9

What would be the name of the position/profile of someone in charge of building the data warehousing architecture/ETL pipelines? I my view, they need make sure the warehouse model is a correct representation of the business and that it can be leveraged to answer basic or not-so-basic questions using SQL. They also need to promote it's usage internally by ensuring it is accessible and easy to use and guide other team…

Most common would be a DevOps or SRE on an observability team.

Re: Building a data team at a mid-stage startup

#19

Post by pram »

IMTDb wrote: post #9

What would be the name of the position/profile of someone in charge of building the data warehousing architecture/ETL pipelines? I my view, they need make sure the warehouse model is a correct representation of the business and that it can be leveraged to answer basic or not-so-basic questions using SQL. They also need to promote it's usage internally by ensuring it is accessible and easy to use and guide other team…

I’ve done this for the past 6 years and my title was “Big Data Infrastructure Engineer” but I don’t think there’s any consistency at companies from what I’ve seen

Re: Building a data team at a mid-stage startup

#20

Post by skrtskrt »

sischoel wrote: post #11
IMTDb wrote: post #9

What would be the name of the position/profile of someone in charge of building the data warehousing architecture/ETL pipelines? I my view, they need make sure the warehouse model is a correct representation of the business and that it can be leveraged to answer basic or not-so-basic questions using SQL. They also need to promote it's usage internally by ensuring it is accessible and easy to use and guide other team…

What about "data engineer"? There seem to be a lot of jobs for that title nowadays.

Yeah we would call this Data Engineer (likely Senior level or up for someone that has had experience building multiple data warehouses) plus the DevOps/SRE work required to stitch all the architecture together
Post reply on HN