Live data from Hacker News

Building a data team at a mid-stage startup

erikbern.com

81–90 of 91 posts

Re: Building a data team at a mid-stage startup

#81
post #78

Earlier quoted context omitted.

Heap might be good but they are crazy expensive. We were quoted something like a quarter million dollars. Good luck getting that signed off, plus you still need quite technical analysts to run the thing. I've found https://contentsquare.com/ to be much better received by juniors and seniors alike, and it's a fraction of the cost of heap.

I don’t know the specifics of what you were quoted, but a quarter million dollars (guessing per year?) does strike me as high. Were you a later-stage startup by chance? The price point for pre-Series-C startups should be much, much lower.

That's odd. Why would you charge more for a post-series C startup or enterprise versus a pre-series C?

Re: Building a data team at a mid-stage startup

#82
post #81

Earlier quoted context omitted.

I don’t know the specifics of what you were quoted, but a quarter million dollars (guessing per year?) does strike me as high. Were you a later-stage startup by chance? The price point for pre-Series-C startups should be much, much lower.

That's odd. Why would you charge more for a post-series C startup or enterprise versus a pre-series C?

That’s generally how pricing works for SAAS products - most later stage customers have stricter or more customized needs. Think support SLAs, SSO, ACLs for their employees, etc.

Re: Building a data team at a mid-stage startup

#83
post #55

I liked this article, but I have two questions: 1. Is it definitely a good idea to build a separate data team, rather than embedding people with analytics knowledge in feature teams? Is it possible to do the latter, but still have end up with a well-curated source-of-truth for your data? 2. Is A/B testing and driving your business by metrics really a good idea? My (uninformed) impression is that data-driven is respon…

Is driving your business by the highest paid person’s opinion any different than driving it by A/B testing? I see those as two extreme end positions.

A/B testing can help you with optimizing existing processes for incremental improvement, but big bets, which can sometimes have data and sometimes don’t, help with step change improvements.

Even with big bets you need a way to show that it’s better than the previous way. Either by coming up with ways to cheaply test the hypothesis or committing to being “agile” (I hate that term) and continuing to iterate.

What is statistical significance anyways? If the p-value is 0.06 is that good enough? Practical significance is something that also needs to be accounted for.

If something can’t be measured, is there a way to find some proxy metric for it?

If not, then you can try to negotiate a pilot study of the problem and have specific criteria to determine success.

Just because something can’t be measured with existing processes doesn’t mean it can’t be measured at all.

For example, there were complaints about systems crashing and having intermittent behavior, and the claim was that’s affecting sales. Technology said nothing in our logs shows any issues, our service center shows no reporting of issues, so we think they are overreacting. We put a team together and went to several different locations to observe the process and get feedback. From the feedback we put together a data collection sheet and went back for a week to collect more data. That finally convinced the Tech team that it was a problem they needed to investigate. They went to the stores, determined it’s true, and amended logging to capture what’s truly going on.

Re: Building a data team at a mid-stage startup

#84
post #71

Earlier quoted context omitted.

can you point me at some of the vendors - I am missing a chunk of knowledge i suspect. Or is this - for exmaple - people taking google analytics and producing analysis on top of that.?

+1. @Zippy - May I ask for some of the vendors you refer to, please? Also love the Uranium analogy.

So for example, the author saw that supply chain team had difficulty managing the complexity and scale of their analysis in large part due to the scalability of their spreadsheet solution. I would have pushed them to use Airtable which is basically a more scalable spreadsheet. By choosing the data pipeline route, the people who understand how to improve the supply chain model and the history of decisions that went into it, as well as previous missteps, now have limited ability to experiment with improving it. In my experience, every rewrite of a system has something lost in translation which makes me think that in the authors example that the life of the analysts got better but may have made the quality of supply chain model worse.

In the long run, there is plenty of useful logistics software that should do everything they want but the most important thing is to empower the people with domain expertise in the data to be as close to the solution as possible. Better decisions are often a result of better information/experience than better analysis. Unfortunately I haven’t studied these vendors well enough to make any suggestions though I believe that the solutions are well defined enough to write textbooks on them, which suggests to me that existing software and I would mostly implement similar methodologies.

On the marketing and product analytics tools, I think 80% of the problems boil down to measuring conversion rates and the comparing those rates across different contexts to select for the contexts which improves those rates.

Another user mentioned heap, which is great product if you know you don’t know what contextual data is meaningful but you suspect that it’s partially in how they interact with other parts of your website. Personally I’d use heap judiciously since I suspect there will be limitations to how useful the historical data will be in the future and collecting everything is expensive. One limitation is that site interactions are only part of the potentially important context. Another limitation is that startups change rapidly, so their historical data often depreciates in terms providing insight into their current problems. For an extreme example, I’m sure zoom’s conversion data before and during pandemic look completely different. But even a small tweak to google’s search algorithm could totally change what type of customer finds your site.

Personally I’d advocate talking to customers, potential customers, and other stake holders to understand what is important and measure that. Most companies, currently do the opposite where they take a lot of measurements and then try to figure out what’s important. The first approach can probably be done in google analytics. The second I might try and use Amplitude which is I what imagine a tool like heap will eventually try to evolve into.

The hardest person to help with data in the organization is the CEO because really they use data as form sales tool and reporting. The closest I have seen a tool to doing this in a way the CEO could mostly self service is Sisu data. Though it’s the CEO so it’s probably reasonable to hire some help anyway.

Lastly data warehouses were the gold standard in the early 2010s but Presto is better fit these days for companies whose data is distributed across many different places.

Re: Building a data team at a mid-stage startup

#85
post #49

When I had started reading this article, I had thought that it would be a sad story about another startup failure. The blogpost turned out to be a fascinating story of the success. I really liked it. But after I had finished reading it, I have realized that it is a sad story, if we look from the eyes of data scientists in the team. People were hired to do cool machine learning projects, but it turned out there is no…

Very interesting perspective. As a early-mid stage startup, you definitely want to invest in generalists who are able to build infrastructure before hiring specialized ICs.

I honestly had flashbacks when the author mentioned the carrot dangling thing. I’ve personally experienced this and as a naive early career swe, I gave the manager the benefit of doubt for a year even though I knew there was no way they could guarantee it. This is just pure manipulation.

The worst part is that he wrote the job description himself and resorted to manipulation to cover up his mistake of hiring for the wrong job role.

Re: Building a data team at a mid-stage startup

#86
post #55

I liked this article, but I have two questions: 1. Is it definitely a good idea to build a separate data team, rather than embedding people with analytics knowledge in feature teams? Is it possible to do the latter, but still have end up with a well-curated source-of-truth for your data? 2. Is A/B testing and driving your business by metrics really a good idea? My (uninformed) impression is that data-driven is respon…

In my experience AB testing has a time and place - and that is after a certain level of traffic load and product/feature maturity and only to "validate" certain hypotheses.

For low volumes of traffic AB testing would takes ages to wield significant results and for products still maturing and shaping there is lot of "wisdom of crowds" data already available to help make decisions faster (ie: do you really need an AB test to know offering timely promotion to users helps convert?)

If you got a young product trying to grow, fast, it's a lot more effective to rely on experienced product people and off-the-shelf simple analytics to iterate quickly and to take some bets so one day you get to a point where AB testing "optimisations" starts to make sense.

It's a quite an interesting topic! I agree with you too - A/B test driven sites tends to culminate in terrible "cumulative experience" for users

Re: Building a data team at a mid-stage startup

#87

Earlier quoted context omitted.

can you point me at some of the vendors - I am missing a chunk of knowledge i suspect. Or is this - for exmaple - people taking google analytics and producing analysis on top of that.?

Highly recommend Heap [1] - they have a neat approach that doesn’t require you to ‘decide’ which analytics you want to track ahead of time. Disclaimer: I was an early engineer at Heap. [1] https://heap.io/

Very happy heap customer here. Been using it since 2016 or so and brought it from last company to my current startup. Autocapture is magic.

Re: Building a data team at a mid-stage startup

#88
post #51

This was wonderfully written and if your gonna start a data team, this is how you do it. But I can see that I’m the only one who thought it was crazy to start a data team in the first place. This company makes 10M and spends 3M on the team and infrastructure to make data a core competency? A vast majority of wins discussed were lowly differentiated web / mobile / supply chain analytics which they could have gotten an…

> My take away was that startups benefit tremendously from a data advisor role to get the data competency, as well as the educational and cultural benefits, but realistically the data infrastructure and analytics at that scale should have been bought not built.

I really like your takeaway about data teams at tech companies. They try to make "data" a core competency of their business, at huge cost for fixed value.

I also appreciated the very subtle implication that the OP is shrouding empire building under an otherwise informative growth story.

Re: Building a data team at a mid-stage startup

#89
Extremely relatable content throughout. Especially around teams beating their own drums while CEO questions around metrics. ;)

Will wait for a follow up post on how decentralised data team created data silos and how we solve it using data discovery and data standardisation. :P

Disclaimer: I have built decentralised data teams and it scales well.

Re: Building a data team at a mid-stage startup

#90

Wow, a story where things start out a mess and end up a lot better! Can we write one of these for society too?

There are bright spots. You might enjoy this book: Radical Equations: Civil Rights from Mississippi to the Algebra Project by Bob Moses [1]

See also: The Algebra Project https://algebra.org/wp/

[1]: https://en.wikipedia.org/wiki/Bob_Moses_(activist)

Post reply on HN