Earlier quoted context omitted.
Heap might be good but they are crazy expensive. We were quoted something like a quarter million dollars. Good luck getting that signed off, plus you still need quite technical analysts to run the thing. I've found https://contentsquare.com/ to be much better received by juniors and seniors alike, and it's a fraction of the cost of heap.
I don’t know the specifics of what you were quoted, but a quarter million dollars (guessing per year?) does strike me as high. Were you a later-stage startup by chance? The price point for pre-Series-C startups should be much, much lower.
Building a data team at a mid-stage startup
81–90 of 91 posts
Re: Building a data team at a mid-stage startup
#82Earlier quoted context omitted.
I don’t know the specifics of what you were quoted, but a quarter million dollars (guessing per year?) does strike me as high. Were you a later-stage startup by chance? The price point for pre-Series-C startups should be much, much lower.
That's odd. Why would you charge more for a post-series C startup or enterprise versus a pre-series C?
Re: Building a data team at a mid-stage startup
#83I liked this article, but I have two questions: 1. Is it definitely a good idea to build a separate data team, rather than embedding people with analytics knowledge in feature teams? Is it possible to do the latter, but still have end up with a well-curated source-of-truth for your data? 2. Is A/B testing and driving your business by metrics really a good idea? My (uninformed) impression is that data-driven is respon…
A/B testing can help you with optimizing existing processes for incremental improvement, but big bets, which can sometimes have data and sometimes don’t, help with step change improvements.
Even with big bets you need a way to show that it’s better than the previous way. Either by coming up with ways to cheaply test the hypothesis or committing to being “agile” (I hate that term) and continuing to iterate.
What is statistical significance anyways? If the p-value is 0.06 is that good enough? Practical significance is something that also needs to be accounted for.
If something can’t be measured, is there a way to find some proxy metric for it?
If not, then you can try to negotiate a pilot study of the problem and have specific criteria to determine success.
Just because something can’t be measured with existing processes doesn’t mean it can’t be measured at all.
For example, there were complaints about systems crashing and having intermittent behavior, and the claim was that’s affecting sales. Technology said nothing in our logs shows any issues, our service center shows no reporting of issues, so we think they are overreacting. We put a team together and went to several different locations to observe the process and get feedback. From the feedback we put together a data collection sheet and went back for a week to collect more data. That finally convinced the Tech team that it was a problem they needed to investigate. They went to the stores, determined it’s true, and amended logging to capture what’s truly going on.
Re: Building a data team at a mid-stage startup
#84Earlier quoted context omitted.
can you point me at some of the vendors - I am missing a chunk of knowledge i suspect. Or is this - for exmaple - people taking google analytics and producing analysis on top of that.?
+1. @Zippy - May I ask for some of the vendors you refer to, please? Also love the Uranium analogy.
In the long run, there is plenty of useful logistics software that should do everything they want but the most important thing is to empower the people with domain expertise in the data to be as close to the solution as possible. Better decisions are often a result of better information/experience than better analysis. Unfortunately I haven’t studied these vendors well enough to make any suggestions though I believe that the solutions are well defined enough to write textbooks on them, which suggests to me that existing software and I would mostly implement similar methodologies.
On the marketing and product analytics tools, I think 80% of the problems boil down to measuring conversion rates and the comparing those rates across different contexts to select for the contexts which improves those rates.
Another user mentioned heap, which is great product if you know you don’t know what contextual data is meaningful but you suspect that it’s partially in how they interact with other parts of your website. Personally I’d use heap judiciously since I suspect there will be limitations to how useful the historical data will be in the future and collecting everything is expensive. One limitation is that site interactions are only part of the potentially important context. Another limitation is that startups change rapidly, so their historical data often depreciates in terms providing insight into their current problems. For an extreme example, I’m sure zoom’s conversion data before and during pandemic look completely different. But even a small tweak to google’s search algorithm could totally change what type of customer finds your site.
Personally I’d advocate talking to customers, potential customers, and other stake holders to understand what is important and measure that. Most companies, currently do the opposite where they take a lot of measurements and then try to figure out what’s important. The first approach can probably be done in google analytics. The second I might try and use Amplitude which is I what imagine a tool like heap will eventually try to evolve into.
The hardest person to help with data in the organization is the CEO because really they use data as form sales tool and reporting. The closest I have seen a tool to doing this in a way the CEO could mostly self service is Sisu data. Though it’s the CEO so it’s probably reasonable to hire some help anyway.
Lastly data warehouses were the gold standard in the early 2010s but Presto is better fit these days for companies whose data is distributed across many different places.
Re: Building a data team at a mid-stage startup
#85When I had started reading this article, I had thought that it would be a sad story about another startup failure. The blogpost turned out to be a fascinating story of the success. I really liked it. But after I had finished reading it, I have realized that it is a sad story, if we look from the eyes of data scientists in the team. People were hired to do cool machine learning projects, but it turned out there is no…
I honestly had flashbacks when the author mentioned the carrot dangling thing. I’ve personally experienced this and as a naive early career swe, I gave the manager the benefit of doubt for a year even though I knew there was no way they could guarantee it. This is just pure manipulation.
The worst part is that he wrote the job description himself and resorted to manipulation to cover up his mistake of hiring for the wrong job role.
Re: Building a data team at a mid-stage startup
#86I liked this article, but I have two questions: 1. Is it definitely a good idea to build a separate data team, rather than embedding people with analytics knowledge in feature teams? Is it possible to do the latter, but still have end up with a well-curated source-of-truth for your data? 2. Is A/B testing and driving your business by metrics really a good idea? My (uninformed) impression is that data-driven is respon…
For low volumes of traffic AB testing would takes ages to wield significant results and for products still maturing and shaping there is lot of "wisdom of crowds" data already available to help make decisions faster (ie: do you really need an AB test to know offering timely promotion to users helps convert?)
If you got a young product trying to grow, fast, it's a lot more effective to rely on experienced product people and off-the-shelf simple analytics to iterate quickly and to take some bets so one day you get to a point where AB testing "optimisations" starts to make sense.
It's a quite an interesting topic! I agree with you too - A/B test driven sites tends to culminate in terrible "cumulative experience" for users
Re: Building a data team at a mid-stage startup
#87Earlier quoted context omitted.
can you point me at some of the vendors - I am missing a chunk of knowledge i suspect. Or is this - for exmaple - people taking google analytics and producing analysis on top of that.?
Highly recommend Heap [1] - they have a neat approach that doesn’t require you to ‘decide’ which analytics you want to track ahead of time. Disclaimer: I was an early engineer at Heap. [1] https://heap.io/
Re: Building a data team at a mid-stage startup
#88This was wonderfully written and if your gonna start a data team, this is how you do it. But I can see that I’m the only one who thought it was crazy to start a data team in the first place. This company makes 10M and spends 3M on the team and infrastructure to make data a core competency? A vast majority of wins discussed were lowly differentiated web / mobile / supply chain analytics which they could have gotten an…
I really like your takeaway about data teams at tech companies. They try to make "data" a core competency of their business, at huge cost for fixed value.
I also appreciated the very subtle implication that the OP is shrouding empire building under an otherwise informative growth story.
Re: Building a data team at a mid-stage startup
#89Will wait for a follow up post on how decentralised data team created data silos and how we solve it using data discovery and data standardisation. :P
Disclaimer: I have built decentralised data teams and it scales well.
Re: Building a data team at a mid-stage startup
#90Wow, a story where things start out a mess and end up a lot better! Can we write one of these for society too?
See also: The Algebra Project https://algebra.org/wp/