Nothing really new here that isn't covered in Lean Startup materials. >We've spent millions of dollars ... so you don't have to. This two person company that's had, from what I can tell, a hand full of clients, has spent millions of dollars testing their marketing? I find that a little hard to believe. That said, their Playbook page is pretty cool and informative, even if seemingly most of it is behind a paywall. It'…
We've actually just hired our 15th person and for the past 2 years we've worked with 70+ startups and spent around $2m. That's not counting one client who spends about $40k a day on FB ads. Before that I worked at Efficient Frontier (now part of Adobe), Travelzoo and ShopStyle. In each place I managed or optimized multi-million dollar budgets; the learnings from which I applied whilst building our playbook, and I apply every time I write a post.
ROI for our playbook is calculated based on the average of the experiments we've run. Looking at the numbers we've averaged about 2 concluded tests per tactic, but those numbers are highly skewed; there are some that have run 10+ times, others that have never 'run' (just things I've tried before but don't have conclusive data to hand, or things I've read about in blog posts). Either way, this data should be taken with a pinch of salt and used directionally; while eventually we dream of being able to build a recommendation engine that auto-prioritizes these tactics, the reality is that we'll need a lot more people using the system (hence the incessant blogging). :-)
Hope that helps explain where we're coming from?