Lessons for Building AI-Driven Products
blog.insightdatascience.com
Lessons for Building AI-Driven Products
1–10 of 19 posts
Re: Lessons for Building AI-Driven Products
#2If I could add to this: realize that “AI” will be a small fraction of the overall product. And important one, one without which the product would likely not exist, but in terms of effort spent it will be small. So no, you don’t need an army of ML researchers to ship it. You need maybe one or two, and the rest of your money is better spent on hiring engineers, sales/bizdev people, and a great product manager. People don’t give a shit about models. They do give a shit about whether your product solves a problem. Use AI as a means to an end, not an end in itself. Focus narrowly on a concrete problem, don’t try to do everything at once.
Re: Lessons for Building AI-Driven Products
#3If I could add to this: realize that “AI” will be a small fraction of the overall product. And important one, one without which the product would likely not exist, but in terms of effort spent it will be small. So no, you don’t need an army of ML researchers to ship it. You need maybe one or two, and the rest of your money is better spent on hiring engineers, sales/bizdev people, and a great product manager. People d…
Yes, exactly. And most of the time, you can get by with a simple model like a random forest, so you just need to have a few analysts/engineers doing feature engineering, and feeding it into your model.
Almost everyone using more complicated architectures (e.g. neural networks) are adding unnecessary complexity. [1]
[1]: Unless you're doing computer vision.
Re: Lessons for Building AI-Driven Products
#4If I could add to this: realize that “AI” will be a small fraction of the overall product. And important one, one without which the product would likely not exist, but in terms of effort spent it will be small. So no, you don’t need an army of ML researchers to ship it. You need maybe one or two, and the rest of your money is better spent on hiring engineers, sales/bizdev people, and a great product manager. People d…
I used to think this but after working in a large image processing team, I've come around to believe it's something management just rationalizes. In a lot of cases you _do_ need a big team with diverse research skills, otherwise you are deeply unable to pivot your own products and services for changes customer demands or adapt to situations when new models or competition elevate the minimal level of features that customers expect.
Building this stuff in a way that won't fall apart is really hard and you need a lot of people who are happy to be focused on the integration, deployment, and maintenance for sure. But you do also need a lot of people who are dedicated to the research, prototype, and model performance understanding side too, without trying to squeeze extra work out of them by trying to sell roles as a "hybrid" between integration and research.
Saying that "people don't give a shit about models" is vacuous, because ultimately the stack trace of what they do give a shit about will bottom out at issues with models that require a diverse team of experts to debug and solve.
I think companies are still stuck in a dreamworld right now where they want to believe that very junior engineers with tangential interest in machine learning are going to slot in at your standard $130k - $160k job offer (in major urban areas) and do just as adequate of a job as senior experts who might earn twice that.
But it doesn't work, and this type of penny pinching on the modeling side, rationalized by a false belief that you only need a tiny bit of research expenditure, I think is responsible for many start-ups failing and many projects being unwound in larger corporations.
Re: Lessons for Building AI-Driven Products
#5If I could add to this: realize that “AI” will be a small fraction of the overall product. And important one, one without which the product would likely not exist, but in terms of effort spent it will be small. So no, you don’t need an army of ML researchers to ship it. You need maybe one or two, and the rest of your money is better spent on hiring engineers, sales/bizdev people, and a great product manager. People d…
I used to think this but after working in a large image processing team, I've come around to believe it's something management just rationalizes. In a lot of cases you _do_ need a big team with diverse research skills, otherwise you are deeply unable to pivot your own products and services for changing customer demands or adapt to situations when new models or competition elevate the minimal level of features that customers expect.
Building this stuff in a way that won't fall apart is really hard and you need a lot of people who are happy to be focused on the integration, deployment, and maintenance for sure. But you do also need a lot of people who are dedicated to the research, prototype, and model performance understanding side too, without trying to squeeze extra work out of them by trying to sell roles as a "hybrid" between integration and research.
Saying that "people don't give a shit about models" is vacuous, because ultimately the stack trace of what they do give a shit about will bottom out at issues with models that require a diverse team of experts to debug and solve.
I think companies are still stuck in a dreamworld right now where they want to believe that very junior engineers with tangential interest in machine learning are going to slot in at your standard $130k - $160k job offer (in major urban areas) and do just as adequate of a job as senior experts who might earn twice that.
But it doesn't work, and this type of penny pinching on the modeling side, rationalized by a false belief that you only need a tiny bit of research expenditure, I think is responsible for many start-ups failing and many projects being unwound in larger corporations.
Re: Lessons for Building AI-Driven Products
#6Question: looking at the article's "AI Product Management" infographic, one item includes the phrase, "Adopt a bread-first approach to building a product". Is that supposed to be "breadth-first"?
Re: Lessons for Building AI-Driven Products
#7Their lesson, "Build Breadth-first (Data/Pipeline/Model) instead of Depth-first (AI model)" is insightful. Once the pipeline/etc. are built perhaps the AI part is not even needed any more. Finding this out early can save you a lot of trouble.
Another perspective, in which we talk about on integrating AI into existing workflows, among other lessons:
Smith, Reid G., and Joshua Eckroth. "Building AI Applications: Yesterday, Today, and Tomorrow." AI Magazine 38.1 (2017): 6-22.
https://www2.stetson.edu/~jeckroth/downloads/smith-eckroth-2...
Re: Lessons for Building AI-Driven Products
#8If I could add to this: realize that “AI” will be a small fraction of the overall product. And important one, one without which the product would likely not exist, but in terms of effort spent it will be small. So no, you don’t need an army of ML researchers to ship it. You need maybe one or two, and the rest of your money is better spent on hiring engineers, sales/bizdev people, and a great product manager. People d…
Yes, exactly. And most of the time, you can get by with a simple model like a random forest, so you just need to have a few analysts/engineers doing feature engineering, and feeding it into your model. Almost everyone using more complicated architectures (e.g. neural networks) are adding unnecessary complexity. [1] [1]: Unless you're doing computer vision.
Agreed, we've also written about why the best use of your time is to start with a baseline (even in Computer Vision!) https://blog.insightdatascience.com/always-start-with-a-stup...
Re: Lessons for Building AI-Driven Products
#9As someone who is a product manager in the commercialized AI SaaS space, the most important pieces of feedback I would give a new PM here:
1)Don't let your -brilliant- colleagues try to force their -brilliantly complex- solution of a problem - clearly define market problems, and don't let the team try to go the route of trying to force fit a solution to a market problem. Market problems come first.
2)Frame the market problems appropriately for your ML/AI teams, and practice trying to frame the problem from a variety of angles. Framing from different angles promotes the 'Ah-ha' moment in terms of the right way to solve the problem from the ML side.
3)Don't commit serious time to a model before having a naive solution to benchmark against. Always have a naive solution to compare against the AI solution. 'Naive' here may be a simple linear regression, RMSE, or multi armed bandit/Thompson sampling.
Re: Lessons for Building AI-Driven Products
#10Earlier quoted context omitted.
Yes, exactly. And most of the time, you can get by with a simple model like a random forest, so you just need to have a few analysts/engineers doing feature engineering, and feeding it into your model. Almost everyone using more complicated architectures (e.g. neural networks) are adding unnecessary complexity. [1] [1]: Unless you're doing computer vision.
Agreed, we've also written about why the best use of your time is to start with a baseline (even in Computer Vision!) https://blog.insightdatascience.com/always-start-with-a-stup...
And before a baseline, start with formulating a problem in a way that’s applicable to a real world product. Not just “image classification” but “image classification to improve QA in underwater basket weaving, a 10 billion dollar industry, from which we could generate $100m/yr in revenue by selling a product that does A, B, and C to an estimated 100k customers each of which would have $1000 LTV”.
I’m in an AI startup right now where the founders are hell bent on doing nothing in particular, and I’ll be bailing in a month or so.