I view AI as the application of ML and ML as the implement (tool). Therefor, tooling efficiency is a competitive advantage of good ML projects.
Andreessen-Horowitz craps on “AI” startups from a great height
211–220 of 256 posts
Re: Andreessen-Horowitz craps on “AI” startups from a great height
#212Earlier quoted context omitted.
"The data" means more than pure computer science people want to admit. In any "advanced" application, that means annotators. Radiologists drawing circles around cancer, attorneys labeling contract clauses as unacceptable, drivers labeling stop signs, etc. ML is a mining problem. Digitizers are the miners. Annotators are the refiners.
This is correct, however, the distinction between labeling and training is artificial, and probably arises from the fact that ML came from academia, where it was not part of the business process. I.e. a modern ML system should just plug into the business process from day 0, where the ML task should be performed by human and recorded by the machine. After a while, the machine would train on this recorded data, and sta…
> a modern ML system should just plug into the business process from day 0, where the ML task should be performed by human and recorded by the machine.
Ah, this is a typical thing I hear people in the Valley say: just push it all ... somewhere. No.
If we digitized all microscopy slides, it would require YouTube-scale storage several times over. People think genomics is big. People think reconnaissance imaging is big. They're big, but there's only so much of them.
IF it were digitized, there would be far more pathology whole slide imaging being generated every day. I did some estimates at one point and had to throw a couple orders of magnitude into the genomics data to even make it competitive at enterprise scale.
And keep in mind, we're talking clinical medicine. We want the data now. We're looking at the slides while the glue is still wet. You don't have the bandwidth, no one has the bandwidth, to do some of this stuff they way you propose and maintain the current "business process" of clinical medicine.
Building models and iterating, the old fashioned way, is the only way it makes sense.
Re: Andreessen-Horowitz craps on “AI” startups from a great height
#213> In the old days of on-premise software, delivering a product meant stamping out and shipping physical media – the cost of running the software, whether on servers or desktops, was borne by the buyer. Today, with the dominance of SaaS, that cost has been pushed back to the vendor. Most software companies pay big AWS or Azure bills every month – the more demanding the software, the higher the bill.
This irrational sheep mentality amuses me. Yes, tehre are some very specific cases where AWS & ca. is clearly a better choice, but for the most cases I saw the TCO with hosting it on premises or renting servers is much lower, sometimes by an order of magnitude (in some cases even more). But people insist on doing it because others do it. We'll soon have an entire generation of engineers completely hooked on AWS & co. and not even realizing other solutions are possible, not to mention lower TCO.
Re: Andreessen-Horowitz craps on “AI” startups from a great height
#214I just spent $50K on coloc hardware. I'm taking a $10K/mo Azure spend down to a $1K/mo hosting cost. But the real kicker is that I get x5 the cores, x20 RAM, x10 storage, and a couple of GPUs. I'm running last-generation Infiniband (56gb/sec) and modern U.2 SSDs (say 500MB/sec per device). I figure it is going to take me about $10K in labor to move and then $1K/mo to maintain and pay for services that are bundled in…
the point of Cloud is that it solves the problem of variable demand.
I used to run on-prem back in the 2000's, and we were constantly dealing with demand fluctuation crises. Spinning up new physical servers to deal with new demand, or being massively over-specced when demand dropped, was a real pain.
I'm starting a new thing this week, and using the Cloud for it because I have no idea what our demand will be. I can start small, scale up with our customer growth, and never have to worry about ordering new servers a month in advance so I have enough capacity when (or if) I need it.
At some point in the future, when our needs are clear and relatively stable, it might make sense to migrate to on-prem and save those costs.
Re: Andreessen-Horowitz craps on “AI” startups from a great height
#215I interviewed at some AI companies a year or two back. They all had teams of people dedicated to support each client: to clean their data, train their models, integrate the domain-specific requirements, customize UIs, etc. They sold themselves as the next AI-powered mega-unicorns, but they were more like boutique consultancies with no obvious path to scale up.
Re: Andreessen-Horowitz craps on “AI” startups from a great height
#216I interviewed at some AI companies a year or two back. They all had teams of people dedicated to support each client: to clean their data, train their models, integrate the domain-specific requirements, customize UIs, etc. They sold themselves as the next AI-powered mega-unicorns, but they were more like boutique consultancies with no obvious path to scale up.
"Boutique Consultancy" is quite recapitulative for most AI companies for now. But this may be the only way to empower their clients. One of these startups will find the path to scale up eventually.
Re: Andreessen-Horowitz craps on “AI” startups from a great height
#217Earlier quoted context omitted.
because you are trading speed for cash. Say you have $8M in funding, and you need to train a model to do x You can either: a) gain access to a system that scale ondemand and allows instant, actionable results. b) hire a infrastructure person, someone to write a K8s deployment system. Another person to come in a throw that all away. Another person to negotiate and buy the hardware, and another to install it. Option b…
Why in the world do you need a Kubernetes deployment system to run a single, manual, one-time (or a handful of times), high-compute job?
because its not a one time operation.
also, how else do you sensibly deploy and manage a multi-stage programme on >500 nodes?
I mean we use AWS batch, which is far superior for this sort of thing. SLURM might work for real steel, as would tractor from pixar.
Re: Andreessen-Horowitz craps on “AI” startups from a great height
#218> most people haven’t figured out that ML oriented processes almost never scale like a simpler application would. You will be confronted with the same problem as using SAP; there is a ton of work done up front; all of it custom. I’ll go out on a limb and assert that most of the up front data pipelining and organizational changes which allow for [ML to be used operationally by an org] are probably more valuable than t…
That's because, ML and operations-research problems can be simplified to set of optimization problems and the underlying math and statistics are all very similar if not identical in some cases. And the input matters, a lot. So the differentiating factor isn't the models, it's the data and companies like Google figured it out a long time ago. In short, find interesting problems, then the solutions -- not the other way…
> And the input matters, a lot. So the differentiating factor isn't the models, it's the data and companies like Google figured it out a long time ago.
The models are likely also a differentiating factor in a sense that there are models that perform much better than others, to a point of completely new functionality. But also all of these models are basically open source currently... So they can't by definition be differentiating between different companies, because all of the companies generally have access to all of the algorithms. At leat to all of the types of algorithms.
Re: Andreessen-Horowitz craps on “AI” startups from a great height
#219Earlier quoted context omitted.
In the nineties, there was a huge difference between 1995 and 1998. It wasn't all that apparent to some of us until later, but things moved really fast in that timeframe. The years leading up to 2000 were almost like the imagining of approaching an event horizon or asymptote.
What you’re describing is so hard to convey to people. In 1994 the we were building raised-floor data centers with halon for suppressors and marveling at our 2GB behemoth UNIX boxes. And writing our own web application framework using CGI. In ‘99 we were renting a suite at a colo and putting our own hardware there, running ColdFusion web apps. In ‘04 we were renting half a rack at the same colo and trying not to writ…
CGI to ColdFusion to Java servlets. Sounds enterprise-y.
Re: Andreessen-Horowitz craps on “AI” startups from a great height
#220I agree with the author's opinion about > I’ll go out on a limb and assert that most of the up front data pipelining and organizational changes which allow for it are probably more valuable than the actual machine learning piece. Especially at non-tech companies with outdated internal technology. I've consulted at one of these and the biggest wins from the project (I left before the whole thing finished unfortunately…
I was a consultant at one of the giant outsourcers and nod my head vigorously at this comment. The least sexy projects were MDM (master data management) but they were absolutely essential to the success of any other fancy analytics/BI/ML project.
Interestingly I too worked on MDM systems about ten years ago, when I was at IBM Research. Ironically, one of my first ideas for applying machine learning was in de-duplication of data in an MDM server. However the technology was a bit too primitive back in 2010 and the project was a hard sell so it was abandoned.