Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Adventures in Improving AI Economics

a16z.com

Post reply on HN
Print view View original

61–70 of 80 posts

Re: Adventures in Improving AI Economics

#61

Post by visarga »

eanzenberg wrote: post #60
visarga wrote: post #55

Earlier quoted context omitted.

Yes, I've come to the same conclusion. When interviewing ML engineers, I prefer to know they are exceptional programmers with passable knowledge of ML than the other way around. If they haven't learned to be good software engineers it's improbable they will in the future, but ML can be learned. In fact a ML team needs a large number of regular software engineers, there's a lot of non-ML code to work on, such as label…

I wouldn’t have ML engineers doing ML. They should be working on scaffolding, maintenance, production side, etc. The worst ML scientists (producing ML models) I’ve experienced were software developers who transitioned.

You just need one ML scientist for 4-5 software (or ML) engineers. If you wand to optimise time to delivery of products, you have much more to gain by improving the software engineering part, because regular SWE it's 90% of the product.

One of the main differences between ML in academia and industry is related to sourcing the training data. In academia they just use available pre-tagged datasets such as ImageNet, in industry you have to collect, clean up, organise, train, iterate with new data.

Re: Adventures in Improving AI Economics

#62

Post by sbierwagen »

jointpdf wrote: post #35
mdorazio wrote: post #11

Earlier quoted context omitted.

> Tesla has more than enough data to unlock autonomous vehicles Many people in the automotive industry, myself included, disagree with this statement pretty strongly. Driving data quantity is not equivalent to quality and they are severely lacking in advanced sensor data.

So is the claim by Elon Musk that current iterations of Tesla vehicles have all of the sensors and compute power needed to be fully autonomous (Level 4+ I guess?) in the future, via software updates only, a specious one?

Musk also said he was taking Tesla private at $420 a share, funding secured.

He says a lot of things.

Re: Adventures in Improving AI Economics

#63

Post by nl »

visarga wrote: post #57
nl wrote: post #38

Earlier quoted context omitted.

Then you haven't really looked. Most credible machine learning systems work well on unseen data, which by definition isn't memorizing.

> Most credible machine learning systems work well on unseen data, which by definition isn't memorizing. Sorry, but no. ML models don't generalise well outside the training data, but they can interpolate inside. This question becomes very interesting in the case of GPT-3 which has had a huge corpus of text to train on, so it's probably seen 'everything'. It's still memorising for GPT-3 but also learning to manipulate…

ML models don't generalise well outside the training data, but they can interpolate inside.

I'm unsure if you just misstated this or don't know, but this is wrong.

ML models don't generalise well on data outside the distribution of their training data. But that's an entirely different thing, and doesn't mean at all they are memorising data.

Imagine something training on the US unemployment rate until 2020 being hit with the COVID rate. It wouldn't know what to do, but that doesn't mean it wouldn't work fine on a rate of 5.342% even if it had never seen that rate before.

This is a simplified example, but applies to everything.

GPT-3 generation of text does pull from memorised training data. There's a lot of stuff going on there, and amongst other things there has never really been a system that does textual generation well. It's also hugely overparameterised, so lots of potential for overfitting. I don't think it's a good example of a "good" AI system - it's very interesting, full of potential, but there are lots of issues.

Re: Adventures in Improving AI Economics

#64

Post by blueblisters »

eanzenberg wrote: post #19

In my experience, there are just a lot of "bad" AI/ML engineers who don't fundamentally understand what data can do, what ML algorithms can handle, and how to piece it together to produce something of value to the end user. A couple of these people on a team can torpedo a project. Worse are those who sabotage projects or are general pain points of hindering progress. These may be jaded people who don't believe that M…

There are people all across the system who fail have a poor understanding of how machine learning works, especially in the enterprise AI market. The most common hurdle I have seen is having to explain to salespeople and customers that models are not perfect - they often fail on known unknowns and unknown unknowns. No matter how simple to understand you try to make it using charts and lift curves, every model failure has a potential for customer escalation, costing hundreds of engineering and support hours and depressing margins.

Re: Adventures in Improving AI Economics

#65

Post by mensetmanusman »

mdorazio wrote: post #11
mensetmanusman wrote: post #3

Good analogy about discovery of Pharma molecules. It’s really fun to think about the fact that Tesla has more than enough data to unlock autonomous vehicles, but all that is missing is the correct AI architecture to get it working... Who will figure out how to code that? Will it be a breakthrough, or can sub-optimal architectures eventually reach equilibrium with 10x or 100x the amount of time/data processing.

> Tesla has more than enough data to unlock autonomous vehicles Many people in the automotive industry, myself included, disagree with this statement pretty strongly. Driving data quantity is not equivalent to quality and they are severely lacking in advanced sensor data.

I don’t know, the existence proof is that it takes a 16 year old a few days of driving before they get it well enough...

Re: Adventures in Improving AI Economics

#66

Post by PaulHoule »

Gr8 article.

I'd add that caveat that software dev processes can be well controlled or not well controlled. AIML is not so much a new kind of project but it is a project likely to be poorly controlled.

Another thing they don't mention is that AIML projects break the agile assumption that you can manage with only punchclock, not calendar time.

Imagine you have a 2 week sprint and it takes 1 week to train a model. You have to get the training started in the first week, and any tasks that need to be done to start training have to start before that.

This of course means applying PERT chart thinking even if you don't make PERT charts. It often isn't that hard but in an agile shop that mistakes the map for the territory they will start the 1 week job consistently on the last day of the sprint.

The 'containerization' process they describe is close to the methods used by East coast defense contractors (in a band between research triangle park and the applied physics dept at John Hopkins in baltimore) to get high accuracy. Also they were what IBM Watson did as opposed to what people thought they did.

It's amazing those methods have remained so obscure, but the mind that is impressed with BERT is going to be impervious to asymtopes. That article should be telling people to run not walk away from those kind of models -- it is how you always be a bridesmaid but never a bride.

Re: Adventures in Improving AI Economics

#67

Post by mlthoughts2018 »

oliver101 wrote: post #10

> This is the crux of the AI business dilemma. If the economics are a function of the problem – not the technology per se – how can we improve them? The article focusses on the costs of resources to build a model (annotated data + compute) but the economics are also affected by the ongoing cost of making a prediction error. False positives and false negatives usually have a different cost and each user might have the…

Great way of putting it. The trouble there is that it takes an exceptional kind of senior ML person to basically wear a product manager hat all the time and press to translate project success criteria into revenue impact or cost reduction terms.

Having these “glue people” that connect ML engineering to product management is probably the most important thing to running an ML organization.

Re: Adventures in Improving AI Economics

#68

Post by jrib »

visarga wrote: post #55
jahewson wrote: post #41

Earlier quoted context omitted.

This is equally true if you replace “ML engineer” with simply “engineer”.

Yes, I've come to the same conclusion. When interviewing ML engineers, I prefer to know they are exceptional programmers with passable knowledge of ML than the other way around. If they haven't learned to be good software engineers it's improbable they will in the future, but ML can be learned. In fact a ML team needs a large number of regular software engineers, there's a lot of non-ML code to work on, such as label…

> ML can be learned

Curious: do you have a recommended pathway for that?

Re: Adventures in Improving AI Economics

#69

Post by PaulHoule »

RussianCow wrote: post #59
logicslave12 wrote: post #58

Earlier quoted context omitted.

Real ML and not just plug and play models takes a serious amount of knowledge

On the other hand, unless ML is your core competency as a business, plug and play models can get you really far.

When you consider the chain from inputs to action you can go far with a small toolbox (e.g. logistic regresssion.)

Plug-and-Play models in visual recognition basically work at this point in time; in NLP they don't.

Re: Adventures in Improving AI Economics

#70

Post by PaulHoule »

nl wrote: post #47
KorfmannArno wrote: post #45

Earlier quoted context omitted.

Can you link an example you find to generalize particularly well?

Sure. Apple's Face Unlock. It generalises to almost every face, and the ones it doesn't its failure mode is safe. Or something like word embeddings. Works incredibly well, and most "failure" modes are around things like bias, where the behavior reflects the real world. Or something like AlphaZero. Not only is every new game of Go it plays brand new, it learnt to play Chess without knowing the rules. That just isn't m…

Word embeddings suck. Take a look at the graphs under " 2. Linear substructures" in

https://nlp.stanford.edu/projects/glove/

and note that they all involve looking at a small number of points. It is easy to reproduce plots like that but if you try to increase the number of points the result breaks down completely.

It is a curve fitting problem: for a small enough set of points compared to the number of dimensions, you can find a matrix that projects a set of random points to an exactly specified set of points in the plane. If you relax the problem to something like "put colors on the left side, put smells on the right side" you will get better than random performance from that kind of model, but not that much better than random.

Word embeddings are a strategy that approaches an asymptote. Systems that are destined to low performance will perform better if you use a word embedding, but they throw away information up front that makes high performance impossible.

Post reply on HN