Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Adventures in Improving AI Economics

a16z.com

Post reply on HN
Print view View original

51–60 of 80 posts

Re: Adventures in Improving AI Economics

#51

Post by KorfmannArno »

nl wrote: post #47
KorfmannArno wrote: post #45

Earlier quoted context omitted.

Can you link an example you find to generalize particularly well?

Sure. Apple's Face Unlock. It generalises to almost every face, and the ones it doesn't its failure mode is safe. Or something like word embeddings. Works incredibly well, and most "failure" modes are around things like bias, where the behavior reflects the real world. Or something like AlphaZero. Not only is every new game of Go it plays brand new, it learnt to play Chess without knowing the rules. That just isn't m…

https://arxiv.org/pdf/1805.12177.pdf

I only read the abstract.

Re: Adventures in Improving AI Economics

#52

Post by nl »

KorfmannArno wrote: post #51
nl wrote: post #47

Earlier quoted context omitted.

Sure. Apple's Face Unlock. It generalises to almost every face, and the ones it doesn't its failure mode is safe. Or something like word embeddings. Works incredibly well, and most "failure" modes are around things like bias, where the behavior reflects the real world. Or something like AlphaZero. Not only is every new game of Go it plays brand new, it learnt to play Chess without knowing the rules. That just isn't m…

https://arxiv.org/pdf/1805.12177.pdf I only read the abstract.

Not sure what you think this shows, but there's a lot of reasons why the results they show don't really matter much - or at least might actually reflect an accuracy approximatly what a human would also achieve.

Their headline claim is a "1 pixel change reduces accuracy by 30%". The test process for that number is this:

We choose a random square within the original image and resize the square to be 224x224. The size and location of the square are chosen randomly according to the distribution described in (Szegedy et al., 2015). We then shift that square by one pixel diagonally to create a second image that differs from the first one by translation by a single pixel.

So... they are taking a random square, downsampling to 224, moving and then predicting on that subset of the original image, and measuring the performance against the accuracy of the original prediction.

What this seems to show is that "CNNs aren't as accurate at making predictions on subsets of an image as on the whole image". This is of course to be expected, and is exactly how a human would perform.

There are a bunch of other criticisms too.

Read the ICLR (for which it was rejected) reviews: https://openreview.net/forum?id=HJxYwiC5tm

Re: Adventures in Improving AI Economics

#53

Post by motohagiography »

jmatthews wrote: post #31
motohagiography wrote: post #18

Good analysis and great of them to share their thinking. Does feel like this could have been a tweet that said the necessary condition for successful ML solution is applying it to a problem that has asymmetric upside. Great for telling people they should get tested for diseases, terrible for diagnosis. In the alerting first case, consequences of being wrong are no better than base rate as they wouldn't have been test…

I appreciate the thoughtful commentary. I couldn't disagree more with you more of course. There are 2 instances where AI breaks the mold you've cast. Executing rote tasks that no humans need do, and relatedly, while there does seem to be a tough hurdle when it comes to "better than human" execution there is also an inverted survivors bias. Once a technology is production ready it is no longer AI. Cars aren't robots,…

Our disagreement might be subtle. An old saw of mine is that the Turing test thought experiment is covered by prior art in economics, where the idea of an indifference curve describes the points between amounts of things where people are indifferent to substituting between them.

I agree these things you state aren't intelligent, but nor are computers, or can they be - people just become indifferent to whether we are dealing with a human or a computer.

My assertion is that we are highly sensitive to substitutes when the downside risk is large, but largely indifferent to them and even like them when they resemble a lottery with good upside at low cost or risk.

Self driving cars are a good example, where someone asked me whether, if I had kids, would I send one to school in traffic in an autonomous vehicle. I told them it would depend on how many kids I had.

But this pretty much describes the dynamic.

Re: Adventures in Improving AI Economics

#54

Post by Jack000 »

kippinitreal wrote: post #48
Jack000 wrote: post #42

Earlier quoted context omitted.

This is a great feature of the AI space for startups - in the short term it reduces competition, in the long term it's not really a problem. If your business is break-even currently, it will be profitable in 2-3 years due to declining cost of compute. In 10 years the compute costs will fall by an order of magnitude and more efficient models will become available, making the economics closer to traditional SaaS. This…

Not true if there’s enough competition that you need more resources for a bigger model in 2-3 years. Anecdotally, it seems like SOTA model training costs are rising much faster than computer cost is falling.

maybe, but model quality doesn't scale linearly with model size. The performance/dollar metric is more important, and that definitely will decline over time.

Personally, I have a boots trapped business that uses a transformer model. I'm not worried about supermassive models like gpt3 because it would be way too expensive to deploy for my use case, with marginal additional quality.

Re: Adventures in Improving AI Economics

#55

Post by visarga »

jahewson wrote: post #41
eanzenberg wrote: post #19

In my experience, there are just a lot of "bad" AI/ML engineers who don't fundamentally understand what data can do, what ML algorithms can handle, and how to piece it together to produce something of value to the end user. A couple of these people on a team can torpedo a project. Worse are those who sabotage projects or are general pain points of hindering progress. These may be jaded people who don't believe that M…

This is equally true if you replace “ML engineer” with simply “engineer”.

Yes, I've come to the same conclusion. When interviewing ML engineers, I prefer to know they are exceptional programmers with passable knowledge of ML than the other way around. If they haven't learned to be good software engineers it's improbable they will in the future, but ML can be learned. In fact a ML team needs a large number of regular software engineers, there's a lot of non-ML code to work on, such as labelling interfaces, data pipelines and CI/CD for models.

Re: Adventures in Improving AI Economics

#56

Post by visarga »

KorfmannArno wrote: post #27
antipaul wrote: post #4

Earlier quoted context omitted.

Perhaps. It seems it’s still an open question whether AI is just about memorizing your data, or can it actually make reliable decisions during previously unseen scenarios. Have we already observed, or collected, all that is possible in the “driving” world?

I've yet to see an "AI" that is not just memorizing data.

Just memorising data is simple, use a file on disk. The hard part is recognising data when it is slightly different than what the model 'memorised', deciding which of the millions of things it learned best fits the answer we desire.

Re: Adventures in Improving AI Economics

#57

Post by visarga »

nl wrote: post #38
KorfmannArno wrote: post #27

Earlier quoted context omitted.

I've yet to see an "AI" that is not just memorizing data.

Then you haven't really looked. Most credible machine learning systems work well on unseen data, which by definition isn't memorizing.

> Most credible machine learning systems work well on unseen data, which by definition isn't memorizing.

Sorry, but no. ML models don't generalise well outside the training data, but they can interpolate inside. This question becomes very interesting in the case of GPT-3 which has had a huge corpus of text to train on, so it's probably seen 'everything'. It's still memorising for GPT-3 but also learning to manipulate data, like software algorithms.

Re: Adventures in Improving AI Economics

#58

Post by logicslave12 »

visarga wrote: post #55
jahewson wrote: post #41

Earlier quoted context omitted.

This is equally true if you replace “ML engineer” with simply “engineer”.

Yes, I've come to the same conclusion. When interviewing ML engineers, I prefer to know they are exceptional programmers with passable knowledge of ML than the other way around. If they haven't learned to be good software engineers it's improbable they will in the future, but ML can be learned. In fact a ML team needs a large number of regular software engineers, there's a lot of non-ML code to work on, such as label…

Real ML and not just plug and play models takes a serious amount of knowledge

Re: Adventures in Improving AI Economics

#59

Post by RussianCow »

logicslave12 wrote: post #58
visarga wrote: post #55

Earlier quoted context omitted.

Yes, I've come to the same conclusion. When interviewing ML engineers, I prefer to know they are exceptional programmers with passable knowledge of ML than the other way around. If they haven't learned to be good software engineers it's improbable they will in the future, but ML can be learned. In fact a ML team needs a large number of regular software engineers, there's a lot of non-ML code to work on, such as label…

Real ML and not just plug and play models takes a serious amount of knowledge

On the other hand, unless ML is your core competency as a business, plug and play models can get you really far.

Re: Adventures in Improving AI Economics

#60

Post by eanzenberg »

visarga wrote: post #55
jahewson wrote: post #41

Earlier quoted context omitted.

This is equally true if you replace “ML engineer” with simply “engineer”.

Yes, I've come to the same conclusion. When interviewing ML engineers, I prefer to know they are exceptional programmers with passable knowledge of ML than the other way around. If they haven't learned to be good software engineers it's improbable they will in the future, but ML can be learned. In fact a ML team needs a large number of regular software engineers, there's a lot of non-ML code to work on, such as label…

I wouldn’t have ML engineers doing ML. They should be working on scaffolding, maintenance, production side, etc. The worst ML scientists (producing ML models) I’ve experienced were software developers who transitioned.
Post reply on HN