Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Adventures in Improving AI Economics

a16z.com

Post reply on HN
Print view View original

71–80 of 80 posts

Re: Adventures in Improving AI Economics

#71

Post by nmca »

mdorazio wrote: post #11
mensetmanusman wrote: post #3

Good analogy about discovery of Pharma molecules. It’s really fun to think about the fact that Tesla has more than enough data to unlock autonomous vehicles, but all that is missing is the correct AI architecture to get it working... Who will figure out how to code that? Will it be a breakthrough, or can sub-optimal architectures eventually reach equilibrium with 10x or 100x the amount of time/data processing.

> Tesla has more than enough data to unlock autonomous vehicles Many people in the automotive industry, myself included, disagree with this statement pretty strongly. Driving data quantity is not equivalent to quality and they are severely lacking in advanced sensor data.

I'm likely biased because I spend some of my time doing perception research, but I find the "advanced sensors are necessary" argument so odd. We have clear evidence from humans that you don't need them. I expect we'll be doing this sort of thing [0] in toy dynamic scenes from monocular vision in ~1year, and in real-time on city scenes in ~2. Perception-wise, what more do you need?

Planning and control seem much harder, but that's not a sensing problem.

[0] https://nerf-w.github.io/

Re: Adventures in Improving AI Economics

#72

Post by PaulHoule »

visarga wrote: post #61
eanzenberg wrote: post #60

Earlier quoted context omitted.

I wouldn’t have ML engineers doing ML. They should be working on scaffolding, maintenance, production side, etc. The worst ML scientists (producing ML models) I’ve experienced were software developers who transitioned.

You just need one ML scientist for 4-5 software (or ML) engineers. If you wand to optimise time to delivery of products, you have much more to gain by improving the software engineering part, because regular SWE it's 90% of the product. One of the main differences between ML in academia and industry is related to sourcing the training data. In academia they just use available pre-tagged datasets such as ImageNet, in…

Wow! This is the elephant in the room that doesn't get talked about in that article at all. You're also the only person to mention it, despite the fact that it is the only essential step to a commercial machine learning project.

Re: Adventures in Improving AI Economics

#73

Post by MisterPea »

visarga wrote: post #61
eanzenberg wrote: post #60

Earlier quoted context omitted.

I wouldn’t have ML engineers doing ML. They should be working on scaffolding, maintenance, production side, etc. The worst ML scientists (producing ML models) I’ve experienced were software developers who transitioned.

You just need one ML scientist for 4-5 software (or ML) engineers. If you wand to optimise time to delivery of products, you have much more to gain by improving the software engineering part, because regular SWE it's 90% of the product. One of the main differences between ML in academia and industry is related to sourcing the training data. In academia they just use available pre-tagged datasets such as ImageNet, in…

Yep, you pretty much hit the nail on the head. A few people in this thread are equating ML scientist's responsibilities to those of an ML engineer, however, they are very different.

Production development of a model is a very hard problem and it's interesting because I see few companies trying to tackle it. One of them is tecton.ai (heard about them on SED) and I'll be interested to see how they evolve their feature set because it still seems incomplete.

Re: Adventures in Improving AI Economics

#74

Post by nl »

PaulHoule wrote: post #70
nl wrote: post #47

Earlier quoted context omitted.

Sure. Apple's Face Unlock. It generalises to almost every face, and the ones it doesn't its failure mode is safe. Or something like word embeddings. Works incredibly well, and most "failure" modes are around things like bias, where the behavior reflects the real world. Or something like AlphaZero. Not only is every new game of Go it plays brand new, it learnt to play Chess without knowing the rules. That just isn't m…

Word embeddings suck. Take a look at the graphs under " 2. Linear substructures" in https://nlp.stanford.edu/projects/glove/ and note that they all involve looking at a small number of points. It is easy to reproduce plots like that but if you try to increase the number of points the result breaks down completely. It is a curve fitting problem: for a small enough set of points compared to the number of dimensions, yo…

This is true, but I don't think you are using word embedding like most people use them.

The linear relationship between things like king/queen etc is a cute demo but not really useful or used in practice.

The real usefulness of word embeddings is that similar concepts are close to each other so they make a great representation for other models (vs something like TF-IDF). These days they have been mostly surpassed in terms of state of the art by full language models, but the point is that simple techniques like average embedding of words in sentences generalised really well to unseen data.

And if you add in subword embeddings they generalise to unseen words, too.

We could talk about how context lets language models do this even better, but I'm still back trying to persuade the OP that this isn't just memorisation and good ML models work well on unseen data!

Re: Adventures in Improving AI Economics

#75

Post by nl »

motohagiography wrote: post #53
jmatthews wrote: post #31

Earlier quoted context omitted.

I appreciate the thoughtful commentary. I couldn't disagree more with you more of course. There are 2 instances where AI breaks the mold you've cast. Executing rote tasks that no humans need do, and relatedly, while there does seem to be a tough hurdle when it comes to "better than human" execution there is also an inverted survivors bias. Once a technology is production ready it is no longer AI. Cars aren't robots,…

Our disagreement might be subtle. An old saw of mine is that the Turing test thought experiment is covered by prior art in economics, where the idea of an indifference curve describes the points between amounts of things where people are indifferent to substituting between them. I agree these things you state aren't intelligent, but nor are computers, or can they be - people just become indifferent to whether we are…

Self driving cars are a good example, where someone asked me whether, if I had kids, would I send one to school in traffic in an autonomous vehicle. I told them it would depend on how many kids I had.

Pretty sure that answer is much less convincing than you think.

Infact, I thought you were right until that, then realised that was an answer no parent would ever give which made me realise there's a lot missing in your hypothesis.

Re: Adventures in Improving AI Economics

#77

Post by PaulHoule »

nl wrote: post #74
PaulHoule wrote: post #70

Earlier quoted context omitted.

Word embeddings suck. Take a look at the graphs under " 2. Linear substructures" in https://nlp.stanford.edu/projects/glove/ and note that they all involve looking at a small number of points. It is easy to reproduce plots like that but if you try to increase the number of points the result breaks down completely. It is a curve fitting problem: for a small enough set of points compared to the number of dimensions, yo…

This is true, but I don't think you are using word embedding like most people use them. The linear relationship between things like king/queen etc is a cute demo but not really useful or used in practice. The real usefulness of word embeddings is that similar concepts are close to each other so they make a great representation for other models (vs something like TF-IDF). These days they have been mostly surpassed in…

It's not so straightforward to go from a word representation to a query, sentence, or document representation.

If you come from the tfidf direction you can first tune up BM25 or something based on the ks-divergence, then you can use a random matrix, LDA, or the deep-network autoencoder that I worked on that crushed conventional tfidf vectors to 50-d vectors.

(Like many things people want to apply word vectors to, you go from 50% accuracy here to 70%, but we know it because we tested it on TREC gov2)

Today I'm interested in systems that have an input-to-action orientation and there you have to be able to put together a story like: "these 10 messages are parsed correctly and not by accident" and that requires that certain 'king/queen' inferences be done correctly or alternately the system has paths to recover from missing an inference.

Often there is no path to go from "popular models in the new A.I." to "something that can serve customers off the leash" and that's the problem.

Now I do like subword embeddings, but that just points out the problem that there is no such thing as a "word".

Let me justify that.

You can split up English into words like "some text".split() but it is not easy to do it from audio. Speech is punctuated by silences, often in the middle of words whenever you make a "[st]op" sound enough that separating words is equivalent to the whole speech understanding problem.

We can turn words into subwords and mash them together with subwords to make words. (e.g. "Fourthmeal", "Juneteenth", "Nihilego")

Also there are many cases you can replace a phrase with a word or a word with a phrase. Putting 'word' at the center of a model means the system is going to be in trouble w/ linguistic phenomena that happen 30% of the time.

Re: Adventures in Improving AI Economics

#78

Post by pc86 »

blueblisters wrote: post #64
eanzenberg wrote: post #19

In my experience, there are just a lot of "bad" AI/ML engineers who don't fundamentally understand what data can do, what ML algorithms can handle, and how to piece it together to produce something of value to the end user. A couple of these people on a team can torpedo a project. Worse are those who sabotage projects or are general pain points of hindering progress. These may be jaded people who don't believe that M…

There are people all across the system who fail have a poor understanding of how machine learning works, especially in the enterprise AI market. The most common hurdle I have seen is having to explain to salespeople and customers that models are not perfect - they often fail on known unknowns and unknown unknowns. No matter how simple to understand you try to make it using charts and lift curves, every model failure…

Not trying to nit pick (but here we go), but allowing an escalation to cost hundreds of hours seems like a systemic corporate issue. It's absolutely a customer service hurdle, you can't just tell a customer sending in a ticket to shove it, and throwing something back them that they signed acknowledging it is never helpful. But even involving engineers in these cut-and-dry scenarios (whether the customer agrees or not) seems like a poor decision.

Re: Adventures in Improving AI Economics

#79

Post by nl »

PaulHoule wrote: post #77
nl wrote: post #74

Earlier quoted context omitted.

This is true, but I don't think you are using word embedding like most people use them. The linear relationship between things like king/queen etc is a cute demo but not really useful or used in practice. The real usefulness of word embeddings is that similar concepts are close to each other so they make a great representation for other models (vs something like TF-IDF). These days they have been mostly surpassed in…

It's not so straightforward to go from a word representation to a query, sentence, or document representation. If you come from the tfidf direction you can first tune up BM25 or something based on the ks-divergence, then you can use a random matrix, LDA, or the deep-network autoencoder that I worked on that crushed conventional tfidf vectors to 50-d vectors. (Like many things people want to apply word vectors to, you…

To expand on input/action thing I guess you have to deal with the issue of representation of opposites being similar in many representations.

That leads to parsing, which is... Ok a lot of the time and completely wrong sometimes, and it's difficult to know which is which.

I think that's one of the big problems in NLP still.

Re: Adventures in Improving AI Economics

#80

Post by nimithryn »

eanzenberg wrote: post #60
visarga wrote: post #55

Earlier quoted context omitted.

Yes, I've come to the same conclusion. When interviewing ML engineers, I prefer to know they are exceptional programmers with passable knowledge of ML than the other way around. If they haven't learned to be good software engineers it's improbable they will in the future, but ML can be learned. In fact a ML team needs a large number of regular software engineers, there's a lot of non-ML code to work on, such as label…

I wouldn’t have ML engineers doing ML. They should be working on scaffolding, maintenance, production side, etc. The worst ML scientists (producing ML models) I’ve experienced were software developers who transitioned.

Curious why you think the worst ML scientists were the former engineers (as someone who does a bit of both).
Post reply on HN