Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Past Performance is Not Indicative of Future Results (2020)

locusmag.com

Post reply on HN
Print view View original

241–250 of 285 posts

Re: Past Performance is Not Indicative of Future Results (2020)

#241

Post by amelius »

lelanthran wrote: post #226
gonehome wrote: post #135

Earlier quoted context omitted.

I'm not sure I buy that - biology is often messier because of nature related constraints, it gets optimized for other things (energy, head size, etc.) The way a plane flies is quite different than the way a bird flies in complexity - they share an underlying mechanism, but planes don't need to flap wings. It's possible that scaling up does lead to generality and we've seen hints of that. - https://deepmind.com/blog/a…

> Few would have predicted the results we’re seeing today in 2010. That's hardly accurate - didn't Musk and Co. promise self-driving cars by 2012 ? We're in 2020 , and the SDC's are great for making youtube videos, but not any good at piloting a vehicle without human intervention. Since the 90s it has been clear that the only thing holding back what we have today is limited processing power. While there may be some n…

Waymo is doing better, I believe.

> plot the improvements in AI and the usage of computational power for AI on the same chart.

Would that be meaningful? I mean, I use an infinity times the computational power for writing a letter than people did 100 years ago, still producing more or less the same results.

Re: Past Performance is Not Indicative of Future Results (2020)

#242

Post by lizardmancan »

lelanthran wrote: post #226
gonehome wrote: post #135

Earlier quoted context omitted.

I'm not sure I buy that - biology is often messier because of nature related constraints, it gets optimized for other things (energy, head size, etc.) The way a plane flies is quite different than the way a bird flies in complexity - they share an underlying mechanism, but planes don't need to flap wings. It's possible that scaling up does lead to generality and we've seen hints of that. - https://deepmind.com/blog/a…

> Few would have predicted the results we’re seeing today in 2010. That's hardly accurate - didn't Musk and Co. promise self-driving cars by 2012 ? We're in 2020 , and the SDC's are great for making youtube videos, but not any good at piloting a vehicle without human intervention. Since the 90s it has been clear that the only thing holding back what we have today is limited processing power. While there may be some n…

With animal intelligence all involved parts are optimized. Remove the thumb and catching a tennisball becomes many fold as complex. A self driving car is an attempt to make carts work without rails while behaving just like railed vehicles. its an unintelligent idea. Perhaps if roads grew naturally it would make some sense?

Some CAI guy made a facinating remark. Self driving cars are the ultimate weapon.

Currently the goal is to avoid all kinds of harm, when we have that it is easy to allow something specific. There could even be plausible deniability.

Where is the intelligence?

Re: Past Performance is Not Indicative of Future Results (2020)

#243

Post by jbuhbjlnjbn »

pnt12 wrote: post #222
epgui wrote: post #173

What baffles me is the number of humans who think they are in the personal possession of some super special sacred form of magical and unexplainable intelligence. "AI is just stats" yes, indeed, but so is human intelligence. In many ways, AI from 2010 was already better than human intelligence. Three remarks: - The task many people seem to be benchmarking against is not just a measure of general intelligence, but a m…

"We have a very selective memory indeed. We have absolutely terrible judgment, are super irrational, and pretty reliably make decisions that are against our own interests, " This is a really bad argument - human intelligence is not highly rational, but it is deeply nuanced, using social cues, emotions, instincts and a miriad of other things. Computers can never be anti-knowledge because they lack the free will and so…

They most certainly could learn to be that. But we don't want that in a machine.

The human body is also functioning like a machine, there is no magic, just new stuff build upon very old stuff.

Re: Past Performance is Not Indicative of Future Results (2020)

#244

Post by kittiepryde »

kaba0 wrote: post #224
dtech wrote: post #153

Earlier quoted context omitted.

> It’s hard to predict timelines for this kind of thing, and people are notoriously bad at it. Few would have predicted the results we’re seeing today in 2010. What would you expect to see in the years leading up to AGI? Does what we’re seeing look like failure? Few have predicted a reasonably-capable text-writing engine or automatic video face replacement, but many have predicted self-driving cars would have been re…

We used to think that machines would be bad at arithmetic and pure logical reasoning, and good at the more primitive animalistic ones, but it turns out the latter is a much harder problem. Also, self-driving cars were mostly hyped up by companies, FSD is quite obviously a hard problem, much closer to general intelligence than the average NN application.

When did we think that?

Re: Past Performance is Not Indicative of Future Results (2020)

#245

Post by SamoyedFurFluff »

mgraczyk wrote: post #199
cratermoon wrote: post #194

Earlier quoted context omitted.

Easy to fix technically , but first the issue must be recognized and demonstrated, then the delicate process of negotiating the social and economic realities in which the technology operates. And that's the problem with ML in general: its failure to recognize the implicit biases in choice of dataset and training and the resulting problems, of which Microsoft racist chatbot Tay[1] is merely the most blatantly ludicrou…

And the first cars didn't have seatbelts. It's fine, these are not complicated problems, and they are much easier to spot and fix than most problems in software engineering at scale. Don't be fooled by the negative PR campaigns and clickbait, there's no reason to be skeptical about ML in general because of this. Also, Tay attempted to solve a much harder problem than image classification. It's hard to build a safe hy…

Forgive me, because I’m not an expert in ML. If this is an easy problem to solve why is it still a problem years after it’s so widespread that msm both knows about it and have written continual investigative journalism about it? It’s clearly not cutting edge anymore once it gets to that point and yet it’s still a problem. Why?

Re: Past Performance is Not Indicative of Future Results (2020)

#246

Post by SamoyedFurFluff »

spacedcowboy wrote: post #80
3gg wrote: post #56

Earlier quoted context omitted.

> The deepmind team etc type of group who actually know what they’re doing and the boundaries of what they are working with. You claim that they "know what they are doing and the boundaries of what they are working with" -- and yet they recklessly make public a racist vision product?

I have a PhD in neural networks, haven't used it in many a year, but some of the knowledge is still there. Some of the memories of racking my brains to understand what the hell is going on are still there, too. It is easy to have a theory of what is going on, to model the processes of how things are playing out inside the system, to make external predictions of the system, and to be utterly wrong. Not because your mo…

Forgive me because I myself do not have a PhD in ML, but if this is hard (making a non-racist system) why are there not serious guardrails you prevent releasing racist systems to the public?

Re: Past Performance is Not Indicative of Future Results (2020)

#247

Post by Jetrel »

cycomanic wrote: post #227
Nevermark wrote: post #200

Earlier quoted context omitted.

Actually you do have to be an expert to make sweeping statements with any credibility in a young field making advances every day. Huge ones and surprising ones every year. If you can’t characterize the technical problem that creates a limitation then you are just expressing an uninformed opinion. Even if you were an expert!

Not to get into the rest of the discussion, but I disagree with the classification of ML as a young field. AI is an established field and I would argue that nothing in modern ML is _fundamentally_ so different that it would justify classifying it as a new field.

Yeah, it's almost as old as computing - likely 60-70 years old. The thing about it is we had the blueprints for a lot of stuff like neural networks almost at the dawn of computing, but it took almost half a century for us to even begin to try out some of the ideas, because the computing hardware wasn't even close - it would have been like trying to build a CPU out of vacuum tubes.

Once we finally had the tools to even start trying, in the late 80s/early 90s, it took us a very long time to "calibrate" these general ideas and figure out the "devils in the details" that were necessary to make certain ideas viable (for example, neural networks were discarded as a dead end in the 80s, and only considerably later were we able to discover that multi-layer networks essentially "salvaged" the idea).

Machine learning without the era of "modern computers" was a bit like flight before we'd really mastered the internal combustion engine - we understood quite a bit about it, and had theories about a lot of stuff (like the basic shape of a wing), and could successfully build gliders and such. Contrary to a lot of propaganda, the Wright Brothers didn't just arrive in the world like "lightning from a clear sky", but ... it had to become practical to do for us to then move on to putting the ideas through the paces, and all of the established theory from beforehand ran into the usual treatment of "no plan of battle survives contact with the enemy".

Re: Past Performance is Not Indicative of Future Results (2020)

#248

Post by hwillis »

Author makes some interesting parallels to infernal combustion engines not being possible without machine tools.

The Antikythera mechanism was built 1800 years before the first metal lathe. It is a fantastically sophisticated[1] clockwork with dozens of gears, concentric shafts, and brilliant, practiced fabrication. It is not a unique device. It was built by someone who knew what they were doing and had made this thing many times. It is obvious in the same way that you can tell when code was written from the start knowing how the finished product would look.

The device displays the relative positions of stars and planets from their underlying orbits, and was built with bronze hammers and some small fragments of steel. All that to say, you can do incredible things with practice, care, and tools that are thousands of years too primitive.

[1]: https://en.wikipedia.org/wiki/File:AntikytheraMechanismSchem...

Re: Past Performance is Not Indicative of Future Results (2020)

#249

Post by Mehdi2277 »

SamoyedFurFluff wrote: post #245
mgraczyk wrote: post #199

Earlier quoted context omitted.

And the first cars didn't have seatbelts. It's fine, these are not complicated problems, and they are much easier to spot and fix than most problems in software engineering at scale. Don't be fooled by the negative PR campaigns and clickbait, there's no reason to be skeptical about ML in general because of this. Also, Tay attempted to solve a much harder problem than image classification. It's hard to build a safe hy…

Forgive me, because I’m not an expert in ML. If this is an easy problem to solve why is it still a problem years after it’s so widespread that msm both knows about it and have written continual investigative journalism about it? It’s clearly not cutting edge anymore once it gets to that point and yet it’s still a problem. Why?

It's some work, but not hard to solve technically having been at companies that deal with very similar problems. The main difficulty is less technical and more investment needed vs value + investment is partly outside of the modeling engineers making the system. Part of the improvement can be done by classical computer vision techniques. But mixing classical computer vision techniques with modern ones both feels somewhat like a hack and complicates the system. The other big area though is dataset improvement. Engineers building ml systems and the people collecting and organizing the needed datasets are normally different people with mild connections to each other. For companies that rely mostly on existing datasets and finetune from them, having to add a data curation process is a big pain point. Most companies have immature data curation processes. Many of the popular open source ml datasets have poor racial diversity. The most popular face generation dataset is celebA, full of celebrities (mostly white ones).

Other issue is for many of these systems having a racial bias in the error rate has mild business impact which makes it harder to prioritize in fixing. Last issue the work needed to fix this tends to be less interesting than most of the other work to make the system.

So overall, the main issues are lack of good open source fair datasets with loose licensing, cross organizational need to solve it (engineers can not code up a fair dataset), and business prioritization.

edit: Also solve here is getting accuracy across races to be close not zero. ML models will always have an error rate and if your goal is 0 errors related to racial factors that is extremely hard. Modeling is about making estimates of data not knowing the truth of that data.

Re: Past Performance is Not Indicative of Future Results (2020)

#250

Post by Jetrel »

shannifin wrote: post #30

> I don’t see any path from continuous improvements to the (admittedly impressive) ‘machine learning’ field that leads to a general AI any more than I can see a path from continuous improvements in horse-breeding that leads to an internal combustion engine. While I also don't expect that AGI will emerge solely through optimizing statistical inference models, I also don't think "improvements to the machine learning fi…

It's worth pointing out that "machine learning" is a specific term of art, not a term for AI in general. It refers very specifically to the type of "convolutional neural networks" that have made a bunch of progress over the past 15-25 years.

The moment you have a paradigm shift, sure, it can be considered "learning done by machines", but it's not "Machine Learning™" anymore.

--

This is why the author put it in quotes; because, since it's a term comprehensible to anybody, it's got this unfortunate side effect where people on the outside of the field take the "plain english" meaning of it rather than realizing it's loaded with some extra specific meaning for the practitioners in the field.

Post reply on HN