Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will we run out of ML data? Evidence from projecting dataset size trends (2022)

epochai.org

Post reply on HN
Print view View original

111–120 of 126 posts

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#111

Post by lostmsu »

amscanne wrote: post #106
lostmsu wrote: post #92

Earlier quoted context omitted.

I don't think you correctly calculate bandwidth in this case. You assume 1 bit per neuron per tick, but time when it fires within the tick also matters, and that information is missing from multipliers. Also, there's no reason to use data from optical nerves as input, as it is already precompressed. You should be counting optical receptors instead (120 000 000).

I don’t think it matters that much. The firing itself takes a couple of milliseconds, and there’s a refactory period of a millisecond. I’m approximating 250hz as the maximum rate of firing. You’re arguing that the neuron can encode more information with the phase (e.g. fire, recover, wait 2ms, fire) but I think information theory tells us the 250hz actually still bounds the information. Maybe there’s a small constant…

I disagree with the later point, as unlike camera sensors the cells in question already include trainable parameters for every single one of 100M+ inputs.

But it matters little as even with 100x reduction the estimate blows GPT out of the water in the first year, making it very sample inefficient in comparison.

As for signal I am a layman in its most extreme here (only mist-like idea about information theory and frequency relationship), but don't the bandwidth limits only apply to fixed rate measurements? E.g. there's basically infinite (sans plank limits) number of values between 4ms and 5ms and as long as the receiver can separate them, they can encode information?

To put it in other words, if the neurons can control the impulse peak delay down to a nanosecond, then shouldn't the limit be measured based on 10^9Hz of that control vs 250Hz of max firing rate?

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#112

Post by eru »

jacooper wrote: post #109
replygirl wrote: post #103

Earlier quoted context omitted.

What you're responding to doesn't propose the abolition of obligatory work or private capital, it proposes a decrease in labor hours commensurate with, or conservative in comparison to, an expected increase in productivity

Yeah sorry about that, but do you think its realistic? I mean productivity has been going high since a long time yet we are still 5 or 6 workweek.

Now that's a reasonable debate we can have!

Gains from increases in productivity in the last hundred years[0] seem to be spread between more consumption, shorter working hours[1].

Some people expected that most gains would go towards decreased working hours instead of the spread we have actually seen. Not sure there's much significance behind that?

[0] Or any span of time you might want to pick.

[1] And bigger bureaucratic overheads, but you can count that either as a weird form of consumption or as just productivity not having increased quite as fast.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#113

Post by eru »

sebzim4500 wrote: post #91
eru wrote: post #73

Earlier quoted context omitted.

They still take in a lot of sensory data, eg related to touch and proprioception.

True but surely that doesn't come to terabytes of data

I would expect similar amount of data (or more) compared to usual visual inputs.

After all, your skin is a pretty big organ. And your sense of balance and proprioception in all the joints is quiet a few different channels and pretty high temporal resolution.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#114

Post by eru »

auggierose wrote: post #108
eru wrote: post #68

Earlier quoted context omitted.

Seems unlikely, that never worked in the past. And humans don't actually use logic (especially formal logic) to come up with anything. They just use it to justify what they came up with. Not even mathematicians think in terms of logic when trying to solve problems.

Of course mathematicians also think in terms of logic. It’s what you learn when you study mathematics, you soak it up automatically, although few study logic explicitly. And before 2015 a machine beating worlds best go player also seemed pretty unlikely.

I've studied mathematics.

You only do (formal) logic as an afterthought when communicating your proofs to other people or writing them down. Otherwise it's mostly intuition.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#115

Post by eru »

subjectsigma wrote: post #102
eru wrote: post #82

Earlier quoted context omitted.

Interesting. Do you have any sources on Google Translate using rule-based reasoning?

Machine Translation, by Thierry Poibeau, 2017.

Alas, that was around the time Google Translate switched to Neural Networks:

See https://blog.google/products/translate/found-translation-mor... and https://en.wikipedia.org/wiki/Google_Neural_Machine_Translat...

It doesn't look like they are still using any rule-based reasoning?

The blog post says:

> With this update, Google Translate is improving more in a single leap than we’ve seen in the last ten years combined. [...]

Which seems pretty strong evidence to me that moving away from rule-based reasoning or even a hybrid approach that includes rule-based reasoning, was a clear win?

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#116

Post by auggierose »

eru wrote: post #114
auggierose wrote: post #108

Earlier quoted context omitted.

Of course mathematicians also think in terms of logic. It’s what you learn when you study mathematics, you soak it up automatically, although few study logic explicitly. And before 2015 a machine beating worlds best go player also seemed pretty unlikely.

I've studied mathematics. You only do (formal) logic as an afterthought when communicating your proofs to other people or writing them down. Otherwise it's mostly intuition.

I've studied mathematics, too. Yes, formal logic is an afterthought when you do mathematics. But formal logic is just an explicit representation of what goes on internally in a mathematician. Or at least that's how I approach formal logic (most logicians don't). I would describe these internal processes inside a mathematician (and outside, when used for communication) as intuition + "logic to keep intuition in check". Sounds like ML + logic to me.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#117

Post by auggierose »

blatant303 wrote: post #83
auggierose wrote: post #62

Earlier quoted context omitted.

But we can devise generally applicable rules of reasoning from first principles. It's called logic. I am pretty sure the next step is to properly combine machine learning and logic properly.

First principles don't work in the space of systems geared towards extreme generalization such as LLMs. You need to be ready to compare anything with anything and build bridges between many principles. In fact there is a deep link between the progress of structuralism in mathematics culminating with homotopy type theory and its parallel (r)evolution in the humanities with the discovery of manuscripts by the founder o…

There is no reason why logic cannot follow various different threads of reasoning, interweave them, merge them, split them again, etc. Logic constitutes a first principle of utmost generality, actually I cannot imagine anything more general. Identity is not equivalent to equivalence, equivalence is a quotient of identity, consisting of two classes: Those values which are identical to True, and those which are not.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#118

Post by HybridCurve »

This take is a bit silly in that they are implying the problem training models will be that we will run out of data. It's more likely that the problem is that the current models require too much data to reach convergence.

We've been trying to speed run neural networks science for the past decade but we still don't fully understand how they work. It's like being a bad programmer who doesn't understand algorithms so you compensate by spending money on hardware to make your programs run faster. At some point we will reach a limit where you can't buy your way out of the problem with more data or money and we'll all be forced to return to studying the foundations of the science rather than just trying to scale the existing models up.

I am certain when we get to that point everyone will realize we've been trying to feed these models too much data. It makes more sense that our current architectures are just not effective at assimilating the data they have.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#119

Post by blatant303 »

auggierose wrote: post #117
blatant303 wrote: post #83

Earlier quoted context omitted.

First principles don't work in the space of systems geared towards extreme generalization such as LLMs. You need to be ready to compare anything with anything and build bridges between many principles. In fact there is a deep link between the progress of structuralism in mathematics culminating with homotopy type theory and its parallel (r)evolution in the humanities with the discovery of manuscripts by the founder o…

There is no reason why logic cannot follow various different threads of reasoning, interweave them, merge them, split them again, etc. Logic constitutes a first principle of utmost generality, actually I cannot imagine anything more general. Identity is not equivalent to equivalence, equivalence is a quotient of identity, consisting of two classes: Those values which are identical to True, and those which are not.

> Identity is not equivalent to equivalence

When talking about identity/equivalence of types in the context of homotopy type theory, yes. This is literally what the univalence axiom states.

Auggierose, I'm curious about your thoughts on how we can provide more rigor to LLMs when it comes to large-scale program transformations and proof synthesis. Given the complexity and versatility of these systems, what kind of foundational framework do you believe would enable GPT and similar models to synthesize and execute proofs rigorously? How can we ensure that they are both reliable and adaptable while dealing with various mathematical and logical domains?

More importantly, how whould this relate to NLP tasks such as: alright, the story is good, but can you rewrite it in the style of Auggierose ?

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#120

Post by [deleted] »

airgapstopgap wrote: post #40
nologic01 wrote: post #27

Brute force approaches always hit some wall. ML will be no different. In the decades to come it us quite likely that algorithms will develop in directions orthogonal to current approaches. The idea that you improve performance by throwing gazillions of data into gargantuan models might be even come to be seen as laughable. Keep in mind (pun) that the only real intelligence here is us, and we are pretty good at figuri…

We won't hit the wall. Somewhat counterintuitively, scaling datasets is the lazy and economical approach. If you have the compute already, might as well dig an OOM more text tokens. But there are other sources of data, and slightly different ways to utilize it. Multimodality, in very large training runs, will almost inevitably increase sample efficiency (for obvious reasons of context richness), synthetic data is alr…

[deleted]
Post reply on HN