Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will we run out of ML data? Evidence from projecting dataset size trends (2022)

epochai.org

Post reply on HN
Print view View original

121–126 of 126 posts

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#121

Post by [deleted] »

auggierose wrote: post #116
eru wrote: post #114

Earlier quoted context omitted.

I've studied mathematics. You only do (formal) logic as an afterthought when communicating your proofs to other people or writing them down. Otherwise it's mostly intuition.

I've studied mathematics, too. Yes, formal logic is an afterthought when you do mathematics. But formal logic is just an explicit representation of what goes on internally in a mathematician. Or at least that's how I approach formal logic (most logicians don't). I would describe these internal processes inside a mathematician (and outside, when used for communication) as intuition + "logic to keep intuition in check"…

[deleted]

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#122

Post by YeGoblynQueenne »

eru wrote: post #115
subjectsigma wrote: post #102

Earlier quoted context omitted.

Machine Translation, by Thierry Poibeau, 2017.

Alas, that was around the time Google Translate switched to Neural Networks: See https://blog.google/products/translate/found-translation-mor... and https://en.wikipedia.org/wiki/Google_Neural_Machine_Translat... It doesn't look like they are still using any rule-based reasoning? The blog post says: > With this update, Google Translate is improving more in a single leap than we’ve seen in the last ten years combined.…

It's not "strong evidence" but a strong claim.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#123

Post by auggierose »

blatant303 wrote: post #119
auggierose wrote: post #117

Earlier quoted context omitted.

There is no reason why logic cannot follow various different threads of reasoning, interweave them, merge them, split them again, etc. Logic constitutes a first principle of utmost generality, actually I cannot imagine anything more general. Identity is not equivalent to equivalence, equivalence is a quotient of identity, consisting of two classes: Those values which are identical to True, and those which are not.

> Identity is not equivalent to equivalence When talking about identity/equivalence of types in the context of homotopy type theory, yes. This is literally what the univalence axiom states. Auggierose, I'm curious about your thoughts on how we can provide more rigor to LLMs when it comes to large-scale program transformations and proof synthesis. Given the complexity and versatility of these systems, what kind of fou…

I am not a fan of HOTT, as nobody managed to explain to me its supposed advantages in terms that didn't border on mysticism.

Anyway, your question is very interesting! :-)

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#124

Post by amscanne »

lostmsu wrote: post #111
amscanne wrote: post #106

Earlier quoted context omitted.

I don’t think it matters that much. The firing itself takes a couple of milliseconds, and there’s a refactory period of a millisecond. I’m approximating 250hz as the maximum rate of firing. You’re arguing that the neuron can encode more information with the phase (e.g. fire, recover, wait 2ms, fire) but I think information theory tells us the 250hz actually still bounds the information. Maybe there’s a small constant…

I disagree with the later point, as unlike camera sensors the cells in question already include trainable parameters for every single one of 100M+ inputs. But it matters little as even with 100x reduction the estimate blows GPT out of the water in the first year, making it very sample inefficient in comparison. As for signal I am a layman in its most extreme here (only mist-like idea about information theory and freq…

I don’t think the connections between the receptors (rods and cones) and the ganglion cells are “trainable”? You seem to be assuming Brain-like learning functionality inside the eye. I’m not a biologist, but I don’t think this is the case unless you’re considering evolution as training. If it were true, wouldn’t people have wildly varying fovea? I feel that these connections are anatomical and not learned in exactly the same way the number of arms, legs or teeth is not trainable.

Regarding the nanosecond point — I don’t believe that’s how information works, and there should be many obvious problems with the idea of an infinite information channel not to mention the obvious practical ones (propagation variability, lack of a reference point, etc.). There may be some optimizations, but generally the frequency (or frequency bandwidth, which is where the generic computing term comes from) determines the information capacity, and phase modulation doesn’t magically change this (it is actually what is used in many radio systems).

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#125

Post by airgapstopgap »

nologic01 wrote: post #50
airgapstopgap wrote: post #40

Earlier quoted context omitted.

We won't hit the wall. Somewhat counterintuitively, scaling datasets is the lazy and economical approach. If you have the compute already, might as well dig an OOM more text tokens. But there are other sources of data, and slightly different ways to utilize it. Multimodality, in very large training runs, will almost inevitably increase sample efficiency (for obvious reasons of context richness), synthetic data is alr…

You are masquerading personal preferences (and possibly professional interests) as rules of nature. If anything, Godlike conceit definetely applies to some ML accolytes. In any case, with your last point "we should allow complex smartness to emerge" you essentially agree with my point that new levels will emerge from orthogonal (new) directions. The good thing about brute force is that it summons so many resources it…

It is interesting that you don't even hide having strong personal emotional preference at stake. Now, does this not suggest that your predictions are a priori less credible, by your logic?

No, I don't think "orthogonal" directions will be fruitful.

I also disagree on evaluations. What you call brute search is not brute search at all, nor a deux ex machina, it is a lawful and honest method of algorithmic discovery of true regularities. "Smarter approaches", meanwhile, usually amount to stilted expressions of narcissism of researchers overly proud with having come up with shallow tricks aping some aspect of explicit human reasoning. They're not actually smart, nor do they work far outside of the toy distribution for which they were developed.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#126

Post by brianr »

haldujai wrote: post #9
brianr wrote: post #7

This analysis misses the impact of AI models being deployed, like is happening rapidly right now. Production applications built on AI will provide ample (infinite?) additional training data to feed back into the underlying models.

Not sure that synthetic or LLM-generated training data is as useful as human generated text. It seems "good enough" (for now) but synthetic makes up a very small proportion of the training set being used in current models that have been trained on it, if that proportion ends up being mostly synthetic we'll likely see whatever weird hallucinations and biases in the dominant backend (GPT4 or whatever) become amplified.…

Agree about synthetic data. My point is that AI-powered applications that are deployed in production generate more _real_ data which can be used for training. For example, self-driving cars generate tons of data about how their models perform, as a result of the cars driving around. Similarly, code-writing AI applications will generate feedback in the form of errors, logs, etc. which is can be fed back into the models as training data.
Post reply on HN