Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ML promises to be profoundly weird

aphyr.com

Post reply on HN
Print view View original

401–410 of 641 posts

Re: ML promises to be profoundly weird

#401

Post by drob518 »

fwip wrote: post #320
drob518 wrote: post #177

Earlier quoted context omitted.

Which is what percent of the world’s content? 0.000000001% or something similar. It’s nothing in the scheme of things. To put it another way, if we were to digitize that continent and train on it, our AIs would not get noticeably better in any way. It doesn’t move the needle.

1.1 million being 0.000000001% implies a total count of 1e17 books in the world - the real number is closer to 1e8.

You’re missing the point. And we’re not just talking about books, whatever that might mean. We’re talking about all documents ever made. Every magazine article, every blog and web page, every Word doc, etc. I’m pretty sure that whatever is in the Vatican archives is tiny by comparison. Given the age of the Vatican archives, I can also guarantee that many of those “books” are nothing more than page fragments. Very few will be full codices or long scrolls. Many will date before the printing press when document production was slow and laborious.

Re: ML promises to be profoundly weird

#402

Post by xyzzyz »

munificent wrote: post #150

There is a whole giant essay I probably need to write at some point, but I can't help but see parallels between today and the Industrial Revolution. Prior to the industrial revolution, the natural world was nearly infinitely abundant. We simply weren't efficient enough to fully exploit it. That meant that it was fine for things like property and the commons to be poorly defined. If all of us can go hunting in the woo…

Prior to Industrial Revolution, nobody could go hunt in the woods, because the woods were King’s, and poaching King’s game carried death penalty. Situation was similar on the continent: the tiny slivers of remaining wood lands were off limits.

Granted, things were different in the New World, as a result of mass depopulation event following the Columbian exchange. But even there, the megafauna was hunted to extinction soon after the humans first appeared there.

Anyway, the point is that no, prior to Industrial Revolution, the world was of full of scarcity, not abundance.

Re: ML promises to be profoundly weird

#403

Post by Quarrelsome »

squigz wrote: post #396
Quarrelsome wrote: post #385

Earlier quoted context omitted.

> This is just wildly incorrect. from an global perspective it isn't. Some places sure, like Western Europe, who in some cases had completed enclosure, but remember the new world had only been discovered a few hundred years ago at that point. Just google maps the north part of South America, even today there are large swathes of undeveloped land across it and back then it was considerably less exploited. At that time…

> remember the new world had only been discovered a few hundred years ago at that point. By White people*

we're talking about the fucking industrial revolution, of course this defaults to the European perspective. Unless you wanna spit some new bars about Aztec foundries and train lines connecting meso-america in the 19th century, then the point stands. At that time, the world appeared to the industrialists of the industrial revolution to be infinite. Nor had humanity discovered the terrible side effects of fossil fuels on the atmosphere.

Why are you weirdly making this about race?

Re: ML promises to be profoundly weird

#404

Post by squigz »

Quarrelsome wrote: post #403
squigz wrote: post #396

Earlier quoted context omitted.

> remember the new world had only been discovered a few hundred years ago at that point. By White people*

we're talking about the fucking industrial revolution, of course this defaults to the European perspective. Unless you wanna spit some new bars about Aztec foundries and train lines connecting meso-america in the 19th century, then the point stands. At that time, the world appeared to the industrialists of the industrial revolution to be infinite. Nor had humanity discovered the terrible side effects of fossil fuels…

Sure, of course it's convenient to ignore the native peoples and pretend that prior to the Industrial Revolution the rest of the world outside of Europe was some untapped well of resources that Europeans had a natural right to.

Who might be swept underfoot in this "Information Revolution", I wonder?

Re: ML promises to be profoundly weird

#405

Post by Quarrelsome »

squigz wrote: post #404
Quarrelsome wrote: post #403

Earlier quoted context omitted.

we're talking about the fucking industrial revolution, of course this defaults to the European perspective. Unless you wanna spit some new bars about Aztec foundries and train lines connecting meso-america in the 19th century, then the point stands. At that time, the world appeared to the industrialists of the industrial revolution to be infinite. Nor had humanity discovered the terrible side effects of fossil fuels…

Sure, of course it's convenient to ignore the native peoples and pretend that prior to the Industrial Revolution the rest of the world outside of Europe was some untapped well of resources that Europeans had a natural right to. Who might be swept underfoot in this "Information Revolution", I wonder?

Yes, just the other day I saw someone make a comment about write performance in SQLite without considering the plight of the Baltic peoples in the Northern Crusades. It was really convenient of them to do that, fucking typical.

Re: ML promises to be profoundly weird

#406

Post by squigz »

Quarrelsome wrote: post #405
squigz wrote: post #404

Earlier quoted context omitted.

Sure, of course it's convenient to ignore the native peoples and pretend that prior to the Industrial Revolution the rest of the world outside of Europe was some untapped well of resources that Europeans had a natural right to. Who might be swept underfoot in this "Information Revolution", I wonder?

Yes, just the other day I saw someone make a comment about write performance in SQLite without considering the plight of the Baltic peoples in the Northern Crusades. It was really convenient of them to do that, fucking typical.

Sure, because working on a database plugin is the same as, for example, working on mass surveillance tech.

This sort of handwashing is exactly why the natives were treated the way they were.

Re: ML promises to be profoundly weird

#407

Post by famouswaffles »

joefourier wrote: post #334
zozbot234 wrote: post #329

Earlier quoted context omitted.

Transformers' greatest improvement over RNN/LSTM was to enable better parallelization of large-scale training. This is what enabled language models to become "large". But when controlling for overall size, more RNN/LSTM-like approaches seem to be more efficient, as seen e.g. in state space models. The transformer architecture does add some notable capabilities in accounting for long-range dependencies and "needle in…

With modern training techniques, RNNs (not just linear SSMs, potentially even vanilla LSTMs) can scale just as well as transformers or even better when it comes to enormous context lengths. Dot-product attention has better performance in a number of domains however (especially for exact retrieval) so the best architectures are likely to remain hybrid for now.

>With modern training techniques, RNNs (not just linear SSMs, potentially even vanilla LSTMs) can scale just as well as transformers or even better when it comes to enormous context lengths.

That's not true. Modern training techniques aren't enough. Vanilla RNNs with modern training techniques still scale poorly. You have to make some pretty big architectural divergences (throwing away recurrency during training) to get a RNN to scale well. None of the big labs seem to be bothered with hybrid approaches.

Re: ML promises to be profoundly weird

#408

Post by hatthew »

cgh wrote: post #164
drob518 wrote: post #151

Earlier quoted context omitted.

What large caches of undigitized content exists? Surely, not everything has been digitized, but I can’t think it’s much in percentage terms.

The Vatican Library contains roughly 1.1 million printed books and around 75,000 codices, only a small percentage of which have been digitised.

Reddit alone contains about the same quantity of text (~10 billion posts * 10 words per post, vs 1 million books * 100k words per book). Messaging and document platforms (google docs, slack, discord, telegram, etc.) probably each have 1-3 orders of magnitude more than reddit. To your/GP's point though, those private platforms probably haven't been slurped up by LLMs yet.

Re: ML promises to be profoundly weird

#409

Post by Quarrelsome »

squigz wrote: post #406
Quarrelsome wrote: post #405

Earlier quoted context omitted.

Yes, just the other day I saw someone make a comment about write performance in SQLite without considering the plight of the Baltic peoples in the Northern Crusades. It was really convenient of them to do that, fucking typical.

Sure, because working on a database plugin is the same as, for example, working on mass surveillance tech. This sort of handwashing is exactly why the natives were treated the way they were.

How do you think they're enabling the mass surveillance tech? SQLite got reach bruv.

Your continued erasure of the Baltic people's continues to cut deep into my heart, and your callous candour to their plight, as you discard any chance to mention them, continues to shock me.

Re: ML promises to be profoundly weird

#410

Post by elros »

ryandrake wrote: post #250
munificent wrote: post #173

Earlier quoted context omitted.

Agreed, totally! I still write and put stuff online. But it definitely feels different now. It used to feel like I was tending a public garden filled with other people who might enjoy it. It still kind of feels like that, but there are a handful of giant combine machines grinding their way around the garden harvesting stuff and making billionaires richer at the same time. It's not enough to dissuade me from contribut…

> But as Amazon got bigger and bigger and the externalities more apparent, it felt less like we were helping each other and more like we were help Bezos buy yet another yacht or media empire. The Internet-circulating quote comes to mind: Planet Earth is pretty much a vacation resort for around 500 rich people, and the remaining 8 billion of us are just their staff. The Relative Few have got the system set up perfectl…

> The Internet-circulating quote comes to mind: Planet Earth is pretty much a vacation resort for around 500 rich people, and the remaining 8 billion of us are just their staff. The Relative Few have got the system set up perfectly so that whatever we do, we're probably serving/enriching them. AI doesn't really change this, but it does further it.

I don't necessarily disagree with the analysis on how Planet Earth is currently setup to be, but something that I've been thinking about lately, is that to the extent we can consume the public image of some of the Relative Few, they seem oddly unhappy.

Post reply on HN