Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta AI Unleashes Megabyte, a Scalable Model Architecture

artisana.ai

Post reply on HN
Print view View original

21–30 of 213 posts

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#21

Post by intelVISA »

p-e-w wrote: post #15
nemothekid wrote: post #11

Earlier quoted context omitted.

> Think of a human with below average intelligence. Then think of a human genius. LLMs are not AGI. A human with below average intelligence is still a league above a chimpanzee. A chimpanzee will never be able to read, not because "it's too dumb", but because a chimp's brain lacks the actual hardware for reading. The LLM is the chimpanzee. The gap between an LLM and a "human with below average intelligence" is far mo…

> The gap between an LLM and a "human with below average intelligence" is far more than 10x. In which direction? GPT-4 passes the bar exam with a top 10% score. How do you think a human with below average intelligence (or even with average intelligence) would fare? Copilot generates programming code that solves problems, and in most cases the code is correct. It outperforms many junior professional developers. Do you…

> Copilot generates programming code that solves problems, and in most cases the code is correct. It outperforms many junior professional developers. Do you think a human with below average intelligence could do that?

Absolutely, it's why Javascript is so popular.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#22

Post by p-e-w »

slowmovintarget wrote: post #16
epups wrote: post #13

Earlier quoted context omitted.

By what criteria? In several professional capacities, including coding, LLM's are far superior to an average human already.

No they're not. They're superior at specific tests when prompted. But asking for a complete application nearly always fails, when an average human that knows programming should be able to make one that works.

> when an average human that knows programming should be able to make one that works.

Very few people who "know programming" are actually capable of creating a complete application that serves a specific purpose. Many professional programmers struggle to implement basic algorithms like prime number testing. Coding AIs absolutely outperform the average programming professional, because the average programming professional can barely program. The software industry's demand for programmers is far too great for every coding position to be filled with top-class full stack engineers.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#25

Post by p-e-w »

intelVISA wrote: post #21
p-e-w wrote: post #15

Earlier quoted context omitted.

> The gap between an LLM and a "human with below average intelligence" is far more than 10x. In which direction? GPT-4 passes the bar exam with a top 10% score. How do you think a human with below average intelligence (or even with average intelligence) would fare? Copilot generates programming code that solves problems, and in most cases the code is correct. It outperforms many junior professional developers. Do you…

> Copilot generates programming code that solves problems, and in most cases the code is correct. It outperforms many junior professional developers. Do you think a human with below average intelligence could do that? Absolutely, it's why Javascript is so popular.

A human with below average intelligence can outperform many junior professional software engineers because of JavaScript?

Do you mean a programmer with below average intelligence (for a programmer)? Because I'm having a hard time believing that you actually believe what you wrote.

BTW, JavaScript is actually a fairly difficult language to learn and use. Python, BASIC, and even Fortran are much simpler conceptually and have far fewer pitfalls. JavaScript is popular because it's the only language that every modern computer has an interpreter for, not because it's so easy that idiots can use it.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#26

Post by amoss »

doctor_eval wrote: post #19
vlovich123 wrote: post #12

Earlier quoted context omitted.

I don’t think I follow this argument. AI has been dropping in costs and complexity the more engineering time is spent on it. It seems like the bottleneck is humans creating new AI techniques right? If an AI is capable of developing new AI techniques unsupervised, isn’t that by definition the singularity? Heck doesn’t even need to be unsupervised. If it can even do most of the heavy lifting for a human I feel like tha…

I consider the singularity to be the point at which the certainty of our predictions about our future becomes close to zero. By this definition I reckon we are already in the singularity. It’s not necessarily bad. The problem with the singularity is that that we can’t tell if it’s bad or not.

Certainty about predictions of the future has always been close to zero. If you take a person from an appropriate time and ask them what the future will look like in 1000, 100 or even 20 years then their predictions will bear little resemblence to what actually occurs.

As humans we have a tendancy to make linear future predictions based on past observations. Over the timescales that matter significant effects occur from previously unseen kinds of events that become important via interactions with other events.

The important measurement would be what the length of the event horizon - the length of time before our predictions rapidly decrease to zero certainty. In a singularity we would expect that length of time to decrease close to zero. What has it been historically? I would claim that 5-10 years is a difficult period to make meaningful predictions about. I think that 20 years has proven to be very difficult but possible in the past. I am unaware of any 100+ year predictions that have landed with better than random chance.

Interestingly I think we are entering a period where 5 years will be the upper bound, and even predictions over shorter 2-3 year timespans are going to become difficult.

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#28

Post by [deleted] »

jacobn wrote: post #3

My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall). This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of t…

[deleted]

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#29

Post by BulgarianIdiot »

visarga wrote: post #20
p-e-w wrote: post #4

Earlier quoted context omitted.

I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It's not like one has 10x the number of neurons/synapses/connections etc. of the other. They're both healthy human brains, and you need pow…

> I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Scaling can happen in two dimensions - model size and dataset size. What counts is the product of n_examples x n_parameters. That's why we have the super-Chinchilla laws, where n_examples >> 20*n_parameters. Scale the data, keep the model lean. Not to mention dataset quality - if you got clean diverse data…

Scaling can happen in many other places. Such as deeper iterative thought during inference such as Chain of Thought, Tree of Thought, which extracts increasingly better performance out of existing parameter and data sizes.

Tried to explain here: https://news.ycombinator.com/item?id=36054809

Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture

#30

Post by doctor_eval »

amoss wrote: post #26
doctor_eval wrote: post #19

Earlier quoted context omitted.

I consider the singularity to be the point at which the certainty of our predictions about our future becomes close to zero. By this definition I reckon we are already in the singularity. It’s not necessarily bad. The problem with the singularity is that that we can’t tell if it’s bad or not.

Certainty about predictions of the future has always been close to zero. If you take a person from an appropriate time and ask them what the future will look like in 1000, 100 or even 20 years then their predictions will bear little resemblence to what actually occurs. As humans we have a tendancy to make linear future predictions based on past observations. Over the timescales that matter significant effects occur f…

I would say that we are really saying the same thing. I did intend to imply “short term predictions”.

However I feel that these LLMs are not like the internet or the release of the iPhone. We’ve gone in a very short time from LLMs in the lab to ChatGPT to passing the bar exam.

Considering the rate of research that’s being published and the fact that what we see today is generally at least many months behind the state of the art, I don’t feel that we can even have any certainty about the next 6 months.

Post reply on HN