Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 details leaked?

threadreaderapp.com

Post reply on HN
Print view View original

201–210 of 648 posts

Re: GPT-4 details leaked?

#201

Post by why_only_15 »

TeMPOraL wrote: post #133
DANmode wrote: post #46

Earlier quoted context omitted.

Parameters: In the context of AI and language models, parameters refer to the internal settings or variables that an AI model uses to make predictions or generate responses. Think of them as the knobs and switches that can be adjusted to fine-tune how the AI understands and generates language. These parameters are learned during the training process, where the AI model analyzes vast amounts of data to optimize its pe…

> "Mixture Of Experts": An "MoE" is an approach in AI that combines multiple specialized AI models, known as "experts," (...) Wonder when that stopped being called just an "ensemble model", which is a term I recall from 10 years ago. Terminology churn?

Mixture of experts is different from ensembles because MoE happens at every layer as opposed to joining the models once at the end

Re: GPT-4 details leaked?

#202

Post by kristiandupont »

aaron695 wrote: post #29
doctor_eval wrote: post #10

Earlier quoted context omitted.

I had to ask GPT what MoE means: "MoE" in the context of artificial intelligence typically stands for "Mixture of Experts". This is a machine learning technique that is based on the idea of dividing a problem into sub-problems, solving each sub-problem with a specialized "expert" (or model), and then combining their outputs.

[flagged]

You should consider asking ChatGPT for help in clarifying your point(s).

Re: GPT-4 details leaked?

#203

Post by capableweb »

twayt wrote: post #108
potatoman22 wrote: post #25

I wonder what the legal implications of them using SciHub and Libgen would be if that's true. I'd imagine OpenAI is big enough to make deals with publishers.

Libgen / Scihub or not, if the model can provide details about the book other than just high level info like the summary and no explicit deal with the publisher has been made, you can make a strong argument that it is plagiarism. Even if bits and pieces of the book text are distributed across the internet and you end up picking up portions of the book, you still read the book. It is extremely sad but ChatGPT will be…

I'm not a lawyer and obviously we won't get any definite answer unless it actually goes to court, all of this is just hand waving and guessing.

But I think that unless GPT starts reciting large parts outside of the context of learning/education/research, reciting smaller snippets would fall into "fair use" and not be illegal.

Re: GPT-4 details leaked?

#204

Post by why_only_15 »

dr_kiszonka wrote: post #160
DANmode wrote: post #46

Earlier quoted context omitted.

Parameters: In the context of AI and language models, parameters refer to the internal settings or variables that an AI model uses to make predictions or generate responses. Think of them as the knobs and switches that can be adjusted to fine-tune how the AI understands and generates language. These parameters are learned during the training process, where the AI model analyzes vast amounts of data to optimize its pe…

Great explanations! How about Multi-Query Attention?

This is the original paper: https://arxiv.org/abs/1911.02150 . The idea is that with a transformer you have many heads, say 64 for LLaMa, and for each head you have 1 "query" vector one "key" vector and one "value" vector per token. Most of the cost of inferencing models is loading the key and value vectors from GPU memory to the GPU itself. the idea behind MQA is that instead of having 64 queries, 64 keys, and 64 values, you have 64 queries, 1 key, and 1 value ("Multi-Query" as opposed to "Multi-Head", the original name). This means that there is much less data to load from GPU memory to the GPU during inference.

Re: GPT-4 details leaked?

#205

Post by why_only_15 »

RC_ITR wrote: post #165
flir wrote: post #121

Earlier quoted context omitted.

I bet it'll be 6 figures within 18 months.

The thing about these models is compute scales quadratically with model dimensionality and memory scales quadratically with sequence length. We are nowhere near diminishing returns for either variable, so sure current models maybe scale quickly but the cutting edge will want as much compute as possible for a long time. That’s kind of the humor of everyone saying this leak somehow leaves OpenAI vulnerable. The work is…

Memory does not scale quadratically with sequence length.

Re: GPT-4 details leaked?

#206

Post by XorNot »

borissk wrote: post #153
vouaobrasil wrote: post #95

Earlier quoted context omitted.

I would love to see your rebuttals, especially since I have never seen any strong arguments in favour of AI being a net benefit to society, and I have thought and read about this at lenght. Of course, I always expect downvotes on my posts here since there is a strong tendency towards loving technology here. But what I find most interesting is that there is absolutely no taking of responsibility of any technological c…

If we were a rational civilization we'd stop all scientific research immediately. First there's a good chance the great filter is ahead of us and will be triggered by a technology break-trough. Second with nuclear weapons we got lucky in that it's extremely hard to separate fission capable isotope of uranium from mineral ores; if in the future we invent a powerful weapon that's easy to produce organizations like al-Q…

I can't figure out if this is a rebuttal by absurdity or serious?

Re: GPT-4 details leaked?

#207

Post by ikekkdcjkfke »

aaron695 wrote: post #29
doctor_eval wrote: post #10

Earlier quoted context omitted.

I had to ask GPT what MoE means: "MoE" in the context of artificial intelligence typically stands for "Mixture of Experts". This is a machine learning technique that is based on the idea of dividing a problem into sub-problems, solving each sub-problem with a specialized "expert" (or model), and then combining their outputs.

[flagged]

[flagged]

Re: GPT-4 details leaked?

#208

Post by ChildOfChaos »

PostOnce wrote: post #19

"Open" AI, a charity to benefit us all by pushing and publishing the frontier of scientific knowledge. Nevermind, fuckers, actually it's just to take your jobs and make a few VCs richer. We'll keep the science a secret and try to pressure the government into making it illegal for you to compete with us. https://github.com/ggerganov/llama.cpp https://github.com/openlm-research/open_llama https://huggingface.co/TheBlok…

You forgot to link where I can buy your tin foil hat at the end.

Re: GPT-4 details leaked?

#209

Post by ZiiS »

dacryn wrote: post #187
ed_mercer wrote: post #142

Earlier quoted context omitted.

AFAIK you can get away with a swapfile, no need for large amounts of RAM.

wont that nearly kill your ssd if you do it for extended periods of time?

Most of the ram is for storing the model once it is loaded it is read only so will not harm an SSD.

Re: GPT-4 details leaked?

#210

Post by potatototoo99 »

KronisLV wrote: post #112
antupis wrote: post #92

Earlier quoted context omitted.

I bet that open models win in the end because porn. There is already very weird and vibrant community creating "waifus" and tinkering with these models.

Huh, more power to those folks then, I guess. But I can easily imagine more conventional forms of entertainment, as well. Like a game of D&D that's narrated by the AI, or a text based adventure set in the Mass Effect universe, Lord of the Rings, Warhammer or any other fandom, really. Maybe like those old Choose Your Own Adventure games. I think some companies are also experimenting with characters in video games that…

Do you seriously think D&D and other choose your adventure games will be more popular than porn? Seriously?
Post reply on HN