The Full Story of Large Language Models and RLHF • Hacker News Board

Re: The Full Story of Large Language Models and RLHF

#11

Post by dragonwriter » Wed, May 03, 2023, 5:30 PM UTC

jason-phillips wrote: post #9

galaxyLogic wrote: post #4

> LLMs with coding abilities could be employed to create sophisticated malware with unprecedented ease. If that is possible then shouldn't it also be possible to ask the AI to find and code remediation to the vulnerabilities it found? So AI could be used to find all possible code-vulnerabilities and then how to neutralize those? This would advance software security in general. In other words AI could be used like a m…

> If that is possible then shouldn't it also be possible to ask the AI to find and code remediation to the vulnerabilities it found? It's not that simple. Mal-AI will probably operate orders of magnitude faster than Bene-AI. Patching affected software/hardware is a bureaucratic or formalized process that can take time. Time dilates for AI relative to humans because it can accomplish so much work in our time horizons.…

> Mal-AI will probably operate orders of magnitude faster than Bene-AI. Patching affected software/hardware is a bureaucratic or formalized process that can take time.

“Bureaucracy” misstates the fundamental issue: errors in patching the software against malware (e.g., breaking the prime function to support security) are more costly than errors in attempts to break in, so the former fundamentally demands more costly (even if it, too, is fully automated) verification.

Re: The Full Story of Large Language Models and RLHF

#12

Post by runnerup » Wed, May 03, 2023, 5:30 PM UTC

I found this to be a particularly lucid writeup of the past 5 years of advancement in LLMs. I sent it to some undergrads to read.

Re: The Full Story of Large Language Models and RLHF

#13

Post by chessgecko » Wed, May 03, 2023, 5:53 PM UTC

jason-phillips wrote: post #9

galaxyLogic wrote: post #4

> LLMs with coding abilities could be employed to create sophisticated malware with unprecedented ease. If that is possible then shouldn't it also be possible to ask the AI to find and code remediation to the vulnerabilities it found? So AI could be used to find all possible code-vulnerabilities and then how to neutralize those? This would advance software security in general. In other words AI could be used like a m…

> If that is possible then shouldn't it also be possible to ask the AI to find and code remediation to the vulnerabilities it found? It's not that simple. Mal-AI will probably operate orders of magnitude faster than Bene-AI. Patching affected software/hardware is a bureaucratic or formalized process that can take time. Time dilates for AI relative to humans because it can accomplish so much work in our time horizons.…

I'd hope that people would run good-ai before pushing or a white hat vuln finder before adding packages. Not sure people will or if it will be as good, but it will probably be available.

Re: The Full Story of Large Language Models and RLHF

#14

Post by samstave » Wed, May 03, 2023, 5:59 PM UTC

jason-phillips wrote: post #9

galaxyLogic wrote: post #4

> LLMs with coding abilities could be employed to create sophisticated malware with unprecedented ease. If that is possible then shouldn't it also be possible to ask the AI to find and code remediation to the vulnerabilities it found? So AI could be used to find all possible code-vulnerabilities and then how to neutralize those? This would advance software security in general. In other words AI could be used like a m…

> If that is possible then shouldn't it also be possible to ask the AI to find and code remediation to the vulnerabilities it found? It's not that simple. Mal-AI will probably operate orders of magnitude faster than Bene-AI. Patching affected software/hardware is a bureaucratic or formalized process that can take time. Time dilates for AI relative to humans because it can accomplish so much work in our time horizons.…

At some point, every FAANG will have entire pods designated for RED/BLUE AI pentesting/vulnhunt, if they dint already...

Also will every other major country.

I'd bet china already has RED/BLUE challenges going on internally..

Re: The Full Story of Large Language Models and RLHF

#15

Post by jason-phillips » Wed, May 03, 2023, 6:01 PM UTC

chessgecko wrote: post #13

jason-phillips wrote: post #9

Earlier quoted context omitted.

> If that is possible then shouldn't it also be possible to ask the AI to find and code remediation to the vulnerabilities it found? It's not that simple. Mal-AI will probably operate orders of magnitude faster than Bene-AI. Patching affected software/hardware is a bureaucratic or formalized process that can take time. Time dilates for AI relative to humans because it can accomplish so much work in our time horizons.…

I'd hope that people would run good-ai before pushing or a white hat vuln finder before adding packages. Not sure people will or if it will be as good, but it will probably be available.

We already do that in a general sense (static and dynamic scanning) in our build pipelines, but all of this will surely accelerate.

Re: The Full Story of Large Language Models and RLHF

#16

Post by pixl97 » Wed, May 03, 2023, 7:44 PM UTC

dragonwriter wrote: post #11

jason-phillips wrote: post #9

Earlier quoted context omitted.

> If that is possible then shouldn't it also be possible to ask the AI to find and code remediation to the vulnerabilities it found? It's not that simple. Mal-AI will probably operate orders of magnitude faster than Bene-AI. Patching affected software/hardware is a bureaucratic or formalized process that can take time. Time dilates for AI relative to humans because it can accomplish so much work in our time horizons.…

> Mal-AI will probably operate orders of magnitude faster than Bene-AI. Patching affected software/hardware is a bureaucratic or formalized process that can take time. “Bureaucracy” misstates the fundamental issue: errors in patching the software against malware (e.g., breaking the prime function to support security) are more costly than errors in attempts to break in, so the former fundamentally demands more costly…

Malware has to find one flaw.

Bene-ware has to avoid screwing up of any one of thousands (millions) of calls in a system.

'Undocumented but valid' is also a nightmare here. Recently the software I support broke for a bunch of clients in the field because they were using an undocumented method that worked, but that we did not actually test internally. This did come a surprise to our testing and development team as our customer base is adverse to any kind of data collection that shows how customers use the system.

Re: The Full Story of Large Language Models and RLHF

#17

Post by paulrchds » Wed, May 03, 2023, 8:11 PM UTC

I have been meaning to get a better overview of LLMs, this was a useful article.

Re: The Full Story of Large Language Models and RLHF

#18

Post by 1024core » Wed, May 03, 2023, 8:38 PM UTC

Word to the wise: the RLHF part comes 80% of the way down.

Re: The Full Story of Large Language Models and RLHF

#19

Post by chaxor » Thu, May 04, 2023, 1:01 AM UTC

One important but that is often left out is that ChatGPT is not the first model to come out using RLHF to train LLMs.

As is typical in the AI field, Deepmind was key in the development of the process. Deepmind 's Sparrow came out just before ChatGPT (regarding language modeling with RLHF), and much of the RLHF work was explored in their robotics/agent exploration work just prior to application in language.

OpenAI was integral in PPO, but it's important to know and understand it wasn't ChatGPT or OpenAI that is solely leading these advancements.

Re: The Full Story of Large Language Models and RLHF

#20

Post by galaxyLogic » Thu, May 04, 2023, 6:45 AM UTC

pixl97 wrote: post #16

dragonwriter wrote: post #11

Earlier quoted context omitted.

> Mal-AI will probably operate orders of magnitude faster than Bene-AI. Patching affected software/hardware is a bureaucratic or formalized process that can take time. “Bureaucracy” misstates the fundamental issue: errors in patching the software against malware (e.g., breaking the prime function to support security) are more costly than errors in attempts to break in, so the former fundamentally demands more costly…

Malware has to find one flaw. Bene-ware has to avoid screwing up of any one of thousands (millions) of calls in a system. 'Undocumented but valid' is also a nightmare here. Recently the software I support broke for a bunch of clients in the field because they were using an undocumented method that worked, but that we did not actually test internally. This did come a surprise to our testing and development team as our…

If hackers can use AI to find the vulnerabilities then surely the good guys can find the same vulnerabilities with the same AI. And once you are aware of them you can protect against them.

I don't think the bad guys have any special advantage that would allow them to produce AI better than the good guys. Hackers are good at finding vulnerabilities because as you say they only need to find one vulnerability. But you can't IMPROVE the state of the art in AI with just a single discovery. So in the AI-race the hackers don't have the same advantage as they have in finding vulnerabilities. I think.