Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI's plans according to sama

humanloop.com

Post reply on HN
Print view View original

111–120 of 269 posts

Re: OpenAI's plans according to sama

#111

Post by visarga »

dr_dshiv wrote: post #74
arugulum wrote: post #23

Earlier quoted context omitted.

As someone who is in the field: papers proposing to solve the context length problem come out every month. Almost none of the solutions stick or work as well as a dense or mostly dense model. You'll know when the problem is solved when model after consistently use a method. Until then (and especially if you're not in the field as a researcher), assume that every paper claiming to tackle context length is simply a nic…

What about Meta’s megabyte? Also nice proposal?

Yes. Solving context length has been tried in hundreds of different approaches, and yet most LLMs are almost identical to the original one from 2017.

Just to name a few families of approaches: Sparse Attention, Hierachical Attention, Global-Local Attention,Sliding Window Attention, Locality sensitive hashing Attention, State space model, EMA gated attention.

Re: OpenAI's plans according to sama

#113

Post by nonethewiser »

purplecats wrote: post #32

> Cheaper and faster GPT-4 — This is their top priority. In general, OpenAI’s aim is to drive “the cost of intelligence” down as far as possible and so they will work hard to continue to reduce the cost of the APIs over time. this certainly aligns with the massive (albeit subjective and anecdotal) degradation in quality i've experienced with ChatGPT GPT-4 over the past few weeks. hopefully a superior (higher quality)…

I wonder of it actually is because they’re tuning it to make it less offensive (by their standards). Thats the only explanation I keep seeing repeated.

Re: OpenAI's plans according to sama

#114

Post by CSMastermind »

invaliduser wrote: post #107
CSMastermind wrote: post #94

Earlier quoted context omitted.

US, France, Israel ... then who? Maybe another five eyes country like the UK? Possibly China? I'm pretty skeptical Russia would be able to get someone in there but maybe.

Hi. French here. I may be wrong, but I really feel like you are overestimating us.

DGSE essentially puts all of it's money/effort into industrial espionage and they're the best in the world at it.

Re: OpenAI's plans according to sama

#115

Post by reaperman »

nonethewiser wrote: post #113
purplecats wrote: post #32

> Cheaper and faster GPT-4 — This is their top priority. In general, OpenAI’s aim is to drive “the cost of intelligence” down as far as possible and so they will work hard to continue to reduce the cost of the APIs over time. this certainly aligns with the massive (albeit subjective and anecdotal) degradation in quality i've experienced with ChatGPT GPT-4 over the past few weeks. hopefully a superior (higher quality)…

I wonder of it actually is because they’re tuning it to make it less offensive (by their standards). Thats the only explanation I keep seeing repeated.

I would be very surprised. Things that are very, very far from that are also much worse. I'm having difficulty finding the difference between GPT-3.5 and GPT-4 for a lot of my programming tasks lately. It's noticeably degraded.

Re: OpenAI's plans according to sama

#116

Post by visarga »

ilaksh wrote: post #102
ren_engineer wrote: post #55

Earlier quoted context omitted.

if this was currently possible wouldn't it lead to sentient/superhuman AI rapidly? >tell AI to make itself more efficient by finding performance improvements in human written code >that newly available processing power can now be used to find more ways to improve itself >flywheel effect of AI improving itself as it gets smarter and smarter eventually you'd turn it loose on improving the actual hardware it runs on. I…

Within a couple of years of improvement processes like you suggest will actually be really dangerous and stupid. Also don't confuse all other types of human/animal characteristics like sentience with intelligence. They are different things. Things like sentience, subjective stream of experience, or other aspects of being alive don't just accidentally fall out of larger training datasets. And we should be glad. The mo…

Intelligence is about action, and sentience is about qualia, which I equate to perceptions coloured by values. Action is visible and qualia are hidden, but they are closely interconnected: we choose our actions in accordance with our values and situation at hand.

Re: OpenAI's plans according to sama

#117

Post by MacsHeadroom »

boringuser2 wrote: post #3

>Dedicated capacity offering is limited by GPU availability. OpenAI also offers dedicated capacity, which provides customers with a private copy of the model. To access this service, customers must be willing to commit to a $100k spend upfront. How many shell corporations are intelligence agencies seeding right now?

Private instance means a dedicated endpoint fully managed by OpenAI. You do not get model access or anything a regular API user doesn't already get, except your API url will be something like customer123.openai.com/api instead of api.openai.com/api

Re: OpenAI's plans according to sama

#118

Post by brucethemoose2 »

purplecats wrote: post #32

> Cheaper and faster GPT-4 — This is their top priority. In general, OpenAI’s aim is to drive “the cost of intelligence” down as far as possible and so they will work hard to continue to reduce the cost of the APIs over time. this certainly aligns with the massive (albeit subjective and anecdotal) degradation in quality i've experienced with ChatGPT GPT-4 over the past few weeks. hopefully a superior (higher quality)…

Anthropic's Claude is said to be very good.

Instruction tuned LLaMA 65B/Falcon 40B are good, especially with an embeddings database.

...But OpenAI has all the name recognition and ease of use now, so it might not even matter if others ambiguously surpass OpenAI models.

Re: OpenAI's plans according to sama

#119

Post by mustacheemperor »

ilaksh wrote: post #101
mustacheemperor wrote: post #21

Earlier quoted context omitted.

I enjoy using GPT4 as a co-programmer, and funny enough it is very challenging to get advice on Microsoft's own .NET MAUI because that framework was in prerelease at the time the model was trained. My understanding is right now they essentially need to train a new model on a new updated corpus to fix this, but maybe some other techniques could be devised...or they'll train something more up to date.

You might actually get pretty far if you just went through the Microsoft docs and created a bunch of really concise examples and fed that as the start of the prompt. Use like 6-7kb for that and then the question at the end.

I have had some luck doing exactly that, and not even as efficiently as you describe - If my question is limited enough that the discussion won't overwhelm the context window I've found I can just paste in big chunks of the docs wholesale like a 'zero shot.'

Re: OpenAI's plans according to sama

#120

Post by razcle »

sharkjacobs wrote: post #24

> He reiterated his belief in the importance of open source and said that OpenAI was considering open-sourcing GPT-3. Part of the reason they hadn’t open-sourced yet was that he was skeptical of how many individuals and companies would have the capability to host and serve large LLMs. Am I reading this right? "We're not open sourcing GPT-3 because we don't think it would be useful to anyone else"

I think I worded this poorly. What he said was that a lot of people say they want open-source models but they underestimate how hard it is to serve them well. So he wondered how much real benefit would come from open-sourcing them.

I think this is reasonable. Giving researchers access is great but for most small companies they're likely better off having a service provider manage inference for them rather than navigate the infra challenge.

Post reply on HN