Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New models and developer products

openai.com

Post reply on HN
Print view View original

421–430 of 561 posts

Re: New models and developer products

#421

Post by dbmikus »

esafak wrote: post #410
wokwokwok wrote: post #354

Earlier quoted context omitted.

Then you’re either not testing your prompts or doing something trivial. Remember: a good model with a good prompt will generate bad outputs sometimes. A bad model with a bad prompt will generate a good output sometimes. That is simply a fact with these non deterministic models. You have to do many iterations for each prompt to verify they are working correctly. > I’ve not had much problems moving between LLMs… If you…

How many people are even writing tests for these things?

Very few. Many deployed apps don't have a good quantitative grasp of the quality of their LLMs. Some are doing testing or evaluation, through things like unit tests, A/B testing different prompts, collecting user feedback.

I think we're exiting the phase where people can launch an AI app and have people use it just because of the initial "wow factor" and moving into the phase where users will start churning and businesses will need to make sure that their AI agent is performing and they they understand how well it's performing.

Re: New models and developer products

#422

Post by endorphine »

steno132 wrote: post #276

For all the hate: Elon ships. And OpenAI ships. People claim OpenAI is closed, that they are controlled by Microsoft, that they don't care enough about safety... But the fact is, Anthropic, Google Brain, even Meta -- OpenAI blows them all out of the water when it comes to shipping new innovations. Just like Twitter ships much more now with Elon, and how SpaceX ships much more than NASA and Blue Origin. If you disagre…

As if shipping was the end goal...

Re: New models and developer products

#423

Post by ssnistfajen »

openquery wrote: post #378

If I had no contact with society from the 29th of November 2022 (the day before ChatGPT was released according to Wikipedia) and came back today to see the OpenAI keynote I would have lost my mind. The progress and usefulness of these products is absolutely incredible.

I remember opening Twitter that night and seeing a bunch of tech people I follow sharing screenshots of conversations with a little green icon. I thought "oh neat, yet another chatbot fad to try out for 5 minutes". I could not have been more wrong.

Re: New models and developer products

#424

Post by dragonwriter »

m3kw9 wrote: post #388
oezi wrote: post #242

Earlier quoted context omitted.

OpenAI is still way too expensive to run a corporate knowledge base on top

It’s more reliable than chatpdfs that relies on vector search. With vector db all you are doing is doing a fuzzy search and then sending in that relevant portion near that text and send it to a LLM model as part of a prompt. It misses info.

I'd be very surprised if the Assistants API is not doing RAG with a vector DB behind the scenes with the supplied files.

Re: New models and developer products

#425

Post by asaddhamani »

minimaxir wrote: post #4

Most of the products announced (and the price cuts) appear to be more about increasing lock-in to the OpenAI API platform, which is not surprising given increased competition in the space. The GPTs/GPT Agents and Assistants demos in particular showed that they are a black box within a black box within a black box that you can't port anywhere else. I'm mixed on the presentation and will need to read the fine print on…

Their products are incredible though. I’ve tried the alternatives and even Claude is not nearly as good as even ChatGPT. Claude gives an ethics lecture with every second reply, which costs me money each time and makes their product very difficult to (want to) embed.

Re: New models and developer products

#426

Post by freedomben »

qingcharles wrote: post #380
openquery wrote: post #378

If I had no contact with society from the 29th of November 2022 (the day before ChatGPT was released according to Wikipedia) and came back today to see the OpenAI keynote I would have lost my mind. The progress and usefulness of these products is absolutely incredible.

I was in prison when ChatGPT came out. All I knew of it was a headline that flashed past really fast on CNN and I called my buddy and said "What the hell is Chat OPT?" I'd just finished reading The Singularity is Near for the second time too...

The singularity is near is a great book. Hilarious that you read that for the second time and then got out and saw chat gpt!

I love kurzweil but his estimates of timeline are often pretty over optimistic, so I'd be really wondering.

Re: New models and developer products

#427

Post by ChatGTP »

asaddhamani wrote: post #425
minimaxir wrote: post #4

Most of the products announced (and the price cuts) appear to be more about increasing lock-in to the OpenAI API platform, which is not surprising given increased competition in the space. The GPTs/GPT Agents and Assistants demos in particular showed that they are a black box within a black box within a black box that you can't port anywhere else. I'm mixed on the presentation and will need to read the fine print on…

Their products are incredible though. I’ve tried the alternatives and even Claude is not nearly as good as even ChatGPT. Claude gives an ethics lecture with every second reply, which costs me money each time and makes their product very difficult to (want to) embed.

What are you using it for? I want to know what people actually use these things for damn it !

Re: New models and developer products

#428

Post by Someone1234 »

vissidarte_choi wrote: post #415

Having ever increasing context is not the silver bullet. For those who believe that the larger the context, the smarter the model, you will find the model still talking nonsense even if it were fed with much larger context.

Sure; but before you couldn't even use it for some problems because the problems were bigger than the context window.

For example, I was trying to generate an XSLT 3.0 transformation from one Json format to another. The two formats and description alone almost depleted my context window. In essence, it killed using GPT-4 for this project.

I use it daily, and I haven't had it spit out too much "nonsense" in spite of everyone constantly telling me how that's all it does. The quality of results are on-par with Stackoverflow (in good and bad ways).

Re: New models and developer products

#429

Post by eichin »

throw2321 wrote: post #268
zizee wrote: post #258

In people's experience with these sorts of tools, have they assisted with maintainance of codebases? This might be directly, or indirectly via more readable, bette organized code. The reason I ask is that these tools seem to excel in helping to write new code. In my experience I think there is an upper limit to the amount of code a single developer can maintain. Eventually you can't keep everything in your head, so m…

I've been thinking about this for a while now, wrt two points: 1. This will be the end of traditional SWEs and the rise of the age of debuggers, human debuggers who spend their days setting up breakpoints and figuring bugs in a sea of LLM generated code. 2. Hiring will switch from using Leetcode questions to "pull out your debugger and figure out what's wrong with this code".

The first interview where I was handed some (intentionally) broken C code and gdb was about 15 years ago. I'm not sure that part is a change in developer workflow (this may apply more in systems and embedded though.)

I've been paying attention to this too (mostly by following Simon Willison) and I'm still solidly in the "get back to me when this stuff can successfully review a pull request or even interpret a traceback" camp...

Re: New models and developer products

#430

Post by jeswin »

anotherpaulg wrote: post #330
zizee wrote: post #258

In people's experience with these sorts of tools, have they assisted with maintainance of codebases? This might be directly, or indirectly via more readable, bette organized code. The reason I ask is that these tools seem to excel in helping to write new code. In my experience I think there is an upper limit to the amount of code a single developer can maintain. Eventually you can't keep everything in your head, so m…

My open source ai coding tool aider is unique in that it is designed to work with existing code bases. You can jump into an existing git repo and start aaking for changes, new features, etc. https://github.com/paul-gauthier/aider It helps gpt understand larger code bases by building a "repository map" based on analyzing the abstract syntax tree of all the code in the repo. This is all built using tree-sitter, the sam…

> It helps gpt understand larger code bases by building a "repository map" based on analyzing the abstract syntax tree of all the code in the repo.

What I do with codespin[1] (another AI code gen tool) is to give a file/files to GPT and ask for signatures (and comments and maybe autogenerate a description), and then cache it until the file changes. For a lot of algorithmic work, we could just use GPT now. Sure it's less efficient, but as these costs come down it matters less and less. In a way, it's similar to higher level (but inefficient) programming languages vs lower level efficient languages.

[1]: https://github.com/codespin-ai/codespin-cli

Post reply on HN