New models and developer products • Hacker News Board

Re: New models and developer products

#541

Post by anotherpaulg » Tue, Nov 07, 2023, 5:12 PM UTC

ukuina wrote: post #539

anotherpaulg wrote: post #515

Earlier quoted context omitted.

Aider has a "token budget" for the repository map (--map-tokens, default of 1k). It analyzes the AST of all the code in the repo, the call graph, etc... and uses a graph optimization algorithm to select the most relevant parts of the repo map that will fit in the budget. There's some more detail in the recent writeup about the new tree-sitter based repo map that was linked in my comment above.

Any plans to open support up for other languages that tree-sitter supports?

Aider supports the repo map for bunch of languages already, see below. Is there one in particular you need that is missing?

https://github.com/paul-gauthier/aider/tree/main/aider/queri...

Re: New models and developer products

#542

Post by kevindamm » Tue, Nov 07, 2023, 5:32 PM UTC

DerJacques wrote: post #397

mvkel wrote: post #315

Earlier quoted context omitted.

> Assistants demos in particular showed that they are a black box within a black box within a black box that you can't port anywhere else. I'd argue the opposite. The new "Threads" interface in the OpenAI admin section lets you see exactly how it's interpreting input/output specifically to address the black box effect. Source: https://platform.openai.com/docs/api-reference/runs/listRunS... tells you exactly how it's…

I agree that some parts of the process now seem more like “open”, but there is definitely a lot more magic in the new processing. Namely, threads can have an arbitrary length, and OpenAI automatically handles context window management for you. Their API now also handles retrieval of information from raw files, so you don’t need to worry about embeddings. Lastly, you don’t even need any sort of database to keep track…

Indeterminate context, unknown/hidden values and a stateful API are usually reasons for me to look elsewhere for a solution.

Re: New models and developer products

#543

Post by usaar333 » Tue, Nov 07, 2023, 5:49 PM UTC

anotherpaulg wrote: post #512

doctoboggan wrote: post #57

In the keynote @sama claimed GPT-4-turbo was superior to the older GPT-4. Have any benchmarks or other examples been shown? I am curious to see how much better it is, if it all. I remember when 3.5 got its turbo version there was some controversy on whether it was really better or not.

My early benchmarking seems to show that it's somewhat better for coding. https://aider.chat/docs/benchmarks-1106.html

I'm really surprised your benchmark shows gpt-3.5-turbo-0301 outperforming gpt-4 (non tubo) on first try coding problems.

Re: New models and developer products

#544

Post by icelancer » Tue, Nov 07, 2023, 6:01 PM UTC

dri_ft wrote: post #486

icelancer wrote: post #473

Earlier quoted context omitted.

> GPT4 also recently developed this annoying tendency to only give you one or two examples of what you asked for, then say “you can write the rest on your own based on this template”. I can’t overstate how annoying this was. The last model "update" has really ruined GPT-4 in this regard.

When was this? I noticed chatGPT becoming succinct almost to the point of being standoffish about a week or two ago. Probably exacerbated by my having some custom instructions to tame its prior prolixity.

About a week ago, I noticed it, tweeted it, and a bunch of people said that /r/chatGPT and other forums noticed the really poor context-awareness around the same time.

Re: New models and developer products

#545

Post by zoogeny » Tue, Nov 07, 2023, 6:01 PM UTC

ShakataGaNai wrote: post #200

bluck wrote: post #164

Copyright Shield > OpenAI is committed to protecting our customers with built-in copyright safeguards in our systems. Today, we’re going one step further and introducing Copyright Shield—we will now step in and defend our customers, and pay the costs incurred, if you face legal claims around copyright infringement. This applies to generally available features of ChatGPT Enterprise and our developer platform. So essen…

For large-scale usage, it doesn't matter what the devs want. If the lawyers show up and say "We can't use this technology because we're probably going to get sued for copyright infringement", it's dead in the water. It's a logical "feature" for them to offer this "shield" as it significantly mitigates one of the large legal concerns to date. It doesn't make the risks fully go away, but if someone else is going to ste…

To add to this, indemnification of this type is pretty standard in sensitive fields. One example I recall was closed captioning providers. If they caption some content incorrectly in a way that exposes their customers to legal action they guarantee that they will take the blame and they have insurance specifically to handle any settlements based on legal actions.

I would expect this is a critical piece for medium to large enterprises that want to adopt LLMs. There are organizations for which this kind of indemnification isn't a nice to have, it is a requirement before even considering a product.

Re: New models and developer products

#546

Post by gen220 » Tue, Nov 07, 2023, 6:04 PM UTC

bertil wrote: post #392

gen220 wrote: post #314

Earlier quoted context omitted.

> If these tools help to write more code, but do not assist with maintainance, I wonder if we're going to see masses of new code written really quickly, and then everything grinds to a halt, because no one has an intimate understanding of what was written? Yep. Companies using LLMs to "augment" junior developers will get a lot of positive press, but I guess it remains to be seen how much the market consistently rewar…

What are the tasks that you envision are key to maintenance? - bug finding and fixing - parsing logs to find optimisation options - refactoring (after several local changes) - given new features, recommending a refactoring? I feel like code assistants are already reasonable help for doing the first two, and the later two are mostly a question of context window. I feel we might end up with code bases split by context…

I guess the issue is that programmers work with a really big context window, and need to reason at multiple levels of abstraction depending on the problem.

A great idea to solve a problem at one level of abstraction / context might be a terrible "strategic" idea at a higher level of abstraction. This is what separates the "junior" engineers from "senior" engineers, speaking very loosely.

IDK, I'm not convinced by all that I've seen, that GPT is capable of that higher-order thinking. I fear it requires a degree of epistemology that GPT fundamentally doesn't possess as a stochastic token-guesser. It never pushes back against a request, or asks if you really intend another question by your first question. It never tries to read through your requirements to grasp the underlying problem that's prompting them.

Maybe some combination of static tools, senior caretakers and prompt hackery can get us to a solution that maintains code effectively. But I don't think you can throw out the senior caretakers, their verification involvement is really necessary. And I don't know how conducive this environment would be to developing the next generation of "senior caretakers".

Re: New models and developer products

#547

Post by anotherpaulg » Tue, Nov 07, 2023, 6:13 PM UTC

usaar333 wrote: post #543

anotherpaulg wrote: post #512

Earlier quoted context omitted.

My early benchmarking seems to show that it's somewhat better for coding. https://aider.chat/docs/benchmarks-1106.html

I'm really surprised your benchmark shows gpt-3.5-turbo-0301 outperforming gpt-4 (non tubo) on first try coding problems.

One thing to notice is that the gpt-3.5 bars are blue and the gpt-4 bars are green. This is because aider uses different prompting strategies for GPT-3.5 and 4.

GPT-3.5 is only able to reliably edit a file by returning a whole new copy of the file with the edits included. This is the "whole" edit format.

GPT-4 is able to use a more efficient "diff" edit format, where it species blocks of code to search and replace.

All of this is described and quantified in more detail in the original aider benchmarking writeup:

https://aider.chat/docs/benchmarks.html

The original article benchmarked both models using both edit formats (and some others). And indeed, gpt-4/whole beats gpt-3.5/whole. But it's very slow and very expensive to ask gpt-4 to return a whole copy of any file that it edits. So it's just much more practical to use the gpt-4/diff, even though it performs a bit worse than gpt-4/whole.

Aider will let you do gpt-4/whole if you'd like to spend the time and money:

  aider --model gpt-4 --edit-format whole

Once OpenAI relaxes the rate limits, I will benchmark gpt-4-1106-preview/whole.

Re: New models and developer products

#548

Post by zoogeny » Tue, Nov 07, 2023, 6:14 PM UTC

9dev wrote: post #448

lucubratory wrote: post #278

Earlier quoted context omitted.

They are my content, actually, from the last ~15 years of being on the internet. I don't care about it personally, and even if I did it is really obviously fair use so even if I find it objectionable I don't get to actually legally compel someone to stop.

What on earth is fair use about a public company deriving its whole valuation from the processing of content taken from the internet without any regard for licensing, or robots.txt rules?? The technology is cool, I get it. But saying ”I don’t mind, they can use my content“ is on par with ”I don’t need privacy, I have nothing to hide“ in terms of statement quality.

I can see a specific argument you are making as something along the lines of, just because some people are ok with their content being used by large corporations to train their for-profit LLMs doesn't mean it should be ok for those companies to take my content and use it to train their LLMs without my permission.

But, I suppose I see it the other way as well. Just because you don't want large corporations to train their LLMs using your content doesn't mean that society has to settle on making it illegal. As an imperfect analogy: just because some people don't want to have their picture taken when they are out in public doesn't mean that taking pictures of people in public ought to be illegal.

So I think we have to get passed the "I don't like this, so it is evil" kind of thinking. As in the analogy to pictures of people in public, there is some expectation of privacy that we give up when we enter out into public. Perhaps there is some analogy there to content that we freely release into public. Perhaps we need stricter guidelines on LLM attribution. I don't have an answer, but I'm not going to allow this decision to be de facto made by the strong emotions of individuals who have already made up their minds.

Re: New models and developer products

#549

Post by dragonwriter » Tue, Nov 07, 2023, 6:21 PM UTC

gen220 wrote: post #546

bertil wrote: post #392

Earlier quoted context omitted.

What are the tasks that you envision are key to maintenance? - bug finding and fixing - parsing logs to find optimisation options - refactoring (after several local changes) - given new features, recommending a refactoring? I feel like code assistants are already reasonable help for doing the first two, and the later two are mostly a question of context window. I feel we might end up with code bases split by context…

I guess the issue is that programmers work with a really big context window, and need to reason at multiple levels of abstraction depending on the problem. A great idea to solve a problem at one level of abstraction / context might be a terrible "strategic" idea at a higher level of abstraction. This is what separates the "junior" engineers from "senior" engineers, speaking very loosely. IDK, I'm not convinced by all…

> IDK, I'm not convinced by all that I've seen, that GPT is capable of that higher-order thinking. I fear it requires a degree of epistemology that GPT fundamentally doesn't possess as a stochastic token-guesser. It never pushes back against a request, or asks if you really intend another question by your first question. It never tries to read through your requirements to grasp the underlying problem that's prompting them.

It can if prompted appropriately. If you are just using the default ChatGPT interface and system prompt, it doesn't, but then, it is intended to be compliant outside of its safety limits in that application. (I am not arguing it has the analytical capacity to be suited for for the role being discussed, but the particular complaint about excessive compliance is a matter of prompting, not model capacity.)

Re: New models and developer products

#550

Post by vunderba » Tue, Nov 07, 2023, 9:30 PM UTC

anotherpaulg wrote: post #330

zizee wrote: post #258

In people's experience with these sorts of tools, have they assisted with maintainance of codebases? This might be directly, or indirectly via more readable, bette organized code. The reason I ask is that these tools seem to excel in helping to write new code. In my experience I think there is an upper limit to the amount of code a single developer can maintain. Eventually you can't keep everything in your head, so m…

My open source ai coding tool aider is unique in that it is designed to work with existing code bases. You can jump into an existing git repo and start aaking for changes, new features, etc. https://github.com/paul-gauthier/aider It helps gpt understand larger code bases by building a "repository map" based on analyzing the abstract syntax tree of all the code in the repo. This is all built using tree-sitter, the sam…

I think Cursor.sh is another tool designed around the same principle at least from a repository level awareness scope, but rather than just being CLI it's a full fledged VS Code fork.