Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

tokens.billchambers.me

Post reply on HN
Print view View original

531–540 of 620 posts

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#531

Post by bobkb »

matheusmoreira wrote: post #423
rectang wrote: post #420

Earlier quoted context omitted.

Are the benchmarks being used to measure these models biased towards completing huge and highly complex tasks, rather than ensuring correctness for less complex tasks? It seems like they're working hard to prioritize wrapping their arms around huge contexts, as opposed to handling small tasks with precision. I prefer to limit the context and the scope of the task and focus on trying to get everything right in increme…

I don't think there's a bias here. I'd say my task is of somewhat high complexity. I'm using Claude to assist me in implementing exceptions in my programming language. It's a SICP chapter 5.4 level task. There are quite a few moving parts in this thing. Opus 4.6 once went around in circles for half an hour trying to trace my interpreter's evaluator. As a human, it's not an easy task for me to do either. I think the p…

‘effort high/max’ seems to be working though

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#532

Post by [deleted] »

vanviegen wrote: post #524
mrandish wrote: post #455

Earlier quoted context omitted.

> when the model won't actually be able to provide one This is key. In my experience, asking an LLM why it did something is usually pointless. In a subsequent round, it generally can't meaningfully introspect on its prior internal state, so it's just referring to the session transcript and extrapolating a plausible sounding answer based on its training data of how LLMs typically work. That doesn't necessarily mean th…

> In a subsequent round, it generally can't meaningfully introspect on its prior internal state It can't do any better in the moment it's making the choices. Introspection mostly amounts to back-rationalisation, just like in humans. Though for humans, doing so may help learning to make better future decisions in similar situations.

[deleted]

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#533

Post by adrian_b »

einpoklum wrote: post #100
pixelatedindex wrote: post #21

Earlier quoted context omitted.

I’m impressed with anyone who can read English right to left.

Right to Left English - read can, who? Anyone with [which] impressed am I.

English can be read in a different order than the normal order when the sentences contain words for which it is easy to guess whether they are agents or patients, e.g. when the agents are animate nouns and the patients are inanimate nouns, or when pronouns are used for the agents or patients.

Otherwise, the non-standard order can be understood incorrectly. While the distinction between agents and patients is the most important that depends on word order in English, there are also other order-dependent distinctions, e.g. between beneficiary and patient, when the beneficiary is not marked by a preposition, or between a noun and its attribute, e.g. "police dog" is not the same as "dog police" and unless there is a detailed context you cannot know what is meant when the word order is wrong.

English is one of the languages with the most rigid word order. There are languages, especially among older languages, where almost any word order can be used without causing ambiguities, because all the possible roles of the words are marked by prepositions, postpositions or affixes (or sometimes by accentuation shifts).

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#534

Post by zozbot234 »

vanviegen wrote: post #528
krackers wrote: post #407

Earlier quoted context omitted.

>pay for reinitializing the cache Why can't they save the kv cache to disk then later reload it to memory?

Isn't that how the kv cache currently works? Of course they could decide to hold on to cache items for longer than an hour, but the storage requirements are pretty significant while the chance of sessions resumption slinks rapidly.

The storage requirements for large-model KV caches are actually comparatively tiny: the per-token size grows far less than model parameters. Of course, we're talking "tiny" for stashing them on bulk storage and slowly fetching them back to RAM. But that should still be viable for very long context, since the time for running prefill is quadratic.

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#535

Post by lelanthran »

gbro3n wrote: post #529
lelanthran wrote: post #422

Earlier quoted context omitted.

> Unfortunately our knowledge of language APIs and syntax has diminished in value, but we have so many more skills that will be just as valuable as ever. There were always jobs that required those "many more skills" but didn't require any programming skills. We call those people Business Analysts and you could have been doing it for decades now. You didn't, because those jobs paid half what a decent/average programme…

I guess we will need to wait and see if AI can remove ALL of the complexity that requires a software engineer over a business analyst. I can't currently believe that it will. BA's I've worked with vary in technical capability from 'having coded before and understanding DB schema basics and network architecture' to 'I know how the business works but nothing about computers'. If we got to the point in the future where…

> If we got to the point in the future where every computer system ran on the same frameworks in the same way, and AI understood it perfectly, then maybe.

They don't need to all run on the same frameworks, they just need to run on documented frameworks.

What possible value can you bring to a BA?

The system topology (say, if the backend was microservices vs Lambda vs something-else)? The LLM can explain to the BA what their options are, and the impact of those options.

The framework being used (Vue, or React, or something else)? The AI can directly twiddle that for the BA.

Solving a problem? If the observability is setup, the LLM can pinpoint almost all the problems too,and with a separate UAT or failover-type replica, can repro, edit, build, deploy and test faster than you can.

Like I already said, if[1] you're now able to build or enhance a system without actually needing programming skills, why are you excited about that? You could always do that. It's just that it pays half what programming skills gets you.

You (and many others who boast about not writing code since $DATE) appear to be willingly moving to a role that already pays less, and will pay even less once the candidates for that role double (because now all you programmers are shifting towards it).

It's supply and demand, that's all.

--------------

[1] That's a very big "If", I think. However, the programmers who are so glad to not program appear to believe that it's a very small "If", because they're the ones explaining just how far the capabilities have come in just a year, and expect the trend to continue. Of course, if the SOTA models never get better than what we have now, then, sure - your argument holds - you'll still provide value.

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#536

Post by AlexCoventry »

mrandish wrote: post #455
codethief wrote: post #430

Earlier quoted context omitted.

> > Why are you handwaving things away though? I've got you on max effort. I even patched the system prompts to reduce this. In my experience, prompts like this one, which 1) ask for a reason behind an answer (when the model won't actually be able to provide one), 2) are somewhat standoff-ish, don't work well at all. You'll just have the model go the other way. What works much better is to tell the model to take a st…

> when the model won't actually be able to provide one This is key. In my experience, asking an LLM why it did something is usually pointless. In a subsequent round, it generally can't meaningfully introspect on its prior internal state, so it's just referring to the session transcript and extrapolating a plausible sounding answer based on its training data of how LLMs typically work. That doesn't necessarily mean th…

> In a subsequent round, it generally can't meaningfully introspect on its prior internal state

It has the K/V cache, no?

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#537

Post by xpe »

ozgrakkurt wrote: post #522
xpe wrote: post #480

Earlier quoted context omitted.

> Those tools seem mostly useful for a Google alternative, scaffolding tedious things, code reviewing, and acting as a fancy search. Just to get a sense for the rate of change, imagine if you took a survey. Compare what people said about AI tools... 3 years ago, 2 years ago, 1 year ago, 6 months ago. Then think about what is plausible that people will be saying in 3 months, 6 months, 9 months ... Moving the goalposts…

Can you explain this literature review process? I don't believe you can do a same quality job with an LLM in 5 minutes.

I don't mean writing a literature review. I mean reviewing the literature to find what I need. My point is that this was not practical with "fancy search" three years ago by which I mean Google-like search engines.

My example: I wanted to get a sense for the feasibility of doing a project that blends Gaussian Processes, active learning, and pairwise comparisons. So I want to dig into the literature to find out what is out there. This was around 5 minutes with Claude. In this case, I don't think I could have found what I wanted in 10 hours of searching and reading. This is the kind of thing that great LLMs unlock.

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#539

Post by danny_codes »

alex_sf wrote: post #467
danny_codes wrote: post #251

Earlier quoted context omitted.

I just don’t see how they’ll be able to make a profit. Open models have the same performance on coding tasks now. The incentives are all wrong. Why pay more for a model that’s no better and also isn’t open? It’s nonsense

Open models, in actual practice, don't match up to even one or two generation prior models from Anthropic/OpenAI/Google. They've clearly been trained on the benchmarks. Entirely possible it was by mistake, but it's definitely happening.

That hasn’t been my experience. For coding at least I find little difference between closed and open models

Re: Anonymous request-token comparisons from Opus 4.6 and Opus 4.7

#540

Post by xpe »

manmal wrote: post #517
xpe wrote: post #480

Earlier quoted context omitted.

> Those tools seem mostly useful for a Google alternative, scaffolding tedious things, code reviewing, and acting as a fancy search. Just to get a sense for the rate of change, imagine if you took a survey. Compare what people said about AI tools... 3 years ago, 2 years ago, 1 year ago, 6 months ago. Then think about what is plausible that people will be saying in 3 months, 6 months, 9 months ... Moving the goalposts…

> Today, in ~5 minutes I can do a literature review that would have taken me easily 10+ hours five years ago. And it will not yield the same outcome you would have had. Your own taste in clicking links and pre-filtering as you do your research, is no longer being done if you outsource this. I‘m guilty of this myself. But let’s not kid ourselves. I’ve had GPT Pro think 40 minutes about the ideal reverse osmosis setup…

I don't claim using AI is the same as doing it yourself. My point is that AI capabilities are much more extensive than "fancy search". By giving a metric and an example I hoped to make that point without getting into hair-splitting.
Post reply on HN