Viewing profile — lambda

Can you provide some references to the research you're referring to? Sounds interesting but you haven't really provided enough information to find it.

They do maintain the Transformers library which is pretty much the core library for how you interact with LLM models in the open source world. So while they weren't using a model t…

3.6 Flash scores exactly the same as 3.5 Flash on the Artificial Analysis index. Better on some tasks, worse on others. Mostly within what I'd consider the noise window. Looks pret…

> I think that in 10-15 years, we are going to have consumer PCs (and phones!) running models doing pretty much anything that frontier models can do right now 10-15 years? The curr…

The article describes it. Both Codex and Claude Code have it, but they work slightly differently. Claude Code uses Haiku to read through the transcript and decide if the goal has b…

I've tried doing a loop of rending the SVG and then tweaking based on that, with local models (so, not nearly as strong). It wasn't very successful; it would mostly report that the…

You can always ask them to draw something else, as a way to avoid any possible pelican related data contamination; given how popular the pelican test is, I'm sure there's some peli…

It's really frustrating, because now when I want to write something like a "not X but Y" or "you're absolutely right," I have to stop and decide if I want to self-censor to avoid s…

It drives us crazy because everyone is using the same 2-3 different machines. So rather than each person having their own unique speaking style, the whole world (or, everyone that …

He is saying that weaker models, as measured by a benchmark to distinguish "frontier" models, would be exempted. So an academic lab or startup that isn't yet producing frontier mod…

Yeah, Kraken SDR removed some functionality due to these concerns, if I remember correctly. Odd, because export controls don't generally apply to published material (like open sour…

But there are already benchtop or handheld signal analyzer for that purpose. This seems more like a tool for checking across entire large assemblies like an entire building, car, a…

I think that for a single device, this probably wouldn't help much over just having a more traditional signal analyzer, either benchtop or handheld. If you know what you're testing…

This is really weird. Someone creating 4 new accounts just to call this project fraudulent because it can't statically analyze every property you'd like? Does this person have a pe…

Tried it out. I'm compring against Qwen 3.5 122B-A10B, so a much larger model. It gets some correct, but Qwen 3.5 122B-A10B has done much better. Gemma 4 12B even hallucinated some…

I haven't run a proper eval, but I've been getting better luck with Qwen models than Gemma on plant and animal identification using vision. I do like Gemma for translation, however…

How could the harness fix this? It's the jinja template used by the inference engine to render the API requests into the raw text that gets tokenized and completed by the model. Un…

Much more complex than that. Even if it does give you a speedup at certain tasks, is it worth the cost and risks? You go faster, but now you have more code that you don't understan…

It means that even if it works for certain tasks, I think that the problems caused by use of LLMs outweigh their benefits. I think it's a bad idea to generate large piles of code t…

Huh? There is a Claude 4 Opus. It was released about a year ago. It is retired by now, in fact, just retired yesterday: https://platform.claude.com/docs/en/about-claude/model-depre…

Not a harness issue. The harness (pi in my case) passes back the cot for all previous turns. The jinja template is what renders the openai-format request sent by the harness, into …

Right. Local models haven't quite hit that level yet. The biggest open models, which you need tens of thousands of dollars of hardware to run at reasonable speed, have pretty much …

Which Opus? Anthropic has been releasing models named Opus since 2024 with Claude 3 Opus. Opus has gotten vastly more capable since then. Local model far surpass Opus 3. They even …

So, one of the ways that this problem manifests is that most local models aren't trained on preserving the full reasoning between turns. Every turn, they skip passing the reasoning…

Yeah, definitely.