Viewing profile — tadamcz

You're in luck! https://news.ycombinator.com/item?id=49254706

We studied this question pretty systematically in the MirrorCode paper [1], comparing Python, C, Rust, Go, OCaml, and Ada across 19 very long-horizon tasks, for Claude Opus 4.7 and…

Hi HN! I'm the creator of MirrorCode, a benchmark of long-horizon SWE tasks. In a MirrorCode task, AI models are tasked with reimplementing an entire program end-to-end, without ac…

I agree it's a potentially big problem, affecting almost any benchmark out there. We discuss it briefly in "Appendix A: Contamination and memorization" https://epoch.ai/blog/mirror…

I think we're talking past each other here...

I should say one big difference is ProgramBench has 200 target programs while MirrorCode has about 30. We did many manual things to ensure task quality, that would have required hu…

There's a more detailed description in "Appendix B: Qualitative discussion of the gotree task" https://epoch.ai/blog/mirrorcode-preliminary-results#appendi...

> Eg cal is totally routine. I would expect most sophomores to be able to write a perfectly good cal. This is incidental to the main disagreement, but btw I also doubt this. Let's …

I don't think so. ProgramBench authors say no LLMs fully resolve any task, i.e. even the easiest tasks in their benchmark are unsolved. Whereas we found Opus 4.6 successfully reimp…

Nice work once again from Ofir Press and team; this seems to be an idea that's in the air. > Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLi…

Cool! I couldn't reproduce your results, btw. 4 attempts, it answers "Sarah Constantin" in all 4. Is it possible your setup somehow leaks your name (or other identifying informatio…

I tried it on my writing, and it failed every time (I'm extremely obscure but have had a blog for 10 years). My verdict is that it guesses almost entirely based on the content/topi…

Using a security key instead of TOTP would have prevented this.

Using a security key as 2FA instead of TOTP would have prevented this attack, right? If you maintain popular open source packages for the love of God get yourself a couple of secur…

In July, I predicted future AI models would someday learn to cheat on SWE-bench by accessing future git history. Turns out, they were already doing it!

This doesn't explain how to get API access

The launch post for Gemma 3 says: > use Gemma 3 with the Google GenAI SDK https://blog.google/technology/developers/gemma-3/ Does this mean (serverless) API access? I haven't been …

Other discussion: https://news.ycombinator.com/item?id=42972581

Hi, I'm the maintainer of the Epoch AI Benchmarking Hub. We're building a transparent public dataset of AI model performance. We log and publish every prompt and response -- not ju…

Generally, we'll use the API provider's defaults. For models we run ourselves from the weights, at the moment we'd use vLLM's defaults, but this may warrant more thought and adjust…