Viewing profile — nielstron

Yes that's a great summary and I agree broadly. Note with different prompt types I refer to different types of meta-prompts to generate the AGENTS.md. All of these are quite useles…

It could... but as pointed out by other the significance is unclear and per-model results have even less samples than the benchmark average. So: maybe :)

Hey thanks for your review, a paper author here. Regarding the 4% improvement for human written AGENTS.md: this would be huge indeed if it were a _consistent_ improvement. However,…

This is life of an LLM researcher. We literally ran the last experiments only a month ago on what were the latest models back then...

Exactly my thoughts... the model should just auto ingest README and CONTRIBUTING when started.

Hey, paper author here. We did try to get an even sample - we include both SWE-bench repos (which are large, popular and mostly human-written) and a sample of smaller, more recent …

Hey, a paper author here :) I agree, if you know well about LLMs it shouldn't be too surprising that autogenerated context files are not helping - yet this is the default recommend…

Debunking the Claims of K2-Think https://www.sri.inf.ethz.ch/blog/k2think

Debunking the Claims of K2-Think https://www.sri.inf.ethz.ch/blog/k2think

noted. we'll make sure to critizise turing complete type systems more thoroughly next time :))

Yes this work is super cool too! Note that LSPs can not guarantee resolving the necessary types that we use to ensure the prefix property, which we leverage to avoid backtracking a…

thank you!

re detecting and switching language: you could run several constraint systems in parallel and switch as soon as one of them rejects the input and another accepts it re backtracking…

the problem with LSPs is that they don't guarantee generating a type annotation that we can use for constraints, i.e. we can not ensure the prefix property using LSPs. so we had to…

The downside is that you need to properly preprocess code, have less non-code Training Data, and can not adapt easily to new programming languages

we were thinking about doing exactly this, the closest current work is probably the amazing "Learning Formal Mathematics from Intrinsic Motivation" by Poesia et al (they use constr…