Viewing profile — hansonw

The skill source is here: https://github.com/openai/skills/blob/main/skills/.curated/p... $skill-installer playwright-interactive in Codex! the model writes normal JS playwright co…

Rest assured that we are better at training models than naming them ;D - New benchmark SOTAs with 77.9% on SWE-Bench-Verified, 79.9% on SWE-Lancer, and 58.1% on TerminalBench 2.0 -…

More about that here! https://platform.openai.com/docs/codex#advanced-configuratio...

The ELI5 of the paper is that most "unlearning" methods can be regarded as adding some delta `w` to the parameters of the network, but most of `w` just gets "rounded away" during q…

It looks like they didn't want to make a public submission in order to avoid disclosing the model internals: https://cosine.sh/blog/genie-technical-report#:~:text=SWE%2D... .

It’s probably more. Pretty conservatively, if the KV embedding dimension for each token is ~10K x 100 attention layers (this is roughly the scale of Llama3.1 405B) that’s already 1…

https://news.ycombinator.com/item?id=40675577

This is also a good paper on the subject: What Algorithms can Transformers Learn? A Study in Length Generalization https://arxiv.org/abs/2310.16028

https://predibase.com

Yes. But also note that the new function calling is actually “tool calling” where the model is also fine-tuned to expect and react to the output of the function (and there are vari…

Not the author, but really nice that they shared some real data points: > Once our Text-to-SQL solution was in production, we were also able to observe how users interacted with th…

If you think about it, RAG is a relatively primitive “first pass attention layer” that is binary and semi-heuristic based. I think it’s fairly safe to say that in the long term RAG…

If sub-quadratic architectures (eg Mamba) become a thing, it will become feasible to precompute most of the work for a fixed prefix (i.e. system prompt) and the latency can be pret…

Indeed: https://arxiv.org/pdf/2402.01032.pdf Perhaps future iterations of SSMs will accommodate dynamically sized (but still non-linearly-growing) hidden states / memories!

“RNN-mode inference” is also extremely exciting because you can precompute the hidden state of any prompt prefix (i.e. a long system prompt, or statically retrieved context) and co…

Our startup is building https://arcwise.app , which allows you to embed full-fledged SQL tables inside Google Sheets! We’re in the process of building out support for joins & subqu…

I’m building a solution that works like this - we directly connect spreadsheet models to company databases (even converting pivots/formulas to SQL). Would love to chat with anyone …

This is the best comparison I've found that benchmarks the current OSS inference solutions: https://hamel.dev/notes/llm/inference/03_inference.html IME the streaming API in text-ge…

gcsfuse has controllable built-in caching of _metadata_ but not contents: https://cloud.google.com/storage/docs/gcsfuse-performance-an... You'd have to use your own cache otherwise…

What about datasets like https://huggingface.co/datasets/b-mc2/sql-create-context ? An OSS model based on StarCoder was also recently published which is roughly between GPT-3.5 and…

Arcwise | https://arcwise.app | Engineering | Full-time | SF or Remote (US) At Arcwise, we’re on a mission to make analytics and data science accessible to anybody - starting with …

It appears the indexing for the model parts is deliberately not contiguous; the 03-82 range represents the main 80 transformer layers. https://github.com/yandex/YaLM-100B/blob/main…