Viewing profile — kir-gadjello

I don't think it's strictly better than GLM 5, more like they are peers (but in math competitions StepFun is stronger than most), and in my experience have similar coding/bugfix ce…

Much worse - from my experience minimax is not suitable for high autonomy on hard projects. The real distant second in my experience is mimo flash v2 (but I did not try the latest …

Yeah, my github is in the profile. Soon (tm). Feel free to follow.

I use pi, but I'm almost done writing a better alternative that doesn't have pi's stability issues. 80K Rust SLOC and a few hundred tests btw.

They are not equivalent 1:1, esp. in knowledge coverage (given OOM param size difference) and in taste (Sonnet wins, but for taste one can also use Kimi K2.5), but in my hardcore u…

I think we are at this point where the hard ceiling of a strong model is pretty hard to delineate reliably (at least in coding, in research work it's clearer ofc) - and in a good s…

I just use openrouter, it's free for now. But I would pay 30-100$ to use it 24/7.

Respectfully, from my experience and a few billions of tokens consumed, some opensource models really are strong and useful. Specifically StepFun-3.5-flash https://github.com/stepf…

While llama3-8b might be slightly more brittle under quantization, llama3-70b really surprised myself and others[1] in how well it performs even in the 2..3 bits per parameter regi…

Synthetic data researchers, connoisseurs and artisans, obviously. Feedback loop architects. Mindscaping artists.

It is quite likely GPT-4 uses one or even two sparsity approaches on top of each other (namely, coarse grained switch transformer-like and fine grained intra-tensor block sparsity)…

It is a guess informed by some familiarity with the literature and by going over the papers authored by researchers credited in the OpenAI's "GPT-4 contributors" web page. I have a…

It allows for modifications and commercial use: https://creativecommons.org/licenses/by-sa/4.0/ >You are free to: >Share — copy and redistribute the material in any medium or forma…

Impressive model, thank you for releasing it under a business-friendly license! Have you considered using Google's sparse "scaling transformer" architecture as the base? Even at 3B…

Thank you for developing the pipeline and amassing considerable compute for gathering and preprocessing this dataset! I'm not sure if this is the right place to ask about this, but…

That's cool, thanks for noting, Alan! Would you mind adding a reference link to the source, so that other people could visit my blog? I'm just starting out with blogging, it would …

This is cool, but SSD read bandwidth is still the bottleneck. On my non-mac machine it still takes several seconds to load the model.

I think envying closed source closed weights proto-AGI systems is counterproductive. We have opensource models with available weights that are almost as powerful: https://huggingfa…

This document doesn't contain the architecture and training details of GPT-4. As an engineer, these details would be the most interesting part of it! Driven by interest in GPT-4 an…

No, a typical LLM is a pure function of its input, if you (and not the LLM hosting company) control all of input context , and if your sampler uses pseudorandom number generator. B…

Given a list of contributors it's not that hard to reverse-engineer the specific engineering choices made by looking up their publication history. My analysis https://kir-gadjello.…

It could be done in a dozen ways. One beautiful method is just using the xPos positional embedding pioneered by Microsoft and scale the context window size at runtime (even better …

If you have questions about my rationale for this or that technique included in the list, please, ask! For example, I think Google's paper "Sparse is enough for scaling transformer…

It's no problem to put model's architecture and even some python code into the generous 32k context window, the real problem seems to be as you say "awareness" - at least the facet…