Viewing profile — in-silico

Does the free tier really not have access to reasoning models? That would explain a lot of the terrible AI/LLM takes online.

This algorithm: sample a bunch of latents, train the model using the one with the lowest error. IWAE: sample a bunch of latents, weight the loss of training the model using each on…

The training method is very similar to Importance Weighted Autoencoders: https://arxiv.org/abs/1509.00519

Did the model really need to hack huggingface to get access to ExploitGym data? I'd imagine that once it had full internet access it could have just used the HF API or website (but…

What would evidence of "intelligence" or "creativity" look like for you?

One "solution" to these problems is to have the probes land on planets instead of asteroids, and build the necessary infrastructure there.

The difference is that artistic sensibility is largely subjective. This means that: 1. It's hard to measure (and people can disagree about it) 2. It can't really be improved using …

I feel like the data should have been generated by a much less predictable policy. It often feels like the model is ignoring my inputs and just doing what it would expect the bot t…

This post misses a very important point: humans aren't 100% correct either. This means that the bar for being useful (at tasks that humans usually do) isn't perfection, it's human-…

Why can't a next token predictor do math? Humans aren't calculators either, but we can do math. If you want proof just look at the benchmarks. Modern frontier models can get basica…

Additionally, maybe it's easier for a model to realize that it doesn't know the answer when the question is easier. If Opus gets all but the hardest questions right, it might have …

Neither of these strike me as particularly groundbreaking. The first idea (as I understand it as retrieving token ids rather than hidden states) is going to really struggle to do u…

> LLMs cannot do math This is plainly not true anymore

These types of ablation studies are always good. However, I'm not sure how generalizable the language model findings here are. Their 1.2B model was trained on only 10B tokens, whic…

> The problem is "public schools". The idea itself is wrong, and it can't be made to work. Do you have an alternative idea in mind?

Care to elaborate?

This is really semantics, but I wouldn't call attending to the KV cache re-reading the context. The model takes in the context, encodes it into a "memory" (the KV cache), and acces…

> It's not changed by the experience The entire file is not changed, but the KV cache is. > It doesn't remember anything The model definitely remembers previous exchanges within th…

> Would be interesting whether it is possible to write a LLM-like program just using compression and function interpolation algoritms. gzip can be used as a (not very good) LLM-lik…

We know how the models are built and trained, but we have a very limited understanding of how the final products work . That is to say, we don't know why they give the outputs that…

> But it is another good example that "AI" is just glorified search and there is not reasoning or thinking going on behind the covers A bold claim given that the current top post o…