Viewing profile — shawntan

The question I keep coming back to is whether ARC-AGI is intended to evaluate generalisation to the task at hand. This would then mean that the test data has a meaningful distribut…

This would not help if no proper constraints are established on what data can and cannot be trained on. And maybe just figuring out what the goal of the benchmark is. If it is to t…

You can have benchmarks with specifically constructed train-test splits for task-specific models. Train only on the train, then your results on test should be what is reported. You…

This is a point I wish more people would recognise.

I should probably also add: It's long been known that Universal / Recursive Transformers are able to solve _simple_ synthetic tasks that vanilla transformers cannot. Just check out…

> Do you mean that HRM and TRM are specifically trained on a small dataset of ARC-AGI samples, while LLMs are not? Or which difference exactly do hint at? Yes, precisely this. The …

Right. There should really be a vanilla Transformer baseline. With recurrence: The idea has been around: https://arxiv.org/abs/1807.03819 There are reasons why it hasn't really bee…

That analysis provided a very non-abrasive wording of their evaluation of HRM and its contributions. The comparison with a recursive / universal transformer on the same settings is…

I think everyone should read the post from ARC-AGI organisers about HRM carefully: https://arcprize.org/blog/hrm-analysis With the same data augmentation / 'test time training' set…

Not sure if you mean in general, but I'll answer both branches of the question. In general: Depending on the method of compression, you can have lossy or non-lossy compression. Usi…

The compression is lossy.

Sup!

2nd employee at Semantics3 here. Considering all the AI available today I think things like product disambiguation becomes wayyy easier. We were trying many tricks and heuristics t…

Systems might want to anticipate changes in LLM architectures (even small changes can make a big difference kernel wise), so it's good to not "bake" too much in ahead of time. That…

I'm curious how the speed is achieved is this is the technique used. Generally I expected this "masked language model" technique to be far slower since the full vocab projection ne…

The formulations in attention as rnn have similar issues as rwkv. Fundamentally it's a question of what we call an RNN. Personally I think it's important not to call some of these …

Although marketed as such, RWKV isn't really an RNN. In the recent RWKV7 incarnation, you could argue it's a type of Linear RNN, but past versions had an issue of taking its previo…

> The actual result of the paper is that any poly-time computable function can be computed with poly-many tokens. You're right. Re: NAND of two inputs. Isn't this doable even by a …

If a "problem we care about" is not stated as a formal language, does it mean it does not exist in the hierarchy of formal languages? Or is it just as yet unclassified?

Using CoT implicitly increases the depth of the circuit. But yes, poorly worded.

You can't really be blamed though, the language in the paper does seem to state what you originally said. Might be a matter of taste but I don't think it's quite accurate. The prio…

Generally, literature on the computational power of the SAME neural architecture can differ on their conclusions based on their premises. Assuming finite precision will give a more…

Theoretical results exist that try to quantify the number of CoT tokens needed to reach different levels of computational expressibility: https://arxiv.org/pdf/2310.07923 TL;DR: Ge…

There's a general trap people working on deep learning tend to fall into, thinking "Why don't we learn the activation function as well?" The answer to that really should be that a …

Yes this is a good empirical study on the types of tasks that's been shown to be impossible for transformers to generalise on. With both empirical and theoretical support I find it…