Viewing profile — itkovian_

Causal masking allow model to learn implicit positional embeddings. The meme that a transformer block is permutation invariant is not true.

Intrinsic motivation/curiosity driven exploration/novelty seeking feels like it has a breakthrough paper waiting. If someone could get those methods working for LLMs, we start gett…

the difference is all the current capex is going to durable, hard to get physical assets + things like PPAs. In your online shopping analogy, the hyperscalars are acting like Amazo…

[dead]

I just don’t understand this view. This is the most significant technology ever developed. The uncertainty currently is whether it 1) has massive impact, completely altering societ…

‘640kb of ram should be enough for anyone’

Contrarian view; I think he’s right. Many of these ideas it’s almost shocking how many you can find sketched out in his old papers. To the point where I think it’s very wise to rea…

You can’t run a closed llm locally. Strange to frame the dichotomy as between local and open. One begets the other.

The entirety of Anthropic believe ai is going to eat everything, not just software, and result in major societal disruption within a year. They do not have a sliver of a doubt on t…

This is called linear mode connectivity and seems to work for almost every large model. So well that in most cases it’s an explicit part of the training process; do many training ‘…

Projects like pluralis agora solve this problem. Really what you want is the model to be collectively owned and governed, not local

People forget the people in charge of these companies are some of the smartest people out there full stop. Far more shadowy strategy/things like this going on than people think.

What are the odds this is partially them making the point; you were all complaining about monitoring/access/safeguards: remember we don’t have to give this to you at all. And using…

In many cases the GPs are not, at least nowhere near as much as you’d think. Obviously there are power laws here as well. Non partners, forget about it. The other thing is it is ma…

The extreme privilege of being forced to pay a major portion of all income you make, regardless of where you earn it, to the us gov indefinitely. And they make it hard for you to a…

The US isn’t what it used to be. It’s definitely not the best place in the world to live for quality of life, on basically any metric. The requirement of being permanently obligate…

The other thing people don’t understand is exponential curves are self similar. The start of an exponential looks like an exponential. People always look at and think ‘well that’s …

The argument is that there is no incentive to carefully review a paper (I agree), however what used to occur is people would do the right thing without explicit incentives. This ha…

Whether it’s actually 20% or not doesn’t matter, everyone is aware the signal of the top confs is in freefall. There are also rings of reviewer fraud going on where groups of peopl…

These are some of the richest entities - forget about universities - just entities full stop, in the entire country.

I don’t think people understand the point sutton was making; he’s saying that general, simple systems that get better with scale tend to outperform hand engineered systems that don…

I’m gonna go ahead and guess they didn’t raise 8.3b on SAFEs

Article doesn’t say jobs aren’t about to be evicerated, says this is already happening and it’s due to capitalism, a lack of consumer protections and we require more government reg…

The reason for this is it’s horrifying to consider that things like the Ukrainian war didn’t have to happen. It provides a huge amount of phycological relief to view these events a…