Viewing profile — SEGyges

i hate "stochastic parrot" because it's not even really meaningful I think it's true that models are statistical, inasmuch as P(A|B) where B is the prior sequence is what the loss …

This is true of the system as a whole, but the core neural network is still a next-token predictor.

fortunately i wrote an entire post about what the difference is between the parts of this that it is easy to make sense of and the parts of it that it is prohibitively difficult to…

I agree, what we do is much closer to growing them than to engineering them. We basically engineer the conditions for growth, and then check the results and try again. My best argu…

My short explanation would be that even for RL, you are training on a next token objective; but the next token is something that has been selected very very carefully for solving t…

Every LLM provider caches their KV-cache, it's a publicly documented technique (go stuff that KV in redis after each request, basically) and a good engineering team could set it up…

Because this specific number comes up constantly and is incredibly frustrating it seemed like it really needed to be addressed directly. If we can get past this specific thing we c…

I am not deep into the Linus weeds but my impression is that he doesn't especially care if he's on the receiving end of this. It only started to feel different from "well, the Linu…

it is not necessarily 16x if you, e.g., decrease model width by a factor of 4 or so also, but yeah naively the RAM and FLOPs scale up by n^2

you might have better luck giving the LM the original document and having it generate its own OCR independently, then asking the llm to tiebreak between its own generation and the …

tokens are on average four characters and the number of residual streams (and therefore RAM) the LLM allocates to a given sequence is proportionate to the number of units of input.…

In a world where the options were to 1) pay the author, 2) implement guaranteed citation of the author any time the model gave an answer that was directly derivative, with an optio…

These are fancy Markov chains in the sense that humans are just chemicals and computers just do math. Technically true, but not even "overly reductive"; it is just wrong if it is u…

This is the correct one.

The counterparties on related legal action are sufficiently litigious that it is probably smarter to DM the magnet link.

You are uploading 5 billion examples of . You cannot filter it manually, of course, because there are five billion of it. Given that it is the year 2024, how hard is it to be posit…

I am pretty sure if the authors were trying to license their works for this purpose we would just not use them at all; it is difficult to see under what circumstances they would st…

By "these algorithms", do you mean the ones that currently exist, or the ones that will exist next month, next year, or in 2034?

Huckabee v Bloomberg, Meta, et al

See my other comment replying to that.

You and the guy below you in this thread should probably tag me on twitter, same tag as here, I can point you. I do not especially want to leave the discord link in a frontpage hn …

I have good news about who wrote this paper

I picked the wrong day to refactor an npm build.