Viewing profile — marcelroed

Yes! I will publish a Rust crate soon. If you have thoughts about how to structure the API I would love to hear them.

The output tokens are identical in either case, but there are quite a few additional settings and formats that huggingface compat mode can generate. In general it also requires inp…

Thanks for the kind words, Craig! I'm planning to do a technical writeup+paper and a presentation video on the project in the near future. Will make sure to share it with the Disco…

Added numbers here: https://news.ycombinator.com/item?id=49015014

Author here: Actually, depending on the nature of the inference you're doing it can be quite significant. Here are some numbers for time-to-first-token (time to process the entire …

Running the numbers now

It's usually not as binary as "hit" or "miss" with a prefix cache, and you need to know the token boundaries to know where the cache hit ends. The current structures used for KV-ca…

Time to first token refers to the time until the model outputs one token, which includes the time to process the entire prompt (doing prefill). The GPU time per token is much lower…

I can add some benchmarks for compatibility mode in the future. I have a little more juice to squeeze out of the Python interop though, so not quite ready for it yet.

Numbers are for the Gigatoken API, but compatibility mode just means eating a bunch of Python overhead (creating lists, reading strings to bytes). You can expect a modest ~200-300x…

You can, but this usually results in sequences with padding/truncation, since you won't know how many tokens your inputs map to before you actually tokenize them. This also makes s…

Author here! In my case it's mostly pretraining experiments, where you might want to change your data mixture/filtering/processing of training data, and splits are usually done at …

We have autograding for code through tests written by hand, and additionally do manual code audits if we see suspicious behavior. We also do grading the old-fashioned way for write…

TA here. Noted! I now have more resources to test more environments, and will do so whenever possible. I think freezing due to memory overuse is going to be a problem with anything…

We do provide resources for enrolled students. The online suggestions are for external students or Stanford students who we weren't able to admit.

TA here. Biggest changes are in the second assignment (distributed) where we added a bunch of memory, profiling and distributed tasks, as well as in the fifth assignment (alignment…

TA here. Definitely not! In fact we explicitly added sections in the first assignment to allow for scaling down to even local compute (M-series GPUs). For assignment 2 there are a …

I am only familiar with MLIR for accelerator-specific compilation, but my understanding is that by describing operations at a higher level, you don’t need the frontend to know what…

After trying this out on my iphone, I can say your conclusion is exactly correct. The icons look subtly out of focus in a way that's quite unsettling.

It’s worth noting that Shreyas (the first author) was a student with Tenenbaum at MIT before he went to Berkeley