100M Token Context Windows
magic.dev
100M Token Context Windows
1–10 of 24 posts
Re: 100M Token Context Windows
#2It should be benchmarked against something like RULER[1]
1: https://github.com/hsiehjackson/RULER (RULER: What’s the Real Context Size of Your Long-Context Language Models)
Re: 100M Token Context Windows
#3Based on Mamba ?
Re: 100M Token Context Windows
#4It should be benchmarked against something like RULER[1] 1: https://github.com/hsiehjackson/RULER (RULER: What’s the Real Context Size of Your Long-Context Language Models)
[deleted]
Re: 100M Token Context Windows
#5It should be benchmarked against something like RULER[1] 1: https://github.com/hsiehjackson/RULER (RULER: What’s the Real Context Size of Your Long-Context Language Models)
> To incorporate this, we ask the model to complete a chain of hashes instead (as recently proposed by RULER):
They did mention it but didn't provide concrete benchmarks
Re: 100M Token Context Windows
#6FYI wouldn't interview here. Got rejected after a 30 minute behavioral screen after spending 8 hours on an unpaid take-home.
Re: 100M Token Context Windows
#7Long context windows are IMO, “AGI enough.”
100M context window means it can probably store everything you’ve ever told it for years.
Couple this with multimodal capabilities, like a robot encoding vision and audio into tokens, you can get autonomous assistants than learn your house/habits/chores really quickly.
Re: 100M Token Context Windows
#8I was wondering how they could afford 8000 H100’s, but I guess I accidentally skipped over this part:
> We’ve raised a total of $465M, including a recent investment of $320 million from new investors Eric Schmidt, Jane Street, Sequoia, Atlassian, among others, and existing investors Nat Friedman & Daniel Gross, Elad Gil, and CapitalG.
Yeah, I guess that'd do it. Who are these people and how'd they convince them to invest that much?
Re: 100M Token Context Windows
#9Context windows are becoming larger and larger, and I anticipate more research focusing on this trend. Could this signal the eventual demise of RAG? Only time will tell.
I recently experimented with RAG and the limitations are often surprising (https://www.lycee.ai/blog/rag-fastapi-postgresql-pgvector). I wonder if we will see some of the same limitations for long context LLM. In context learning is probably a form of semantic / lexical cues based arithmetic.
Re: 100M Token Context Windows
#10I was wondering how they could afford 8000 H100’s, but I guess I accidentally skipped over this part: > We’ve raised a total of $465M, including a recent investment of $320 million from new investors Eric Schmidt, Jane Street, Sequoia, Atlassian, among others, and existing investors Nat Friedman & Daniel Gross, Elad Gil, and CapitalG. Yeah, I guess that'd do it. Who are these people and how'd they convince them to in…
For those names (access to $billions), curious how much due diligence they do any more. Just make a “chump change” investment in every hot trend? One phony AI startup pitch deck will look identical (if not better) to one with a real edge.