Experiments with 1M Token Context Windows

I am quite new to LLMs, so probably something is lost on me. What I don't get is why we need millions or billions of tokens. Like... 3 words=4 tokens more or less, right? Surely you don't need millions of words to build context for something, right? I guess someone out there has tried LLMs with hundreds or thousands of tokens. Is there a noticeable difference for most use cases?

Maybe with image generation tokens have a different significance, and you may need millions or billions of tokens, but when it comes to pure text I don't get that.