Earlier quoted context omitted.
It falls under the "nominative use" defense to an assertion of trademark infringement. Free speech has nothing to do with it. Trademarks are intended to provide consumers with the confidence that the products they're buying really come from the producer they claim to come from. As such, you're prohibited from assembling your own computer and branding it Apple; you've never been prohibited from referring to Apple.
Why do retail entities refer to the Super Bowl by pseudonyms (e.g "The Big Game") rather than its true name?
In summary:
"If “Super Bowl” is used by an individual or business to describe the event and not used for commercial intent, then individuals and businesses can use the term. For example, a grocery store can say “Make sure to stock up on chips for the Super Bowl” but the store cannot say they are the “Official Super Bowl Chip Stop.”"
That said, you can just avoid infringing entirely by calling it The Big Game