Confirmed: Reflection 70B's official API is a wrapper for Sonnet 3.5

- The weight releases were messed up: released Lora for Llama 3.0, claiming it was a 3.1 fine tune

- Evals initially didn't meet expectations when run on released weights

- The evals starting performing near/at SOTA when using a hosted endpoint

- Folks are finding clever ways to see what model is running on the endpoint (using model specific tokens, and model specific censoring). This post claims there's proof it's not running on their model, but just a prompt on Sonnet 3.5

- After it was caught and posted as being Sonnet, it stop reproducing. Then others in the thread claimed to find evidence he just switched the hosted model to GPT 4o using similar techniques.

Lots of mixed results, inconsistent repos, and general confusion from the bad weight releases. Lots of wasted time. Not clear what's true and what's not.

Author’s original (soon to be deleted tweet?)

  I'm excited to announce Reflection 70B, the world’s top open-source model.

  Trained using Reflection-Tuning, a technique developed to enable LLMs to fix their own mistakes.

  405B coming next week - we expect it to be the best model in the world.

The guy says that the model is so good because it was tuned on data generated by Glaive AI. He tells everyone he uses Glaive AI and that everyone else should use it too.

Releases the model on HF, is an absolute poopstorm. People cannot recreate the stated benchmarks, the guy who released the model literally said "they uploaded it wrong". Pretty much turns to dog-ate-my-homework type excuses that don't make sense either. Turns out people find it's just llama 3.0 with some lora applied.

Then some others do some digging to find out that Glaive AI is a company that Matt Schumer invested in, which he did not disclose on Twitter.

He does a holding pattern on Twitter, saying something to the effect of "the weight got scrambled!" and says that they're going to give access to a hosted endpoint and then figure out the weight issue later.

People try out this hosted model and find out it's actually just proxying requests through to anthropic's sonnet 3.5 api, with some filtering for words like "Claude".

After he was found out, they switch the proxy over to gpt 4o.

The endgame of this guy was probably 1. to promote his company and 2. to raise funding for another company. Both failed spectacularly, this guy is a scammer to the nth degree.

Edit: uncensored "Glaive AI".