Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Evidence suggesting Quasar Alpha is OpenAI's new model

blog.kilocode.ai

Post reply on HN
Print view View original

11–20 of 49 posts

Re: Evidence suggesting Quasar Alpha is OpenAI's new model

#11

Post by daemonologist »

To some extent the "mystery" (and temporary free-as-in-beer-ness) of this model might be getting to me, but I think it's pretty interesting. Given the token throughput (250B this week) it's obvious there's a pretty major player behind the model, but why is it stealthed? Maybe there's something about the architecture or training that would put people off if it was public right off the bat? Maybe they're purely collecting usage/acceptance data and want unbiased users?

On the Aider Polyglot leaderboard it's ~middle of the leading pack, comparable to DeepSeek V3 and 3.5 Sonnet. I ran NoLi(teral)Ma(tching), an unsaturated long-context benchmark, on it and was impressed though:

  = Model =========== Base Score = 8K Context = 16K Context =
  Quasar Alpha:       >=97.8%      89.2%        85.1%
  GPT-4o:             99.3%        89.2%        81.6%
  Llama 3.3 70B:      97.3%        72.1%        59.5%
  Gemini 1.5 Pro:     92.6%        63.9%        55.5%
  Claude 3.5 Sonnet:  87.6%        61.7%        45.7%
  Gemini 1.5 Flash:   84.7%        44.4%        35.5%
  GPT-4o mini:        84.9%        32.6%        20.6%
  Llama 3.1 8B:       76.7%        31.9%        22.6%
It also performs well - slightly better than GPT-o1 - on the "hard" subset at 16K context with 62.8%. Latency is quite good as well.

More details: https://old.reddit.com/r/LocalLLaMA/comments/1ju1czn/quasar_...

Re: Evidence suggesting Quasar Alpha is OpenAI's new model

#12

Post by anotherpaulg »

The graph in the article about Quasar Alpha’s coding skill is taken from my tweet [0]. It shows QA’s results on the aider polyglot coding benchmark [1].

QA seems to be a skilled coder, and is very fast.

Aider supports Quasar Alpha as of v0.81, released last week.

[0] https://x.com/paulgauthier/status/1907996176605220995

[1] https://aider.chat/docs/leaderboards/

Re: Evidence suggesting Quasar Alpha is OpenAI's new model

#13

Post by atemerev »

svachalek wrote: post #10

Oof, it started strong and then went with the "who created you" which shows no understanding of how LLMs work. (They don't know a thing about themselves, they will either regurgitate their prompt or hallucinate something, usually that they are ChatGPT since that is the most likely LLM to appear in training data.)

And yet, if you ask Claude, Llama, or xAI who they are, they answer correctly. Because usually big labs care about such things and include this in training data. So, not a proof, but some evidence.

Re: Evidence suggesting Quasar Alpha is OpenAI's new model

#14

Post by daemonologist »

krackers wrote: post #8
fpgaminer wrote: post #7

I ran an interesting benchmark/experiment yesterday, which did not do Quasar Alpha any favors (from best to worst, score is an average of four runs): "google/gemini-2.5-pro-preview-03-25" => 67.65 "anthropic/claude-3.7-sonnet:thinking" => 66.76 "anthropic/claude-3.7-sonnet" => 66.23 "deepseek/deepseek-r1:free" => 54.38 "google/gemini-2.0-flash-001" => 52.03 "openai/o3-mini" => 47.82 "qwen/qwen2.5-32b-instruct" => 44.…

Didn't they say they were going to open-source some model? "Fast and good but not too cutting-edge" would be a good candidate for a "token model" to open-source without meaningfully hurting your own bottom line.

I'd be pleasantly surprised - GPT-4o is their bread and butter (it powers paid ChatGPT) and QA seems to be slightly ahead on benchmarks at similar or lower latency (so very roughly, it might be cheaper to run).

Re: Evidence suggesting Quasar Alpha is OpenAI's new model

#15

Post by alphabettsy »

atemerev wrote: post #13
svachalek wrote: post #10

Oof, it started strong and then went with the "who created you" which shows no understanding of how LLMs work. (They don't know a thing about themselves, they will either regurgitate their prompt or hallucinate something, usually that they are ChatGPT since that is the most likely LLM to appear in training data.)

And yet, if you ask Claude, Llama, or xAI who they are, they answer correctly. Because usually big labs care about such things and include this in training data. So, not a proof, but some evidence.

It’s likely not in training data but system instructions. It makes sense that a stealth model wouldn’t.

Re: Evidence suggesting Quasar Alpha is OpenAI's new model

#16

Post by jug »

The most fun way I’ve seen users explore its origin is to give it a single period (”.”) as your first query. Only OpenAI answers in this way with a smiley at the end, and it’s probably a more certain way to check it than asking about its arch, because many will incorrectly answer OpenAI and GPT-4 due to (?) tainted training data as ChatGPT has been so much in the news and became a de facto LLM early.

Re: Evidence suggesting Quasar Alpha is OpenAI's new model

#18

Post by boleary »

jug wrote: post #16

The most fun way I’ve seen users explore its origin is to give it a single period (”.”) as your first query. Only OpenAI answers in this way with a smiley at the end, and it’s probably a more certain way to check it than asking about its arch, because many will incorrectly answer OpenAI and GPT-4 due to (?) tainted training data as ChatGPT has been so much in the news and became a de facto LLM early.

So I just did this and got this response:

“ Hello! How can I assist you today? “

Re: Evidence suggesting Quasar Alpha is OpenAI's new model

#19

Post by energy123 »

atemerev wrote: post #13
svachalek wrote: post #10

Oof, it started strong and then went with the "who created you" which shows no understanding of how LLMs work. (They don't know a thing about themselves, they will either regurgitate their prompt or hallucinate something, usually that they are ChatGPT since that is the most likely LLM to appear in training data.)

And yet, if you ask Claude, Llama, or xAI who they are, they answer correctly. Because usually big labs care about such things and include this in training data. So, not a proof, but some evidence.

The labs put themselves in the system prompt at inference time. Without that the model will hallucinate a creator, most likely ChatGPT

Re: Evidence suggesting Quasar Alpha is OpenAI's new model

#20

Post by atemerev »

energy123 wrote: post #19
atemerev wrote: post #13

Earlier quoted context omitted.

And yet, if you ask Claude, Llama, or xAI who they are, they answer correctly. Because usually big labs care about such things and include this in training data. So, not a proof, but some evidence.

The labs put themselves in the system prompt at inference time. Without that the model will hallucinate a creator, most likely ChatGPT

This is what Llama4 replies (locally running, no system prompt):

"I'm Llama, a Meta-designed model here to adapt to your conversational style. Whether you need quick answers, deep dives into ideas, or just want to vent, joke or brainstorm—I'm here for it. What's on your mind?"

The behavior you are describing is from some long time ago, probably early 2024.

Post reply on HN