Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Muse Spark 1.1

ai.meta.com

Post reply on HN
Print view View original

41–50 of 228 posts

Re: Muse Spark 1.1

#41

Post by pa7ch »

Lomlioto wrote: post #33
kilroy123 wrote: post #5

I personally do not like Meta, but I'll say this. The more competition, the better for regular consumers. (Enterprise too) - Chinese models - Grok - Meta - Google - OpenAI - Anthropic I think this is a win. I'm building like crazy to take advantage of all these subsidized tokens while I can.

Its the biggest technology race we have ever seen. Richest companies, smartest people, richest countries. I do not know if competition is good, we will see in a few years. Looking forward having a physical job for a change :D

A bit much describing our tech leadership as smartest people we've ever seen.

Re: Muse Spark 1.1

#42

Post by carimura »

I missed the fact that Meta was developing and releasing closed-weights models... bummer. Would be great to see some more progress with American open-weights models.

Re: Muse Spark 1.1

#43

Post by alansaber »

winstonp wrote: post #37
alansaber wrote: post #11

Earlier quoted context omitted.

This sounds... kind of useless? Really good JSON or similar constrained decoder performance is interesting, but normal decoder > tool validator loop with good error message > tool retry is almost always able to get a tool to work second try, and input is cached so it's not expensive.

The avg coding session has hundreds or thousands of tool calls. Even a 5% failure rate noticeably notches up token use and cost. See Gemini.

Yes, but each tool call has a different failure %. The tool calls that make up the majority of volume like grep are going to have nowhere near a 5% failure. A custom user-defined skill having a 5% failure rate is probably fine.

Re: Muse Spark 1.1

#44

Post by anematode »

pa7ch wrote: post #41
Lomlioto wrote: post #33

Earlier quoted context omitted.

Its the biggest technology race we have ever seen. Richest companies, smartest people, richest countries. I do not know if competition is good, we will see in a few years. Looking forward having a physical job for a change :D

A bit much describing our tech leadership as smartest people we've ever seen.

Greediest, perhaps?

Re: Muse Spark 1.1

#45

Post by 4d4m »

wolttam wrote: post #28
kilroy123 wrote: post #5

I personally do not like Meta, but I'll say this. The more competition, the better for regular consumers. (Enterprise too) - Chinese models - Grok - Meta - Google - OpenAI - Anthropic I think this is a win. I'm building like crazy to take advantage of all these subsidized tokens while I can.

To expand on Chinese models: - DeepSeek - GLM (Z.ai) - Minimax - Kimi (Moonshot) - Hy3 (Tencent) - Qwen (Alibaba) (Each one of these with weights available to download and run locally)

GLM 5.2 is great, but is so rate limited now I no longer recommend it

Re: Muse Spark 1.1

#46

Post by anthonypasq »

Everyone has been loving to shit on the Alexander Wang acquisition but this seems legitimately impressive to me?

Meta's AI org when from a total mismanaged dumpster fire for multiple years to delivering a competitive model in less than a year on essentially their first try?

Re: Muse Spark 1.1

#47

Post by Lomlioto »

pa7ch wrote: post #41
Lomlioto wrote: post #33

Earlier quoted context omitted.

Its the biggest technology race we have ever seen. Richest companies, smartest people, richest countries. I do not know if competition is good, we will see in a few years. Looking forward having a physical job for a change :D

A bit much describing our tech leadership as smartest people we've ever seen.

I would call the founders of DeepMind (Demis Hassabis, Mustafa Suleyman, Shane Legg) very smart people. Im pretty sure with the amount of funding everyone of these companies have, they have a long list of very smart researchers in their companies.

I do not mean Suckerberg or Eric Schmidt.

Re: Muse Spark 1.1

#48

Post by ycui7 »

fallingbananna wrote: post #36
Tiberium wrote: post #4

The pricing is insane: $1.25/$4.5 for 1M tokens, and $0.15 for cached input! https://dev.meta.ai/docs/getting-started/pricing-rate-limits

Meta isn’t right now on the radar for most folks picking models. If they have a really good model, it makes sense to subsidise it, to gain users, before they align prices with competitors.

this is not subsidizing. this is way too expensive for a no-name model.

Re: Muse Spark 1.1

#49

Post by GodelNumbering »

Lot more details in the linked report https://ai.meta.com/static-resource/muse-spark-1-1-evaluatio...

From Terminal-bench-2.1 details,

> We use a bash-tool-only agent harness to evaluate 89 Terminal-Bench 2.1 tasks from the official repository, where resources are capped at 6 CPU cores and 8GB RAM.

This disqualifies the results. Each terminal bench task has a cpu upper limit and RAM upper limit. Overriding either is disqualification.

For reference, in tbench-2.1,

1. 0 out of 89 task allow 6 cpu cores (highest is 4, and i think only 1 task)

2. 8 out of 89 tasks allow 8GB RAM

This kind of shady benchmarking (I was talking about it just yesterday in a different context https://news.ycombinator.com/item?id=48838212) takes all joy out of building a harness to improve benchmark performance of a model because no matter what you do, you won't beat the headline (cheating) number. This is presumably why this model is not in the official benchmark leaderboard https://www.tbench.ai/leaderboard/terminal-bench/2.1

As an ex Meta employee, this is a little sad but not massively surprising. 'Number go up' is the core performance evaluation metric until PSC is done and you move on.

Post reply on HN