Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Muse Spark 1.1

ai.meta.com

Post reply on HN
Print view View original

121–130 of 228 posts

Re: Muse Spark 1.1

#121

Post by rsanek »

GodelNumbering wrote: post #49

Lot more details in the linked report https://ai.meta.com/static-resource/muse-spark-1-1-evaluatio... From Terminal-bench-2.1 details, > We use a bash-tool-only agent harness to evaluate 89 Terminal-Bench 2.1 tasks from the official repository, where resources are capped at 6 CPU cores and 8GB RAM. This disqualifies the results. Each terminal bench task has a cpu upper limit and RAM upper limit. Overriding either is…

This doesn't seem that big of a deal to me? I mean, in any other area where I want an assessment of a product, I'm not going to trust what the product producer says about it at face value -- obviously they're going to be biased. This is the whole raison d'etre for independent testing, like https://artificialanalysis.ai.

Re: Muse Spark 1.1

#123

Post by meric_ »

solarkraft wrote: post #119
phillipcarter wrote: post #32

My trust factor is gone with Meta right now. Has there been any independent analysis to confirm they didn't cheat on benchmarks again?

They cheated again: https://news.ycombinator.com/item?id=48847019

No they didn't. Please read: https://www.anthropic.com/engineering/infrastructure-noise

Re: Muse Spark 1.1

#124

Post by disgruntledphd2 »

Lomlioto wrote: post #68
ianm218 wrote: post #51

Earlier quoted context omitted.

> At least in China a lot of software developers are now struggling. Do you think that Chinese software industry is that relevant to the kind of software market talked about on HN? I.e. lots of enterprise b2b and infra companies. Chinese companies have always had a very low willingness to pay for software which kinda breaks the flywheel of B2B SaaS companies and companies to service those companies all the way down.

Its a signal. They were earning well and AI crashed the market in China.

China is in a very different economic state to the West (broadly construed).

They have had real issues with deflation rather than the inflation most Western countries have seen over the past five years.

Re: Muse Spark 1.1

#125

Post by mchusma »

Tiberium wrote: post #4

The pricing is insane: $1.25/$4.5 for 1M tokens, and $0.15 for cached input! https://dev.meta.ai/docs/getting-started/pricing-rate-limits

Yeah, this is most directly comparable to xAI Grok 4.5. In both cases, directionally "opus level intelligence for haiku prices" which is a really big deal for application developers who want to include models like this in their applications. I have been testing switching out haiku and sonnet for Grok 4.5, and may give this a try too (it is quite a bit cheaper, particularly for cached).

Re: Muse Spark 1.1

#126

Post by GodelNumbering »

meric_ wrote: post #120
GodelNumbering wrote: post #49

Lot more details in the linked report https://ai.meta.com/static-resource/muse-spark-1-1-evaluatio... From Terminal-bench-2.1 details, > We use a bash-tool-only agent harness to evaluate 89 Terminal-Bench 2.1 tasks from the official repository, where resources are capped at 6 CPU cores and 8GB RAM. This disqualifies the results. Each terminal bench task has a cpu upper limit and RAM upper limit. Overriding either is…

Huh? What are you talking about? https://www.anthropic.com/engineering/infrastructure-noise Is anthropic benchmark maxxing and cheating on terminal bench too? They don't follow the strict resource "limits" either

What that link describes is basically the motivation to go from terminal bench 2.0 to 2.1. The latter simply fixed the common issues/complaints. There is a long github discussion on tbench's about it

Re: Muse Spark 1.1

#127

Post by hraxz »

cpt100 wrote: post #19
kilroy123 wrote: post #5

I personally do not like Meta, but I'll say this. The more competition, the better for regular consumers. (Enterprise too) - Chinese models - Grok - Meta - Google - OpenAI - Anthropic I think this is a win. I'm building like crazy to take advantage of all these subsidized tokens while I can.

Yeah, I think it is definitely great. Having said that, I am still debating in my mind whether the volume of software engineers needed in the AI era is going to increase or decrease because of all of these advancements. On the one hand, because it is easy to build products, more and more people will build. And more and more products and features will be built. However, a lot of people who are non-technical will also…

The big thing to me is why are we even running these models on top of an operating system?

What I really want is Claude as a deep part of the operating system.

If that happens then a whole lot of the abstraction of software vanishes along with what we think of today as software jobs. I think many new forms of knowledge work would emerge from this though.

I would think that needs massive local compute but I can't imagine that is not the future down the line.

Re: Muse Spark 1.1

#128

Post by meric_ »

GodelNumbering wrote: post #126
meric_ wrote: post #120

Earlier quoted context omitted.

Huh? What are you talking about? https://www.anthropic.com/engineering/infrastructure-noise Is anthropic benchmark maxxing and cheating on terminal bench too? They don't follow the strict resource "limits" either

What that link describes is basically the motivation to go from terminal bench 2.0 to 2.1. The latter simply fixed the common issues/complaints. There is a long github discussion on tbench's about it

Yes but my point is - Resource limits are a "recommendation" and are not strictly enforced - Significantly boosting resources up to 3 did not statistically shift performance results

Sure for old tasks you could argue that now its not required to boost because infra errors are alleviated with better default limits. My point more so is that its a strange thing to index on because if you wanted to cheat on the benchmark, it does not particularly seem like something that shifts results? Once the API is out maybe I'll eat my words, but I don't really believe that if you manually tried to reproduce the results with lower limits you'd see significantly different results

Re: Muse Spark 1.1

#129

Post by iFire »

ai_fry_ur_brain wrote: post #86
Tiberium wrote: post #4

The pricing is insane: $1.25/$4.5 for 1M tokens, and $0.15 for cached input! https://dev.meta.ai/docs/getting-started/pricing-rate-limits

This is still ridiculously expensive imagine having to pay $10 for 100 search results on Google, thats essentially what this is. I really dont see how anyone's willing spend more than $1.50 per mm output. Let alone $15-50. Does anyone actually pay for usage based billing as a consumer?

This is pretty cheap compared to anthropic opus and fable.

https://platform.claude.com/docs/en/about-claude/pricing

Model Base Input Tokens 5m Cache Writes 1h Cache Writes Cache Hits & Refreshes Output Tokens

Claude Fable 5 $10 / MTok $12.50 / MTok $20 / MTok $1 / MTok $50 / MTok

Claude Opus 4.8 $5 / MTok $6.25 / MTok $10 / MTok $0.50 / MTok $25 / MTok

Note Fable costs $50 MTok and Opus 4.8 costs $25 / MTok.

Re: Muse Spark 1.1

#130

Post by solarkraft »

meric_ wrote: post #123
solarkraft wrote: post #119

Earlier quoted context omitted.

They cheated again: https://news.ycombinator.com/item?id=48847019

No they didn't. Please read: https://www.anthropic.com/engineering/infrastructure-noise

Thanks for the read. It seems to confirm that resource limits are an important factor for terminal benchmarks:

> The extra resources enable the agent to try approaches that only work with generous allocations, such as pulling in large dependencies, spawning expensive subprocesses, and running memory-intensive test suites.

> An agent that writes lean, efficient code very fast will do well under tight constraints. An agent that brute-forces solutions with heavyweight tools will do well under generous ones. Both are legitimate things to test, but collapsing them into a single score without specifying the resource configuration makes the differences—and real-world generalizability—hard to interpret.

So changing the resource limits changes the benchmark. Yet their score table claims their score to be for Terminal-Bench 2.1, not Terminal-Bench 2.1 with raised limits.

Post reply on HN