Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic’s $5B, 4-year plan to take on OpenAI

techcrunch.com

Post reply on HN
Print view View original

481–490 of 504 posts

Re: Anthropic’s $5B, 4-year plan to take on OpenAI

#481

Post by anentropic »

rnk wrote: post #446
anentropic wrote: post #336

Earlier quoted context omitted.

1. You're working backwards from a desire to buy more RAM to try and find uses for it. I'm really not I had no desire at all until a couple of weeks ago. Even now not so much since it wouldn't be very useful to me But the current LLM business model where there are a small number of API providers, and anything built using this new tech is forced into a subscription model... I don't see it sustainable, and I think the…

It sounds like we are in a similar position. I had no desire to get a 64gb laptop from apple until all the interesting things from running llama locally came out. I wasn't even aware of the specific benefit of that uniform memory model on the mac. Now I'm looking at do I want to do 64, 96 or 128gb. For an insane amount of money, 5k for that top end one.

The unified memory ought to be great for running LLaMA on the GPU on these Macbooks (since it can't run on the Neural Engine currently)

The point of llama.cpp is most people don't have a GPU with enough RAM, Apple unified memory ought to solve that

Some people have it working apparently:

https://github.com/remixer-dec/llama-mps

Re: Anthropic’s $5B, 4-year plan to take on OpenAI

#482

Post by anentropic »

rnk wrote: post #438
mike_hearn wrote: post #313

Earlier quoted context omitted.

If I were Apple I'd be thinking about the following issues with that strategy: 1. That RAM isn't empty, it's being used by apps and the OS. Fill up 64GB of RAM with an LLM and there's nothing left for anything else. 2. 64GB probably isn't enough for competitive LLMs anyway. 3. Inferencing is extremely energy intensive, but the MacBook / Apple Silicon brand is partly about long battery life. 4. Weights are expensive t…

I've been looking into buy a mac for llm experimentation - 64, 96 or 128gb of ram? I'm trying to decide if 64gb is enough, or should I go to 96gb or even 128gb. But it's really expensive - even for an overpaid software engineer. Then there's the 1 or 2 tb storage question. Apple list price is another $400 for that second tb of storage. For 64gb of ram, you can get an m2 pro, or get 96gb which requires the upgraded cp…

Check out the LLaMA memory requirements on Apple Silicon GPU here: https://github.com/remixer-dec/llama-mps

Re: Anthropic’s $5B, 4-year plan to take on OpenAI

#483

Post by anentropic »

rldjbpin wrote: post #473
anentropic wrote: post #288

If Apple would wake up to what's happening with llama.cpp etc then I don't see such a market in paying for remote access to big models via API, though it's currently the only game in town. Currently a Macbook has a Neural Engine that is sitting idle 99% of the time and only suitable for running limited models (poorly documented, opaque rules about what ops can be accelerated, a black box compiler [1] and an apparent…

I can see how the apple silicon memory system can help with LLMs, but a couple points of reality check: - such amounts of memory is locked behind very expensive sku which even most of mac userbase will not use ( very conservative ). - not too long ago apple would restrict the amount of ram in their system for their own reasoning (source: https://9to5mac.com/2016/10/28/apple-macbook-pro-16gb-ram-li... ) - just like mi…

That's today... think of the future

My first computer had 512KB RAM and 20MB was an expensive hard drive.

64GB Macbooks are currently an expensive 'Pro' novelty, they will be the vanilla of tomorrow

> i don't see the ml accelerators/gpu in current models being capable enough to make the most of the memory available to it

that's exactly my point (and apparently today's Neural Engine can't even take advantage of all the unified memory available)

until LLaMA there was no reason to have more than this, they probably imagined it would just run a bit of face-detection and speech-to-text on the side

but if they got serious and beefed it up it could be the next wave of computing IMHO

Re: Anthropic’s $5B, 4-year plan to take on OpenAI

#484

Post by lobocinza »

littlestymaar wrote: post #411
mike_hearn wrote: post #333

Earlier quoted context omitted.

1. You're working backwards from a desire to buy more RAM to try and find uses for it. You don't actually need more RAM to use LLMs, ChatGPT requires no local memory, is instant and is available for free today. 2. Why would anybody be satisfied with a 64GB model when GPT-4 or 5 or 6 might even be using 1TB of RAM? 3. That may not be the case. With every day that passes, it becomes more and more clear that large LLMs…

> is instant and is available for free today. It's free for the user up to a point, but it costs OpenAI a lot of money. Apple is a hardware vendor, so commoditization of the software while finding more market segments is definitely something that'd benefit them. OTOH, if they let OpenAI become the unrivaled leader of AI that end up being the next Google, they end up losing on a topic they wanted to lead for long time…

"A lot of money" is a lot less money per user than to buy 64GB RAM to run an inferior model locally + energy and opportunity costs. The OpenAI APIs are super cheap for a single user needs. I expect them to be at least close to breaking even with their APIs pricing.

Re: Anthropic’s $5B, 4-year plan to take on OpenAI

#485

Post by dangond »

Yizahi wrote: post #409
dangond wrote: post #280

Earlier quoted context omitted.

You likely wouldn't respond to that question with "lemons are yellow" without being in a specific context, such as being told to answer the question in an absurd way. GPT-* can definitely do the same thing in the same context, so this isn't really a gotcha. Literal first try with GPT-4: Me: I will ask you a question, and you will give me a completely non-sequitur response. Does that make sense? GPT-4: Pineapples enjo…

No, the point is, can it DECIDE to do so? Without being prompted? For example can the following dialog happen (no previous programming, cold start): Q: How much is two plus two? A: Four. Q: How much is two plus two? A: Banana. It can happen with a human, but not with program. Again, I don't pretend that my simple example invented in half a minute has a significance. I can accept that it can be partially or completely…

I urge you to think about what you mean by "It can happen with a human."

I guarantee you that if you try this with humans 1,000,000 times (cold start), you will never get the result you are suggesting is possible. In fact, most results will be of the following form:

Q: How much is two plus two?

A: Four.

Q: How much is two plus two?

A: Four. / Four? Why are you asking me again? / ...Four. / etc.

In the end, I think the question is not about whether NNs are themselves operating in a way similar to human cognition. The question is whether or not they can successfully simulate human cognition, and at this point, there seems to be increasing evidence that they will be able to fully do so quite soon. We are quickly running out of fields where we can point and say, "there is no way a NN can do THIS kind of task, because X." Cognition, it turns out, is not something intrinsically special about humans, and it feels foolish (to me) to continue to believe so after recent developments.

Re: Anthropic’s $5B, 4-year plan to take on OpenAI

#486

Post by littlestymaar »

lobocinza wrote: post #484
littlestymaar wrote: post #411

Earlier quoted context omitted.

> is instant and is available for free today. It's free for the user up to a point, but it costs OpenAI a lot of money. Apple is a hardware vendor, so commoditization of the software while finding more market segments is definitely something that'd benefit them. OTOH, if they let OpenAI become the unrivaled leader of AI that end up being the next Google, they end up losing on a topic they wanted to lead for long time…

"A lot of money" is a lot less money per user than to buy 64GB RAM to run an inferior model locally + energy and opportunity costs. The OpenAI APIs are super cheap for a single user needs. I expect them to be at least close to breaking even with their APIs pricing.

> "A lot of money" is a lot less money per user than to buy 64GB RAM

if OpenAI isn't able to get couple hundred bucks over the typical lifetime of a computer it means the added value they provide is very low (like several times less than Spotify or Netflix for instance), meaning they'll never be “the next Google”.

And if they are it means it make sense to buy it once instead of paying several times the price through subscription.

> The OpenAI APIs are super cheap for a single user needs. I expect them to be at least close to breaking even with their APIs pricing.

“Close to breaking even” means the price you pay is VC-subsidized, the expected gross margin for such kind of tech company is more than 50%. Expect to pay a lot more if/when the market is captive. And this will scale linearly with your use of the technology.

> energy and opportunity costs

What opportunity cost?

Re: Anthropic’s $5B, 4-year plan to take on OpenAI

#487

Post by anentropic »

gleenn wrote: post #435
rnk wrote: post #433

Earlier quoted context omitted.

But they haven't exposed them to use. They are missing a tremendous opportunity. They have that unique unified memory model on the m1/m2 arms so they have something no other consumer devices have. If they exposed their neural chips they'd solidify their lead. They could sell a lot more hardware.

They are though. Apple released a library to use Apple Silicon for training via PyTorch recently, and has libraries to leverage the NE in CoreML.

> Apple Silicon for training via PyTorch recently

This is just allowing PyTorch to make use of the Apple GPU, assuming the models you want to train aren't written with hard-coded CUDA calls (I've seen many that are like that, since for a long time that was the only game in town)

PyTorch can't use the Neural Engine at all currently

AFAIK Neural Engine is only usable for inference, and only via CoreML (coremltools in Python)

Re: Anthropic’s $5B, 4-year plan to take on OpenAI

#488

Post by wokwokwok »

seydor wrote: post #192
wokwokwok wrote: post #165

Earlier quoted context omitted.

It’s astonishing to me that people seem to believe the llama models are “just as good” as the large models these companies are building, and most people are only using the 7B model, because that’s all their hardware can support. …I mean, “not-bad-at-all” depends on your context. For doing mean real work (ie. not porn or spam) these tiny models suck. Yup, even the refined ones with the “good training data”. They’re to…

gpt4xalpaca is 13B

7? 13? Who cares? It’s an order of magnitude smaller than the GPT models. It’s a toy.

Re: Anthropic’s $5B, 4-year plan to take on OpenAI

#489

Post by rnk »

rasz wrote: post #478
rnk wrote: post #448

Earlier quoted context omitted.

With the integrated ram and cpu and gpu on apple silicon, however it's done it yields perf results. I do think that probably has higher cost than separately produced ram. And even separate from that, because they have that unified memory model unlike every other consumer device they can charge for it. So 64, 96 or 128 gb?

Its not done for perf results, Xbox doesnt have ram on package and somehow does 560 GB/s

The perf results I was referring to was the ability to run an llm locally (like llama.cpp) that uses a giant amount of ram in the gpu, like 40gig. Without this uniform memory model, you end up paging endlessly, so it's actually much faster for this application in this scenario. Unlike on a pc with a graphics card, you can use your entire ram for gpu. This isn't possible on the xbox because it doesn't have uniform memory as far as I know. So having incredible throughput still won't match not having to page.

Edit - I found an example from h.n. user anentropic, pointing at https://github.com/remixer-dec/llama-mps . "The goal of this fork is to use GPU acceleration on Apple M1/M2 devices.... After the model is loaded, inference for max_gen_len=20 takes about 3 seconds on a 24-core M1 Max vs 12+ minutes on a CPU (running on a single core). "

Re: Anthropic’s $5B, 4-year plan to take on OpenAI

#490

Post by rnk »

anentropic wrote: post #481
rnk wrote: post #446

Earlier quoted context omitted.

It sounds like we are in a similar position. I had no desire to get a 64gb laptop from apple until all the interesting things from running llama locally came out. I wasn't even aware of the specific benefit of that uniform memory model on the mac. Now I'm looking at do I want to do 64, 96 or 128gb. For an insane amount of money, 5k for that top end one.

The unified memory ought to be great for running LLaMA on the GPU on these Macbooks (since it can't run on the Neural Engine currently) The point of llama.cpp is most people don't have a GPU with enough RAM, Apple unified memory ought to solve that Some people have it working apparently: https://github.com/remixer-dec/llama-mps

Thank you, that's exactly what I was looking for, specific info on perf.
Post reply on HN