241–250 of 396 posts
Earlier quoted context omitted.
> I paid around $4000 for 4090m laptop That's how much many developers currently spend on tokens - every day. Whatever "Apple Tax" applies to a device that can run a capable model offline will amortise itself in a blink.
In what sustainable world outside of Bay Area jobs do devs spend 120k on tokens monthly?
Earlier quoted context omitted.
Consider also that right now LLMs run slowly enough you can watch them think. I've seen a demo of an LLM running at an absurdly high speed and it reminds me of when I moved from a 2400 baud modem to a 14.4 - BBS screens that I could watch draw were all of a sudden nigh-interactive. Faster-than-realtime video generation is also coming, and will also continue to require huge hardware for a long while yet. I love local…
If anyone wishes to see the future. A fast LLM is quite eye-opening. I think chatjimmy uses Talaas' chips where models are hardcoded into the silicon. https://chatjimmy.ai/
Earlier quoted context omitted.
I recently bought a Mac mini M4 16 GB - mostly to run Immich. I assumed I needed a Linux box. After a lot of researched I was quite surprised that the mac was the cheapest option. So not always an Apple tax.
> After a lot of researched I was quite surprised that the mac was the cheapest option For Immich, the cheapest option will either be a NAS or a used laptop depending on the amount of data you need, I wouldn't buy a mac for that.
(I just run the defaults on my CPU, works for me)
How about we release M5 Ultra first?
Apple is actually interesting. They are one of the few companies with a chip / PC play with real power AND basically no play I'm the hyperscalar market. That means they're actually incentivized at least short term, to benefit PCs becoming strong enough to do local LLMs. Which makes this play make even more sense. Though, I've been saying for a while that the local AI inflectiom point is the death knell for these fron…
Tangential: About 8 years ago ex-Apple chip engineers left to design server-grade chips, this was Nuvia, and they got sued by Apple to the point that they had to get acquired by Qualcomm.
So maybe they were assholes.
Earlier quoted context omitted.
Wouldn't this help Intel compete? If they have Apple's designs months prior to launch, rather than after launch.
Ripping off designs from your own fab customers is a pretty sure way to crater your fab business, and get sued into the ground at the same time.
The real advantage is knowing exactly what Apple is launching months or years in advance, because that can inform strategic planning.
Earlier quoted context omitted.
> After a lot of researched I was quite surprised that the mac was the cheapest option For Immich, the cheapest option will either be a NAS or a used laptop depending on the amount of data you need, I wouldn't buy a mac for that.
Maybe he wants really fast or large AI models inside immich? (I just run the defaults on my CPU, works for me)
I think the decision comes primarily on how much data you would like to store for Immich, if you want to go cheaply, a 100 bucks used laptop will do the job, if you have too much data, a NAS will be more suitable (and you are certainly not going to get a mac where you can plug multiple internal hard drives for the price of a NAS)
Earlier quoted context omitted.
In what sustainable world outside of Bay Area jobs do devs spend 120k on tokens monthly?
Nobody said anything about sustainable, or outside of the Bay Area really
1. NVidia aggressively segments the market on VRAM and will continue to do so. A 5090 with 32GB of RAM, ~21k CUDA cores and 1800GB/s of memory bandwidth is $3-4k. An RTX 6000 Pro with 96GB of RAM, ~24k CUDA cores and 1800GB/s memory bandwidth is ~$11k;
2. The 5090 won't be replaced until late 2028 or even 2029. There has been no mid-cycle refresh (eg 4080 Super vs 4080) and likely won't be either at all or for at least a year. If there is in a year, it basically confirms that the 6000 series won't be until 2028/2029. Also, the x090 never got a mid-cycle refresh so the current consumer high-end is staying that way for years;
3. The 6090 whenever it comes will still have 32GB of VRAM unless the memory market drastically changes;
4. Many have anticipated an M5 Max/Ultra refresh of the Mac Studio line in Q3. Given that Apple chose to hike the prices on Studios rather than discontinue them, I now think this isn't going to happen. We may not see a Studio refresh for up to 2 years. Apple has done this before with the Mac Pro;
5. M7 Max/Ultra will probably go to a memory bandwidth of 1.2-1.8TB/s vs the current tops of M3 Ultra, M4 Max and M5 Max of 600-900GB/s. This simply needs to go up to boost inference speed;
6. You'll also see the number of GPU cores go up. All of this will add up to an M7 Max being 50-80%+ of the performance of a 5090. That's huge given the shared memory architecture;
7. We may see the return of Apple using its massive cash pile for vendor-financing of an exclusive memory supply. This was one of Tim Apple's [sic] big innovations.