Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data on the cheapest MacBook

duckdb.org

Post reply on HN
Print view View original

311–320 of 321 posts

Re: Big data on the cheapest MacBook

#311

Post by CryptoBanker »

meatmanek wrote: post #266
api wrote: post #33

Earlier quoted context omitted.

Yeah, this is really about how ludicrously overpriced big cloud is. I’ve got a first gen M1 Max and it destroys all but the largest cloud instances (that cost its entire current market value per month!), at least in compute. It’s a laptop! A decent bare metal server in a rack will destroy any laptop. It’s staggering. Jaw dropping. Bandwidth is even worse, like 10000X markup. Yet cloud is how we do things. There’s a g…

> I’ve got a first gen M1 Max and it destroys all but the largest cloud instances (that cost its entire current market value per month!) You're either underestimating how big cloud instances can get or overestimating how much it costs to rent a cloud instance that would beat an M1 Max at any multi-core processing. According to Geekbench, the M1 Max macbook pro has a single-core performance of 2374 and multicore of 12…

> That c8i.4xlarge would cost you $246/mo at current spot pricing of $0.3425/hr, which is, what, 20% of the cost of that M1 Max MBP?

A 5 month ROI on a hardware investment would be excellent, so not sure what you're trying to say here?

Re: Big data on the cheapest MacBook

#312

Post by jorvi »

whatever1 wrote: post #297
jorvi wrote: post #229

Earlier quoted context omitted.

People usually forget 8GB isn't 8GB. Memory compression means you can store ~2x (lz4) to 3x (zstd) as much data in memory as ordinarily. And in the worst case, reading swap from disk (writes don't matter as they can be predicted) is so much faster with NVMe SSDs. The worst corner they cut is no keyboard backlighting. That saves them what, $1 BoM per MacBook Neo? Especially because now they have to put up an entire ne…

This is PR speak. No 8GB, compressed or whatever was not enough for MacOS when M1 was released. Even for simple outlook, web browser, excel type of workflows. After 3-4 hours of work, the window manager process itself is consuming gigabytes of memory. Not even considering any browser or electron apps. My M1 Mac mini was choking up so much that I had to trade it in. That was back in 2021. Today apps are even more bloa…

Yeah, you never owned a Mac and are just making stuff up. I have no urge to debate zealots. Bye.

Re: Big data on the cheapest MacBook

#314

Post by whatever1 »

jorvi wrote: post #312
whatever1 wrote: post #297

Earlier quoted context omitted.

This is PR speak. No 8GB, compressed or whatever was not enough for MacOS when M1 was released. Even for simple outlook, web browser, excel type of workflows. After 3-4 hours of work, the window manager process itself is consuming gigabytes of memory. Not even considering any browser or electron apps. My M1 Mac mini was choking up so much that I had to trade it in. That was back in 2021. Today apps are even more bloa…

Yeah, you never owned a Mac and are just making stuff up. I have no urge to debate zealots. Bye.

I am not the one claiming that apple does magic with memory. FYI right now I just opened monitor (M1 16GB) and sorted by memory:

1. WindowServer : 4.33GB 2. Safari pages (~10 open): ~ 5GB (monitor reports per tab) 3. Outlook: 1.45 GB 4. VS Code: 1.43 GB 5. Excel: 1.25 GB

Swap Used: 6.46GB.

Memory pressure is already orange and machine is slow.

Re: Big data on the cheapest MacBook

#315

Post by meatmanek »

CryptoBanker wrote: post #311
meatmanek wrote: post #266

Earlier quoted context omitted.

> I’ve got a first gen M1 Max and it destroys all but the largest cloud instances (that cost its entire current market value per month!) You're either underestimating how big cloud instances can get or overestimating how much it costs to rent a cloud instance that would beat an M1 Max at any multi-core processing. According to Geekbench, the M1 Max macbook pro has a single-core performance of 2374 and multicore of 12…

> That c8i.4xlarge would cost you $246/mo at current spot pricing of $0.3425/hr, which is, what, 20% of the cost of that M1 Max MBP? A 5 month ROI on a hardware investment would be excellent, so not sure what you're trying to say here?

5 months is a lot worse than 1 month, which is what the parent claimed.

Re: Big data on the cheapest MacBook

#316

Post by antonyh »

brudgers wrote: post #269
antonyh wrote: post #163

Earlier quoted context omitted.

I think the definition of big is smaller than that. Mine was "too big to fit on a maxed-out laptop", effectively >8TB. Our photo collection is bigger than that, it's not 'big data'. Or one could define it as too big to fit on a single SSD/HDD, maybe >30TB. Still within the reach of a hobbyist, but too large to process in memory and needs special tools to work with. It doesn't have to be petabyte scale to need 'big da…

“Your data is not big” comes from this thread… https://news.ycombinator.com/item?id=7192839 8TB is a couple hundred hours of 4k RAW video assets.

This is true, but 8TB is big data if it's text.

Re: Big data on the cheapest MacBook

#317

Post by astrange »

whatever1 wrote: post #314
jorvi wrote: post #312

Earlier quoted context omitted.

Yeah, you never owned a Mac and are just making stuff up. I have no urge to debate zealots. Bye.

I am not the one claiming that apple does magic with memory. FYI right now I just opened monitor (M1 16GB) and sorted by memory: 1. WindowServer : 4.33GB 2. Safari pages (~10 open): ~ 5GB (monitor reports per tab) 3. Outlook: 1.45 GB 4. VS Code: 1.43 GB 5. Excel: 1.25 GB Swap Used: 6.46GB. Memory pressure is already orange and machine is slow.

Run `IOAccelMemory` in the terminal to see what's causing that (dirty and wired columns). It's probably an app with a lot of windows.

Or for general memory `footprint` is good.

Re: Big data on the cheapest MacBook

#318

Post by whatever1 »

astrange wrote: post #317
whatever1 wrote: post #314

Earlier quoted context omitted.

I am not the one claiming that apple does magic with memory. FYI right now I just opened monitor (M1 16GB) and sorted by memory: 1. WindowServer : 4.33GB 2. Safari pages (~10 open): ~ 5GB (monitor reports per tab) 3. Outlook: 1.45 GB 4. VS Code: 1.43 GB 5. Excel: 1.25 GB Swap Used: 6.46GB. Memory pressure is already orange and machine is slow.

Run `IOAccelMemory` in the terminal to see what's causing that (dirty and wired columns). It's probably an app with a lot of windows. Or for general memory `footprint` is good.

Thank you so much for this! At least now I can see the worst offenders. How did you even find this tool? The internet has almost no records of it. Amazing.

Still does not explain why this balloons over time. Aka if I restart my Mac right now and reopen the same exactly apps with the same exactly windows the WindowServer will take 80% less memory.

Re: Big data on the cheapest MacBook

#319

Post by astrange »

whatever1 wrote: post #318
astrange wrote: post #317

Earlier quoted context omitted.

Run `IOAccelMemory` in the terminal to see what's causing that (dirty and wired columns). It's probably an app with a lot of windows. Or for general memory `footprint` is good.

Thank you so much for this! At least now I can see the worst offenders. How did you even find this tool? The internet has almost no records of it. Amazing. Still does not explain why this balloons over time. Aka if I restart my Mac right now and reopen the same exactly apps with the same exactly windows the WindowServer will take 80% less memory.

Well, now you get to file a bug with Feedback Assistant…

If you send me the number it will get looked at.

Re: Big data on the cheapest MacBook

#320

Post by tosh »

szarnyasg wrote: post #251
tosh wrote: post #36

For the TPC-DS results it would also have been nice to show how the macbook neo compares to the AWS instances. Or am I missing something?

Indeed, it would have been interesting but I really wanted to get the blog post out on the launch day of the MacBook Neo and did not have the bandwidth to run additional cloud experiments. I ran TPC-DS SF300 now on the c6a.4xlarge. It turns out that it's still quite limited by the EBS disk's IO: while 32 GB memory is much more than 8 GB, DuckDB needs to spill to disk a lot and this shows on the runtimes. Running all…

ty for the follow up!
Post reply on HN