Cleaning up my 200GB iCloud with some JavaScript • Hacker News Board

Re: Cleaning up my 200GB iCloud with some JavaScript

#21

Post by sgloutnikov » Fri, Jan 05, 2024, 11:51 AM UTC

cnicolaou wrote: post #11

Surprisingly, I've received a similar storage notice from Apple before the holidays and I decided to download all photos/videos into my own media server instead of hosting it on iCloud. There is not an easy, straight-forward way to download your archive from iCloud. I am slowly getting there using multiple machines and devices. The issue with the recent changes with Apple is that they increase the prices for no good…

What about the Photos app on MacOS if you have one? I keep a local copy of my iCloud Photo Library and sync it through the MacOS Photos app.

Re: Cleaning up my 200GB iCloud with some JavaScript

#22

Post by Nextgrid » Fri, Jan 05, 2024, 11:51 AM UTC

o-o- wrote: post #15

Nextgrid wrote: post #9

Earlier quoted context omitted.

The opaqueness is definitely a problem. My iMessage database has a corrupted spot somewhere in the message history; if I accidentally make the Messages app read that spot (via searching or scrolling back through old conversations) it'll silently start maxing out the CPU and burning energy for no reason. There is no error message, nothing obviously wrong-looking, and as far as I know everything works.

Oh, so that's what's happening? You made me realise I have the same exact issue!

Well I can't for sure prove it's indeed DB corruption, but I don't have many other theories.

It's unlikely to be a "static" bug affecting everyone because it would've been caught in QA and/or user reports and fixed already (this is not a new issue, been happening for a year at least).

Therefore my hypothesis is that it's dependent on corrupt persistent data that a relative minority of users have which is not easy for Apple to detect/replicate.

Re: Cleaning up my 200GB iCloud with some JavaScript

#23

Post by captn3m0 » Fri, Jan 05, 2024, 11:52 AM UTC

The thing I detest the most is the weird tier that jumps from 200GB to 2TB. There’s no way to pay for incremental storage for 500GB or 1TB, which would be perfect for many families.

In today’s age where storage is a commodity, this ought to be priced per GB used.

Re: Cleaning up my 200GB iCloud with some JavaScript

#24

Post by Nextgrid » Fri, Jan 05, 2024, 11:53 AM UTC

farconics wrote: post #13

Nextgrid wrote: post #9

Earlier quoted context omitted.

The opaqueness is definitely a problem. My iMessage database has a corrupted spot somewhere in the message history; if I accidentally make the Messages app read that spot (via searching or scrolling back through old conversations) it'll silently start maxing out the CPU and burning energy for no reason. There is no error message, nothing obviously wrong-looking, and as far as I know everything works.

Sounds like a nightmare. Do you know what could have caused the database corruption?

Couple years of macOS/iOS reinstalls along with the typical shittiness of iCloud and the iMessage clients. I would be more surprised if it wasn't corrupt.

Re: Cleaning up my 200GB iCloud with some JavaScript

#25

Post by InsomniacL » Fri, Jan 05, 2024, 11:54 AM UTC

I'm pretty sure that iCloud stores the full version and downloads a lower quality optimised version to your phone.

If you open iPhone settings and browse to:

  > Apple ID > iCloud  > Photos

There is an option to 'Optimise iPhone Storage' which is enabled by default. This states:

  > If your phone is low on space, full resolution photos and videos are automatically replaced with smaller device sized versions. Full-Resolution versions can be downloaded from iCloud at any time.

This seems perfectly reasonable to me.

Re: Cleaning up my 200GB iCloud with some JavaScript

#26

Post by sliq » Fri, Jan 05, 2024, 11:55 AM UTC

Can confirm, have some ideas: Apple uses different file formats "internally" and also different resolutions, maybe saving them all in parallel as "one" video, which might cause this issue. So the total space per image is actually the sum of original .mov, optimized .mp4 and thumbnail image.

Also, (automated) version history might have caused this! Maybe (automatic) image optimiztion saved old and new versions of everything.

Re: Cleaning up my 200GB iCloud with some JavaScript

#27

Post by darkstar_16 » Fri, Jan 05, 2024, 11:56 AM UTC

wouldbecouldbe wrote: post #16

It would make sense if they save the video in a few formats for being able to load them fast in the Apple apps, most video hosts do the same.

But that shouldn't count in the user's storage quota.

Re: Cleaning up my 200GB iCloud with some JavaScript

#28

Post by AnonC » Fri, Jan 05, 2024, 12:02 PM UTC

saagarjha wrote: post #6

I kind of hate opaque cloud solutions like this, because their storage usage is inscrutable and their interfaces are basically never optimized to help you clear things out. Whether that’s intentional and malicious or just not a priority, I don’t really care, I just want to be in control of the stuff I’m using. I have a very similar issue with iMessage in iCloud, and unlike photos, there’s no web interface. You have t…

Search on the messages app is broken for me on iOS and macOS. It really can’t find things that are there. So I have to scroll and look around. The sizes reported are also inaccurate. While iCloud will report some large number, when I dig into the largest messages section it doesn’t show anything close to that size (with just a few items in that list).

Since the macOS Messages app is some Catalyst abomination, it won’t even support expected keyboard shortcuts for navigation, selection and deletion. Having to do everything using the trackpad is a slow process.

It’s frustrating that messages is so bad.

Re: Cleaning up my 200GB iCloud with some JavaScript

#29

Post by Nextgrid » Fri, Jan 05, 2024, 12:06 PM UTC

mft_ wrote: post #18

Okay, so it could be a bug, but it is possible that iCloud is secretly storing more than one version of the file in some cases? (After all, Apple does similar things with other media files.) The example given at the end is interesting: > So iCloud says the video is 128MB, I download it and the video is actually 48MB, and my free storage increases by ~170MB when I deleted it. Interesting! This suggests that iCloud isn…

That's probably indeed the case, but it doesn't excuse it.

When you buy a hard drive or USB stick, you get a certain amount of GBs to use as you please. If you put a 1GB file on it your free space decreases by 1GB (yes this is filesystem dependent and you might lose a few KBs for metadata, but the choice of filesystem is up to you and not mandated by the storage decide). It doesn't matter that the NAND controller probably used a few megabytes of the overprovisioned area to store its block mapping tables, or maybe even duplicated your data for its convenience - you were never charged for that overprovisioned area.

Here, you are sold a storage device (that you access over HTTP instead of SATA/PCIe), but when you write a 1GB file, they duplicate/convert/etc it for their convenience yet still charge you to store those duplicates you haven't asked for. That's new and unexpected.

Re: Cleaning up my 200GB iCloud with some JavaScript

#30

Post by Nextgrid » Fri, Jan 05, 2024, 12:08 PM UTC

InsomniacL wrote: post #25

I'm pretty sure that iCloud stores the full version and downloads a lower quality optimised version to your phone. If you open iPhone settings and browse to: > Apple ID > iCloud > Photos There is an option to 'Optimise iPhone Storage' which is enabled by default. This states: > If your phone is low on space, full resolution photos and videos are automatically replaced with smaller device sized versions. Full-Resoluti…

Downloads should still return the original file though, so that can't be used to explain the storage size discrepancy.