Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hyperspace

hypercritical.co

Post reply on HN
Print view View original

471–480 of 503 posts

Re: Hyperspace

#471

Post by mattgreenrocks »

kccqzy wrote: post #443
mattgreenrocks wrote: post #321

What jumped out to me: > Finally, at WWDC 2017, Apple announced Apple File System (APFS) for macOS (after secretly test-converting everyone’s iPhones to APFS and then reverting them back to HFS+ as part of an earlier iOS 10.x update in one of the most audacious technological gambits in history). How can you revert a FS change like that if it goes south? You'd certainly exercise the code well but also it seems like yo…

You lack imagination. This is not some crown jewel only achievable by Apple. In the open source world we have tools to convert ext file systems to btrfs and (1) you could revert back; (2) you could mount the original ext file system while using the btrfs file system.

Cool!

Re: Hyperspace

#472

Post by crowselect »

karparov wrote: post #395
jonhohle wrote: post #381

Earlier quoted context omitted.

Replying to myself now that I've had a chance to try the scan, but not the deduplication. I work with disc images, program binaries, intermediate representations in a workspace that's 7.6G. A few notes: * By default it doesn't scan everything. It ignores all files but those in an allow list. The way the allow list is structured, it seems like Hyperspace needs to understand the content of a file. As an end user, I hav…

I'm a little surprised that folks here are investing so much time into this app. It's closed source, only available for a non-obious amount, time-limited or subscription-based and lots of details of how it works are missing. With a FOSS project this would have been expected, but with a ShareWare-style model? Idk..

Non obvious amount? Scroll to the bottom of the app store page, all the in app purchase prices are listed there.

App store prices are localized. If the blog post said it costs “$10” or whatever, that doesn’t mean anything to millions of potential customers who live where they don’t use $, and is confusing for millions more that do use $ but don’t know if the price is in their local $ or USD

Re: Hyperspace

#473

Post by alwillis »

david_allison wrote: post #136

> Hyperspace can’t be installed on “Macintosh HD” because macOS version 15 or later is required. macOS 15 was released in September 2024, this feels far too soon to deprecate older versions.

He’s likely using APIs that only exist in macOS 15.

Re: Hyperspace

#474

Post by alwillis »

MadnessASAP wrote: post #247
dewey wrote: post #232

Earlier quoted context omitted.

It's not that nice to call someone's work they spent months on "trivial" without knowing anything about the internals and what they ran into.

I don't think they meant it in a disparaging way, except maybe against Apple. Moreso that typically filesystems that can support deduplication include a deduplication tool in it's standard suite of FS tools. I too find it odd that Apple does not do this.

By now, you know it’s not de-duping anything-nothing is deleted.

Of course the Copy-on-Write clone functionality has been available since APFS became the default file system in 2019.

Re: Hyperspace

#475

Post by EasyMark »

ryandrake wrote: post #157
david_allison wrote: post #136

> Hyperspace can’t be installed on “Macintosh HD” because macOS version 15 or later is required. macOS 15 was released in September 2024, this feels far too soon to deprecate older versions.

Came here to post the same thing. Would love to try the application, but I guess not if the developer is deliberately excluding my device (which cannot run the bleeding edge OS).

It's a new app, seems logical they might limit the audience at the start, with plans to expand in the future.

Re: Hyperspace

#476

Post by jonhohle »

mort96 wrote: post #399
jonhohle wrote: post #364

Earlier quoted context omitted.

The key is “unmodified” and how APFS knows or doesn’t know whether they are modified. How many apps write on block boundaries or even mutate just in disk data that has changed vs overwriting or replacing atomically? For most applications there is no benefit and a significant risk of corruption. So APFS supports it, but there is no way to control what an app is going to do, and after it’s done it, no way to know what…

For apps which write a new file and replace atomically, the CoW mechanism doesn't come into play at all. The new file is a new file. I don't understand what makes you think there's a significant risk of corruption. Are you talking about the risk of something modifying a file while the dedupe is happening? Or do you think there's risk associated with just having deduplicated files on disk?

The vast majority of apps using structured data and not block oriented data formats. A major exception is databases, but common file formats that most people work with - images, text, etc. often aren't best mutated directly on disk, but rewritten either to the same file or a new file. Without some transactional capability, mutating a file directly on disk can corrupt a file if the writer fails in the middle of the write. More than a few text editors use this as their method of saving to ensure that there is never an inconsistent state of that file on disk.

Re: Hyperspace

#477

Post by jonhohle »

pornel wrote: post #454
jonhohle wrote: post #351

Earlier quoted context omitted.

To make dedup[0] fast, I use a tree with device id, size, first byte, last byte, and finally SHA-256. Each of those is only used if there is a collision to avoid as many reads as possible. dedup doesn’t do a full file compare, because if you’ve found a file with the same size, first and last bytes, and SHA-256 you’ve also probably won the lottery several times over and can afford data recovery. This is the default fo…

Reading just the first byte is probably wasting a read of the whole block. Hashing the whole file after that is wasteful. You need to read (and hash) only as much as needed to demonstrate uniqueness of the file in the set. The tree concept can be extended to every byte in the file: https://github.com/kornelski/dupe-krill?tab=readme-ov-file#n...

Yeah, there is definitely some merit to more efficient hashing. Trees with a lot of duplicates require a lot of hashing, but hashing the entire file would be required regardless of whether partial hashes or done or not.

I have one data set where `dedup` was 40% faster than `dupe-krill` and another where `dupe-drill` was 45% faster than `dedup`.

`dupe-krill` uses blake3, which last I checked, was not hardware accelerated on M series processors. What's interesting is that because of hardware acceleration, `dedup` is mostly CPU-idle, waiting on the hash calculation, while `dupe-krill` is maxing out 3 cores.

Thanks for the link!

Re: Hyperspace

#478

Post by jonhohle »

bmn__ wrote: post #428
jonhohle wrote: post #345

Earlier quoted context omitted.

If you don’t mind CLI tools, You can try dedup - https://github.com/ttkb-oss/dedup . Use the —-dry-run option to get a list of files that would be merged without modifying anything and how much space would be saved.

On good file systems (see https://news.ycombinator.com/item?id=43174685 >) also identical chunks of files can be merged, resulting in more savings than with just whole files. As of now, dedup cannot help with this, but duperemove or jdupes do.

Yeah, not having an extent API or even an API that can clone to a file descriptor is unfortunate.

That said, keeping track of blocks and extents for deduplication would be a much more expensive problem to solve.

Re: Hyperspace

#479

Post by least »

karparov wrote: post #466
notatallshaw wrote: post #457

Earlier quoted context omitted.

As I understand it, from listening to the podcast, a better summary is that if it becomes popular, he wants it to be worthwhile for him to keep working on. Apps like this can easily bit rot, and more users does often mean more work e.g. answering or filtering emails, finding more edge cases, etc. From his perspective that means having a income to dedicate time to this. I don't think he's interested in being an "infra…

I still maintain that if that's the case then something is wrong. More users reporting bugs for relevant edge cases is not a nuisance, it's the crowdsourcing of testing and each such reported issue is gold because then he can fix it before he as a user of his own software runs into it. Assuming he actually uses the software. (I also do maintain a bunch of packages and I do use them daily.) Making software proprietary…

> why should I spend the time to for free report sth to somebody who is making money off my testing and doesn't even bother to be transparent about how things work exactly (i.e. the source)?

You don’t have to. No one is saying you are compelled to report bugs in software you paid for. Most people don’t. The benefit to you as a customer is it can help get the bug fixed. That is clearly a mutual benefit.

> If you really care about the quality of your work then maximizing the eye ball count and incentivise high qualith issue reporting.

I think you’re vastly overestimating the value in the “higher quality” bug reports you’re getting from free users. You might get some higher quality reports but you’ll mostly get a lot more noise.

Re: Hyperspace

#480

Post by [deleted] »

cmckn wrote: post #434
0x0203 wrote: post #413

Earlier quoted context omitted.

Absolutely no judgement for however people want to licence and distribute their software, but I've seen the support burden used as justification for closed source/selling software quite a bit recently, and wonder how often people might be conflating open source with open development. There's no reason an open source project has to accept bug reports or pull requests from anyone. See SQLite or many of the tools from F…

Didn’t SQLite developer(s) famously receive a flood of phone calls because McAfee antivirus used it in a way that was visible (and “suspicious”) to its users? One does not simply “not accept bug reports”. https://github.com/sqlite/sqlite/blob/e8346d0a889c89ec8a78e6...

[deleted]
Post reply on HN