Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hyperspace

hypercritical.co

Post reply on HN
Print view View original

441–450 of 503 posts

Re: Hyperspace

#441

Post by kccqzy »

rzzzt wrote: post #217
borland wrote: post #158

Earlier quoted context omitted.

I don't know exactly what Siracusa is doing here, but I can take an educated guess: For each candidate file, you need some "key" that you can use to check if another candidate file is the same. There can be millions of files so the key needs to be small and quick to generate, but at the same time we don't want any false positives. The obvious answer today is a SHA256 hash of the file's contents; It's very fast, not t…

I experimented with a similar, "hardlink farm"-style approach for deduplicated, browseable snapshots. It resulted in a small bash script which did the following: - compute SHA256 hashes for each file on the source side - copy files which are not already known to a "canonical copies" folder on the destination (this step uses the hash itself as the file name, which makes it easy to check if I had a copy from the same f…

Hard links are not a suitable alternative here. When you deduplicate files, you typically want copy-on-write: if an app writes to one file, it should not change the other. Because of this, I would be extremely scared to use anything based on hard links.

In any case, a good design is to ask the kernel to do the dedupe step after user space has found duplicates. The kernel can double-check for you that they are really identical before doing the dedupe. This is available on Linux as the ioctl BTRFS_IOC_FILE_EXTENT_SAME.

Re: Hyperspace

#442

Post by kccqzy »

Sesse__ wrote: post #436
bob1029 wrote: post #48

> There is no way for Hyperspace to cooperate with all other applications and macOS itself to coordinate a “safe” time for those files to be replaced, nor is there a way for Hyperspace for forcibly take exclusive control of those files. This got me wondering why the filesystem itself doesn't run a similar kind of deduplication process in the background. Presumably, it is at a level of abstraction where it could safel…

This is the standard API for deduplication on Linux (used for btrfs and XFS); you ask the OS nicely to deduplicate a given set of ranges, and it responds by locking the ranges, verifying that they are indeed identical and only then deduplicates for you (you get a field back saying how many bytes were deduplicated from each range). So there's no way a userspace program can mess up your files.

Yup. This is the ioctl BTRFS_IOC_FILE_EXTENT_SAME.

Re: Hyperspace

#443

Post by kccqzy »

mattgreenrocks wrote: post #321

What jumped out to me: > Finally, at WWDC 2017, Apple announced Apple File System (APFS) for macOS (after secretly test-converting everyone’s iPhones to APFS and then reverting them back to HFS+ as part of an earlier iOS 10.x update in one of the most audacious technological gambits in history). How can you revert a FS change like that if it goes south? You'd certainly exercise the code well but also it seems like yo…

You lack imagination. This is not some crown jewel only achievable by Apple. In the open source world we have tools to convert ext file systems to btrfs and (1) you could revert back; (2) you could mount the original ext file system while using the btrfs file system.

Re: Hyperspace

#444

Post by kccqzy »

diggan wrote: post #435
Tagbert wrote: post #433

Earlier quoted context omitted.

Judging by this sub-thread, the process really is harder to explain that it appears on the surface. The basic idea is simple but the implementation requires deeper knowledge.

But why would you discuss the implementation to end-users who probably wouldn't even understand what "implementation" means? The discussions you see in the subthread is not a discussion that would appear on less-technical forums, and I wouldn't draw any broader conclusions based on HN conversations in general.

Because the implementation leaks to the user experience. The user at least needs to know whether after running the utility, the duplicate files will be gone, or whether changing one of the files will change the other.

Symbolic links, hard links, ref links are all part of the file system interface, not the implementation.

Re: Hyperspace

#446

Post by wubrr »

codemusings wrote: post #408
karparov wrote: post #395

Earlier quoted context omitted.

I'm a little surprised that folks here are investing so much time into this app. It's closed source, only available for a non-obious amount, time-limited or subscription-based and lots of details of how it works are missing. With a FOSS project this would have been expected, but with a ShareWare-style model? Idk..

John has reiterated multiple times on his podcast that he doesn't want to deal with thousands of support requests when making his apps open source and free. All his apps are personal itches he scratched and he sells them not to make a profit but to make the barrier of entry high enough to make user feedback manageable.

> he doesn't want to deal with thousands of support requests when making his apps open source and free.

Who says you have to deal with support requests if you open source something?

> All his apps are personal itches he scratched and he sells them not to make a profit but to make the barrier of entry high enough to make user feedback manageable.

That makes no sense

Re: Hyperspace

#447

Post by karparov »

codemusings wrote: post #408
karparov wrote: post #395

Earlier quoted context omitted.

I'm a little surprised that folks here are investing so much time into this app. It's closed source, only available for a non-obious amount, time-limited or subscription-based and lots of details of how it works are missing. With a FOSS project this would have been expected, but with a ShareWare-style model? Idk..

John has reiterated multiple times on his podcast that he doesn't want to deal with thousands of support requests when making his apps open source and free. All his apps are personal itches he scratched and he sells them not to make a profit but to make the barrier of entry high enough to make user feedback manageable.

Sorry, that's a BS reason. If you don't want that, just ignore all opened issues. That's it. If you are nice then you put a README that explains this in a sentence or two. If a community forms that wants to fix issues, for example critical ones that could lead to data loss then the community will deal with it, e.g. by forking.

Just keeping everything closed is really missing the point of how trust in infra that handles critical data is built nowadays.

Re: Hyperspace

#448

Post by p_ing »

amzin wrote: post #146
taneliv wrote: post #88

Earlier quoted context omitted.

On ZFS it consumes a lot of RAM. In part I think this is because ZFS does it on the block level, and has to keep track of a lot of blocks to compare against when a new one is written out. It might be easier on resources if implemented on the file level. Not sure if the implementation would be simpler or more complex. It might also be a little unintuitive that modifying one byte of a large file would result in a lot d…

Is there a FS that keeps only diffs in clone files? It would be neat

Not an FS but attempting to mimic NTFS, SharePoint does this within it's content database(s).

https://www.microsoft.com/en-us/download/details.aspx?id=397...

Re: Hyperspace

#449

Post by p_ing »

galaxyLogic wrote: post #302

On Windows there is "Dev Drive" which I believe does a similar "copy-on-write" -thing. If it works it's a no-brainer so why isn't it the default? https://learn.microsoft.com/en-us/windows/dev-drive/#dev-dri...

CoW is a function of ReFS, shipped with Server 2016. "DevDrive" is just a marketing term for a ReFS volume which has file system filters placed in async mode or optionally disabled altogether.

Re: Hyperspace

#450

Post by jamil7 »

ryandrake wrote: post #382
jamil7 wrote: post #379

Earlier quoted context omitted.

This isn’t true you can set the target multiple versions back. The main problem right now is a huge amount of churn in the language, APIs and multiple UI frameworks means everything is a moving target. SwiftUI has only really become useable in the last coupe of versions.

Every time Xcode updates, it seems a few more older macOS and iOS versions are removed from the list of "Minimum Deployment Versions". My current Xcode lets me target macOS back to 10.13 (High Sierra, 7 years old) and iOS 12.0 (6 years old). This seems... rather limiting. Like, I'd be leaving a lot of users out in the cold if I were actually releasing apps anymore. And this is Xcode 15.2, on a dev host Mac forever st…

You can keep using an older version of Xcode if you like. I mean, every other tool chain that I can think of does more or less the same thing. There are plenty of reasons to criticise Apple's developer tooling and relations, but I don't see this as being especially different to other platforms
Post reply on HN