Live data from Hacker News

Keep Your Stuff, for Life

perkeep.org

101–110 of 119 posts

Re: Keep Your Stuff, for Life

#101
post #78
post #22

It sounds like this preserves your data from lots of different services. This is something I need! But I couldn't figure out what it actually supports. Suggestion: the very first paragraph should describe the specific inputs it can handle.

For the most part, it's just an object storage (think Amazon S3). Content addressable (think Git): you put an object (file bytes) in, and you can get it out by its hash, that's it. There are some bits (permanodes and claims) for adding metadata to objects (filename, timestamp, geo location and other attributes, I think even arbitrary jsons) and for authentication/sharing. A few really cool bits around modularity: blo…

Thank you so much for a description of what it actually does which the website seems to struggle so much to convey.

One thing that I'm still trying to figure out is, if you do happen to know: how does it handle data deduplication (if at all)? How about redundancy and backups? I've been glancing over the docs and I do see mention of replication to another Perkeep instance but that's not quite what I'm looking for.

Re: Keep Your Stuff, for Life

#102

Last released version 0.10 is from May 2018. Is this project still alive? Last commit is from March 11 2020 so maybe they are just "slow" at releasing.

Development paused when Brad had a kid in 2018.

It probably needs someone new to adopt it.

Re: Keep Your Stuff, for Life

#103

Ah, nostalgia :). It seems like with the recent wave of news about social media migrations (reddit, facebook, twitter, twitch, tiktok), people are hopefully starting to get more and more warmed up to the idea of protocolization of their social data. But most of the projects doing it are still just too immature. Solid, Perkeep, Blockstack, etc. just seem like vaporware. Seems like the only serious projects in use are…

How can you call a pile of open source code with no marketing "vaporware"?

Re: Keep Your Stuff, for Life

#106

Earlier quoted context omitted.

Re: 4Tb drives, I do the dollars-per-mb calculation before buying hard drives. The most recent time I included the enclosure cost and found that it was actually cheaper to go huge. Granted the enclosure was a Synology, but buying 16TB drives is the closest I’ve been so ‘solving’ storage in a long time. Formatting them and adding them to the array was brutal, and they are noisy, but it has been worth it.

Buying enough 16tb drives for an efficient raid array is an expensive way to save money. Something that's easy to overlook with larger drives is that their rebuild times are worse. "Shucking" drives throws the economics way off even if it means having to do some hacks and losing warranty... Usually the drives that come in enclosures are smaller.

> Buying enough 16tb drives for an efficient raid array is an expensive way to save money.

A lot of ways to be efficient with money start by having or using a lot of it:)

Re: Keep Your Stuff, for Life

#107
post #85

Earlier quoted context omitted.

> Does the first sentence not cover that fairly well? It does not. The first sentence on the site ("Perkeep [...] is a set of open source formats...") describes literally what the thing is, but not at all what it _does_. Not to slam on these cats, because marketing copy is _hard_. For project collaborators, or open-source dorks who live in this kind of world anyway, the sentence on the homepage is probably perfectly…

What it _does_ is: > modeling, storing, searching, sharing and synchronizing [...] files or objects, tweets or 5TB videos, and you can access it via a phone, browser or FUSE filesystem I mean maybe it could have been more explicit or they could have added more detail, but having this as the first sentence is WAY better than most of the 'professional' landing pages for startups that get posted here. 'Harmonizes synerg…

Them: “The project description isn’t clear to me.”

You: “Well I’m sorry it wasn’t clear to you but it was clear to me and better than these other things and here’s why it should have been clear to you.”

If someone tells you something is unclear to them, arguing about it doesn’t change the fact that it wasn’t clear to them.

Re: Keep Your Stuff, for Life

#108
post #78

Earlier quoted context omitted.

For the most part, it's just an object storage (think Amazon S3). Content addressable (think Git): you put an object (file bytes) in, and you can get it out by its hash, that's it. There are some bits (permanodes and claims) for adding metadata to objects (filename, timestamp, geo location and other attributes, I think even arbitrary jsons) and for authentication/sharing. A few really cool bits around modularity: blo…

Thank you so much for a description of what it actually does which the website seems to struggle so much to convey. One thing that I'm still trying to figure out is, if you do happen to know: how does it handle data deduplication (if at all)? How about redundancy and backups? I've been glancing over the docs and I do see mention of replication to another Perkeep instance but that's not quite what I'm looking for.

Deduplication is naturally handled by content-addressable property of this object store: the address of each object is its cryptographic hash, SHA224 in Perkeep. So if you try to put a duplicate copy, you'll find that the address at which you try to put it at is already occupied by the first copy. Perkeep assumes that you never delete anything (deletion is even simply not implemented, not even for garbage collection/compaction purposes), so if you see that one copy of an object was already put, you can discard any further puts as no-ops.

Then there is also some logic to chunk large objects into small pieces or "blobs". These small chunks are actually what the storage layer works with, rather than with the original unlimited-length blobs that the user uploaded. Chunking helps to space-efficiently store multiple versions of same large file (say, a large VM image) - the system only needs to store the set of unique chunks, which can be much smaller than N full but slightly-different copies of the same file. But I personally I find that it deteriorates its performance to the point of making it unusable for my use case of multi-TB multi-million-files storage of immutable media files. If chunking/snapshotting/versioning is important for your use case, I'd look more towards backup-flavored tools like restic, which share many of these storage ideas with Perkeep.

Redundancy and backup is handled by configuring storage layer ("blobserver") to do it. Perkeep's blobservers are composable - you can have leaf servers storing your blobs, say, directly in a local filesystem directory, remote server over sftp, or an S3 bucket, and you can compose them using special virtual blobserver implementations into bigger and more powerful systems. One such virtual blobserver is https://github.com/perkeep/perkeep/blob/master/pkg/blobserve... - which takes addresses of 2+ other blobservers and replicates your reads and writes to them.

Re: Keep Your Stuff, for Life

#109
post #108

Earlier quoted context omitted.

Thank you so much for a description of what it actually does which the website seems to struggle so much to convey. One thing that I'm still trying to figure out is, if you do happen to know: how does it handle data deduplication (if at all)? How about redundancy and backups? I've been glancing over the docs and I do see mention of replication to another Perkeep instance but that's not quite what I'm looking for.

Deduplication is naturally handled by content-addressable property of this object store: the address of each object is its cryptographic hash, SHA224 in Perkeep. So if you try to put a duplicate copy, you'll find that the address at which you try to put it at is already occupied by the first copy. Perkeep assumes that you never delete anything (deletion is even simply not implemented, not even for garbage collection/…

Backup as in backing up one perkeep instance to another is the "pk sync" command (https://github.com/perkeep/perkeep/blob/master/cmd/pk/sync.g...).

You give it the addresses of source and destination blobservers, it enumerates blobs in both, and copies the source blobs missing from destination into the destination server.

Re: Keep Your Stuff, for Life

#110

Ah, nostalgia :). It seems like with the recent wave of news about social media migrations (reddit, facebook, twitter, twitch, tiktok), people are hopefully starting to get more and more warmed up to the idea of protocolization of their social data. But most of the projects doing it are still just too immature. Solid, Perkeep, Blockstack, etc. just seem like vaporware. Seems like the only serious projects in use are…

I'm bullish on the Solid model, but they have a chicken-egg problem. Nobody is going to develop apps until a lot of people have their own pod, and nobody is going to use a pod until there are great apps for accessing their data.

The same thing happened with remoteStorage. There's initially a flurry of proof-of-concept apps, but no commercial quality killer apps to attract daily users.

AFAIK the only cloud storage protocol really used for app development is Google Drive. GDrive got successful by making a great cloud storage solution first, then once everyone had one app developers started making apps for it.

Post reply on HN