Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nix × IPFS – Milestone 1

blog.ipfs.io

Post reply on HN
Print view View original

51–60 of 92 posts

Re: Nix × IPFS – Milestone 1

#51

Post by hinkley »

brabel wrote: post #49
Taek wrote: post #8

Earlier quoted context omitted.

We built an alternative to IPFS called Skynet that attempts to solve a couple of the major issues with IPFS. The biggest one being data durability and uptime. On Skynet, pinning doesn't mean hosting the file from your machine, it means paying a bunch of service providers to host the file for you. When you pin content to Skynet, you can turn off your computer 5 minutes later and the data will still be available global…

This looks interesting. Why is the payment method based on a crypto coin? How does using a blockchain help make Skynet and Sia work? I am wary of crypto coins as they tend to have wild swings in value over time as they are used primarily for speculation... can I be sure that if I upload something today worth $2/month, in 2 years I will still be paying this much or less?

A common theme in distributed filesystem conversations is the idea of "socializing" the costs of intermittent loads. If I 'pay in' a little more than my mean traffic capacity, then my surplus during low traffic cancels out some of my peak traffic. Peak shaving and trough filling.

If you are doing business in India, you get payed in rupees. If the workers are in India, you just pay them in rupees. If you have to exchange currencies you end up with several types of friction that just create headaches and potential losses. If you periodically cash out or inject cash it's easier to deal with than on every transaction.

Denominating file replication services in a "coin of the realm" just seems like the same sort of rationale.

One of the problems with capacity planning is that you get punished for being wrong in either direction. You bought too much hardware or not enough, too soon or too late. With an IPFS or a Skynet, putting your hardware online two months before you need the capacity at least affords you some opportunity to make use of the hardware while your Development or PR team figures out how to cross the finish line.

Re: Nix × IPFS – Milestone 1

#52

Post by 3np »

bityard wrote: post #41
akerro wrote: post #34

Earlier quoted context omitted.

I hear it in every discussion, yet I'm still waiting for any of my SD cards to fail. I've had the same rpi0 for 3 years as CCTV with motiond (motion detector). Recording all activity from my window, filling the card in 100% and removing all recorded data, every 2 days, no errors, no failure. I no longer need that CCTV so using the rpi, with the same SD card, as IPFS node.

IME, the people who complain about SD cards failing on raspberry pi bought cheap cards that were intended for bulk storage, not frequent writes or running an OS card. Regular SD cards are not SSDs. On my network of random Pis and other stuff, I use only high-endurance SD cards which can withstand lots of writes have durability much closer to SSDs.

Here’s from someone who only bought the supposedly highest recommended ones and had several failures before I switched to usb. Maybe roughly average 10-20% failure/ year uptime for something that has heavy log writing?

A thing people tend to forget is temperature. Keep them cool and they have higher chances of surviving longer.

Re: Nix × IPFS – Milestone 1

#53

Post by e12e »

vander_elst wrote: post #30

Does anyone know if there are authz features planned for ipfs? Like having acls based on certificates or something...

As far as I'm aware the "official" ipfs line is ipfs host and share data - if you want to protect it; encrypt it.

Re: Nix × IPFS – Milestone 1

#54

Post by e12e »

Avamander wrote: post #11
imhoguy wrote: post #9

Open source plus distributed p2p file sharing is the killer combo. I don't get why public stuff like NPM, DEB or Docker registries haven't switched to use it as primary way of distribution. P2P, such as IPFS and recently BitTorrent 2.0 with its hash tree per file, is the only free (as beer and speech) reliable way to host things online forever - at least as long there is the last of the veterans who keeps a copy and…

I don't get why machines running the same Linux distribution couldn't actually do the same. BT2 would indeed work very well for this.

Nixos/guix might be a good fit for rootfs on ipfs? Comes with a local cache, and you could overlay local changes (eg: /etc/passwd)?

Re: Nix × IPFS – Milestone 1

#55

Post by kordlessagain »

brabel wrote: post #49
Taek wrote: post #8

Earlier quoted context omitted.

We built an alternative to IPFS called Skynet that attempts to solve a couple of the major issues with IPFS. The biggest one being data durability and uptime. On Skynet, pinning doesn't mean hosting the file from your machine, it means paying a bunch of service providers to host the file for you. When you pin content to Skynet, you can turn off your computer 5 minutes later and the data will still be available global…

This looks interesting. Why is the payment method based on a crypto coin? How does using a blockchain help make Skynet and Sia work? I am wary of crypto coins as they tend to have wild swings in value over time as they are used primarily for speculation... can I be sure that if I upload something today worth $2/month, in 2 years I will still be paying this much or less?

> Why is the payment method based on a crypto coin?

A blockchain gives a way to make a payment and provide identity/ecrypt functions to keep the resource private while it is active.

Yes, one could create a system that attaches other authentication (user/pass or oauth), but then one has to create/connect a payment system which then uses that login information in conjunction with credit card information. To sell a product online taking credit cards requires an excess of 10 pieces of information that must be provided by the user.

In the case of compute resources, I may want to deal with 100s of providers to host my resources (blog, images, video, code) and I'd need to use an intermediary if I wanted to be efficient about it.

With something like Lightning payments, a system like this can provide resources without the need for a "signup" process or intermediary.

> can I be sure that if I upload something today worth $2/month, in 2 years I will still be paying this much or less?

What does the future value of a fiat currency have to do with the current rate of storage on something like AWS? Would those prices not go up if the currency was undergoing deflation? Would you not have paid less integer values before the deflation? Would you not pay more integer values after? Where does something like AWS provide cost protection, other than spot instances?

If you create a crypto contract and put the funds in escrow, then there is ZERO ways for the cost to go up over the life of the contract. Other than a bad actor scenario, which is why having multiple providers is the way to go.

Re: Nix × IPFS – Milestone 1

#56

Post by takeda »

na85 wrote: post #31
imhoguy wrote: post #9

Open source plus distributed p2p file sharing is the killer combo. I don't get why public stuff like NPM, DEB or Docker registries haven't switched to use it as primary way of distribution. P2P, such as IPFS and recently BitTorrent 2.0 with its hash tree per file, is the only free (as beer and speech) reliable way to host things online forever - at least as long there is the last of the veterans who keeps a copy and…

In a lot of the world, upload speeds are very limited and ISPs charge an arm and a leg if you exceed your upstream data caps. On top of that, Bit torrent has a tendency to overwhelm routers and make them slow to a crawl. Those are a lot of downsides to ask your users to accept just so you can cheap out on download servers.

Perhaps there are routers that don't handle well large amounts of data, but from my experience what typically is happening are two things:

1. the fairness in TCP is done per TCP connection, bittorrent opens many connections so they take over the whole bandwidth. You can limit number of connections or throttle the bandwidth used by BT (of course you will get slower speeds), maybe there are some other ways.

2. when you maximize your throughput ACKs might get dropped which slows everything down (I suspect that might be the issue you're talking about) again you could throttle or enable traffic shaping and give highest priority to ACKs. This worked well for me.

Re: Nix × IPFS – Milestone 1

#57

Post by Ericson2314 »

arianvanp wrote: post #3

Does IPFS actually solve the problem they set out here though? IPFS is a distributed CDN; but not very good for storing things persistently or reliably from my experience. At the moment; the nixos cache is stored in very durable and reliable S3 storage; with very high durability guarantees. Why is that not good enough? sure it's centralised. But IPFS doesn't offer distributed durability; it offers a CDN. It doesn't s…

Author here!

I absolutely agree it's hard to go head-to-head with s3 in the short term --- this is why I am most excited about sharing sources. Once the ecosystem is bootstrapped, it will make more sense to use IPFS for binaries too. (e.g. if you wanted to build some fancy multiple build farms and reputation system.)

> https://www.softwareheritage.org/

I do really want to work with then!

- IPFS as CDN means software heritage can be "seeder of last resort"

- Original authors uploading to CDN, using IPNS or similar for git tags/versions, should make it easier for software heritage to archive the code in the first place.

Re: Nix × IPFS – Milestone 1

#58

Post by takeda »

e12e wrote: post #54
Avamander wrote: post #11

Earlier quoted context omitted.

I don't get why machines running the same Linux distribution couldn't actually do the same. BT2 would indeed work very well for this.

Nixos/guix might be a good fit for rootfs on ipfs? Comes with a local cache, and you could overlay local changes (eg: /etc/passwd)?

Actually not exactly rootfs on ipfs, but with NixOS is fairly easy to make it wipe data on reboot, here's an example article[1]. The idea is to only keep state we care about and rebuild the system on every boot.

[1] https://grahamc.com/blog/erase-your-darlings

Re: Nix × IPFS – Milestone 1

#59

Post by akerro »

solarkraft wrote: post #26
imhoguy wrote: post #9

Open source plus distributed p2p file sharing is the killer combo. I don't get why public stuff like NPM, DEB or Docker registries haven't switched to use it as primary way of distribution. P2P, such as IPFS and recently BitTorrent 2.0 with its hash tree per file, is the only free (as beer and speech) reliable way to host things online forever - at least as long there is the last of the veterans who keeps a copy and…

> I don't get why public stuff like NPM, DEB or Docker registries haven't switched to use it as primary way of distribution 2 of them are run by commercial entities. I don't think those are all that interested in reducing dependency on them.

>2 of them are run by commercial entities. I don't think those are all that interested in reducing dependency on them.

Docker is also removing some old images and putting limits on existing images, it's too expensive for them to host.

Re: Nix × IPFS – Milestone 1

#60

Post by akerro »

na85 wrote: post #31
imhoguy wrote: post #9

Open source plus distributed p2p file sharing is the killer combo. I don't get why public stuff like NPM, DEB or Docker registries haven't switched to use it as primary way of distribution. P2P, such as IPFS and recently BitTorrent 2.0 with its hash tree per file, is the only free (as beer and speech) reliable way to host things online forever - at least as long there is the last of the veterans who keeps a copy and…

In a lot of the world, upload speeds are very limited and ISPs charge an arm and a leg if you exceed your upstream data caps. On top of that, Bit torrent has a tendency to overwhelm routers and make them slow to a crawl. Those are a lot of downsides to ask your users to accept just so you can cheap out on download servers.

> In a lot of the world, upload speeds are very limited and ISPs charge an arm and a leg if you exceed your upstream data caps.

Not all 1st world countries have data caps. I pay ~20$ for month for 37mbps (15 upload) with no data caps (it's literally the cheapest plan I could find, if there was cheaper available, I would take it).

Post reply on HN