Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New improvements to IPFS Bitswap for faster container image distribution

blog.ipfs.io

Post reply on HN
Print view View original

41–50 of 61 posts

Re: New improvements to IPFS Bitswap for faster container image distribution

#41

Post by Grustaf »

bastawhiz wrote: post #35
Grustaf wrote: post #11

Earlier quoted context omitted.

”Netflix and IPFS began collaborating on ways to incorporate peer-to-peer services into Netflix’s developer tooling” Developer tooling is a pretty limited part of their traffic probably

The volume of traffic doesn't really matter that much. Developer productivity is just as important as the service you're actually selling when you're the size of Netflix. If your N thousand engineers are suddenly unable to work, or slowed down by X%, that's a huge problem. Large companies treat (or should treat) developer tooling issues as seriously as application outages. If Netflix is using IPFS for anything worth…

Of course the service itself is more important. If there are outages in the service, you will lose customers. If developers lose time at most your new features risk delays. If developers are less productive over time you lose a bit of money.

I say this as a developer for a FANG company.

It’s still an endorsement, but not nearly as strong as if the broadcasting was somehow relying on IPFS. As it is, this is probably just some engineering manager that made some non-crucial tool and put that on ipfs.

Re: New improvements to IPFS Bitswap for faster container image distribution

#42

Post by humblebee »

tln wrote: post #28

"the container runtime can be modified to retrieve layers identified by their CIDs" How do you do this? Exercise for the reader? :) For the case of distributing containers in a datacenter with P2P, theres also this work: https://github.com/uber/kraken

This might be related: https://github.com/docker/distribution/pull/2906

Re: New improvements to IPFS Bitswap for faster container image distribution

#43

Post by zapita »

jude- wrote: post #32
asdkhadsj wrote: post #23

Earlier quoted context omitted.

I would think because IPNS is IPFS.. and you asked how IPNS compares to BitTorrent. Maybe I misunderstand your question, but the reply seems totally on topic and a valid answer to your question.

IPNS is a system that runs on top of IPFS. You do not need IPNS to use IPFS.

That's technically correct, but in practice the term "IPFS" is commonly used to refer to both ipfs and its optional feature ipns.

Re: New improvements to IPFS Bitswap for faster container image distribution

#44

Post by 411111111111111 »

jungong wrote: post #5
tyingq wrote: post #2

"Web 2.0" Hadn't seen that gem for a while.

3.0, no?

pretty sure we're already at 4.0 with the Internet of Shit (IoT) and Mobile Internet, depending on the one using the buzzword

but thats besides the point of the parent. web 2.0 hasn't really been mentioned in ages.

Re: New improvements to IPFS Bitswap for faster container image distribution

#45

Post by rochaporto_ »

etaioinshrdlu wrote: post #39

One big optimization that could help in some cases for container platforms like Fargate is not downloading the entire image just to run the container. Instead read files (or even just blocks) from network storage on demand. This is basically how booting from a disk image works on most cloud platforms too.

That should be there soon : https://github.com/containerd/containerd/issues/3731

Re: New improvements to IPFS Bitswap for faster container image distribution

#46

Post by hinkley »

tylersmith wrote: post #27
hinkley wrote: post #22

> The node sends out a want for each CID to several peers in the session in parallel, because not all peers will have all blocks. If the node starts receiving a lot of duplicate blocks, it sends a want for each CID to fewer peers. If the node gets timeouts waiting for blocks, it sends a want for each CID to more peers. Trying to recall how the protocol works. Doesn’t this pattern of behavior mean that a lot of machin…

Files are not downloaded sequentially, they're chunked into blocks which are sent in parallel.

Each client downloads in random order, or all clients download in the same order?

Re: New improvements to IPFS Bitswap for faster container image distribution

#47

Post by viraptor »

hobofan wrote: post #29
hiccuphippo wrote: post #12

Earlier quoted context omitted.

You can't update a torrent, if the content changes you have to create a new one. IPNS helps with that. And you can't share pieces across different torrents, if some still have the old torrent, they share it separately from the new one even if the differences are minimal.

> And you can't share pieces across different torrents So make each layer an own torrent?

That's still not great. Imagine the only difference between two versions of the layer is that you updated a single jar in a 200MB app bundle. The effective difference could be a few tens of blocks, but you still need to redownload the whole thing.

If we can manage the assignment/padding to match ipfs fragments, that could result in a massive saving.

Re: New improvements to IPFS Bitswap for faster container image distribution

#48

Post by viraptor »

DaniFong wrote: post #9
js4ever wrote: post #7

I'm shocked to see IPFS used to make something Faster...

it's remarkable! a bootstrap technique to improve time coherence -- what an innovation!

I think that was a dig at ipfs's issues with real world usage where a lot of traffic until recently was wasted on metadata and every node used lots of bandwidth to gossip. Meanwhile the actual throughout on a non-tuned node was not great at all.

Re: New improvements to IPFS Bitswap for faster container image distribution

#49

Post by viraptor »

marshmellowtest wrote: post #16
nine_k wrote: post #15

Earlier quoted context omitted.

I thought that ipfs is about high availability, fault tolerance, including some resistance against addressed censorship. It never looked like an anonymizing tool to me; did anybody advertise it as such?

"resistance against addressed censorship" does not work at all when all your traffic is made public. People can be prosecuted or otherwise harassed for sharing contents on a P2P system. > It never looked like an anonymizing tool to me; did anybody advertise it as such? You are confusing "anonymizing" with "leaking a lot of information to the whole world". They constantly "forget" to tell people about the huge securit…

Ipfs helps you distribute content which may get taken down. It does not help you evade local police.

For the second scenario, you want another layer which maintains secrecy. (Like the tor transport https://ipfs.io/ipfs/QmYKQvBsbYrRhdaGvQXcEoSam7s5gKVYULfRgNP...)

Re: New improvements to IPFS Bitswap for faster container image distribution

#50

Post by StavrosK »

viraptor wrote: post #48
DaniFong wrote: post #9

Earlier quoted context omitted.

it's remarkable! a bootstrap technique to improve time coherence -- what an innovation!

I think that was a dig at ipfs's issues with real world usage where a lot of traffic until recently was wasted on metadata and every node used lots of bandwidth to gossip. Meanwhile the actual throughout on a non-tuned node was not great at all.

Has this changed now?
Post reply on HN