How We Made IPFS Content Publishing 10x Faster
1–10 of 69 posts
Re: How We Made IPFS Content Publishing 10x Faster
#2Is it also possible to speed up lookup? I never used IPFS much as it took several minutes to find a cid.
Re: How We Made IPFS Content Publishing 10x Faster
#3> Return control back to the user after most (not all) of the PUT RPCs have succeeded and continue with the remaining ones in the background.
Making things faster by doing less (and not the same) been speeding up computing since forever! Can't help but feel like it's slightly misleading to call the providing ("publishing") faster when it's not actually doing the same, it's just that most parts turned async instead of waiting for confirmation.
Wouldn't this lead to the problem where the user things everything been provided properly, but once others try to find it, the records haven't yet been published? As far as I understand, it'd still take mostly the same amount of time until the entire CID (not just some of them) are available to others, the only thing that got "faster" is the end-user UX of the one providing?
Re: How We Made IPFS Content Publishing 10x Faster
#4Is it also possible to speed up lookup? I never used IPFS much as it took several minutes to find a cid.
Actually, lookup is super fast - CID lookup is consistently [0] https://probelab.io/ipfs/dht/#chart-ipfs-dht-lookup-performa...
Re: How We Made IPFS Content Publishing 10x Faster
#5Is it also possible to speed up lookup? I never used IPFS much as it took several minutes to find a cid.
When’s the last time you tried? It must be much faster now. Check: https://probelab.io/ipfs/dht/#chart-ipfs-dht-lookup-performa...
Re: How We Made IPFS Content Publishing 10x Faster
#6Are the defaults still leaking your whole internal and external IP allocations to the dHT still?
Its security posture was absolutely fucking gross the last time I reviewed it.
And of course, there's a shitcoin bolted on as well. Last thing I want to do is feed into FileCoin. Of course, everything new these days has some financial interaction crap bolted on to entice speculators and ilk.
Re: How We Made IPFS Content Publishing 10x Faster
#7I'll add to the "is it still...?" questions.
Last I was told about it, there was no way to delete stuff from IPFS. Nothing enforceable, at least. Setting aside that public stuff is "impossible" to delete on the internet, there's something appealing to me about being able to shut off my server. Feels like that is less possible with IPFS hosted content.
Does anyone have some perspective for me about removing content?
Re: How We Made IPFS Content Publishing 10x Faster
#8> Return control back to the user after most (not all) of the PUT RPCs have succeeded and continue with the remaining ones in the background. Making things faster by doing less (and not the same) been speeding up computing since forever! Can't help but feel like it's slightly misleading to call the providing ("publishing") faster when it's not actually doing the same, it's just that most parts turned async instead of…
The "Early Return" sections describe it more, I don't think it's as bad as it sounds in that first bullet. They're returning after 15 out of 20 complete,and it sounds like even if only those 15 end up succeeding it'll still generally be fine. (Exactly how fine / is that violating some common expectations and will cause problems: I dunno. Not familiar enough with IPFS's internals)
That said:
>In practice, at least one of the 20 follow-up requests fails in the vast majority of operations, and a single unresponsive peer can stall the entire phase waiting for a timeout.
It continually surprises me how often systems lack a Fast Fallback-like strategy¹. Or at least sound like it. Just an absolute flood of apps and websites and systems that try to do something once and then never tried an alternate route until that finishes, something like a minute or two later... for a process that usually takes less than a second. It's maddening. By the time you're considering one to be "stalled" and delaying everything unnecessarily, you probably should've already started trying two or three alternate routes!
Re: How We Made IPFS Content Publishing 10x Faster
#9> Return control back to the user after most (not all) of the PUT RPCs have succeeded and continue with the remaining ones in the background. Making things faster by doing less (and not the same) been speeding up computing since forever! Can't help but feel like it's slightly misleading to call the providing ("publishing") faster when it's not actually doing the same, it's just that most parts turned async instead of…
As far as I understand, the producer is publishing to the 20 nearest nodes it finds, but the consumer is also searching the 20 nearest nodes it finds, and there is quite a big safety margin built into that number 20. Almost all consumers should still be able to find your object once it has only published to 10 or 15.
This is a probabilistic system anyway. Even if publication finishes to 20 nodes, why is that enough to return to the caller? Shouldn't it be 30, or 50, just in case?
I'd say it makes sense to return control once zero PUTs have been made and do the whole thing in the background, to avoid serializing operations that usually don't need to be serialized, such as publishing multiple objects.
Re: How We Made IPFS Content Publishing 10x Faster
#10Is it also possible to speed up lookup? I never used IPFS much as it took several minutes to find a cid.
When’s the last time you tried? It must be much faster now. Check: https://probelab.io/ipfs/dht/#chart-ipfs-dht-lookup-performa...
Do you have any examples of actual content to look up, rather than benchmark graphs?