Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

YouTube as Storage

github.com

Post reply on HN
Print view View original

151–160 of 163 posts

Re: YouTube as Storage

#151

Post by grishka »

thrdbndndn wrote: post #17

I don't get how it works. > Encoding: Files are chunked, encoded with fountain codes, and embedded into video frames Wouldn't YouTube just compress/re-encode your video and ruin your data (assuming you want bit-by-bit accurate recovery)? If you have some redundancy to counter this, wouldn't it be super inefficient? (Admittedly, I've never heard of "fountain codes", which is probably crucial to understanding how it wo…

He encodes bits as signs of DCT coefficients. I do feel like this is not as optimal as it could be. A better approach IMO would be to just ignore the AC coefficients altogether and instead encode several bits per block into the DC. Not using the chrominance also feels like a waste.

Re: YouTube as Storage

#152

Post by Nevermark »

lysp wrote: post #138
repeekad wrote: post #4

I once asked one of the original YouTube infra engineers “will you ever need to delete the long tail of videos no one watches” They said it didn’t matter, because the sheer volume of new data flowing in growing so fast made the old data just a drop in the bucket

I seem to recall reading that the HD variations may get removed leaving only 480p or lower for older unwatched videos. The original upload would likely still be stored, but not available for viewing.

That would be an odd thing to do. HD is low resolution already, and 480 is noticeably worse.

If they really wanted to compress, take out every other frame, and regenerate those frames with a neural decoder. But I don't know why that would be worth the effort for a stable number of low res files either.

Re: YouTube as Storage

#154

Post by bentcorner »

tempestn wrote: post #126
donkeyboy wrote: post #122

Earlier quoted context omitted.

This is amazing.

Would've been, once. These days I assume bentcorner asked their favourite LLM to generate a poem parodying Ozymandias about once-popular youtube videos.

Nope, I hand wrote this.

I actually considered using an LLM but in my experience they "warp" the content too much for anything like this. The effort required to get them to retain what I would consider something to my taste would take longer than just writing the poem myself. (Although tbf it's been awhile since I've asked a LLM to do parody work, so I could be wrong)

Re: YouTube as Storage

#155

Post by [deleted] »

1bpp wrote: post #141
tempestn wrote: post #126

Earlier quoted context omitted.

Would've been, once. These days I assume bentcorner asked their favourite LLM to generate a poem parodying Ozymandias about once-popular youtube videos.

It doesn't feel like it at all (I'd never expect an LLM to say 'pfp' like that, or 'lossly[sic] compressed', ASCII instead of fancy quotes) but who knows at this point. I may have gotten incredibly neurotic about online text since 2022.

[deleted]

Re: YouTube as Storage

#156

Post by water9 »

paulryanrogers wrote: post #131
MagicMoonlight wrote: post #96

Earlier quoted context omitted.

Now that they can harvest it all for AI training, that decision was the cheapest and greatest thing they ever did. Imagine trying to pay for all that content, nobody on earth would be able or willing to supply it.

PeerTube is a thing. I like to think without centralized players like YT, that P2P supported federation may have gained a better foothold.

There’s still time

Re: YouTube as Storage

#157

Post by brandonli28 »

grishka wrote: post #151
thrdbndndn wrote: post #17

I don't get how it works. > Encoding: Files are chunked, encoded with fountain codes, and embedded into video frames Wouldn't YouTube just compress/re-encode your video and ruin your data (assuming you want bit-by-bit accurate recovery)? If you have some redundancy to counter this, wouldn't it be super inefficient? (Admittedly, I've never heard of "fountain codes", which is probably crucial to understanding how it wo…

He encodes bits as signs of DCT coefficients. I do feel like this is not as optimal as it could be. A better approach IMO would be to just ignore the AC coefficients altogether and instead encode several bits per block into the DC. Not using the chrominance also feels like a waste.

This actually won't work against YouTube's compression. The DC coefficient is always quantized, rounded, scale, and any other things. That means that these bits are pretty much guaranteed to be destroyed immediately. If this is the case for every single block, then data is unrecoverable. Also, chrominance is not used on purpose, because chrominance is compressed much more aggressively compared to luminance.

Re: YouTube as Storage

#158

Post by compsciphd »

catlikesshrimp wrote: post #95
compsciphd wrote: post #33

Earlier quoted context omitted.

par2 is very limited. It only support 32k parts in total (or in reality that means in practice 16k parts of source and 16k parts of parity). Lets take 100GB of data (relatively large, but within realm of reason of what someone might want to protect), that means each part will be ~6MB in size. But you're thinking you also created 100GB of parity data (6MB*16384 parity parts) so you're well protected. You're wrong. Now…

you can split files so you can have more par blocks (100GB in 100 1GB parts 32k blocks per part)

yes, but it just moves the needle a bit, if you lose 1 of the 1GB parts in totality, you also can't recover, so it really depends in on your error model you are trying to protect from.

In practice a DVD like PI/PO model would be the best for many people (protect the 1GB parts like you said with 5-10% redundancy, and then protect all 100 1GB parts together with 5-10% redundancy. the PI will repair as much as it can at the 1GB size, while the PO will be able to repair 1GB blocks that can't be repaired otherwise.

It be interesting if Par2 or something like it could implement it natively without people having to hack together their own one off solutions.

Re: YouTube as Storage

#159

Post by grishka »

brandonli28 wrote: post #157
grishka wrote: post #151

Earlier quoted context omitted.

He encodes bits as signs of DCT coefficients. I do feel like this is not as optimal as it could be. A better approach IMO would be to just ignore the AC coefficients altogether and instead encode several bits per block into the DC. Not using the chrominance also feels like a waste.

This actually won't work against YouTube's compression. The DC coefficient is always quantized, rounded, scale, and any other things. That means that these bits are pretty much guaranteed to be destroyed immediately. If this is the case for every single block, then data is unrecoverable. Also, chrominance is not used on purpose, because chrominance is compressed much more aggressively compared to luminance.

I meant choosing multiple values, e.g. 4 to represent 2 bits. Say, 0.25, 0.5, 0.75, and 1. Then when decoding you would pick the closest valid value, so for example for 0.20 it would be 0.25. Not using AC coefficients would mean that theoretically you would get more bitrate for the DC ones.

Re: YouTube as Storage

#160

Post by tempestn »

bentcorner wrote: post #154
tempestn wrote: post #126

Earlier quoted context omitted.

Would've been, once. These days I assume bentcorner asked their favourite LLM to generate a poem parodying Ozymandias about once-popular youtube videos.

Nope, I hand wrote this. I actually considered using an LLM but in my experience they "warp" the content too much for anything like this. The effort required to get them to retain what I would consider something to my taste would take longer than just writing the poem myself. (Although tbf it's been awhile since I've asked a LLM to do parody work, so I could be wrong)

Ah, well, kudos then!
Post reply on HN