Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 83 posts

Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

#71

Post by saulpw »

ufocia wrote: post #36
thin_carapace wrote: post #28

Earlier quoted context omitted.

by this definition, a random dataset could apparently present no patterns, while presenting non apparent patterns.

Sounds like presenting no patterns, apparently or otherwise, would be a pattern in itself.

That's like a teenage "i am very smart" thinking. I mean sure we can look at some string of random bits and say "that looks random" but you can't just generate any old string of random bits to replace it (which would be the only 'pattern' that could be leveraged for compression here). If it's encrypted it'll also appear random, and therefore not be compressible, but you have to encode every byte exactly or the message won't be decryptable.

Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

#72

Post by andai »

purple-leafy wrote: post #32
andai wrote: post #22

Earlier quoted context omitted.

I was working on a multiplayer game a while ago, and one of the iterations of the netcode was "thin client" where clients just sent input, server simulated the game, and it dumped world state onto the pipe at 60hz. I didn't ship that version but I estimated a $3000 bandwidth bill with that approach! I started looking into diffing the state, compression, etc... until I realized, wait a minute! My player movement is li…

$3000 bill wow!! Clever insight :) yes a specialised solution usually wins! Good effort Did you end up publishing your game?

My game has been in development hell for several years. I've worked out most of the single-node issues and now I'm working out how to scale it to multiple nodes. (It is embarrassingly parallel, but I am embarrassingly stupid!)

Meanwhile I'm learning how to ship a trivial single player game -- I went with Pong, and it turns out polishing Pong and getting it ready for mass consumption is a surprisingly labor-intensive endeavour (yes, even with the fabled miracle machine assistants!)

But, it should probably be online by the end of July. Like, for real this time :)

What are you working on?

Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

#73

Post by r-w »

anyg wrote: post #39
purple-leafy wrote: post #38

Dumb question: can you train a model to predict the next byte of ANOTHER MODEL So apply this same logic to compressing a bigger model within a smaller model I know this is absolutely regarded, but humour me please

Not dumb at all. It's a whole field of active research - Speculative Decoding. A recent paper goes one level deeper with Speculative Speculative Decoding - https://arxiv.org/abs/2603.03251

Is model distillation also related?

Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

#74

Post by echoangle »

gcr wrote: post #68
gnabgib wrote: post #25

Earlier quoted context omitted.

No.. they're not. Do you understand random (the apparent or actual lack of definite patterns or predictability[0]) or compression (reduces bits by identifying and eliminating statistical redundancy[1])? [0]: https://en.wikipedia.org/wiki/Randomness [1]: https://en.wikipedia.org/wiki/Data_compression

I could write a program to generate the first 100MB of pi in a couple kilobytes. That certainly counts as “data compression” but isn’t useful outside this particular problem instance.

Yeah, because the digits of pi aren’t random.

Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

#75

Post by purple-leafy »

andai wrote: post #72
purple-leafy wrote: post #32

Earlier quoted context omitted.

$3000 bill wow!! Clever insight :) yes a specialised solution usually wins! Good effort Did you end up publishing your game?

My game has been in development hell for several years. I've worked out most of the single-node issues and now I'm working out how to scale it to multiple nodes. (It is embarrassingly parallel, but I am embarrassingly stupid!) Meanwhile I'm learning how to ship a trivial single player game -- I went with Pong, and it turns out polishing Pong and getting it ready for mass consumption is a surprisingly labor-intensive…

I hope you can get around to shipping!

I’m working on a game engine/platform and my first game on the engine.

Lots of compression work so that I can fit the games into qr codes which I’m super stubborn about achieving and finally achieved for 30 minute+ games for N number of players

Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

#77

Post by spidy__ »

fennecbutt wrote: post #76

Given it's an overfitted transformer isn't is still terribly inaccurate/is liable to add random bytes here and there?

If I were only using a transformer that would have been true, but we use arithmetic coding alongside our transformer to fix those mistakes (layman terms). You can read about arithmetic coding, its a pretty cool topic.

Re: Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

#79

Post by freakynit »

Recently read about some upcoming GPU chip vendor that speeds up multiply-and-add (the core of [pretty much any neural network, including LLM's) oepration by transforming them to log scale where the multiplication changes to addition.

There issue was the size of such log tables that would be needed, and hence, they settled for some trade-off.

My question is: can the expensive log-domain addition / correction function be implemented as fixed hardware lookup tables or approximate units?

Update: found the vendor: https://www.tensordyne.ai/silicon-and-math

Post reply on HN