Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

πFS

github.com

Post reply on HN
Print view View original

191–200 of 224 posts

Re: πFS

#191

Post by frotaur »

janalsncm wrote: post #121
jamwise wrote: post #31

Reminds me of when I tried to use the library of babel as a data compression tool. It led me down a fun rabbit hole and was my first introduction to information theory. The conclusion being that you basically need the same amount of data to represent the address of your data as the data itself, so it's not really effective at compression, just a fun thought experiment. The cool part of this in modern times is that LL…

The level of compression is pretty impressive when you think about it. I wrote a comment a while back which is still true (although bytes should be bits, so in that sense it’s still wrong): https://news.ycombinator.com/item?id=39559969 Back of the envelope calculation for storing valid 4-grams (sequences of four words) is around 10 billion x 14 bits per word = 17 gb for all 10 billion. There are LLMs 100x smaller whi…

If you combine the LLM probability distribution with arithmetic coding you can actually use them to compress text losslessly. When people reports 'bits per byte', it is actually the compression rate for text.

GPT-2 for instance achieves roughly 1 bit per byte, so it can be used to compress (english) text 8-fold. Modern models are likely much better.

Re: πFS

#192

Post by ChrisMarshallNY »

khimaros wrote: post #156
Levitating wrote: post #88

Earlier quoted context omitted.

Even using algolia, I don't see a way to generate a list in this exact format. I think ChrisMarshallNY is right, dang has access to eldritch powers.

the Glider HN app for Android shows related posts with high overlap to dang's list, so it must be possible for mere mortals after all.

It was really just a rhetorical joke, but I wrote an app that is a system, based on a custom backend and native frontend.

I wrote a special native management app, and often use that, to implement dashboard functionality, like the kind of thing that the HN mods do.

Yeah, I could, for example, feed the logs into an LLM, and get fancy reports, but it’s a lot easier to simply hit the charts button in the navbar, and view interactive graphs, customized exactly for my workflow.

Re: πFS

#193

Post by utopiah »

"This file doesn't look like what I remember.

Are you sure? It's been a while since you last opened it. Memory is funny like that. The file is fine — maybe take another look with fresh eyes."

from https://github.com/philipl/inferencefs/

Maybe I do not indeed remember properly. Anyway, back to watching "Eternal Sunshine of the Spotless Mind" for the first time, I think.

Re: πFS

#195

Post by Galanwe »

russfink wrote: post #105
mondrian wrote: post #55

Earlier quoted context omitted.

The index of your 20 line file is

Unless, in turn, you locate the index itself in pi at a much smaller index. And so on... Find k candidate indices for your data, then locate each of them. If the smallest one is a significantly smaller index space, repeat.

It's recursive as well, you now need to store how many levels of indirection of indices you had to resolve, which will in turn take 20TB to store, unless you store that in pi as well, which in turn...

Re: πFS

#196

Post by initramfs »

nairboon wrote: post #182
fc417fc802 wrote: post #170

Earlier quoted context omitted.

That's known as cover traffic and is a tactic employed by many of the anonymity oriented overlay networks. I'll note that any observer already has this problem to the extent that video streams are also encrypted. However most observers presumably recognize the endpoints as well as being able to classify the traffic by means of statistical analysis. What might be useful would be a tool to generate arbitrary user data…

Local llms and diffusion models can help you with that.

https://github.com/marcus0x62/quixotic bot and LLM obduscator

Re: πFS

#198

Post by briansm »

jamwise wrote: post #31

Reminds me of when I tried to use the library of babel as a data compression tool. It led me down a fun rabbit hole and was my first introduction to information theory. The conclusion being that you basically need the same amount of data to represent the address of your data as the data itself, so it's not really effective at compression, just a fun thought experiment. The cool part of this in modern times is that LL…

LLM's seem to be the weird interesting outcome of applying lossy (de)compression concepts to text instead of the audio/image/video domains where they have traditionally been used.

Re: πFS

#199

Post by nz »

ithkuil wrote: post #119
jamwise wrote: post #31

Reminds me of when I tried to use the library of babel as a data compression tool. It led me down a fun rabbit hole and was my first introduction to information theory. The conclusion being that you basically need the same amount of data to represent the address of your data as the data itself, so it's not really effective at compression, just a fun thought experiment. The cool part of this in modern times is that LL…

You'll find this an interesting watch: Reinventing Entropy Compression is Intelligence Part 1 3blue1brown https://youtu.be/l6DKRf-fAAM?is=ne73FCJ7ErXhzZ-v

You, and the HN users, `lojban`, `klingon`, `ido`, `brithenig`, `solresol`, `babm`, and `tokipona`, may want to start a club. Amusingly, nobody seems to have registered the `esperanto`, `volapuk`, `interslavic`, `balaibalan`, and `dothraki` usernames.

Re: πFS

#200

Post by jnovek »

jamwise wrote: post #31

Reminds me of when I tried to use the library of babel as a data compression tool. It led me down a fun rabbit hole and was my first introduction to information theory. The conclusion being that you basically need the same amount of data to represent the address of your data as the data itself, so it's not really effective at compression, just a fun thought experiment. The cool part of this in modern times is that LL…

If you set temperature to 0.0 you almost have a key-value store, but finding the right key for your value might take some effort.
Post reply on HN