Knowledge Compressor
githubnext.com
Knowledge Compressor
1–6 of 6 posts
Re: Knowledge Compressor
#2How far can documentation be compressed without losing its meaning?
Re: Knowledge Compressor
#3How far can documentation be compressed without losing its meaning?
This is a good question! The answer is, unfortunately, uncomputable – or so we think. But if you manage to find a more satisfying answer, you'll have solved one of the more beguiling problems of the last century, which is the basis for most of modern Information Theory. You can learn more about it here: https://homepages.cwi.nl/~paulv/kolmogorov.html
Re: Knowledge Compressor
#4Hello! Idan (from Next) here, and others from the team will drop by the thread if you have questions!
Re: Knowledge Compressor
#5Did you consider extracting knowledge from the doc into a graph database (while passing the Q&A) and compressing the graph instead?
Re: Knowledge Compressor
#6Did you consider extracting knowledge from the doc into a graph database (while passing the Q&A) and compressing the graph instead?
https://huggingface.co/datasets/ladybugdb/github-knowledge-c...
These methods could not beat the 480 token result because of the small dataset. But for a sufficiently large corpus with graph reordering and columnar parquet compression, they could be quite competitive.
The parquet files are self explanatory. For knowledge.lbdb.zst, first decompress with zstd and then run a cypher such as:
lbug> MATCH (a:E)-[t:TRIPLE]->(b:E) RETURN a.name AS subj, t.predicate AS pred, b.name AS obj, t.props AS props, t.scope AS scope;