Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM Wiki – example of an "idea file"

gist.github.com

Post reply on HN
Print view View original

101–109 of 109 posts

Re: LLM Wiki – example of an "idea file"

#101

Post by npodbielski »

thoughtpeddler wrote: post #94

I’m surprised Karpathy thinks this is a viable solution for a quasi-continual learning system. Yes, it’s cool to experiment with these sort of ‘intermediate knowledge systems’, but the real goal within this current LLM paradigm remains clear: new information within a knowledge system should be manifest through updating the weights ! Many efforts are taking a crack at this, which this really helpful talk [0] by Jack M…

> [0] “Stuffing Context is not Memory, Updating Weights Is":

This guy is not a good speaker. Is there any article about it?

Re: LLM Wiki – example of an "idea file"

#103

Post by vbarsoum » 

  I built an implementation of this and tested it on 3 Alex Hormozi books (~155K words, 68 source files). Some data for the skeptics:
                                                                                                                                                                                              
  The naive version (each book as 1 file) produced exactly the slop people are describing here. But splitting into chapter-level files and recompiling changed the output categorically. Same model, same prompts — the only variable was source granularity.                                                                                                                             
                  
  The compiler produced 210 concept pages with 4,597 cross-references (19.2 avg links per page). 20+ concepts synthesized across all 3 books unprompted — one pulled from 11 source files and found a genuine contradiction between two books that neither makes explicit. 173K words of output from 155K input. It's not compression — it's synthesis.
                                                                                                                                                                                              
  The thing I think the "this is just RAG" comments are missing: a vector database is only useful to machines. You can't open a .faiss file and browse it. A wiki is useful to both. I open these files in Obsidian, browse the graph, follow links, read concept pages — no AI needed. But when I do ask the AI a question, it reads the same wiki pages I do, and the answers are better than RAG because the knowledge is already structured and cross-referenced instead of retrieved as raw chunks.                                                                        
                  
  That's the key insight in Karpathy's idea. The compiled wiki is the interface for humans AND the knowledge layer for AI. Same artifact, two audiences.                                      
   
  ~Cost: 12M tokens, ~10-15 min. Repo: https://github.com/vbarsoum1/llm-wiki-compiler

Re: LLM Wiki – example of an "idea file"

#105

Post by asakin »

I'd been working on a personal wiki system for a while and adapted it to implement this pattern. Ended up as a git template:

https://github.com/asakin/llm-context-base

Main additions on top of the pattern: a training period where the AI learns how you work over 30 days then gets quieter over time, a metadata standard so files are queryable by summary, and a lint pass for stale content and context loading optimization. Never have to design a taxonomy upfront.

Re: LLM Wiki – example of an "idea file"

#108

Post by saberience »

claudiug wrote: post #98

i had the feeling that Karpathy is not a software developer at all, reading his tweets is like he is discovering half a century ago ideas. like specification...

He's not, he was an AI researcher and then led AI research teams.

He's doing "engineering" now since he can use LLMs, but he never really spent years doing what we would call software engineering. I.e. building distributed systems, writing terraform/ansible, maintaining old databases or optimizing MySQL indexes, debugging Kafka or AzureMQ, etc.

Don't get me wrong, he's knowledgeable about LLMs specifically, (although his knowledge is rapidly becoming out of date) but he's not a software engineer, which is why his ideas around engineering often seem totally deluded.

Re: LLM Wiki – example of an "idea file"

#109

Post by saberience »

Karpathy is at his best when working on teaching materials for ML beginners.

When it comes to other stuff he seems to be in a X/Twitter induced AI Psychosis like Garry Tan where he thinks everything he does is amazing and novel because he gets glazed by 1000 X bots who just post "You're amazing" after anything he tweets.

This is most definitely an (old) solution in search of a problem. Plenty of people have been trying variants of this for several years at this point but the issue isn't putting stuff into a wiki, or git, or markdown files. It's how you keep then up to date, how you deal with conflicts, how you deal with bloat, how you decide what to keep and delete over time, and also, when you've got this big mass of notes and markdown, when do you surface it?

It sounds great on paper until you try and use it and realize that in reality it isn't that useful and doesn't become part of your daily life. That is, it's more fun to build than to actually use, and you don't end up using it outside of the initial novelty.

Post reply on HN