Personal computing paves the way for personal library science
1–10 of 52 posts
Re: Personal computing paves the way for personal library science
#2> Personal Library Science is the leverage of LLM technology, applied to a personal library. A personal library differs from a impersonal library in the fact that a personal library is an interpretation of a source material. These interpretations include: photographs from different photographers at the same event, or favorite scenes from a movie, or favorite passages from books, parts of songs that bring you to tears, etc. Importantly, these interpretations create unique sets that go on to create unique problems which require unique, idiosyncratic solutions.
Would an LLM-driven "Personal Library" require manually annotated textual interpretation of each curated item, or could it derive personal interpretations from user history and the uniqueness of curated items/sets?
For those who have been using local, offline LLMs with a manually curated text/image corpus, what have been the most valuable or surprising use cases?
Author demo video (2023), https://youtube.com/watch?v=7TgqMRz2r3M & tooling comment (2024), https://news.ycombinator.com/item?id=39789712
> Inspired by the commonplace book format, I take highlights from Kindle and embed them in a DB. From there I build (multiple) downstream apps but the central one, Commonplace Bot is a bot that serves as a retrieval and transformer for said highlights.
Re: Personal computing paves the way for personal library science
#3> Personal Library Science is the leverage of LLM technology, applied to a personal library. A personal library differs from a impersonal library in the fact that a personal library is an interpretation of a source material. These interpretations include: photographs from different photographers at the same event, or favorite scenes from a movie, or favorite passages from books, parts of songs that bring you to tears…
> Would an LLM-driven "Personal Library" require manually annotated textual interpretation of each curated item
No. In something like this you’d probably have the LLM annotate and curate your personal library for you.
Potentially by creating and assigning tags or topics based on the content of your library.
Re: Personal computing paves the way for personal library science
#4there may be an important divergence implied by this essay .. people here ask about using an LLM.. but the essay refers to "different photographs of the same scene from different photographers" or other personal collection items that are related but subjective or not-authoritative
There is a rush in public to condense and summarize many authoritative publications to find patterns, or to replace a human expert with automated results.. yet that is fundamentally different than taking multiple incomplete perspectives to add to a human library-owners knowledge and investigations.
It is subtle to speak it but not subtle in its implications.. taking "data as facts" and condensing them or reordering them or rewriting an output based on them, using automation, is different than a human mind taking in many inputs for human mind knowledge and enabling new outputs from a human author.
Re: Personal computing paves the way for personal library science
#5> Personal Library Science is the leverage of LLM technology, applied to a personal library. A personal library differs from a impersonal library in the fact that a personal library is an interpretation of a source material. These interpretations include: photographs from different photographers at the same event, or favorite scenes from a movie, or favorite passages from books, parts of songs that bring you to tears…
> Would an LLM-driven "Personal Library" require manually annotated textual interpretation of each curated item No. In something like this you’d probably have the LLM annotate and curate your personal library for you. Potentially by creating and assigning tags or topics based on the content of your library.
Yea. I had this discussion not too long ago about this. I'd love to have a combination of a library (Personal Knowledge Management style), data ingestions, and a current world view/state.
The PKM is the stored info to write to and query against (both for LLMs and humans). The data ingests are just a pipeline of digital inputs to the system, like chat logs, maybe (transcribed) webcam feeds, files i'm currently editing on desktop, browsing history, etc. The current world view is the interpretation of what i'm doing - to tie all the ingests together and give them context. Eg in isolation browsing some Rust crates might not be that useful. But if i'm also editing Project X on my computer then it's reasonable to assume the searching is related to X. However if it's been 8 hours since any Project X activity, it's less likely related. Same goes for context-less chat logs (as happens frequently in my house) where they are extensions of a voice conversation, etc.
All of this stuff is of course insanely privacy invading, so i'd only implement this locally. I also wouldn't even store most of it for fear of data invasion, but using it to fuel a PKM automatically seems pretty sexy. Like browser history, but for your life.
This is all just wishful thinking though, LLMs have been moving too fast for me to even bother toying with this. I should note though that i did not intend for LLMs to be "smart". Rather, in a RAG-like fashion (i think is the term), i want to just let LLMs do what they're good at - summarization & autocomplete, and let the world view / PKM store the real data.
Re: Personal computing paves the way for personal library science
#6> The "during" is hard work, and very lonely work. There are no promises of success, and indeed, the path is one where you can't see more than three feet ahead of you and you exist on the cliff's edge of extinction by any silly mishap. The work of "during" is exhausting, and it constantly holds you taut and alert, afraid of the shadows that lurk beyond the campfire's edge.
Well said. All anyone can do is to do the lonely work till you can't anymore or you find friends to not be lonely at that work anymore.
Re: Personal computing paves the way for personal library science
#7I've had related ideas lurking at the back of my mind for a while now. Essentially, I want to save more things locally and and interact with it. For example, I have a bunch of book notes stored in Bear. I'd like to be able to ask questions about those notes, and also show the pages of the book itself.
Re: Personal computing paves the way for personal library science
#8> Personal Library Science is the leverage of LLM technology, applied to a personal library. A personal library differs from a impersonal library in the fact that a personal library is an interpretation of a source material. These interpretations include: photographs from different photographers at the same event, or favorite scenes from a movie, or favorite passages from books, parts of songs that bring you to tears…
> Would an LLM-driven "Personal Library" require manually annotated textual interpretation of each curated item, or could it derive personal interpretations from user history and the uniqueness of curated items/sets?
I’ve personally found that tagging is less robust than LLM embeddings (mainly due to dimensionality), but human appended thoughts about a source — also embedded — serve even better as tags.
Example: “this is a quote about dinosaurs…” (Old way of doing things) Tags: dinosaurs, jurassic, history Query: “dinosaurs” > results = 1…
(New way of doing things) Embedded Quote: [0.182…] User Added Thought: “this dinosaur reminds me of a time i went to six flags with my cousins and…” Embedded User Added Thought: [0.284…]
Query: “dinosaurs” > results = 2 (indexes = sources, thoughts)
The "thoughts" index can do a second layer cosine similarity search and serve as a tag on its own to fetch similar concepts. Basically a tree search created by similarity from user input/feedback loops.
Re: Personal computing paves the way for personal library science
#9there may be an important divergence implied by this essay .. people here ask about using an LLM.. but the essay refers to "different photographs of the same scene from different photographers" or other personal collection items that are related but subjective or not-authoritative There is a rush in public to condense and summarize many authoritative publications to find patterns, or to replace a human expert with au…
> There is a rush in public to condense and summarize many authoritative publications to find patterns, or to replace a human expert with automated results.. yet that is fundamentally different than taking multiple incomplete perspectives to add to a human library-owners knowledge and investigations.
It is subtle to speak it but not subtle in its implications.. taking "data as facts" and condensing them or reordering them or rewriting an output based on them, using automation, is different than a human mind taking in many inputs for human mind knowledge and enabling new outputs from a human author.
You nailed it! Thanks for noticing the divergence!
Re: Personal computing paves the way for personal library science
#10Personal computing has stagnated for such a long time, it creates substantial uncertainty about what state it might evolve to if and when the next step actually happens.
In this respect local LLM's are simply the tip of the iceberg, pointing out the vast amount of personal information processing that is available in principle but does not actually happen.