The holy grail [
https://beepb00p.xyz/pkm-search.html#future] of this really resonated with me and fully mirrors what I've been thinking about the past few months. In my observations, it's input capture, information organization, and subsequent retrieval:
Information Capture:
Input Capture - You’re going to have all-encompassing tracking and recording of all activity, but want configurable privacy on the extent to which you want your daily conversations and observations of external things you encounter and are exposed to. Capturing input needs to be holistic and incorporate all properties of encounters and new information.
Potential sources of input:
Vision — point of view recording, see snapchat spectacles, etc as primitive examples.
Audio (voice notes and multi-party conversations) - voice calls, video, etc. and other forms of audio transmission where there is more than a single party in the interaction.
Digital interactions
You will need to keep track of web pages you visit at what times
Conversations you see on Twitter, etc.
Properties and cues must be extrapolated from the information that is captured on input, in the case of audio, transcriptions are sufficient for transcription and retrieval purposes, however since video is a visual medium, it includes significantly more properties that need to be accounted for.
The aim here is to identify sufficient data points (cues) that are subsequently represented in such a way that they are easy to search across things you have encountered but only seem to recall a certain property or cue from. This is because of the fact that human beings tend to remember things in fragments, for instance, you might remember a certain color on a page that you visited within the last 6 months and nothing else.
So long as you are capturing sufficient input and actions then you should be able to go back to any given point in time. How and where are you going to store this information? Storing everything is going to be a large amount of data. The essence of the information and context must be preserved. If you want to wind back to an arbitrary position in time with the original context intact, you want to retain as much as you can in the most efficient manner possible, so determining which data points to retain is essential. (Once the content structure has been figured out, this will be viable).
Examples of Primary Cues:
Time - humans generally keep track of things in a linear time-based fashion.
Color - invokes emotion and is memorable.
Physical Location - the efficiency of information retrieval is highly influenced by the location at which it is originally synthesized, encountered, and stored.
Keywords - the default conventional mode. Can and should be extracted from video/imagery and audio.
Imagery - search for images based on their contents and ambience.
Potential Secondary Cue — Music - see historical associated input and actions while certain music was played.
(What else?)
Meta Cues — Subjects - Automated tagging of keywords/encountered content.
Any combination of these queries is possible, but ultimately the killer feature is the ability to backtrack through time to find a certain piece of information that is made available thanks to the always-on recorded nature of your interactions with the physical and digital worlds combined.
Knowing what to store, and how, + displaying it needs to be worked on further.