Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I made an open source Windows app to rewind and search everything on screen

tonoko.notion.site

Post reply on HN
Print view View original

111–120 of 172 posts

Re: I made an open source Windows app to rewind and search everything on screen

#111

Post by dbish »

modeless wrote: post #52

Has anyone built something like this using accessibility APIs instead of (or in addition to) OCR? It seems like a waste to OCR everything when you could just get the text directly from the accessibility APIs. Also seems like potentially a good way to connect LLMs to UIs, and something like this would be the way to collect the training data.

We've done a bit of both for our screen seachable loom-like screen recorder, the problem is that the accessibility APIs differ greatly between Mac and Windows if you want to be OS agnostic and even on Windows all the apps tend to do things a little differently making it hard to say what did you actually "see", with some apps missing key data or implementing it incorrectly. OCR ends up being easier many times desptie thinking accessibility would be.

Re: I made an open source Windows app to rewind and search everything on screen

#112

Post by dbish »

novok wrote: post #50
poisonborz wrote: post #8

Congrats on the great execution of this novel idea! Inspiring for everyone with a "why isn't there an app that does x" idea.

rewind.ai is another example of this, but their recent pivot into cloud only storage for this and renaming to limitless.ai makes me glad that open source stuff like this popping up so you are not forced into a cloud storage situation. And I say this as a paying customer who will probably stop being one.

they also seem to have shifted to hardware/"AI" wearables (lavalier mics?). hardware is not a promising shift imho

Re: I made an open source Windows app to rewind and search everything on screen

#113

Post by zamadatix »

yarg wrote: post #91
Intralexical wrote: post #68

Earlier quoted context omitted.

Previously commented with some other details. TL;DR: It's an old PhD project that requires custom patches to an ancient kernel version. So I guess you can't download it, and even if you could it wouldn't work on any system you'd want to use today: > compsciphd on Oct 11, 2022 | parent | context | favorite | on: Linux NILFS file system: automatic continuous snap... > we used NILFS 15 years ago in dejaview - https://ww…

> vnc didn't work for us as we needed better performance 15 years is a long time, performance wise VNC would be adequate and we're at the point now where OCRing a render is probably fast enough for most use cases (it's actually the most reasonable choice a lot of the time, e.g.: PDF parsing). (But how's VNC support looking on Wayland based window managers?)

VNC is available for pretty much any Wayland window manager but your distro may opt to default build the more modern RDP instead (e.g. Ubuntu Gnome).

Re: I made an open source Windows app to rewind and search everything on screen

#114

Post by dbish »

bee_rider wrote: post #30
hu3 wrote: post #22

It would be cool to: 1) Make a Meta Smart Glasses take a photo every 2 seconds. 2) Send images to some server in the cloud. 3) Run OCR and object detection/labeling on these images. 4) Then present an app that allows searching (and chatting with) your past. I could then ask the LLM things like: 1) Where did I left my wallet? 2) Did I get my credit card back after paying the restaurant yesterday? (ADHD things, don't a…

Generally Black Mirror episodes are not intended to be aspirational.

really? i tend to look to them for ideas

Re: I made an open source Windows app to rewind and search everything on screen

#115

Post by dbish »

ppqqrr wrote: post #59

Anyone know if there’s an audio equivalent of this software?

Not "always on" but if you enable a recording with us, it's all searchable and even attempts to write a doc of what you were talking about to scan or answer questions about later: https://augmend.com. We do screen + audio, but audio alone does work.

Re: I made an open source Windows app to rewind and search everything on screen

#116

Post by Sephr »

dbish wrote: post #111
modeless wrote: post #52

Has anyone built something like this using accessibility APIs instead of (or in addition to) OCR? It seems like a waste to OCR everything when you could just get the text directly from the accessibility APIs. Also seems like potentially a good way to connect LLMs to UIs, and something like this would be the way to collect the training data.

We've done a bit of both for our screen seachable loom-like screen recorder, the problem is that the accessibility APIs differ greatly between Mac and Windows if you want to be OS agnostic and even on Windows all the apps tend to do things a little differently making it hard to say what did you actually "see", with some apps missing key data or implementing it incorrectly. OCR ends up being easier many times desptie…

OCR is easier for the developer, but worse for the user in terms of battery drain / energy use.

Re: I made an open source Windows app to rewind and search everything on screen

#117

Post by LoganDark »

somenameforme wrote: post #58
Cyph0n wrote: post #47

Earlier quoted context omitted.

Have you watched the Black Mirror episode in question (S1E3)? Because the point of it was the tech itself and had nothing to do with sharing or the cloud.

An extremely similar take on this, that predates Black Mirror, was in some sci-fi media whose name, appropriately enough, currently escapes me. Their premise is that there was a species who evolved to develop a flawless memory. It's a subset of the Black Mirror episode (and this dubious idea) because there, at least, memory is a purely private thing. Yet the end too was quite predictable and logical -- frequently des…

> What need is there for the rest of your life when you can simply endlessly relive, in perfect clarity, the best moments of your life - ones that you, in many cases, will likely never surpass?

Even without perfect memory, this can still happen, especially if something like PTSD is involved. I myself have a dissociative disorder and sometimes it can be difficult not to just relive the past over and over. (It's not like this is a daily struggle, but during low points it comes up.)

Re: I made an open source Windows app to rewind and search everything on screen

#118

Post by dbish »

Sephr wrote: post #116
dbish wrote: post #111

Earlier quoted context omitted.

We've done a bit of both for our screen seachable loom-like screen recorder, the problem is that the accessibility APIs differ greatly between Mac and Windows if you want to be OS agnostic and even on Windows all the apps tend to do things a little differently making it hard to say what did you actually "see", with some apps missing key data or implementing it incorrectly. OCR ends up being easier many times desptie…

OCR is easier for the developer, but worse for the user in terms of battery drain / energy use.

For sure, we made a privacy tradeoff to do it server side (given some screen change delta) because of this. Accessibility is a good "in addition to" but there are just so many apps that don't handle it well

Re: I made an open source Windows app to rewind and search everything on screen

#119

Post by akoboldfrying »

jackvalentine wrote: post #110

I have had this thought for ages and I'm stoked to see someone having built it. It seems to create a huge risk for your data though. Any ideas about how you might go about securing against the application itself doing something untoward?

It runs locally, so you could just block outgoing network connections for the app.

Re: I made an open source Windows app to rewind and search everything on screen

#120

Post by pests »

LeonB wrote: post #29
wartijn_ wrote: post #25

Earlier quoted context omitted.

That’s fast :) Do you get notified when someone mentions your product, or are you _that_ active?

Neither, lol. Just happened across the comment within five minutes of it being posted. I think it was one of the first times I visited HN today.

The first first time, or the second first time, maybe the third?
Post reply on HN