Nice constraint. I've been building in this space for a while now and the lossiness question is a real one. The accessibility tree works for a lot of things but the more I worked on it the more stuff I found needed pixels.
YMMV.
Show HN: Screen memory without screenshots, just text to Markdown
21–30 of 30 posts
Re: Show HN: Screen memory without screenshots, just text to Markdown
#22Earlier quoted context omitted.
OCR starts from screenshots though. This only grabs text via accessibility APIs, before rendering or any screenshots.
Yes, I know. Like I said : “Project reasons aside”. I’m not suggesting OCR for this, I’m imparting the general information to be used in other situations that in macOS you can OCR without requiring third-party tools.
Fair! It is actually new and interesting information for me (not into macOS dev)
Re: Show HN: Screen memory without screenshots, just text to Markdown
#23I've liked the focused window tracker in https://activitywatch.net
Re: Show HN: Screen memory without screenshots, just text to Markdown
#24A cool project, but feels like fundamentally the wrong approach when what you do most likely leaves a string of structured digital footprints anyway. I’ve got an agent that fills out my timesheets by looking at git commits, agent history, Slack messages, emails, and time-tracker tickets. I guess I could add relevant web-browsing?
Re: Show HN: Screen memory without screenshots, just text to Markdown
#25[flagged]
Re: Show HN: Screen memory without screenshots, just text to Markdown
#26neat. why not screenshot and tesseract (videos/images/viewport/etc)
When I built rem, I spent significant effort getting screenshot -> ocr + screenshot -> ffmpeg loop energy efficient, but it definitely is more expensive than accessibility API.
You also save a lot of disk space and writes to disk.
That being said, you lose the cool swipe to go back in time and search through history and visually see, features.
And situations where accessibility isn't supported.
And as others have mentioned, built in ocr is definitely better than tesseract.
Re: Show HN: Screen memory without screenshots, just text to Markdown
#27I've liked the focused window tracker in https://activitywatch.net
Oh interesting, I'll check it out
Re: Show HN: Screen memory without screenshots, just text to Markdown
#28Update: I've added a prompt to process the raw output into something more useful: https://github.com/dragthelake/ambient-context/blob/main/doc...
This is pretty magical tbh
Also added MIT license and added some more pruning to reduce initial capture by around 10%.
Re: Show HN: Screen memory without screenshots, just text to Markdown
#29The per-app breakage is the real problem with text extraction, not accuracy, like etchalon pointed out. Screenshots dodge that since they look the same no matter what is rendering. Cropping to the window and marking it up before it goes anywhere also skips the full screen recording and the orange icon. https://slimsnap.ai already works that way.
Re: Show HN: Screen memory without screenshots, just text to Markdown
#30I've released the v 1.0.0 with a bunch of improvements + a signed app. It's the first time I've done this, so still working out exactly what's required: https://github.com/dragthelake/ambient-context or ambientcontext.app