Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Screen memory without screenshots, just text to Markdown

github.com

Post reply on HN
Print view View original

11–20 of 30 posts

Re: Show HN: Screen memory without screenshots, just text to Markdown

#11

Post by ShinyLeftPad »

zahlman wrote: post #10
etchalon wrote: post #3

I experimented with this exact same approach earlier this year. It's barely sufficient, because, bluntly, most apps just aren't wired up right. So you end up having to hand code a lot of specific profiles for specific apps to make this work well, and even then, you don't quite get the right level of detail to make it work out. Will try this app, to see if it improved on my own approach, but man, the hope levels are l…

One of the reasons people like TUIs is because the text is always just right there.

Not if it's rendered on GPU, I guess?

Re: Show HN: Screen memory without screenshots, just text to Markdown

#12

Post by albert_e »

etchalon wrote: post #3

I experimented with this exact same approach earlier this year. It's barely sufficient, because, bluntly, most apps just aren't wired up right. So you end up having to hand code a lot of specific profiles for specific apps to make this work well, and even then, you don't quite get the right level of detail to make it work out. Will try this app, to see if it improved on my own approach, but man, the hope levels are l…

I hope in near future _that_ layer of abstraction -- looking at a fairly standard application window with minor UI variations and reasoning about what area / labels within the UI mean what (possibly paired with app documentation) -- could probably become a light-weight fine-tuned vision model it itself that can run fully locally.

Re: Show HN: Screen memory without screenshots, just text to Markdown

#13

Post by Dramatize »

zahlman wrote: post #9

> reads the text of your focused window every few seconds through the Accessibility API > It writes plain markdown Where are the formatting decisions coming from?

So the markdown is scaffolding the app generates deterministically.

The ## heading is built from the block's timestamps, the app name and the window title. The frontmatter is per-day boilerplate. The file:/url: line is the window's backing document where the app exposes one. The captured text underneath is written exactly as the tree handed it over: plain lines, no reconstruction.

That lossiness is also why the file/URL references exist. Trying to rebuild a document's formatting from its accessibility tree is a losing game, so instead each block records where the real document lives, and the LLM reading the file can open the original if the fragments aren't enough. "Plain markdown" was meant as "a markdown file you can open anywhere", not "faithful markdown conversion of what you saw"

Re: Show HN: Screen memory without screenshots, just text to Markdown

#14

Post by Dramatize »

hetsaraiya wrote: post #8
etchalon wrote: post #3

I experimented with this exact same approach earlier this year. It's barely sufficient, because, bluntly, most apps just aren't wired up right. So you end up having to hand code a lot of specific profiles for specific apps to make this work well, and even then, you don't quite get the right level of detail to make it work out. Will try this app, to see if it improved on my own approach, but man, the hope levels are l…

I haven't read about how the Codex Appshots work yet, but this can be used to extract text properly. I guess. How does this idea look to you?

Appshots is fine for what it is, but not great for building ambient context about your daily usage/work.

Re: Show HN: Screen memory without screenshots, just text to Markdown

#15

Post by hetsaraiya »

Dramatize wrote: post #14
hetsaraiya wrote: post #8

Earlier quoted context omitted.

I haven't read about how the Codex Appshots work yet, but this can be used to extract text properly. I guess. How does this idea look to you?

Appshots is fine for what it is, but not great for building ambient context about your daily usage/work.

But we can clean that extracted text from Appshots.

UPDATE: As i just checked it is also using Accessibility API. So i guess we will have access to the same set to data.

Re: Show HN: Screen memory without screenshots, just text to Markdown

#17

Post by jwr »

Hmm. What would be really helpful for me would be an enhancement of the TypeWhisper app for dictation that would be able to get the context of what I am dictating into and send it along with the prompt to an LLM.

Probably much simpler and much less of a privacy problem (I run my own local LLM for that purpose so that nothing leaves the machine).

Re: Show HN: Screen memory without screenshots, just text to Markdown

#19

Post by hzwanip »

latexr wrote: post #18
purplecats wrote: post #2

neat. why not screenshot and tesseract (videos/images/viewport/etc)

Project reasons aside, macOS has perfectly capable, local, native OCR that you can access programmatically, without having to install something else.

OCR starts from screenshots though. This only grabs text via accessibility APIs, before rendering or any screenshots.

Re: Show HN: Screen memory without screenshots, just text to Markdown

#20

Post by latexr »

hzwanip wrote: post #19
latexr wrote: post #18

Earlier quoted context omitted.

Project reasons aside, macOS has perfectly capable, local, native OCR that you can access programmatically, without having to install something else.

OCR starts from screenshots though. This only grabs text via accessibility APIs, before rendering or any screenshots.

Yes, I know. Like I said: “Project reasons aside”. I’m not suggesting OCR for this, I’m imparting the general information to be used in other situations that in macOS you can OCR without requiring third-party tools.
Post reply on HN