Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hister – A private, full content search index that you control

hister.org

Post reply on HN
Print view View original

41–50 of 110 posts

Re: Hister – A private, full content search index that you control

#41

Post by zargon »

asciimoo wrote: post #7
zuzululu wrote: post #3

Earlier quoted context omitted.

interesting you used AGPL 3 licensing, are you planning a hosted version? would've been great with a more liberal license

> are you planning a hosted version Not in the near term. Right now I am focused on developing Hister rather than operating a hosted service. There are already plenty of centralized hosted search engines, so my longer term interest is in federation and distributed search. I want to make the core system mature first. > would've been great with a more liberal license It depends on how do you define liberal. =] I chose…

AGPLv3 is awesome, great choice!

Re: Hister – A private, full content search index that you control

#42

Post by Groxx »

asciimoo wrote: post #36
Groxx wrote: post #21

Earlier quoted context omitted.

I assume it content-extracts that on upload? I'd really like to move storage into hister too, if possible. That way you could also switch from an extracted view to a "full" view in the UI. Though I assume that'd be fairly simple to build later. Overall I really like what I'm seeing, it ticks a lot of important boxes for me and it's pleasantly straightforward. Hopefully I'll find time to contribute!

> I assume it content-extracts that on upload? Hister always stores the original material. > That way you could also switch from an extracted view to a "full" view in the UI. It isn't even needed, we just need a SingleFile specific extractor (an interface in Hister to parse specific page content and provide custom previews) that provides the full original HTML for the preview panel. > Hopefully I'll find time to cont…

Oooh, now I see the extractor-view setting in the UI. Yeah, that's essentially perfect \o/

Thank you again!

Re: Hister – A private, full content search index that you control

#43

Post by wmchen »

asciimoo wrote: post #23
wmchen wrote: post #19

I set this up a few months ago based on asciimoo's comments on HN, and barely used it at first, but I realized not too long ago that it could be a pretty useful research tool for one of my hobbies (award travel), that revolves around being in the know around various concepts and quirks. I scraped and imported posts from the blogs I regularly reference for award travel, then hooked it up to OpenCode/Codex as an MCP se…

Wow, this is a really inspiring use case and blog post. Thanks for sharing it. What tools or features would Hister need to support your complete search workflow?

Thank you so much for building this! It's a really awesome piece of work and I'm grateful for your work. I'd love to sponsor you on Github in the near future.

The only thing that I think would be interesting to see is native support for crawling via a sitemap.xml instead of recursively. I worked around this by implementing a basic scraper that fetched pages exclusively from the sitemap.xml to add into Hister.

I think you're already aware of this, but I also experienced some data loss during the import because I was running a concurrent reindex. I clocked it pretty quickly so I didn't think too much of it. [1]

[1] "TODO store new documents in both indexes while running reindex to guarantee not losing any data." @ https://github.com/asciimoo/hister/blob/master/server/indexe...

Re: Hister – A private, full content search index that you control

#44

Post by exe34 »

asciimoo wrote: post #39
exe34 wrote: post #34

Earlier quoted context omitted.

Is this reply from Gemini? Because I'm used to gemini gaslighting me. The answer is the opposite of my question.

Doh, sorry, the answer was coming from me who did not read the question properly. It currently cannot import browser bookmarks, but it is a good idea. Added to my TODO.

Ah sorry didn't mean to be mean, I just wondered if you had automated it haha!

Re: Hister – A private, full content search index that you control

#45

Post by Carrok »

asciimoo wrote: post #20
Carrok wrote: post #11

Earlier quoted context omitted.

It seemed like the public mode was the default when I set it up. If so, that’s a fairly dangerous default as keeping a “clean” history with no secrets leaked seems neigh impossible.

The default configuration binds only to localhost, and a fresh installation starts with an empty database/index. Could you clarify which specific attack surface you are concerned about in that scenario?

I’m not concerned about an attack scenario. I’m just saying that using the docker image, if someone (or their agent) isn’t careful, they could expose their browsing history publicly fairly easily. It might just be nice to default to at least a user and pass login rather than just wide open.

Re: Hister – A private, full content search index that you control

#46

Post by zuzululu »

asciimoo wrote: post #7
zuzululu wrote: post #3

Earlier quoted context omitted.

interesting you used AGPL 3 licensing, are you planning a hosted version? would've been great with a more liberal license

> are you planning a hosted version Not in the near term. Right now I am focused on developing Hister rather than operating a hosted service. There are already plenty of centralized hosted search engines, so my longer term interest is in federation and distributed search. I want to make the core system mature first. > would've been great with a more liberal license It depends on how do you define liberal. =] I chose…

it prevents certain innovations to be derived from it but with LLMs I do think these licensing are pretty much moot.

i still think it could've benefited by Apache 2.0 which more or less gets you to your goals

Re: Hister – A private, full content search index that you control

#47

Post by mfld »

asciimoo wrote: post #2

Ohi, author here! Thanks for posting Hister. Feel free to A.M.A. My first free software search project was Searx, a privacy respecting metasearch engine, but because of the limitations of the metasearch concept, I've decided to take a different approach. Hister builds a personal search index from pages you visit, bookmarks, browser history, local files, and crawled websites. It stores extracted content with offline r…

What would be a typical size of the search index, let's say after 5 years of intense browsing?

Re: Hister – A private, full content search index that you control

#49

Post by genxy »

asciimoo wrote: post #15
Groxx wrote: post #6

Earlier quoted context omitted.

> Hister builds a personal search index from pages you visit, bookmarks, browser history, local files, and crawled websites. Immediately interested and will check it out, thank you! I've wanted a "search stuff you've seen online" tool for a long time, but everything seems to be research-oriented or "archive but don't search" or some weird combination that means it's nigh useless to me. I've got decades of bookmarks a…

Exactly! I had the very same issues before Hister. > One question if ya don't mind, while I explore: any chance of singlefile support? Yes, partially. Hister can already import HTML files created by SingleFile, but there is no direct integration yet. In the longer term, I would like the SingleFile extension to be able to send snapshots directly to Hister.

I came here to mention https://github.com/gildas-lormeau/singlefile, I couldn't get it to do what I want so I built my own, for watched domains it pushes a copy of the serialized DOM to a local search database. For structured data, either extract and enrich in the browser or enrich on the server side.

Would Hister support this basic workflow? I'd love to retire my own software.

The next phase was going to move to a recording proxy.

Re: Hister – A private, full content search index that you control

#50

Post by asciimoo »

mfld wrote: post #47
asciimoo wrote: post #2

Ohi, author here! Thanks for posting Hister. Feel free to A.M.A. My first free software search project was Searx, a privacy respecting metasearch engine, but because of the limitations of the metasearch concept, I've decided to take a different approach. Hister builds a personal search index from pages you visit, bookmarks, browser history, local files, and crawled websites. It stores extracted content with offline r…

What would be a typical size of the search index, let's say after 5 years of intense browsing?

It depends on what you consider intense browsing. An indexed document uses about 100KB on average because Hister stores the full original HTML for offline previews. If storage is a concern, you can disable offline previews, reducing the average document size significantly.
Post reply on HN