Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hister – A private, full content search index that you control

hister.org

Post reply on HN
Print view View original

71–80 of 110 posts

Re: Hister – A private, full content search index that you control

#71

Post by p-e-w »

asciimoo wrote: post #2

Ohi, author here! Thanks for posting Hister. Feel free to A.M.A. My first free software search project was Searx, a privacy respecting metasearch engine, but because of the limitations of the metasearch concept, I've decided to take a different approach. Hister builds a personal search index from pages you visit, bookmarks, browser history, local files, and crawled websites. It stores extracted content with offline r…

I used to be interested in the prophecies of Nostradamus a lifetime ago. You may be surprised to learn that “Hister” is a name mentioned in those prophecies that is commonly claimed by believers to refer to the future Adolf Hitler.

Academics usually take it to mean the Danube river instead, but in conspiracy/New Age contexts, the association with Hitler is prevalent, and mentioned in several pop culture works, including at least one feature film.

Re: Hister – A private, full content search index that you control

#72

Post by tolerance »

ebr4him wrote: post #51

Is the design vibe coded in Codex by any chance? Seeing a lot of similar designs out there and it’s really starting to annoy me.

I think that the tells, if they are in fact legitimate, are negligible at best or subtle at worst. It looks like the site uses Tailwind, so if AI was involved it wasn't generating the design from scratch. If it did then I'm impressed because usually it sucks. This is alright.

Re: Hister – A private, full content search index that you control

#73

Post by yreg »

tim-projects wrote: post #48
thataccount wrote: post #24

[flagged]

I didn't know this link. But, I immediately read Hitler when I saw it. Coming back days later... Nope brain still reads Hitler. It's a terrible choice of name

I too read it that way. I think that the problem is in the shape of the word being too similar.

Unlike say Hipster with the descender on the p.

Re: Hister – A private, full content search index that you control

#74

Post by mmargenot »

This is so cool! For the time being I’m still locked into notion for my handwritten knowledge base, but I love this for incorporating external information.

For the semantic search is there any chunking/processing that happens with the content or do you need to be diligent about having a large embedding context (and/or small content)?

Re: Hister – A private, full content search index that you control

#76

Post by leetrout »

Groxx wrote: post #6
asciimoo wrote: post #2

Ohi, author here! Thanks for posting Hister. Feel free to A.M.A. My first free software search project was Searx, a privacy respecting metasearch engine, but because of the limitations of the metasearch concept, I've decided to take a different approach. Hister builds a personal search index from pages you visit, bookmarks, browser history, local files, and crawled websites. It stores extracted content with offline r…

> Hister builds a personal search index from pages you visit, bookmarks, browser history, local files, and crawled websites. Immediately interested and will check it out, thank you! I've wanted a "search stuff you've seen online" tool for a long time, but everything seems to be research-oriented or "archive but don't search" or some weird combination that means it's nigh useless to me. I've got decades of bookmarks a…

Promnesia was a prior attempt at this

https://beepb00p.xyz/promnesia.html

Re: Hister – A private, full content search index that you control

#77

Post by human_llm »

asciimoo wrote: post #2

Ohi, author here! Thanks for posting Hister. Feel free to A.M.A. My first free software search project was Searx, a privacy respecting metasearch engine, but because of the limitations of the metasearch concept, I've decided to take a different approach. Hister builds a personal search index from pages you visit, bookmarks, browser history, local files, and crawled websites. It stores extracted content with offline r…

Thanks you for making Hister. I have been using it since May and find it immensely useful. It's a piece of mind to know that I always will be able to easily find what I looked at. So I don't feel the need to bookmark things, which I used to do, but then never looked at anyways ;).

Re: Hister – A private, full content search index that you control

#79

Post by idontneedcoffee »

asciimoo wrote: post #2

Ohi, author here! Thanks for posting Hister. Feel free to A.M.A. My first free software search project was Searx, a privacy respecting metasearch engine, but because of the limitations of the metasearch concept, I've decided to take a different approach. Hister builds a personal search index from pages you visit, bookmarks, browser history, local files, and crawled websites. It stores extracted content with offline r…

Thanks for your contribution to the Universe! :)

I'm hacking on a in-process DB on top of LMDB+Lance(for now, hilbert space kung-fu with a custom matryoshka embedding setup with separate spatial + temporal + internal and content derived anchors will replace that ~last-century~ last-year tech) + roaring bitmaps as the primary indexing engine with the same or similar goal[0] and will definitely deep-dive into yours.

I'm also trying to index users unstructured documents and workflows(tabs, emails, files, notes, identities etc)

- Organize them into semantically meaningful user or agent created context or directory-like virtual trees (the same photo of a nice kitchen may be surfaced under `/travel/barcelona` and `/arch/interieour/kitches`)

- ..where tree nodes are mapped to bitmaps - `/travel/barcelona` does a fast and cheap `travel` AND `barcelona`, want to "zoom-out" you just go one directory up to `/travel` and see all documents tagged with travel)

- You can use multiple timelines - extract that fancy md-converted en-wiki hf dataset into a wikipedia db dataset + timeline, tag your personal timeline as "personal" - wanna know the zeitgeist of your grandmothers birth date - search for it with timelines personal + wikipedia in layered mode and you'll get everything that happened or was happening during that time.

- You can have long-running stateful query sessions and refine your searches dynamically - search for "winter" and get all documents with a winter scenery or mentioning winter - refine with "nice view" then "laptop" - citing a recent example[1]

- Documents have relations that would be cumbersome to map in a virtual tree structure(worth an experiment due to the zoom-in/out you get with context bitmap trees though) - hence on top of the initial structure you can use graph edges(also powered by bitmaps - as most indexes are)

- All vector queries always run on top of a candidate set you get by the bitmap/bitmap-based filter algebra hence searching through 100k+ docs is usually pretty fast

Anyhow, let me stop here, thank you once again!

[0] https://github.com/canvas-ui/canvas-synapsd (sorry for the sloppy ai readme, no time to resurrect my old one with the updated APIs)

[1] https://demo.cnvs.ai/pub/c/aks6zaf8

Re: Hister – A private, full content search index that you control

#80

Post by vivzkestrel »

- lots of stupid questions to the author from a guy who has no idea about search engines

- let us say I want to index every blog ever listed on HN

- should be a small subset of the 400 billion pages out there on the internet no?

- First I need to gather data, what do you use to load so many webpages rapidly? asyncio with aiohttp in python? are there better options?

- how do you handle proxies? rotation? are there libraries you recommend for this?

- what about pages that use cloudflare? or block your request or present a captcha or a challenge of some kind?

- what are the filetypes you collect? only html or media as well?

- where and in what format do you store all these collected files? flat file storage? duckdb? postgres? hstore? something else?

- what is the frequency at which you refresh each page? once a day? once a week? something else?

- what kind of pre-processing do you use on the collected data? remove extra spaces? special characters? some kind of complex regex pipeline? LLM?

- how do you match the incoming query with processed data? simple text matching? regex? vector embedding match? something else?

Post reply on HN