If you want to index HTML documents using SQLite FTS5, I have a HTML (pseudo) tokenizer plugin https://github.com/gyf304/sqlite3-fts5-html
SQLite FTS5 Extension
21–30 of 61 posts
Re: SQLite FTS5 Extension
#22FTS5 works great. The only issue I had is that the syntax for the queries is quite irregular. Had problems both with understanding it exactly and translating to it from the usual format of space delimited keywords, exact strings in quotes, etc.
I wish it would have an API instead where I could put the parts of the query exactly (as nodes in a tree) without the need to translate to this irregular syntax.
Re: SQLite FTS5 Extension
#23This is hugely underrated in my opinion: it’s a very competent search engine. It also ships as part of the Python standard library, so if your machine has Python installed you have a high quality search engine ready to use without installing anything else. I have a CLI tool (and Python library) for working with it here: https://sqlite-utils.datasette.io/en/stable/cli.html#configu...
Shh, it's a secret, don't tell people.
Re: SQLite FTS5 Extension
#24Earlier quoted context omitted.
Not by default, no. But you could kind of implement it by providing a custom tokenizer that emits multiple terms for the same position in the document, with different variants of the same token. This would not be "proper" fuzzy search, but might be enough depending on the use case. See https://www.sqlite.org/fts5.html#synonym_support for more details on the different approaches for implementing synonyms in custom tok…
I don't think that will work for me, since I needed something that can handle mistakes in the words, like Du'ha to duha etc, Rahman to rehman, basically whatever looks closest.
One thing you could do: FTS5 has the `fts5vocab` virtual table [1] that has all the terms. You could provide a user-defined function that computes the levenshtein distance between your query terms and the terms in that table, obtain candidate terms that way and build a big query that searches for all those lexically close terms.
[1] https://www.sqlite.org/fts5.html#the_fts5vocab_virtual_table...
Re: SQLite FTS5 Extension
#25It's a very useful feature of sqlite and it also works great in-browser using wa-sqlite[0]. Example, if anyone's curious [1]. [0]: https://github.com/rhashimoto/wa-sqlite [1]: https://github.com/iansinnott/prompta/blob/master/src/lib/mi...
I've been trying something similar, but using https://github.com/mmomtchev/sqlite-wasm-http to stream the database over http for a SPA without a backend. It's actually able to do searches (for queries that aren't super short) without downloading the entire FTS table.
Re: SQLite FTS5 Extension
#26I once tried to use sql.js [1] on a static site for full text search. It worked, but the resulting database size for that site was too large for the web, even with things like detail=none and content='' applied, and requiring the user to download a database each time was just no go. (I guess things should work better for sites with less content or those not requiring a trigram tokenizer.)
I switched to Pagefind [2] afterwards before finding out a sql.js-httpvfs [3] fork of sql.js that removes exactly the need to fully download a database (with HTTP range requests). I haven't got the chance to test sql.js-httpvfs out though, but it looks pretty sound and could be much more flexible than Pagefind. (Previously discussed at https://news.ycombinator.com/item?id=27016630 .)
Re: SQLite FTS5 Extension
#27This is hugely underrated in my opinion: it’s a very competent search engine. It also ships as part of the Python standard library, so if your machine has Python installed you have a high quality search engine ready to use without installing anything else. I have a CLI tool (and Python library) for working with it here: https://sqlite-utils.datasette.io/en/stable/cli.html#configu...
I've got half of my mind to use this as a search for legal documents and "pretend" it's an LLM.
Re: SQLite FTS5 Extension
#28Re: SQLite FTS5 Extension
#29Using this for our new Svelte community website. Very pleased with it so far.