51–60 of 61 posts
Earlier quoted context omitted.
Eh? The tokenizers are fully pluggable, the Unicode and ICU tokenizers will be good for many purposes and you can always drop something specific in. Eg https://github.com/wangfenjin/simple
I know it’s extensible, with original work. ICU isn’t helpful for some languages
Earlier quoted context omitted.
though useless for many non english languages
Of all/many opensource projects I looked at, only meilisearch is good for non-English languages: https://www.meilisearch.com/docs/learn/resources/language
For instance searching for ‘___missilestrike’ in Kyiv would ideally surface all incidents, regardless of launch mode (surface-to-surface, air-to-surface…), range, propulsion, warhead, or guidance system. In these languages a number of these would form the prefix of that compound word.
Does anyone have successfully worked with Non-English text with FTS5 in Sqlite? I could not find any reference for German, e.g. and the default stemming does not seem to work properly (given some short tests).
Should work well for German, I’m using it with Nordic languages.
We are running FTS5 with a bunch of sqlite databases for an internal search tool, one with 15m records (we sharded that one, one shard per core), others with 200k-2m records. Speed is very fast -- the biggest dbs are hundreds of ms response time, but others are sub 10 or tens of ms. By _far_ the simplest search option for a small organization where a lot of the data is already in a SQL db, and we don't have the resou…
Do you have to do anything special to guarantee each sqlite instance executes in parallel with the others on its own core? Does the language you’re calling them from support concurrency or parallelism?
Earlier quoted context omitted.
How’d it go?
GRDB on iOS and macOS It’s easy and works well It is slower than Realm in some benchmarking (I forget if it’s reads or writes) and might be less memory friendly (Realm does lazy evaluation). You need to use some third party open source to make sqlite faster by precompiling model definitions, which I didn’t bother with. I currently only use it for fts5 and Realm for other needs, but would like to try using it for more…
Earlier quoted context omitted.
GRDB on iOS and macOS It’s easy and works well It is slower than Realm in some benchmarking (I forget if it’s reads or writes) and might be less memory friendly (Realm does lazy evaluation). You need to use some third party open source to make sqlite faster by precompiling model definitions, which I didn’t bother with. I currently only use it for fts5 and Realm for other needs, but would like to try using it for more…
Thanks for the input! Did you need to ship your own binaries of sqlite that include fts5 or does skip do that for you?
We are running FTS5 with a bunch of sqlite databases for an internal search tool, one with 15m records (we sharded that one, one shard per core), others with 200k-2m records. Speed is very fast -- the biggest dbs are hundreds of ms response time, but others are sub 10 or tens of ms. By _far_ the simplest search option for a small organization where a lot of the data is already in a SQL db, and we don't have the resou…
I’m interested in the sharding approach myself but those hundreds of ms sound a bit slow. Do you have to do anything special to guarantee each sqlite instance executes in parallel with the others on its own core? Does the language you’re calling them from support concurrency or parallelism?
- 15m row db will return a search for an extremely common term in ~3s (9m rows have that term);
- When adding terms, so that only ~200k rows have the combination, response is ~2s
- That 15m db was in 16 shards. (I’m actually not sure why it should be so slow; doesn’t actually make sense given that a single 1m row db is not that slow, might look into it but not a huge priority)
- For a ~200k row db, search is ~150ms for three terms that appear in ~1.5k entries
- Obviously the network io is the slower piece anyway
This perf is fine for our use case — a research system for internal users. Certainly worth the trade-off of not having to deal with a more complex database like elasticsearch, or even PG with the tantivy extension (which we might switch to some day).
For sharding — we only shard the huge one. To make sure bm25 rankings are still _roughly_ the same as they would be if we did not shard, we just randomised the rows assigned to the num_core shards. It’s a 16 core machine so 16 shards.
We know/ensure each db file gets its own core because we use aioprocessing in python. It handles both multicore and async. Running a query with htop shows all the cores light up (unlike with unsharded).
It’s all about trade-offs in the end. We cache the first search then pre cache the next page (and so on) so the user only has to wait 2-3s when searching the big db for that initial search and after that it’s pretty snappy. Most searches are on much smaller dbs (thousands to hundreds of thousands of rows) and results there are often 10ms or something. No user complaints so far.
Earlier quoted context omitted.
Wait, why they revoked git access?
Someone unexpectedly ingesting all git repos at once would look a lot like a compromised workstation being used to harvest company data. I assume everything was fine once they explained what was going on.