Earlier quoted context omitted.
Any tips for scaling Postgres-only fulltext search?
https://github.com/postgrespro/rum
Good find, bookmarked!
Ask HN: Are Lucene/Solr/ES Still Used for Search?
211–220 of 223 posts
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#212Earlier quoted context omitted.
Do you use ts_rank? PostgreSQL FTS is very efficient until you want to rank the results according to their relevance. This is because the data necessary to the ranking are not in the GIN or GIST index. They are in the heap, and this triggers a lot of random IOs.
Ah, this is good to know. My site doesn't yet need to scale, so this is definitely A Problem I Would Love To Have ;) EDIT: This seems to help with the ranking problem: https://github.com/postgrespro/rum
Yes, RUM is great. I'd hope it will be built-in one day.
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#213Earlier quoted context omitted.
I've looked at Vespa a bit. It looks pretty good. It's also readily apparent that it's an ancient system that's grown out of an in-house project, and that its design has accrued a lot of oddities over the years from lack of careful, co-ordinated design. It includes a bunch of esoteric features (like the "predicate" function) that have obviously grown out of Yahoo's own internal architecture. One curiosity is Vespa's…
Just an FYI to your last paragraph: The core indexing/ranking/storage components of Vespa are C++, and run in a separate process (no jni). In my own attempt to compare the two, I found the memory consumption of Vespa was easier to predict and understand (there are formulas for it in the documentation).
Thanks, I didn't know that!
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#214From my experience, yes. Lucene is a "production" state of art library and Solr/Elasticsearch is very used in many scenarios. This expertise is very on demand. My company personally migrated from ElasticSearch to https://vespa.ai/ and could not be happier. Faster and way easier to maintain a cluster. The "Application Packages" feature present in Vespa opened many opportunities to improve our product( Curiously we use…
I've looked at Vespa a bit. It looks pretty good. It's also readily apparent that it's an ancient system that's grown out of an in-house project, and that its design has accrued a lot of oddities over the years from lack of careful, co-ordinated design. It includes a bunch of esoteric features (like the "predicate" function) that have obviously grown out of Yahoo's own internal architecture. One curiosity is Vespa's…
Predicate fields are indeed an oddity, but not an architectural one - it's for situations where the documents need to specify criteria (predicates) for when they should match - like only match for certain users, certain times of day etc. It's probably an underused feature imho since most people don't know this can be done efficiently.
If you have dynamic fields like in your SaaS example I recommend using a single map field rather than let data not under your control drive changes to the set of fields.
> Do you know how long a package update takes in Vespa, to add, say, a single field?
A few seconds. However, rather than having operators do any of this manually, set up an automatic process which deploys on each change made to the repo (i.e do CD).
> all the reference documentation in one place
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#215Are people still using Sphinx Search ( http://sphinxsearch.com ) at all? It doesn't seem like it gets many releases anymore...since they unpublished the source code, it's hard to see how much activity there is.
https://manticoresearch.com/ is the lively, open source, fork of Sphinxsearch. that's where some of the earlier developers from the project moved to. it's used as a text-search backend on craigslist.
this is cool! Definitely will give it a look.
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#216We have been using https://www.algolia.com/ completely as a replacement for ES. Pros: - Managed search engine - Great API / Developer experience Cons: - Cloud only makes it hard for local development - Expensive (I guess it depends on the usage)
It is a similar cost to elastic/solr cloud options, cheaper if you have to get to feature parity with Algolia.
Oh that's a good point. I was referring to Algolia vs self hosted ES and that's why I pointed out that it depends on the usage (how much power you need and how many people to maintain it).
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#217Earlier quoted context omitted.
I've looked at Vespa a bit. It looks pretty good. It's also readily apparent that it's an ancient system that's grown out of an in-house project, and that its design has accrued a lot of oddities over the years from lack of careful, co-ordinated design. It includes a bunch of esoteric features (like the "predicate" function) that have obviously grown out of Yahoo's own internal architecture. One curiosity is Vespa's…
Predicate fields are indeed an oddity, but not an architectural one - it's for situations where the documents need to specify criteria (predicates) for when they should match - like only match for certain users, certain times of day etc. It's probably an underused feature imho since most people don't know this can be done efficiently. If you have dynamic fields like in your SaaS example I recommend using a single map…
Thanks. I'm still learning about Vespa, and it's still not clear how map fields work.
Edit: Documentation says: "Accessing attributes in maps and arrays of struct in ranking is not possible". So maps aren't really usable.
Regarding how long it takes to update a field, the application I described would have to do this programmatically. It would have to keep track of all known fields in some kind of registry, and then if a new unknown field came in, it would have to perform an "application package" deploy just for that field, using the REST API. (Unless there's a less cumbersome way to do it?)
Reference docs: That's nice, but that's just a bunch of links. Good reference documentation has tables of contents. Bonus points for runnable examples in multiple languages. For an example of good reference API documentation, look at Stripe's [1].
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#218Earlier quoted context omitted.
I've looked at Vespa a bit. It looks pretty good. It's also readily apparent that it's an ancient system that's grown out of an in-house project, and that its design has accrued a lot of oddities over the years from lack of careful, co-ordinated design. It includes a bunch of esoteric features (like the "predicate" function) that have obviously grown out of Yahoo's own internal architecture. One curiosity is Vespa's…
Predicate fields are indeed an oddity, but not an architectural one - it's for situations where the documents need to specify criteria (predicates) for when they should match - like only match for certain users, certain times of day etc. It's probably an underused feature imho since most people don't know this can be done efficiently. If you have dynamic fields like in your SaaS example I recommend using a single map…
Another thing is that Vespa doesn't seem to support indexing of nested data, either structs or arrays of structs. For example:
{
"location": {
"city": "Washington",
"state: "District of Columbia"
},
"friends": [
{"firstName: "Bill", "lastName": "Clinton"}
]
}
An application's search module could flatten the location key (e.g. "location_city", "location_state") for simple attributes, but the same is not possible for the array, since there can be arbitrary array elements. And you can't split it to an array of strings:
"friends_firstName_elems": ["Bill"]
"friends_lastName_elems": ["Clinton"]
This seems unnecessarily restrictive. A search engine should be able to index the data you already have, not force the application to contort its data to whatever shape the engine requires.
Is there no way around this?
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#219Nobody seems to have mentioned that GitHub uses ElasticSearch.
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#220AFAIK, Reddit, Slack, Dice, Bloomberg, IBM, Apple all use Solr. Jira and Confluence use Lucene. Others use Elasticsearch and Fusion (commercial product on top of Solr). See, for example: https://www.activate-conf.com/more-events for presentations from several past years on who and how uses Lucene/Solr. Also, the new trend in jobs is "Relevancy Engineering", which is less about just setting up search engines and more…
Until about 2014-2015, many large companies wouldn't have looked twice at Elasticsearch. The companies you list using Solr have been invested in search for 10 years or longer (predating Elasticsearch), and may have high switching costs.
For sure they would have high switching costs. They would be switching from an open-source (and free product) to which they contributed changes to a commercial product. License alone would be a serious discussion point. So, that's a fact.
Was there an opinion in there as well that you tried to convey?