Is there alternative to ES that scales nicely? I'm running ELK stack for logging using AWS Elasticsearch. Logs have unpredictable traffic volume and even overprovisioned ES cluster gets clogged sometimes. I wonder is there something more scalable than ES, and have nice GUI like Kibana?
Postgres Full-Text Search: A search engine in a database
111–120 of 141 posts
Re: Postgres Full-Text Search: A search engine in a database
#112Earlier quoted context omitted.
This is the key one for us that makes Postgres a non-starter for FTS (We use postgres for everything else ) We begrudgingly use Solr instead (we started before ES was really a thing and haven't found a need to switch yet) When you get more than about two different types of filters (e.g. types of filters could be 'tags', 'categories', 'geotags', 'media type', 'author' etc), the combinatorial explosion of Postgres quer…
This is exactly the issue I’m currently facing. We do a bunch of count queries to calculate facets and am looking for something that can do this out of the box. I’m glad I came to the same conclusion myself, either solr or elasticsearch might be the way to go. Starting this from scratch, which of the two would you recommend and why?
I myself is currently setting up Solr for a project as my experience with ES as a DevOps is not a happy one. Always nodes/indexes having some kind of problem. In the same time i have also worked with Solr for years before and never met any major issues. It just works and does the job well.
Re: Postgres Full-Text Search: A search engine in a database
#113Earlier quoted context omitted.
This is the key one for us that makes Postgres a non-starter for FTS (We use postgres for everything else ) We begrudgingly use Solr instead (we started before ES was really a thing and haven't found a need to switch yet) When you get more than about two different types of filters (e.g. types of filters could be 'tags', 'categories', 'geotags', 'media type', 'author' etc), the combinatorial explosion of Postgres quer…
This is exactly the issue I’m currently facing. We do a bunch of count queries to calculate facets and am looking for something that can do this out of the box. I’m glad I came to the same conclusion myself, either solr or elasticsearch might be the way to go. Starting this from scratch, which of the two would you recommend and why?
Unfortunately I don't know enough about elasticsearch to provide a useful comparison. I like the way ES queries are structured JSON instead of obscure compact querystring parameters, but that's not a good reason to choose one over the other :)
Re: Postgres Full-Text Search: A search engine in a database
#114Is there alternative to ES that scales nicely? I'm running ELK stack for logging using AWS Elasticsearch. Logs have unpredictable traffic volume and even overprovisioned ES cluster gets clogged sometimes. I wonder is there something more scalable than ES, and have nice GUI like Kibana?
loki
Re: Postgres Full-Text Search: A search engine in a database
#115Something that's missing from this which I'm curious about is how far can't postgres search take you? That is, what tends to be the "killer feature" that makes teams groan and set up Elasticsearch because you just can't do it in Postgres and your business needs it? Having dealt with ES, I'd really like to avoid the operational burden if possible, but I wouldn't want to choose an intermediary solution without being ab…
Used to work on Google Search, used ES extensively for a startup I founded (which was sort of quasi-search...it was built around feed ranking, where the query is constant and a stream of documents is constantly coming in), and have also used Postgres extensively in other companies. The big problem with all the off-the-shelf search solutions (RDBMS full-text search, ES, Algolia) is that search ranking is a complicated…
> Google's big insight is that how other people talk about a website is more important than how the website talks about itself, and its ranking algorithm weights accordingly.
I'm having trouble believing that seeing how top results on opinionated keywords are all SEO spam of websites no one visits by themselves.
Re: Postgres Full-Text Search: A search engine in a database
#116Earlier quoted context omitted.
Used to work on Google Search, used ES extensively for a startup I founded (which was sort of quasi-search...it was built around feed ranking, where the query is constant and a stream of documents is constantly coming in), and have also used Postgres extensively in other companies. The big problem with all the off-the-shelf search solutions (RDBMS full-text search, ES, Algolia) is that search ranking is a complicated…
> Google's big insight is that how other people talk about a website is more important than how the website talks about itself, and its ranking algorithm weights accordingly. I'm having trouble believing that seeing how top results on opinionated keywords are all SEO spam of websites no one visits by themselves.
I remember when Google came into being and how amazed I was. It was quite something totally different and I switched and never looked back for a lot of years.
In recent years however I tend to massively agree with your sentiment and experience. Every day I do not find the things that are really helpful on page 1 - 3, sadly.
Re: Postgres Full-Text Search: A search engine in a database
#117Huh, just yesterday I blogged[0] about using FTS in SQLite[1] to search my PDF database. SQLite's full-text search is really excellent. The thing that tripped me up for a while was `GROUP BY` with the `snippet`/`highlight` function but that's the point of the blog post. [0] https://jcuenod.github.io/bibletech/2021/07/26/full-text-sea... [1] https://www.sqlite.org/fts5.html
We've been using SQLite's FTS capabilities to index customer log files since 2017 or so. It's been a wonderful approach for us. Even if we move to our own in-house data store (event sourced log), we would still continue using SQLite for tracing because it brings so many of these sorts of benefits.
Re: Postgres Full-Text Search: A search engine in a database
#118Earlier quoted context omitted.
Used to work on Google Search, used ES extensively for a startup I founded (which was sort of quasi-search...it was built around feed ranking, where the query is constant and a stream of documents is constantly coming in), and have also used Postgres extensively in other companies. The big problem with all the off-the-shelf search solutions (RDBMS full-text search, ES, Algolia) is that search ranking is a complicated…
> Google's big insight is that how other people talk about a website is more important than how the website talks about itself, and its ranking algorithm weights accordingly. I'm having trouble believing that seeing how top results on opinionated keywords are all SEO spam of websites no one visits by themselves.
You don't remember what a game-changer Google was back in 2000 or so. Its results are pretty awful now, probably because of Goodhart's Law, but they were such a huge improvement over Jeeves et al..
Re: Postgres Full-Text Search: A search engine in a database
#119Is there alternative to ES that scales nicely? I'm running ELK stack for logging using AWS Elasticsearch. Logs have unpredictable traffic volume and even overprovisioned ES cluster gets clogged sometimes. I wonder is there something more scalable than ES, and have nice GUI like Kibana?
It's more a matter of configuring it right. I'd recommend trying out Elastic Cloud. It's a bit easier to deal with than Amazon's offering and much better supported. AWS has always been a bit hands-off on that front. Their opensearch project does not seem to break that pattern so far.
Also, with Elastic Cloud you get some access to useful features for logging (like life cycle management and data streams) that will help you scale the setup.
Kibana in recent iterations has actually improved quite a bit. The version you are getting from Amazon is probably a bit bare bones in comparison. One nice thing with Elastic is that going with the defaults gets you some useful dashboards out of the box if you use e.g. file or docker beats for collecting logs.
Re: Postgres Full-Text Search: A search engine in a database
#120I used Postgres full-text search for mail log feature on my email forward app https://hanami.run
Essentially allow arbitraty query in from/to/subject/body. One thing that make full-text serch work great for me is that I don't need to sort or rank the relevant of query. I just show a list of email that match the query order by their id.
I also don't do pagination and counting, instead users has to load more paged and the ID of the email is pass to the query as a point to compare( where id And with those strategy, full text search works great for us since we don't really want to bring in ElasticSearch because only about 20% of users use this features.