Lucene is the carry.
Ask HN: Are Lucene/Solr/ES Still Used for Search?
151–160 of 223 posts
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#152I guess more and more people are turning towards search engines directly integrated into databases, like ArangoSearch of ArangoDB to combine search with other needs https://www.arangodb.com/why-arangodb/full-text-search-engin...
I assume you work for ArangoDB?
Looking at his/her post history, he/she does but rarely discloses it.
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#153Earlier quoted context omitted.
We host a smallish (12 large node) cluster ourselves on AWS, and it's been nothing but SUPER SOLID for us. Literally 0 issues in the year. We use it for analytics, aggregations and the like, as well as domain-specific search, for which it's a great fit.
I agree, if ES is scaled well for your usage it just keeps going. However, a big time sink for me has been reindexing, lost data due to bad mapping, and version incompatibilities.
I've found https://www.npmjs.com/package/elasticdump to be VERY useful for doing things like copying indices and recovering data. Much more than the built in stuff.
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#154Afaik almost everything runs Lucene under the hood, it's 20 years old, no one is going to build something as good any time soon. I suppose some company like Google have their own in house solution but otherwise it'll always be something built on top of Lucene. I guess you don't see much demand because for a lot of use cases the basic setups are good enough.
About seven years ago, I got a contracting gig for a website that wanted a "search engine". I remember thinking "Solr/Lucene is old, not pure-functional, and therefore awful!" and decided to build my own. Somehow I even managed to convince the client that this was a good idea. I ended up trying to reinvent Solr for the client, realizing after about two days of trying to reinvent stemming and indexing, that this was s…
[deleted]
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#155Earlier quoted context omitted.
> Elasticsearch, definitely. I always recommend using it in hosted form and not running your own cluster. That allows you to focus on getting data in and out of your cluster instead of sinking time into doing devops. I think everyone doing Elasticsearch well has to bring it in-house eventually. AWS's hosted solution is poor, Logz.io and ElasticCloud are expensive. There's a 7-figure/yr Elastic Cloud customer I work w…
> so tired of Elastic just randomly killing their clusters out of nowhere and having to spend basically triple to deal with it that they're bringing it all in house. That is because Elastic Cloud is not a fully managed Elasticsearch. People often don't get that with Elastic Cloud you are still responsible for your ES cluster. That's one of the differentiators that e.g. Sematext has (disclaimer: founder). > AWS's host…
Why are all ELK stack based logging solutions so much more expensive than custom rolled solutions like logdna, datadog, papertrail,etc.
The per gb cost on the other ones start at 1.20 $/gb and goes till 2$/gb. While almost all hosted ELK solutions start off at 3$/gb.
Im asking because i would very much like to adopt an ELK based hosted solution..but I'm not able to justify paying double. Is it that running+resource costs for ELK are so high that the extra charge needs to happen ?
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#156Elasticsearch, definitely. I always recommend using it in hosted form and not running your own cluster. That allows you to focus on getting data in and out of your cluster instead of sinking time into doing devops. While I have not used Solr recently, it has evolved along with ES as a solid product with a solid community. Nothing against it; it's a solid choice and there are probably people offering to host that as w…
> Elasticsearch, definitely. I always recommend using it in hosted form and not running your own cluster. That allows you to focus on getting data in and out of your cluster instead of sinking time into doing devops. I think everyone doing Elasticsearch well has to bring it in-house eventually. AWS's hosted solution is poor, Logz.io and ElasticCloud are expensive. There's a 7-figure/yr Elastic Cloud customer I work w…
Elastic Cloud is pretty reasonable IMHO. We have a logging cluster that costs us around 250 euro per month. Yes, running it ourselves would be cheaper but doing that in Amazon would eliminate most of the cost benefits. If you then consider devops cost needed to that, there is no difference. Most companies getting started with Elasticsearch should probably not do that on day 1. You can always start doing this later if there's some reason to.
I've never had clusters being killed randomly but I did have a few self inflicted issues with cluster instability due to flooding the cluster with too much data and not having suitable data retention policies in place. With the recently added index life cycle management (an x-pack feature), this is easier to manage these days.
If you are spending seven figures a year on elastic search, you clearly are not a beginning user and there's some cost savings that you might be able to realize by taking ownership of the problem of hosting it somewhere cheaper. For that, their recently open sourced kubernetes helm charts are worth a look. Those scripts take care of a lot of things and get you a self hosted version of Elastic Cloud.
Amazon hosted clusters are indeed a bit bare-bones (as is their support for these clusters) and I would also not recommend them; you get more value for money by using Elastic Cloud.
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#157Earlier quoted context omitted.
> AWS's hosted solution is poor What has been poor about it in your experience?
Not the GP, but you aren't allowed to touch settings like the shard recovery rate. If a configuration change (changing # of nodes, instance type, etc) goes wrong, your cluster indefinitely gets stuck in Processing due to a race condition. The only way to get unstuck is to file a ticket. The company I'm at doesn't pay for AWS support, so at one point we ended up completely tearing down our cluster and rebuilding a new…
Check out the recently open sourced kubernetes scripts: https://www.elastic.co/elasticsearch-kubernetes. There's no need to reinvent all of that.
ES is indeed super solid if you know what you are doing. However, most users getting started with this are probably going to find out a few things the hard way though; which is why I recommend hosted solutions as it removes quite a bit of non trivial devops from the equation.
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#158Earlier quoted context omitted.
> so tired of Elastic just randomly killing their clusters out of nowhere and having to spend basically triple to deal with it that they're bringing it all in house. That is because Elastic Cloud is not a fully managed Elasticsearch. People often don't get that with Elastic Cloud you are still responsible for your ES cluster. That's one of the differentiators that e.g. Sematext has (disclaimer: founder). > AWS's host…
Why are all ELK stack based logging solutions so much more expensive than custom rolled solutions like logdna, datadog, papertrail,etc. The per gb cost on the other ones start at 1.20 $/gb and goes till 2$/gb. While almost all hosted ELK solutions start off at 3$/gb. Im asking because i would very much like to adopt an ELK based hosted solution..but I'm not able to justify paying double. Is it that running+resource c…
Datadog, etc are not cheap when you're paying per agent/per month to ship your logs in the first place. Which they want you to do, of course.
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#159Very much so. For retail/catalog search SOLR dominates. There's a lot more customization available for relevancy/ranking OOB than Elastic. Drawbacks are managing indexing - SOLR cloud is much harder to manage. For commodity search workloads (general retrieval/faceting) Elastic does a fine job. It scales well and there is good documentation and support. Lucene is the core engine behind both of these solutions. For fun…
>"For retail/catalog search SOLR dominates." Interesting, could you elaborate on why SOLR is dominant in that space over say Elasticsearch?
I think a big reason is simply that Solr has been around a few years longer and most older sites would have been using that by the time Elasticsearch came along and never saw a good reason to go through a painful process of replacing one with the other.
These days, I would say there's very little that either product does that the other product can't do though obviously there are lots of strengths and weaknesses on both sides.
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#160Earlier quoted context omitted.
Why are all ELK stack based logging solutions so much more expensive than custom rolled solutions like logdna, datadog, papertrail,etc. The per gb cost on the other ones start at 1.20 $/gb and goes till 2$/gb. While almost all hosted ELK solutions start off at 3$/gb. Im asking because i would very much like to adopt an ELK based hosted solution..but I'm not able to justify paying double. Is it that running+resource c…
Datadog, etc are not cheap when you're paying per agent/per month to ship your logs in the first place. Which they want you to do, of course.
The pricing is not per agent per month. The pricing for everyone is always either per million events or per gb.
You do not need an agent. You can ship directly from syslog ( https://docs.datadoghq.com/integrations/rsyslog/?tab=datadog...)