41–50 of 153 posts
Interesting although a 15 second response time on certain queries is not a very good user experience.
On the other hand, under 1.5 seconds on common / basic search terms is pretty good.
It is a cool project. S3 can be cost efficient, but only if you don't touch data :) Their price calculation doesn't mention cost of S3 requests, which very quickly adds up and is often neglected. It costs $1 for 2.5M GET requests to S3. They have 180 shards, in a general case query seems to fetch all of them. Presumably they don't download full shard per request, but download an index + some relevant ranges. Lets say…
For that kind of use case, I'd probably start using minio.
> Seems comparable to AWS ElasticSearch service costs: > - 3 nodes m5.2xlarge.elasticsearch = $1,200 > - 20TB EBS storage = $1,638
Don't forget S3 includes replication. Also EBS throughput (even with SSD) is not good at all. Also our memory footprint is tiny. This is necessary to make it run on two servers.
Finally, cpu-wise, our search engine is almost 2x faster than lucene.
If you don't believe us, try to replicate our demo on an elastic search :D.
Chatnoir.eu is the only other common crawl cluster we know of. It consists of 120 nodes.
Earlier quoted context omitted.
Depending on the requirements https://www.hetzner.com/dedicated-rootserver/ax101/ May be an actually better fit Once they available again
Holy smokes. 8TB SSD + 128GB RAM + Ryzen 9 for 100 euro a month. Can you get anywhere close to this with AWS or even DO?
If you're going for low cost, you could do better: https://www.hetzner.com/dedicated-rootserver/dell/dx181/conf... Basic configuration in Finland 1 224,91 € 1.92 TB SATA SSD Datacenter Edition 4 95,20 € 320,11 € 320 Euro equals 385.90 United States Dollar
This is a complex misunderstanding... First, we are getting better throughput from S3 than I we were using a SATA SSD. (and slower than a NVMe SSD). This is a bit of a secret. Of course, single sequential throughput on S3 sucks. At the end of the day the data is stored on spining disk and we cannot do anything against the law physics. ... but we can concurrently read many disks using s3. Network is our only bottlenec…
Sometimes we pay a bit too much for this multi-replication, battle tested stuff. It's not like the probability of loosing data is THAT huge. For the 4x extra cost you could easily take a backup every 24h.
"It means that we can host 100 different indices on S3, and use the same pool of search server to deal with the CPU-bound stuff"
You can do that with NFS.
It's amazing how much we are willing to pay for a bunch of computers in the cloud. Leasing a new car costs around $350/month. You could have three new cars at your disposal for the same price as this search implementation.
Earlier quoted context omitted.
We really need to make this clear in our next blog post. This is not grep here. We are using the same datastructure that are used in Elasticsearch or google. We just adapted them to be object storage friendly. I would not call Object Storage dumb by any mean. They are a very powerful bottom-up abstraction. We do manage to get SSD-like throughput from them. The latency is the big issue. We had to redesign our search t…
Appreciate the response. I wasn't trying to say this is grep, I fully understand that this is an inverted index which is way more interesting to build on top of S3. I merely wanted to say that by using S3 within AWS you always have the fallback option of brute-force "grep" across your semi-structured "data lake" or whatever it's called thanks to the aggregate bandwidth and Athena.
This does decouple compute and storage in a trivial manner. There is indeed a realm in which this brute force approach is the best approach.
We could probably make a 4D chart with QPS, data size, latency, and retention period and define regions where the elastic/SOLR approach, Humio, and quickwit are the most relevant.
It is a cool project. S3 can be cost efficient, but only if you don't touch data :) Their price calculation doesn't mention cost of S3 requests, which very quickly adds up and is often neglected. It costs $1 for 2.5M GET requests to S3. They have 180 shards, in a general case query seems to fetch all of them. Presumably they don't download full shard per request, but download an index + some relevant ranges. Lets say…
If you're going for low cost, you could do better: https://www.hetzner.com/dedicated-rootserver/dell/dx181/conf... Basic configuration in Finland 1 224,91 € 1.92 TB SATA SSD Datacenter Edition 4 95,20 € 320,11 € 320 Euro equals 385.90 United States Dollar
Depending on the requirements https://www.hetzner.com/dedicated-rootserver/ax101/ May be an actually better fit Once they available again
If you're going for low cost, you could do better: https://www.hetzner.com/dedicated-rootserver/dell/dx181/conf... Basic configuration in Finland 1 224,91 € 1.92 TB SATA SSD Datacenter Edition 4 95,20 € 320,11 € 320 Euro equals 385.90 United States Dollar
Earlier quoted context omitted.
Depending on the requirements https://www.hetzner.com/dedicated-rootserver/ax101/ May be an actually better fit Once they available again
Holy smokes. 8TB SSD + 128GB RAM + Ryzen 9 for 100 euro a month. Can you get anywhere close to this with AWS or even DO?