Interesting although a 15 second response time on certain queries is not a very good user experience.
Searching the web for under $1000/month
31–40 of 153 posts
Re: Searching the web for under $1000/month
#32It is a cool project. S3 can be cost efficient, but only if you don't touch data :)
Their price calculation doesn't mention cost of S3 requests, which very quickly adds up and is often neglected.
It costs $1 for 2.5M GET requests to S3. They have 180 shards, in a general case query seems to fetch all of them. Presumably they don't download full shard per request, but download an index + some relevant ranges. Lets say that is 10 requests per shard. So that would be 1800 S3 GET requests = ~1400 search queries cost them $1.
Assuming their service is reasonably popular and serve 1 req/second on average, that would be $1,440 per 30 days in addition to advertised $1,000 spent on EC2 and S3 storage.
Seems comparable to AWS ElasticSearch service costs:
- 3 nodes m5.2xlarge.elasticsearch = $1,200
- 20TB EBS storage = $1,638
Re: Searching the web for under $1000/month
#33What an amazing project, good luck to you guys and thanks for sharing.
Re: Searching the web for under $1000/month
#34If you're going for low cost, you could do better: https://www.hetzner.com/dedicated-rootserver/dell/dx181/conf... Basic configuration in Finland 1 224,91 € 1.92 TB SATA SSD Datacenter Edition 4 95,20 € 320,11 € 320 Euro equals 385.90 United States Dollar
Depending on the requirements https://www.hetzner.com/dedicated-rootserver/ax101/ May be an actually better fit Once they available again
Holy smokes. 8TB SSD + 128GB RAM + Ryzen 9 for 100 euro a month.
Can you get anywhere close to this with AWS or even DO?
Re: Searching the web for under $1000/month
#35Earlier quoted context omitted.
Depending on the requirements https://www.hetzner.com/dedicated-rootserver/ax101/ May be an actually better fit Once they available again
Holy smokes. 8TB SSD + 128GB RAM + Ryzen 9 for 100 euro a month. Can you get anywhere close to this with AWS or even DO?
Of course not. But that's why the "cloud" (as in the typical DO/AWS/Azure/GCP offerings) are a scam.
Re: Searching the web for under $1000/month
#36If you're going for low cost, you could do better: https://www.hetzner.com/dedicated-rootserver/dell/dx181/conf... Basic configuration in Finland 1 224,91 € 1.92 TB SATA SSD Datacenter Edition 4 95,20 € 320,11 € 320 Euro equals 385.90 United States Dollar
This is a complex misunderstanding...
First, we are getting better throughput from S3 than I we were using a SATA SSD. (and slower than a NVMe SSD). This is a bit of a secret.
Of course, single sequential throughput on S3 sucks. At the end of the day the data is stored on spining disk and we cannot do anything against the law physics.
... but we can concurrently read many disks using s3. Network is our only bottleneck. The theoretical upper bound on our instances is 2GB/s. On throughput intensive 1s query, we observe an average of 1GB/s.
Also you are not accounting for replication. S3 costs include battle tested, multi-DC replication.
Last but not least, S3 trivially decouples compute and storage. It means that we can host 100 different indices on S3, and use the same pool of search server to deal with the CPU-bound stuff.
This last bit is really what drives the price an extra 5x down for many use case.
Re: Searching the web for under $1000/month
#37Interesting although a 15 second response time on certain queries is not a very good user experience.
On the other hand, under 1.5 seconds on common / basic search terms is pretty good.
Re: Searching the web for under $1000/month
#38Interesting although a 15 second response time on certain queries is not a very good user experience.
This demo is not indeed quite misleading.
The high response time is due to the fact that we generate 18k snippets to generate the tag cloud. Imagine this is the equivalent of clicking on page 1 to 900 on Google!
A "barack obama" phrase query generating 20 snippets runs in less than 2seconds on our 2 cheap servers.
I'll set up a "normal 20 results search setting" next week and share it an API to show the latency again.
Re: Searching the web for under $1000/month
#39Is this reliant on S3 or can it be used on something like minio or digital ocean spaces or backblaze2 too? Backblaze to cloudflare data transfers is free so that can reduce costs a lot plus B2 is much cheaper than S3.
It can work on any object storages. I really want to test on Minio to see performance fly :)
Re: Searching the web for under $1000/month
#40Earlier quoted context omitted.
Holy smokes. 8TB SSD + 128GB RAM + Ryzen 9 for 100 euro a month. Can you get anywhere close to this with AWS or even DO?
Of course not. But that's why the "cloud" (as in the typical DO/AWS/Azure/GCP offerings) are a scam.
Just because you don't understand the value proposition of something doesn't make it a scam.