Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Searching the web for under $1000/month

quickwit.io

Post reply on HN
Print view View original

61–70 of 153 posts

Re: Searching the web for under $1000/month

#61

Post by fulmicoton »

2Gkashmiri wrote: post #54
fulmicoton wrote: post #36

Earlier quoted context omitted.

This is a complex misunderstanding... First, we are getting better throughput from S3 than I we were using a SATA SSD. (and slower than a NVMe SSD). This is a bit of a secret. Of course, single sequential throughput on S3 sucks. At the end of the day the data is stored on spining disk and we cannot do anything against the law physics. ... but we can concurrently read many disks using s3. Network is our only bottlenec…

How about setting up minio on these hertzner setups? You get benefit of s3 on cheap hardware without aws costs

Absolutely! I want to try that.. We are especially interested in testing the latency minio could offer.

Re: Searching the web for under $1000/month

#62

Post by Retric »

marcinzm wrote: post #40
kuschku wrote: post #35

Earlier quoted context omitted.

Of course not. But that's why the "cloud" (as in the typical DO/AWS/Azure/GCP offerings) are a scam.

Just because you don't understand the value proposition of something doesn't make it a scam.

AWS is a scam not because it can’t save you money, but because they actively try to trick you into spending more money. That’s practically the definition of a scam.

Go to the AWS console and try to answer even simply things like how much did the last hour/day/week cost me? Or how about some notifications if that new service you just added is going to cost vastly more than you where expecting.

I know of a few people getting fired after migrating to AWS and it’s not because the company was suddenly saving money.

Re: Searching the web for under $1000/month

#63

Post by rossmohax »

fulmicoton wrote: post #42
nopurpose wrote: post #32

It is a cool project. S3 can be cost efficient, but only if you don't touch data :) Their price calculation doesn't mention cost of S3 requests, which very quickly adds up and is often neglected. It costs $1 for 2.5M GET requests to S3. They have 180 shards, in a general case query seems to fetch all of them. Presumably they don't download full shard per request, but download an index + some relevant ranges. Lets say…

I tend to agree :). If we get 1 req/s, even for a dataset of that size, this is not as cost efficient. For that kind of use case, I'd probably start using minio. > Seems comparable to AWS ElasticSearch service costs: > - 3 nodes m5.2xlarge.elasticsearch = $1,200 > - 20TB EBS storage = $1,638 Don't forget S3 includes replication. Also EBS throughput (even with SSD) is not good at all. Also our memory footprint is tiny…

> If we get 1 req/s, even for a dataset of that size, this is not as cost efficient.

How many req/s do you have in mind for your system to be a viable option?

> Also EBS throughput (even with SSD) is not good at all.

It is not worse than S3 still, right?

> Chatnoir.eu is the only other common crawl cluster we know of. It consists of 120 nodes.

I have no deep ES experience. Are you saying, that to host 6TB of indexed data (before replication) you'd need 120 nodes ES cluster? If so, then reducing it to just 2 nodes is the real sales pitch, not S3 usage :)

Re: Searching the web for under $1000/month

#64

Post by rossmohax »

fulmicoton wrote: post #61
2Gkashmiri wrote: post #54

Earlier quoted context omitted.

How about setting up minio on these hertzner setups? You get benefit of s3 on cheap hardware without aws costs

Absolutely! I want to try that.. We are especially interested in testing the latency minio could offer.

I've heard minio metadata handling isn't great,it queries all servers. SeaweedFS might give you a better results.

Re: Searching the web for under $1000/month

#65

Post by marcinzm »

Retric wrote: post #62
marcinzm wrote: post #40

Earlier quoted context omitted.

Just because you don't understand the value proposition of something doesn't make it a scam.

AWS is a scam not because it can’t save you money, but because they actively try to trick you into spending more money. That’s practically the definition of a scam. Go to the AWS console and try to answer even simply things like how much did the last hour/day/week cost me? Or how about some notifications if that new service you just added is going to cost vastly more than you where expecting. I know of a few people g…

AWs is pretty bad at telling you how much something you're not running will cost if you run it but I've never had any issues knowing what something has cost me in the past.

>Go to the AWS console and try to answer even simply things like how much did the last hour/day/week cost me?

Click user@account in top right, click My Billing Dashboard, spend this month is on that page in giant font, click Cost Explorer for more granular breakdown (day, service, etc.), click Bill Details for list breakdown of spend by month.

>Or how about some notifications if that new service you just added is going to cost vastly more than you where expecting.

Billing Dashboard and then Budgets.

edit: This assumes you have permissions to see billing details, by default non-root accounts do not which might be why you're confused.

Re: Searching the web for under $1000/month

#66

Post by curryst »

plater wrote: post #44
fulmicoton wrote: post #36

Earlier quoted context omitted.

This is a complex misunderstanding... First, we are getting better throughput from S3 than I we were using a SATA SSD. (and slower than a NVMe SSD). This is a bit of a secret. Of course, single sequential throughput on S3 sucks. At the end of the day the data is stored on spining disk and we cannot do anything against the law physics. ... but we can concurrently read many disks using s3. Network is our only bottlenec…

"S3 costs include battle tested, multi-DC replication." Sometimes we pay a bit too much for this multi-replication, battle tested stuff. It's not like the probability of loosing data is THAT huge. For the 4x extra cost you could easily take a backup every 24h. "It means that we can host 100 different indices on S3, and use the same pool of search server to deal with the CPU-bound stuff" You can do that with NFS. It's…

> For the 4x extra cost you could easily take a backup every 24h.

It's also worth considering the cost to simply regenerate the data for something like this that isn't the source of truth. You'll lose any content that you indexed that has disappeared from the web, but that seems like a feature more than a bug.

> You can do that with NFS.

You're going to be bound by your NIC speed. You can bond them together, but the upper bounds on NFS performance are going to be significantly lower than on S3. Whether that's going to be an issue for them or not, I don't know, but a big part of the reason for separating compute and storage is so that one of them can scale massively without the other.

Re: Searching the web for under $1000/month

#68

Post by thejosh »

natpat wrote: post #2

This is super interesting. I've recently also been working on a similar concept: we have a reasonable amount (in the terabytes) of data, that's fairly static, that I need to search fairly infrequently (but sometimes in bulk). A solution we came up with was a small , hot, in memory index, that points to the location of the data in a file on S3. Random access of a file on S3 is pretty fast, and running in an EC2 instan…

You might want to check out Snowflake for something like this, it makes searching pretty easy, especially as it seems your data is semi-static? We use it pretty extensively at work and it's great.

For your usecase it'll be very cheap if you don't access it constantly (you can probably get away with the extra small instances, which you are billed per minute).

Not affiliated in anyway, just a suggestion.

Re: Searching the web for under $1000/month

#69

Post by gallexme »

potiuper wrote: post #47
gallexme wrote: post #22

Earlier quoted context omitted.

Depending on the requirements https://www.hetzner.com/dedicated-rootserver/ax101/ May be an actually better fit Once they available again

Why a Ryzen instead of an Epyc in a data center?

Also cause the 5950x is likely for many workloads faster which do not linearly scale across more cores than a zen2 epyc (since zen3 has huge singlethread performance improvements)

Re: Searching the web for under $1000/month

#70

Post by klohto »

fulmicoton wrote: post #42
nopurpose wrote: post #32

It is a cool project. S3 can be cost efficient, but only if you don't touch data :) Their price calculation doesn't mention cost of S3 requests, which very quickly adds up and is often neglected. It costs $1 for 2.5M GET requests to S3. They have 180 shards, in a general case query seems to fetch all of them. Presumably they don't download full shard per request, but download an index + some relevant ranges. Lets say…

I tend to agree :). If we get 1 req/s, even for a dataset of that size, this is not as cost efficient. For that kind of use case, I'd probably start using minio. > Seems comparable to AWS ElasticSearch service costs: > - 3 nodes m5.2xlarge.elasticsearch = $1,200 > - 20TB EBS storage = $1,638 Don't forget S3 includes replication. Also EBS throughput (even with SSD) is not good at all. Also our memory footprint is tiny…

What about d3en instances? Clustered, and together with minio you might reach similar performance. Only issue is the inter-region traffic, it would need to be inside the same AZ

EDIT: Realizing that d3 has just slow HDD

Post reply on HN