Searching the web for under $1000/month • Hacker News Board

Re: Searching the web for under $1000/month

#151

Post by arcturus17 » Sat, May 08, 2021, 5:43 PM UTC

vagrantJin wrote: post #144

arcturus17 wrote: post #105

Earlier quoted context omitted.

> AWS is almost never cost efficient. A ridiculous blanket statement, despite the "almost never" cop-out... It is cost-efficient in a wide array of scenarios. Many companies pay for it because they have calculated the different investment scenarios and AWS comes on top of alternatives such as owning the hardware or using competing cloud vendors. I own a consultancy that builds complex web apps and while I appreciate…

> I consider it cheaper in the long run. Really?

Yes, I charge $60-$90 per hour of dev time to my customers and the time saved from using simple Elastic Beanstalk deployments pays for itself in saved dev time. The architecture is also infinitely easier to reason about and scale than cramming parts in a single image.

Re: Searching the web for under $1000/month

#152

Post by kuschku » Mon, May 10, 2021, 5:01 PM UTC

heipei wrote: post #56

kuschku wrote: post #35

Earlier quoted context omitted.

Of course not. But that's why the "cloud" (as in the typical DO/AWS/Azure/GCP offerings) are a scam.

Huge fan of Hetzner, but dedicated servers do not invalidate the value proposition of the cloud. Ordering a server at Hetzner can take anywhere between a few minutes and a few days. Each server has a fixed setup cost of around the monthly rent. They only have two datacenters in Europe. They don't have any auxillary services (databases, queues, scalable object storage, etc.). They are unbeatable for certain use-cases,…

> Ordering a server at Hetzner can take anywhere between a few minutes and a few days

At the start of the pandemic, ordering bare metal servers anywhere was faster than getting a new EC2 VM up and running...

The cloud doesn't even fulfil the value proposition of the cloud. It's significantly more expensive, and when you actually need the flexibility, none of it is available.

Re: Searching the web for under $1000/month

#153

Post by fulmicoton » Wed, May 12, 2021, 2:19 AM UTC

djdjdjdjdj wrote: post #115

fulmicoton wrote: post #28

Earlier quoted context omitted.

The bandwidth is free if you are in the same region.

But you pay for requests.

A normal search experience (displaying a 20 hits search page) requires num segments * (1 + num terms * 2) + 20 GET requests.

We have 180 segments for our commoncrawl index. So we can consider a generous upper bound of 1000 requests.

The GET request costs adds $0.0004 per commoncrawl search request. Storage costs us $5 per day, so the cost of GET request starts topping storage cost at >10k request per day.

Our search engine is meant for searching large datasets, with a low number of queries: Logs, SIEM, e-discovery, exotic big data datasets, etc. These use case have typically a low daily query rate.

For high request rate, (1 query per second) like e-commerce, entirely decoupling storage and compute is actually a bad idea. For low request rate (< 1000 per day), using S3 without caring about the GET request cost is perfectly fine. And in the middle, you might probably want to use another object model with a more favorable pricing model.