Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Searching the web for under $1000/month

quickwit.io

Post reply on HN
Print view View original

81–90 of 153 posts

Re: Searching the web for under $1000/month

#81

Post by Retric »

whoknew1122 wrote: post #71
Retric wrote: post #62

Earlier quoted context omitted.

AWS is a scam not because it can’t save you money, but because they actively try to trick you into spending more money. That’s practically the definition of a scam. Go to the AWS console and try to answer even simply things like how much did the last hour/day/week cost me? Or how about some notifications if that new service you just added is going to cost vastly more than you where expecting. I know of a few people g…

Disclosure: Work at AWS. I've never seen AWS actively try to trick people into spending more money. I've seen Premium Support, product service teams, solutions architects, and account managers all suggest not to use AWS services if it doesn't fit the customer usecase. I've personally recommended non-AWS options for customers who are trying to fit a square peg into a round hole. Can the billing console be better? Yes.…

I haven’t used AWS in a while but one trick that I recall was enabling service X also enabled sub dependencies. Instantly disabling service X didn’t stop services XYZ which you continued to be billed for. Granted not that expensive, but it still felt like a trap.

Other stuff was more debatable, but it just felt like dancing in a mine field.

Re: Searching the web for under $1000/month

#82

Post by Retric »

marcinzm wrote: post #65
Retric wrote: post #62

Earlier quoted context omitted.

AWS is a scam not because it can’t save you money, but because they actively try to trick you into spending more money. That’s practically the definition of a scam. Go to the AWS console and try to answer even simply things like how much did the last hour/day/week cost me? Or how about some notifications if that new service you just added is going to cost vastly more than you where expecting. I know of a few people g…

AWs is pretty bad at telling you how much something you're not running will cost if you run it but I've never had any issues knowing what something has cost me in the past. >Go to the AWS console and try to answer even simply things like how much did the last hour/day/week cost me? Click user@account in top right, click My Billing Dashboard, spend this month is on that page in giant font, click Cost Explorer for more…

> Click user@account in top right, click My Billing Dashboard, spend this month is on that page in giant font, click Cost Explorer for more granular breakdown (day, service, etc.), click Bill Details for list breakdown of spend by month.

Sure, you see a number but I was just talking with someone at AWS who said it you still can’t trust it to be up to date especially across zone boundaries. That means it’s useful when everything is working as expected but can be actively misleading when troubleshooting.

Re: Searching the web for under $1000/month

#83

Post by rossmohax »

whoknew1122 wrote: post #71
Retric wrote: post #62

Earlier quoted context omitted.

AWS is a scam not because it can’t save you money, but because they actively try to trick you into spending more money. That’s practically the definition of a scam. Go to the AWS console and try to answer even simply things like how much did the last hour/day/week cost me? Or how about some notifications if that new service you just added is going to cost vastly more than you where expecting. I know of a few people g…

Disclosure: Work at AWS. I've never seen AWS actively try to trick people into spending more money. I've seen Premium Support, product service teams, solutions architects, and account managers all suggest not to use AWS services if it doesn't fit the customer usecase. I've personally recommended non-AWS options for customers who are trying to fit a square peg into a round hole. Can the billing console be better? Yes.…

I do not support view that AWS is a scam, but price is something AWS tries to make developers not to think about. Every blog post, documentation or quick start tells you about features, but never about costs.

You read "you can run Lambda in VPC", great, but there is a fine print somewhere on a remote page, that you'd also need NAT gateway if you want said Lambda to access internet, public network wont do.

You read "you can enable SSE on S3", but it is not immediately obvious, that every request then incurs KMS call and billed accordingly (that was before bucket key feture).

Want to enable Control Tower? It creates so many services, it is impossible to predict costs until you enable it and wait to be billed.

Re: Searching the web for under $1000/month

#84

Post by hansvm »

chris_f wrote: post #79

Nice! Maybe at one point you can release a general web search engine for the Common Crawl corpus? It seems even simpler than this proof of concept, but potentially more useful for people looking for a true full text web search. There isn't an easy way today to explore or search what is contained in the Common Crawl index.

> There isn't an easy way today to explore or search what is contained in the Common Crawl index.

By that you mean searching the full text contents of their crawl, right?

The index is super easy to search nowadays -- in pretty much any language you can slap a few lines of code around a get request (using range requests [0] if needed), and explore a columnar representation of the index [1].

[0] https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec1...

[1] https://commoncrawl.org/2018/03/index-to-warc-files-and-urls...

Re: Searching the web for under $1000/month

#85

Post by fulmicoton »

chris_f wrote: post #79

Nice! Maybe at one point you can release a general web search engine for the Common Crawl corpus? It seems even simpler than this proof of concept, but potentially more useful for people looking for a true full text web search. There isn't an easy way today to explore or search what is contained in the Common Crawl index.

That's on my to-do list for next week. :)

Re: Searching the web for under $1000/month

#87

Post by ywelsch »

Interesting! We've built similar support for decoupling compute from storage into Elasticsearch and, as coincidence would have it, just shared some performance numbers today:

https://www.elastic.co/blog/querying-a-petabyte-of-cloud-sto...

It works just as any regular Elasticsearch index (with full Kibana support etc.).

The data being indexed by Lucene allows queries to access index structures and return results orders of magnitude faster than doing a full table scan.

It is complemented with various caching layers to make repeat queries fast.

We expect this new functionality to be used for less frequently queried data (e.g. operational or security investigations, legal discoveries, or historical performance comparisons on older data), trading query speed for cost.

It supports Google Cloud Storage, Azure Blob Storage, Amazon S3 (+ S3 compatible stores), HDFS, and shared file systems.

Re: Searching the web for under $1000/month

#88

Post by fizx »

nopurpose wrote: post #32

It is a cool project. S3 can be cost efficient, but only if you don't touch data :) Their price calculation doesn't mention cost of S3 requests, which very quickly adds up and is often neglected. It costs $1 for 2.5M GET requests to S3. They have 180 shards, in a general case query seems to fetch all of them. Presumably they don't download full shard per request, but download an index + some relevant ranges. Lets say…

It's easy to put a block cache in front of the index, and I'm sure they'll get to it sooner or later.

The benefit of using S3 in that case is that unlike e.g. Elastic, your block cache servers don't need replication, and you can tear them down when you're done. You can put them in a true autoscaling group as well.

Re: Searching the web for under $1000/month

#89

Post by rossmohax »

whoknew1122 wrote: post #71
Retric wrote: post #62

Earlier quoted context omitted.

AWS is a scam not because it can’t save you money, but because they actively try to trick you into spending more money. That’s practically the definition of a scam. Go to the AWS console and try to answer even simply things like how much did the last hour/day/week cost me? Or how about some notifications if that new service you just added is going to cost vastly more than you where expecting. I know of a few people g…

Disclosure: Work at AWS. I've never seen AWS actively try to trick people into spending more money. I've seen Premium Support, product service teams, solutions architects, and account managers all suggest not to use AWS services if it doesn't fit the customer usecase. I've personally recommended non-AWS options for customers who are trying to fit a square peg into a round hole. Can the billing console be better? Yes.…

Another example of a bit darkish pattern is listing ridiculously small prices ($0.0000166667 per GB-second, $0.0004 per 1000 GET requests). It's hard to reason about very small and very big numbers, order of magnitude difference "feels" the same. Showing such a small prices is accurate, but deceiving IMHO.

Re: Searching the web for under $1000/month

#90

Post by simonw »

What does your on-S3 storage format look like? Are you storing relatively large blobs and doing HTTP Range requests against them or are you storing lots of tiny objects and fetching the whole object any time you need it?
Post reply on HN