101–110 of 153 posts
What does your on-S3 storage format look like? Are you storing relatively large blobs and doing HTTP Range requests against them or are you storing lots of tiny objects and fetching the whole object any time you need it?
Earlier quoted context omitted.
There are only 180 splits. For this demo we use a file. For more serious stuff we use postgresql.
What does the metadata structure look like?
If you're going for low cost, you could do better: https://www.hetzner.com/dedicated-rootserver/dell/dx181/conf... Basic configuration in Finland 1 224,91 € 1.92 TB SATA SSD Datacenter Edition 4 95,20 € 320,11 € 320 Euro equals 385.90 United States Dollar
do you know if they let you host adult videos?
https://www.hetzner.com/rechtliches/cloud-server/?country=de
i know a couple people have naughty stuff on it (like sex toy shops, sexual services, private for sale adult videos)
It is a cool project. S3 can be cost efficient, but only if you don't touch data :) Their price calculation doesn't mention cost of S3 requests, which very quickly adds up and is often neglected. It costs $1 for 2.5M GET requests to S3. They have 180 shards, in a general case query seems to fetch all of them. Presumably they don't download full shard per request, but download an index + some relevant ranges. Lets say…
It is a cool project. S3 can be cost efficient, but only if you don't touch data :) Their price calculation doesn't mention cost of S3 requests, which very quickly adds up and is often neglected. It costs $1 for 2.5M GET requests to S3. They have 180 shards, in a general case query seems to fetch all of them. Presumably they don't download full shard per request, but download an index + some relevant ranges. Lets say…
AWS is almost never cost efficient. Maybe if you stay in their free tier.
A ridiculous blanket statement, despite the "almost never" cop-out...
It is cost-efficient in a wide array of scenarios. Many companies pay for it because they have calculated the different investment scenarios and AWS comes on top of alternatives such as owning the hardware or using competing cloud vendors.
I own a consultancy that builds complex web apps and while I appreciate how occasionally a dev has tried to save costs for me by cramming every piece of the stack (web server, cache, db, queue, etc.), in a single Docker image to host in a droplet, I'd much rather pay for separate services, as I consider it cheaper in the long run.
Earlier quoted context omitted.
AWS is a scam not because it can’t save you money, but because they actively try to trick you into spending more money. That’s practically the definition of a scam. Go to the AWS console and try to answer even simply things like how much did the last hour/day/week cost me? Or how about some notifications if that new service you just added is going to cost vastly more than you where expecting. I know of a few people g…
Disclosure: Work at AWS. I've never seen AWS actively try to trick people into spending more money. I've seen Premium Support, product service teams, solutions architects, and account managers all suggest not to use AWS services if it doesn't fit the customer usecase. I've personally recommended non-AWS options for customers who are trying to fit a square peg into a round hole. Can the billing console be better? Yes.…
If pricing is intended to be transparent, then why is it completely absent from the user interface? Transparent pricing would be to tell me how much something costs when I order it, not make me use a different tool or find it in the documentation
Reason I ask is I'm trying to get all subdomain a of a certain domain. So I want a reverse host of unique hostnames under a certain domain.
The following is a stupid question, so bare with me.
I have been using search engines for about... 26 years. I have attempted to make really crappy databases and search engines. I have worked for companies that use search products for internal services and customer products. I'm not a search engineer but I have a decent understanding of them and their issues, I think. And I get why people want full-text search. But is it actually a good idea? Should anyone really be using full text search?
I actually work on search products right now. We use Solr as the general full text index. We have separate indexes and algorithms to make context and semantic inferences, and prioritize results based on those, falling back to full text if we don't get anything. The full text sucks. The corpus of relationships of related concepts is what makes the whole thing useful.
Are we (all) only using full-text because some users are demanding that it be there? Or shouldn't we all stop this charade of thinking that full-text search of billions of items of data will ever be useful to a human being? Even when I show my coworkers that I can get something done 10x faster with a curated index of content, they still want a search engine that they know doesn't give them the results they want.
Is full-text search the junk food of information retrieval?
Is there a more recent common crawl data set? 2019 is a long time away. Reason I ask is I'm trying to get all subdomain a of a certain domain. So I want a reverse host of unique hostnames under a certain domain.
It is a cool project. S3 can be cost efficient, but only if you don't touch data :) Their price calculation doesn't mention cost of S3 requests, which very quickly adds up and is often neglected. It costs $1 for 2.5M GET requests to S3. They have 180 shards, in a general case query seems to fetch all of them. Presumably they don't download full shard per request, but download an index + some relevant ranges. Lets say…
AWS is almost never cost efficient. Maybe if you stay in their free tier.