Searching the web for under $1000/month
quickwit.io
Searching the web for under $1000/month
1–10 of 153 posts
Re: Searching the web for under $1000/month
#2This is super interesting. I've recently also been working on a similar concept: we have a reasonable amount (in the terabytes) of data, that's fairly static, that I need to search fairly infrequently (but sometimes in bulk). A solution we came up with was a small , hot, in memory index, that points to the location of the data in a file on S3. Random access of a file on S3 is pretty fast, and running in an EC2 instance means latency is almost nil to S3. Cheap, fast and effective.
We're using some custom Python code to build a Marisa Trie as our index. I was wondering if there were alternatives to this set up?
Re: Searching the web for under $1000/month
#3This is super interesting. I've recently also been working on a similar concept: we have a reasonable amount (in the terabytes) of data, that's fairly static, that I need to search fairly infrequently (but sometimes in bulk). A solution we came up with was a small , hot, in memory index, that points to the location of the data in a file on S3. Random access of a file on S3 is pretty fast, and running in an EC2 instan…
> that I need to search fairly infrequently (but sometimes in bulk).
What do you mean by search ? Full-text-search ? Do you need to run custom code on the original data ?
> A solution we came up with was a small , hot, in memory index, that points to the location of the data in a file on S3.
Yes, it's like keeping the block-index of a sstable (in rocksdb) in-memory. The next step is to have a local cache on the ec2 node. And the next step one is to have a "distributed" cache on your ec2 nodes, so you don't query S3 for a chunk if it's present in any of your other nodes.
Come to think of it, I searched and didn't find a "distributed disk cache with optional replication" that can be used in front of S3 or whatever dataset. You can use nginx/varnish as a reverse-proxy but it doesn't have "distributed". There is Alluxio, but it's single-master.
Re: Searching the web for under $1000/month
#4Very interesting! For some reason I find search engines fascinating...
How dependent is this on AWS? Can it be ported to another cloud provider?
Re: Searching the web for under $1000/month
#5> which is key as each instance issues a lot of parallel requests to Amazon S3 and tends to be bound by the network
I wonder if most of the cost comes from S3, EC2 or the "premium" bandwidth that Amazon charges ridiculously much for. Since it seems to be doing a lot of requests, it wouldn't surprise me if it's the network cost, and if so, I wonder why they would even use AWS at all then.
Re: Searching the web for under $1000/month
#6> which is key as each instance issues a lot of parallel requests to Amazon S3 and tends to be bound by the network I wonder if most of the cost comes from S3, EC2 or the "premium" bandwidth that Amazon charges ridiculously much for. Since it seems to be doing a lot of requests, it wouldn't surprise me if it's the network cost, and if so, I wonder why they would even use AWS at all then.
> I wonder if most of the cost comes from S3
This current cost comes from the big dataset of storage in S3.
> it wouldn't surprise me if it's the network cost
Network cost is only outbound. Inside it's free (except multi region etc). Ec2 S3 is free bandwidth (you pay for requests).
Re: Searching the web for under $1000/month
#7This is super interesting. I've recently also been working on a similar concept: we have a reasonable amount (in the terabytes) of data, that's fairly static, that I need to search fairly infrequently (but sometimes in bulk). A solution we came up with was a small , hot, in memory index, that points to the location of the data in a file on S3. Random access of a file on S3 is pretty fast, and running in an EC2 instan…
Combining data-at-rest with some slim index structure coupled with a common access method (like HTTP) was the idea behind a tool a key-value store for JSON I once wrote: https://github.com/miku/microblob
I first thought of building a custom index structure, but found that I did not need everything in memory all the time. Using an embedded leveldb works just fine.
Re: Searching the web for under $1000/month
#8This looks really interesting, I wonder how they will monetize it though.
As an aside, projects like these are what keep me wondering whether I should switch from cheaper but "dumb" object stores to AWS since on AWS you can use your object store together with things like Athena etc. and get pay-per-use search / grep and a lot of other things, without the egress fees since it's all within AWS.
Re: Searching the web for under $1000/month
#9This is super interesting. I've recently also been working on a similar concept: we have a reasonable amount (in the terabytes) of data, that's fairly static, that I need to search fairly infrequently (but sometimes in bulk). A solution we came up with was a small , hot, in memory index, that points to the location of the data in a file on S3. Random access of a file on S3 is pretty fast, and running in an EC2 instan…
> that I need to search fairly infrequently (but sometimes in bulk). What do you mean by search ? Full-text-search ? Do you need to run custom code on the original data ? > A solution we came up with was a small , hot, in memory index, that points to the location of the data in a file on S3. Yes, it's like keeping the block-index of a sstable (in rocksdb) in-memory. The next step is to have a local cache on the ec2 n…
> What do you mean by search ?
Search maybe is too strong a word - "lookup" is probably more correct. I have a couple of identifiers for each document, from which I want to retrieve the full doc.
I'm not sure what you mean by running custom code on the data. I usually do some kind of transformation afterwards.
I didn't find anything either, which is why I was wondering if I was searching for the wrong thing.
Re: Searching the web for under $1000/month
#10Article title is "Searching the web for Despite mentioning Rust once, of course it had to be added to the title on HN as "Search 1B pages on AWS S3 for 1000$ / month, made in Rust and tantivy".