Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Searching the web for under $1000/month

quickwit.io

Post reply on HN
Print view View original

21–30 of 153 posts

Re: Searching the web for under $1000/month

#21

Post by heipei »

fulmicoton wrote: post #19
heipei wrote: post #8

This looks really interesting, I wonder how they will monetize it though. As an aside, projects like these are what keep me wondering whether I should switch from cheaper but "dumb" object stores to AWS since on AWS you can use your object store together with things like Athena etc. and get pay-per-use search / grep and a lot of other things, without the egress fees since it's all within AWS.

We really need to make this clear in our next blog post. This is not grep here. We are using the same datastructure that are used in Elasticsearch or google. We just adapted them to be object storage friendly. I would not call Object Storage dumb by any mean. They are a very powerful bottom-up abstraction. We do manage to get SSD-like throughput from them. The latency is the big issue. We had to redesign our search t…

Appreciate the response. I wasn't trying to say this is grep, I fully understand that this is an inverted index which is way more interesting to build on top of S3.

I merely wanted to say that by using S3 within AWS you always have the fallback option of brute-force "grep" across your semi-structured "data lake" or whatever it's called thanks to the aggregate bandwidth and Athena.

Re: Searching the web for under $1000/month

#22

Post by gallexme »

jonatron wrote: post #16

If you're going for low cost, you could do better: https://www.hetzner.com/dedicated-rootserver/dell/dx181/conf... Basic configuration in Finland 1 224,91 € 1.92 TB SATA SSD Datacenter Edition 4 95,20 € 320,11 € 320 Euro equals 385.90 United States Dollar

Depending on the requirements https://www.hetzner.com/dedicated-rootserver/ax101/ May be an actually better fit

Once they available again

Re: Searching the web for under $1000/month

#23

Post by ddorian43 »

hungnv wrote: post #11
ddorian43 wrote: post #3

Earlier quoted context omitted.

> that I need to search fairly infrequently (but sometimes in bulk). What do you mean by search ? Full-text-search ? Do you need to run custom code on the original data ? > A solution we came up with was a small , hot, in memory index, that points to the location of the data in a file on S3. Yes, it's like keeping the block-index of a sstable (in rocksdb) in-memory. The next step is to have a local cache on the ec2 n…

> Come to think of it, I searched and didn't find a "distributed disk cache with optional replication" that can be used in front of S3 or whatever dataset. You can use nginx/varnish as a reverse-proxy but it doesn't have "distributed". There is Alluxio, but it's single-master. If you think more about this, it will be like distributed key value store with support both disk and memory access. You can write one using so…

> If you think more about this, it will be like distributed key value store with support both disk and memory access. You can write one using some opensource Raft libraries, or a possible candidate is Tikv from PingCap

My whole point was not building it ;)

There's also https://github.com/NVIDIA/aistore

Re: Searching the web for under $1000/month

#24

Post by fulmicoton »

natpat wrote: post #2

This is super interesting. I've recently also been working on a similar concept: we have a reasonable amount (in the terabytes) of data, that's fairly static, that I need to search fairly infrequently (but sometimes in bulk). A solution we came up with was a small , hot, in memory index, that points to the location of the data in a file on S3. Random access of a file on S3 is pretty fast, and running in an EC2 instan…

There might be much better alternative but it really depends on the nature of your key.

Because the crux of S3 is the latency you can also decide to encode the docs in blocks, and retrieve more data than is actually needed.

For this demo, the index from DocID to offset in S3 takes 1.2 bytes per doc. For a log corpus, we end up with 0.2 bytes per doc.

Re: Searching the web for under $1000/month

#25

Post by ddorian43 »

natpat wrote: post #9
ddorian43 wrote: post #3

Earlier quoted context omitted.

> that I need to search fairly infrequently (but sometimes in bulk). What do you mean by search ? Full-text-search ? Do you need to run custom code on the original data ? > A solution we came up with was a small , hot, in memory index, that points to the location of the data in a file on S3. Yes, it's like keeping the block-index of a sstable (in rocksdb) in-memory. The next step is to have a local cache on the ec2 n…

> What do you mean by search ? Search maybe is too strong a word - "lookup" is probably more correct. I have a couple of identifiers for each document, from which I want to retrieve the full doc. I'm not sure what you mean by running custom code on the data. I usually do some kind of transformation afterwards. I didn't find anything either, which is why I was wondering if I was searching for the wrong thing.

How big is each document ? If documents are big, keep each of them as a separate file and store the ids in a database. If documents are small, then you want something like https://github.com/rockset/rocksdb-cloud for a building block

Re: Searching the web for under $1000/month

#26

Post by ddorian43 »

gallexme wrote: post #22
jonatron wrote: post #16

If you're going for low cost, you could do better: https://www.hetzner.com/dedicated-rootserver/dell/dx181/conf... Basic configuration in Finland 1 224,91 € 1.92 TB SATA SSD Datacenter Edition 4 95,20 € 320,11 € 320 Euro equals 385.90 United States Dollar

Depending on the requirements https://www.hetzner.com/dedicated-rootserver/ax101/ May be an actually better fit Once they available again

You'll be waiting 1+ month to get the server above.

Re: Searching the web for under $1000/month

#27

Post by fulmicoton »

bambax wrote: post #4

Very interesting! For some reason I find search engines fascinating... How dependent is this on AWS? Can it be ported to another cloud provider?

We have a storage abstraction that boils down to being able to perform Range queries.

Anything that allows us to do range queries is ok.

That includes basically all object storage I know of (Google Cloud Storage, Azure Storage, Minio, you name it), but also HDFS, or even a remote HTTP2 server.

Re: Searching the web for under $1000/month

#30

Post by [deleted] »

jonatron wrote: post #16

If you're going for low cost, you could do better: https://www.hetzner.com/dedicated-rootserver/dell/dx181/conf... Basic configuration in Finland 1 224,91 € 1.92 TB SATA SSD Datacenter Edition 4 95,20 € 320,11 € 320 Euro equals 385.90 United States Dollar

[deleted]
Post reply on HN