Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Searching the web for under $1000/month

quickwit.io

Post reply on HN
Print view View original

11–20 of 153 posts

Re: Searching the web for under $1000/month

#11

Post by hungnv »

ddorian43 wrote: post #3
natpat wrote: post #2

This is super interesting. I've recently also been working on a similar concept: we have a reasonable amount (in the terabytes) of data, that's fairly static, that I need to search fairly infrequently (but sometimes in bulk). A solution we came up with was a small , hot, in memory index, that points to the location of the data in a file on S3. Random access of a file on S3 is pretty fast, and running in an EC2 instan…

> that I need to search fairly infrequently (but sometimes in bulk). What do you mean by search ? Full-text-search ? Do you need to run custom code on the original data ? > A solution we came up with was a small , hot, in memory index, that points to the location of the data in a file on S3. Yes, it's like keeping the block-index of a sstable (in rocksdb) in-memory. The next step is to have a local cache on the ec2 n…

> Come to think of it, I searched and didn't find a "distributed disk cache with optional replication" that can be used in front of S3 or whatever dataset. You can use nginx/varnish as a reverse-proxy but it doesn't have "distributed". There is Alluxio, but it's single-master.

If you think more about this, it will be like distributed key value store with support both disk and memory access. You can write one using some opensource Raft libraries, or a possible candidate is Tikv from PingCap

Re: Searching the web for under $1000/month

#12

Post by heipei »

natpat wrote: post #2

This is super interesting. I've recently also been working on a similar concept: we have a reasonable amount (in the terabytes) of data, that's fairly static, that I need to search fairly infrequently (but sometimes in bulk). A solution we came up with was a small , hot, in memory index, that points to the location of the data in a file on S3. Random access of a file on S3 is pretty fast, and running in an EC2 instan…

You could look at AWS Athena, especially if you only query infrequently and can wait a minute on the search results. There are some data layout patterns in your S3 bucket that you can use to optimize the search. Then you have true pay-per-use querying and don't even have to run any EC2 nodes or code yourself.

Re: Searching the web for under $1000/month

#13

Post by artembugara »

Francois, Adrien, that's a super nice demo.

Stateless search engine is something new, for sure.

I'd be super interested to see how it evolves over time. We're [1] indexing over 1,000,000 news articles per day. We're using ElasticSearch to index our data.

Would be interested to see if there's a way to make a cross-demo? Let me know.

[1] https://newscatcherapi.com/

Re: Searching the web for under $1000/month

#14

Post by fulmicoton »

artembugara wrote: post #13

Francois, Adrien, that's a super nice demo. Stateless search engine is something new, for sure. I'd be super interested to see how it evolves over time. We're [1] indexing over 1,000,000 news articles per day. We're using ElasticSearch to index our data. Would be interested to see if there's a way to make a cross-demo? Let me know. [1] https://newscatcherapi.com/

That sounds interesting indeed.

Can you schedule a meeting with me? https://calendly.com/paul-quickwit/30min

Re: Searching the web for under $1000/month

#15

Post by giovannibonetti »

natpat wrote: post #2

This is super interesting. I've recently also been working on a similar concept: we have a reasonable amount (in the terabytes) of data, that's fairly static, that I need to search fairly infrequently (but sometimes in bulk). A solution we came up with was a small , hot, in memory index, that points to the location of the data in a file on S3. Random access of a file on S3 is pretty fast, and running in an EC2 instan…

This is the kind of thing I value in Rails. Active storage [1] has been around for a few years and it solves all of this. All the metadata you care about is in the database - content type, file size, image dimensions, creation date, storage path.

[1] https://guides.rubyonrails.org/active_storage_overview.html

Re: Searching the web for under $1000/month

#17

Post by artembugara »

fulmicoton wrote: post #14
artembugara wrote: post #13

Francois, Adrien, that's a super nice demo. Stateless search engine is something new, for sure. I'd be super interested to see how it evolves over time. We're [1] indexing over 1,000,000 news articles per day. We're using ElasticSearch to index our data. Would be interested to see if there's a way to make a cross-demo? Let me know. [1] https://newscatcherapi.com/

That sounds interesting indeed. Can you schedule a meeting with me? https://calendly.com/paul-quickwit/30min

Merci

Re: Searching the web for under $1000/month

#19

Post by fulmicoton »

heipei wrote: post #8

This looks really interesting, I wonder how they will monetize it though. As an aside, projects like these are what keep me wondering whether I should switch from cheaper but "dumb" object stores to AWS since on AWS you can use your object store together with things like Athena etc. and get pay-per-use search / grep and a lot of other things, without the egress fees since it's all within AWS.

We really need to make this clear in our next blog post. This is not grep here. We are using the same datastructure that are used in Elasticsearch or google.

We just adapted them to be object storage friendly. I would not call Object Storage dumb by any mean. They are a very powerful bottom-up abstraction.

We do manage to get SSD-like throughput from them. The latency is the big issue. We had to redesign our search to reduce the number of random read in the critical to the bear minimum.

Post reply on HN