Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How to make a globally fast search engine without using AWS?

news.ycombinator.com

Post reply on HN
Print view View original

1–7 of 7 posts

Ask HN: How to make a globally fast search engine without using AWS?

#1

Post by aw123 »

Forgive me as I'm sort of a novice but after looking at how sluggish some of the small search engines like Kagi are compared to Google or DDG I've been thinking about what the best most cost effective way for a single person/small team to build an app that is low latency across the world without breaking the bank and without relying heavily on the big cloud providers.

If we only have one database in one location then that wouldn't work, right? We'd need a distributed solution so what could we use? For serverless functions to start maybe we use Cloudflare workers? Maybe we'd then have a strategy for moving off cloud and towards buying/renting more datacenter space as we scale?

I'm interested in hearing thoughts about this. If you were to build a fast search engine that could scale to millions of users today, how would you do it?

Re: Ask HN: How to make a globally fast search engine without using AWS?

#3

Post by benoau »

As a starting point I'd go all-in on serverless because even though it's not necessarily the fastest response times or cheapest, it does let you completely ignore several sets of scaling challenges till the price matters. So I'd go serverless for spidering, indexing and user-facing APIs. UIs I'd go static HTML on a CDN.

For database, first I'd lean on plain file s3-like storage as much as possible for spidering and indexing data and try to keep the user-facing database under a couple hundred terabytes, I'd probably favor some bare metal at this point but that will fit on some "db as a service" providers.

Re: Ask HN: How to make a globally fast search engine without using AWS?

#4

Post by conradklnspl »

benoau wrote: post #3

As a starting point I'd go all-in on serverless because even though it's not necessarily the fastest response times or cheapest, it does let you completely ignore several sets of scaling challenges till the price matters. So I'd go serverless for spidering, indexing and user-facing APIs. UIs I'd go static HTML on a CDN. For database, first I'd lean on plain file s3-like storage as much as possible for spidering and i…

While I agree that for user-facing APIs, serverless can both help handling load spikes and keep costs low, how would serverless help with spidering and indexing?

Spidering and indexing are processes that would most likely run continuously for any search engine like Google or Kagi. There is always data to update and new web pages being created.

Thus, they would benefit from dedicated servers from the get go for costs. Ahrefs posted an interesting article on this topic: https://tech.ahrefs.com/how-ahrefs-saved-us-400m-in-3-years-...

Then again, you could use serverless on your own dedicated servers to benefit from the advantages of both.

Re: Ask HN: How to make a globally fast search engine without using AWS?

#5

Post by benoau »

conradklnspl wrote: post #4
benoau wrote: post #3

As a starting point I'd go all-in on serverless because even though it's not necessarily the fastest response times or cheapest, it does let you completely ignore several sets of scaling challenges till the price matters. So I'd go serverless for spidering, indexing and user-facing APIs. UIs I'd go static HTML on a CDN. For database, first I'd lean on plain file s3-like storage as much as possible for spidering and i…

While I agree that for user-facing APIs, serverless can both help handling load spikes and keep costs low, how would serverless help with spidering and indexing? Spidering and indexing are processes that would most likely run continuously for any search engine like Google or Kagi. There is always data to update and new web pages being created. Thus, they would benefit from dedicated servers from the get go for costs.…

> how would serverless help with spidering and indexing?

Because these are tasks that lend themselves to massive concurrency and serverless makes that instantly possible. If you were going to put this on your own hardware that will be a pretty big project of its own.

How many months of dev time can you budget for rolling your own indexing/spidering infrastructure and getting that just right? How does that affect your launch date?

Re: Ask HN: How to make a globally fast search engine without using AWS?

#6

Post by [deleted] »

benoau wrote: post #5
conradklnspl wrote: post #4

Earlier quoted context omitted.

While I agree that for user-facing APIs, serverless can both help handling load spikes and keep costs low, how would serverless help with spidering and indexing? Spidering and indexing are processes that would most likely run continuously for any search engine like Google or Kagi. There is always data to update and new web pages being created. Thus, they would benefit from dedicated servers from the get go for costs.…

> how would serverless help with spidering and indexing? Because these are tasks that lend themselves to massive concurrency and serverless makes that instantly possible. If you were going to put this on your own hardware that will be a pretty big project of its own. How many months of dev time can you budget for rolling your own indexing/spidering infrastructure and getting that just right? How does that affect your…

[deleted]
Post reply on HN