31–40 of 82 posts
If you haven't felt it, look at companies like Apple, Microsoft, or "The most important AI research lab in the world" OpenAI, for example, their products have terrible search features even though their resources - money - technology can be considered top-notch.
About a decade ago, I was working with a guy who was getting a PhD in search engine design, which I knew/know nothing about. It was actually a lot of fun to chat with him, because he was so enthusiastic about how searching works and how it can integrate with databases, and he was eager to explain this all to anyone who would listen. I learned a fair amount from him, though admittedly I still don't know much about the…
People who work on really obscure things love to talk about their work, heck if someone would listen to me I could talk for hours about what I do. Unfortunately very few people care about the minutia of making a behemoth system work.
Searching in general is difficult. It is really a difficult thing. If you haven't felt it, look at companies like Apple, Microsoft, or "The most important AI research lab in the world" OpenAI, for example, their products have terrible search features even though their resources - money - technology can be considered top-notch.
Getting search to work well requires a lot of fiddling with ranking parameters, work that is difficult bordering on impossible to plan or track. The work requires a degree of trust that developers are rarely afforded these days.
The idea behind search itself is very simple, and it's a fun problem domain that I encourage anyone to explore[1]. The difficulties in search are almost entirely dealing with the large amounts of data, both logistically and in handling underspecified queries. A DBMS-backed approach breaks down surprisingly fast. Probably perfectly fine if you're indexing your own website, but will likely choke on something the size o…
The idea behind search itself is very simple, and it's a fun problem domain that I encourage anyone to explore[1]. The difficulties in search are almost entirely dealing with the large amounts of data, both logistically and in handling underspecified queries. A DBMS-backed approach breaks down surprisingly fast. Probably perfectly fine if you're indexing your own website, but will likely choke on something the size o…
The idea behind search itself is very simple, and it's a fun problem domain that I encourage anyone to explore[1]. The difficulties in search are almost entirely dealing with the large amounts of data, both logistically and in handling underspecified queries. A DBMS-backed approach breaks down surprisingly fast. Probably perfectly fine if you're indexing your own website, but will likely choke on something the size o…
I love your https://marginalia-search.com :)
The idea behind search itself is very simple, and it's a fun problem domain that I encourage anyone to explore[1]. The difficulties in search are almost entirely dealing with the large amounts of data, both logistically and in handling underspecified queries. A DBMS-backed approach breaks down surprisingly fast. Probably perfectly fine if you're indexing your own website, but will likely choke on something the size o…
I would expect the difficulty to be deciding which item to return when there are multiple that contain the search term. Is wikipedia's article on Gilligan's Island better than some guy's blog post? Or is that guy a fanatic who has spent his entire life pondering whether Wrongway Feldman was malicious or how Irving met Bingo Bango and Bongo?
Add in rank hacking, keyword stuffing, etc. and it seems like a very hard problem, while scaling... is scaling? ¯\_(ツ)_/¯
Entire cubicle farms of people have been devoted to this problem for years, and if you dare to do this for work because "I think I can", you will find yourself in an ocean of hurt.
"Hey, so it won't be so hard to add 'did you mean' functionality, right? And we were thinking of adding a taxonomy next year for easy navigation..."
Check. Mate.
The idea behind search itself is very simple, and it's a fun problem domain that I encourage anyone to explore[1]. The difficulties in search are almost entirely dealing with the large amounts of data, both logistically and in handling underspecified queries. A DBMS-backed approach breaks down surprisingly fast. Probably perfectly fine if you're indexing your own website, but will likely choke on something the size o…
> The difficulties in search are almost entirely dealing with the large amounts of data, both logistically and in handling underspecified queries. I would expect the difficulty to be deciding which item to return when there are multiple that contain the search term. Is wikipedia's article on Gilligan's Island better than some guy's blog post? Or is that guy a fanatic who has spent his entire life pondering whether Wr…
Searching in general is difficult. It is really a difficult thing. If you haven't felt it, look at companies like Apple, Microsoft, or "The most important AI research lab in the world" OpenAI, for example, their products have terrible search features even though their resources - money - technology can be considered top-notch.