The Implications of Facebook Indexing a Trillion Posts
techcrunch.com
The Implications of Facebook Indexing a Trillion Posts
1–10 of 33 posts
Re: The Implications of Facebook Indexing a Trillion Posts
#2> Most obviously, the News Feed could learn to mimic our external dialogue, showing us posts with similar content to what we spread. Never talk about sports or babies? Facebook could eventually filter those out of your feed. Just shared your thoughts on Syria, celebrity gossip, or the police state? The algorithm could pull an audible and show you more about related news.
I was pretty sure Facebook was already doing this, even before the public search. No?
Re: The Implications of Facebook Indexing a Trillion Posts
#3I'm pretty interested in how this is implemented, actually. Naively you could search a global index and then filter results down to your friends, but that filter seems impossibly slow. Alternatively you could maintain a separate index per person, but querying 500 indexes seems unreasonable.
Maybe they partition the graph into cliques which they index, search and then fix up with a second pass to add and remove noncliqued friends?
Re: The Implications of Facebook Indexing a Trillion Posts
#4I'm pretty interested in how this is implemented, actually. Naively you could search a global index and then filter results down to your friends, but that filter seems impossibly slow. Alternatively you could maintain a separate index per person, but querying 500 indexes seems unreasonable. Maybe they partition the graph into cliques which they index, search and then fix up with a second pass to add and remove noncli…
You could build an index per person, but rather than search 500 indexes, the index of each person would be of all their and their friends' posts, so you would only search one index.
Re: The Implications of Facebook Indexing a Trillion Posts
#5I wonder what Stallman might be thinking. He has been against Facebook since inception and in those days the main concern was not the state but private entities getting too much power over the consumers/users. I still think that this should be the main worry for the average Joe.
Re: The Implications of Facebook Indexing a Trillion Posts
#6I'm pretty interested in how this is implemented, actually. Naively you could search a global index and then filter results down to your friends, but that filter seems impossibly slow. Alternatively you could maintain a separate index per person, but querying 500 indexes seems unreasonable. Maybe they partition the graph into cliques which they index, search and then fix up with a second pass to add and remove noncli…
You could build an index per person, but rather than search 500 indexes, the index of each person would be of all their and their friends' posts, so you would only search one index.
Doesn't that make updating rather expensive - for each new post it needs to go into a lot of different search indexes?
Off-topic: Wasn't there a start-up a while back that was allowing people to build their own personal search index for their social media content?
Re: The Implications of Facebook Indexing a Trillion Posts
#7Earlier quoted context omitted.
You could build an index per person, but rather than search 500 indexes, the index of each person would be of all their and their friends' posts, so you would only search one index.
Doesn't that make updating rather expensive - for each new post it needs to go into a lot of different search indexes? Off-topic: Wasn't there a start-up a while back that was allowing people to build their own personal search index for their social media content?
Yes. Writes can be delayed and queued though. Reads need to be fast.
Re: The Implications of Facebook Indexing a Trillion Posts
#8I'm pretty interested in how this is implemented, actually. Naively you could search a global index and then filter results down to your friends, but that filter seems impossibly slow. Alternatively you could maintain a separate index per person, but querying 500 indexes seems unreasonable. Maybe they partition the graph into cliques which they index, search and then fix up with a second pass to add and remove noncli…
I'm not sure how similar this will be but Twitter have a bit of information on how they indexed "roughly half a trillion documents " when they moved to indexing all tweets here: https://blog.twitter.com/2014/building-a-complete-tweet-inde...
Edit: I don't think Twitter do nearly the same filtering with this and their social graph so maybe not as close as I first thought.
Re: The Implications of Facebook Indexing a Trillion Posts
#9And all of this is brought to you by the friendly folks of PHP. Next time when you bash PHP because the argument order for functions is inconsistent, think about how some people overlook semantics and build great things with the tools they have at their disposal.
Re: The Implications of Facebook Indexing a Trillion Posts
#10And all of this is brought to you by the friendly folks of PHP. Next time when you bash PHP because the argument order for functions is inconsistent, think about how some people overlook semantics and build great things with the tools they have at their disposal.
Yeah, it's all brought to us by the friendly PHP folks who found it to be so dog slow they had to write a compiler to get it running at a decent speed. Further, they found the semantics so flaky, they actually wrote a statically typed variant on top of it, just because they're stuck with PHP.