41–50 of 91 posts
This is mostly why I left Reddit. The API allows far too much control and I started questioning what was even real. Being able to quickly find keywords and then have a network of bots that creates replies/upvotes/downvotes is very disturbing thought to me. I can't even imagine something like that on a large scale to change opinions.
Seems a bit over the top imho. Maybe a better approach is to ask for a 1,000 and look for any missing — which you can grab individually.
I’d be a little annoyed at people not using batch mode and making so many request but that’s just me.
> The other 95% of it is just wasted bandwidth. You can save a lot of bandwith by requesting compressed responses: $ curl -s --user-agent moo/1 -H 'Accept-Encoding: gzip' "$pretty_long_url" > test.gz $ wc -c (OK, that's 85% saved, not 95%, but hey.)
> So here’s the approach I ended up using, which worked much better: request each post by its ID. That’s right, instead of asking for posts in batches of 100, we’re going to need to ask for each post individually by its post ID. We’ll do the same for comments. Seems a bit over the top imho. Maybe a better approach is to ask for a 1,000 and look for any missing — which you can grab individually. I’d be a little annoye…
Their default listing mode works very poorly. It would certainly be more requests to use a hybrid system like you're talking about.
It is difficult for me to describe just how angry it makes me that reddit doesn't provide a way for users to even do basic things like "see all of my own comments" or "see all of the posts made to the subreddit I moderate". They keep nerfing the search APIs and claim it is so they could make the indexes more efficient, but while that might make sense for a full-text search interface, that is entirely unreasonable for…
(I worked at reddit) It's not really related to search, most of the cause is a pretty bizarre optimization method that reddit decided to implement fairly early on. The database structure is unusual and not very conducive to indexing (it's similar to an EAV model), so at some point they decided to basically write their own "secondary indexing"-like system that stores the "listing indexes" in memcached/Cassandra (with…
Earlier quoted context omitted.
(I worked at reddit) It's not really related to search, most of the cause is a pretty bizarre optimization method that reddit decided to implement fairly early on. The database structure is unusual and not very conducive to indexing (it's similar to an EAV model), so at some point they decided to basically write their own "secondary indexing"-like system that stores the "listing indexes" in memcached/Cassandra (with…
Thanks for the explanation. The APIs make a lot more sense in light of it. It seems odd they didn't just use Postgres indexing though. Do you know if they benchmarked it at any point?
Aho-Corasick is really great. It’s a bit complicated to set up, but once you have the modified true set up it’s really fast. By the way, > Basically I use the selftext, subreddit, permalink, url and title. The other 95% of it is just wasted bandwidth. It’d probably be better for Reddit if they allowed for specifying the fields we care about rather than just returning the whole thing…
I only took a quick skim, but it doesn't look like they do the optimization to aho-corasick where you store connections directly to "leaf" nodes (ie: nodes where you can have a finished match). If I'm right, that would probably speed things up significantly.
Earlier quoted context omitted.
(I worked at reddit) It's not really related to search, most of the cause is a pretty bizarre optimization method that reddit decided to implement fairly early on. The database structure is unusual and not very conducive to indexing (it's similar to an EAV model), so at some point they decided to basically write their own "secondary indexing"-like system that stores the "listing indexes" in memcached/Cassandra (with…
Thanks for the explanation. The APIs make a lot more sense in light of it. It seems odd they didn't just use Postgres indexing though. Do you know if they benchmarked it at any point?
> The other 95% of it is just wasted bandwidth. You can save a lot of bandwith by requesting compressed responses: $ curl -s --user-agent moo/1 -H 'Accept-Encoding: gzip' "$pretty_long_url" > test.gz $ wc -c (OK, that's 85% saved, not 95%, but hey.)
Trading cpu for bandwidth, so optimize for what you want.
If you do it right you can even keep the content stored compressed without re-compressing by saving the compressed byte stream directly.
> I mean do you really want subreddit name and subreddit_name_prefixed? They’re the same, one just has an “r/” in front of it. This is (unfortunately) not quite true. Since Reddit introduced "profile posts," there can be a post where the subreddit name is something like "u_Shitty_Watercolour" but the subreddit_name_prefixed is actually "u/Shitty_Watercolour", rather than "r/u_Shitty_Watercolour". Example: https://www…
I'm not sure that's true. I think they both work, see: https://www.reddit.com/r/u_Shitty_Watercolour/ Maybe one is just an alias though? I wonder if you can make a r/u_$unused_username and then later register $unused_username edit: nope, you can't make a sub that starts with "u_"