> Twitter’s backend was initially built on Ruby on Rails, a rudimentary web-application framework that made it nearly impossible to find a technical solution to the harassment problem.
This is probably a new low in journalism.
Twitter and the Internet War
31–40 of 59 posts
Re: Twitter and the Internet War
#32> There are two main components to Harvey’s job, this person told me: to formulate a clear set of rules for what constitutes abusive speech, and to be consistent in enforcing them. The odds are against Harvey being the first person in human history to solve this problem without false positives and negatives.
The big question in my eyes is not how a human can reduce the error rate, but rather, which category of errors one should bias towards.
I know where I stand on this one: be biased against the mean-spirited.
Re: Twitter and the Internet War
#33This is the excerpt I've seen being tweeted about this piece: At the same time, her defenders say, Harvey has been forced to clean up a mess that Twitter should have fixed years ago. Twitter’s backend was initially built on Ruby on Rails, a rudimentary web-application framework that made it nearly impossible to find a technical solution to the harassment problem. If Twitter’s co-founders had known what it would becom…
Hi, early Twitter employee here. The reliability team called the infrastructure Fisher-Price internally so this wasn't just one random executive coming up with a term on his/her own.
The problem wasn't ruby. The problem was the way that Twitter used Ruby. We had one big mono repo with every single function and every form of business logic baked into a single place. That logic relied on monkey patching and all sorts of crazy horrible glue to keep it working together. Every time we had to scale up we would glue infra in place to keep things working while we came up with a real solution (which never really materialized).
In my time there we had memcache instances which held timelines. Populating them took hours/days and while they were unpopulated the site was offline. Rebooting/restarting the caches was simply not an option. We had a data sharding strategy that was temporal. We would spin up a new database cluster every few weeks to handle all of the incoming tweets and failing to spin up a new cluster in time meant we would have a global site outage. Don't even get me started on the "load bearing mac mini".
In reality the only problem rails really contributed on its own was that it could only process a single request per process at a time. Each machine would spin up 16 or 32 processes to handle requests in parallel but each process needed its own connection to the database, to memcache, etc. At one point we had something like 100k processes all trying to talk to a single mysql master. Much of this could have been mitigated by better design of course, but rails encourages models that don't scale up to crazy dimensions.
In reality moderation was virtually impossible because we were in a 24/7 fight with ourselves about how to keep the system alive for the next couple of days. Constant infighting, managerial changes (I had 9 different managers in 3 years), focus changes (we didn't finish the last major site redesign before starting the next one) and a general unwillingness to pause features long enough to stabilize the system meant we were always on the losing end of a infra battle.
Re: Twitter and the Internet War
#34This is the excerpt I've seen being tweeted about this piece: At the same time, her defenders say, Harvey has been forced to clean up a mess that Twitter should have fixed years ago. Twitter’s backend was initially built on Ruby on Rails, a rudimentary web-application framework that made it nearly impossible to find a technical solution to the harassment problem. If Twitter’s co-founders had known what it would becom…
I followed this story back in 2006 to 2008. I’m sorry I don’t have time to write a real comment with the crucial details. The basic plot is that Twitter initially set out to be a CMS sort of like WordPress, but then they realized they had actually built a messaging platform, which needed a different architecture. You can catch a glimpse of the issues in “Amateur hour is over at Twitter” from 2008. https://www.google.…
Hi! I am one of the dudes that actually helped kill the fail whale so I feel I can speak to this topic. =)
Steve and John were both good developers that wrote some seriously critical components of the system and were a big part of getting twitter more stable. But reading Techcrunch to figure out what was happening inside of Twitter is asking for serious misinformation. At that time it was very common for people with no idea what they were talking about to write articles about what was going on inside of Twitter, yet do so with authority that they didn't really have. By the time a team of reliability focused people started to get hired Twitter was finally getting smart about not letting employees/outsides control the messaging sent to Techcrunch and the like.
Twitter has completely reinvented the architecture since I left so that is not impossible by any means.
Lets be clear about the rate that Twitter was growing though. In any 9 month window while I worked there they never failed to grow an order of magnitude in request volume. When I started we would have spikes of a few thousand requests per second. We had a major site outage at 451 tweets per second and we simply couldn't keep up. 9 months later we were spiking 50k requests per second, and we never dropped below 451 tps (barring outage). By the time I left both of those numbers simply couldn't be seen in the graphs anymore.
Every time we got technology in place to deal with the scaling issues we were already behind the curve because of how fast things were growing. It felt like being in a car that has amazing 0-60 time, but it just kept accelerating well beyond 60mph.
Re: Twitter and the Internet War
#35Earlier quoted context omitted.
You're going to think I'm trolling, but I'm not-- What value does Twitter provide to ordinary people? I have friends who are celebrities in their industries or trying to market themselves-- but other than that it seems like the most hostile place imaginable. Why would I want to participate in that?
In the US, most ordinary people pay attention to twitter's top customer: donald trump
I suspect the vast majority of people read what Trump said on Twitter on their favorite news site, rather than using Twitter itself.
Re: Twitter and the Internet War
#36“I think what religion and politics have in common is that they become part of people's identity, and people can never have a fruitful argument about something that's part of their identity. By definition they're partisan.
Which topics engage people's identity depends on the people, not the topic. For example, a discussion about a battle that included citizens of one or more of the countries involved would probably degenerate into a political argument. But a discussion today about a battle that took place in the Bronze Age probably wouldn't. No one would know what side to be on. So it's not politics that's the source of the trouble, but identity. When people say a discussion has degenerated into a religious war, what they really mean is that it has started to be driven mostly by people's identities.
...
More generally, you can have a fruitful discussion about a topic only if it doesn't engage the identities of any of the participants. What makes politics and religion such minefields is that they engage so many people's identities. But you could in principle have a useful conversation about them with some people. And there are other topics that might seem harmless, like the relative merits of Ford and Chevy pickup trucks, that you couldn't safely talk about with others.“
Re: Twitter and the Internet War
#37Lost? How is Twitter losing the Internet War? Their platform is more influential than ever and helped to seriously impact a presidential election (the results of which not too many people in media are happy with, fair enough). Yes, the platform is a failure in so many ways - failure to protect the identity and safety of the people on it. (Though as someone who has served a subpoena to Twitter, let me tell you somethi…
You're going to think I'm trolling, but I'm not-- What value does Twitter provide to ordinary people? I have friends who are celebrities in their industries or trying to market themselves-- but other than that it seems like the most hostile place imaginable. Why would I want to participate in that?
> What value does Twitter provide to ordinary people?
I don’t tweet. I do read newspapers. Twitter appears to have made journalists quicker at spotting trends. (By making it easier to find quotes, it also makes them more productive.)
Re: Twitter and the Internet War
#38> Twitter’s backend was initially built on Ruby on Rails, a rudimentary web-application framework that made it nearly impossible to find a technical solution to the harassment problem. This is probably a new low in journalism.
Can you elaborate? I don't know much about ror and would like to know more.
Re: Twitter and the Internet War
#39> Twitter’s backend was initially built on Ruby on Rails, a rudimentary web-application framework that made it nearly impossible to find a technical solution to the harassment problem. This is probably a new low in journalism.
Can you elaborate? I don't know much about ror and would like to know more.
The web framework (Ruby on Rails) has nothing to do with the ability to build anti-harassment tech.
Re: Twitter and the Internet War
#40My general feeling when it comes to questions about harassment on Twitter is that people look at it too much as a policy issue ("you're not banning and deleting enough accounts according to my ideological stance"), which has its place, but there's a lot of product aspects that make it such a shouty place. A good example is the way the 'quote tweet' feature is often used to start a pile-on. It's not a bad feature in i…
That's a cool idea, if you really want to quote-tweet someone you can follow them, do it and then unfollow them, so you're not limiting behavior really, but it would still reduce the number of times it happened.
However I think twitter used to have a lot of weird secret features like how DM's could accidentally become public if you formatted them wrong, and another secret rule to twitter could be a bad idea.