I don't know nearly enough about large scale projects or the startup world maybe this is a silly question but when it became apparent that it was blowing up far beyond what you were expecting, was it not an option to have another team (hire or split existing) in parallel, to start writing a re-implmentation in another language or system that would work better for your new quickly expanding needs? Was that just not fe…
Thats what kept happening and it was an abysmal failure. Every so often a person would have a brilliant idea on how to solve our scaling issues. They would then disappear into a corner to invent yet-another-bird-themed-datastore. After a few weeks/months they would appear with a magical new thing that would fix all our problems and would make everybody happy. Every single time it would fail. Having a team that is not…
Maybe I'm being dumb here, but twitter doesn't look like a product from the outside that has many features. Are these focused on advertisers, analytics or what?
I followed this story back in 2006 to 2008. I’m sorry I don’t have time to write a real comment with the crucial details. The basic plot is that Twitter initially set out to be a CMS sort of like WordPress, but then they realized they had actually built a messaging platform, which needed a different architecture. You can catch a glimpse of the issues in “Amateur hour is over at Twitter” from 2008. https://www.google.…
Hi! I am one of the dudes that actually helped kill the fail whale so I feel I can speak to this topic. =) Steve and John were both good developers that wrote some seriously critical components of the system and were a big part of getting twitter more stable. But reading Techcrunch to figure out what was happening inside of Twitter is asking for serious misinformation. At that time it was very common for people with…
If Twitter couldn't keep up with scaling then it wasn't being architected to scale, only to grow.
There are financial systems that handle millions of transactions per second because their designers sat down and designed systems that could scale by magnitudes.
Hi! I am one of the dudes that actually helped kill the fail whale so I feel I can speak to this topic. =) Steve and John were both good developers that wrote some seriously critical components of the system and were a big part of getting twitter more stable. But reading Techcrunch to figure out what was happening inside of Twitter is asking for serious misinformation. At that time it was very common for people with…
If Twitter couldn't keep up with scaling then it wasn't being architected to scale, only to grow. There are financial systems that handle millions of transactions per second because their designers sat down and designed systems that could scale by magnitudes.
Yea but those designers knew the magnitudes that they were dealing with and built them into every single layer of the system. Its easy to build a system that handles "x million transactions a second". Its MUCH harder to build a system that handles somewhere between x, and x*100 million messages a second. Especially when in startup mode because cost and complexity matter greatly.
Ultimately it wasn't "build a db that handles x writes" though. That is actually a fairly known and solvable problem. Our issue was that we were migrating from a design model that encouraged rabid development by using existing patterns to a model that would scale up much better but required a lot more planning and design, all while dealing with management that expected the feature development cycle to not slow down at all during or after the transition.
Thats what kept happening and it was an abysmal failure. Every so often a person would have a brilliant idea on how to solve our scaling issues. They would then disappear into a corner to invent yet-another-bird-themed-datastore. After a few weeks/months they would appear with a magical new thing that would fix all our problems and would make everybody happy. Every single time it would fail. Having a team that is not…
Maybe I'm being dumb here, but twitter doesn't look like a product from the outside that has many features. Are these focused on advertisers, analytics or what?
At the time Twitter had a ton of features under the hood that kept being supported and maintained, all of which just added complexity to the system.
We had an API service, a web interface, the legacy web interface that was still used for select devices because the new UI didn't quite work right on them, the even older legacy interface that was necessary because a bunch of badly behaved early day clients still relied on the functionality and they were popular enough that turning them off would cause outrage, the "zero" interface used in countries with low bandwidth capabilities, the mobile interface.
Each interface had to implement all the different variations on functionality. Timelines with inline tweet rendering (automatic expansion of images, etc), list (alternate view time lines), the whole following graph (duplicated for lists as well), verified users and all the infra around that, search, public/private designations, direct messages, notifications via email, text message, and mobile app, favorites, retweets, replies, plus a slew of statistics and information tracking data integrated directly into the site.. Thats only the user visible stuff. There are a TON of experiments and projects that run behind that interface in a way the user will never completely see.
We heard over and over that twitter was so simple that it could run on a laptop and every time it reminded me just how clueless most developers are when it comes to seeing the body of work needed to make something like twitter work, even more so at the scale we are talking about.
This is the excerpt I've seen being tweeted about this piece: At the same time, her defenders say, Harvey has been forced to clean up a mess that Twitter should have fixed years ago. Twitter’s backend was initially built on Ruby on Rails, a rudimentary web-application framework that made it nearly impossible to find a technical solution to the harassment problem. If Twitter’s co-founders had known what it would becom…
Hi, early Twitter employee here. The reliability team called the infrastructure Fisher-Price internally so this wasn't just one random executive coming up with a term on his/her own. The problem wasn't ruby. The problem was the way that Twitter used Ruby. We had one big mono repo with every single function and every form of business logic baked into a single place. That logic relied on monkey patching and all sorts o…
So this is really just an issue of the reporter/editor not bothering to fact-check the statements of a non-technical former executive with, one assumes, an axe (or two) to grind.
I am not a coder by any means, but I am a heavy social media user and know a lot about politics, sociology, psych, organizing, health, comms---and I admit, until I read this thread, I liked this article because it gave me a better reason for all the screw ups, beyond, "The Executives are dithering and have no business skills and social skills." Or the theory "The Executives do nothing because they don't care if women and vulnerable people die." Which is far more disturbing....it can't be true, even if it feels true.
I would give anything if they'd listen to users who have been around awhile. So few people worked there and also used it at the same time. (And users longingly miss the Fail Whale logo...)
As a user (250,000+ tweets) under a pseudonym, twitter has incredible uses, things Facebook and other platforms didn't do, because they were so closed and hard to search. They created echo Chambers, because you could only see people you already knew or were slightly connected too. Or worse, your mom or mother-in-law could find you. Twitter was so open--if I want to discuss philosophy or Japanese food or an MRI result, 24/7 I just search and people who love that are there. And pseudonyms had to exist, because thousands of people in real life have the same name. (just like the rest of the Internet back then and yes, now too) They thought people would just post status updates like, hey eating lunch, but we did way way more. They made it 140 chars with 20 chars reserved for names because many people had expensive tiny data plans, but could update by text message; allowing a much wider demographic to use it, and across multiple countries, even low tech ones with limited access.
The most critical piece tho is that users invented everything good about twitter (Sorry creators--but we did). TW allowed some html symbols, so a user put an @ in front of a user name to reply, and it worked! Users also invented hashtags, the first one was for organizing BarCamp--which was kind of a conference not about alcohol and the original manual RT, and MT for modified tweet, and commenting on top of someone's tweet, or at the end of it and we figured out how to shrink long urls to post links. bit.ly had no purpose til then, and many users became Developers and got full access to the API and we crowdsourced hundreds of changes and ideas, from pictures to videos to emoji to gifs to analytics, to accessible apps for people with vision, hearing, speech issues. All while making lots and lots of jokes.
They didn't start out with a heavy respect for pseudonyms, and privacy, and free speech--but TW learned it fast after the Green Iran Revolution, and after many patient users wanted to keep privacy because they had rare diseases, and mental health issues, and parents of kids with autism and speech issues took to it and found each other and felt less alone. It was less complicated than blogging and writing long stories on laptops after events happened. I could do everything from live tweeting a doctor's appt to an ER visit and get reactions from friends who could tell me what to ask.
We still tweet everything from recipes to exact instructions on how to ride a bike, how to organize groups like #occupy to crowd sourcing Flu symptoms and rashes, to who is watching what TV show and how cool it is to watch live sports "together" even for people who can't leave the house that night and meet.
Weekly hour long Chats take place under special hashtags like #hcsm for Health Care Social Media or #meded for medical education. (doctors and science fans and academics found twitter and they debate articles, techniques, crowdsource diagnoses)
Police and emergency responders and good Samaritans have used it to (swear to god, it's true) befriend people who sound troubled or suicidal and validate their pain and sadness and find them help.
It is to this day, the only platform I know that allows people from many different areas to find like-minded people and to bring together people across multiple subject areas.
Nothing else does it quite the same way. Especially with the ridiculous algorithms other platforms use.
> There are two main components to Harvey’s job, this person told me: to formulate a clear set of rules for what constitutes abusive speech, and to be consistent in enforcing them. The odds are against Harvey being the first person in human history to solve this problem without false positives and negatives.
The big question in my eyes is not how a human can reduce the error rate, but rather, which category of errors one should bias towards. I know where I stand on this one: be biased against the mean-spirited.
My general feeling when it comes to questions about harassment on Twitter is that people look at it too much as a policy issue ("you're not banning and deleting enough accounts according to my ideological stance"), which has its place, but there's a lot of product aspects that make it such a shouty place. A good example is the way the 'quote tweet' feature is often used to start a pile-on. It's not a bad feature in i…
Jeff Vogel:
> Twitter was designed, from Day 1, to enable any random person to send messages directly to any public figure. In other words, from Day 1, it was designed to be an abuse and harassment engine. It's not a bug. It's a feature. All that abuse and controversy is how it gets clicks and money.