451–460 of 500 posts
Earlier quoted context omitted.
A Twitter-like service that fits on a single server could probably get by with the reduced revenue that comes with not offering obsessively fine-grained analytics and ad targeting.
Best comment. And one you don’t hear nearly enough from typical product and eng managers.
$$$$$$ income on $$$ costs
$$$ income on $ costsEarlier quoted context omitted.
I'm a CS undergrad, do you have any recommendations for effectively combating this?
No approach is a silver bullet here, but what I've found effective is to seek out friendships with / mentorship from senior and staff-level engineers. They'll have tons of war stories from 10-20 years ago, and may even have some snarky opinions on what "new technologies" are just re-inventions of something that the industry had already solved decades ago. As a current undergrad, you can also look to your professors f…
And the thing is that what was a bad idea in 2000 might now be an idea whose time has come, because the surrounding context has changed - be it browser technology or the size of machine memory or the capabilities of programming languages.
So, like, when I point out that kubernetes is just DCOM all over again I’m not actually dismissing kubernetes (just because we don’t use DCOM any more doesn’t mean it wasn’t a good choice then); less still suggesting that we should go back to using DCOM; I’m just saying ‘maybe there are some lessons we can learn from how people used DCOM back in the day about what cases kubernetes is suited for and what the pitfalls might be’. And, also, maybe raising the possibility that in a few years time we will look back at kubernetes as a bloated outdated approach and be glad to see the back of it - even though right now it might be a great technology to use.
But I’m not sure how a new junior dev can possibly pick up all that nuance from just listening to old farts like me talking about how this reminds us of how we used to do things back in the old days.
Earlier quoted context omitted.
True. You'd also save a ton of less operations and engineering staff. Running anything on a single server, however, is really a non starter for anything remotely serious. What do you do if you need to do an OS update? I suppose you could just never do those, like a former employer (1000+ day uptimes...)
> You'd also save a ton of less operations and engineering staff. You absolutely would not. The cost of having developers put out extremely optimized code (due to the scaling limits) and cuddle that single server to never, ever fail easily eclipses the cost of a having a multiple servers by a few orders of magnitude. EDIT: To the downvoters, I'd really love to see the calculation on how engineering time would be chea…
(You are right, servers are cheap compared to employee costs though.)
I'm going to preface this criticism by saying that I think exercises like this are fun in an architectural/prototyping code-golf kinda way. However, I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. e.g.: Quote tweets do not go against the size limit of the tweet field at Twitter. Likely they are embedding a tweet reference in some manner or ot…
When interviewing developers I always ask them what is the largest public web site they ever worked on and then probe about performance issues they encountered and how they resolved them in order gague how far along they are in their skill development.
I would never plan to run a production service on a single server just because coordinating changes in the active dataset among two or more production servers often changes your design significantly, and you want to plan for that because the consumer grade hardware we all use has a nasty habit of not working after power cycles (which still tends to be the most strain a system goes through, even in a world of SSD storage).
Nice. While there may be some impracticalities to actually doing this for twitter, 99% of the software out there could run on a fraction of a single commodity server. People complain about the carbon burn of crypto, and they are right, but I bet it is dwarfed by the carbon burn of all the shitty over-provisioned and over-architected CRUD apps running interpreted languages. Unfortunately with universities teaching pyt…
I think people overestimate how much CPU time a typical CRUD app spends on actual business logic, even with an interpreted language like Ruby or Python. What I’ve seen is the bottleneck is largely memory, such that you can pack a ton of these apps on a machine with a few cores and a lot of RAM. The stuff that actually is CPU-bound often ends up being written in an appropriate language, or uses C extensions (e.g. ML a…
Earlier quoted context omitted.
Keyword search is not that hard. Pagerank is not that hard. The comparable goal to the article is to be a search engine, not to fight google for best results.
I think you severely underestimate the effort search engines have to put to filter out all the SEO garbage and come up with some sensible results
Earlier quoted context omitted.
Keyword search is not that hard. Pagerank is not that hard. The comparable goal to the article is to be a search engine, not to fight google for best results.
> Keyword search is not that hard. Pagerank is not that hard. George Hotz, is that you ?
Hotz was trying to make a car controller that had never been done before, by himself, and then he wanted to """improve""" search with no explanation of what that meant that I saw.
I think if he was tasked with taking twitter from no search to "has a search" he probably could have managed it. A team of five people definitely could have managed it.
Earlier quoted context omitted.
> With a second OS partition, the server can alternate between the working copy and a copy that is updated in a VM. For a free service, customers cannot complain even if there is a reboot every day and the service is down for a couple of minutes. Are you seriously suggesting that a service (the size of Twitter, no less) has an acceptable downtime of a few minutes a day? > Realistically, there would be a mirroring ser…
> has an acceptable downtime of a few minutes a day? We kinda know the answer to this: Twitter was struggling with harm to its reputation for a long time because of regular Fail Whales. It absolutely was a huge problem for them at the time.
Who needs Twitter to be a service without any downtime?
Earlier quoted context omitted.
It does not fit on one machine. Tweets alone generate petabytes of data a year, and other events are petabytes per day. https://blog.twitter.com/engineering/en_us/topics/infrastruc... https://ankush-chavan.medium.com/twitter-data-storage-and-pr...
> Tweets alone generate petabytes of data a year Nope. It's not Tweets that generate that data. It's the insane amount of (mostly unnecessary) noise that gets thrown into the mix: analytics, logs, metrics, you name it. Every time you scroll Twitter sends multiple events to the server. That alone will generate a large chunk of those petabytes.
Tweets alone generate petabytes of data a year.
https://ankush-chavan.medium.com/twitter-data-storage-and-pr...
Also, many people would disagree that stuff required to run a business is "mostly unnecessary".