261–270 of 500 posts
The ultimate extension of this "run it all on one machine" meme would be to run the bots on the single machine along with the service.
Earlier quoted context omitted.
> which is how many hard drives you can plug in to a non-mainframe machine for historical image storage. You would be surprised . First off, SSDs are denser than hard drives now if you're willing to spend $$$. Second, "plug in" doesn't necessarily mean "in the chassis". You can expand storage with external disk arrays in all sorts of ways. Everything from external PCI-e cages to SAS disk arrays, fibre channel, NVMe-o…
That's really cool! Each year of historical images I estimate at 2.8PB, so it would need to scale quite far to handle multiple years. How would you actually connect all those external drive chassis, is there some kind of chainable SAS or PCIe that can scale arbitrarily far? I consider NVMe-over-fabrics to be cheating and just using multiple machines and calling it one machine, but "one machine" is kinda an arbitrary…
[0] https://www.liqid.com/products/liqid-elements/liqid-48-port-...
Very cool exercise. I enjoyed reading it. I see a lot of comments here assuming that this proves something about Twitter being inefficient. Before you jump to conclusions, take a look at the author’s code: https://github.com/trishume/twitterperf Notably absent are things like serving HTTP , not to even mention HTTPS. This was a fun exercise in algorithms, I/O, and benchmarking. It wasn’t actually imitating anything t…
Which I think I'm perfectly clear about in the blog post. The post is mostly about napkin math systems analysis, which does cover HTTP and HTTPS. I'm now somewhat confident I could implement this if I tried, but it would take many years, the prototype and math is to check whether there's anything that would stop me if I tried and be a fun blog post about what systems are capable of. I've worked on a team building a s…
Of course. I was commenting here to counter all of the comments declaring that this proves Twitter doesn’t need all of their servers, etc.
It’s a fun article, but the comments here interpreting it as proving something about Twitter engineers being bad are kind of depressing.
Earlier quoted context omitted.
Quote tweets I'd do as a reference and they'd basically have the cost of loading 2 tweets instead of one, so increasing the delivery rate by the fraction of tweets that are quote tweets. Hashtags are a search feature and basically need the same posting lists as for search, but if you only support hashtags the posting lists are smaller. I already have an estimate saying probably search wouldn't fit. But I think hashta…
I don’t think that hashtags are a search only feature. In the posts themselves, the hashtags are clickable to view other tweets. I don’t think that qualifies as a search.
Earlier quoted context omitted.
I don’t think that hashtags are a search only feature. In the posts themselves, the hashtags are clickable to view other tweets. I don’t think that qualifies as a search.
It does strike me as a feature you'd typically serve out of some sort of search index since if you had to build search, you'd essentially get indexing of hashtags "for free"
Earlier quoted context omitted.
[flagged]
If an inspector reviews your house and finds no issues, that is indeed evidence of absence.
Absence of evidence, in your case via a clean building inspection, does not mean the building is safe. It just means the checklist of known items was considered and nothing bad found.
Ask a building inspector if their clean report proves nothing is wrong with the building.
They will be firm and quick to inform you that it’s not a warranty — anything not checked was not covered. Items not covered could still be significant problems.
That’s the whole point of the saying. Absence of evidence is not evidence of absence.
I am both embarrassed and disappointed with the negativity this post has attracted. A litanny of "gotchas", where someone attempts to best the OP. What about x, y and z? It can't possibly scale. Twitter is so much more than this, etc. The OP isn't making the assertion that Twitter should replace their current system with a single large machine. The whole thread paints a picture of HN like it is full of a bunch of hal…
If he would have called it anything else but Twitter and stated it is theoretical or a hobby project or something for fun, I would be fine with it, but don't mention Twitter anywhere near because it has nothing to do with Twitter.
I'm going to preface this criticism by saying that I think exercises like this are fun in an architectural/prototyping code-golf kinda way. However, I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. e.g.: Quote tweets do not go against the size limit of the tweet field at Twitter. Likely they are embedding a tweet reference in some manner or ot…
Twitter's "linked" tweets seems to be strangely unattached from context. What I mean is, Twitter seems to be processing data based on whatever it is in the tweet and doesn't maintain some grand coherent database. So I changed my Twitter handle and opened a new account with my original Twitter handle and to my surprise, I was receiving notifications of engagement with tweets my old account sent previously. I also hear…
Interesting optimization idea: If 95% of the users are bots, and your ML algorithms are smart enough to figure who's bot and who's bio, you could save a lot of traffic by not publishing tweets to bots as no one is going to read them anyway. Of course if that traffic included advertisements, you'd also lose 95% of your ad revenue. The ultimate extension of this "run it all on one machine" meme would be to run the bots…
Not so, serving an ad to a bot gains you no revenue, because ad networks charge for clicks, not impressions. If a significant percentage of your ad clicks are from bots, you're running a defective advertising platform and won't have customers for long regardless.