Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Malicious attack on Wikipedia – what we know and what we’re doing

wikimediafoundation.org

Post reply on HN
Print view View original

141–150 of 320 posts

Re: Malicious attack on Wikipedia – what we know and what we’re doing

#141

Post by DuskStar »

patsplat wrote: post #80
didip wrote: post #68

Earlier quoted context omitted.

Not to diminish Wikipedia engineers talent, of course... But, I'd consider Wikipedia traffic to skew heavily towards anonymous read-only, with very few logged-in write traffic. This allows for tons of caching opportunities: Varnish, Memcache, etc. And these techniques are well known.

The proportion of read/write may skew towards reads, but Wikipedia still is an application where any user can create state visible to all other users. It's not as simple as this comment makes it out to be.

But how quickly must those writes be reflected in the reads of others? If you can accept a few minutes of latency there, I imagine things would get easier

Re: Malicious attack on Wikipedia – what we know and what we’re doing

#142

Post by jolmg »

betaby wrote: post #133
jolmg wrote: post #129

Earlier quoted context omitted.

What I'm saying is that it's very simple to inspect a car and make sure it's not going to spontaneously catch fire while parked. You can get a reliable answer in a day from a mechanic. You can't get a reliable answer on whether a computing device is programmed to send malicious packets. There's too much code, most is compiled, there's too many ways to hide it. You can probably gather the smartest people in the world…

We are in the area of probability in both cases. Oftentimes is obvious if PC/device is infected. Sometimes is really hard to find out, https://en.wikipedia.org/wiki/Stuxnet

The question is whether you feel good enough about that probability to be held liable if the malicious code was hidden good enough. Also, some cases might be obvious on Windows PCs, but I don't think that's necessarily the case with phones. Take note that websites can also send malicious packets. When you load a webpage, the code is downloaded and immediately executed. Are you OK with being held liable for visiting a webpage that decided to send malicious packets?

Re: Malicious attack on Wikipedia – what we know and what we’re doing

#143

Post by progval »

mpfundstein wrote: post #105
ollybritton wrote: post #92

Earlier quoted context omitted.

There's also a read-only IPFS mirror of Wikipedia in English: https://ipfs.io/ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1m...

I love ipfs. Can this actually be ddos’d as well?

Yes. It's based on a bittorrent-like protocol with a DHT, so you can get a list of nodes hosting a particular file; then you can DoS them.

Re: Malicious attack on Wikipedia – what we know and what we’re doing

#144

Post by lkoolma »

dangxiaopin wrote: post #120
judge2020 wrote: post #6

Someone claimed the attack on twitter with some details (DDoS) - and proved it later by stopping the attack for x minutes then restarting it at a specific time. https://twitter.com/fs0c131y/status/1170093562878472194?s=20 - the attacker also went on to DDoS the twitch ingest servers (not twitch.tv itself) knocking some big streamers offline.

It looks like a volumetric attack from this tweet. Wikipedia needs to use Verisign BGP mitigation. They create GRE tunnels to your routers and are capable of handling 2Tbps. During an attack, you make a BGP announcement and the traffic goes via Verisign scrubbing/tunnels. No application changes are required, no Matthew Prince selectively and benevolently enforcing CF neutrality. It's used by large banks.

After working with a few large corporations and their DDoS protection solutions, I did not have a good experience with Verisign, and they were not able to handle attacks or get things working. However, I have great experiences with Akamai and Cloudflare. I trust the people at Wikimedia will choose wisely. I would I have learned that Verisign has one of the worst BGP mitigation/scraping solutions out there. There are a few alternatives that have more experience and provide much better uptime, include solutions from Cloudflare and Akamai.

Re: Malicious attack on Wikipedia – what we know and what we’re doing

#145

Post by innocenat »

DuskStar wrote: post #141
patsplat wrote: post #80

Earlier quoted context omitted.

The proportion of read/write may skew towards reads, but Wikipedia still is an application where any user can create state visible to all other users. It's not as simple as this comment makes it out to be.

But how quickly must those writes be reflected in the reads of others? If you can accept a few minutes of latency there, I imagine things would get easier

It must be immediate, because Wikimedia can detect edit conflict (when someone update the article you are in the middle of editing)

Re: Malicious attack on Wikipedia – what we know and what we’re doing

#146

Post by beachy »

blauditore wrote: post #137
chillydawg wrote: post #47

Earlier quoted context omitted.

It's because they're just serving a big site, not running the world's most sophisticated surveillance and ad serving machine. Serving giant websites isn't all that hard if you're just spewing out SQL queries into html templates. It all scales in all directions with a properly thought through architecture.

> Serving giant websites isn't all that hard if you're just spewing out SQL queries into html templates. It all scales in all directions with a properly thought through architecture. No. 1. Your comment makes it sound like Wikipedia is just, or mostly, serving read-only content, which is far from true. Yes, static read-only content is significantly easier to serve than dynamic, editable one, but Wikipedia is the latt…

I've never heard anyone in my life say they could rebuild MS office in a weekend.

What, in your opinion, would be the work needed to go from a 100k monthly active user site to a wikipedia scale site - that would be comparable to rebuilding MS office?

Re: Malicious attack on Wikipedia – what we know and what we’re doing

#147

Post by betaby »

jolmg wrote: post #142
betaby wrote: post #133

Earlier quoted context omitted.

We are in the area of probability in both cases. Oftentimes is obvious if PC/device is infected. Sometimes is really hard to find out, https://en.wikipedia.org/wiki/Stuxnet

The question is whether you feel good enough about that probability to be held liable if the malicious code was hidden good enough. Also, some cases might be obvious on Windows PCs, but I don't think that's necessarily the case with phones. Take note that websites can also send malicious packets. When you load a webpage, the code is downloaded and immediately executed. Are you OK with being held liable for visiting a…

Liability boundary is an important question. There is no simple answer. Couple of years back owner of the abandoned building was declared liable of the death of the kid who entered to the fenced building despite all the signs no trespassing and so on. Neither your example nor mine negates that there should liability for various voluntarily and involuntarily acts, own or third party.

Re: Malicious attack on Wikipedia – what we know and what we’re doing

#148

Post by dangxiaopin »

lkoolma wrote: post #144
dangxiaopin wrote: post #120

Earlier quoted context omitted.

It looks like a volumetric attack from this tweet. Wikipedia needs to use Verisign BGP mitigation. They create GRE tunnels to your routers and are capable of handling 2Tbps. During an attack, you make a BGP announcement and the traffic goes via Verisign scrubbing/tunnels. No application changes are required, no Matthew Prince selectively and benevolently enforcing CF neutrality. It's used by large banks.

After working with a few large corporations and their DDoS protection solutions, I did not have a good experience with Verisign, and they were not able to handle attacks or get things working. However, I have great experiences with Akamai and Cloudflare. I trust the people at Wikimedia will choose wisely. I would I have learned that Verisign has one of the worst BGP mitigation/scraping solutions out there. There are…

Any serious mitigation solution must be BGP based, not proxy. Besides its technical merits and convenience, it also minimizes the risk of a benevolent controller (e.g. Matthew Prince of Cloudflare) ruining your company, because it becomes your upstream provider only during the attacks. Otherwise the GRE tunnels are not in use. The IP addresses are still yours always.

We used Verisign for mitigation of a 44Gbps volumetric attack and it worked very well. We also evaluated Neustar, but Verisign's infrastructure seemed to be more robust.

Re: Malicious attack on Wikipedia – what we know and what we’re doing

#149

Post by RijilV »

softwaredoug wrote: post #11

Just want to mention, WMF has a very small but elite team of engineers. Amazed they maintain an Alexa top 5 site with many orders of magnitude less engineering staff than Facebook or Reddit. I think they must count ~100 engineers? I can't imagine what such a small team must be going through with a major DDOS - wish them well in their efforts!

It’s sorta interesting how the replies to this digressed into linguistics about systems architecture, and nobody called out the “elite engineers” statement.

I’ve worked with a couple of engineers who are now on Wikipedia’s SRE team. They’re good engineers, but not elite by any means. Not “10x” developers or wizards in castles or whatever. Good solid engineers who I would work with again and fight to hire. But they’re not savants or even the top 10% of folks I’ve worked with. Solid mid to sr level engineers I’d be happy to hand a project off to with ambiguous goals and little oversight, and I’d expect them to get a team of 4 or so other engineers to be more productive.

These are the engineers who meet the job requirements for SRE positions.

Re: Malicious attack on Wikipedia – what we know and what we’re doing

#150

Post by input_sh »

woadwarrior01 wrote: post #77
softwaredoug wrote: post #11

Just want to mention, WMF has a very small but elite team of engineers. Amazed they maintain an Alexa top 5 site with many orders of magnitude less engineering staff than Facebook or Reddit. I think they must count ~100 engineers? I can't imagine what such a small team must be going through with a major DDOS - wish them well in their efforts!

I used to work at FB and now work at Reddit. The engineering staff count at Reddit is within the same order of magnitude as the number you cite above. :)

Yeah, but unlike on Reddit, I never see "something went wrong" on Wikipedia.

No offense to you nor your team, but to me, as a consumer, reddit's product doesn't appear nowhere near as polished as Wikimedia's projects.

Post reply on HN