Gitlab servers are being exploited in DDoS attacks • Hacker News Board

Re: Gitlab servers are being exploited in DDoS attacks

#61

Post by 1vuio0pswjnm7 » Fri, Nov 05, 2021, 12:36 AM UTC

Scaevolus wrote: post #51

1vuio0pswjnm7 wrote: post #34

Seems like one solution would be to require users uploading images to remove metadata themselves before uploading. The website could reject images that contain matadata. Was Gitlab was extracting the metadata and using it for some purpose. If not, what is the reason to accept images with metadata. Perhaps they assume their customers prefer less "security" and more "convenience", instead of vice versa (less "convenien…

Theoretically the pure-perl implementation will be immune to most categories of exploits that sloppy binary parsing runs into-- buffer overflows, use after frees, and so on. That better security goes out the window once you start using eval(), of course.

Both exiv2 and jhead, as examples, have had numerous such bugs. However, there is a difference IMO between running these programs on one's own images (created by oneself), e.g., before uploading them to the internet, versus running them on untrusted images from others.

Re: Gitlab servers are being exploited in DDoS attacks

#62

Post by PragmaticPulp » Fri, Nov 05, 2021, 12:37 AM UTC

Someone1234 wrote: post #44

magicalhippo wrote: post #10

From the issue report[1] comments: > Thanks in no small part to your recent findings, [GitLab] are rolling back our policy about paying half-bounties for third party findings. These have great impact on GitLab and we want to continue to incentivize research for high+ severity issues in that area. At least they're taking these things a bit more seriously now. [1]: https://hackerone.com/reports/1154542

Anyone have background on what a "third party finding" is? Are they talking about dependencies? Because, to me, this specific bug is both a first AND third party issue (e.g. GitLab not pre-verifying before calling ExifTool).

> Anyone have background on what a "third party finding" is? Are they talking about dependencies?

Yes.

I've had some experience with bug bounty programs. The situation with vulnerabilities in dependencies is complicated.

On one hand you want to incentivize researchers to find vulnerabilities in your dependencies, which likely don't have their own bug bounty programs.

On the other hand, you run into situations where researchers discover a bug in a 3rd-party library and, instead of reporting it ASAP, they keep it secret as long as possible while they build up an arsenal of bug bounty exploits to report to multiple companies that use the library. This is a weird disincentive to fix the underlying bug quickly because the researchers know they only have so much time to exploit it in bug bounty programs before one of the companies fixes it upstream.

We also had a problem where amateurs would spam our bug bounty program any time a CVE came out for one of our dependencies, even if they couldn't exploit it. It was relatively easy to close these bug reports because they couldn't provide a proof of concept exploit, but it still wasted a lot of time arguing with them, especially when they'd try to blackmail us on social media for not paying them out (for a bug that didn't exist in our product).

I do not miss my days of dealing with bug bounty programs. Met a few great researchers, but most of the (attempted) participants were trying harder to exploit the bug bounty program than find exploits in our code.

Re: Gitlab servers are being exploited in DDoS attacks

#63

Post by Gigachad » Fri, Nov 05, 2021, 12:41 AM UTC

ed25519FUUU wrote: post #50

Jach wrote: post #41

It's somewhat refreshing that the underlying bug isn't from some C or C++ utility, but instead a Perl program using eval: https://github.com/exiftool/exiftool/blob/11.70/lib/Image/Ex... Another instance of "avoid eval as much as possible" for languages that have it.

Public facing? Handles user input data?? Uses ‘eval’???

The line existed in the 2014 commit which migrated the repo to git. It wasn't designed in the current era of mass automated abuse and internet connected everything.

Re: Gitlab servers are being exploited in DDoS attacks

#64

Post by djbusby » Fri, Nov 05, 2021, 12:42 AM UTC

jeffbee wrote: post #56

djbusby wrote: post #9

Earlier quoted context omitted.

ExifTool is hackjob? I think not. But also, file uploads should be handled in a jail or box of some type - and never let their analysis make network calls.

It's a perl program that evaluates untrusted strings it finds in user files. What exactly is your standard for "is hackjob"? It appears to be a complete piece of shit.

It's clearly not complete shit, else it wouldn't be used by literally millions of people/systems. ExifTool is so far away from shit that in fact it was chosen by a highly respected company with a very good team.

A hackjob usually has less deploys than my own stuff (which, outside of Windows 2000 components, is less than a few millions)

Re: Gitlab servers are being exploited in DDoS attacks

#65

Post by Gigachad » Fri, Nov 05, 2021, 12:42 AM UTC

Someone1234 wrote: post #38

buildbuildbuild wrote: post #2

This happened to all Gitlab instances that I manage around 2 days ago. Good to see publicity, I’m still dealing with not-so-understanding abuse departments at my hosting providers. Sure, my fault for not keeping it up to date. But there is much noise to filter through in the many tools we juggle these days, especially if an organization prefers to self-host.

> But there is much noise to filter through in the many tools we juggle these days, especially if an organization prefers to self-host. If an organization is too overloaded to patch for six months, maybe they should re-evaluate if self-hosting is the best course of action. Seems like this is a foot-gun of your own creation.

There is very little reason to even self host Gitlab unless you are insanely paranoid or for philosophy reasons like debian/gnome.

Re: Gitlab servers are being exploited in DDoS attacks

#66

Post by Gigachad » Fri, Nov 05, 2021, 12:43 AM UTC

charcircuit wrote: post #49

SV_BubbleTime wrote: post #5

Earlier quoted context omitted.

Article said GitLab patched back in April. Safe to say you didn’t deploy these patches? No judgment. I’m paid to make things, not apply patches. This is however why I don’t use self-hosted, pros and cons, etc.

Not him, but how are you supposed to know about the update? Do you need to check some page every day if there's an update? Why can't security updates just autoupdate like apps on phones or at least email the admin saying there is an important update.

Yes, you are meant to be paying attention to the security news for all tools you self host. That's part of self hosting. If you can't manage that (which is completely fair), then you shouldn't be self hosting.

Re: Gitlab servers are being exploited in DDoS attacks

#67

Post by djbusby » Fri, Nov 05, 2021, 12:45 AM UTC

wswope wrote: post #39

djbusby wrote: post #9

Earlier quoted context omitted.

ExifTool is hackjob? I think not. But also, file uploads should be handled in a jail or box of some type - and never let their analysis make network calls.

I’ve had a really hard time finding good guides for hardening VMs for malware analysis or processing untrusted inputs as you suggest - any guidance on learning resources?

I learned the hard way. But for these restricted jails I'd start by making a VM that is not allowed out at all, like DROP on iptavles OUTPUT chain.

Re: Gitlab servers are being exploited in DDoS attacks

#68

Post by xxpor » Fri, Nov 05, 2021, 12:47 AM UTC

Jach wrote: post #41

It's somewhat refreshing that the underlying bug isn't from some C or C++ utility, but instead a Perl program using eval: https://github.com/exiftool/exiftool/blob/11.70/lib/Image/Ex... Another instance of "avoid eval as much as possible" for languages that have it.

With dlopen() and libclang/libgccjit, you could argue C (on an OS that supports dynamic loading) has eval too ;)

Re: Gitlab servers are being exploited in DDoS attacks

#69

Post by majou » Fri, Nov 05, 2021, 12:50 AM UTC

Gigachad wrote: post #63

ed25519FUUU wrote: post #50

Earlier quoted context omitted.

Public facing? Handles user input data?? Uses ‘eval’???

The line existed in the 2014 commit which migrated the repo to git. It wasn't designed in the current era of mass automated abuse and internet connected everything.

2014 was very much that era, I'd accept this excuse maybe for 2007

Re: Gitlab servers are being exploited in DDoS attacks

#70

Post by pm90 » Fri, Nov 05, 2021, 12:54 AM UTC

PragmaticPulp wrote: post #62

Someone1234 wrote: post #44

Earlier quoted context omitted.

Anyone have background on what a "third party finding" is? Are they talking about dependencies? Because, to me, this specific bug is both a first AND third party issue (e.g. GitLab not pre-verifying before calling ExifTool).

> Anyone have background on what a "third party finding" is? Are they talking about dependencies? Yes. I've had some experience with bug bounty programs. The situation with vulnerabilities in dependencies is complicated. On one hand you want to incentivize researchers to find vulnerabilities in your dependencies, which likely don't have their own bug bounty programs. On the other hand, you run into situations where r…

Fascinating. I would love to read more about your experience if you have a blog.