Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ARIN Public Incident Report – 4.10 Misissuance Error

arin.net

Post reply on HN
Print view View original

11–20 of 41 posts

Re: ARIN Public Incident Report – 4.10 Misissuance Error

#11

Post by Aurornis »

_zagj wrote: post #6
progbits wrote: post #3

I like how frank the report is, no sugarcoating. "We relied on manual error prone verification and made a mistake. We have to automate the process." As ARIN block owner this situation is kinda scary but reading this actually makes me think it's less likely to happen again .

You don't find this part > We have to automate the process. to be ominous?

I don’t. The report says part of this process relied on flat files and spreadsheets. Automating that with software is a good idea.

“Automate the process” doesn’t mean feeding everything to an LLM.

Re: ARIN Public Incident Report – 4.10 Misissuance Error

#13

Post by aaomidi »

_zagj wrote: post #6
progbits wrote: post #3

I like how frank the report is, no sugarcoating. "We relied on manual error prone verification and made a mistake. We have to automate the process." As ARIN block owner this situation is kinda scary but reading this actually makes me think it's less likely to happen again .

You don't find this part > We have to automate the process. to be ominous?

Certificate issuance was once only possible manually.

Re: ARIN Public Incident Report – 4.10 Misissuance Error

#14

Post by galaxygate »

Affected customer here, if you're curious on our original NANOG post on the whole situation:

Hey NANOG,

After receiving a BGPAlerter notification that one of our subnets (23.150.164.0/24) had been hijacked, I checked and noticed the prefix in question was missing RPKI. Assuming I had fat fingered something and butchered the ROA, I logged into ARIN and found that the prefix was missing from our resource list entirely, and had been reallocated to another organization and announced from their network. I created a ticket in ARIN and called immediately.

They confirmed that our subnet had been accidentally reallocated to another customer, and that they are currently working on returning it to us. After a couple hours, they told us the other organization will stop announcing the prefix, and WHOIS will be returned shortly.

I’m guessing there’s no way to prevent this kind of thing on our side if the RPKI ROA itself is removed along with the allocation? I’m planning on adding checks to look for missing ROAs (in addition to invalid/expiring ones), which I'm guessing would've caught this earlier.

Have any of you had anything like this happen with ARIN or another RIR? I’m especially curious what might have happened if we’d only noticed and reached out a few weeks later instead of within a few minutes.

Re: ARIN Public Incident Report – 4.10 Misissuance Error

#16

Post by thaumaturgy »

galaxygate wrote: post #14

Affected customer here, if you're curious on our original NANOG post on the whole situation: Hey NANOG, After receiving a BGPAlerter notification that one of our subnets (23.150.164.0/24) had been hijacked, I checked and noticed the prefix in question was missing RPKI. Assuming I had fat fingered something and butchered the ROA, I logged into ARIN and found that the prefix was missing from our resource list entirely,…

[flagged]

Re: ARIN Public Incident Report – 4.10 Misissuance Error

#17

Post by yoan9224 »

The transparency in this incident report is refreshing. "We relied on manual Excel-based verification and screwed up" - no corporate speak, just honest assessment.

What's scary is that IPv4 allocations are literally internet infrastructure. Having your /24 suddenly reassigned to someone else could be catastrophic for a business.

The fact that RPKI didn't catch this is interesting. The ROA was deleted along with the allocation, so from RPKI's perspective everything was valid. This is a good reminder that RPKI protects against hijacking but not against the RIR itself making mistakes.

Glad they're automating this. Anything involving copy-pasting IP ranges in Excel is an accident waiting to happen.

Re: ARIN Public Incident Report – 4.10 Misissuance Error

#18

Post by 12_throw_away »

mlhpdx wrote: post #10

So at least a good chunk of the Internet does indeed operate on a spreadsheet. Good to know.

All data begins life in a spreadsheet and dies in a spreadsheet. Automation is an illusion; databases are illusions. Only Excel is real.

Re: ARIN Public Incident Report – 4.10 Misissuance Error

#19

Post by Titan2189 »

galaxygate wrote: post #14

Affected customer here, if you're curious on our original NANOG post on the whole situation: Hey NANOG, After receiving a BGPAlerter notification that one of our subnets (23.150.164.0/24) had been hijacked, I checked and noticed the prefix in question was missing RPKI. Assuming I had fat fingered something and butchered the ROA, I logged into ARIN and found that the prefix was missing from our resource list entirely,…

The original report says

> The incorrect state persisted for approximately seven days before detection

However you're saying you've reached out "within a few minutes" ?

Re: ARIN Public Incident Report – 4.10 Misissuance Error

#20

Post by BlueMatt »

Titan2189 wrote: post #19
galaxygate wrote: post #14

Affected customer here, if you're curious on our original NANOG post on the whole situation: Hey NANOG, After receiving a BGPAlerter notification that one of our subnets (23.150.164.0/24) had been hijacked, I checked and noticed the prefix in question was missing RPKI. Assuming I had fat fingered something and butchered the ROA, I logged into ARIN and found that the prefix was missing from our resource list entirely,…

The original report says > The incorrect state persisted for approximately seven days before detection However you're saying you've reached out "within a few minutes" ?

It was re-allocated to the new/wrong ARIN customer for seven days before they started announcing it, at which point the OP detected the issue. Prior to that their prefix was routing to them just fine, just without RPKI protection.
Post reply on HN