Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ARIN Public Incident Report – 4.10 Misissuance Error

arin.net

Post reply on HN
Print view View original

31–40 of 41 posts

Re: ARIN Public Incident Report – 4.10 Misissuance Error

#31

Post by galaxygate »

thaumaturgy wrote: post #16
galaxygate wrote: post #14

Affected customer here, if you're curious on our original NANOG post on the whole situation: Hey NANOG, After receiving a BGPAlerter notification that one of our subnets (23.150.164.0/24) had been hijacked, I checked and noticed the prefix in question was missing RPKI. Assuming I had fat fingered something and butchered the ROA, I logged into ARIN and found that the prefix was missing from our resource list entirely,…

[flagged]

Yup, another engineer that works on our team mentioned seeing the report here, I figured I'd make an account to add some further context

Re: ARIN Public Incident Report – 4.10 Misissuance Error

#32

Post by rmoriz »

icedchai wrote: post #30
rmoriz wrote: post #29

Earlier quoted context omitted.

A single AS resource and a single PI assignment cost more than the ARIN fee.

Are you sure? For RIPE I see a 50 ASN plus 75 euro PI fee. ARIN is $275. Maybe I’m looking at it wrong. It’s cheaper as a hobbyist to use a RIPE LIR. Even in the US. That’s what I’ve been doing for years.

afaik that's +VAT and also for LIRs only. LIRs apply markup, see https://www.lir.services/lir-sponsoring/ they charge 200€ per resource, so ASN + PI would be at last 400€/year that's way above the price of ARIN and you have a middleman.

You must have a sponsoring LIR for your resources or become a LIR yourself. The only exception is LEGACY resources (IPv4, no ASN) but that's a different story.

Re: ARIN Public Incident Report – 4.10 Misissuance Error

#33

Post by icedchai »

rmoriz wrote: post #32
icedchai wrote: post #30

Earlier quoted context omitted.

Are you sure? For RIPE I see a 50 ASN plus 75 euro PI fee. ARIN is $275. Maybe I’m looking at it wrong. It’s cheaper as a hobbyist to use a RIPE LIR. Even in the US. That’s what I’ve been doing for years.

afaik that's +VAT and also for LIRs only. LIRs apply markup, see https://www.lir.services/lir-sponsoring/ they charge 200€ per resource, so ASN + PI would be at last 400€/year that's way above the price of ARIN and you have a middleman. You must have a sponsoring LIR for your resources or become a LIR yourself. The only exception is LEGACY resources (IPv4, no ASN) but that's a different story.

There are more competitive LIRs out there. Example: https://lagrange.cloud/products/lir

It’s also cheaper for me because I have legacy ARIN space. All I really needed was an ASN. The LIR gives me some PA v6 space for cheap, too.

Re: ARIN Public Incident Report – 4.10 Misissuance Error

#34

Post by rmoriz »

icedchai wrote: post #33
rmoriz wrote: post #32

Earlier quoted context omitted.

afaik that's +VAT and also for LIRs only. LIRs apply markup, see https://www.lir.services/lir-sponsoring/ they charge 200€ per resource, so ASN + PI would be at last 400€/year that's way above the price of ARIN and you have a middleman. You must have a sponsoring LIR for your resources or become a LIR yourself. The only exception is LEGACY resources (IPv4, no ASN) but that's a different story.

There are more competitive LIRs out there. Example: https://lagrange.cloud/products/lir It’s also cheaper for me because I have legacy ARIN space. All I really needed was an ASN. The LIR gives me some PA v6 space for cheap, too.

Okay, but that is not enough to operate independently. PA v6 is another dependency. With ARIN you get your personal IPv6 assignment.

Re: ARIN Public Incident Report – 4.10 Misissuance Error

#35

Post by stefan_ »

jonathanlydall wrote: post #22
stefan_ wrote: post #8

Earlier quoted context omitted.

I'm curious how these fellas took something like IP block allocation and turned it into an Excel based workflow.

“Workflow” is probably a bit generous to describe how they probably use Excel. Having worked at a mom and pop ISP a couple of decades ago where we used Excel to track a lot of things, I can see how this might have happened. To actually know who is allocated what is ultimately just a list. And when there are only a few people who edit the list (and probably no more than 1 person at a time) you can get by with even a p…

It's ARIN, this is essentially their only job

Re: ARIN Public Incident Report – 4.10 Misissuance Error

#36

Post by mmooss »

stefan_ wrote: post #8
progbits wrote: post #3

I like how frank the report is, no sugarcoating. "We relied on manual error prone verification and made a mistake. We have to automate the process." As ARIN block owner this situation is kinda scary but reading this actually makes me think it's less likely to happen again .

I'm curious how these fellas took something like IP block allocation and turned it into an Excel based workflow.

The world's financial systems run on Excel, to a great extent.

I'm more surprised that a single person, apparently without seniority, could delete a block. IME deleting user data is usually a significant event; an IP block would especially be a big deal, especially for the IP block issuers. From the OP:

> RSD has implemented additional process controls that require a dual review for all ticketing type workflows that include a network delete.

> Only a limited set of experienced analysts are permitted to perform this function.

Great that they didn't blame the person who deleted it. ARIN seems to have put them in position where a failure was likely, eventually. Without any inside knowledge, I'd hope the culture would have any engineer leary about pressing that button without a second set of eyes reviewing it carefully and without clear authorization; I don't imagine they delete many blocks each day so it shouldn't interfere with productivity.

Re: ARIN Public Incident Report – 4.10 Misissuance Error

#37

Post by icedchai »

rmoriz wrote: post #34
icedchai wrote: post #33

Earlier quoted context omitted.

There are more competitive LIRs out there. Example: https://lagrange.cloud/products/lir It’s also cheaper for me because I have legacy ARIN space. All I really needed was an ASN. The LIR gives me some PA v6 space for cheap, too.

Okay, but that is not enough to operate independently. PA v6 is another dependency. With ARIN you get your personal IPv6 assignment.

For a hobbyist, the difference is academic. You can announce PA space with your own ASN, which is what I do. If I change LIRs I’ll have to renumber my IPv6 space.

Re: ARIN Public Incident Report – 4.10 Misissuance Error

#38

Post by patmorgan23 »

autoexec wrote: post #5

I can't remember a screw up by ARIN this bad before. I'm not too concerned about it. I understand that mistakes can happen. That said, I'm a little surprised at how easy it was to make this one. I'm entirely unsurprised that this mistake involved an excel spreadsheet. Out of all the databases and IP management software they could be using which would have prevented this the first thing the employee reached for was ex…

From the nanog thread it seemed like the IP allocations for the IPv6 transition space (4.10) was the only space using this manual Excel process. That's probably how they initially started managing these allocations with the intention to build it into their automated systems but hadn't gotten around to it. And it sounds like they're prioritizing that work now, and have implemented an additional lay of checks in the mean time.

This is a really big egg on face moment for ARIN, but it sounds like they are responding appropriately.

Re: ARIN Public Incident Report – 4.10 Misissuance Error

#39

Post by bigbuppo »

stefan_ wrote: post #8
progbits wrote: post #3

I like how frank the report is, no sugarcoating. "We relied on manual error prone verification and made a mistake. We have to automate the process." As ARIN block owner this situation is kinda scary but reading this actually makes me think it's less likely to happen again .

I'm curious how these fellas took something like IP block allocation and turned it into an Excel based workflow.

They've improved over the decades. At one point the authoritative database was a physical paper notebook.

Re: ARIN Public Incident Report – 4.10 Misissuance Error

#40

Post by rmoriz »

icedchai wrote: post #37
rmoriz wrote: post #34

Earlier quoted context omitted.

Okay, but that is not enough to operate independently. PA v6 is another dependency. With ARIN you get your personal IPv6 assignment.

For a hobbyist, the difference is academic. You can announce PA space with your own ASN, which is what I do. If I change LIRs I’ll have to renumber my IPv6 space.

Companies offering LIR services to hobbyists are probably not going to stay in business forever, as many of them are 1 person companies, too. Also keep in mind that they may change pricing. I understand, that with IPv6 the numbering strategy is almost always automatic and a renumbering can be done in a couple of hours, but it's still an inconvenience, especially when you have to update a lot of AAAA records.

I really think that when you start to operate an AS that you should have a direct RIR membership. And as mentioned above, RIPE has a higher financial entry barrier. I remember they had an object volume based pricing scheme 15 years ago, just like ARIN still has.

Post reply on HN