Affected customer here, if you're curious on our original NANOG post on the whole situation:
Hey NANOG,
After receiving a BGPAlerter notification that one of our subnets (23.150.164.0/24) had been hijacked, I checked and noticed the prefix in question was missing RPKI. Assuming I had fat fingered something and butchered the ROA, I logged into ARIN and found that the prefix was missing from our resource list entirely, and had been reallocated to another organization and announced from their network. I created a ticket in ARIN and called immediately.
They confirmed that our subnet had been accidentally reallocated to another customer, and that they are currently working on returning it to us. After a couple hours, they told us the other organization will stop announcing the prefix, and WHOIS will be returned shortly.
I’m guessing there’s no way to prevent this kind of thing on our side if the RPKI ROA itself is removed along with the allocation? I’m planning on adding checks to look for missing ROAs (in addition to invalid/expiring ones), which I'm guessing would've caught this earlier.
Have any of you had anything like this happen with ARIN or another RIR? I’m especially curious what might have happened if we’d only noticed and reached out a few weeks later instead of within a few minutes.
Off-topic, but: I see you've got a green username (new account). How did you know this post was on the HN front page? ARIN's writeup doesn't mention your service by name. I looked it up out of curiosity from the CIDR they mentioned, before clicking over into the comments here. Unless you've got a regular HN account and just set up a new business-facing one for this?
I periodically see people showing up early in comment threads posted about things they've written or articles where they're the subject. Usually I figure they've got a Google alert or some other whatsit, or they've got something monitoring referers in their web traffic. But this is a case where neither would apply.
A couple of years ago ARIN increased their fees considerably - way higher than fees paid to RIPE for way less resources - and had a call with their management to express my frustration, not because I was paying from my pocket but because of the high discrepancy of the what they wanted to get and the quantity/quality of their services. Now I can see that their backbone services haven't really improved while their income for sure has.
On a sidenote, what I appreciate in both RIPE and ARIN is that you can have at least a proper discussion when you have valid arguments with their support teams.
All the RIRs are, in my experience, a very consistent and safe set of hands. This sort of things is vanishing rare to the point of borderline inconsequence by many providers of major internet infrastructure. The fact they care enough to take it seriously and publish shows how much they care about getting it right.
I just completed a fairly major reorganisation of resources with RIPE, and I’ve interacted with them for two decades, and my experience is they remain as steady and consistent as ever.
Sure, you may not like a particular policy at some moment, or may not agree with the charging structure at some point in time when it’s not advantageous to you, but they do at least do what they say and say what they do.
The transparency in this incident report is refreshing. "We relied on manual Excel-based verification and screwed up" - no corporate speak, just honest assessment.
What's scary is that IPv4 allocations are literally internet infrastructure. Having your /24 suddenly reassigned to someone else could be catastrophic for a business.
The fact that RPKI didn't catch this is interesting. The ROA was deleted along with the allocation, so from RPKI's perspective everything was valid. This is a good reminder that RPKI protects against hijacking but not against the RIR itself making mistakes.
Glad they're automating this. Anything involving copy-pasting IP ranges in Excel is an accident waiting to happen.
I can't remember a screw up by ARIN this bad before. I'm not too concerned about it. I understand that mistakes can happen. That said, I'm a little surprised at how easy it was to make this one.
I'm entirely unsurprised that this mistake involved an excel spreadsheet. Out of all the databases and IP management software they could be using which would have prevented this the first thing the employee reached for was excel. Almost every company I've worked for has employees using excel for data that would be better managed/stored/presented outside of an office document.
I've considered setting up an ASN and grabbing an IPv6 block for myself for a while now, but have never had the gumption, time, and funds at the same time.
Hey NANOG,
After receiving a BGPAlerter notification that one of our subnets (23.150.164.0/24) had been hijacked, I checked and noticed the prefix in question was missing RPKI. Assuming I had fat fingered something and butchered the ROA, I logged into ARIN and found that the prefix was missing from our resource list entirely, and had been reallocated to another organization and announced from their network. I created a ticket in ARIN and called immediately.
They confirmed that our subnet had been accidentally reallocated to another customer, and that they are currently working on returning it to us. After a couple hours, they told us the other organization will stop announcing the prefix, and WHOIS will be returned shortly.
I’m guessing there’s no way to prevent this kind of thing on our side if the RPKI ROA itself is removed along with the allocation? I’m planning on adding checks to look for missing ROAs (in addition to invalid/expiring ones), which I'm guessing would've caught this earlier.
Have any of you had anything like this happen with ARIN or another RIR? I’m especially curious what might have happened if we’d only noticed and reached out a few weeks later instead of within a few minutes.
I periodically see people showing up early in comment threads posted about things they've written or articles where they're the subject. Usually I figure they've got a Google alert or some other whatsit, or they've got something monitoring referers in their web traffic. But this is a case where neither would apply.
On a sidenote, what I appreciate in both RIPE and ARIN is that you can have at least a proper discussion when you have valid arguments with their support teams.
I just completed a fairly major reorganisation of resources with RIPE, and I’ve interacted with them for two decades, and my experience is they remain as steady and consistent as ever.
Sure, you may not like a particular policy at some moment, or may not agree with the charging structure at some point in time when it’s not advantageous to you, but they do at least do what they say and say what they do.
What's scary is that IPv4 allocations are literally internet infrastructure. Having your /24 suddenly reassigned to someone else could be catastrophic for a business.
The fact that RPKI didn't catch this is interesting. The ROA was deleted along with the allocation, so from RPKI's perspective everything was valid. This is a good reminder that RPKI protects against hijacking but not against the RIR itself making mistakes.
Glad they're automating this. Anything involving copy-pasting IP ranges in Excel is an accident waiting to happen.
As ARIN block owner this situation is kinda scary but reading this actually makes me think it's less likely to happen again .
> We have to automate the process.
to be ominous?
“Automate the process” doesn’t mean feeding everything to an LLM.
I'm entirely unsurprised that this mistake involved an excel spreadsheet. Out of all the databases and IP management software they could be using which would have prevented this the first thing the employee reached for was excel. Almost every company I've worked for has employees using excel for data that would be better managed/stored/presented outside of an office document.