Affected customer here, if you're curious on our original NANOG post on the whole situation:
Hey NANOG,
After receiving a BGPAlerter notification that one of our subnets (23.150.164.0/24) had been hijacked, I checked and noticed the prefix in question was missing RPKI. Assuming I had fat fingered something and butchered the ROA, I logged into ARIN and found that the prefix was missing from our resource list entirely, and had been reallocated to another organization and announced from their network. I created a ticket in ARIN and called immediately.
They confirmed that our subnet had been accidentally reallocated to another customer, and that they are currently working on returning it to us. After a couple hours, they told us the other organization will stop announcing the prefix, and WHOIS will be returned shortly.
I’m guessing there’s no way to prevent this kind of thing on our side if the RPKI ROA itself is removed along with the allocation? I’m planning on adding checks to look for missing ROAs (in addition to invalid/expiring ones), which I'm guessing would've caught this earlier.
Have any of you had anything like this happen with ARIN or another RIR? I’m especially curious what might have happened if we’d only noticed and reached out a few weeks later instead of within a few minutes.
The "incorrect state" being talked about is the IP prefix being misregistered in ARIN's database.
The "hijacking" happened later, when the IP prefix was announced via BGP by the registrant who it was incorrectly assigned to. Those are two different events.
It was re-allocated to the new/wrong ARIN customer for seven days before they started announcing it, at which point the OP detected the issue. Prior to that their prefix was routing to them just fine, just without RPKI protection.
Off-topic, but: I see you've got a green username (new account). How did you know this post was on the HN front page? ARIN's writeup doesn't mention your service by name. I looked it up out of curiosity from the CIDR they mentioned, before clicking over into the comments here. Unless you've got a regular HN account and just set up a new business-facing one for this?
I periodically see people showing up early in comment threads posted about things they've written or articles where they're the subject. Usually I figure they've got a Google alert or some other whatsit, or they've got something monitoring referers in their web traffic. But this is a case where neither would apply.
A couple of years ago ARIN increased their fees considerably - way higher than fees paid to RIPE for way less resources - and had a call with their management to express my frustration, not because I was paying from my pocket but because of the high discrepancy of the what they wanted to get and the quantity/quality of their services. Now I can see that their backbone services haven't really improved while their income for sure has.
On a sidenote, what I appreciate in both RIPE and ARIN is that you can have at least a proper discussion when you have valid arguments with their support teams.
All the RIRs are, in my experience, a very consistent and safe set of hands. This sort of things is vanishing rare to the point of borderline inconsequence by many providers of major internet infrastructure. The fact they care enough to take it seriously and publish shows how much they care about getting it right.
I just completed a fairly major reorganisation of resources with RIPE, and I’ve interacted with them for two decades, and my experience is they remain as steady and consistent as ever.
Sure, you may not like a particular policy at some moment, or may not agree with the charging structure at some point in time when it’s not advantageous to you, but they do at least do what they say and say what they do.
The transparency in this incident report is refreshing. "We relied on manual Excel-based verification and screwed up" - no corporate speak, just honest assessment.
What's scary is that IPv4 allocations are literally internet infrastructure. Having your /24 suddenly reassigned to someone else could be catastrophic for a business.
The fact that RPKI didn't catch this is interesting. The ROA was deleted along with the allocation, so from RPKI's perspective everything was valid. This is a good reminder that RPKI protects against hijacking but not against the RIR itself making mistakes.
Glad they're automating this. Anything involving copy-pasting IP ranges in Excel is an accident waiting to happen.
“Workflow” is probably a bit generous to describe how they probably use Excel.
Having worked at a mom and pop ISP a couple of decades ago where we used Excel to track a lot of things, I can see how this might have happened.
To actually know who is allocated what is ultimately just a list.
And when there are only a few people who edit the list (and probably no more than 1 person at a time) you can get by with even a plain text file, but Excel is quite a bit nicer as you can do things like filtering and sorting easily, maybe even some formulas to help with things.
Building a program backed by a database might be nice, but hard to justify when the manual system has never been a problem before.
They’ve probably been thinking for a while they should, but it’s just never been enough of a pain point for them to invest the effort.
Looks like they see this incident as justification that they need a system with hard coded rules and constraints, no more manual checking.
I can't remember a screw up by ARIN this bad before. I'm not too concerned about it. I understand that mistakes can happen. That said, I'm a little surprised at how easy it was to make this one.
I'm entirely unsurprised that this mistake involved an excel spreadsheet. Out of all the databases and IP management software they could be using which would have prevented this the first thing the employee reached for was excel. Almost every company I've worked for has employees using excel for data that would be better managed/stored/presented outside of an office document.
This reads like a joke, but I've known two DBAs who don't use database management tools beyond exporting whole tables to excel, making manual changes, and importing to update the tables. Scary stuff.
I've considered setting up an ASN and grabbing an IPv6 block for myself for a while now, but have never had the gumption, time, and funds at the same time.
Affected customer here, if you're curious on our original NANOG post on the whole situation:
Hey NANOG,
After receiving a BGPAlerter notification that one of our subnets (23.150.164.0/24) had been hijacked, I checked and noticed the prefix in question was missing RPKI. Assuming I had fat fingered something and butchered the ROA, I logged into ARIN and found that the prefix was missing from our resource list entirely, and had been reallocated to another organization and announced from their network. I created a ticket in ARIN and called immediately.
They confirmed that our subnet had been accidentally reallocated to another customer, and that they are currently working on returning it to us. After a couple hours, they told us the other organization will stop announcing the prefix, and WHOIS will be returned shortly.
I’m guessing there’s no way to prevent this kind of thing on our side if the RPKI ROA itself is removed along with the allocation? I’m planning on adding checks to look for missing ROAs (in addition to invalid/expiring ones), which I'm guessing would've caught this earlier.
Have any of you had anything like this happen with ARIN or another RIR? I’m especially curious what might have happened if we’d only noticed and reached out a few weeks later instead of within a few minutes.
The original report says
> The incorrect state persisted for approximately seven days before detection
However you're saying you've reached out "within a few minutes" ?
The "incorrect state" being talked about is the IP prefix being misregistered in ARIN's database.
The "hijacking" happened later, when the IP prefix was announced via BGP by the registrant who it was incorrectly assigned to. Those are two different events.
It was re-allocated to the new/wrong ARIN customer for seven days before they started announcing it, at which point the OP detected the issue. Prior to that their prefix was routing to them just fine, just without RPKI protection.
Off-topic, but: I see you've got a green username (new account). How did you know this post was on the HN front page? ARIN's writeup doesn't mention your service by name. I looked it up out of curiosity from the CIDR they mentioned, before clicking over into the comments here. Unless you've got a regular HN account and just set up a new business-facing one for this?
I periodically see people showing up early in comment threads posted about things they've written or articles where they're the subject. Usually I figure they've got a Google alert or some other whatsit, or they've got something monitoring referers in their web traffic. But this is a case where neither would apply.
> Unless you've got a regular HN account and just set up a new business-facing one for this?
This is likely; I can't imagine a regular HN user would appreciate having their subnet publicly available in their comment history.
Maybe some college of theirs on HN recognized the story and shared it with them.
A couple of years ago ARIN increased their fees considerably - way higher than fees paid to RIPE for way less resources - and had a call with their management to express my frustration, not because I was paying from my pocket but because of the high discrepancy of the what they wanted to get and the quantity/quality of their services. Now I can see that their backbone services haven't really improved while their income for sure has.
On a sidenote, what I appreciate in both RIPE and ARIN is that you can have at least a proper discussion when you have valid arguments with their support teams.
Now ARIN is much cheaper than RIPE for small entities.
fee schedules FYI
- ARIN 2026 PDF: https://www.arin.net/resources/fees/images/2026feeschedule.p...
- RIPE 2026 : https://www.ripe.net/membership/payment/
Enthusiasts, trainees and small orgs are paying a lot more with RIPE.
All the RIRs are, in my experience, a very consistent and safe set of hands. This sort of things is vanishing rare to the point of borderline inconsequence by many providers of major internet infrastructure. The fact they care enough to take it seriously and publish shows how much they care about getting it right.
I just completed a fairly major reorganisation of resources with RIPE, and I’ve interacted with them for two decades, and my experience is they remain as steady and consistent as ever.
Sure, you may not like a particular policy at some moment, or may not agree with the charging structure at some point in time when it’s not advantageous to you, but they do at least do what they say and say what they do.
The transparency in this incident report is refreshing. "We relied on manual Excel-based verification and screwed up" - no corporate speak, just honest assessment.
What's scary is that IPv4 allocations are literally internet infrastructure. Having your /24 suddenly reassigned to someone else could be catastrophic for a business.
The fact that RPKI didn't catch this is interesting. The ROA was deleted along with the allocation, so from RPKI's perspective everything was valid. This is a good reminder that RPKI protects against hijacking but not against the RIR itself making mistakes.
Glad they're automating this. Anything involving copy-pasting IP ranges in Excel is an accident waiting to happen.
I like how frank the report is, no sugarcoating. "We relied on manual error prone verification and made a mistake. We have to automate the process."
As ARIN block owner this situation is kinda scary but reading this actually makes me think it's less likely to happen again .
The road to automation is always full of outages.
I'm curious how these fellas took something like IP block allocation and turned it into an Excel based workflow.
“Workflow” is probably a bit generous to describe how they probably use Excel.
Having worked at a mom and pop ISP a couple of decades ago where we used Excel to track a lot of things, I can see how this might have happened.
To actually know who is allocated what is ultimately just a list.
And when there are only a few people who edit the list (and probably no more than 1 person at a time) you can get by with even a plain text file, but Excel is quite a bit nicer as you can do things like filtering and sorting easily, maybe even some formulas to help with things.
Building a program backed by a database might be nice, but hard to justify when the manual system has never been a problem before.
They’ve probably been thinking for a while they should, but it’s just never been enough of a pain point for them to invest the effort.
Looks like they see this incident as justification that they need a system with hard coded rules and constraints, no more manual checking.
You don't find this part
> We have to automate the process.
to be ominous?
I don’t. The report says part of this process relied on flat files and spreadsheets. Automating that with software is a good idea.
“Automate the process” doesn’t mean feeding everything to an LLM.
Certificate issuance was once only possible manually.
Domains too, well into the 90s.
I can't remember a screw up by ARIN this bad before. I'm not too concerned about it. I understand that mistakes can happen. That said, I'm a little surprised at how easy it was to make this one.
I'm entirely unsurprised that this mistake involved an excel spreadsheet. Out of all the databases and IP management software they could be using which would have prevented this the first thing the employee reached for was excel. Almost every company I've worked for has employees using excel for data that would be better managed/stored/presented outside of an office document.
So at least a good chunk of the Internet does indeed operate on a spreadsheet. Good to know.
All data begins life in a spreadsheet and dies in a spreadsheet. Automation is an illusion; databases are illusions. Only Excel is real.
This reads like a joke, but I've known two DBAs who don't use database management tools beyond exporting whole tables to excel, making manual changes, and importing to update the tables. Scary stuff.
I've considered setting up an ASN and grabbing an IPv6 block for myself for a while now, but have never had the gumption, time, and funds at the same time.