Blog of network observability company Kentik, with analyses of BGP incidents, internet routing and outages, plus company job posts.
HN links it for BGP and internet routing analysis, such as the history of major BGP incidents, and Kentik shows up in hiring threads.
Kentik | Infrastructure Engineering Manager | Remote (Eastern US or EU time-zones preferred) | Full Time
Kentik is Network Observability. Our saas and onprem platform ingests massive amounts of network and infrastructure data - trillions of record per day - from routers, hosts and clouds to give network engineers, operators and SREs a full view of their networks. Check out https://kentik.com for more.
OPEN ROLES
- We're looking for a technical engineering manager to help run and scale our infrastructure and tooling team. The team works on a lot of the pieces that power Kentik's systems: our API gateway infrastructure, production network and VPN, CI/CD, SQL datastores, and more.
If you're an experienced engineer looking to try managing a team, or you're now on the management side of the Engineer/Manager Pendulum, you could be a great fit!
Contact me at stefan@kentik.com
- We’re also looking for a front-end-focused fullstack engineer for our new Rapid Application Development team.
This team is charged with exploring new product and feature ideas by experimenting, gathering feedback, and iterating quickly. If the flexibility to try out many concepts and find out what works appeals to you, apply! https://www.kentik.com/careers/jobs/?gh_jid=4019954005
- More postings here: https://www.kentik.com/careers/#postings
We've been remote-first for years and offer deep problems to solve in a work-life balanced environment with competitive pay, stock, benefits, unlimited PTO and…
There are many discussion threads at the RIPE Network Coordination Centre (NCC) Services Working Group (ncc-services-wg) mailing list with a lot of direct and ancillary information [0].
There is also a detailed factual exposition of the technical changes forced by Russia since 2014 in Crimea and later Kherson "Rerouting of Kherson follows familiar gameplan" [1].
In [3] below Viktoriia Opanasiuk writes:
"responding on «As this only affects LIRs, I doubt there will be significant participation by non-LIRs» I must say that it affects all internet service providers that use IP resources. In Ukraine there are more than 5 000 registered ISPs, but not all of them are LIRs.
The situation that we have now is that when russian troops withdraw from our territory, they dismantle and take with them not only washing machines, but also telecom equipment, TV- and radio-transmitters, leaving people who live there without any information channels and means of communication. We have to re-build telecom and internet infrastructure, and yes, we could bring new equipment there, but if the IP addresses of local Internet Providers will be transferred to other parties, we will not be able to restore even our critical infrastructure there, and there is no place to
get new IPv4 addresses.
Therefore any solutions that involve allowing transfers now and somehow reverting them later are not solving this immediate problem that we have."
The threads with most focus seem to be:
"Protecting Resource…
> (The CTO was roaming out-of-country, had zero bars and thought nothing of it... how they had no idea an enterprise-risking update was scheduled, we'll never know)
How does one know ahead of time if any particular change is "enterprise-risking"? It appeared to be a fairly routine set of changes that were going just fine:
> The report summary says that in the weeks leading up to the outage, Rogers was undergoing a seven-phase process to upgrade its network. The outage occurred during the sixth's phase of the upgrade.
* https://www.cbc.ca/news/politics/rogers-outage-human-error-system-deficiencies-1.7255641
It turns out that they self-DoSed certain components:
> Staff at Rogers caused the shutdown, the report says, by removing a control filter that directed information to its appropriate destination.
> Without the filter in place, a flood of information was sent into Rogers' core network, overloading and crashing the system within minutes of the control filter being removed.
* Ibid
> In a letter to the CRTC, Rogers stated that the deletion of a routing filter on its distribution routers caused all possible routes to the internet to pass through the routers, exceeding the capacity of the routers on its core network.
* https://en.wikipedia.org/wiki/2022_Rogers_Communications_outage#Postmortem
> Rogers staff removed the Access Control ListFootnote 5 policy filter from the configuration of the distribution routers. This consequently resulted in a flood of IP routing…
> Defense Digital Service (DDS) authorized a pilot effort advertising DoD Internet Protocol (IP) space using Border Gateway Protocol (BGP). This pilot will assess, evaluate and prevent unauthorized use of DoD IP address space. Additionally, this pilot may identify potential vulnerabilities. This is one of DoD’s many efforts focused on continually improving our cyber posture and defense in response to advanced persistent threats. We are partnering throughout DoD to ensure potential vulnerabilities are mitigated.
Interesting, seems an effort to find out who was abusing ranges that were exclusively allowed or disallowed based on the ranges. Malware that tries to look like something else that uses a state level IP range to evade blocking, or check for blocks.[1]
>I interpret this to mean that the objectives of this effort are twofold. First, to announce this address space to scare off any would-be squatters, and secondly, to collect a massive amount of background internet traffic for threat intelligence.
>On the first point, there is a vast world of fraudulent BGP routing out there. As I’ve documented over the years, various types of bad actors use unrouted address space to bypass blocklists in order to send spam and other types of malicious traffic.
Cloudflare example shows how much traffic some of these ranges that are included/excluded have when turned on.
>On the second, there is a lot of background noise that can be scooped up when announcing large ranges of IPv4…
I recently fired up an LTE router using a T-Mobile SIM. Within an hour I had two different WAN IPs that belong to AS749 DOD NIC (in 33.79.135.0/24 & 21.140.100.0/24).
Again, those were my WAN IPs. My public IPs were in T-Mobile's AS21928 network space.
And for clarity, T-Mobile likes using DoD IPs for it's internal routing. It's safe as long as DoD never starts advertising it's IPs as publicly routable. And if DoD does... beats me.
Not really related: The DoD consolidated it's addresses into the previously unused AS749 in 2021, after a weird dead FL company started publicly advertising DoD addresses within it's own ASN.
The move evokes the 2021 mystery surrounding AS8003, an ASN associated with a defunct Florida company that started announcing all of the formerly unused IPv4 address space belonging to the US Department of Defense. Later that year, the DoD quietly moved the address space to a formerly unused ASN, AS749, which continues to announce more IPv4 address space than anything else in the history of the internet. ref: https://www.kentik.com/analysis/aws-announces-super-aggregates-massively-expands-bgp-footprint/
dead fl biz ref: https://www.kentik.com/blog/the-mystery-of-as8003/
Hi, Avi Freedman here, CEO of Kentik (formerly CloudHelix). Thanks for the mention, Ryan... And for being a human vulnerability scanner for netaxs back in the 90s :)
Re: data backend and analytics - happy to discuss... A poster we did earlier this year is @ http://avi.net/flocon-hydrasql.pdf or the marketed-up version is a white paper on the site - https://www.kentik.com/wp-content/uploads/2015/06/Kentik-KDE-whitepaper.pdf
We've considered OSSing the backend (what sits behind postgres via FDWs) but it'd take require expanding the team working on it beyond the current ~ 1.5 FTEs. Maybe post-B...
Another note of potential interest: We applied to YC Winter 2014 and didn't get an interview. Same team, last minute sub, not everyone was yet full time, silly idea (though we did mention as an alt what we started Kentik to do). Lessons as are noted in many places: 1) Read the reqs, 2) Work on proposal, 3) Don't stop because you don't get {in,an interview}.