Ivan Pepelnjak writes about network engineering, covering IPv6, BGP, NAT, data center design, and network automation with Ansible.
HN commenters treat it as a trusted networking resource, describe Pepelnjak as a networking expert, and link it for IPv6, routing, and automation topics.
> We want to dual bound the network connections to increase performance and reliability. This will allow us to take routers out of service during low traffic times, for example to restart them after a software upgrade.
does not really agree with
> Each of the two physical network connections will connect to a different top of rack router.
Sure, you can do it with something like MLAG, but that's really just moving your SPOF to somewhere else (the router software running MLAG). Router software being super buggy, I wouldn't rely on MLAG being up at all times.
> N1 Which router should we purchase?
Pick your favorite. For what you're looking for here, everything is largely using the same silicon (broadcom chipsets).
> N2 How do we interconnect the routers while keeping the network simple and fast?
Don't fall into the trap of extending vlans everywhere. You should definitely be routing (not switching) between different routers. You can read through http://blog.ipspace.net/ for some info on layer 3 only datacenter networks.
You'd want to use something like OSPF or BGP between routers.
> N3 Should we have a separate network for Ceph traffic?
Yes, if you want your Ceph cluster to remain usable during rebuilds. Ceph will peg the internal network during any sort of rebuild event.
> N4 Do we need an SDN compatible router or can we purchase something more affordable?
You probably don't need SDN unless you actually have a SDN use case in mind. I'd bet you can get away…
> Also, NAT is desirable for security/network isolation reasons […]
This is security theatre. People have been saying that NAT is not a security feature for over a decade:
* https://blog.ipspace.net/2011/12/is-nat-security-feature/
but the message still has not sunk in. The "Zero Trust" paper was published by John Kindervag in 2010:
* https://media.paloaltonetworks.com/documents/Forrester-No-More-Chewy-Centers.pdf
Most modern attacks start from a compromised internal host (e.g., from phishing), or through stolen credentials via a remote access method. The above is "castle-and-moat" thinking that tends to have weaker internal controls because it is thought the internal network is "hidden" from the dangerous outside network.
Set your firewall to default deny, then add a rule for allow outgoing connections, followed by only allow incoming connections if they are replies. For most machines (and networks), most of the time, this is what's needed: the above is applicable for both IPv6 and IPv4 (with or without NAT).
The protection comes from filtering (generally) and stateful packet inspection, not from hiding addresses.
> […] and having no distinction between a local IP and a public IP has a lot of disadvantages.
Just because something has a global addresses does not mean global reachability (see default deny above). Further you can layout your IPv6 address plan so that you can tell at a glance if hosts are externally accessible. Using a /48 a basis, you break out…
Also, IPV6 has far more scalable systems for dynamic address allocation compared to IPv4.
SLAAC scales really, really well, and is fairly stateless compared to DHCPv4 or V6.
Also, slaac allows one to easily change the global prefix of a vm/host if it lives on another layer 3 network.
A good example of this:
you have two seperate datacenter networks based on an EVPN-VXLAN solution, and you do no want to stretch layer 2 across both datacenters because stretching layer 2 is a terrible, terrible idea[0].
Before IPv6, moving VM;s across datacenters which have different public ip space was a major pain for two reasons:
- IPV4 has no concept of using multiple addresses per interface without it having unexpected bahaviour on a host.
- address management at scale is a major PITA because DHCP is not scalable.
How to solve this with IPv6? It's fairly easy:
- Use slaac + prefix delegation of your globally unique address space (which is different per Datacenter).
- Use a different (site local) address to reach the VM for management purposes. Because this address is not globally routable, you are sure it will never leak into the greater internet and be reachable from the outside world.
- If you are using anycast, you can easily announce this prefix to upstream BGP neighbours because you can use BGP Neighbour Autodiscovery[1].
Trying to do this with Ipv4 will result in a mess of administration, not to mention have to do some clever technical hacks to make it work on some…
> Exactly. IPv6 was developed in the ivory towers where it was still assumed that everyone wanted to be a full participant of the internet.
IPv6 was developed in the open on mailing lists that anyone could subscribe to:
The criteria presented here were culled from several sources,
including "IP Version 7" [1], "IESG Deliberations on Routing and
Addressing" [2], "Towards the Future Internet Architecture" [3], the
IPng Requirements BOF held at the Washington D.C. IETF Meeting in
December of 1992, the IPng Working Group meeting at the Seattle IETF
meeting in March 1994, the discussions held on the Big-Internet
mailing list (big-internet-at-munnari.oz.au, send requests to join to
big-internet-request-at-munnari.oz.au), discussions with the IPng Area
Directors and Directorate, and the mailing lists devoted to the
individual IPng efforts.
* https://datatracker.ietf.org/doc/html/rfc1726
Just like all current IETF discussions are in the open and free for all to participate. If you don't like the direction things are going in participate: as Gandhi did (not) say, “Be the change you want to see in the world.”
One of the co-authors on that RFC worked at BBN: you know, the folks that actually built the first the routers (IMPs) that created the ARPA/Internet in the first place. I would hazard to guess they have know something about network operations.
* https://www.goodreads.com/book/show/281818.Where_Wizards_Stay_Up_Late
> But the…
>https://blog.ipspace.net/2011/12/is-nat-security-feature/ >>Basic NAT (as defined in RFC 2663) performs just the IP address translation (one inside host to one IP address in the NAT pool). The moment the inside host starts a session through the NAT, it becomes fully exposed to the outside world.
This is a lie. A "session through the NAT" does not really expose the host to the outside world, because in 99% of the cases this is a TCP session, and the NAT machine would drop all "out of order" packets.
>Most modern attacks start from a compromised internal host (e.g., from phishing), or through stolen credentials via a remote access method.
Your statement is a perfect example of https://en.wikipedia.org/wiki/Survivorship_bias.
Most modern attacks start from an internal host exactly because NAT makes external attacks infeasible for the majority of scenarios.
>Set your firewall to default deny, then add a rule for allow outgoing connections, followed by only allow incoming connections if they are replies.
What about I don't do it, and the system is still _automatically_ secure, because NAT does exactly that while being _required_ for the system to work.
>See Figure A-5
LOL. What about I don't see any figures, and the system still works and is secure for the 99% of the cases.
I'm the Snabb Switch originator.
The project is new: I and other open source contributors are currently under contract to build a Network Functions Virtualization platform for Deutsche Telekom's TeraStream project [1] [2]. This is called Snabb NFV [3] and it's going to be totally open source and integrated with OpenStack.
Currently we are doing a lot of virtualization work from the "outside" of the VM: implementing Intel VMDq hardware acceleration and providing zero-copy Virtio-net to the VMs. So the virtual machine will see normal Virtio-net but we will make that operate really fast.
Inside the VMs we can either access a hardware NIC directly (via IOMMU "PCI passthrough") or we can write a device driver for the Virtio-net device.
So, early days, first major product being built, and lots of potential both inside and outside VMs, lots of fantastic products to build with nobody yet building them :-)
[1] http://blog.ipspace.net/2013/11/deutsche-telekom-terastream-designed.html
[2] https://ripe67.ripe.net/archives/video/3/
[3] https://github.com/SnabbCo/snabbswitch/blob/snabbnfv-readme/src/designs/nfv/README.md