(SWM) ACI Topology and Hardware part 1

Oh yeah, 2 part series because I like to compartmentalize. On these Study-With-Me posts I will usually list questions and the material will be in the answers - I may throw in some notes at the end. Part 1 will cover Topology and Part 2 will cover Hardware.

As always, I’m following my blueprint. And all acronyms are documented at the bottom under the Glossary.

Study time:

What is an ACI fabric?

1. loaded question. ACI consists of a Clos architecture (Leaf & Spine) + APIC controllers.

What makes a Clos architecture?

1. Every SPINE switch connects to every LEAF switch.
   • Remote leafs are an exception; so might be extended fabrics.
2. LEAF to LEAF connections are not allowed.
3. SPINE to SPINE connections are not allowed.
   • (I believe interfaces get disabled if they connect in either of those two ways)

What is an APIC?

1. It’s the fabric controller.
   • Deployed in a cluster with a minimum of 3 servers.
   • Cluster size depends on fabric size.
2. Sets up and manages the config that gets pushed to the switches.
3. Monitors the fabric’s health.
4. REST API for fabric management.
5. Provides 3rd-party integration to the fabric (LBs, FWs, vCenter, etc.).

What is a LEAF switch?

1. It’s an edge, Top-of-Rack switch.
   • It connects to traditional Ethernet devices (servers, FWs, routers, etc.).
2. Connects the APICs to the fabric.
3. First device discovered in the fabric.
   • if directly connected to an APIC.
4. Loopback interface acts as a VTEP/PTEP.
5. Routes/bridges traffic.
6. Applies network policy to the traffic.

What is a SPINE switch?

1. Interconnects LEAF switches.
2. Stores endpoint-to-VTEP information.
3. Can connect to external devices for Multi-Site connectivity.
   • only situation where I’ve seen them connected to something other than a LEAF - probably Multi-Pod as well.

What’s a VTEP?

1. A VTEP or VXLAN Tunnel Endpoint is the IP used to encapsulate the traffic over the fabric.

What is VXLAN?

1. Transport encapsulation mechanism used by ACI to move packets through the fabric.
   • L2 over L3.
   • if endpoint traffic must traverse over a spine it probably is encapsulated on a VXLAN frame.
2. Uses VNI to identify the network segments.
   • 24-bits (16 million segments)

VXLAN Header from ACI COOKBOOK

Does ACI support vPC?

1. Yes.
2. Switches in a vPC pair must be same gen model.

Can ACI connect to network devices external to the fabric?

1. Yes.
2. L3Outs are used to connect to devices over an L3 protocol.
   • Protocols: BGP, OSPF, EIGRP, and static routes.
   • Supports SVI, Routed Interfaces, Routed Sub-Interfaces.
3. L2 extension outside the fabric is also supported.
4. SPINE L3Outs run MP-BGP EVPN.
   • VXLAN also possible for GOLF.
5. LEAF L3Outs run VRF-Lite (configured under Tenant).
   • not recommended to configure directly connected servers over an L3Out.

Can a FEX be connected to a Leaf?

1. Yes.
2. No L3Out support.
3. Upgrades are more difficult because the FEX is not aware the Leaf it connects to is unavailable.
4. There are some VLAN restrictions per FEX port (20 max?).

Server could be connected with link aggregation. For some reason I drew an active standby topology.

Extra-Notes:

• ACI forwarding is based on VXLAN overlay with IS-IS in the underlay.
• ACI maintains a mapping database with endpoints MAC/IP address info.
  • COOP database.
• Leaf nodes = Identified by VTEP/PTEP
• Spine nodes = Identified by Proxy TEP
  • Proxy TEP is the Anycast IP used for lookups.
• L2 traffic carries VNID to identify Bridge Domains.
• L3 traffic carries VNID to identify VRF.
• VXLAN traffic is carried over Overlay-1 VRF.
  • Overlay-1 is part of the Infra Tenant
  • Contains /32 IP routes to all VTEPs/vPC VIPs/APICs/Spine Proxy TEPs.
• FTEP is the loopback that identifies the fabric.
• 9000 MTU is enabled by default in the fabric’s access ports.
• Minimum MTU size is 1500 + 50bytes for encapsulation.
• Fabric’s uplinks MTU is configured to 9150 bytes.

Resources

• ACI Design Guide White Paper
• INE Application Centric Infrastructure (ACI) Part 1 - Network Centric Mode
  • ACI Network Topology
• Cisco ACI COOKBOOK by Stuart Fordham

Glossary

APIC:  Application Policy Infrastructure Controller

VTEP:  VXLAN Tunnel Endpoint

PTEP:  Physical Tunnel Endpoint

FTEP: Fabric Tunnel Endpoint

VXLAN: Virtual Extensible LAN

FEX:  Fabric Extender

MP-BGP: Multi-Protocol Border Gateway Protocol

EVPN: Ethernet VPN 

Final Thoughts

As always this is a learning experiment so feel free to reach out with any feedback.