The devices that terminate overlays within the network virtualization space are referred to as Virtual Tunnel Endpoints (VTEPs). I don’t believe the Top of Rack, End of Row, or Core need to terminate overlay tunnels in a fully virtualized environment for VM to VM traffic. Believe me, if my viewpoint changes, I’ll be the first to write about it.
Where I do like hardware terminating overlays with Network Virtualization
But even with an end to end virtualized network, terminating overlays in hardware in other parts of the network may make sense. Where? I like WAN edge and integration into physical workloads as two examples.
What if you deployed a great multi-tenant Private Cloud based on the latest and greatest technology and then needed to get a WAN router that resided in the same physical data center to communicate with a single tenant (maybe for a b2b connection or high speed WAN for a particular business application)? You can do some cool stuff and peer using BGP, use your favorite IGP, create VRFs, and ultimately have a L3 gateway into the overlay network, or you can simply plug a WAN router into a physical switch and configure the port that the router plugs into to be part of the same VXLAN (overlay) segment. This even allows you to put the router on the same subnet as the hosts if you wanted. This can offer up high capacity routing or other services where needed for those that still refuse virtual services L3-L7. But if you’re deploying a solution as described, I suppose you’ve succumbed to the flexibility L2 overlays, and L3+ are next anyway :).
Integration with a TOR switch also makes sense for physical servers. If an application heartbeat is needed or if there is a P2V migration going on, having a TOR terminate an overlay makes sense.
So why don’t I like TORs or Core switches participating in a [general] NV solution?
It reminds me too much of what you can already do with VLANs, VRFs, and GRE. That is deploy solid networks that take a team to setup that are very difficult to automate. Remember locally connected routes that start communicating automagically too? Not a good thing, hence needing a VLAN per VRF for some deployments. As I said in previous posts, while overlays do give the ability to deploy many more than 4000 segments, one of the best side effects not often discussed, of a platform that uses overlays is that they are very easy (so I’ve heard) to automate using off the shelf tools and platforms.
Would you rather automate the deployment of the endpoints or try and automate every device along the path? I also feel like terminating overlays on all TORs or in the Core would be like trying to terminate IPSec with your local carrier, having them terminate IPSec to their remote device, which would finish with a peering to your remote device, all to setup a point to point VPN connection. Imagine trying to automate that? Isn't it easier to go from edge device to edge device using a control server (DMVPN/NHRP server) to do the setup?
Of course, this is only what I think...