[ixpmanager] Design of IXP

[Nishal Goburdhan]

On 28 May 2019, at 14:40, Shahab Vahabzadeh wrote:

> Dear Friends,
> This is Shahab from Tehran IX, We are redesigning our IXP like this:
>
> Tehran IX will be in two pop:
>    1. POP A: Two Leaves and One Spine (all routers are Nexus 9508)
>    2. POP B: Two Leaves and One Spine (all routers are Nexus 9508)
>
> What we have designed is that We want to put leaf switches into VPC 
> and run
> VXLAN between them and spine switch, We want to put all members in one 
> VLAN
> belongs to route server and assign VLAN per peer for them too not per
> customer (customer port will be trunk). We are doing this because for 
> our
> customer's graph of each peer is important and as we are not sure that 
> how
> much Nexus 9K is good inflow export we are not going through looking 
> to
> running flow accounting and etc.
>
> 1st of all I want to know what is you guys ideas about our design and 
> If
> you have any comment in this phase help me. 2nd I want to change our
> software platform to IXP Manager too, I want to does it support 
> providing
> too much VLAN per customer or not? (one route server, too many per 
> peer).


hi shahab,

welcome to the list  :-)

is your intention to provide two sites that allow peers to connect to 
the same shared fabric from either site, (ie.  more locations for 
connections), or, to have two separate fabrics (ie. redundancy via 
completely separate fabric).
if you are aiming for “more ports in different places, but one 
fabric”, then you’ll want to extend your VXLAN setup between the two 
sites as well, which is missing in your description above.  in 
ixpmanager terms, you’ll have one “infrastructure” spread across 
different “facilities”
if you’re aiming for separate infrastructure (ie. different fabrics), 
then, of course, there’s no need to extend your VXLAN setup between 
sites/facilities.

the concept of an IXP, is one single shared vlan, so that peers can 
easily setup BGP sessions, which, in turn, creates traffic flow.  route 
servers, that are attached to these, make it easier for those sessions 
to go live, and, with the number of peers that you already have at 
tehran-ix, this is a Good Thing, for you to do.  the idea of a single, 
shared vlan, is a central one;  because it emphasises that an IXP is an 
easy to use facility, where no special agreements are needed, other than 
a simply willingness to peer, on the part of the providers.

it is certainly possible to create virtual private network interconnects 
(PNIs) between peers on a vlan-to-vlan basis, but this is the exception, 
rather than the norm.   it’s also, something you really don’t want 
to have the IXP do for every single peer, as it breaks the concept of 
“easy to peer with anyone”.  ixpmanager allows you to record private 
VLANs that you’ve allocated to peers, but, again, this that’s really 
just a good housekeeping habit, and *not* encouragement that individual 
vlans are needed to peer between participants.  in fact, mostly everyone 
here will likely tell you to *not* do that!

the manner that other IXPs use to show peer-to-peer graphs (ie. traffic 
between two peers) is through sflow.  for ixpmanager,  that’s 
documented quite well here:  
https://docs.ixpmanager.org/features/sflow/.  the nexus 9K does support 
sflow, but, i’m afraid i can’t offer advice on that.  what i will 
reiterate though, is that even where you need peer-to-peer graphs, the 
recommended design choice is still a single peering vlan.  it really 
does make your life easier as an IX operator!
it’s possible to use ixpmanager, and support more than a single vlan 
per customer.  as i’ve written above, you can record the 
point-to-point vlan used, in ixpmanager.  the graphs, as presented by 
MRTG will be for the port (and not for the vlan) but you’ll be able to 
get a graph for the generic peering vlan, quite easily via sflow.  but 
again - you really want to be using a single vlan for all peers   :-)

hth,
—n.