Main
Basically 2 KVM VM with Debian9 and FRR software used for RR roles – rr{1,2}v.itim.vn
- packages with configs
- coccoc-net-config-rr1v
- coccoc-net-frr-config-rr1v
- coccoc-net-config-rr2v
- coccoc-net-frr-config-rr1v
RRs grouped inside of subnet 10.196.112.0/24 to keep a simple config of RR cluster FRRouting software used for the next reasons:
- Support of BGP peer-groups
- Support of BGP dynamic neighbors (with a different ranges per group)
- Future support of BFD
- frr 3.0.2-1~debian9+1 (Deb9) – fork of Quagga
- FRR has its roots in the Quagga project. In fact, it was started by many long-time Quagga developers who combined their efforts to improve on Quagga’s well-established foundation in order to create the best routing protocol stack available.
- Sources <https://github.com/FRRouting/frr/>
- Packages <https://github.com/FRRouting/frr/releases>
- Latest stable release version is 3.0.2 <https://github.com/FRRouting/frr/releases/download/frr-3.0.2/frr_3.0.2-1-debian9.1_amd64.deb>
- Install dependencies and frr package
apt-get install libc-ares2 libjson-c3 wget https://github.com/FRRouting/frr/releases/download/frr-3.0.2/frr_3.0.2-1-debian9.1_amd64.deb dpkg -i frr_3.0.2-1-debian9.1_amd64.deb - Bug: BGP third-party next-hop resolution via kernel default route doesn’t work <https://github.com/FRRouting/frr/issues/1578>
Currently frr package added into our Deb9 repo, so it can be installed by apt-get Typical BGP config
conf t router bgp 100000 bgp router-id 10.196.112.N bgp cluster-id 10.196.112.0 neighbor PG-BGP-RRs peer-group neighbor PG-BGP-RRs remote-as 100000 neighbor PG-Calico peer-group neighbor PG-Calico remote-as 100000 neighbor PG-Core peer-group neighbor PG-Core remote-as 100000 bgp listen range 10.196.112.0/24 peer-group PG-BGP-RRs bgp listen range 10.3.0.0/22 peer-group PG-Calico bgp listen range 10.10.0.0/23 peer-group PG-Calico bgp listen range 10.10.64.0/23 peer-group PG-Calico bgp listen range 10.101.0.0/22 peer-group PG-Calico bgp listen range 10.196.126.0/24 peer-group PG-Core neighbor aaa.bbb.ccc.ddd peer-group PG-BGP-RRs neighbor PG-Core route-reflector-client neighbor PG-Calico route-reflector-client bgp client-to-client reflection ip route 10.0.0.0/8 10.196.112.1Static route is a workaround for the bug, described above.
Installation of new node
To add additional RR node:
- create 2 more packages
- coccoc-net-config-rrNv – network interface config
- coccoc-net-frr-config-rrNv – frr configs
- add all RRs as a neighbors (exclude itself)
neighbor 10.196.112.16 peer-group PG-BGP-RRs neighbor 10.196.112.17 peer-group PG-BGP-RRs neighbor 10.196.112.N peer-group PG-BGP-RRs - bgp router-id is ip of eth0/ens3/… interface (we don’t need a dummy/lo interface here, because we have just a single interface per RR)
- install packages on VM/Bare metal node
- in vtysh cli make sure that your RRs have a peers:
sh ip bgp summary
output should show a static (RRs) and dynamic (Calico nodes and core/border routers)
rr1v# sh ip bgp summary IPv4 Unicast Summary: BGP router identifier 10.196.112.16, local AS number 100000 vrf-id 0 BGP table version 49 RIB entries 1, using 136 bytes of memory Peers 4, using 84 KiB of memory Peer groups 3, using 216 bytes of memory Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 10.196.112.17 4 100000 10121 10159 0 0 0 01w0d00h 0 *10.3.4.149 4 100000 214 188 0 0 0 03:04:08 1 *10.196.126.1 4 100000 10120 10156 0 0 0 01w0d00h 0 *10.196.126.2 4 100000 10120 10156 0 0 0 01w0d00h 0 Total number of neighbors 4 * - dynamic neighbor 3 dynamic neighbor(s), limit 100
Security
The next things can be done fore BGP security
MD5 based passwords for BGP sessions between peers
- seems like neither Quagga nor FRR supports MD5 auth for Peer Groups with Dynamic peers (support MD5 auth only for Static peers)
- ISSUE request to FRR developers <https://github.com/FRRouting/frr/issues/1631>
- MD5 auth for dynamic neighbors doesn’t work in FRR 3.0.2 due to the limitation in Linux kernel (patched in kernel version 4.14), feature should be implemented in future versions of FRR). It means that on dev BareMetal? nodes non-privileged user can run a BGP speaking software to establish rogue BGP session and advertise a prefixes, that may be already advertised from another BareMetal? node. It can cause installation of 2 ECMP routes into routing table of Nodes and Routers, and can be used for:
- traffic capture
- service outages (blackhole routes)
- All above means that we should control connections to TCP port 179 of RRs on BareMetal? nodes. It may be implemented by using of cgroups + iptables, to allow outbound connection to TCP port 179 only for process in exact cgroup. At least until MD5SIG feature will be implemented in FRR.
Filter inbound updates from Calico peers – allow only routes to the prefixes inside of allocated big prefixes for projects
- We should use a filters for inbound route updates, to avoid announces of prefixes that we did not define in IP plan. Such prefixes can be installed with the same technique that i described above.
- We can use BGP ORF technique, so with the next config Calico node will announce only prefixes that our RR allow to sent in prefix-list
- Let’s say that RR is our CE
conf t ip prefix-list calico-in description "Allow only allocated prefixes in updates" ip prefix-list calico-in seq 10 permit 10.196.160.0/21 le 32 ip prefix-list calico-in seq 1000 deny any router bgp 100000 neighbor PG-Calico capability orf prefix-list send neighbor PG-Calico prefix-list calico-in in - Let’s say that Calico node is our PE (here is an example for FRR software)
conf t router bgp 100000 neighbor 10.196.112.16 capability orf prefix-list receive
- Let’s say that RR is our CE
- NOTE: If BGP ORF will not be supported at the Calico node side, i will implement usual prefix-list based filter. It requires much more CPU in case of big numbers of routes.