{"id":681,"date":"2018-07-07T23:55:16","date_gmt":"2018-07-07T16:55:16","guid":{"rendered":"https:\/\/lagonet.vn\/?p=681"},"modified":"2018-07-07T23:55:16","modified_gmt":"2018-07-07T16:55:16","slug":"goi-y-su-dung-border-router-1","status":"publish","type":"post","link":"https:\/\/kb.lagonet.vn\/?p=681","title":{"rendered":"G\u1ee2I \u00dd S\u1eec D\u1ee4NG BORDER ROUTER #1"},"content":{"rendered":"<h2 id=\"BorderRouters\">Border Routers<a id=\"section_1\" class=\"anchor\" title=\"Edit this section\" href=\"https:\/\/trac.coccoc.com\/coccoc\/wiki\/NCC\/net-structure\/Border-Routers?action=edit&amp;section=1\"><\/a><\/h2>\n<p>We are using 2 hosts for Production requests routing purposes:\u00a0<a class=\"ext-link\" href=\"http:\/\/racktables.itim.vn\/index.php?page=object&amp;object_id=148\"><span class=\"icon\">\u200b<\/span>rb1v.itim.vn<\/a>\u00a0and\u00a0<a class=\"ext-link\" href=\"http:\/\/racktables.itim.vn\/index.php?page=object&amp;object_id=149\"><span class=\"icon\">\u200b<\/span>rb2v.itim.vn<\/a><\/p>\n<p>and 1 router for hosts in Production and Dev, that requires outgoing connections to Internet\u00a0<a class=\"ext-link\" href=\"https:\/\/racktables.itim.vn\/index.php?page=object&amp;object_id=107\"><span class=\"icon\">\u200b<\/span>rb3v.itim.vn<\/a><\/p>\n<h2 id=\"NetworkInterfaces\">Network Interfaces<a id=\"section_2\" class=\"anchor\" title=\"Edit this section\" href=\"https:\/\/trac.coccoc.com\/coccoc\/wiki\/NCC\/net-structure\/Border-Routers?action=edit&amp;section=2\"><\/a><\/h2>\n<p>rb{1,2} hosts has 2 integrated 1G ports and 4x1G ports from installed\u00a0<a class=\"ext-link\" href=\"http:\/\/ark.intel.com\/products\/49187\/Intel-Gigabit-ET2-Quad-Port-Server-Adapter\"><span class=\"icon\">\u200b<\/span>Intel ET2 Quad port NIC<\/a><\/p>\n<p>All 6 ports aggregated into LACP etherchannel (bond0 inteface on server):<\/p>\n<pre class=\"wiki\">auto bond0\niface bond0 inet manual\n    bond-slaves eth0 eth1 eth2 eth3 eth4 eth5\n    bond-mode 4\n    bond-miimon 100\n    bond-downdelay 200\n    bond-updelay 200\n    bond_lacp_rate fast\n    bond_xmit_hash_policy layer3+4\n<\/pre>\n<h3 id=\"SMPAffinity\">SMP Affinity<a id=\"section_3\" class=\"anchor\" title=\"Edit this section\" href=\"https:\/\/trac.coccoc.com\/coccoc\/wiki\/NCC\/net-structure\/Border-Routers?action=edit&amp;section=3\"><\/a><\/h3>\n<h3 id=\"RXTXQueuetuning\">RX\/TX Queue tuning<a id=\"section_4\" class=\"anchor\" title=\"Edit this section\" href=\"https:\/\/trac.coccoc.com\/coccoc\/wiki\/NCC\/net-structure\/Border-Routers?action=edit&amp;section=4\"><\/a><\/h3>\n<h3 id=\"NICoffloads\">NIC offloads<a id=\"section_5\" class=\"anchor\" title=\"Edit this section\" href=\"https:\/\/trac.coccoc.com\/coccoc\/wiki\/NCC\/net-structure\/Border-Routers?action=edit&amp;section=5\"><\/a><\/h3>\n<hr \/>\n<h3 id=\"Failover\">Failover<a id=\"section_6\" class=\"anchor\" title=\"Edit this section\" href=\"https:\/\/trac.coccoc.com\/coccoc\/wiki\/NCC\/net-structure\/Border-Routers?action=edit&amp;section=6\"><\/a><\/h3>\n<ul>\n<li>rb3v is working as primary default gateway for:\n<ul>\n<li><strong>Core routers &#8211; rc*N<\/strong>\u00a0(inject\u00a0<strong>default-information originate metric 10 metric-type 1<\/strong>\u00a0into OSPF Area 1 (vlan 998); OSPF-based failover)<\/li>\n<li>Internet-required-hosts inside 172.16.16.0\/24\u00a0<strong>Frontend&#8217;s vlan 999<\/strong><\/li>\n<\/ul>\n<\/li>\n<li>rb1v acts as\u00a0<strong>PRIMARY<\/strong>\u00a0router for non-internet-required hosts inside 172.16.16.0\/24\u00a0<strong>Frontend&#8217;s vlan 999<\/strong>\u00a0(<strong>Keepalived<\/strong>:\u00a0<strong>1.2.12<\/strong>\u00a0&#8211; VRRP-based failover ); look into keepalived.conf:<\/li>\n<li>rb2v acts as\u00a0<strong>BACKUP<\/strong>\u00a0router for non-internet-required hosts inside 172.16.16.0\/24\u00a0<strong>Frontend&#8217;s vlan 999<\/strong>\u00a0(<strong>Keepalived<\/strong>:\u00a0<strong>1.2.12<\/strong>\u00a0&#8211; VRRP-based failover ); look into keepalived.conf:\n<pre class=\"wiki\">virtual_ipaddress {\n    172.16.16.1\/24  dev eth0.999\n    }\n<\/pre>\n<pre class=\"wiki\">eth0.999    inet 172.16.16.1\/24 scope global secondary eth0.999\n<\/pre>\n<\/li>\n<li>If rb1v is down, keepalived on rb2v must change state to\u00a0<strong>MASTER<\/strong><\/li>\n<\/ul>\n<h3 id=\"Routing\">Routing<a id=\"section_7\" class=\"anchor\" title=\"Edit this section\" href=\"https:\/\/trac.coccoc.com\/coccoc\/wiki\/NCC\/net-structure\/Border-Routers?action=edit&amp;section=7\"><\/a><\/h3>\n<p>We are using quagga package ( 0.99.17-2+squeeze3 BGP\/OSPF\/RIP routing daemon ) from Debian Squeeze repo for OSPF.<\/p>\n<p>OSPF-based failover schema:<\/p>\n<figure id=\"attachment_691\" aria-describedby=\"caption-attachment-691\" style=\"width: 1435px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-691\" src=\"https:\/\/lagonet.vn\/wp-content\/uploads\/2018\/07\/vdc-RB-diag-v0.2.jpeg\" alt=\"Border Router\" width=\"1435\" height=\"1000\" \/><figcaption id=\"caption-attachment-691\" class=\"wp-caption-text\">Border Router<\/figcaption><\/figure>\n<p>Traffic from\/to all subnets (<strong>exclude lb*v hosts and hosts with external ip addresses<\/strong>) going via rb1v, so SNAT and<\/p>\n<h3 id=\"Firewall\">Firewall<a id=\"section_8\" class=\"anchor\" title=\"Edit this section\" href=\"https:\/\/trac.coccoc.com\/coccoc\/wiki\/NCC\/net-structure\/Border-Routers?action=edit&amp;section=8\"><\/a><\/h3>\n<p>All rules in \/etc\/itim\/firewall\/<\/p>\n<ul>\n<li><strong>firewall<\/strong>\u00a0&#8211; main rules<\/li>\n<li><strong>firewall-itim-inet-allowed<\/strong>\u00a0&#8211; rules, that mark traffic (in PREROUTING) from local subnets, aloowed for SNAT (<strong>actual for rb3v<\/strong>)<\/li>\n<\/ul>\n<h3 id=\"Sysctl\">Sysctl<a id=\"section_9\" class=\"anchor\" title=\"Edit this section\" href=\"https:\/\/trac.coccoc.com\/coccoc\/wiki\/NCC\/net-structure\/Border-Routers?action=edit&amp;section=9\"><\/a><\/h3>\n<p><strong>rp.filter\u00a0<\/strong>must be disabled!!!<\/p>\n<p><strong>Without it load-balancing will NOT work!!!<\/strong><\/p>\n<p>Also on rb3v is increased size of net.nf_conntrack_max and are tuned up timeouts for conntrack table:<\/p>\n<pre class=\"wiki\"># Networking tuning\nnet.ipv4.tcp_orphan_retries = 1\nnet.ipv4.conf.default.proxy_arp = 0\nnet.ipv4.conf.all.rp_filter = 0\nnet.core.somaxconn = 65535\nnet.ipv4.tcp_max_tw_buckets = 1048576\nnet.ipv4.tcp_ecn = 0\nnet.ipv4.conf.default.send_redirects = 1\nnet.ipv4.conf.all.send_redirects = 0\n\nnet.ipv6.conf.all.disable_ipv6 = 1\nnet.ipv6.conf.default.disable_ipv6 = 1\nnet.ipv6.conf.lo.disable_ipv6 = 1\n\n# core backlog\nnet.core.netdev_max_backlog=10000\n\n# conntrack\nnet.netfilter.nf_conntrack_generic_timeout = 120\nnet.netfilter.nf_conntrack_tcp_timeout_syn_sent = 60\nnet.netfilter.nf_conntrack_tcp_timeout_syn_recv = 30\nnet.netfilter.nf_conntrack_tcp_timeout_established = 300\nnet.netfilter.nf_conntrack_tcp_timeout_fin_wait = 60\nnet.netfilter.nf_conntrack_tcp_timeout_close_wait = 30\nnet.netfilter.nf_conntrack_tcp_timeout_last_ack = 30\nnet.netfilter.nf_conntrack_tcp_timeout_time_wait = 60\nnet.netfilter.nf_conntrack_tcp_timeout_close = 10\nnet.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300\nnet.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300\n\nnet.nf_conntrack_max = 2048576\nnet.netfilter.nf_conntrack_acct = 0\n\n\n<\/pre>\n<h2 id=\"Serviceson-board\">Services on-board<a id=\"section_10\" class=\"anchor\" title=\"Edit this section\" href=\"https:\/\/trac.coccoc.com\/coccoc\/wiki\/NCC\/net-structure\/Border-Routers?action=edit&amp;section=10\"><\/a><\/h2>\n<h3 id=\"DC-Officetunnel\">DC &#8211; Office tunnel<\/h3>\n","protected":false},"excerpt":{"rendered":"<p>Border Routers We are using 2 hosts for Production requests routing purposes:\u00a0\u200brb1v.itim.vn\u00a0and\u00a0\u200brb2v.itim.vn and 1 router for hosts in Production and [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[6],"tags":[],"class_list":["post-681","post","type-post","status-publish","format-standard","hentry","category-networking"],"_links":{"self":[{"href":"https:\/\/kb.lagonet.vn\/index.php?rest_route=\/wp\/v2\/posts\/681","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kb.lagonet.vn\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kb.lagonet.vn\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kb.lagonet.vn\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kb.lagonet.vn\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=681"}],"version-history":[{"count":0,"href":"https:\/\/kb.lagonet.vn\/index.php?rest_route=\/wp\/v2\/posts\/681\/revisions"}],"wp:attachment":[{"href":"https:\/\/kb.lagonet.vn\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=681"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kb.lagonet.vn\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=681"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kb.lagonet.vn\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=681"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}