{"id":686,"date":"2018-07-07T23:58:44","date_gmt":"2018-07-07T16:58:44","guid":{"rendered":"https:\/\/lagonet.vn\/?p=686"},"modified":"2018-07-07T23:58:44","modified_gmt":"2018-07-07T16:58:44","slug":"behavior-of-keepalived-healthchecker-in-depends-of-return-tcp-flags-example-based-on-configured-http_get-health-check","status":"publish","type":"post","link":"https:\/\/kb.lagonet.vn\/?p=686","title":{"rendered":"Behavior of Keepalived healthchecker in depends of return TCP flags (example based on configured HTTP_GET health check)"},"content":{"rendered":"<p>About option\u00a0<strong>connect_timeout<\/strong>\u00a0&#8211; we can have 2 cases about healthchecker behavior with tcp flags:<\/p>\n<ul>\n<li>case\u00a0<strong>A<\/strong>\n<ul>\n<li>healthchecker sent TCP SYN<\/li>\n<li>remote host return TCP RESET<\/li>\n<li>keepalived ignore\u00a0<strong>connect_timeout<\/strong>\u00a0and drop RS (or reduce weight in depends of config)\n<pre class=\"wiki\">Test schema:\n\ntest RS:\n[root@test5v.dev:~]# date; iptables -A INPUT -s 192.168.2.1 -p tcp --dport 8081 -j REJECT\nWed Nov 25 14:08:44 ICT 2015\n\ndump - normal tcp healthcheck connect:\n14:08:40.252848 IP 192.168.2.1.37274 &gt; 192.168.2.10.8081: Flags [S], seq 318940877, win 29200, options [mss 1460,sackOK,TS val 629869104 ecr 0,nop,wscale 7], length 0\n14:08:40.252877 IP 192.168.2.10.8081 &gt; 192.168.2.1.37274: Flags [S.], seq 548455626, ack 318940878, win 28960, options [mss 1460,sackOK,TS val 3908139897 ecr 629869104,nop,wscale 7], length 0\n14:08:40.253034 IP 192.168.2.1.37274 &gt; 192.168.2.10.8081: Flags [.], ack 1, win 229, options [nop,nop,TS val 629869104 ecr 3908139897], length 0\n14:08:40.253067 IP 192.168.2.1.37274 &gt; 192.168.2.10.8081: Flags [P.], seq 1:76, ack 1, win 229, options [nop,nop,TS val 629869104 ecr 3908139897], length 75\n14:08:40.253079 IP 192.168.2.10.8081 &gt; 192.168.2.1.37274: Flags [.], ack 76, win 227, options [nop,nop,TS val 3908139897 ecr 629869104], length 0\n14:08:40.253220 IP 192.168.2.10.8081 &gt; 192.168.2.1.37274: Flags [P.], seq 1:225, ack 76, win 227, options [nop,nop,TS val 3908139897 ecr 629869104], length 224\n14:08:40.253238 IP 192.168.2.10.8081 &gt; 192.168.2.1.37274: Flags [FP.], seq 225:320, ack 76, win 227, options [nop,nop,TS val 3908139897 ecr 629869104], length 95\n14:08:40.253350 IP 192.168.2.1.37274 &gt; 192.168.2.10.8081: Flags [.], ack 225, win 237, options [nop,nop,TS val 629869104 ecr 3908139897], length 0\n14:08:40.253503 IP 192.168.2.1.37274 &gt; 192.168.2.10.8081: Flags [R.], seq 76, ack 321, win 237, options [nop,nop,TS val 629869104 ecr 3908139897], length 0\n\nhere the last check before drop RS:\n14:08:46.254560 IP 192.168.2.1.37275 &gt; 192.168.2.10.8081: Flags [S], seq 3293277997, win 29200, options [mss 1460,sackOK,TS val 629870605 ecr 0,nop,wscale 7], length 0\n\ntest LB:\nNov 25 14:08:46 test2v Keepalived_healthcheckers[10690]: Error connecting server [192.168.2.10]:80.\nNov 25 14:08:46 test2v Keepalived_healthcheckers[10690]: Removing service [192.168.2.10]:80 from VS [10.3.0.144]:80\n<\/pre>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<ul>\n<li>case\u00a0<strong>B<\/strong>\n<ul>\n<li>healthchecker sent TCP SYN<\/li>\n<li>remote host return nothing (DROP all packets)<\/li>\n<li>keepalived use\u00a0<strong>connect_timeout<\/strong>\n<ul>\n<li>configured\u00a0<strong>connect_timeout<\/strong>\u00a0value (8sec) &lt; than sumary GET time of nb_get_retry * delay_befor_retry = 10sec\n<pre class=\"wiki\">    HTTP_GET {\n      connect_port 8081\n      url {\n        path \/status\/\n        status_code 200\n        digest 6fb9c6eed1b7f0a50854944905dc9481\n      }\n      connect_timeout 8\n      nb_get_retry 5\n      delay_befor_retry 2\n    }\n<\/pre>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<pre class=\"wiki\">Test schema:\n\ntest RS:\ntest5v.dev:~]# date; iptables -A INPUT -s 192.168.2.1 -p tcp --dport 8081 -j DROP\nWed Nov 25 14:22:40 ICT 2015\n\nthe last normal healthcheck TCP session:\n14:47:00.340636 IP 192.168.2.1.37687 &gt; 192.168.2.10.8081: Flags [S], seq 3820409120, win 29200, options [mss 1460,sackOK,TS val 630444127 ecr 0,nop,wscale 7], length 0\n14:47:00.340665 IP 192.168.2.10.8081 &gt; 192.168.2.1.37687: Flags [S.], seq 2338668368, ack 3820409121, win 28960, options [mss 1460,sackOK,TS val 3908714919 ecr 630444127,nop,wscale 7], length 0\n14:47:00.340785 IP 192.168.2.1.37687 &gt; 192.168.2.10.8081: Flags [.], ack 1, win 229, options [nop,nop,TS val 630444127 ecr 3908714919], length 0\n14:47:00.340864 IP 192.168.2.1.37687 &gt; 192.168.2.10.8081: Flags [P.], seq 1:76, ack 1, win 229, options [nop,nop,TS val 630444127 ecr 3908714919], length 75\n14:47:00.340879 IP 192.168.2.10.8081 &gt; 192.168.2.1.37687: Flags [.], ack 76, win 227, options [nop,nop,TS val 3908714919 ecr 630444127], length 0\n14:47:00.340949 IP 192.168.2.10.8081 &gt; 192.168.2.1.37687: Flags [P.], seq 1:225, ack 76, win 227, options [nop,nop,TS val 3908714919 ecr 630444127], length 224\n14:47:00.340969 IP 192.168.2.10.8081 &gt; 192.168.2.1.37687: Flags [FP.], seq 225:320, ack 76, win 227, options [nop,nop,TS val 3908714919 ecr 630444127], length 95\n14:47:00.341134 IP 192.168.2.1.37687 &gt; 192.168.2.10.8081: Flags [.], ack 225, win 237, options [nop,nop,TS val 630444127 ecr 3908714919], length 0\n14:47:00.341184 IP 192.168.2.1.37687 &gt; 192.168.2.10.8081: Flags [R.], seq 76, ack 321, win 237, options [nop,nop,TS val 630444127 ecr 3908714919], length 0\n...here is send 4 SYN\n14:47:06.342363 IP 192.168.2.1.37688 &gt; 192.168.2.10.8081: Flags [S], seq 862050682, win 29200, options [mss 1460,sackOK,TS val 630445627 ecr 0,nop,wscale 7], length 0\n14:47:07.339158 IP 192.168.2.1.37688 &gt; 192.168.2.10.8081: Flags [S], seq 862050682, win 29200, options [mss 1460,sackOK,TS val 630445877 ecr 0,nop,wscale 7], length 0\n14:47:09.343156 IP 192.168.2.1.37688 &gt; 192.168.2.10.8081: Flags [S], seq 862050682, win 29200, options [mss 1460,sackOK,TS val 630446378 ecr 0,nop,wscale 7], length 0\n14:47:13.351145 IP 192.168.2.1.37688 &gt; 192.168.2.10.8081: Flags [S], seq 862050682, win 29200, options [mss 1460,sackOK,TS val 630447380 ecr 0,nop,wscale 7], length 0\n...and drop RS due to connect_timeout 8 seconds\n\ntest LB:\nNov 25 14:47:14 test2v Keepalived_healthcheckers[10782]: Timeout connect, timeout server [192.168.2.10]:80.\nNov 25 14:47:14 test2v Keepalived_healthcheckers[10782]: Removing service [192.168.2.10]:80 from VS [10.3.0.144]:80\n<\/pre>\n<p>so here RS dropped due to connect_timeout 8 seconds<\/p>\n<ul>\n<li>configured\u00a0<strong>connect_timeout<\/strong>\u00a0value (12sec) &gt; than sumary GET time of nb_get_retry * delay_befor_retry = 10sec\n<pre class=\"wiki\">    HTTP_GET {\n      connect_port 8081\n      url {\n        path \/status\/\n        status_code 200\n        digest 6fb9c6eed1b7f0a50854944905dc9481\n      }\n      connect_timeout 12\n      nb_get_retry 5\n      delay_befor_retry 2\n    }\n<\/pre>\n<\/li>\n<\/ul>\n<pre class=\"wiki\">Test schema:\n\ntest5v.dev:~]# date; iptables -A INPUT -s 192.168.2.1 -p tcp --dport 8081 -j DROP\nWed Nov 25 14:57:04 ICT 2015\n\nthe last normal healthcheck TCP session:\n14:57:02.912134 IP 192.168.2.1.37751 &gt; 192.168.2.10.8081: Flags [S], seq 3492153841, win 29200, options [mss 1460,sackOK,TS val 630594770 ecr 0,nop,wscale 7], length 0\n14:57:02.912168 IP 192.168.2.10.8081 &gt; 192.168.2.1.37751: Flags [S.], seq 2204287936, ack 3492153842, win 28960, options [mss 1460,sackOK,TS val 3908865562 ecr 630594770,nop,wscale 7], length 0\n14:57:02.912284 IP 192.168.2.1.37751 &gt; 192.168.2.10.8081: Flags [.], ack 1, win 229, options [nop,nop,TS val 630594770 ecr 3908865562], length 0\n14:57:02.912432 IP 192.168.2.1.37751 &gt; 192.168.2.10.8081: Flags [P.], seq 1:76, ack 1, win 229, options [nop,nop,TS val 630594770 ecr 3908865562], length 75\n14:57:02.912453 IP 192.168.2.10.8081 &gt; 192.168.2.1.37751: Flags [.], ack 76, win 227, options [nop,nop,TS val 3908865562 ecr 630594770], length 0\n14:57:02.912534 IP 192.168.2.10.8081 &gt; 192.168.2.1.37751: Flags [P.], seq 1:225, ack 76, win 227, options [nop,nop,TS val 3908865562 ecr 630594770], length 224\n14:57:02.912553 IP 192.168.2.10.8081 &gt; 192.168.2.1.37751: Flags [FP.], seq 225:320, ack 76, win 227, options [nop,nop,TS val 3908865562 ecr 630594770], length 95\n14:57:02.912597 IP 192.168.2.1.37751 &gt; 192.168.2.10.8081: Flags [.], ack 225, win 237, options [nop,nop,TS val 630594770 ecr 3908865562], length 0\n14:57:02.912789 IP 192.168.2.1.37751 &gt; 192.168.2.10.8081: Flags [R.], seq 76, ack 321, win 237, options [nop,nop,TS val 630594770 ecr 3908865562], length 0\n...here is send 5 SYN\n14:57:08.913861 IP 192.168.2.1.37752 &gt; 192.168.2.10.8081: Flags [S], seq 714362941, win 29200, options [mss 1460,sackOK,TS val 630596270 ecr 0,nop,wscale 7], length 0\n14:57:09.911049 IP 192.168.2.1.37752 &gt; 192.168.2.10.8081: Flags [S], seq 714362941, win 29200, options [mss 1460,sackOK,TS val 630596520 ecr 0,nop,wscale 7], length 0\n14:57:11.915050 IP 192.168.2.1.37752 &gt; 192.168.2.10.8081: Flags [S], seq 714362941, win 29200, options [mss 1460,sackOK,TS val 630597021 ecr 0,nop,wscale 7], length 0\n14:57:15.927067 IP 192.168.2.1.37752 &gt; 192.168.2.10.8081: Flags [S], seq 714362941, win 29200, options [mss 1460,sackOK,TS val 630598024 ecr 0,nop,wscale 7], length 0\n14:57:23.943070 IP 192.168.2.1.37752 &gt; 192.168.2.10.8081: Flags [S], seq 714362941, win 29200, options [mss 1460,sackOK,TS val 630600028 ecr 0,nop,wscale 7], length 0\n...and drop RS due to connect_timeout 15 seconds (&gt; defined 12sec)\n\ntest LB:\nNov 25 14:57:28 test2v Keepalived_healthcheckers[11075]: Timeout connect, timeout server [192.168.2.10]:80.\nNov 25 14:57:28 test2v Keepalived_healthcheckers[11075]: Removing service [192.168.2.10]:80 from VS [10.3.0.144]:80\n<\/pre>\n<p>so here RS dropped due to connect_timeout 15 seconds<\/p>\n<p><strong>Conclusion: if we have errors on L4 (TCP), L7 checks will not work due to problem on another layer.<\/strong><\/p>\n<hr \/>\n<h3 id=\"Nowletstestwithenabledoptioninhibit_on_failure-itsetweightto0onhealthcheckerfailureandshouldkeepexistingconnections.\">Now let&#8217;s test with enabled option\u00a0<em>inhibit_on_failure<\/em>\u00a0&#8211; it set weight to 0 on healthchecker failure (and should keep existing connections).<a id=\"section_2\" class=\"anchor\" title=\"Edit this section\" href=\"https:\/\/trac.coccoc.com\/coccoc\/wiki\/NCC\/net-structure\/Load-Balancers\/RS_heathcheckers\/keepalived_healthcheck_tests?action=edit&amp;section=2\"><\/a><\/h3>\n<p>Request from test1v.dev to VIP 10.3.0.144:80<\/p>\n<pre class=\"wiki\">test1v.dev:~]# curl http:\/\/test.localhost\n&lt;!DOCTYPE html&gt;\n&lt;html&gt;\n&lt;head&gt;\n&lt;title&gt;Welcome to nginx!&lt;\/title&gt;\n&lt;style&gt;\n    body {\n        width: 35em;\n        margin: 0 auto;\n        font-family: Tahoma, Verdana, Arial, sans-serif;\n    }\n&lt;\/style&gt;\n&lt;\/head&gt;\n&lt;body&gt;\n&lt;h1&gt;Welcome to nginx!&lt;\/h1&gt;\n&lt;p&gt;If you see this page, the nginx web server is successfully installed and\nworking. Further configuration is required.&lt;\/p&gt;\n\n&lt;p&gt;For online documentation and support please refer to\n&lt;a href=\"http:\/\/nginx.org\/\"&gt;nginx.org&lt;\/a&gt;.&lt;br\/&gt;\nCommercial support is available at\n&lt;a href=\"http:\/\/nginx.com\/\"&gt;nginx.com&lt;\/a&gt;.&lt;\/p&gt;\n\n&lt;p&gt;&lt;em&gt;Thank you for using nginx.&lt;\/em&gt;&lt;\/p&gt;\n&lt;\/body&gt;\n&lt;\/html&gt;\n<\/pre>\n<pre class=\"wiki\">Test LB:\nEvery 1.0s: ipvsadm -L                                                                                                                                                                      Wed Nov 25 17:27:59 2015\n\nIP Virtual Server version 1.2.1 (size=4096)\nProt LocalAddress:Port Scheduler Flags\n  -&gt; RemoteAddress:Port           Forward Weight ActiveConn InActConn\nTCP  10.3.0.144:http wlc\n  -&gt; 192.168.2.10:http            Route   10     0          15\n<\/pre>\n<p>Now reject TCP SYN for \/status<\/p>\n<pre class=\"wiki\">test5v.dev:~]# date; iptables -A INPUT -s 192.168.2.1 -p tcp --dport 8081 -j REJECT\nWed Nov 25 17:28:53 ICT 2015\n<\/pre>\n<pre class=\"wiki\">Test LB:\nEvery 1.0s: ipvsadm -L Wed Nov 25 17:29:01 2015\n\nIP Virtual Server version 1.2.1 (size=4096)\nProt LocalAddress:Port Scheduler Flags\n  -&gt; RemoteAddress:Port           Forward Weight ActiveConn InActConn\nTCP  10.3.0.144:http wlc\n  -&gt; 192.168.2.10:http            Route   0      0          15\n\n\nNov 25 17:28:59 test2v Keepalived_healthcheckers[11371]: Error connecting server [192.168.2.10]:80.\nNov 25 17:28:59 test2v Keepalived_healthcheckers[11371]: Disabling service [192.168.2.10]:80 from VS [10.3.0.144]:80\n\n<\/pre>\n<p>&#8230;still keep connections, but we can not establish a new one<\/p>\n<pre class=\"wiki\">test1v.dev:~]# curl http:\/\/test.localhost\ncurl: (7) couldn't connect to host\n<\/pre>\n<p>Let&#8217;s add RS back:<\/p>\n<pre class=\"wiki\">test5v.dev:~]# date; iptables -D INPUT -s 192.168.2.1 -p tcp --dport 8081 -j REJECT\nWed Nov 25 17:31:17 ICT 2015\n<\/pre>\n<pre class=\"wiki\">Test LB:\nEvery 1.0s: ipvsadm -L Wed Nov 25 17:31:24 2015\n\nIP Virtual Server version 1.2.1 (size=4096)\nProt LocalAddress:Port Scheduler Flags\n  -&gt; RemoteAddress:Port           Forward Weight ActiveConn InActConn\nTCP  10.3.0.144:http wlc\n  -&gt; 192.168.2.10:http            Route   10     0          0\n\n\nNov 25 17:31:17 test2v Keepalived_healthcheckers[11371]: MD5 digest success to [192.168.2.10]:80 url(1).\nNov 25 17:31:23 test2v Keepalived_healthcheckers[11371]: Remote Web server [192.168.2.10]:80 succeed on service.\nNov 25 17:31:23 test2v Keepalived_healthcheckers[11371]: Enabling service [192.168.2.10]:80 to VS [10.3.0.144]:80<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>About option\u00a0connect_timeout\u00a0&#8211; we can have 2 cases about healthchecker behavior with tcp flags: case\u00a0A healthchecker sent TCP SYN remote host [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[6],"tags":[],"class_list":["post-686","post","type-post","status-publish","format-standard","hentry","category-networking"],"_links":{"self":[{"href":"https:\/\/kb.lagonet.vn\/index.php?rest_route=\/wp\/v2\/posts\/686","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kb.lagonet.vn\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kb.lagonet.vn\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kb.lagonet.vn\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kb.lagonet.vn\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=686"}],"version-history":[{"count":0,"href":"https:\/\/kb.lagonet.vn\/index.php?rest_route=\/wp\/v2\/posts\/686\/revisions"}],"wp:attachment":[{"href":"https:\/\/kb.lagonet.vn\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=686"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kb.lagonet.vn\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=686"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kb.lagonet.vn\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=686"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}