Re: HAProxy Hot Reconfiguration with s6

From: Colin Booth <cathexis_at_gmail.com>
Date: Wed, 18 May 2016 20:32:16 -0700

On Mon, May 16, 2016 at 12:57 PM, Remko Tronçon <remko_at_el-tramo.be> wrote:
> Hi,
>
> This question has probably been asked before, but I couldn't find anything.
>
> I want to run HAProxy under a system using s6. I want to make HAProxy hot
> reload its configuration to have minimal downtime on configuration updates.
> The way this is done in HAProxy is by starting a second instance, which
> takes over the port and asks the first instance to exit. However, if
> haproxy runs under s6, when the first haproxy process goes away, s6 will
> automatically restart it I assume? Is there a way around this? How is
> HAproxy generally used with s6?
>
> thanks!
> Remko
Hi Remko,

I hacked together a thing today at work to handle this (basically
taking my plan from yesterday and implementing it). Remember, the
design here is for emulating config hot reloads, not for handling any
rollback or other reversable cutover schenarios. This is runit
specific but should be pretty easy to rewrite in execline.

run:
#!/bin/sh
exec 2>&1
if [ -f /tmp/haproxy.pid ]; then
    HANDOFFPID=`cat /tmp/haproxy.pid`
    rm /tmp/haproxy.pid
fi
exec /usr/sbin/haproxy -db -f /etc/haproxy/haproxy.cfg \
    -f /opt/haproxy-config/combined.cfg \
    -sf $HANDOFFPID

reload.sh:
#!/bin/sh
/usr/sbin/haproxy \
    -f /etc/haproxy/haproxy.cfg \
    -f /opt/haproxy-config/combined.cfg \
    -p /tmp/haproxy.pid \
    -sf $(sv s /opt/zsv/jboss/haproxy | awk '{print substr($4,1,length($4)-1)}')

A couple of notes: the -sf option to haproxy is fine with taking a
null option so you don't need to worry about having two exec lines. If
you're in a startup or crash restart situation, ./run skips all the
pid detection stuff and while $HANDOFFPID is empty, like I said it
doesn't matter. If you're reloading, reload.sh starts a daemonized
copy of haproxy (this is important, I'll explain in a sec) having dug
out the process ID of the supervised copy from the output of sv status
and saving the current pid in a well-known location so that when runsv
restarts haproxy it's able to re-acquire all its fds.

So, the special note about daemonization. if you run haproxy in
foreground mode (-db), the -p flag does nothing. Since I need to get
the process id for a clean hand-back, I can either scrape the pid via
/proc or ps, or daemonize. Yes there's the standard pidfile race, but
I figure the 1-2 second turnaround between termination of the
supervised copy and the give-back is short-enough that it isn't too
bad.

Cheers!
-Colin

-- 
"If the doors of perception were cleansed every thing would appear to
man as it is, infinite. For man has closed himself up, till he sees
all things thru' narrow chinks of his cavern."
  --  William Blake
Received on Thu May 19 2016 - 03:32:16 UTC

This archive was generated by hypermail 2.3.0 : Sun May 09 2021 - 19:44:19 UTC