Re: s6-rc shutdown timing issue

From: Laurent Bercot <ska-skaware_at_skarnet.org>
Date: Sun, 13 Sep 2015 23:34:55 +0200

On 13/09/2015 20:25, Colin Booth wrote:
> My current issue is that I'm initially remounting my root filesystem
> as r/w as one of the first steps for s6-rc, which means that if I'm
> doing everything correctly, s6-rc attempts to remount root as
> read-only as part of its shutdown.

  Yeah, indeed, that won't work in all cases, because the situation is not
symmetrical. Remounting your rootfs rw at boot time will always succeed,
but remounting it ro before killing everything may fail.

  You have absolutely no way of ensuring that nothing will attempt to write
to your rootfs before you nuke everything. That's the fundamental difference
between startup and shutdown: during shutdown, and until your nuke, other
stuff may still be running that you have no control on.

  If it's only about your rootfs, I'd simply keep the "remount it rw/ro" part
out of s6-rc.
  If it's about a more complex fs infrastructure and you may still have
processes with open handles to the mounted filesystems, it's more annoying,
and I don't have a perfect solution. The simplest thing is to do all the
unmounts outside of s6-rc, but it's asymmetrical and doesn't feel right.
Another possibility is to have a nuke in a ./down script of a oneshot that
depends on all your filesystem-mounting service, so you're sure to have
already killed everything when you get to unmounting; but that's not great
either, especially if you switch databases and s6-rc-update computes that
it has to remount one of your filesystems - oops, it just killed everything
on your machine and can't complete its work. That shouldn't happen if your
"mount stuff" services are very low-level and always up, but if you give
users a set of teeth, they will manage to bite themselves.

  I'm afraid there's no real solution to the stragglers problem, and the
only safe approach is to keep everything mounted all the time and only
perform the unmounts in stage 3 after everything else has been done and
the processes have been killed.


> Cool, thanks. That's what I thought but I wasn't sure to what degree
> execlineb cared about script validity and I don't have a terribly
> great test methodology for oneshots figured out yet.

  If you run s6-rc -v3, it should show you exactly what commands it's
running.

  In other news, I'm now in the process of testing s6-rc-update. I've
finished brooming the obvious bugs, now is the most annoying part: so
many test cases, so many things that can go wrong, and I have to try
them one by one with specifically crafted evil databases. Ugh. I said
I'd release that thing in September, but that won't happen if I die of
boredom first.
  If you're totally crazy, you can try running it, but unless you're
doing something trivial such as switching to the exact same database,
chances are that something will blow up in your face - in which case
please let me analyze the smoke and ashes.

-- 
  Laurent
Received on Sun Sep 13 2015 - 21:34:55 UTC

This archive was generated by hypermail 2.3.0 : Sun May 09 2021 - 19:38:49 UTC