publish/subscribe data between services

From: Buck Evan <buck_at_yelp.com>
Date: Mon, 7 Mar 2016 17:41:23 -0800

First, I'd like to explain that I'm using process supervision in a context
that is a bit unusual. Rather than supervising system services, I'm
supervising services for development of particular projects. For example, a
webserver and database for a website project. Also, a prime use case is
many developers developing the same project on the same machine. This means
that the supervised services cannot use a statically defined port number;
they will need to carefully select their ports to not collide among
developers services. I've taken the route of binding services to port 0 for
this purpose, to listen on an arbitrary available port. While I've chosen
the port for my example, the same problem applies to several small pieces
of data. In general, there are data that are generated by one service and
required by another.

At this point I have a design issue. How do services become aware of each
others' port numbers? I've considered using ftrig or envdirs or both to
represent and notify this data. Still, I don't see how I can keep the
overall state consistent in the case of a single service's restart; I don't
believe in general I can assume that its old port number will still be
available, which means that I will sometimes need to restart any other
services in the "playground" (collection of supervised developer services)
that know this port number. This implies dependency resolution, which is
too complex for my taste.

I've also considered plopping data into an envdir as it becomes available
and failing fast when it's unavailable, but it seems too easy to become
inconsistent under this scheme. It could easily be that the data found in
the envdir was created by a service that's no longer running, and the data
is invalid.

The system I'm replacing solves this by assigning a particular localhost IP
to the whole playground, then we may hard-code the port numbers throughout
the code base. There's complexity in maintaining the state of assigned
localhost IPs that I'd like to factor out.


Do any of you have smart ideas to implement a solution for this problem
simply?
Received on Tue Mar 08 2016 - 01:41:23 UTC