CUSTOMS(8) CUSTOMS(8)
NAME
customs - remote execution daemon
SYNOPSIS
customs [-master ] [-check check-interval] [-localjobs
allowed] [-idle min-idle] [-idlecrit criteria] [-load min-
load-average] [-jobs max-jobs] [-swap min-swap-pct] [-proc
min-free-procs] [-cpu cpu-time-limit] [-memory memory-use-
limit] [-nice nice-level] [-npri priority] [-evict evic
tion-delay] [-access user-access-level] [-bias factor]
[-server server-name] [-net net-num] [-arch architecture-
num] [-attr attribute-string] [-allow address] [-deny
address] [-version] [-verbose] [-debug] [-nolog] client
[client...]
DESCRIPTION
Customs is a daemon to provide remote-execution facilities
for pmake(1). It attempts to do some load balancing, but
it's not particularly intelligent about it. It was
designed to allow a network of workstations to make use of
idle machines in that network.
In order to use the system, two conditions must be met:
1) All participating machines must be running this
daemon.
2) The filesystem should be uniform across all the
machines.
This last condition is very important, as customs makes no
attempt to mount any absent filesystems that are needed by
an imported job. The attempt to change directories simply
fails and the job perishes ignominiously. (However, there
is a path normalization routine that can be customized at
compile time to deal with automounters or other cross-
mounting schemes.)
For a group of machines to cooperate via customs they must
be attached to the same network. The customs agents on a
network elect among themselves a Master Agent to keep a
central record of machine availability. Only daemons
started with the -master flag can become master agents.
There needs to be at least one potential master daemon;
having several is a good idea in the face of potential
network partitions, downtimes, etc. Having too many -mas
ter daemons, on the other hand, can cause long delays in
settling the election, or failure to do so.
Each agent in the network is responsible for checking
whether its machine is available for use and reporting
this status to the Master Agent on a regular basis. The
Master Agent uses these reports to determine which machine
should be given to a client for its use. An agent's
PMake 2.1 (ICSI) $Date: 1996/08/18 15:51:58 $ 1
CUSTOMS(8) CUSTOMS(8)
decision is based on four criteria:
1) The current load average. There is a maximum
load average below which the current load
must be. This is set by the -load switch (it
must be at least .25). It defaults to 0.5.
2) The amount of free swap space. This is
expressed as a percentage of all available
swap space and is set by the -swap flag. It
can be at most 40% and defaults to 25%.
3) The number of jobs imported from other
machines. This is set by the -jobs flag and
defaults to 2. It must be at least 1, for
obvious reasons.
4) Workstation idle time. This is defined as
the most recent access time on various ter
minal devices. The minimum time for which
the workstation must have been idle is set
by the -idle flag. It is expressed as min
utes:seconds and defaults to 15 minutes. A
second flag, -idlecrit, determines what cri
teria are used to compute idle times. Pos
sible argument values are
1 Keyboard and/or mouse input, if the
system supports special devices for
this purpose (e.g., /dev/kbd on Sun
workstations).
2 Logins (local or remote) by the user
owning /dev/console.
4 Local logins by any user.
8 Remote logins by any user. Requires
that the system record the origin of
login sessions (otherwise all logins
will be treated as local, see
above).
The sum of several of these values signifies
a disjunction of the associated cirteria.
For example, the default is 3, which com
putes idle time from both workstation key
board input and console user logins.
5) Total number of processes. On machines
which have a limited number of simultane
ously running processes a minimum number of
free slots in the process table has to be
available. The minimum number of `free'
PMake 2.1 (ICSI) $Date: 1996/08/18 15:51:58 $ 2
CUSTOMS(8) CUSTOMS(8)
processes can be set using the -proc option.
By default it is 10 and can be at most 20.
If any of these quantities is set to 0, it is ignored when
determining the machine's availability.
The interval at which an availability check is made is
governed by the -check flag. The interval defaults to 10
seconds, but this is excessive as the availability isn't
likely to change in such a short while. There are also
other times when the agent reports its availability to the
Master Agent. A more reasonable time is every 30 seconds.
This interval should not be made too long however, as the
master agent uses it as the upper bound on the time for
which a machine may be silent without being considered
down. If the machine crashes within this interval, nothing
evil will happen, save that the system will be slowed a
bit as the master agent tries to contact the down machine
and fails.
By default only jobs from remote machines can be started
through customs. However, if -localjobs specifies a non-
zero value then even local client programs can start pro
cesses through customs, subject to the same availability
criteria as used for remote clients.
A number of resource limitations may be imposed on
imported processes, governed by the following options.
-cpu Specifies a cpu time limit (in minutes:seconds
format). Processes exceeding the set limit are
sent a SIGXCPU signal, and if they choose to
catch that signal, given an additional 15 sec
onds to terminate. A value of 0 disables CPU
time limitation, and if enabled, the limit has
to be at least 1 minute. CPU time limits are
disabled by default. The CPU time limit might
be used to implement a usage policy that allows
only short-lived jobs (e.g, compilations) to use
the customs system.
-memory Limits the real memory usage (resident set size)
of any imported processes. The argument speci
fies memory size in Kbytes. A value of 0
imposes no limit (the default). At least 1
Mbyte must be specified if a limit is imposed.
-nice Directs customs to increase the nice level of
imported jobs to at least the value specified,
i.e., execute them at lower priority. The
default value for -nice is 4.
-npri allows customs to run imported jobs at the
PMake 2.1 (ICSI) $Date: 1996/08/18 15:51:58 $ 3
CUSTOMS(8) CUSTOMS(8)
specified non-degrading priority. (This is a
feature only supported on SGI systems.) The
value given must be in the range assigned to
low-priority batch jobs (128 to 255, see
npri(1)). The default is to use standard inter
active scheduling (value 0).
Both -memory and -nice (or -npri where available) could be
used to implement a remote execution policy whereby
machines take on imports even though they are not idle
(i.e., -idle 0), while making sure that imported jobs
yield CPU and memory to interactive processes. Presumably
one would also choose -localjobs 1 to run local jobs sub
ject to the customs availability criteria and resource
limits.
An alternative policy (used as the default) is to allow
unlimited utilization as long as a machine is idle, but
force eviction of all current imports once it ceases to be
idle.
The -evict flag sets a time after which imported jobs are
terminated when a host machine stops being idle (other
availability criteria are ignored for this purpose since
they are highly dependent on the imported job itself). To
allow graceful handling of eviction by the evicted, the
following protocol is used. Immediately after the host
ceases to be idle, jobs are sent a SIGUSR2 signal as a
notice of impending eviction. After the period specified
by -evict (2 minutes by default) customs attempts termina
tion with SIGXCPU (if available) or SIGTERM. Finally,
after another 15 second grace period, jobs are killed.
Long running applications that want to use a customs net
work to distribute computation should be prepared to catch
SIGUSR2 and save any important information or state.
Short-lived applications (such as jobs originating from
pmake(1)) shouldn't have to worry about evictions as cus
toms provides for SIGUSR2 to be ignored by default. The
eviction-delay parameter should be chosen with this in
mind. -evict 0 disables the eviction mechanism.
The customs daemon does not perform sophisticated user
authentication since it is assumed that customs networks
will only span mutually trusted hosts with consistent user
and group privileges. However, the -access flag can be
used to specify various degrees of user-based access con
trol on the server side. Level 0 allows unconditional
access to all users, i.e., if a user was successful in
obtaining an export permit on the client side that is con
sidered sufficient authorization. Level 1 requires that
imported jobs carry a UID that at least has a valid local
passwd entry. Level 2 requires additionally that an
authorized shell is defined for the user (as per
PMake 2.1 (ICSI) $Date: 1996/08/18 15:51:58 $ 4
CUSTOMS(8) CUSTOMS(8)
/etc/shells). Level 3 behaves as level 2 but will record
any denied access attempts in the customs log file.
Access levels are most useful in a homogeneous network
which implements host access restrictions by excluding
certain users from /etc/passwd on certain hosts. Access
level 0 allows users use idle hosts even if would not be
able to login to the host (which leaves open some back-
doors for abuse). Levels 1 and 2 can be used to restrict
customs use only to hosts where rlogin/rsh use would oth
erwise be possible. Finally, level 3 may be useful where
no such restrictions are ever supposed to take effect
(e.g., because of independent access control mechanisms,
such as the client host list, see below).
The preceding flags may also be given to the
importquota(8) program to change these quantities, or find
out the current criteria, on the fly. The following flags
are available only to customs itself.
The customs master daemon, when performing host alloca
tions, ranks the available hosts according to a ``rating''
that is proportional to the difference between the maximum
allowed and the actual current load on the machines. This
alone wouldn't account for inherent speed (or desirabil
ity) differences between machines. The rating for a
machine can therefore be scaled by an integer factor set
with the -bias flag.
The -server option may be used to ensure that diskless
machines are only considered available if their server
host is up. To this end, the local customs daemon will
ping the daemon on machine server-name, if specified, and
declare itself available only if a response is received.
Therefore, this feature assumes that servers also run cus
toms agents.
Each type of machine on the network is given an unique
architecture number to allow clients to request export to
the same architecture. Machines should likely have the
same architecture number if they are binary-compatible,
though processor speed might also be taken into account.
The number itself has no intrinsic meaning as far customs
is concerned. The architecture for a machine is set by the
-arch flag.
For finer distinctions between available machines in a
customs network, the -attr flag can be used to specify an
arbitrary number of strings that are stored as attributes
for that machine. Typical uses include amount of avail
able memory, disks space etc. Clients requesting machines
for exportation may specify that the selection be
restricted to any (conjunctive) combination of such
attributes.
PMake 2.1 (ICSI) $Date: 1996/08/18 15:51:58 $ 5
CUSTOMS(8) CUSTOMS(8)
Since there are times when several distinct groups operate
machines on the same Ethernet cable, but do not want to
share their machines outside their own group, customs also
may be given a network number. Only agents with the same
network number will work together. An agent's network num
ber (note this is different from its network address) is
given with the -net flag. The groups must cooperate to the
extent of allocating an unique network number to each
group.
The options -allow and -deny each take IP numbers in dot
ted notation as arguments. They may be repeated and per
form additions or deletions to the access control list
used by the RPC system underlying customs operation. This
feature is useful to prevent prviledged uses on non-
trusted, but reachable hosts from controlling local cus
toms daemons.
By default, all hosts on the networks that the local host
is connected to are allowed to interact with the RPC sys
tem (subject to additional constraints imposed by customs
itself). -allow adds the given host IP address or network
number to the access list. Conversely, -deny removes all
matching access list entries (such that specifying a full
address could be used to remove a matching network num
ber). When RPC debugging is enabled using cctrl(8) the
effective access control list is logged.
The other four flags are mostly for debugging purposes:
-version
just prints a version message and exits.
-nolog causes log information to be written to customs's
standard output. It also keeps customs from detach
ing from its parent.
-verbose
tells customs to place large amounts of verbose
output into its log file or, if -nolog was given,
to its standard output.
-debug keeps it from detaching from its parent.
Every machine must import jobs from at least one other
machine. The names of the machines for which it will per
form this service are given as the final arguments. If the
sole argument is ALL, the agent will accept jobs from any
machine on its network. This is generally the way I set it
up.
SIGNALS
SIGHUP Causes the daemon to restart itself (unless
there are jobs still running under its
PMake 2.1 (ICSI) $Date: 1996/08/18 15:51:58 $ 6
CUSTOMS(8) CUSTOMS(8)
control).
SIGINT Terminates the daemon.
SIGTERM Terminates the daemon.
SIGUSR1 Enables customs debugging (like -verbose).
SIGUSR2 Disables customs debugging.
FILES
/etc/passwd
/etc/shells For user access control.
/dev/kbd For console idle time checking.
/usr/adm/customs-log Log file (unless configured to
use syslog(3)).
SEE ALSO
importquota(8), cctrl(8), reginfo(1), pmake(1), logd(8),
export(1)
BUGS
UNIX is still lacking a general mechanism for saving and
restarting evicted jobs.
The -cpu and -memory functionality relies on BSD resource
limits and may not be supported on all systems.
AUTHOR
Adam de Boor, adam@bsw.uu.net (...!uunet!bsw!adam).
Bugfixes and enhancements by Andreas Stolcke (stol
cke@icsi.berkeley.edu).
PMake 2.1 (ICSI) $Date: 1996/08/18 15:51:58 $ 7
Man(1) output converted with
man2html