Netplex_admin

Netplex Administration Guide

Applications created with the Netplex framework all share the following configuration settings and allow some basic administration commands. This is only a common minimum - the applications typically define more than this.

Configuration

The Netplex config file has the following layout:

netplex {
  controller { <settings> };    (* only one controller section *)
  service { <settings> };       (* as many service sections as running services *)
  ...
  service { <settings> };
  (* The application can define further types of sections *)
}

Configuration: The controller section

This section configures the controller component. The task of the controller is to start the containers for the workload, and logging.

netplex {
  controller {
    socket_directory = "<path>";
    max_level = "<debuglevel>";
    logging { <settings> };    (* several logging destinations possible *)
    ...
    logging { <settings> };
  };
  ...
}

Settings:

Log levels are (same as for syslog):

Every logging section defines a logging destination. Log messages are written to all destinations that do not filter the messages out. There are several types of logging sections:

Logging to stderr

This type writes log messages to stderr:

netplex {
  controller {
    logging {
      type = "stderr";                        (* mandatory *)
      format = "<format string>";             (* optional *)
      component = "<name_of_component>";      (* optional *)
      subchannel = "<name_of_subchannel>";    (* optional *)
      max_level = "<max_level>";              (* optional *)
    };
    ...
  };
  ...
}

The settings format, component, subchannel, and max_level may also occur in the other types of logging definitions, and are explained below.

Logging to a file

This writes the log messages to a single file.

netplex {
  controller {
    logging {
      type = "file";                          (* mandatory *)
      file = "<path>";                        (* mandatory *)
      format = "<format string>";             (* optional *)
      component = "<name_of_component>";      (* optional *)
      subchannel = "<name_of_subchannel>";    (* optional *)
      max_level = "<max_level>";              (* optional *)
    };
    ...
  };
  ...
}

Settings:

The settings format, component, subchannel, and max_level may also occur in the other types of logging definitions, and are explained below.

Logging to multiple files

This logging definition directs to create several files in a common directory.

netplex {
  controller {
    logging {
      type = "multi_file";                    (* mandatory *)
      directory = "<path>";                   (* mandatory *)
      format = "<format string>";             (* optional *)
      file { <settings> };
      ...
      file { <settings> };
    };
    ...
  };
  ...
}

The settings in the file section:

  file {
    file = "<name>";                        (* mandatory *)
    format = "<format string>";             (* optional *)
    component = "<name_of_component>";      (* optional *)
    subchannel = "<name_of_subchannel>";    (* optional *)
    max_level = "<max_level>";              (* optional *)
  };

Settings:

The settings format, component, subchannel, and max_level may also occur in the other types of logging definitions, and are explained below. Note that a format setting in the file section overrides the definition in logging for the respective file.

Logging to syslog

The log messages are sent to the syslog device of the system (Unix only).

netplex {
  controller {
    logging {
      type = "syslog";                        (* mandatory *)
      identifier = "<identifier>";            (* optional *)
      facility = "<facility>";                (* optional *)
      format = "<format string>";             (* optional *)
      component = "<name_of_component>";      (* optional *)
      subchannel = "<name_of_subchannel>";    (* optional *)
      max_level = "<max_level>";              (* optional *)
    };
    ...
  };
  ...
}

Settings:

Facility names:

Common settings in logging

format: This parameter defines how the log messages look like. This is a string containing variables in dollar notation ($name or ${name}). The following variable specifications are defined:

The standard format string is

 [${timestamp}] [${component}] [${level}] ${message} 

component: This parameter filters messages by the component emitting the messages. The component name is here the Netplex service name, i.e. the name parameter in the service section (see below). The parameter may be set to the component name, or to a pattern matching component names. The wildcard * can be used in the pattern.

Default: *

subchannel: Netplex allows it to have several log channels per component. There is normally only the main log channel, but components can define additional channels. For example, a web server may have a separate log channel for access logging. This parameter filters messages by the subchannel identifier. Again it is possible to use the wildcard *.

The main log channel has the empty string as subchannel identifier, hence subchannel="" restricts the messages to the main channel.

Default: *

max_level: Restricts the log level of the printed messages. See above for possible levels.

Examples for logging definitions

1. Write everything to stderr:

  logging { type="stderr" }

2. Write a separate file for each log level:

  logging {
    type = "multi_file";
    directory = "/var/log/myapp";
    file {
      max_level = "debug";
      file = "all_debug.log";
    };
    file {
      max_level = "info";
      file = "all_info.log";
    };
    ... (* and so on ... *)
    file {
      max_level = "emerg";
      file = "all_emerg.log";
    };
  }

3. Write errors to syslog, but write access logs to a file:

  logging {
    type = "syslog";
    max_level = "err";
    subchannel = "";    (* i.e. only the main log channel goes here *)
  };
  logging {
    type = "file";
    file = "/var/log/myapp/access.log";
    subchannel = "access";
  }

Configuration: The service section

Each service section instructs Netplex to start containers for the service. If a service type is only defined by the application, but does not appear in the config file, no containers will be started!

A service section looks like:

netplex {
  service {
    name = "<name>";                 (* mandatory *)
    user = "<user>";                 (* optional *)
    group = "<group>";               (* optional *)
    startup_timeout = <float>;       (* optional *)
    conn_limit = <int>;              (* optional *)
    gc_when_idle = <bool>;           (* optional *)
    protocol { <settings> };         (* at least one *)
    ...
    protocol { <settings> };         (* at least one *)
    processor { <settings> };        (* mandatory *)
    workload_manager { <settings> }; (* mandatory *)
  };
  ...
}

The name of the service is a freely chosen identifier. It is used to reference the service, e.g. in log messages.

Each protocol section defines a set of sockets with common properties. The idea here is that a service may define several protocols for accessing it, e.g. an HTTP-based protocol and an RPC-based protocol. For each protocol there can then be several sockets.

The processor section is the connection to the application which must have defined the type of processor that is referenced here. The task of the processor is to accept incoming connections and to process them.

The workload_manager section defines how many containers (i.e. subprocesses or threads) are created for serving incoming connections.

Settings:

The protocol subsection

It looks like:

netplex {
  service {
    protocol {
      name = "<protoname>";      (* mandatory *)
      lstn_backlog = <int>;      (* optional *)
      lstn_reuseaddr = <bool>;   (* optional *)
      so_keepalive = <bool>;     (* optional *)
      tcp_nodelay = <bool>;      (* optional *)
      local_chmod = "...";       (* optional *)
      local_chown = "...";       (* optional *)
      address { <settings> };    (* at least one *)
      ...
      address { <settings> };    (* at least one *)
    };
    ...
  };
  ...
}

Settings:

Specifying socket addresses:

The processor subsection

This section depends on the Netplex processor connected with it. At minimum, this section has only a parameter type that is the name of a defined Netplex processor type:

netplex {
  service {
    processor {
      type = "<type>";
      ... (* rest depends on the type *)
    };
    ...
  };
  ...
}

There are a number of processor types coming with Ocamlnet:

See these modules for how to configure the processors defined by them.

Configuration: The workload_manager section

The workload manager determines how many containers (processes or threads) are running for each service.

The constant workload manager

The constant workload manager starts a fixed number of containers. If containers are shut down or crash, new containers are automatically launched to replace the missing ones.

The config section looks like:

netplex {
  service {
    workload_manager {
      type = "constant";
      threads = <n>;
    }
  }
}

Note that the parameter threads is also interpreted in multi-processing mode (as number of processes).

Since Ocamlnet-3.5, there are two optional settings:

The dynamic workload manager

The dynamic workload manager starts as many containers as needed to handle the current load. Initially, a certain minimum number is started. If it turns out that too many containers become busy, more containers are started until a maximum is reached. If too many containers become idle containers are shut down.

netplex {
  service {
    workload_manager {
      type = "dynamic";
      max_jobs_per_thread = <n>;            (* optional, default: 1 *)
      recommended_jobs_per_thread = <n>;    (* optional *)
      min_free_jobs_capacity = <n>;         (* mandatory *)
      max_free_jobs_capacity = <n>;         (* mandatory *)
      max_threads = <n>;                    (* mandatory *)
    }
  }
}

A thread is here a container, even in multi-processing mode. A job is a TCP connection processed by a container. It is possible that a container can process several jobs at the same time (but only a few service types support this, e.g. RPC servers), and the whole calculation is based on the job capacity, i.e. the number of jobs all containers can execute in parallel.

The workload manager adjusts the number of containers so that there is always free capacity for min_free_jobs_capacity, but the free capacity does not exceed max_free_jobs_capacity. Also, the number of containers is capped by max_threads.

As mentioned, in most cases a container can only run one job at a time (this is meant with max_jobs_per_thread=1). Then min_free_jobs_capacity is just the minimum number of idle containers, and max_free_jobs_capacity the maximum number.

If more than one job can be executed, set max_jobs_per_thread to a value bigger than one. The workload manager assigns the components then up to this number of TCP connections to process. A component is filled up with jobs until it is full before jobs are assigned to the next container.

The latter behavior can be modified by recommended_jobs_per_thread. This must be a number less than or equal to max_jobs_per_thread, and it means that the containers normally only get the recommended number of jobs until they are "full", and only for very high workloads this scheme is left, and even more jobs are assigned to the containers until the maximum is reached. A common configuration is to set recommended_jobs_per_thread=1, so that each container gets first only up to one job, and only if the maximum number of containers are running, additional jobs can be assigned.

There are also a few rarely used options for the dynamic workload manager:

The netplex-admin command

Ocamlnet installs a little utility command netplex-admin that can be used to administer a running Netplex program.

$ netplex-admin -help
Usage: netplex-admin [ options ] [ admin_cmd arg ... ]
  -sockdir <dir>  Set the socket directory of the Netplex to administer
  -conf <file>  Get the socket directory from this config file
  -list   List available Netplex services
  -containers   List available Netplex services with container details
  -enable <name>  Enable service <name>
  -disable <name>  Disable service <name>
  -restart <name>  Restart service <name>
  -restart-all   Restart all services
  -shutdown   Shutdown the whole Netplex
  -reopen-logfiles   Reopen logfiles (if possible)
  -unlink   Unlink persistent kernel objects
  -receiver <pat>  Restrict receivers of admin messages to services matching <pat>

The utility talks to the running program, and needs the socket directory for this. Either specify it directly with the -sockdir argument, or indirectly via the config file (-conf).

The -list and -containers switches allow it to get some introspection into the running conglomerate of processes or threads. For -list the services and the socket addresses are printed. For -containers even more details are emitted, including the process IDs.

The -enable, -disable, and -restart commands allow it to manage the set of running services. A service can be disabled, which means that all containers are shut down. Note that this does not mean that incoming TCP connections are rejected. They are just not processed. If enabled again, the containers are started again for the service, and the processing of TCP connections is resumed (including the connections that were accepted during the downtime). Disabling a service is useful for temporarily stopping the service, e.g. because it would interfer with other admin tasks.

A restart means to disable and re-enable the service. It may be useful for cleanly reinitializing a service.

With -restart-all even all services are restarted that are running within the Netplex framework.

A -shutdown starts the shutdown sequence. The shutdown is announced to all containers, and in a second step, the containers are terminated. Finally, the master process is also stopped.

The command -reopen-logfiles is meaningful for all file-based logging definitions. The current set of files is closed, and reopened again. This is useful as post-action after log file rotation (see below).

-unlink: see below.

netplex-admin can also be used to send so-called admin messages to the containers. These messages have a name and optionally arguments:

$ netplex-admin ... name arg1 arg2 ...

Generally, the possible admin commands must be defined by the Netplex processors. A few commands are defined by default, though:

How to configure log file rotation

logrotate is a common utility to perform log file rotation. It can be easily used together with Netplex. The essential point is to run netplex-admin -reopen-logfiles as post action to the rotation. This stanza is an example how to configure the rotation in a logrotate config file:

/var/log/myapp/file.log
{
	weekly
        rotate 10
        missingok
        sharedscripts
        postrotate
                /some/path/bin/netplex-admin \
                        -sockdir /var/lib/myapp/sockdir \
                        -reopen-logfiles
        endscript
}

For some functionality, Netplex programs allocate persistent kernel objects:

These objects are effectively only used temporarily, but the available system interfaces only allow it to allocate them in a kernel-persisting manner. This means that the objects are not automatically released if the program ends, like most other system resources, but that they remain in memory after termination. If they were just left there behind, they would consume memory forever.

Of course, most Netplex programs are carefully written, and delete these objects if the program ends in a regular way. However, after a crash the programs normally don't have a chance to delete the objects. In order to allow administrative removal, many Netplex programs write the names of these objects into a special file netplex.pmanage residing in the socket directory. The netplex-admin command has a special mode in order to perform the deletion:

netplex-admin -sockdir /path/to/sockdir -unlink

Some operating systems have actually means for adminstering the kind of objects managed this way, but some have not. For example, Linux reflects the objects as normal files in the /dev/shm directory. As a counter example, there is no administration possibility in OS X.

(For programmers: The Netplex interface to access netplex.pmanage is Netplex_cenv.pmanage.)