Commands

The check_multi plugin is configured by a simple NRPE stylish ASCII config file.
Format

<keyword> [ <tag> ] = <content>

Elements

• Supported keywords:
  1. attribute
  2. command
  3. eval
  4. eeval
  5. state
  6. statusdat
• '#' comments are allowed, but only at the beginning of the lines and not in continuation lines
• empty lines are allowed.
• spaces / white characters between all elements can provide better readability

With the attribute command you can specify global variables and child check specific settings.

Format

attribute [ <global variable> ] = <global variable value>

attribute [ <tag>::<variable> ] = <child check value>

Elements

• tag: <tag> is the name of a particular child check. see command or eval.
• global variable: any -s variable, which you can set via command line, compare check_multi -hhh
• child check / tag variable: child check attributes, which can be seen via check_multi XML export (-r 256)

Examples

• Check /tmp/ permissions (should be writeable for everybody, but with sticky bit:

command   [ tmp_dir_permissions           ] = ls -ld /tmp
attribute [ tmp_dir_permissions::critical ] = "$tmp_dir_permissions$" !~ /^drwxrwxrwt/

• Set global variable extinfo_in_status, this replaces nthe ewline between plugin_output and long_plugin_output by HTML <br /> in order to present all check_multi output in the status view:

attribute [ extinfo_in_status ] = 1

The command definition specifies the list of child plugin calls which should be executed by check_multi as parent. Format

command [ <tag> ] = <plugin command line>

Elements

• tag: <tag> is a freely choosable expression and provides something like a primary key to identify the plugin call. Especially in large environments you should spent some consideration about a naming scheme. Allowed characters: A–Za-z0–9_:
• plugin command line: This is just a standard plugin command line.
  The directory or full path of the plugin does not have to be mentioned here, it will be either be the default /usr/local/nagios/libexec or can be specified on the check_multi command-line using the –libexec-switch.
  
  New: the standard PATH is evaluated for the execution of programs. Therefore the evaluation order is as follows:
  1. libexec-dir (default or specified with –libexec)
  2. PATH of Nagios user
  3. fully specified path of plugin or command

• Nagios Macros are allowed here: eg. $HOSTNAME$ specifies the current name of the machine. This allows you to write flexible command files.
• Note: the order of the command list determines the latter order of execution.
• PNP: If you want to specify a particular plugin for performance data, use the label format command [ tag::plugin ].

Eval is a powerful means to gather information from any source and create flexible plugin configurations.

• eval just evaluates the given expression and stores the result in $<tag>$. There is no output.
• eeval (speak: e(cho)eval) does the same as eval but prints the result in the output.

Format

eval [ <tag> ] = <perl expression>

eeval [ <tag> ] = <perl expression>

Elements

• tag: like in commands <tag> is a freely choosable expression and provides something like a primary key to identify the eval expression.
• perl expression: This can be an arbitrary perl expression.

Return code A eeval statement is a small perl script, and you can set the RC like in other perl scripts via the $? variable.
Nevertheless it is a little bit special: you have to set the RC first and then, in the last statement, define the output of the eeval statement. Look into the following example to understand this:[code perl]eeval [ RC_CRITICAL_output_xyz ] =

 # the RC has to be shifted to the high byte of $?     
 $? = $CRITICAL << 8;
 # the output itself is the return value (well, that's not quite intuitive ;))
 return "xyz";[/code]

Examples Examples for eval and eeval can be seen here.

The info command is the door to add information to check_multi.

If you did it with dummy command / echo or dummy eval commands, the disadvantage was that every command bears its own status. And documentation should not show any status.

There are multiple forms of info commands:

info [ tag::pre[_<state>  ] = this text comes before a plugin output

info [ tag::post[_<state> ] = and this text comes after the plugin output\nnewlines are allowed

Format

Details

• tag: like in commands <tag> is a freely choosable expression and provides something like a primary key to identify the info expression.
  Note: the child check which info is related to, must exist before the info command. Otherwise info is not able to add its pre or post attributes.
  You will get an error message otherwise.
• 4 infos max per check: There may be multiple info lines for each check. 'info [ tag::pre ]' and 'info [ tag::post ]' commands will be displayed in any case, so you may have two info blocks before and two info blocks after, one for the generic info and one for the specific state info.
• Use raw content: you don't need to use ” or ' to enclose your info blocks, just print them as they are. Newlines will be translated into ”<br />” in HTML mode.

Format

livestatus [ <tag> ] = <host>:<service>

livestatus [ <tag> ] = <host>

Elements

• tag : like in commands <tag> is a freely choosable expression and provides something like a primary key to identify the eval expression.
• host : host on which the service is running.
• service: service_description of the wanted service.

Note: Additional variables

check_multi -s livestatus=</path/to/livestatus-socket>

Multiple Hosts / Services:
If you want to specify more hosts / services, use regular expressions, which are perl like enclosed in '/'
On the other hand: if you don't use slashes, the expression is taken literally.
/host[289]/ e.g. means host2, host8 and host9, while host123 just means host123.

Another Example:

livestatus [ web ] = /srv.*/:httpd

Let's assume that you have three webservers, called h1, h2 and h3, and each of them has a service httpd.
check_multi now expands the child check web to three webservices as follows:

livestatus [ web_h1_httpd ] = h1:httpd
livestatus [ web_h2_httpd ] = h2:httpd
livestatus [ web_h3_httpd ] = h3:httpd

The new tags are built like this:<tag>_<host>_<service>.

You can also specify services with regular expressions.

Question at the end: what will happen, if you specify the following?

livestatus [ all ] = /.*/:/.*/

Yes: all services in your installation will be combined within one check_multi call…
This will most likely break your Nagios buffer sizes and it's not really reasonable . But it shows how this expansion can work.

First of all: For the default function of check_multi the state definition feature is not needed .
The state definition specifies how the results of the child plugin calls should be evaluated. The standard or builtin evaluation looks for any non OK state and returns the state with the highest severity.
The evaluation order is: OK → UNKNOWN → WARNING → CRITICAL.

Format

state [ <OK|UNKNOWN|WARNING|CRITICAL> ] = <expression>

Elements

• state: can be one of the standard Nagios states OK, UNKNOWN, WARNING and CRITICAL.
• expression: a logical expression in Perlish style determines when the given state evaluates to true or false.
  Example: (disk_root == CRITICAL || disk_opt == CRITICAL).
  • Perl operators are allowed. Since the precedence of perl operators is sometimes a little bit sophisticated please use brackets extensively - this also provides better readability .
  • check tags are replaced by their return codes, e.g. disk_root → 2 (CRITICAL)
  • COUNT(<state>) is replaced by the number of child check results for the mentioned state.
    Example: COUNT(WARNING) → 2
  • COUNT(ALL) will be replaced by the number of checks, e.g. for rules like this:
    state [ CRITICAL ] = COUNT(CRITICAL) == COUNT(ALL)
  • Note: all special words (COUNT, OK, UNKNOWN, WARNING, CRITICAL) can be used case-insensitive, so there's no problem to write count(warning).

Examples

• Builtin standard states (note - evaluated in this order):
  

  state [ OK       ] = 1
  state [ UNKNOWN  ] = COUNT(UNKNOWN)  > 0
  state [ WARNING  ] = COUNT(WARNING)  > 0
  state [ CRITICAL ] = COUNT(CRITICAL) > 0

• Ignore WARNING and UNKNOWN and only rise CRITICAL states:
  

  state [ UNKNOWN  ] = (1==0)
  state [ WARNING  ] = (1==0)
  state [ CRITICAL ] = COUNT(CRITICAL) > 0

• Clustering: only rise critical if more than one CRITICAL state encountered, otherwise raise WARNING:
  

  state [ WARNING  ] = COUNT(WARNING) > 0 || COUNT(CRITICAL) > 0
  state [ CRITICAL ] = COUNT(CRITICAL) > 1

Config file vs. command line All states can also be specified on the command line with options -o/-u/-w/-c <state expression>. The precedence in the state evaluation is

1. Builtin evaluation
2. Config file
3. Command line

Format

statusdat [ <tag> ] = <host>:<service>

(and since 2010/06/04)

statusdat [ <tag> ] = <host>

Elements

• tag : like in commands <tag> is a freely choosable expression and provides something like a primary key to identify the eval expression.
• host : host on which the service is running.
• service: service_description of the wanted service.

Note: the status.dat is normally taken from <nagiosdir>/var/status.dat. If you want to specify another path, do it like this:

check_multi -s status_dat=</path/to/status.dat>

Multiple Hosts / Services:
If you want to specify more hosts / services, use regular expressions, which are perl like enclosed in '/'
On the other hand: if you don't use slashes, the expression is taken literally.
/host[289]/ e.g. means host2, host8 and host9, while host123 just means host123.

Another Example:

statusdat [ web ] = /srv.*/:httpd

Let's assume that you have three webservers, called h1, h2 and h3, and each of them has a service httpd.
check_multi now expands the child check web to three webservices as follows:

statusdat [ web_h1_httpd ] = h1:httpd
statusdat [ web_h2_httpd ] = h2:httpd
statusdat [ web_h3_httpd ] = h3:httpd

The new tags are built like this:<tag>_<host>_<service>.

You can also specify services with regular expressions.

Question at the end: what will happen, if you specify the following?

statusdat [ all ] = /.*/:/.*/

Yes: all services in your status.dat will be combined within one check_multi call…
This will most likely break your Nagios buffer sizes and it's not really reasonable . But it shows how this expansion can work.