SICM Configuration and Operation Guide

Prerequisites and Tools

An understanding of xml, ip networking and snmp is assumed.

• Editing Tool:
ConText was used to develop SICM, hence is recommended if you do not already have a preferred editor.
• snmp Query Tool:
Getif for Win32 or net-snmp for Linux are recommended for the snmp querying of devices to assist in determining OIDs, etc.

Configuring SICM (Win32 and Linux)

Up to four files may require modification:

1. sicm.cgi
2. Configuration File (eg. config.xml)
3. Template File (eg. templates.xml)
4. Target File (eg. targets.xml)

When making changes to the Configuration, Template or Target files, SICM will need to be restarted to activate the changes.

Below are details of all configuration attributes, including descriptions, limitations and examples. Each attribute is also identified as optional or mandatory. If you are new to SICM, this may seem daunting. To assist in overcoming this it is recommended to:

1. Start with a minimal configuration of a few devices by removing or commenting out optional components
<!-- an example of xml comments -->
2. Within a console (dos or unix command line), use the test (-t) mode when first executing SICM
3. If test mode is successful, execute SICM in normal mode
4. Modify config.xml, targets.xml and templates.xml as appropriate and restart SICM. In some circumstances .rrd file may need to be deleted (SICM will create new files for you). If there are difficulties, use verbose (-v) mode and logging to assist diagnosis
5. Upon the initial configuration stabilizing, commence expanding the templates and options as required
6. When you are comfortable, continue to add devices as required
sicm.cgi Setup
1. When using Apache 2 in a Win32 environment, line 1 in sicm.cgi may require modification:
• Example for Linux and Win32/IIS:  line 1:  #!/usr/bin/perl -w
• Example for Win32/Apache2:     line 1:  #!c:/perl/bin/perl.exe -w

so that the first line of sicm.cgi points to the Perl executable

2. Line 12 in sicm.cgi may require modification:
• Example for Win32: line 12: $configfile = "C:\\Program Files\\sicm\\config.xml"; • Example for Linux: line 12:$configfile = "/etc/opt/sicm/config.xml";

Where:
\$configfile = The full path of the configuration file

Note: For Win32, use \\ for directories instead of \.

Configuration File Setup

The configuration file, is a XML file with the following properties (refer to config.xml as an example):

   <opt>
  <web_dir>C:\Inetpub\wwwroot\sicm</web_dir>
  <targetfile>C:\Program Files\sicm\targets.xml</targetfile>
  <templatefile>C:\Program Files\sicm\templates.xml</templatefile>
  <nix_mgt user="sicm" group="sicm" PID_path="/var/run/sicm.pid" />
  <alert_file>C:\Program Files\sicm\alert.xml</alert_file>
  <ping_file>C:\WINDOWS\system32\ping.exe</ping_file>
  <log_file>C:\Program Files\sicm\sicm.log</log_file>
  <imgformat>PNG</imgformat>
  <width>480</width>
  <height>120</height>
  <step>300</step>
  <step_timeout>2</step_timeout>
  <port>161</port>
  <retries>2</retries>
  <timeout>5</timeout>
  <ping_count>4</ping_count>
  <graph_font>C:\WINNT\Fonts\unvr55w.ttf</graph_font>
  <graph_font_color>555555</graph_font_color>
  <graph_tab>30</graph_tab>
  <mail_server>192.168.1.50</mail_server>
  <mail_from>sicm@example.com</mail_from>
  <log_purge>14</log_purge>
</opt>

 (mandatory) the full path to a writeable directory, where the cgi and htmt template files are located. The subdirectories, rrd and png files will be created in this directory. Example for Win32/IIS: C:\Inetpub\wwwroot\sicm Example for Win32/Apache: C:\Program Files\Apache Group\Apache2\sicm Example for Linux/Apache: /var/www/html/sicm (mandatory) the full path to the target file, where the targets are defined Example for Win32: C:\Program Files\sicm\targets.xml Example for Linux: /etc/opt/sicm/targets.xml (mandatory) the full path to the template file Example for Win32: C:\Program Files\sicm\templates.xml Example for Linux: /etc/opt/sicm/templates.xml (optional) For Linux/Unix only: Set the user, group and process IDs Note: If you have a requirement for modifying the ownership of the SICM process when running on Linux/Unix, then will enable this. Otherwise use of is unnecessary. Reference: Linux Tutorial - Identifiers To set the user and group that the process is running on behalf of: Example user="sicm" Example group="sicm" To create a PID for SICM: Example PID_path="/var/run/sicm.pid" Default: user, group and process IDs are not set (mandatory) the full path to the alert file Example for Win32: C:\Program Files\sicm\alert.xml Example for Linux: /var/opt/sicm/alert.xml (optional) the full path to the operating system ping file Example for Win32: C:\WINDOWS\system32\ping.exe Example for Linux: /bin/ping If a ping file is not defined, the ping functionality internal to SICM will be used. (recommended) the full path to the log file Example for Win32: C:\Program Files\sicm\sicm.log Example for Linux: /var/log/sicm.log If a log file is not defined, logging will not be activated. (optional) the image format. You may choose: PNG - Portable Network Graphics GIF - Graphics Interchange Format IGIF - Interlaced GIF default = PNG (optional) the WIDTH of the graphic in pixels: minimum = 100 maximum = 1000 default = 400 (optional) the HEIGHT of the graphic in pixels: minimum = 25 maximum = 250 default = 100 (optional) the default period in seconds that data will be fed into the RRD minimum = 60 (1 minute) maximum = 14400 (4 hours) default = 300 (5 minutes) (optional) "step_timeout*frequency" is the timeout value (seconds) that without getting data, rrdtool will plot 0 in the graphics. minimum = 1 maximum = 2 default = 2 (optional) the snmp agents UDP port A natural number, only using digits "0-9" default = 161 (optional) The number of times the snmp agent on the target will be queried for data min = 1 max = 5 default = 2 (optional) the period (in seconds) to wait for an snmp agent to respond to a query min = 1 max = 20 default = 4 (optional) the number of ICMP pings used to determine the max, min and average round-trip-times. min = 1 max = 20 default = 4 (optional) the font path and file name used to create the graphs Example: C:\WINNT\Fonts\times.ttf Example: /usr/share/fonts/bitstream-vera/Vera.ttf (optional) the RRBBGG color to be presented by the graph font: range is from 000000 (black) to FFFFFF (white) default is eeeeee (light gray), example: 008000 is green, example: 0000FF is blue, example: FF0000 in red SEE the wc3 web colors for more information (optional) the RRBBGG color to be presented by the graph font: range is from 000000 (black) to FFFFFF (white), default is ffffff (white) (optional) the RRBBGG color to be presented by the graph font: range is from 000000 (black) to FFFFFF (white), default is 000000 (black) (optional) the tab width (in pixels) used to create the legend in the graphs A natural number, only using digits "0-9" example: 30 default: 40 (optional) the e-mail server that alerts are sent to. server name or ip address example: mail.example.com Note: no mail server dictates no mail (optional) the e-mail address the alerts are sent from. example: sicm@example.com default = sicm@example.com (optional) The number of days before log file entries are purged. A natural number, only using digits "0-9" min = 1 max = 90 default = 14
Template File Setup

The template file is a XML format file. Additional templates should be added, one for each graph required. (refer to templates.xml as an example):

   <templates>
 <template template_id="bandwidth_use">
   <metric oid="1.3.6.1.2.1.2.2.1.10.int" data_source_type="COUNTER"
       convert="8"plot="AREA" color="00BB00" line="95:00BB00">inOctets</metric>
   <metric oid="1.3.6.1.2.1.2.2.1.16.int" data_source_type="COUNTER"
       convert="8"plot="LINE2" color="0000FF" line="95:0000FF">outOctets</metric>
   <k_base>1000</k_base>
   <vertical_label>bit/sec</vertical_label>
   <short_label>bit/s</short_label>
 </template>
 <template template_id="cisco_cpu">
   <metric oid="1.3.6.1.4.1.9.2.1.57.int" data_source_type="GAUGE"
       plot="AREA" color="00BB00" consolidate="MAX">cpuBusy1min</metric>
   <metric oid="1.3.6.1.4.1.9.2.1.58.int" data_source_type="GAUGE"
       plot="LINE2" color="0000FF" consolidate="MAX">cpuBusy5min</metric>
   <k_base>1000</k_base>
   <vertical_label>Utilisation (%)</vertical_label>
   <short_label>%</short_label>
   <graph_max>100</graph_max>
   <graph_min>0</graph_min>
   <number_format>fixed</number_format>
 </template>
</templates>
• Each template produces one graph per target. Each graph has a horizontal and vertical axis.
• The horizontal axis relates to time, and the vertical axis is determined by the metrics assigned to the template.
• When selecting the metrics for a template, ensure they have the same unit of measure. Examples include bit/sec, utilisation (%), etc.
• Note: the 'convert' factor can be used to align similar metrics. Example: Using 'convert=8' can align inOctets with inBitsSec.

Hint: It is recommended to trial each template prior to assigning it to many targets as if a change to the template is required, it may result in the need to manually delete and allow SICM to re-generate the associated .rrd files.
If: 'template_id', 'metric', 'data_source_type', (or 'interface' in the target file) are modified, the structure of the .rrd database will be modified, hence all effected .rrd files for all effected targets will need to be deleted prior to re-starting SICM. At this point in time, it is not possible to modify the structure of the .rrd files, hence manual deletion and re-generation by SICM is the only solution.

 (mandatory) A description of the group of metrics that are being measured template_id's must be unique Allowed characters are "a-zA-Z0-9_-" and " " Note: "ping" is a mandatory template for ICMP ping only example: bandwidth_use example: cisco_cpu (mandatory) A description of the measured value that determines capacity of the target. Example1: outOctets Example2: apache_hits Each template requires one or more metrics. Allowed characters are "a-zA-Z0-9_-" and " " Maximum length is 11 characters Metrics are typically descriptions of snmp OIDs, Modules or ping Hint: use the snmp OID text name (ie MIB) for metric content example: inOctets example: cpuBusy5min Either oid, or module (mandatory) For oid, the snmp OID in dotted number format, excluding "int" Allowed characters are "0-9." Where an interface ID forms part of the OID, the special characters "int" are used. The interface_id (as per the target file) replaces "int" when SICM processing occurs. example: 1.3.6.1.2.1.2.2.1.10.int example: 1.3.6.1.4.1.9.2.1.58.int If using snmp to monitor Windows 2000, it is recommended to install Garth William's SNMP4W2K on each target, refer: http://www.snmp4tpc.com. If using snmp to monitor Windows 2000 disk space, remember to initialise diskperf -y on each target. For module, the program line to execute the module's subroutine, including parameters The program line parameters may include any of the SICM variables %ip_address%, %metric% and %community% Refer to: How to configure Modules in SICM for more information example: Apachestats::stats( '%ip_address%', 'apache_total_hits') data_source_type (mandatory) - refer to rrdtool documentation either GAUGE is for things like temperatures or value of a share, or COUNTER is for continuous incrementing counters like the InOctets counter in a router. or DERIVE will store the derivative of the line going from the last to the current value of the data source. DERIVE works exactly like COUNTER but without overflow checks. or ABSOLUTE is for counters which get reset upon reading. default is COUNTER convert (optional) - a multiplication factor that converts "oid" values into "vertical_label" values a number the number 0 is invalid example: to convert "Octets" to "bits", convert = 8 example: to convert "Errors per Second" to "Errors per Minute", convert = 60 default = 1 plot (mandatory) - the plot type to be presented on the graph: either AREA (the area between 0 and the value will be filled), or LINE1 (a line of width approximately 1 pixel), or LINE2 (a line of width approximately 2 pixels), or LINE3 (a line of width approximately 3 pixels), or STACK (a line or area stacked upon the previous) default is LINE2 AREA or LINEx must always precede STACK SEE the RRD GRAPH manual for more information color (recommended) - the RRBBGG color to be presented on the graph: range is from 000000 (black) to FFFFFF (white), default is 000000 (black), example: 008000 is green, example: 0000FF is blue, example: FF0000 in red SEE the w3c web colors for more information consolidate (optional) When SICM (rrd) consolidates data from daily to weekly, monthly and yearly, the method of consolidation is to either AVERAGE the data or select the MAXimum value of the data to be consolidated. This attribute only effects the graph output and can be changed at the users discretion (and restarting SICM) without impacting the underlying data. either AVERAGE, or MAX default is AVERAGE line (optional) - Draw a horizontal line on the graph of the AVERAGE, MAXimum or 95th percentile Format is: type:color where: type is: AVERAGE, MAX, or 95, and color is from 000000 (black) to FFFFFF (white) example: 95:0000FF (draw a blue line of the 95%ile of the metric) Note: the line is only visible after setting yes in the appropriate interface(s) of the targets.xml file. (optional) The number of units per "k": either 1000 for traffic measurement (1 kbit/s = 1000 bits/s), or 1024 for memory measurement (1 KB = 1024 bytes) default = 1000 (recommended) A description of the vertical scale units of measure: example: "bit/s" for network traffic (note the graph will automatically add G, M or k as required), example: "utilisation (%)" for CPU utilisation, no default (recommended) A short description of the vertical scale units of measure: A maximum of 5 characters is permitted, any additional characters will be trimmed The short_label is used in the legend below the graph example: "b/s" for network traffic (note the graph will automatically add G, M or k as required), example: "%" for CPU utilisation, no default and (optional) The upper and lower limits defined on the graph: A number The graph will change these limits if the values collected exceed the limits graph_max and graph_min are also optionally set in the Target file. The Target file graph_max and graph_min settings take precedence over the Template file graph_max and graph_min settings. example: graph_max = 10e6 eg 10 Mbit/s example: graph_max = 10000000 eg 10 Mbit/s no default (optional) The format of the numbers in the graph legend either SI (use of the appropriate SI magnitude unit ( u, m, k, M, G ) and the value will be scaled accordingly, i.e. 654321 -> 654.32 k), or fixed (number format is fixed at 2 decimal places, SI units are not used) default: SI Hint: It is recommended to initially use SI for all number_formats, only changing them to fixed if the output is overly complex or confusing. Where the number is always 3 digits or less, it may be appropriate to use fixed: e.g. for cpu the number is always ≤ 100, so to avoid 971.12 m% (milli-percent), use fixed i.e. 0.97 %.
Target File Setup

The target file is a XML format file. Additional targets (i.e. devices) should be added, one for each graph required. (refer to targets.xml as an example):

   <targets>
 <target target_id="router_1" ip_address="127.0.0.1" community="public">
 <interface_alert>2:sysadmin@example.com;netmgr@example.com</interface_alert>
 <menu>site_1</menu>
 <target_desc>cisco router at site_1</target_desc>
   <template template_id="ping" get_sysuptime="no">
   </template>
   <template template_id="cisco_cpu">
   </template>
   <template template_id="interface_errors" frequency="3600" create_graph="no">
     <interface>
       <interface_id>1</interface_id>
       <interface_desc>Ethernet 0</interface_desc>
       <graph_max>10e6</graph_max>
       <graph_min>0</graph_min>
       <input_max>40e6</input_max>
       <line>yes</line>
       <hrule>10e6</hrule>
       <hrule_color>660000</hrule_color>
     </interface>
     <interface>
       <interface_id>2</interface_id>
       <interface_desc>Serial 0 to site_2</interface_desc>
     </interface>
   </template>
 </target>
 <target target_id="win_host" ip_address="127.0.0.1" version="2" community="public">
   <menu>Site 2</menu>
   <target_desc>Windows Host at Site 2</target_desc>
   <template template_id="win_disk_cde" frequency="3600">
     <interface>
       <interface_id>1</interface_id>
       <interface_desc>Volumes C,D,E</interface_desc>
       <metric_alert>lt:25:2</metric_alert>
     </interface>
   </template>
   <template template_id="win_cpu">
   </template>
   <template template_id="iis_web_hits">
   </template> 
 </target>
 <target target_id="ups_1" ip_address="127.0.0.1" community="public">
   <port>161</port>
   <menu>site_2</menu>
   <target_desc>APC UPS at Site_2</target_desc>
   <template template_id="apc_ups" frequency="1800">
   </template>
 </target>
</targets>

 (mandatory) The name of the target Allowed characters are "a-zA-Z0-9_.-" example: localhost example: ftp.example.com (mandatory) The ip address of the target Format is nnn.nnn.nnn.nnn where n are natural numbers between 0 and 254 example: 127.0.0.1 (optional) The snmp version of the target Either version = 1 (snmp version 1) Or version = 2 (snmp version 2c) Default version = 1 (mandatory) The snmp read community of the target Any characters are allowed minimum number of characters is 1 example: public (optional) Format is: n:email@example.com n = the number of query failures before an e-mail alert is triggered A natural number, only using digits "0-9" min = 1 max = 999 email@example.com = the e-mail address to send alerts to for the target Must be a valid e-mail address Multiple e-mail addresses can be configured, separated by ; or , (no spaces) example: someone@example.com example: someone@example.com;someone_else@example.com example: 2:sicm@example.com (if there are 2 concurrent failures, send an e-mail) If the interface_alert is not defined, no e-mail will be generated in the event of an alert (optional) The target's snmp agent UDP port A natural number, only using digits "0-9" The in the config file will be used if not specified example: 161 (optional) The web menu The target_id will be the menu if a menu is not specified (optional) A description of the device The target_desc is displayed in the web page title Allowed characters are "a-zA-Z0-9_.-" and " "

Hint: For some templates the graph_max and graph_min is the same for all targets, eg: cisco_cpu where graph_max is fixed at 100% and graph_min is fixed at 0%. In this case use the template to set the graph_max and graph_min. For other templates the graph_max and graph_min is not the same for all targets, eg: bandwidth_use where graph_max can be different for each target (eg 10baseT Ethernet = 10e6 bit/s or E1 serial link = 2048e3 bit/s) and graph_min is fixed at 0 bit/s. In this case use the target to set the graph_max and graph_min.

How to configure Modules in SICM

When configuring the templates file (eg templates.xml), modules may be configured within each template. If modules are used, the following requirements apply:

• Modules must be located in SICM's \lib directory.
• The module must return a single numeric value. SICM validates this, returning UNKNOWN if invalid.
• The module is required to exit gracefully in the event of a failure.
• The module must be written in perl.
• Three optional variables are avaivable within SICM to assist module parameters:
• %ip_address% - the ip address of the target
• %metric%     - the metric as defined in the target file (eg targets.xml)
• %community%  - the snmp community string for the target

An example is apache_hits where %ip_address% is used as a module parameter:

   <metric module="Apachestats::stats( '%ip_address%', 'apache_total_hits')"
   data_source_type="COUNTER" convert="60" plot="LINE2"
   color="00BB00" consolidate="MAX">apache_hits</metric>

It is strongly recommended to test the module before using within SICM.

For further information, refer to: Modules

How Alerts in SICM work

Alerts are triggered by two methods:

• <interface_alert> - failure of the target to respond to a (ping, snmp or module) query. Configuration is only required for sending e-mail alerts.
• <metric_alert>    - a query result does not meet a criteria. Configuration is required to set the criteria.

Note: to e-mail a <metric_alert>, an <interface_alert> must be configured

Alerts are presented in two forms:

• On the SICM Alert web page (sicm.cgi).
• All interface alerts are presented on the SICM Alert web page (no configuration required)
• Metric alerts are presented on the SICM Alert web page if <metric_alert> is configured
• Send via e-mail. The sending of alerts via e-mail is optional. To send e-mail alerts:
• Configure <mail_server> and <mail_from> in the configuration file (config.xml)
• Configure <interface_alert> in the target file (target.xml)
• Optionally configure criteria in <metric_alert> in the target file (target.xml)

How SICM works

There are two program files that comprise SICM, the server program sicm.pl and the browser program sicm.cgi.

1. sicm.pl is the server program that:
1. loads then verifies the configuration, template and target files,
2. queries targets at predetermined intervals,
3. writes the data to .rrd files (creating them if they do not exist),
4. creates graphical images of the captured data,
5. creates an alert file of unsuccessful queries, and
6. sends e-mail alerts.

There are two methods of running the SICM server:

• Console Mode - using the DOS or terminal console. The SICM server has no graphical interface, the method of viewing SICM server messages are via the console or the log file.
• Service Mode - When SICM server is correctly configured and has been operating in console mode without issue for a reasonable time, the SICM server can be configured to operate as a service. This eliminates the use of the console and provides for automatic start-up upon the server booting without user login or interaction.

sicm.pl creates sub-directories in <web_dir>, one for each target. In each target subdirectory, one .rrd file and 4 images will be produced for each template associated with that target.

2. sicm.cgi is a program that presents the graphs via a web browser (Internet Explorer, Firefox, etc).
1. sicm.cgi is executed each time the user requests new or updated presentation of the graphs.
2. Data is developed from sicm.cgi and parsed to template files (.htmt) used to create the presentation. The output is then presented to the user using a standard web browser.
3. The alert file is also queried and displayed as requested by the user.
4. The most recent 100 lines of the log file are also queried and displayed as requested by the user.

Starting SICM

Normal SICM Server Operation

At the console prompt:

1. Go to the directory where sicm.pl resides:
Example: C:\Program Files\sicm>
2. If starting for the first time it is recommended to run the program in test mode:
Win32 Example: C:\Program Files\sicm>sicm.pl -c config.xml -t, or
Linux Example: [root@host sicm] perl sicm.pl -c /etc/opt/sicm/config.xml -t.
If any error messages are displayed after the test, attend to as appropriate.
3. Run the program:
Win32 Example: C:\Program Files\sicm>sicm.pl -c config.xml, or
Linux Example: [root@host sicm] perl sicm.pl -c /etc/opt/sicm/config.xml.
4. In the console a start-up message should appear:
Example: 0000000000 main(): Loop start Day Mth 0 00:00:00 0000
5. If any error messages are displayed after the start-up, attend to as appropriate.
6. Only when SICM has been tested for a time, if appropriate, it can be installed as a service.

At the console prompt, to stop SICM, use Ctrl^C.

Note, it may take up to 10 seconds to stop processing.

SICM Browser Operation

To view the graphs via a web interface, go to the configured web path / cgi file name:
Example: http://localhost/sicm/sicm.cgi

To align SICM Browser (left or center)
• To Align SICM Browser left (default), modify header.htmt, lines 74-77 to:
   <!--<div style="text-align: center;">
<div style="text-align: left; margin: 0em auto; width: 70%;">-->
<div>
<div>
• To Align SICM Browser center, modify header.htmt, lines 74-77 to:
   <div style="text-align: center;">
<div style="text-align: left; margin: 0em auto; width: 70%;">
<!--<div>
<div>-->
SICM Server Operation Options

To test and verify the correct operation of SICM, two settings have been developed.
Also, in the event of anomalies, these two methods can be used to correct the condition:

Note: It is recommended that test and verbose modes are only activated in console mode.

1. Test Mode (sicm.pl -c config.xml -t)
When test mode is activated, SICM will test and log itself including the configuration, template and target files but will not query any targets. Included is a test of RRDs (the perl RRDs shared module) to a test_x.rrd file, where the test .rrd file is created from the command rrdtool create test.rrd DS:test:GAUGE:300:U:U RRA:LAST:0.5:1:10.
2. Verbose Mode (sicm.pl -c config.xml -v)
When verbose mode is activated, SICM will provide verbose logging of all activities to assist in verifying and identifying configuration corrections including target query anomalies.

Both test mode and verbose mode will output to console as well as to the log file (if defined in the configuration file).

SICM Server Maintenance
1. The menu in the web browser provides an alert status.
2. Remember to regularly review the log file if it has been defined (SICM will purge old logs automatically).
SICM v0.95 Copyright 2003-2006 Thomas Price