IBM Tivoli Software IBM Tivoli Software

[ | | | ]


IBM Tivoli Storage Manager for UNIX: Backup-Archive Clients Installation and User's Guide


Appendix A. Using the Tivoli Storage Manager central scheduler


Overview of the TSM scheduler

The Tivoli Storage Manager central scheduler allows client operations to occur automatically at specified times. In order to understand scheduling with Tivoli Storage Manager, several terms need to be defined:

schedule definition
A definition on the Tivoli Storage Manager server which specifies critical properties of the automated activity including the type of action, the time the action should take place, and how frequently the action will take place. There are numerous other properties which can be set (see the appropriate Tivoli Storage Manager Administrator's Reference Guide for a detailed description of the define schedule.)

schedule association
An assignment to a specific schedule definition for a client node. Multiple schedule associations allow single schedule definitions to be used by many client nodes. Because schedule definitions are included with specific policy domains, it is only possible for nodes defined to a certain policy domain to be associated with schedules defined in that domain.

scheduled event
A specific occurrence of when a schedule will be executed for a node. The following conditions must be met before automatic scheduled events will take place for a client:

When creating a schedule definition on the Tivoli Storage Manager server, schedule actions that you can take include incremental, selective, archive, restore, retrieve, imagebackup, imagerestore, command, and macro. The scheduled action that is most frequently used is incremental with the objects parameter left undefined. With this setting, the Tivoli Storage Manager client performs a domain incremental backup of all drives defined by the client domain option. A schedule definition using the command action allows an operating system command or shell script to be executed. When automating tasks for Tivoli Storage Manager for Data Protection clients, you must use command action schedule definitions which invoke the command line utilities for the "Tivoli Storage Manager for" application.

The schedule startup window indicates the acceptable time period for a scheduled event to start. The startup window is defined by these schedule definition parameters: startdate, starttime, durunits, and duration. The startdate and starttime options define the beginning of the startup window for the very first scheduled event. The beginning of the startup windows for subsequent scheduled events will vary depending on the period and perunit values of the schedule definition. The duration of the schedule window defines the length of the startup window. The schedule action is required to start within the startup window. To illustrate, consider the results of the following schedule definition:

define schedule standard test1 action=incremental starttime=12:00:00 period=1
perunits=hour dur=30 duru=minutes

Event Window start Window end Actual start (just an example, times will vary)
1 12:00:00 12:30:00 12:05:33
2 13:00:00 13:30:00 13:15:02
3 14:00:00 14:30:00 14:02:00
and so on


The variation in actual start times is a result of the randomization feature provided by the Tivoli Storage Manager central scheduler which helps to balance the load of scheduled sessions on the Tivoli Storage Manager server.

Handling spaces in file names in schedule definitions

When defining or updating a schedule objects parameter with file specifications that contain blank spaces, use double quotes around each file specification that contains blanks, then single quotes around all of the specifications. Examples:

objects='"/home/proj1/Some file.doc"'
objects='"/home/proj1/Some file.doc" "/home/Another file.txt"
/home/noblanks.txt'
objects='"/home/My Directory With Blank Spaces/"'
 

This will ensure that /home/proj1/Some file.docis treated as a single file name, as opposed to three separate files (/home/proj1/Some, and file.doc).

You can also refer to the objects parameter information for the define schedule and update schedule commands in the appropriate IBM Tivoli Storage Manager Administrator's Reference

Preferential start times for certain nodes

Occasionally, you may want to ensure that a particular node begins its scheduled activity as close as possible to the schedule's defined start time. The need for this typically arises when prompted mode scheduling is in use. Depending on the number of client nodes associated with the schedule and where the node is in the prompting sequence, the node might be prompted significantly later than the start time for the schedule. In this case, you can perform the following steps:

  1. Copy the schedule to a new schedule with a different name (or define a new schedule with the preferred attributes).
  2. Set the new schedule priority attribute so that it has a higher priority than the original schedule.
  3. Delete the association for the node from the original schedule, then associate the node to the new schedule.

Now the Tivoli Storage Manager server will process the new schedule first.


Understanding scheduler processing options

There are several processing options which impact the behavior of the client scheduler. On the Tivoli Storage Manager client, you can define most of these options in the client user options file (dsm.opt) or client system options file (dsm.sys). However, some of these options can be set globally on the Tivoli Storage Manager server for all Tivoli Storage Manager clients. The Managing Throughput of Scheduled Operations section of the Tivoli Storage Manager Administrator's Guide provides detailed information on all topics described in the section.

Option Client defined Server defined Server global override
managedservices X

maxcmdretries X
set maxcmdretries command
maxschedsessions
X
postschedulecmd, postnschedulecmd X

preschedulecmd, prenschedulecmd X

queryschedperiod X
set queryschedperiod command
randomize
X
retryperiod X
set retryperiod command
schedcmddisabled X

schedlogname X

schedlogretention X

schedmode X
set schedmodes command
sessioninitiation X X (update node command)
tcpclientaddress X X (also defined on server when sessioninit=serveronly as part of the node definition)
tcpclientport X X (also defined on server when sessioninit=serveronly as part of the node definition)

Client defined options are defined in the dsm.sys or dsm.opt file depending on the option and platform. The Tivoli Storage Manager server can also define some options in a client options set, or as part of the options parameter of the schedule definition. The Tivoli Storage Manager server can also set some options globally for all clients. By default, the client setting for these options is honored. If the global override on the Tivoli Storage Manager server is set, the client setting for the option is ignored. Defining client options as part of the schedule definition is useful if you want to use specific options for a scheduled action that differ from the option settings normally used by the client node, or are different for each schedule the node will execute.

The schedmode option controls the communication interaction between the Tivoli Storage Manager client and server. There are three variations on the schedule mode: client polling, server prompted, server prompted with sessions only originated by the Tivoli Storage Manager server (sessioninit=serveronly.)

Handling return codes from preschedulecmd and postschedulecmd Scripts

Beginning with Tivoli Storage Manager version 5.1, the scheduler exhibits the following behavior when the preschedulecmd and postschedulecmd options are used:

When interpreting the return code from a command, Tivoli Storage Manager considers 0 to mean success, and anything else to mean failure. While this behavior is widely accepted in the industry, it is not 100% guaranteed. For example, the developer of the widget.exe command might exit with return code 3, if widget.exe ran successfully. Therefore, it is possible that the preschedulecmd or postschedulecmd command may end with a nonzero return code and be successful. To prevent Tivoli Storage Manager from treating such commands as failed, you should wrap these commands in a script, and code the script so that it interprets the command return codes correctly. The script should exit with return code 0 if the command was successful; otherwise it should exit with a nonzero return code. The logic for a script running widget.exe might look like this:

run 'widget.exe'
   if lastcc == 3
      exit 0
   else
      exit 1
 

See the following references for more information:


Using the client acceptor to manage scheduler services versus the legacy scheduler services

You can configure the Tivoli Storage Manager client to manage the scheduler process via the Tivoli Storage Manager Client Acceptor (CAD). The CAD provides a light-weight timer which automatically starts and stops the scheduler process as needed. Alternatively, the traditional method keeps the Tivoli Storage Manager scheduler process running continuously. Generally, using the CAD to manage the scheduler is the preferred method. These methods are compared as follows:

CAD-managed Services

Tivoli Storage Manager legacy scheduler services

Setting the client scheduler process to run as a background task and start automatically at boot time

You can configure the Tivoli Storage Manager client scheduler to run as a background system task which starts automatically when your system is booted. This is true for both CAD-managed and traditional methods of running the Tivoli Storage Manager client scheduler. When running a CAD-managed schedule, only the CAD process should be set to start automatically at boot time; not the scheduler process. For the traditional method, the scheduler process should be set up to start automatically at boot time.

You can configure the CAD to run as a background system task which starts automatically when your system is booted. To configure the CAD to manage scheduled backups, you must set the managedservices option to manage the scheduler, or both the scheduler and web client. The method for setting up the CAD as a system task varies for each platform.

In order for the scheduler to start unattended, you must enable the client to store its password by setting the passwordaccess option to generate, and store the password by running a simple Tivoli Storage Manager client command such as dsmc query session. Note that for testing purposes, you can always start the scheduler in the foreground by running dsmc sched from a command prompt.

To start the scheduler automatically at boot time, use either of the following methods:

CAD-managed
  1. In your client system options file (dsm.sys), set the managedservices option to schedule or schedule webclient.
  2. Add the following entry into the system startup file (/etc/inittab for most platforms):
    tsmcad::once:/usr/bin/dsmcad > /dev/null 2>&1 # TSM Client Acceptor
    Daemon
    
  3. In your client system options file (dsm.sys), set the passwordaccess option to generate.
  4. Run a command like dsmc query sess to store the node password.

Traditional
  1. In your client system options file (dsm.sys), either set the managedservices option to webclient or do not define this option.
  2. Add the following entry into the system startup file (/etc/inittab for most platforms):
    tsmsched::once:/usr/bin/dsmc sched > /dev/null 2>&1 # TSM scheduler
    
    For OS/390 UNIX:
    1. Create a shell script called /tivoli/tsm/client/ba/bin/rundsmc which contains the following entries:
      cd /usr/lpp/Tivoli/tsm/client/ba/bin
      sleep 60
      ./dsmc schedule
      
      This prevents the creation of two jobs with the same name and enables automatic shutdown. You might need to customize the time for your system.
    2. Add the following entries in the /etc/rc file to set environment variables to retrieve the servername and nodename options from dsm.sys and to start the client scheduler, as follows:
         # Set environment variables to retrieve the servername and
         # nodename options from dsm.sys.
         export DSM_DIR=/tivoli/tsm/client/ba/bin
         export DSM_CONFIG=/tivoli/tsm/client/ba/bin/dsm.opt
         # Start the TSM Client scheduler and redirect outputs to
         # schedule.out instead of the /etc/log file.
         _BPX_JOBNAME='ADSMCLNT' /tivoli/tsm/client/ba/bin/rundsmc
         1>/tivoli/tsm/client/ba/bin/schedule.out 2>&1 &
       
      
      Note:
      Enter the _BPX_JOBNAME entry on a single line in the /etc/rc file.
  3. In your client system options file (dsm.sys), set the passwordaccess option to generate.
  4. Run a command like dsmc query sess to store the node password.
  5. To start the client scheduler on your client node and connect to the server schedule, enter the following command:
       dsmc schedule
    

    If the current directory is not in your PATH environment variable, enter the following command:

       ./dsmc schedule 
    

    When you start the client scheduler, it runs continuously until you close the window, end the process, or log off your system.

    To run the schedule command in the background and to keep the client scheduler running, even if you log off your system, enter the following:

       nohup dsmc schedule 2> /dev/null &
    

Managing multiple schedule requirements on one machine

In certain situations it is preferable to have more than one scheduled activity for each client system. Normally, you can do this by associating a node with more than one schedule definition. This is the standard method of running multiple schedules on one machine. You must ensure that the schedule windows for each schedule do not overlap. A single client scheduler process is not capable of executing multiple scheduled actions simultaneously, so if there is overlap, the second schedule to start will be missed if the first schedule does not complete before the end of the startup window of the second schedule. Suppose that most of the drives on your client system must be backed up daily, and that one drive containing critical data must be backed up hourly. In this case, you would need to define two schedules to handle this requirement. To avoid conflict between the hourly and daily backup schedule, the starttime of each schedule needs to be varied.

In certain cases, it is necessary to run more than one scheduler process on a system. Multiple processes require a separate options file for each process and must contain the following information:

The advantages of using multiple schedule processes:

The disadvantages of using multiple schedule processes:

Multiple schedule processes can run on UNIX platforms with either the CAD managed method, or the traditional method of running the scheduler. In either case, there are certain setup requirements:

You must start each dsmc sched command or instance with the -servername option to reference its unique stanza name in dsm.sys. For dsmcad, it is necessary to define the environment variable DSM_CONFIG for each instance of dsmcad to reference its unique option file.

The following is an example configuration of two schedule processes managed by the CAD in the client system options file (dsm.sys). Note that you must use full paths for the log file names to avoid the files being written in the root directory):

+--------------------------------------------------------------------------------+
|servername tsm1_sched1                                                          |
|	nodename        aixsvt01_sched1                                                |
|	tcpserv         firebat                                                        |
|	tcpclientport   1507                                                           |
|	passwordaccess  generate                                                       |
|	domain          /svt1                                                          |
|	schedmode       prompted                                                       |
|	schedlogname    /tsm/dsmsched1.log                                             |
|	errorlogname    /tsm/dsmerror1.log                                             |
|	managedservices schedule                                                       |
|                                                                                |
|servername tsm1_sched2                                                          |
|  nodename        aixsvt01_sched2                                               |
|  tcpserv         firebat                                                       |
|  tcpclientport   1508                                                          |
|  passwordaccess  generate                                                      |
|  domain          /svt1                                                         |
|  schedmode       prompted                                                      |
|  schedlogname    /tsm/dsmsched2.log                                            |
|  errorlogname    /tsm/dsmerror2.log                                            |
|  managedservices schedule                                                      |
|                                                                                |
+--------------------------------------------------------------------------------+

Contents of /test/dsm.opt1:

servername tsm1_sched1

Contents of /test/dsm.opt2:

servername tsm1_sched2

Open two shell command windows:

Note: You should enter these commands into a shell script if you intend to have the dsmcad processes started directly from /etc/inittab so that the proper DSM_CONFIG variable can be set prior to launching dsmcad.


Restarting the scheduler process on a remote machine

When managing a large number of Tivoli Storage Manager clients running scheduler processes, it is helpful to be able to start and stop the client service from a remote machine.

You can create a shell script to search for and kill running Tivoli Storage Manager scheduler or CAD processes, and restart the processes. You can use software products, such as Symark Power Broker, to allow Tivoli Storage Manager administrators limited access to UNIX servers to manage the scheduler processes and copy off the Tivoli Storage Manager schedule log file. The following shell script is an example of how to recycle the Tivoli Storage Manager scheduler process:

#!/bin/ksh
# Use the following script to kill the currently running instance of the
# TSM scheduler, and restart the scheduler in nohup mode.
#
# This script will not work properly if more than one scheduler process is
# running.
 
# If necessary, the following variables can be customized to allow an
# alternateoptions file to be used.
# export DSM_DIR=
# export DSM_CONFIG=
# export PATH=$PATH:$DSM_DIR
 
# Extract the PID for the running TSM Scheduler
PID=$(ps -ef | grep "dsmc sched" | grep -v "grep" | awk {'print $2'});
print "Original TSM scheduler process using PID=$PID"
 
# Kill the scheduler
kill -9 $PID
 
# Restart the scheduler with nohup, redirecting all output to NULL
# Output will still be logged in the dsmsched.log
nohup dsmc sched 2>&1 > /dev/null &
 
# Extract the PID for the running TSM Scheduler
PID=$(ps -ef | grep "dsmc sched" | grep -v "grep" | awk {'print $2'});
print "New TSM scheduler process using PID=$PID"
 

Using the scheduler on clustered systems

Using Tivoli Storage Manager client in cluster environment requires additional configuration steps and can be easily done. There are some concepts to keep in mind when configuring Tivoli Storage Manager client in cluster environment:

It is recommended that you separate the backup of shared and local volumes. If the cluster has several shared disk resources which can migrate from one cluster node to another, it is recommended to run a separate instance of the Tivoli Storage Manager client for each disk resource. This allows Tivoli Storage Manager client instances to migrate together with disk resources, in case of failover or failback. See cluster software documentation for information about how to create a program resource. In case of a failover or failback, the disk resources must be up and running before the Tivoli Storage Manager client starts.

The Tivoli Storage Manager client will prompt for the password if it is not readable from a password file. The Tivoli Storage Manager node password must be available for all Tivoli Storage Manager client cluster nodes where the Tivoli Storage Manager client resource can migrate to. For example, if a cluster has three nodes A, B and C and the Tivoli Storage Manager client resource can execute on nodes A and C, passwords must be readable on nodes A and C but not on B. See Changing your password for more information about how to generate Tivoli Storage Manager client passwords.

For information on how to configure a Tivoli Storage Manager server to manage a cluster configured client, see Appendix B, Configuring the backup-archive client in an HACMP takeover environment.


Using the scheduler through a firewall

See Configuring Tivoli Storage Manager client/server communication across a firewall for more information.


Troubleshooting the Tivoli Storage Manager scheduler

The Tivoli Storage Manager server maintains records of scheduled events which can be helpful when managing Tivoli Storage Manager schedules on several client machines. The Tivoli Storage Manager server query event command allows an administrator to view event records on the Tivoli Storage Manager server. A useful query which shows all of the event results for the previous day is:

query event * * begind=today-1 begint=00:00:00 endd=today-1 endt=23:59:59

You can limit query results to exception cases with:

query event * * begind=today-1 begint=00:00:00 endd=today-1 endt=23:59:59
exceptionsonly=yes

Query results include a status field which gives a summary of the result for a specific event. By using the format=detailed option on the query event command, complete information for events displays, including the return code passed back by the Tivoli Storage Manager client. Table 94 summarizes the meaning of the event status codes which are likely to exist for a scheduled event that has already taken place:

Table 94. Scheduled client event status codes

Status Meaning
Completed The scheduled client event ran to completion without a critical failure. There is a possibility that the event completed with some errors or warnings. Query the event with detailed format to inspect the event result for more information. The result can either be 0, 4, or 8.
Missed The schedule start window elapsed without action from the Tivoli Storage Manager client. Common explanations are that the schedule service is not running on the client or a previous scheduled event is not completing for the same or a different schedule.
Started Normally, this indicates that a scheduled event has begun. However, if an event showing a status of Started is followed by one more Missed events, it is possible that the client scheduler encountered a hang while processing that event. One common cause for a hanging client schedule is the occurrence of a user interaction prompt, such as a prompt for an encryption key, to which the user has not responded.
Failed The client event ran to completion, however, a critical failure occurred.

Investigating abnormal scheduled events

If a particular client node displays multiple consecutive events with a result of missed, the client scheduler process is not running, has terminated, or is hanging. See Restarting the scheduler process on a remote machine for tips on how to remotely recycle the hanging scheduler process. If a scheduled event is missed, but other consecutive scheduled events for that node show a result of completed, investigate the server activity log and the client schedule log to determine the cause. Scheduled events with a result of failed, have encountered a processing error originating either from the Tivoli Storage Manager client or server. The first place to check is the server activity log during the time period in which the scheduled event was processed. The activity log shows Tivoli Storage Manager server processing errors and some client errors which have been remotely logged to the Tivoli Storage Manager server. If the explanation cannot be found in the activity log, check the client schedule log.

Checking the server activity log

When checking the server activity log, narrow the query results down to the time period in which the scheduled event occurred. Begin the event log query at a time shortly before the start window of the scheduled event. For example, if you are investigating the following suspect event:

+--------------------------------------------------------------------------------+
|Scheduled Start      Actual Start         Schedule Name Node Name     Status    |
|-------------------- -------------------- ------------- ------------- -------   |
|08/21/2003 08:27:33                       HOURLY        NODEA         Missed    |
|                                                                                |
|                                                                                |
+--------------------------------------------------------------------------------+

You could use one of the following queries:

query actlog begind=08/21/2003 begint=08:25:00
query actlog begind=08/21/2003 begint=08:25:00 originator=client node=nodea
 

Inspecting the Client Schedule Log

The Tivoli Storage Manager client maintains a detailed log of all scheduled activities. If queries of the server activity log do not explain a failed scheduled event, check the Tivoli Storage Manager client schedule log. Access to the client machine is required for inspecting the schedule log. The schedule log file typically resides in the same directory in which the Tivoli Storage Manager client software is installed, in a file named dsmsched.log. The location of the log file can be specified using the schedlogname option, so you may need to refer to the options file to see if the schedlogname option was used to relocate the log file. When you locate the schedule log, it is easy to search through the file to find the time period corresponding the start date and time of the scheduled event. Here are some tips on what to look for:


[ Top of Page | Previous Page | Next Page | Table of Contents | Index ]