8: Hierarchical Storage Management

This chapter provides an overview of the features of the optional Backup HSM (Hierarchical Storage Management) module and explains how automatic migration works. The HSM module is available for Solaris only.


Use HSM to automatically move data between your local disk and other storage locations. Migration is the process of moving files from a client to the migration store, which is a remote migration storage device; recall is the process of moving files from the remote storage device back to the original location on the client. The purpose of HSM is to more effectively manage a network's storage resources by keeping newer data available for fast access without compromising the availability of older or less frequently accessed data. Except for a relatively longer access time for migrated files, the entire migration and recall process is transparent to the user.

The Backup HSM module moves data from a client's local disk to another storage medium automatically, based on a set of policies specified by an administrator. The data management strategy relies on the administrator's definition of a high water mark, which defines the threshold condition that determines when automatic migration begins and a low water mark, which defines the threshold condition that determines when automatic migration stops. Migration continues until all eligible files are migrated or until the low water mark is reached.

How HSM Relates to Other Backup Features

Hierarchical storage management (HSM) is complimentary to backup, archiving, and save set staging. All Backup features store data on media; however, each one has a specific purpose.

Backup is the process of storing copies of data on media. The original file is left in place. Backup is done on a regular schedule on the entire file system to ensure that data can be recovered if the original file is lost or damaged, a disk crashes, or in event of a disaster.

Archive is the process of storing data that you no longer need to access regularly. Archive and retrieval are usually manual procedures. Users whose archive privileges have been enabled can initiate an archive at any time, usually at the end of a project. The media used for archiving must be safe and durable. After files are archived, they are usually removed from the local file system, thus freeing up disk space.

Save set staging is the process of moving data from one storage medium to another. Currently, you can only employ save set staging from the command line. You can move data that has been backed up, archived, or migrated using save set staging. A staged file does not leave a corresponding stub. Unlike migration, you can use staging to move save sets between any storage media, not just from the local disk to remote storage.

Hierarchical Storage Management (HSM) is a data management strategy where data is automatically migrated from a client's local disk to remote storage, based on a set of rules. The rule often employed is access time - the longer a file is inactive, the more likely it is to migrate. The file is removed from the local disk, leaving a stub (pointer to the new location). When you access the file, it is automatically returned to the original location. The migration and recall process is transparent to the user.

The principal goals of these features differ:

Hierarchical Storage Management (Data Migration) Operations

After you enable the optional HSM module on your Backup server, you can manually migrate files and set up automatic migration.Use the Migration resource to specify one or more Backup clients as HSM clients. For each HSM client, set the policy that defines when files are migrated in the high water mark and low water mark attributes and define the criteria for Backup uses to select files for migration, based on the following attributes:

    For example, on Solaris systems, the following files are automatically excluded from migration:

Files that meet the criteria you specify in the Migration resource are available for pre-migration and later, migration. For step-by-step instructions and definitions of the attributes in Backup resources, see the online help. The automatic migration process is described in the following section.

Caution -

You must enable and register your HSM module within 45 days, or the functions will be automatically disabled.

What Happens During Automatic Migration

The Backup server controls the migration process through the policies specified by the administrator.

    1. When the Backup server conducts a regularly scheduled backup, it checks each client in the backup group for files that are candidates for migration.

    Files that meet the migration criteria for their client resource are pre-migrated. During pre-migration, the file is copied to a Backup storage location (a migration volume), but the file remains on the client machine.

    2. When the local disk on the Backup HSM client reaches the defined high water mark, the Backup server initiates the actual migration of the candidate files from the client machine to free up local disk space.

    First, Backup verifies that pre-migrated files have not changed since the most recent migration sweep. If a file has changed between the pre-migration and the initiation of actual migration, it is no longer a migration candidate. If the file has not changed, Backup deletes the pre-migrated candidate files from the client's local disk, and replaces it with a stub, which contains information about the file and a symbolic link to the physical location of the file.

    Normal backups of the client data from this point forward will only back up the stub (symbolic link) that is left on the client machine.

    To provide additional backup protection for migrated files, perform a super-full backup. A super-full backup clones the most recent full backup of a save set an a clone of all the migration save sets, so it contains the data on the client and the data in the migration store. To perform a super-full backup, become root on the Backup server, then enter the following command from the shell prompt:

# nsrclone -c client-name -N save-set-name ---------------------------------------------------

    When you recover a file system that contains migrated data, Backup recovers only the stub to the local disk. To return the file to the client's local disk, open the file on the client machine. Backup automatically recalls the file from the migration store. If the media containing your migrated file is not currently mounted, Backup notifies the administrator.

    When browsing the client file index, users see the migrated files as symbolic links.

    Migrated data is managed by the Backup server and is subject to all of the usual storage management features such as pools, cloning, and auto media verification.

    Migrated data can be written only to storage volumes associated with a pool of type "migration." Clones of "migration" volumes can be written only to storage volumes from a pool of type "migration clone." Migrated data is exempt from the Backup server's automatic data recycling policies.

    3. When a user or application attempts to read a migrated file, the stub automatically triggers an HSM recall.

    During a recall, Backup replaces the stub on the client machine with the original file. If the local hard disk has insufficient free space to demigrate the file, Backup issues the appropriate notifications. Backup provides preconfigured HSM notifications. For more information on using notifications, see "Preconfigured Notifications" on page 47.

Manual Migration

You can pre-migrate and migrate files manually from the command line. Use manual migration when the file system is full or nearly full, for example, after you receive a Migration Attention notification. First, you pre-migrate the files using the nsrpmig command, then you migrate the files using the nsrmig command. Migrating large files provides the most benefit, because it frees the most local disk space.

To pre-migrate a file manually, enter the following command:

# nsrpmig -s server-name -b pool -g group path --------------------------------------------------------

To migrate a file manually, enter the following command:

# nsrmig -s server-name path ----------------------------------

If you do not specify a path, the current directory is used. Migration continues until the file system capacity reaches the low water mark specified in the Migration resource.

Refer to the nsrpmig(1M) and nsrmig(1M) man pages and see "Troubleshooting" on page 241 for more details about these two commands.

How Does Backup Deal With Renamed or Deleted Files?

The nsrexecd daemon automatically checks and corrects the consistency of HSM-managed file systems every night at 2:00 a.m.

You can also check the consistency of HSM-managed file systems manually. The nsrhsmck command checks and corrects the consistency of HSM managed file systems and deals with cases where the stubs of migrated files are renamed or deleted from local disk. The basic syntax of the nsrhsmck command follows:

# nsrhsmck -cdfv -s server-name path -----------------------------------------

You must specify a path on the command line when you run nsrhsmck. Only files and index entries that fall under the path specified are examined for consistency.

If you rename the stub file on the client machine, Backup cannot recall the file because the stub with the original filename no longer exists. Before you can recall the file, you must use the nsrhsmck -f command to update the HSM file index to reflect the new name.

If you delete the stub file from the client machine, you can recover the stub then recall the actual file.

If you want to delete a migrated file, delete the stub from local disk, then use the nsrhsmck -d command to mark the index entry for the migrated file as possibly deleted. After an index entry that is marked as a possible deletion has passed the 60 day expiration time, use the nsrhsmck -c command to remove the expired entries from the HSM file index.

Before an entry is deleted from the HSM file index, Backup checks to make sure the file does not exist on disk. If a file marked as possibly deleted is detected on disk before the index entry is later deleted, the index entry will be unmarked as a possible deletion.

Migration Monitoring

The Migration Control resource in the Backup administration program displays a list of clients configured for HSM services and statistics for all migration activities that occurred in the last seven days.

You can produce reports on HSM activities using command line instructions. Refer to the man pages for details on using these commands:

To customize the Migration Completion notification, modify the resource configured for the notification. By default, a migration completion notice is sent by email to root any time a migration event occurs.

Migration Volumes

Migration volumes are the media that hold migrated data. You can either use the default Migration pool of volumes to store migrated data or create your own custom pool of volumes to use as the migration store. You can also automatically clone the volume to which migrated data is sent. Cloned data must be sent to a pool of type Clone. Backup provides a default clone pool for migrated data, called Migration Clone.

Caution -

Migration data is in a different format than regular Backup save set data, therefore, it must be written to different volumes. Because of these differences, the client file indexes and bootstrap save set created during a premigration or migration operation is also not written to the same volume as the migrated save sets. By default, they are written to a volume from the Default pool. If you need to direct the client file indexes and bootstrap to a volume pool other than Default, see "Example: Directing Client Indexes and Bootstrap to a Separate Pool" on page 62 for information.

Copyright 1997 Sun Microsystems, Inc., 901 San Antonio Road, Palo Alto, California 94303 USA. All Rights Reserved.