Torrent Suite SoftwareData ManagementGuide

---

Torrent Suite Software space on Ion Community

Data Management Guide TOC

Data Management Overview

Purpose

The goal of the Torrent Browser Data Management system is to give you control over both your analysis data and your Torrent Server disk space.

You use Data Management for both automatic and on-demand control over your files and disk space usage. With this feature, you configure the behavior of the Export, Archive,and Delete options for both specific run reports and for projects.

You can set different data age and disk space thresholds for different categories of analysis files (such as raw data files, debugging files, and output files) as well as different archive locations.

You can set archive and deletion rules to happen automatically. On demand, you can also archive or deleted a specific run (in the Data tab > Data Management tab) or archive, delete, or export one or more runs in a project (in a Data > Projects > project_name page).

You use the Data Management system to balance the rate of incoming new data with your needs to preserve existing results and to maintain control over your disk partitions in an orderly manner.

Important points

Email notifications

Even if you do not enable automatic data management features at this time, please visit the Torrent Browser Data Management Configuration page and complete the following steps:

• Enter the email address to receive disk-full notifications.
• Enable the Data Management system (required to send email notifications)

( Access the configuration page from the Data tab > Data Management tab Data Management section Configure link.)

New installations or upgrades

When your Torrent Server is first installed or is upgraded from a release earlier than 3.6 , you have to configure your data management settings because of system changes. Your previous Keep settings remain in place after the upgrade, but automatic actions are not enabled.

See Settings After an Upgrade from 3.4 or Earlier Version for instructions.

Data management summary

There are two modes of operation:

• Automatic You can configure rules for the automatic archive or deletion of analysis files based on data age, disk space usage, and file category.
• Manual You can archive, delete, or export all or part of a run on the Data tab > Data Management tab Disk Space Management section. You can archive, delete, or export one or more runs in a project from the project page (Data > Projects > project_name ).

There are three supported data management actions:

• Delete Removes files from the Torrent Server data directories.
• Archive Moves files from the Torrent Server to an external drive.
• Export Copies files from the Torrent Server to an external drive.

There are four file categories of files for each analysis. Each category can be handled separately. (To act on an entire run, select all four file categories.)

• Signal Processing Input Files that are required to reanalyze the run from Signal Processing.
• Basecalling Input Files that are required to reanalyze the run from Basecalling.
• Output Files Files that are required to view the run report in the Torrent Browser, including report-rendering and customer deliverable files.
• Intermediate Files Files that are required to debug the run.

On-demand features overview

This section shows how you invoke on-demand data management features. See Data Management Configuration for information on the automatic archive and deletion features.

Archive and export are not supported until an archive directory is configured for the type of analysis file ( file category) involved.

On-demand usage opens a popup called the Manual Data Management Menu. In this menu, you select which type of files to archive, delete, or export:

The destination directory for archive and export is determined by your data management configuration. (Each file category can have a different destination directory. "Local" in the State column means the files are undisturbed on the Torrent Server and have not been archived or deleted.)

Archive, delete, or export a specific run

In the Data tab > Data Management tab Data Management section, the Disk Usage section lists runs by file system partition. The gear menu Actions option opens the Manual Data Management Menu. In this popup (see above ), you delete, archive, or export one or more of the file categories for a specific run.

Archive, delete, or export one or more members of a project

In a project page (Data > Projects > project_name ), you can select one or more members of the project, then click Process Selected... and Data Management . The Manual Data Management Menu opens. In this popup (see above ), you delete, archive, or export one or more of the file categories for the selected run or runs.

Import

In the Data tab > Data Management tab, you can import one or more file categories for a specific archived or exported run. These files must have been archived or exported by the Data Management utility in release 4.0 or higher.

Check the Data Management settings for a run

To find out what the current settings are for a specific run, go to the Data > Data Management tab, scroll down to the Data Management section and then to the Disk Space Management section. Each analysis is listed (by file system partition), with the settings for its file categories.

See Check Data Management Settings for an Analysis for more information.

Prevent a run's files from being deleted or archived

In the Data tab > Data Management tab Disk Space Management section, y ou can mark important analyses so that their files are protected from both automatic or on-demand deletion or archival.

See Protect Analysis Files From Being Deleted or Archived .

Monitor disk space

The Data tab > Data Management tab Disk Space Management section displays graphs of how much archive space is used and free (in blue) for both the Torrent Server file systems and the archive directories. See Monitor Free Disk Space (in the Administration Guide).

Disk full banner

When any storage device reaches 95% full (and again at 99%), a warning banner is displayed across the top of Torrent Browser pages:

*** CRITICAL! /results/: Partition is getting very full - 95% ***

Acknowledge deletion checkboxes

A single a cknowledge deletion checkbox (on the Data Management Configuration page) is used when automatic deletion is configured. If you use this checkbox to enable automatic deletion of Signal Processing Input files, you do not need to acknowledge deletion of every run.

If automatic deletion is not enabled, when the disk usage threshold is exceeded the system identifies candidate runs to be deleted. Each of these deletion candidate runs has its own acknowledge deletion checkbox in the Data tab > Data Management tab Disk Space Management section.

Configuration

Configuration is required for the following:

• To enable automatic archive and deletion
• To enable on-demand archive and export
• To configure the email address to receive disk-full notifications
  
  

Before you configure Torrent Browser Data Management features, only the following is supported:

• On-demand deletion
  
  

During configuration, you do the following:

• Specify the archive media for each type of file category.
• You set data age and disk space thresholds that are used for automatic archive and deletion.
• Provide the email address of the administrator or distribution list to receive disk-full notifications.
• Enable the automatic Data Management system.
• Optionally enable automatic deletion of the Signal Processing Input file category.

See Data Management Configuration .

Handling of run reports and Customer Support Archive s

The Output file category includes analysis results files and files required both to display the analysis run report and to launch plugins on the analysis.

When you archive or delete an Output file category through the Data Management utility, the system first creates a PDF of the run report and a Customer Support Archive (CSA) of the run. Both of these files are left in the original analysis directory on the file system, after the file category is archived or deleted.

After the file category is archived or deleted, the Torrent Browser cannot display the run report for the analysis. If someone later tries to access the run report, the Torrent Browser instead displays a message similar to the following :

The "Restore Report files from archived location" option allows you to restore the report.

Run report notifications

The Data Management system places notices of completed actions in the run report for the affected dataset.

Here is an example:

The View Report Log link opens a log of data management actions on the run result set.

Logs

Data Management logs for a specific run are available on the following Torrent Browser screens, with the gear menu View Log option:

• The Data tab > Data Management tab Disk Usage section
• The Data tab Completed Runs and Reports list view

The following logs in the file system contain also data management information:

• /var/log/ion/data_management.log
• /var/log/ion/celery_periodic.log

---