Configure A Tool to Determine Rules For FTS3 Tape Transfers

Rucio includes functionality to pass instructions to FTS3 that describe how the transfer should be concluded (e.g., placement on tape). This information is passed forward to the storage endpoint as a json of archive_metadata, as described by FTS3.

The plugin ensures the metadata conforms to json encoding and size constraints before adding it to the parameters used in the transfer.

Writing the plugin

Plugins are expected to be subclasses of rucio.transfertool.fts3_plugins.FTS3TapeMetadataPlugin, and registered as fts3_tape_metadata_plugins algorithms. This is done by using the FTS3TapeMetadataPlugin.register method, which tells the algorithm which code to execute during runtime.

The return type of the selected policy code must be a json-encodable dictionary. The only required keys are the keys that will be used by the storage endpoint, not any decorators for FTS3.

During the transfer, the algorithm is passed the existing job parameters generated for the transfer in the form of a dictionary.

from rucio.transfertool.fts3_plugins import FTS3TapeMetadataPlugin

class ExperimentFTSPlugins(FTS3TapeMetadataPlugin):
    def __init__(self, policy_algorithm="def"):
        self.register("policy_algorithm", func=self.plugin_algorithm)  # Name and function for the new algorithm
        super().__init__(policy_algorithm)

    def plugin_algorithm(self, *hints):  # Code executed at runtime
        return {"storage_location": "this_location"}

This will result in the below transfer.

{"files": [{
    "sources": [...],
    ...
    "archive_metadata": {
        "storage_location": "this_location"
        }
    }]
}

Plugins with initialization rules

If the plugin requires set-up that would slow down transfers, using a plugin initialization is recommended. This includes things like querying the configuration file, performing a calculation that would not change between different transfers, or hard-coding parameters.

This is done by including a init_func when registering the plugin.

from rucio.transfertool.fts3_plugins import FTS3TapeMetadataPlugin

class ExperimentFTSPlugins(FTS3TapeMetadataPlugin):
    def __init__(self, policy_algorithm="def"):
        self.register(
            "policy_algorithm",
            func=self.plugin_algorithm,
            init_func=self.plugin_initialization)
        super().__init__(policy_algorithm)

    def plugin_algorithm(self, *hints):
        # Can use `self.extra_params`
        return {
            "storage_location": self.extra_params[hints["name"]]
            }

    def plugin_initialization(self):
        self.extra_params = dict(get_config("transfers", "extra_params"))

To trigger registration it is recommended to run the plugin in the class file with the default plugin name.

class ExperimentFTSPlugins(FTS3TapeMetadataPlugin):
    ...
ExperimentFTSPlugins("def")

Configuration

Configuration set in the rucio.cfg. To use a plugin (here named policy_algorithm_1), modify the config to include the below field. To use multiple plugins, their names can be listed to make each plugin algorithm run in sequence.

[transfers]
fts3tape_metadata_plugins = policy_algorithm_1, policy_algorithm_2

Size constraints can be set with the below. (Default of 4096)

[transfers]
metadata_byte_limit = <byte limit of transfer metadata>

Pre-built plugins

Activity Based Transfer Priority

To assign archive priority based on activity, we include the activity_hints plugin. This plugin assigns an integer priority between 0 and 100 based on the activity of the transfer in question.

Output of the plugin follows the form of

{"archive_metadata": {"scheduling_hints": {"priority": 0}}}

The plugin requires no code modification or subclassing, all set up is done in the rucio.cfg.

To set up, update your configuration as follows:

[transfers]
fts3tape_metadata_plugins = policy_algorithm

[tape_priority]
<Activity 1> = 100
<Activity 2> = 80
<Activity 3> = 50

Baseline priority for any activity not listed is 20.

Tape Collocation

In order to ensure data is placed on tape with similar data, a tape_collocation plugin can be used. This plugin requires custom logic, but behaves as a generic plugin does.

The collocation can be set to up to 4 different levels, such that the passed archive_metadata contains the following:

{
    "collocation_hints":{
        "0": Highest level of grouping,
        "1": Second Highest level,
        ...
        "3": Lowest level of grouping
    }
}

Writing the plugin is similar to any other plugin, it just requires that these 4 levels be filled for a given transfer.

If the collocation wrapper is used, this format is verified and put into the collocation_hints field of the transfer parameters. This is done below, where find_level_hints is an arbitrary function written for an experiment's needs:

from rucio.transfertool.fts3_plugins import FTS3TapeMetadataPlugin

class ExperimentCollocationFTSPlugins(FTS3TapeMetadataPlugin):
    def __init__(self, policy_algorithm="def"):
        super().__init__(policy_algorithm)
        self.register(
            "policy_collocation_algorithm",
            func= lambda x: self._collocation(self._experiment_plugin, x)
        )

    def find_level_hints(self, level, hints):
        ...

    def _experiment_plugin(self, *hints):
        return {
                "0": self.find_level_hints(level=0, hints=hints),
                "1": self.find_level_hints(level=1, hints=hints),
                "2": self.find_level_hints(level=2, hints=hints),
                "3": self.find_level_hints(level=3, hints=hints),
            }

ExperimentCollocationFTSPlugins("def")

Writing the plugin​

Plugins with initialization rules​

Configuration​

Pre-built plugins​

Activity Based Transfer Priority​

Tape Collocation​