Fts3

Classes¶

Fts3TransferStatusReport ¶

Fts3TransferStatusReport(
    external_host, request_id, request=None
)

FTS3CompletionMessageTransferStatusReport ¶

FTS3CompletionMessageTransferStatusReport(
    external_host, request_id, fts_message
)

Parses FTS Completion messages received via the message queue

FTS3ApiTransferStatusReport ¶

FTS3ApiTransferStatusReport(
    external_host,
    request_id,
    job_response,
    file_response,
    request=None,
)

Parses FTS api response

FTS3Transfertool ¶

FTS3Transfertool(
    external_host,
    oidc_support=False,
    vo=None,
    group_bulk=1,
    group_policy="rule",
    source_strategy=None,
    max_time_in_queue=None,
    bring_online=43200,
    default_lifetime=172800,
    archive_timeout_override=None,
    logger=logging.log,
)

FTS3 implementation of a Rucio transfertool

Initializes the transfertool

PARAMETER	DESCRIPTION
`external_host`	The external host where the transfertool API is running TYPE: `str`

Functions¶

submit ¶

submit(transfers, job_params, timeout=None)

Submit transfers to FTS3 via JSON.

PARAMETER	DESCRIPTION
`files`	List of dictionaries describing the file transfers.
`job_params`	Dictionary containing key/value pairs, for all transfers. TYPE: `dict[str, str]`
`timeout`	Timeout in seconds. TYPE: `Optional[int]` DEFAULT: `None`

RETURNS	DESCRIPTION
`str`	FTS transfer identifier.

cancel ¶

cancel(transfer_ids, timeout=None)

Cancel transfers that have been submitted to FTS3.

PARAMETER	DESCRIPTION
`transfer_ids`	FTS transfer identifiers as list of strings. TYPE: `Sequence[str]`
`timeout`	Timeout in seconds. TYPE: `Optional[int]` DEFAULT: `None`

RETURNS	DESCRIPTION
`dict[str, Any]`	True if cancellation was successful.

update_priority ¶

update_priority(transfer_id, priority, timeout=None)

Update the priority of a transfer that has been submitted to FTS via JSON.

PARAMETER	DESCRIPTION
`transfer_id`	FTS transfer identifier as a string. TYPE: `str`
`priority`	FTS job priority as an integer from 1 to 5. TYPE: `int`
`timeout`	Timeout in seconds. TYPE: `Optional[int]` DEFAULT: `None`

RETURNS	DESCRIPTION
`dict[str, Any]`	True if update was successful.

query ¶

query(transfer_ids, details=False, timeout=None)

Query the status of a transfer in FTS3 via JSON.

PARAMETER	DESCRIPTION
`transfer_ids`	FTS transfer identifiers as list of strings. TYPE: `Sequence[str]`
`details`	Switch if detailed information should be listed. TYPE: `bool` DEFAULT: `False`
`timeout`	Timeout in seconds. TYPE: `Optional[int]` DEFAULT: `None`

RETURNS	DESCRIPTION
`Union[Optional[dict[str, Any]], list[dict[str, Any]]]`	Transfer status information as a list of dictionaries.

whoami ¶

whoami()

Returns credential information from the FTS3 server.

RETURNS	DESCRIPTION
`dict[str, Any]`	Credentials as stored by the FTS3 server as a dictionary.

version ¶

version()

Returns FTS3 server information.

RETURNS	DESCRIPTION
`dict[str, Any]`	FTS3 server information as a dictionary.

bulk_query ¶

bulk_query(requests_by_eid, timeout=None)

Query the status of a bulk of transfers in FTS3 via JSON.

PARAMETER	DESCRIPTION
`requests_by_eid`	dictionary {external_id1: {request_id1: request1, ...}, ...} of request to be queried TYPE: `dict[str, dict[str, dict[str, Any]]]`

RETURNS	DESCRIPTION
`dict[str, Any]`	Transfer status information as a dictionary.

list_se_status ¶

list_se_status()

Get the list of banned Storage Elements.

RETURNS	DESCRIPTION
`dict[str, Any]`	Detailed dictionary of banned Storage Elements.

get_se_config ¶

get_se_config(storage_element)

Get the Json response for the configuration of a storage element.

PARAMETER	DESCRIPTION
`storage_element`	the storage element you want the configuration for. TYPE: `str`

RETURNS	DESCRIPTION
`dict[str, Any]`	a Json result for the configuration of a storage element.

set_se_config ¶

set_se_config(
    storage_element,
    inbound_max_active=None,
    outbound_max_active=None,
    inbound_max_throughput=None,
    outbound_max_throughput=None,
    staging=None,
)

Set the configuration for a storage element. Used for alleviating transfer failures due to timeout.

PARAMETER	DESCRIPTION
`storage_element`	The storage element to be configured TYPE: `str`
`inbound_max_active`	the integer to set the inbound_max_active for the SE. TYPE: `Optional[int]` DEFAULT: `None`
`outbound_max_active`	the integer to set the outbound_max_active for the SE. TYPE: `Optional[int]` DEFAULT: `None`
`inbound_max_throughput`	the float to set the inbound_max_throughput for the SE. TYPE: `Optional[float]` DEFAULT: `None`
`outbound_max_throughput`	the float to set the outbound_max_throughput for the SE. TYPE: `Optional[float]` DEFAULT: `None`
`staging`	the integer to set the staging for the operation of a SE. TYPE: `Optional[int]` DEFAULT: `None`

RETURNS	DESCRIPTION
`dict[str, Any]`	JSON post response in case of success, otherwise raise Exception.

set_se_status ¶

set_se_status(
    storage_element, message, ban=True, timeout=None
)

Ban a Storage Element. Used when a site is in downtime. One can use a timeout in seconds. In that case the jobs will wait before being cancel. If no timeout is specified, the jobs are canceled immediately

PARAMETER	DESCRIPTION
`storage_element`	The Storage Element that will be banned. TYPE: `str`
`message`	The reason of the ban. TYPE: `str`
`ban`	Boolean. If set to True, ban the SE, if set to False unban the SE. TYPE: `bool` DEFAULT: `True`
`timeout`	if None, send to FTS status 'cancel' else 'waiting' + the corresponding timeout. TYPE: `Optional[int]` DEFAULT: `None`

RETURNS	DESCRIPTION
`int`	0 in case of success, otherwise raise Exception

__get_transfer_baseid_voname ¶

__get_transfer_baseid_voname()

Get transfer VO name from the external host.

__get_deterministic_id ¶

__get_deterministic_id(sid)

Get deterministic FTS job id.

PARAMETER	DESCRIPTION
`sid`	FTS seed id. TYPE: `str`

RETURNS	DESCRIPTION
`Optional[str]`	FTS transfer identifier.

__query_details ¶

__query_details(transfer_id)

Query the detailed status of a transfer in FTS3 via JSON.

PARAMETER	DESCRIPTION
`transfer_id`	FTS transfer identifier as a string. TYPE: `str`

RETURNS	DESCRIPTION
`Optional[dict[str, Any]]`	Detailed transfer status information as a dictionary.

Functions¶

build_job_params ¶

build_job_params(
    transfer_path,
    bring_online=None,
    default_lifetime=None,
    archive_timeout_override=None,
    max_time_in_queue=None,
    logger=logging.log,
)

Prepare the job parameters which will be passed to FTS transfertool Please refer to https://fts3-docs.web.cern.ch/fts3-docs/fts-rest/docs/bulk.html#parameters for the list of parameters.

bulk_group_transfers ¶

bulk_group_transfers(
    transfer_paths,
    policy="rule",
    group_bulk=200,
    source_strategy=None,
    max_time_in_queue=None,
    logger=logging.log,
    archive_timeout_override=None,
    bring_online=None,
    default_lifetime=None,
)

Group transfers in bulk based on certain criteria

PARAMETER	DESCRIPTION
`transfer_paths`	List of transfer paths to group. Each path is a list of single-hop transfers. TYPE: `Iterable[list[DirectTransfer]]`
`policy`	Policy to use to group. TYPE: `str` DEFAULT: `'rule'`
`group_bulk`	Bulk sizes. TYPE: `int` DEFAULT: `200`
`source_strategy`	Strategy to group sources TYPE: `Optional[str]` DEFAULT: `None`
`max_time_in_queue`	Maximum time in queue TYPE: `Optional[dict]` DEFAULT: `None`
`archive_timeout_override`	Override the archive_timeout parameter for any transfers with it set (0 to unset) TYPE: `Optional[int]` DEFAULT: `None`
`logger`	Optional decorated logger that can be passed from the calling daemons or servers. TYPE: `LoggerFunction` DEFAULT: `log`

RETURNS	DESCRIPTION
`list[dict[str, Any]]`	List of grouped transfers.