Rsemanager

Functions¶

get_scope_protocol ¶

get_scope_protocol(vo='def')

Returns the callable protocol to translate the pfn to a name/scope pair

RETURNS	DESCRIPTION
`Callable`	Callable: Scope Parser function

get_rse_info ¶

get_rse_info(rse=None, vo='def', rse_id=None, session=None)

Returns all protocol related RSE attributes. Call with either rse and vo, or (in server mode) rse_id

PARAMETER	DESCRIPTION
`rse`	Name of the requested RSE DEFAULT: `None`
`vo`	The VO for the RSE. DEFAULT: `'def'`
`rse_id`	The id of the rse (use in server mode to avoid db calls) DEFAULT: `None`
`session`	The eventual database session. DEFAULT: `None`

RETURNS DESCRIPTION

RSESettingsDict

a dict object with the following attributes: id ... an internal identifier rse ... the name of the RSE as string type ... the storage type odf the RSE e.g. DISK volatile ... boolean indicating if the RSE is volatile verify_checksum ... boolean indicating whether RSE supports requests for checksums deterministic ... boolean indicating of the naming of the files follows the defined determinism domain ... indicating the domain that should be assumed for transfers. Values are 'ALL', 'LAN', or 'WAN' protocols ... all supported protocol in form of a list of dict objects with the following structure - scheme ... protocol scheme e.g. http, srm, ... - hostname ... hostname of the site - prefix ... path to the folder where the files are stored - port ... port used for this protocol - impl ... naming the python class of the protocol implementation - extended_attributes ... additional information for the protocol - domains ... a dict naming each domain and the priority of the protocol for each operation (lower is better, zero is not supported)

RAISES	DESCRIPTION
`RSENotFound`	if the provided RSE could not be found in the database.

_get_possible_protocols ¶

_get_possible_protocols(
    rse_settings,
    operation,
    scheme=None,
    domain=None,
    impl=None,
)

Filter the list of available protocols or provided by the supported ones.

PARAMETER	DESCRIPTION
`rse_settings`	The rse settings. TYPE: `RSESettingsDict`
`operation`	The operation (write, read).
`scheme`	Optional filter if no specific protocol is defined in rse_setting for the provided operation. DEFAULT: `None`
`domain`	Optional domain (lan/wan), if not specified, both will be returned DEFAULT: `None`

RETURNS	DESCRIPTION
	The list of possible protocols.

create_protocol ¶

create_protocol(
    rse_settings,
    operation,
    scheme=None,
    domain="wan",
    auth_token=None,
    protocol_attr=None,
    logger=logging.log,
    impl=None,
)

Instantiates the protocol defined for the given operation.

PARAMETER	DESCRIPTION
`rse_settings`	RSE attributes TYPE: `RSESettingsDict`
`operation`	Intended operation for this protocol
`scheme`	Optional filter if no specific protocol is defined in rse_setting for the provided operation DEFAULT: `None`
`domain`	Optional specification of the domain DEFAULT: `'wan'`
`auth_token`	Optionally passing JSON Web Token (OIDC) string for authentication DEFAULT: `None`
`protocol_attr`	Optionally passing the full protocol availability information to correctly select WAN/LAN DEFAULT: `None`
`logger`	Optional decorated logger that can be passed from the calling daemons or servers. DEFAULT: `log`

RETURNS	DESCRIPTION
	An instance of the requested protocol

lfns2pfns ¶

lfns2pfns(
    rse_settings,
    lfns,
    operation="write",
    scheme=None,
    domain="wan",
    auth_token=None,
    logger=logging.log,
    impl=None,
)

Convert the lfn to a pfn

PARAMETER	DESCRIPTION
`rse_settings`	RSE attributes TYPE: `RSESettingsDict`
`lfns`	logical file names as a dict containing 'scope' and 'name' as keys. For bulk a list of dicts can be provided
`operation`	Intended operation for this protocol DEFAULT: `'write'`
`scheme`	Optional filter if no specific protocol is defined in rse_setting for the provided operation DEFAULT: `None`
`domain`	Optional specification of the domain DEFAULT: `'wan'`
`auth_token`	Optionally passing JSON Web Token (OIDC) string for authentication DEFAULT: `None`
`logger`	Optional decorated logger that can be passed from the calling daemons or servers. DEFAULT: `log`

RETURNS	DESCRIPTION
	a dict with scope:name as key and the PFN as value

parse_pfns ¶

parse_pfns(
    rse_settings,
    pfns,
    operation="read",
    domain="wan",
    auth_token=None,
)

Checks if a PFN is feasible for a given RSE. If so it splits the pfn in its various components.

PARAMETER	DESCRIPTION
`rse_settings`	RSE attributes TYPE: `RSESettingsDict`
`pfns`	list of PFNs
`operation`	Intended operation for this protocol DEFAULT: `'read'`
`domain`	Optional specification of the domain DEFAULT: `'wan'`
`auth_token`	Optionally passing JSON Web Token (OIDC) string for authentication DEFAULT: `None`

RETURNS	DESCRIPTION
	A dict with the parts known by the selected protocol e.g. scheme, hostname, prefix, path, name

RAISES	DESCRIPTION
`RSEFileNameNotSupported`	if provided PFN is not supported by the RSE/protocol
`RSENotFound`	if the referred storage is not found i the repository (rse_id)
`InvalidObject`	If the properties parameter doesn't include scheme, hostname, and port as keys
`RSEOperationNotSupported`	If no matching protocol was found for the requested operation

exists ¶

exists(
    rse_settings,
    files,
    domain="wan",
    scheme=None,
    impl=None,
    auth_token=None,
    vo="def",
    logger=logging.log,
)

Checks if a file is present at the connected storage. Providing a list indicates the bulk mode.

PARAMETER	DESCRIPTION
`rse_settings`	RSE attributes TYPE: `RSESettingsDict`
`files`	a single dict or a list with dicts containing 'scope' and 'name' if LFNs are used and only 'name' if PFNs are used. E.g. {'name': '2_rse_remote_get.raw', 'scope': 'user.jdoe'}, {'name': 'user/jdoe/5a/98/3_rse_remote_get.raw'}
`domain`	The network domain, either 'wan' (default) or 'lan' DEFAULT: `'wan'`
`auth_token`	Optionally passing JSON Web Token (OIDC) string for authentication DEFAULT: `None`
`vo`	The VO for the RSE DEFAULT: `'def'`
`logger`	Optional decorated logger that can be passed from the calling daemons or servers. DEFAULT: `log`

RETURNS	DESCRIPTION
	True/False for a single file or a dict object with 'scope:name' for LFNs or 'name' for PFNs as keys and True or the exception as value for each file in bulk mode

RAISES	DESCRIPTION
`RSENotConnected`	no connection to a specific storage has been established

upload ¶

upload(
    rse_settings,
    lfns,
    domain="wan",
    source_dir=None,
    force_pfn=None,
    force_scheme=None,
    transfer_timeout=None,
    delete_existing=False,
    sign_service=None,
    auth_token=None,
    vo="def",
    logger=logging.log,
    impl=None,
)

Uploads a file to the connected storage. Providing a list indicates the bulk mode.

PARAMETER	DESCRIPTION
`rse_settings`	RSE attributes TYPE: `RSESettingsDict`
`lfns`	a single dict or a list with dicts containing 'scope' and 'name'. Examples: [ {'name': '1_rse_local_put.raw', 'scope': 'user.jdoe', 'filesize': 42, 'adler32': '87HS3J968JSNWID'}, {'name': '2_rse_local_put.raw', 'scope': 'user.jdoe', 'filesize': 4711, 'adler32': 'RSSMICETHMISBA837464F'} ] If the 'filename' key is present, it will be used by Rucio as the actual name of the file on disk (separate from the Rucio 'name').
`domain`	The network domain, either 'wan' (default) or 'lan' DEFAULT: `'wan'`
`source_dir`	path to the local directory including the source files DEFAULT: `None`
`force_pfn`	use the given PFN -- can lead to dark data, use sparingly DEFAULT: `None`
`force_scheme`	use the given protocol scheme, overriding the protocol priority in the RSE description DEFAULT: `None`
`transfer_timeout`	set this timeout (in seconds) for the transfers, for protocols that support it DEFAULT: `None`
`sign_service`	use the given service (e.g. gcs, s3, swift) to sign the URL DEFAULT: `None`
`auth_token`	Optionally passing JSON Web Token (OIDC) string for authentication DEFAULT: `None`
`vo`	The VO for the RSE DEFAULT: `'def'`
`logger`	Optional decorated logger that can be passed from the calling daemons or servers. DEFAULT: `log`

RETURNS	DESCRIPTION
	True/False for a single file or a dict object with 'scope:name' as keys and True or the exception as value for each file in bulk mode

RAISES	DESCRIPTION
`RSENotConnected`	no connection to a specific storage has been established
`SourceNotFound`	local source file can not be found
`DestinationNotAccessible`	remote destination directory is not accessible
`ServiceUnavailable`	for any other reason

delete ¶

delete(
    rse_settings,
    lfns,
    domain="wan",
    auth_token=None,
    logger=logging.log,
    impl=None,
)

Delete a file from the connected storage. Providing a list indicates the bulk mode.

PARAMETER	DESCRIPTION
`rse_settings`	RSE attributes TYPE: `RSESettingsDict`
`lfns`	a single dict or a list with dicts containing 'scope' and 'name'. E.g. [{'name': '1_rse_remote_delete.raw', 'scope': 'user.jdoe'}, {'name': '2_rse_remote_delete.raw', 'scope': 'user.jdoe'}]
`domain`	The network domain, either 'wan' (default) or 'lan' DEFAULT: `'wan'`
`auth_token`	Optionally passing JSON Web Token (OIDC) string for authentication DEFAULT: `None`
`logger`	Optional decorated logger that can be passed from the calling daemons or servers. DEFAULT: `log`

RETURNS	DESCRIPTION
	True/False for a single file or a dict object with 'scope:name' as keys and True or the exception as value for each file in bulk mode

RAISES	DESCRIPTION
`RSENotConnected`	no connection to a specific storage has been established
`SourceNotFound`	remote source file can not be found on storage
`ServiceUnavailable`	for any other reason

rename ¶

rename(
    rse_settings,
    files,
    domain="wan",
    auth_token=None,
    logger=logging.log,
    impl=None,
)

Rename files stored on the connected storage. Providing a list indicates the bulk mode.

PARAMETER	DESCRIPTION
`rse_settings`	RSE attributes TYPE: `RSESettingsDict`
`files`	a single dict or a list with dicts containing 'scope', 'name', 'new_scope' and 'new_name' if LFNs are used or only 'name' and 'new_name' if PFNs are used. If 'new_scope' or 'new_name' are not provided, the current one is used. Examples: [ {'name': '3_rse_remote_rename.raw', 'scope': 'user.jdoe', 'new_name': '3_rse_new.raw', 'new_scope': 'user.jdoe'}, {'name': 'user/jdoe/d9/cb/9_rse_remote_rename.raw', 'new_name': 'user/jdoe/c6/4a/9_rse_new.raw'} ]
`domain`	The network domain, either 'wan' (default) or 'lan' DEFAULT: `'wan'`
`auth_token`	Optionally passing JSON Web Token (OIDC) string for authentication DEFAULT: `None`
`logger`	Optional decorated logger that can be passed from the calling daemons or servers. DEFAULT: `log`

RETURNS	DESCRIPTION
	True/False for a single file or a dict object with LFN (key) and True/False (value) in bulk mode

RAISES	DESCRIPTION
`RSENotConnected`	no connection to a specific storage has been established
`SourceNotFound`	remote source file can not be found on storage
`DestinationNotAccessible`	remote destination directory is not accessible
`ServiceUnavailable`	for any other reason

get_space_usage ¶

get_space_usage(
    rse_settings,
    scheme=None,
    domain="wan",
    auth_token=None,
    logger=logging.log,
    impl=None,
)

Get RSE space usage information.

PARAMETER	DESCRIPTION
`rse_settings`	RSE attributes TYPE: `RSESettingsDict`
`scheme`	optional filter to select which protocol to be used. DEFAULT: `None`
`domain`	The network domain, either 'wan' (default) or 'lan' DEFAULT: `'wan'`
`auth_token`	Optionally passing JSON Web Token (OIDC) string for authentication DEFAULT: `None`
`logger`	Optional decorated logger that can be passed from the calling daemons or servers. DEFAULT: `log`

RETURNS	DESCRIPTION
	a list with dict containing 'totalsize' and 'unusedsize'

RAISES	DESCRIPTION
`ServiceUnavailable`	if some generic error occurred in the library.

find_matching_scheme ¶

find_matching_scheme(
    rse_settings_dest,
    rse_settings_src,
    operation_src,
    operation_dest,
    domain="wan",
    scheme=None,
)

Find the best matching scheme between two RSEs

PARAMETER	DESCRIPTION
`rse_settings_dest`	RSE settings for the destination RSE.
`rse_settings_src`	RSE settings for the src RSE.
`operation_src`	Source Operation such as read, write.
`operation_dest`	Dest Operation such as read, write.
`domain`	Domain such as lan, wan. DEFAULT: `'wan'`
`scheme`	List of supported schemes. DEFAULT: `None`

RETURNS	DESCRIPTION
	Tuple of matching schemes (dest_scheme, src_scheme, dest_scheme_priority, src_scheme_priority).

_retry_protocol_stat ¶

_retry_protocol_stat(protocol, pfn)

try to stat file, on fail try again 1s, 2s, 4s, 8s, 16s, 32s later. Fail is all fail

PARAMETER	DESCRIPTION
`protocol`	The protocol to use to reach this file
`pfn`	Physical file name of the target for the protocol stat

__check_compatible_scheme ¶

__check_compatible_scheme(dest_scheme, src_scheme)

Check if two schemes are compatible, such as srm and gsiftp

PARAMETER	DESCRIPTION
`dest_scheme`	Destination scheme
`src_scheme`	Source scheme
`scheme`	List of supported schemes

RETURNS	DESCRIPTION
	True if schemes are compatible, False otherwise.