Rsemanager

Classes¶

Functions¶

get_scope_protocol ¶

get_scope_protocol(vo='def')

Returns the callable protocol to translate the pfn to a name/scope pair

RETURNS	DESCRIPTION
`Callable`	Callable: Scope Parser function

get_rse_info ¶

get_rse_info(rse=None, vo='def', rse_id=None, session=None)

Returns all protocol related RSE attributes. Call with either rse and vo, or (in server mode) rse_id

PARAMETER	DESCRIPTION
`rse`	Name of the requested RSE TYPE: `Optional[str]` DEFAULT: `None`
`vo`	The VO for the RSE. TYPE: `str` DEFAULT: `'def'`
`rse_id`	The id of the rse (use in server mode to avoid db calls) TYPE: `Optional[str]` DEFAULT: `None`
`session`	The eventual database session. TYPE: `Optional[Session]` DEFAULT: `None`

RETURNS DESCRIPTION

RSESettingsDict

a dict object with the following attributes: id ... an internal identifier rse ... the name of the RSE as string type ... the storage type odf the RSE e.g. DISK volatile ... boolean indicating if the RSE is volatile verify_checksum ... boolean indicating whether RSE supports requests for checksums deterministic ... boolean indicating of the naming of the files follows the defined determinism domain ... indicating the domain that should be assumed for transfers. Values are 'ALL', 'LAN', or 'WAN' protocols ... all supported protocol in form of a list of dict objects with the following structure - scheme ... protocol scheme e.g. http, srm, ... - hostname ... hostname of the site - prefix ... path to the folder where the files are stored - port ... port used for this protocol - impl ... naming the python class of the protocol implementation - extended_attributes ... additional information for the protocol - domains ... a dict naming each domain and the priority of the protocol for each operation (lower is better, zero is not supported)

RAISES	DESCRIPTION
`RSENotFound`	if the provided RSE could not be found in the database.

create_protocol ¶

create_protocol(
    rse_settings,
    operation,
    scheme=None,
    domain="wan",
    auth_token=None,
    protocol_attr=None,
    logger=logging.log,
    impl=None,
)

Instantiates the protocol defined for the given operation.

PARAMETER	DESCRIPTION
`rse_settings`	RSE attributes TYPE: `RSESettingsDict`
`operation`	Intended operation for this protocol TYPE: `str`
`scheme`	Optional filter if no specific protocol is defined in rse_setting for the provided operation TYPE: `Optional[str]` DEFAULT: `None`
`domain`	Optional specification of the domain TYPE: `str` DEFAULT: `'wan'`
`auth_token`	Optionally passing JSON Web Token (OIDC) string for authentication TYPE: `Optional[str]` DEFAULT: `None`
`protocol_attr`	Optionally passing the full protocol availability information to correctly select WAN/LAN TYPE: `Optional[RSEProtocolDict]` DEFAULT: `None`
`logger`	Optional decorated logger that can be passed from the calling daemons or servers. TYPE: `LoggerFunction` DEFAULT: `log`

RETURNS	DESCRIPTION
`RSEProtocol`	An instance of the requested protocol

lfns2pfns ¶

lfns2pfns(
    rse_settings,
    lfns,
    operation="write",
    scheme=None,
    domain="wan",
    auth_token=None,
    logger=logging.log,
    impl=None,
)

Convert the lfn to a pfn

PARAMETER	DESCRIPTION
`rse_settings`	RSE attributes TYPE: `RSESettingsDict`
`lfns`	logical file names as a dict containing 'scope' and 'name' as keys. For bulk a list of dicts can be provided TYPE: `Union[list[LFNDict], LFNDict]`
`operation`	Intended operation for this protocol TYPE: `str` DEFAULT: `'write'`
`scheme`	Optional filter if no specific protocol is defined in rse_setting for the provided operation TYPE: `Optional[str]` DEFAULT: `None`
`domain`	Optional specification of the domain TYPE: `str` DEFAULT: `'wan'`
`auth_token`	Optionally passing JSON Web Token (OIDC) string for authentication TYPE: `Optional[str]` DEFAULT: `None`
`logger`	Optional decorated logger that can be passed from the calling daemons or servers. TYPE: `LoggerFunction` DEFAULT: `log`

RETURNS	DESCRIPTION
`dict[str, str]`	a dict with scope:name as key and the PFN as value

parse_pfns ¶

parse_pfns(
    rse_settings,
    pfns,
    operation="read",
    domain="wan",
    auth_token=None,
)

Checks if a PFN is feasible for a given RSE. If so it splits the pfn in its various components.

PARAMETER	DESCRIPTION
`rse_settings`	RSE attributes TYPE: `RSESettingsDict`
`pfns`	list of PFNs TYPE: `list[str]`
`operation`	Intended operation for this protocol TYPE: `str` DEFAULT: `'read'`
`domain`	Optional specification of the domain TYPE: `str` DEFAULT: `'wan'`
`auth_token`	Optionally passing JSON Web Token (OIDC) string for authentication TYPE: `Optional[str]` DEFAULT: `None`

RETURNS	DESCRIPTION
`dict[str, dict[str, str]]`	A dict with the parts known by the selected protocol e.g. scheme, hostname, prefix, path, name

RAISES	DESCRIPTION
`RSEFileNameNotSupported`	if provided PFN is not supported by the RSE/protocol
`RSENotFound`	if the referred storage is not found i the repository (rse_id)
`InvalidObject`	If the properties parameter doesn't include scheme, hostname, and port as keys
`RSEOperationNotSupported`	If no matching protocol was found for the requested operation

exists ¶

exists(
    rse_settings,
    files,
    domain="wan",
    scheme=None,
    impl=None,
    auth_token=None,
    vo="def",
    logger=logging.log,
)

Checks if a file is present at the connected storage. Providing a list indicates the bulk mode.

PARAMETER	DESCRIPTION
`rse_settings`	RSE attributes TYPE: `RSESettingsDict`
`files`	a single dict or a list with dicts containing 'scope' and 'name' if LFNs are used and only 'name' if PFNs are used. E.g. {'name': '2_rse_remote_get.raw', 'scope': 'user.jdoe'}, {'name': 'user/jdoe/5a/98/3_rse_remote_get.raw'} TYPE: `Union[list[dict[str, str]], dict[str, str]]`
`domain`	The network domain, either 'wan' (default) or 'lan' TYPE: `str` DEFAULT: `'wan'`
`auth_token`	Optionally passing JSON Web Token (OIDC) string for authentication TYPE: `Optional[str]` DEFAULT: `None`
`vo`	The VO for the RSE TYPE: `str` DEFAULT: `'def'`
`logger`	Optional decorated logger that can be passed from the calling daemons or servers. TYPE: `LoggerFunction` DEFAULT: `log`

RETURNS	DESCRIPTION
`Union[bool, list[Union[bool, dict[dict[str, str], bool]]]]`	True/False for a single file or a dict object with 'scope:name' for LFNs or 'name' for PFNs as keys and True or the exception as value for each file in bulk mode

RAISES	DESCRIPTION
`RSENotConnected`	no connection to a specific storage has been established

upload ¶

upload(
    rse_settings,
    lfns,
    domain="wan",
    source_dir=None,
    force_pfn=None,
    force_scheme=None,
    transfer_timeout=None,
    delete_existing=False,
    sign_service=None,
    auth_token=None,
    vo="def",
    logger=logging.log,
    impl=None,
)

Uploads a file to the connected storage. Providing a list indicates the bulk mode.

PARAMETER	DESCRIPTION
`rse_settings`	RSE attributes TYPE: `RSESettingsDict`
`lfns`	a single dict or a list with dicts containing 'scope' and 'name'. Examples: [ {'name': '1_rse_local_put.raw', 'scope': 'user.jdoe', 'filesize': 42, 'adler32': '87HS3J968JSNWID'}, {'name': '2_rse_local_put.raw', 'scope': 'user.jdoe', 'filesize': 4711, 'adler32': 'RSSMICETHMISBA837464F'} ] If the 'filename' key is present, it will be used by Rucio as the actual name of the file on disk (separate from the Rucio 'name'). TYPE: `Union[list[LFNDict], LFNDict]`
`domain`	The network domain, either 'wan' (default) or 'lan' TYPE: `str` DEFAULT: `'wan'`
`source_dir`	path to the local directory including the source files TYPE: `Optional[str]` DEFAULT: `None`
`force_pfn`	use the given PFN -- can lead to dark data, use sparingly TYPE: `Optional[str]` DEFAULT: `None`
`force_scheme`	use the given protocol scheme, overriding the protocol priority in the RSE description TYPE: `Optional[str]` DEFAULT: `None`
`transfer_timeout`	set this timeout (in seconds) for the transfers, for protocols that support it TYPE: `Optional[int]` DEFAULT: `None`
`sign_service`	use the given service (e.g. gcs, s3, swift) to sign the URL TYPE: `Optional[str]` DEFAULT: `None`
`auth_token`	Optionally passing JSON Web Token (OIDC) string for authentication TYPE: `Optional[str]` DEFAULT: `None`
`vo`	The VO for the RSE TYPE: `str` DEFAULT: `'def'`
`logger`	Optional decorated logger that can be passed from the calling daemons or servers. TYPE: `LoggerFunction` DEFAULT: `log`

RETURNS	DESCRIPTION
`dict[Union[int, str], Union[bool, str, dict[str, Union[Literal[True], Exception]]]]`	True/False for a single file or a dict object with 'scope:name' as keys and True or the exception as value for each file in bulk mode

RAISES	DESCRIPTION
`RSENotConnected`	no connection to a specific storage has been established
`SourceNotFound`	local source file can not be found
`DestinationNotAccessible`	remote destination directory is not accessible
`ServiceUnavailable`	for any other reason

delete ¶

delete(
    rse_settings,
    lfns,
    domain="wan",
    auth_token=None,
    logger=logging.log,
    impl=None,
)

Delete a file from the connected storage. Providing a list indicates the bulk mode.

PARAMETER	DESCRIPTION
`rse_settings`	RSE attributes TYPE: `RSESettingsDict`
`lfns`	a single dict or a list with dicts containing 'scope' and 'name'. E.g. [{'name': '1_rse_remote_delete.raw', 'scope': 'user.jdoe'}, {'name': '2_rse_remote_delete.raw', 'scope': 'user.jdoe'}] TYPE: `Union[list[LFNDict], LFNDict]`
`domain`	The network domain, either 'wan' (default) or 'lan' TYPE: `str` DEFAULT: `'wan'`
`auth_token`	Optionally passing JSON Web Token (OIDC) string for authentication TYPE: `Optional[str]` DEFAULT: `None`
`logger`	Optional decorated logger that can be passed from the calling daemons or servers. TYPE: `LoggerFunction` DEFAULT: `log`

RETURNS	DESCRIPTION
`Union[bool, list[Union[bool, dict[str, Union[Literal[True], Exception]]]]]`	True/False for a single file or a dict object with 'scope:name' as keys and True or the exception as value for each file in bulk mode

RAISES	DESCRIPTION
`RSENotConnected`	no connection to a specific storage has been established
`SourceNotFound`	remote source file can not be found on storage
`ServiceUnavailable`	for any other reason

rename ¶

rename(
    rse_settings,
    files,
    domain="wan",
    auth_token=None,
    logger=logging.log,
    impl=None,
)

Rename files stored on the connected storage. Providing a list indicates the bulk mode.

PARAMETER	DESCRIPTION
`rse_settings`	RSE attributes TYPE: `RSESettingsDict`
`files`	a single dict or a list with dicts containing 'scope', 'name', 'new_scope' and 'new_name' if LFNs are used or only 'name' and 'new_name' if PFNs are used. If 'new_scope' or 'new_name' are not provided, the current one is used. Examples: [ {'name': '3_rse_remote_rename.raw', 'scope': 'user.jdoe', 'new_name': '3_rse_new.raw', 'new_scope': 'user.jdoe'}, {'name': 'user/jdoe/d9/cb/9_rse_remote_rename.raw', 'new_name': 'user/jdoe/c6/4a/9_rse_new.raw'} ] TYPE: `Union[list[dict[str, str]], dict[str, str]]`
`domain`	The network domain, either 'wan' (default) or 'lan' TYPE: `str` DEFAULT: `'wan'`
`auth_token`	Optionally passing JSON Web Token (OIDC) string for authentication TYPE: `Optional[str]` DEFAULT: `None`
`logger`	Optional decorated logger that can be passed from the calling daemons or servers. TYPE: `LoggerFunction` DEFAULT: `log`

RETURNS	DESCRIPTION
`Union[bool, list[Union[bool, dict[str, Union[Literal[True], Exception]]]]]`	True/False for a single file or a dict object with LFN (key) and True/False (value) in bulk mode

RAISES	DESCRIPTION
`RSENotConnected`	no connection to a specific storage has been established
`SourceNotFound`	remote source file can not be found on storage
`DestinationNotAccessible`	remote destination directory is not accessible
`ServiceUnavailable`	for any other reason

get_space_usage ¶

get_space_usage(
    rse_settings,
    scheme=None,
    domain="wan",
    auth_token=None,
    logger=logging.log,
    impl=None,
)

Get RSE space usage information.

PARAMETER	DESCRIPTION
`rse_settings`	RSE attributes TYPE: `RSESettingsDict`
`scheme`	optional filter to select which protocol to be used. TYPE: `Optional[str]` DEFAULT: `None`
`domain`	The network domain, either 'wan' (default) or 'lan' TYPE: `str` DEFAULT: `'wan'`
`auth_token`	Optionally passing JSON Web Token (OIDC) string for authentication TYPE: `Optional[str]` DEFAULT: `None`
`logger`	Optional decorated logger that can be passed from the calling daemons or servers. TYPE: `LoggerFunction` DEFAULT: `log`

RETURNS	DESCRIPTION
`list[Union[bool, Union[dict[str, int], Exception]]]`	a list with dict containing 'totalsize' and 'unusedsize'

RAISES	DESCRIPTION
`ServiceUnavailable`	if some generic error occurred in the library.

find_matching_scheme ¶

find_matching_scheme(
    rse_settings_dest,
    rse_settings_src,
    operation_src,
    operation_dest,
    domain="wan",
    scheme=None,
)

Find the best matching scheme between two RSEs

PARAMETER	DESCRIPTION
`rse_settings_dest`	RSE settings for the destination RSE. TYPE: `RSESettingsDict`
`rse_settings_src`	RSE settings for the src RSE. TYPE: `RSESettingsDict`
`operation_src`	Source Operation such as read, write. TYPE: `str`
`operation_dest`	Dest Operation such as read, write. TYPE: `str`
`domain`	Domain such as lan, wan. TYPE: `str` DEFAULT: `'wan'`
`scheme`	List of supported schemes. TYPE: `Optional[Union[str, list[str]]]` DEFAULT: `None`

RETURNS	DESCRIPTION
`tuple[str, str, int, int]`	Tuple of matching schemes (dest_scheme, src_scheme, dest_scheme_priority, src_scheme_priority).

__check_compatible_scheme ¶

__check_compatible_scheme(dest_scheme, src_scheme)

Check if two schemes are compatible, such as srm and gsiftp

PARAMETER	DESCRIPTION
`dest_scheme`	Destination scheme TYPE: `str`
`src_scheme`	Source scheme TYPE: `str`
`scheme`	List of supported schemes

RETURNS	DESCRIPTION
`bool`	True if schemes are compatible, False otherwise.