Didclient

Classes¶

DIDClient ¶

DIDClient(
    rucio_host=None,
    auth_host=None,
    account=None,
    ca_cert=None,
    auth_type=None,
    creds=None,
    timeout=600,
    user_agent="rucio-clients",
    vo=None,
    logger=LOG,
)

DataIdentifier client class for working with data identifiers

Functions¶

list_dids ¶

list_dids(
    scope,
    filters,
    did_type="collection",
    long=False,
    recursive=False,
)

List all data identifiers in a scope which match a given pattern.

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`filters`	A nested dictionary of key/value pairs like [{'key1': 'value1', 'key2.lte': 'value2'}, {'key3.gte, 'value3'}]. Keypairs in the same dictionary are AND'ed together, dictionaries are OR'ed together. Keys should be suffixed like ., e.g. key1 >= value1 is equivalent to {'key1.gte': value}, where belongs to one of the set {'lte', 'gte', 'gt', 'lt', 'ne' or ''}. Equivalence doesn't require an operator. TYPE: `Sequence[dict[str, Any]]`
`did_type`	The type of the did: 'all'(container, dataset or file)\|'collection'(dataset or container)\|'dataset'\|'container'\|'file' TYPE: `Literal['all', 'collection', 'dataset', 'container', 'file']` DEFAULT: `'collection'`
`long`	Long format option to display more information for each DID. TYPE: `bool` DEFAULT: `False`
`recursive`	Recursively list DIDs content. TYPE: `bool` DEFAULT: `False`

list_dids_extended ¶

list_dids_extended(
    scope,
    filters,
    did_type="collection",
    long=False,
    recursive=False,
)

List all data identifiers in a scope which match a given pattern (DEPRECATED)

add_did ¶

add_did(
    scope,
    name,
    did_type,
    statuses=None,
    meta=None,
    rules=None,
    lifetime=None,
    dids=None,
    rse=None,
)

Add data identifier for a dataset or container.

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The data identifier name. TYPE: `str`
`did_type`	The data identifier type (dataset\|container). TYPE: `Literal['DATASET', 'CONTAINER']`
`statuses`	Dictionary with statuses, e.g. {'monotonic':True}. TYPE: `Optional[Mapping[str, Any]]` DEFAULT: `None`
`meta`	Meta-data associated with the data identifier is represented using key/value pairs in a dictionary. TYPE: `Optional[Mapping[str, Any]]` DEFAULT: `None`
`rules`	Replication rules associated with the data identifier. A list of dictionaries, e.g., [{'copies': 2, 'rse_expression': 'TIERS1'}, ]. TYPE: `Optional[Sequence[Mapping[str, Any]]]` DEFAULT: `None`
`lifetime`	DID's lifetime (in seconds). TYPE: `Optional[int]` DEFAULT: `None`
`dids`	The content. TYPE: `Optional[Sequence[Mapping[str, Any]]]` DEFAULT: `None`
`rse`	The RSE name when registering replicas. TYPE: `Optional[str]` DEFAULT: `None`

add_dids ¶

add_dids(dids)

Bulk add datasets/containers.

add_dataset ¶

add_dataset(
    scope,
    name,
    statuses=None,
    meta=None,
    rules=None,
    lifetime=None,
    files=None,
    rse=None,
)

Add data identifier for a dataset.

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The data identifier name. TYPE: `str`
`statuses`	Dictionary with statuses, e.g.g {'monotonic':True}. TYPE: `Optional[Mapping[str, Any]]` DEFAULT: `None`
`meta`	Meta-data associated with the data identifier is represented using key/value pairs in a dictionary. TYPE: `Optional[Mapping[str, Any]]` DEFAULT: `None`
`rules`	Replication rules associated with the data identifier. A list of dictionaries, e.g., [{'copies': 2, 'rse_expression': 'TIERS1'}, ]. TYPE: `Optional[Sequence[Mapping[str, Any]]]` DEFAULT: `None`
`lifetime`	DID's lifetime (in seconds). TYPE: `Optional[int]` DEFAULT: `None`
`files`	The content. TYPE: `Optional[Sequence[Mapping[str, Any]]]` DEFAULT: `None`
`rse`	The RSE name when registering replicas. TYPE: `Optional[str]` DEFAULT: `None`

add_datasets ¶

add_datasets(dsns)

Bulk add datasets.

PARAMETER	DESCRIPTION
`dsns`	A list of datasets. TYPE: `Iterable[dict[str, Any]]`

add_container ¶

add_container(
    scope,
    name,
    statuses=None,
    meta=None,
    rules=None,
    lifetime=None,
)

Add data identifier for a container.

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The data identifier name. TYPE: `str`
`statuses`	Dictionary with statuses, e.g.g {'monotonic':True}. TYPE: `Optional[Mapping[str, Any]]` DEFAULT: `None`
`meta`	Meta-data associated with the data identifier is represented using key/value pairs in a dictionary. TYPE: `Optional[Mapping[str, Any]]` DEFAULT: `None`
`rules`	Replication rules associated with the data identifier. A list of dictionaries, e.g., [{'copies': 2, 'rse_expression': 'TIERS1'}, ]. TYPE: `Optional[Sequence[Mapping[str, Any]]]` DEFAULT: `None`
`lifetime`	DID's lifetime (in seconds). TYPE: `Optional[int]` DEFAULT: `None`

add_containers ¶

add_containers(cnts)

Bulk add containers.

PARAMETER	DESCRIPTION
`cnts`	A list of containers. TYPE: `Sequence[dict[str, Any]]`

attach_dids ¶

attach_dids(scope, name, dids, rse=None)

Attach data identifier.

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The data identifier name. TYPE: `str`
`dids`	The content. TYPE: `Sequence[Mapping[str, Any]]`
`rse`	The RSE name when registering replicas. TYPE: `Optional[str]` DEFAULT: `None`

detach_dids ¶

detach_dids(scope, name, dids=None)

Detach data identifier

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The data identifier name. TYPE: `str`
`dids`	The content. TYPE: `Optional[Sequence[Mapping[str, Any]]]` DEFAULT: `None`

attach_dids_to_dids ¶

attach_dids_to_dids(attachments, ignore_duplicate=False)

Add dids to dids.

PARAMETER	DESCRIPTION
`attachments`	The attachments. attachments is: [attachment, attachment, ...] attachment is: {'scope': scope, 'name': name, 'dids': dids} dids is: [{'scope': scope, 'name': name}, ...] TYPE: `Sequence[dict[str, Union[str, Sequence[dict[str, Any]]]]]`
`ignore_duplicate`	If True, ignore duplicate entries. TYPE: `bool` DEFAULT: `False`

add_files_to_datasets ¶

add_files_to_datasets(attachments, ignore_duplicate=False)

Add files to datasets.

PARAMETER	DESCRIPTION
`attachments`	The attachments. attachments is: [attachment, attachment, ...] attachment is: {'scope': scope, 'name': name, 'dids': dids} dids is: [{'scope': scope, 'name': name}, ...] TYPE: `Sequence[dict[str, Union[str, Sequence[dict[str, Any]]]]]`
`ignore_duplicate`	If True, ignore duplicate entries. TYPE: `bool` DEFAULT: `False`

add_datasets_to_containers ¶

add_datasets_to_containers(attachments)

Add datasets_to_containers.

PARAMETER	DESCRIPTION
`attachments`	The attachments. attachments is: [attachment, attachment, ...] attachment is: {'scope': scope, 'name': name, 'dids': dids} dids is: [{'scope': scope, 'name': name}, ...] TYPE: `Sequence[dict[str, Union[str, Sequence[dict[str, Any]]]]]`

add_containers_to_containers ¶

add_containers_to_containers(attachments)

Add containers_to_containers.

PARAMETER	DESCRIPTION
`attachments`	The attachments. attachments is: [attachment, attachment, ...] attachment is: {'scope': scope, 'name': name, 'dids': dids} dids is: [{'scope': scope, 'name': name}, ...] TYPE: `Sequence[dict[str, Union[str, Sequence[dict[str, Any]]]]]`

add_files_to_dataset ¶

add_files_to_dataset(scope, name, files, rse=None)

Add files to datasets.

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The dataset name. TYPE: `str`
`files`	The content. TYPE: `Sequence[Mapping[str, Any]]`
`rse`	The RSE name when registering replicas. TYPE: `Optional[str]` DEFAULT: `None`

add_files_to_archive ¶

add_files_to_archive(scope, name, files)

Add files to archive.

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The dataset name. TYPE: `str`
`files`	The content. TYPE: `Sequence[Mapping[str, Any]]`

add_datasets_to_container ¶

add_datasets_to_container(scope, name, dsns)

Add datasets to container.

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The dataset name. TYPE: `str`
`dsns`	The content. TYPE: `Sequence[Mapping[str, Any]]`

add_containers_to_container ¶

add_containers_to_container(scope, name, cnts)

Add containers to container.

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The dataset name. TYPE: `str`
`cnts`	The content. TYPE: `Sequence[Mapping[str, Any]]`

list_content ¶

list_content(scope, name)

List data identifier contents.

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The data identifier name. TYPE: `str`

list_content_history ¶

list_content_history(scope, name)

List data identifier contents history.

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The data identifier name. TYPE: `str`

list_files ¶

list_files(scope, name, long=None)

List data identifier file contents.

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The data identifier name. TYPE: `str`
`long`	A boolean to choose if GUID is returned or not. TYPE: `Optional[bool]` DEFAULT: `None`

bulk_list_files ¶

bulk_list_files(dids)

List data identifier file contents.

PARAMETER	DESCRIPTION
`dids`	The list of DIDs. TYPE: `list[dict[str, Any]]`

get_did ¶

get_did(scope, name, dynamic=False, dynamic_depth=None)

Retrieve a single data identifier.

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The data identifier name. TYPE: `str`
`dynamic_depth`	The DID type as string ('FILE'/'DATASET') at which to stop the dynamic length/bytes calculation. If not set, the size will not be computed dynamically. TYPE: `Optional[str]` DEFAULT: `None`
`dynamic`	(Deprecated) same as dynamic_depth = 'FILE' TYPE: `bool` DEFAULT: `False`

get_metadata ¶

get_metadata(scope, name, plugin='DID_COLUMN')

Get data identifier metadata

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The data identifier name. TYPE: `str`
`plugin`	Backend Metadata plugin the Rucio server should use to query data. TYPE: `str` DEFAULT: `'DID_COLUMN'`

get_metadata_bulk ¶

get_metadata_bulk(dids, inherit=False, plugin='JSON')

Bulk get data identifier metadata

PARAMETER	DESCRIPTION
`dids`	A list of dids. TYPE: `Sequence[Mapping[str, Any]]`
`inherit`	A boolean. If set to true, the metadata of the parent are concatenated. TYPE: `bool` DEFAULT: `False`
`plugin`	The metadata plugin to query, 'ALL' for all available plugins TYPE: `str` DEFAULT: `'JSON'`

set_metadata ¶

set_metadata(scope, name, key, value, recursive=False)

Set data identifier metadata

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The data identifier name. TYPE: `str`
`key`	the key. TYPE: `str`
`value`	the value. TYPE: `Any`
`recursive`	Option to propagate the metadata change to content. TYPE: `bool` DEFAULT: `False`

set_metadata_bulk ¶

set_metadata_bulk(scope, name, meta, recursive=False)

Set data identifier metadata in bulk.

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The data identifier name. TYPE: `str`
`meta`	the metadata key-values. TYPE: `Mapping[str, Any]`
`recursive`	Option to propagate the metadata change to content. TYPE: `bool` DEFAULT: `False`

set_dids_metadata_bulk ¶

set_dids_metadata_bulk(dids, recursive=False)

Set metadata to a list of data identifiers.

PARAMETER	DESCRIPTION
`dids`	A list of dids including metadata, i.e. [{'scope': scope1, 'name': name1, 'meta': {key1: value1, key2: value2}] . TYPE: `Sequence[Mapping[str, Any]]`
`recursive`	Option to propagate the metadata update to content. TYPE: `bool` DEFAULT: `False`

set_status ¶

set_status(scope, name, **kwargs)

Set data identifier status

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The data identifier name. TYPE: `str`
`kwargs`	Keyword arguments of the form status_name=value. DEFAULT: `{}`

close ¶

close(scope, name)

close dataset/container

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The dataset/container name. TYPE: `str`

delete_metadata ¶

delete_metadata(scope, name, key)

Delete data identifier metadata

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The data identifier. TYPE: `str`
`key`	the key. TYPE: `str`

list_did_rules ¶

list_did_rules(scope, name)

List the associated rules of a data identifier.

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The data identifier name. TYPE: `str`

list_associated_rules_for_file ¶

list_associated_rules_for_file(scope, name)

List the associated rules a file is affected from..

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The file name. TYPE: `str`

get_dataset_by_guid ¶

get_dataset_by_guid(guid)

Get the parent datasets for a given GUID.

PARAMETER	DESCRIPTION
`guid`	The GUID. TYPE: `str`

RETURNS	DESCRIPTION
`Iterator[dict[str, Any]]`	A did

scope_list ¶

scope_list(scope, name=None, recursive=False)

List data identifiers in a scope.

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The data identifier name. TYPE: `Optional[str]` DEFAULT: `None`
`recursive`	boolean, True or False. TYPE: `bool` DEFAULT: `False`

list_parent_dids ¶

list_parent_dids(scope, name)

List parent dataset/containers of a did.

PARAMETER	DESCRIPTION
`scope`	The scope. TYPE: `str`
`name`	The name. TYPE: `str`

create_did_sample ¶

create_did_sample(
    input_scope,
    input_name,
    output_scope,
    output_name,
    nbfiles,
)

Create a sample from an input collection.

PARAMETER	DESCRIPTION
`input_scope`	The scope of the input DID. TYPE: `str`
`input_name`	The name of the input DID. TYPE: `str`
`output_scope`	The scope of the output dataset. TYPE: `str`
`output_name`	The name of the output dataset. TYPE: `str`
`account`	The account.
`nbfiles`	The number of files to register in the output dataset. TYPE: `int`

resurrect ¶

resurrect(dids)

Resurrect a list of dids.

PARAMETER	DESCRIPTION
`dids`	A list of dids [{'scope': scope, 'name': name}, ...] TYPE: `Sequence[Mapping[str, Any]]`

list_archive_content ¶

list_archive_content(scope, name)

List archive contents.

PARAMETER	DESCRIPTION
`scope`	The scope name. TYPE: `str`
`name`	The data identifier name. TYPE: `str`