Uploadclient

UploadClient(_client=None, logger=None, tracing=True)

Initialises the basic settings for an UploadClient object

Parameters:

Name	Type	Description	Default
_client	Optional[Client]	Optional: rucio.client.client.Client object. If None, a new object will be created.	None
logger	Optional[LoggerFunction]	Optional: logging.Logger object. If None, default logger will be used.	None

upload(items, summary_file_path=None, traces_copy_out=None, ignore_availability=False, activity=None)

Parameters:

Name	Type	Description	Default
items	Iterable[FileToUploadDict]	List of dictionaries. Each dictionary describing a file to upload. Keys: path - path of the file that will be uploaded rse - rse expression/name (e.g. 'CERN-PROD_DATADISK') where to upload the file did_scope - Optional: custom did scope (Default: user.) did_name - Optional: custom did name (Default: name of the file) dataset_scope - Optional: custom dataset scope dataset_name - Optional: custom dataset name dataset_meta - Optional: custom metadata for dataset impl - Optional: name of the protocol implementation to be used to upload this item. force_scheme - Optional: force a specific scheme (if PFN upload this will be overwritten) (Default: None) pfn - Optional: use a given PFN (this sets no_register to True, and no_register becomes mandatory) no_register - Optional: if True, the file will not be registered in the rucio catalogue register_after_upload - Optional: if True, the file will be registered after successful upload lifetime - Optional: the lifetime of the file after it was uploaded transfer_timeout - Optional: time after the upload will be aborted guid - Optional: guid of the file recursive - Optional: if set, parses the folder structure recursively into collections	required
summary_file_path	Optional[str]	Optional: a path where a summary in form of a json file will be stored	None
traces_copy_out	Optional[list[TraceBaseDict]]	reference to an external list, where the traces should be uploaded	None
ignore_availability	bool	ignore the availability of a RSE	False
activity	Optional[str]	the activity set to the rule if no dataset is specified	None

Returns:

Type	Description
int	0 on success

Raises:

Type	Description
InputValidationError	if any input arguments are in a wrong format
RSEWriteBlocked	if a given RSE is not available for writing
NoFilesUploaded	if no files were successfully uploaded
NotAllFilesUploaded	if not all files were successfully uploaded

_register_file(file, registered_dataset_dids, ignore_availability=False, activity=None)

Registers the given file in Rucio. Creates a dataset if needed. Registers the file DID and creates the replication rule if needed. Adds a replica to the file did. (This function is meant to be used as class internal only)

Parameters:

Name	Type	Description	Default
file	Mapping[str, Any]	dictionary describing the file	required
registered_dataset_dids	set[str]	set of dataset dids that were already registered	required
ignore_availability	bool	ignore the availability of a RSE	False
activity	Optional[str]	the activity set to the rule if no dataset is specified	None

Raises:

Type	Description
DataIdentifierAlreadyExists	if file DID is already registered and the checksums do not match

_get_file_guid(file)

Get the guid of a file, trying different strategies (This function is meant to be used as class internal only)

Parameters:

Name	Type	Description	Default
file	Mapping[str, Any]	dictionary describing the file	required

Returns:

Type	Description
str	the guid

_collect_file_info(filepath, item)

Collects infos (e.g. size, checksums, etc.) about the file and returns them as a dictionary (This function is meant to be used as class internal only)

Parameters:

Name	Type	Description	Default
filepath	PathTypeAlias	path where the file is stored	required
item	FileToUploadDict	input options for the given file	required

Returns:

Type	Description
FileToUploadWithCollectedInfoDict	a dictionary containing all collected info and the input options

_collect_and_validate_file_info(items)

Checks if there are any inconsistencies within the given input options and stores the output of _collect_file_info for every file (This function is meant to be used as class internal only)

Parameters:

Name	Type	Description	Default
filepath		list of dictionaries with all input files and options	required

Returns:

Type	Description
list[FileToUploadWithCollectedInfoDict]	a list of dictionaries containing all descriptions of the files to upload

Raises:

Type	Description
InputValidationError	if an input option has a wrong format

_convert_file_for_api(file)

Creates a new dictionary that contains only the values that are needed for the upload with the correct keys (This function is meant to be used as class internal only)

Parameters:

Name	Type	Description	Default
file	Mapping[str, Any]	dictionary describing a file to upload	required

Returns:

Type	Description
dict[str, Any]	dictionary containing not more then the needed values for the upload

_upload_item(rse_settings, rse_attributes, lfn, source_dir=None, domain='wan', impl=None, force_pfn=None, force_scheme=None, transfer_timeout=None, delete_existing=False, sign_service=None)

Uploads a file to the connected storage.

Parameters:

Name	Type	Description	Default
rse_settings	RSESettingsDict	dictionary containing the RSE settings	required
rse_attributes	dict[str, Any]	dictionary containing the RSE attribute key value pairs	required
lfn	dict[str, Union[str, int]]	a single dict containing 'scope' and 'name'. Example: {'name': '1_rse_local_put.raw', 'scope': 'user.jdoe', 'filesize': 42, 'adler32': '87HS3J968JSNWID'} If the 'filename' key is present, it will be used by Rucio as the actual name of the file on disk (separate from the Rucio 'name').	required
source_dir	Optional[str]	path to the local directory including the source files	None
force_pfn	Optional[str]	use the given PFN -- can lead to dark data, use sparingly	None
force_scheme	Optional[str]	use the given protocol scheme, overriding the protocol priority in the RSE description	None
transfer_timeout	Optional[int]	set this timeout (in seconds) for the transfers, for protocols that support it	None
sign_service	Optional[str]	use the given service (e.g. gcs, s3, swift) to sign the URL	None

Raises:

Type	Description
RucioException(msg)	general exception with msg for more details.

_retry_protocol_stat(protocol, pfn)

Try to stat file, on fail try again 1s, 2s, 4s, 8s, 16s, 32s later. Fail is all fail

Parameters:

Name	Type	Description	Default
protocol	RSEProtocol	The protocol to use to reach this file	required
pfn	str	Physical file name of the target for the protocol stat	required

_create_protocol(rse_settings, operation, impl=None, force_scheme=None, domain='wan')

Protocol construction.

Parameters:

Name	Type	Description	Default
rse_settings	RSESettingsDict	rse_settings	required
operation	str	activity, e.g. read, write, delete etc.	required
force_scheme	Optional[str]	custom scheme	None
auth_token		Optionally passing JSON Web Token (OIDC) string for authentication	required

_send_trace(trace)

Checks if sending trace is allowed and send the trace.

Parameters:

Name	Type	Description	Default
trace	TraceDict	the trace	required

_recursive(item)

If the --recursive flag is set, it replicates the folder structure recursively into collections A folder only can have either other folders inside or files, but not both of them - If it has folders, the root folder will be a container - If it has files, the root folder will be a dataset - If it is empty, it does not create anything

Parameters:

Name	Type	Description	Default
item	FileToUploadDict	dictionary containing all descriptions of the files to upload	required

preferred_impl(rse_settings, domain)

Finds the optimum protocol impl preferred by the client and supported by the remote RSE.

Parameters:

Name	Type	Description	Default
rse_settings	RSESettingsDict	dictionary containing the RSE settings	required
domain	str	The network domain, either 'wan' (default) or 'lan'	required

Raises:

Type	Description
RucioException(msg)	general exception with msg for more details.