Gfal
Default(protocol_attr, rse_settings, logger=logging.log)
¶
Implementing access to RSEs using the srm protocol.
lfns2pfns(lfns)
¶
Returns a fully qualified PFN for the file referred by path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
The path to the file. |
required |
Returns:
| Type | Description |
|---|---|
|
Fully qualified PFN. |
parse_pfns(pfns)
¶
Splits the given PFN into the parts known by the protocol. During parsing the PFN is also checked for validity on the given RSE with the given protocol.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pfn
|
a fully qualified PFN |
required |
Returns:
| Type | Description |
|---|---|
|
a dict containing all known parts of the PFN for the protocol e.g. scheme, path, filename |
Raises:
| Type | Description |
|---|---|
RSEFileNameNotSupported
|
if the provided PFN doesn't match with the protocol settings |
path2pfn(path)
¶
Returns a fully qualified PFN for the file referred by path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
The path to the file. |
required |
Returns:
| Type | Description |
|---|---|
|
Fully qualified PFN. |
connect()
¶
Establishes the actual connection to the referred RSE. If we decide to use gfal, init should be done here.
get(path, dest, transfer_timeout=None)
¶
Provides access to files stored inside connected the RSE.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
Physical file name of requested file |
required | |
dest
|
Name and path of the files when stored at the client |
required | |
transfer_timeout
|
Transfer timeout (in seconds) |
None
|
Raises:
| Type | Description |
|---|---|
DestinationNotAccessible
|
if the destination storage was not accessible. |
ServiceUnavailable
|
if some generic error occurred in the library. |
SourceNotFound
|
if the source file was not found on the referred storage. |
put(source, target, source_dir, transfer_timeout=None)
¶
Allows to store files inside the referred RSE.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
path to the source file on the client file system |
required | |
target
|
path to the destination file on the storage |
required | |
source_dir
|
Path where the to be transferred files are stored in the local file system |
required | |
transfer_timeout
|
Transfer timeout (in seconds) |
None
|
Raises:
| Type | Description |
|---|---|
DestinationNotAccessible
|
if the destination storage was not accessible. |
ServiceUnavailable
|
if some generic error occurred in the library. |
SourceNotFound
|
if the source file was not found on the referred storage. |
delete(path)
¶
Deletes a file from the connected RSE.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
path to the to be deleted file |
required |
Raises:
| Type | Description |
|---|---|
ServiceUnavailable
|
if some generic error occurred in the library. |
SourceNotFound
|
if the source file was not found on the referred storage. |
rename(path, new_path)
¶
Allows to rename a file stored inside the connected RSE.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
path to the current file on the storage |
required | |
new_path
|
path to the new file on the storage |
required |
Raises:
| Type | Description |
|---|---|
DestinationNotAccessible
|
if the destination storage was not accessible. |
ServiceUnavailable
|
if some generic error occurred in the library. |
SourceNotFound
|
if the source file was not found on the referred storage. |
exists(path)
¶
Checks if the requested file is known by the referred RSE.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
Physical file name |
required |
Returns:
| Type | Description |
|---|---|
|
True if the file exists, False if it doesn't |
Raises:
| Type | Description |
|---|---|
SourceNotFound
|
if the source file was not found on the referred storage. |
close()
¶
Closes the connection to RSE.
stat(path)
¶
Returns the stats of a file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
path to file |
required |
Returns:
| Type | Description |
|---|---|
|
a dict with two keys, filesize and an element of GLOBALLY_SUPPORTED_CHECKSUMS. |
Raises:
| Type | Description |
|---|---|
ServiceUnavailable
|
if some generic error occurred in the library. |
__gfal2_cancel()
¶
Cancel all gfal operations in progress.
__gfal2_copy(src, dest, src_spacetoken=None, dest_spacetoken=None, transfer_timeout=None)
¶
Uses gfal2 to copy file from src to dest.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
src
|
Physical source file name |
required | |
src_spacetoken
|
The source file's space token |
None
|
|
dest
|
Physical destination file name |
required | |
dest_spacetoken
|
The destination file's space token |
None
|
|
transfer_timeout
|
Transfer timeout (in seconds) |
None
|
Returns:
| Type | Description |
|---|---|
|
0 if copied successfully, other than 0 if failed |
Raises:
| Type | Description |
|---|---|
SourceNotFound
|
if source file cannot be found. |
RucioException
|
if it failed to copy the file. |
__gfal2_rm(paths)
¶
Uses gfal2 to remove the file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
Physical file name |
required |
Returns:
| Type | Description |
|---|---|
|
0 if removed successfully, other than 0 if failed |
Raises:
| Type | Description |
|---|---|
SourceNotFound
|
if the source file was not found. |
RucioException
|
if it failed to remove the file. |
__gfal2_exist(path)
¶
Uses gfal2 to check whether the file exists.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
Physical file name |
required |
Returns:
| Type | Description |
|---|---|
|
0 if it exists, -1 if it doesn't |
Raises:
| Type | Description |
|---|---|
RucioException
|
if the error is not source not found. |
__gfal2_rename(path, new_path)
¶
Uses gfal2 to rename a file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
path to the current file on the storage |
required | |
new_path
|
path to the new file on the storage |
required |
Returns:
| Type | Description |
|---|---|
|
0 if it exists, -1 if it doesn't |
Raises:
| Type | Description |
|---|---|
RucioException
|
if failed. |
get_space_usage()
¶
Get RSE space usage information.
Returns:
| Type | Description |
|---|---|
|
a list with dict containing 'totalsize' and 'unusedsize' |
Raises:
| Type | Description |
|---|---|
ServiceUnavailable
|
if some generic error occurred in the library. |
__gfal2_get_space_usage(path, space_token)
¶
Uses gfal2 to get space usage info with space token.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
the endpoint path |
required | |
space_token
|
a string space token. E.g. "ATLASDATADISK" |
required |
Returns:
| Type | Description |
|---|---|
|
a list with dict containing 'totalsize' and 'unusedsize' |
Raises:
| Type | Description |
|---|---|
ServiceUnavailable
|
if failed. |
NoRename(protocol_attr, rse_settings, logger=logging.log)
¶
Implementing access to RSEs using the srm protocol without renaming files on upload/download. Necessary for some storage endpoints.
Initializes the object with information about the referred RSE.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
protocol_attr
|
Properties of the requested protocol. |
required | |
rse_settting
|
The RSE settings. |
required | |
logger
|
Optional decorated logger that can be passed from the calling daemons or servers. |
log
|
rename(pfn, new_pfn)
¶
Allows to rename a file stored inside the connected RSE.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pfn
|
Current physical file name |
required |
CLI(protocol_attr, rse_settings, logger=logging.log)
¶
Implementing access to RSEs using the srm protocol through CLI with 'gfal' commands.
Initializes the object with information about the referred RSE.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
props
|
Properties derived from the RSE Repository |
required |
get(path, dest, transfer_timeout=None)
¶
Provides access to files stored inside connected the RSE.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
Physical file name of requested file |
required | |
dest
|
Name and path of the files when stored at the client |
required | |
transfer_timeout
|
Transfer timeout (in seconds) |
None
|
Raises:
| Type | Description |
|---|---|
RucioException
|
Passthrough of gfal-copy error message. |
put(source, target, source_dir, transfer_timeout=None)
¶
Allows to store files inside the referred RSE.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
path to the source file on the client file system |
required | |
target
|
path to the destination file on the storage |
required | |
source_dir
|
Path where the to be transferred files are stored in the local file system |
required | |
transfer_timeout
|
Transfer timeout (in seconds) |
None
|
Raises:
| Type | Description |
|---|---|
RucioException
|
Passthrough of gfal-copy error message. |