API Reference

dCacheFileSystem

class dcachefs.dCacheFileSystem(*args, **kwargs)[source]

Bases: AsyncFileSystem

File system interface for a dCache storage instance.

Inspired by the fsspec HTTPFileSystem implementation, specific methods interacts with the dCache system either via its API or via the WebDAV protocol.

Parameters:

  • api_url – (str, optional) dCache API URL

  • webdav_url – (str, optional) WebDAV door URL

  • username – (str, optional) username for basic authentication

  • password – (str, optional) password for basic authentication

  • token – (str, optional) token for bearer-token authentication

  • client_kwargs – (dict, optional) keyword arguments passed on to aiohttp.ClientSession, see https://docs.aiohttp.org/en/stable/client_reference.html . For example, {‘auth’: aiohttp.BasicAuth(‘user’, ‘pass’)}

  • request_kwargs – (dict, optional) keyword arguments passed on to the request method of aiohttp.ClientSession (also see client_kwargs)

  • block_size – (int, optional) when creating a file-like object, this is the buffer size (in bytes) for reading and writing. when reading, this is also the size downloaded in one request. if 0, will default to raw requests file-like objects

  • asynchronous – (bool, optional) use in asynchronous mode

  • loop – (optional) if asynchronous, event loop where to run coroutines

  • batch_size – (int, optional) if asynchronous, number of coroutines to submit/wait on simultaneously

  • encoded – use encoded strings when formatting URLs

  • storage_options – (dict, optional) keyword arguments passed on to the super-class

property api_url

dCache API URL.

static close_session(loop, session)[source]

Close the client session.

created(path)[source]

Date and time in which the path was created.

Parameters:

path – (str) target path

Returns:

(datetime.datetime) time of creation

encode_url(url)[source]

Build URL, optionally encoded.

info(path, **kwargs)

Give details about a file or a directory.

Parameters:

  • path – (str) target path

  • kwargs – (dict, optional) arguments passed on to requests

Returns:

(dict) path metadata

ls(path, detail=True, limit=None, **kwargs)

List path content.

Parameters:

  • path – (str) target path (file or directory)

  • detail – (bool, optional) if True, return a list of dictionaries with the (children) path(s) info. If False, return a list of paths

  • limit – (int, optional) set the maximum number of children paths returned to this value

  • kwargs – (dict, optional) arguments passed on to requests

Returns:

(list) if detail is True, list of dictionaries. List of strings otherwise

modified(path)[source]

Date and time in which the path was last modified.

Parameters:

path – (str) target path

Returns:

(datetime.datetime) time of last modification

mv(path1, path2, **kwargs)

Rename path1 to path2.

Parameters:

  • path1 – (str) source path

  • path2 – (str) destination path

  • kwargs – (dict, optional) arguments passed on to requests

open(path, mode='rb', **kwargs)[source]

Return a file-like object from the filesystem.

Parameters:

  • path – (str) target file path

  • mode – (str, optional) choose between “r”, “rb”, “w”, and “wb”

  • kwargs – (dict, optional) keyword arguments passed on to the super-class

Returns:

(dCacheFile or dCacheStreamFile) file-like object

rm(path, recursive=False, **kwargs)

Remove file or directory tree.

Parameters:

  • path – (str) target path

  • recursive – (bool, optional) if True, and the target path is a directory, remove all subdirectories and their files

  • kwargs – (dict, optional) arguments passed on to requests

async set_session()[source]

Set the client session, compatibly with both async/sync execution.

property webdav_url

WebDAV door URL.

dCacheFile

class dcachefs.dcachefs.dCacheFile(fs, url, mode='rb', block_size=None, request_kwargs=None, asynchronous=False, session=None, loop=None, **kwargs)[source]

Bases: HTTPFile

A file-like object pointing to a target file on dCache.

Supports reading, with read-ahead of a pre-determined block-size, and writing, with the file content being first cached and then uploaded to upon file closure.

Parameters:

  • fs – (dCacheFileSystem) file-system instance creating the file

  • url – (str) target file path

  • mode – (str, optional) choose between “r”, “rb”, “w”, and “wb”

  • block_size – (int, optional) The amount of read-ahead to do, in bytes. Default is 5MB, or the value configured for the FileSystem creating this file

  • request_kwargs – (dict, optional) arguments passed on to requests

  • asynchronous – (bool, optional) use in asynchronous mode

  • session – (aiohttp.ClientSession, optional) All calls will be made within this session, to avoid restarting connections

  • loop – (optional) if asynchronous, event loop where to run coroutines

  • kwargs – (dict, optional) arguments passed on to the super-class

close()[source]

Close file. Finalize writes, discard cache.

flush(force=False)[source]

Write buffered data to remote file.

Since byte-range writing is not supported, the file content is only written when force is True (i.e. when the file-like object is closed).

Parameters:

force – (bool, optional) Force writing of the remote file. Disallows further writing to this file.

write_chunked()

Write buffered data to remote file.

dCacheStreamFile

class dcachefs.dcachefs.dCacheStreamFile(fs, url, mode='rb', request_kwargs=None, asynchronous=False, session=None, loop=None, **kwargs)[source]

Bases: HTTPStreamFile

A streaming file-like object pointing to a target file on dCache.

Supports reading and writing by opening request streams to the remote file.

Parameters:

  • fs – (dCacheFileSystem) file-system instance creating the file

  • url – (str) target file path

  • mode – (str, optional) choose between “r”, “rb”, “w”, and “wb”

  • request_kwargs – (dict, optional) arguments passed on to requests

  • asynchronous – (bool, optional) use in asynchronous mode

  • session – (aiohttp.ClientSession, optional) All calls will be made within this session, to avoid restarting connections

  • loop – (optional) if asynchronous, event loop where to run coroutines

  • kwargs – (dict, optional) arguments passed on to the super-class

read(num=-1)[source]

Read bytes from file.

Parameters:

num – (int, optional) Read up this many bytes. If negative, read all content to end of file.

Returns:

bytes read from the target file

write(data)[source]

Write data to remote file.

Can be called only once, consecutive calls will overwrite the file.

Parameters:

data – dict, list of tuples, bytes or file-like object to write