The utilities.srblib module

SRB ("Storage Resource Broker") is a distributed filesystem, developed by the San Diego Supercomputer Center, and intended for "Grid" applications, where geographically-separated groups need to share large quantities of data.

SRB comes with a client Python library, but this is very low level, and the API is not very Pythonic. This module aims to provide a more useable interface to SRB, allowing you to treat an SRB system more like a local filesystem.

Various alternative high-level SRB Python interfaces are available: srboo, SRBpy and pysrb. These mostly offer more features than srblib, but I didn't succeed in getting any of them up-and-running, so here is another one.

Installing the low-level srb module

There are some good build instructions for the low-level bindings at: http://www.vislab.uq.edu.au/research/accessgrid/virtual_fs/building.htm

If building on OS X, I found this useful: https://lists.sdsc.edu/pipermail/srb-chat/2007-November/004506.html

Make sure you create a directory called .srb in your home directory and, in this directory, a file called .MdasEnv, which should look something like this:

mdasCollectionHome '/home/srbdev.sdsc'
mdasDomainHome 'sdsc'
srbUser 'srbdev'
srbHost 'torah.sdsc.edu'
defaultResource 'unix-sdsc'
AUTH_SCHEME 'ENCRYPT1'

Check your configuration works using the "Scommands" before trying the Python interface. You will find full details on the SRB website.

Browsing an SRB filesystem

If you have your .MdasEnv file setup correctly, you can connect to an SRB server using:

>>> from NeuroTools.utilities.srblib import *
>>> S = SRBFileSystem('facets.inria.fr')

(It is also possible to override the contents of .MdasEnv, or even to have no such file at all, by passing the connection parameters (username, password, etc) as keyword arguments to the constructor.) You can now browse the server using method calls similar to the Unix command-line, e.g.:

>>> S.ls()
...
>>> S.cd("/INRIA/home/INRIA/WP5")
>>> S.ls()
>>> S.get("srbtestfile", "/tmp/localfile")
>>> S.put("/tmp/localfile", "srbtestfile_copy")

You can also open files and read to/write from them more-or-less as if they were on your local filesystem:

>>> f = S.open("srbtestfile2", "w")
>>> f.write("Mate, this bird wouldn't 'voom' if you put four million volts through it!\n")
>>> f.close()
>>> f = S.open("srbtestfile2", "r")
>>> print f.read()
Mate, this bird wouldn't 'voom' if you put four million volts through it!
>>> f.close()

Accessing files by URL

This part of the interface was inspired by urllib from the Python standard library. There is no need to create an SRBServer object, just pass an "srb://" URL to the urlopen or urlretrieve functions:

>>> f = urlopen("srb://facets.inria.fr/INRIA/home/INRIA/WP5/srbtestfile2")
>>> print f.read()
Mate, this bird wouldn't 'voom' if you put four million volts through it!
>>> f.close()