Page Comparison

This is the specification of the command set for Synapse clients.  The goal is to align the command sets for clients in different languages to ease users' transitions between languages.  Additionally we define the organization of the file cache so that various clients arrange local copies of file consistently.

Note:  This effort to create common command sets between R and Python has been subsumed by the creation of the "synapser" R package which wraps the Python client, automatically generating matching commands.  Nevertheless this document serves as a valuable reference for the core methods and especially the design of the client file cache mechanism.

Synapse File Management

...

The analytical clients provide for client-side caching of Synapse files, to avoid unnecessarily retrieving (large or many) files that are already available locally.  When a file is uploaded or downloaded the client keeps track of the location along with information to determine if it is later changed.  Specifically, the client maintains a "Cache Map" whose keys are local file locations and whose values are UTC last-modified-on times.  The use of this map is as follows: 

Cache Location

When a file is downloaded, specifying the file location is optional.  By default, the file is placed in a cache folder along with a Cache Map file.  

...

Note: the file separator is "/" regardless of the platform (that means 'native' path strings on Windows must be cross-platform normalized before being written into a cache-map file.)

File locking

...

Problem: Clients may concurrently access files on the same file cache; Only 1 client may access the cache map at a time to prevent multiple processes from overwriting or mangling data.

A client must "lock" a .cacheMap file before accessing it.  Opening a file with write access does not lock it on all platforms, therefore we use the following convention, which all clients must follow: 

• To lock a file <filename> a client must successfully create an (empty) folder in the file's parent folder and named <filename>.lock.  (Folder creation is atomic and exclusive across platforms.  This lock folder acts as a mutex object.)  
• If the client is successful, it has exclusive access for ten (10) seconds from the time the lock folder is created.  
• When the creation time stamp is older than ten seconds, any client may delete the lock folder and the client which has created the folder is obligated to refrain from accessing <filename> unless it locks the file again.  The time limit prevents stale locks from blocking other clients.  
• If the client cannot obtain a lock within seventy (70) seconds, it should throw an appropriate exception.
• The client creating the lock should delete the lock folder when it's done accessing <filename>.

...

File Usage Examples

In each example, we have a project in which the File will reside:

...

# we use 'synapseStore=F' to indicate that we only wish to link
file <- File(path="file:///corporatenfs/sharedproject/genotypedata.csv", synapseStore=F, name="genotypedata", `Id=pid)
# Synapse stores the metadata, but does not upload the file
file <- synStore(file)


# Now assume a new session, perhaps by a different user
# synGet downloads the file to a default location
fileId<-"synXXXXX"
# we use 'downloadFile=F' to indicate that we do not need a new copy on our local disk
file <-synGet(fileId, downloadFile=F)
getFileLocation(file)
> "/corporatenfs/sharedproject/genotypedata.csv"
# now change the meta data and save
synAnnot(file, "data type")<-"SNP"
# since the File was created with "synapseStore=F", synStore does not upload the file
file<-synStore(file)

Examples Provenance

Provenance can be handled either implicitly by creating it when something is saved/created in Synapse or explicitly by modifying the provenance record of something already in Synapse.  For most use cases implicitly creating the provenance record should be enough.

...

myFile <- synStore(File("/path/to/file", parentId="syn123"), used=list('syn445865', 'syn1446185'))

#Lets store some code that was executed to generate another file and reference this code as being executed

...

myCode <- synStore(File("/path/to/script.R", parentId="syn123"))
myFile <- synStore(File("/path/to/file", parentId="syn123"), used=list('syn445865', 'syn1446185'), executed=myCode)

...

#To specify the name of the activity that was performed you can specify the name and description of the activity

...

myFile <- synStore(File("/path/to/file", parentId="syn123"), used=list('syn445865', 'syn1446185'), 
                   activityName="Manual editing of file", 
                   activityDescription="Corrected spelling of variable names")

...

Example 2: Creating a provenance record referencing things files stored on the web

...

The construction of a Table consists of creating column models by performing a POST on /column which returns a ColumnModel and then creating the entity by posting a TableEntity to /entity/.  The Table entity should contain a list of column ids obtainable from the list of columnModels. 

#Create a local representation of a table
table synapseclient.Table(name=foobar, parent=syn123, columns=[{columnType:int, name:'age'}, {columnType:string, name:'gender', enumValues=['m', 'f']}])
table = syn.store(table)

...

syn.describeTable(id|Table)  that outputs the information about the column models. Also perhaps passing a details=true would return information about the size of the table by running a query "select count" and information about the values in columns by doing "select unique(c)" for each  each string column c and perhaps "select min(c), max(c)" for numeric columns. 

Command Set

We conceptual divide the client commands into three levels (1) Common functions, (2) Advanced functions and (3) low-level Web API functions.  The first collection of commands captures the majority of functionality of interest to users. The second collection rounds out the functionality with less frequently used functions.  The third set comprises simple, low level wrappers around the Synapse web service interface.  By including this third set users can access web services in advance of having specialized commands in the analytic clients.

*The endpoint defaults to repoEndpoint and it would be useful to be able to pass arbitrary named arguments that are just passed on to the underlying http library.  For python for example the stream and file parameters could be useful to pass along the the filehandle requests for get and put. 

...

The Web APIs are defined here:

Synapse REST APIs

Common Configuration File

...

• Username, password, session token, or API key

• File cache location (should be private to the user)

• Endpoints for each of the Synapse services

...

[authentication] username = example@user.com password = samplePassword sessionToken = 1234567890asdfghjkl apikey = Some+API+key+retrieved+from+either+the+web+portal+or+via+a+REST+GET+call+to+/secretKey==   [cache] location = ~/.synapseCache   [endpoints] repoEndpoint = https://repo-prod.prod.sagebase.org/repo/v1 authEndpoint = https://auth-prod.prod.sagebase.org/auth/v1 fileHandleEndpoint = https://file-prod.prod.sagebase.org/file/v1 portalEndpoint = https://synapse.org/   

Appendix:  Current implementation of the file cache in the R Client:

• files are cached (meatadata used to be cached in entity.json)

• cache is mix of read/write

• each entity version has a location within the cache is based on its URI (e.g. .synapseCache/proddata.sagebase.org/<entityId>/<locationId>/version/<version>)

  • files.json specifies what resides within the archive
  • <fileName> file which R Client currently assumes to be a zip (this is immutable by convention until storeEntity is called)  (TODO:  What happens when it is not a zip archive)
  • <fileName>_unpacked directory within which all unzipped content lives
    • this subdirectory is writable (by convention)
    • re-stores file if not an archive (both as <fileName> and <fileName>_unpacked/<fileName>)

...