...
| Code Block |
|---|
# we use 'synapseStore=F' to indicate that we only wish to link file <- File(path="file:///corporatenfs/sharedproject/genotypedata.csv", synapseStore=F, name="genotypedata", parentId=pid) # Synapse stores the metadata, but does not upload the file file <- synStore(file) # Now assume a new session, perhaps by a different user # synGet downloads the file to a default location fileId<-"synXXXXX" # we use 'downloadFile=F' to indicate that we do not need a new copy on our local disk file <-synGet(fileId, downloadFile=F) getFileLocation(file) > "/corporatenfs/sharedproject/genotypedata.csv" # now change the meta data and save synAnnot(file, "data type")<-"SNP" # since the File was created with "synapseStore=F", synStore does not upload the file file<-synStore(file) |
Examples Provenance
Provenance can be handled either implicitly by creating it when something is saved/created in Synapse or explicitly by modifying the provenance record of something already in Synapse. For most use cases implicitly creating the provenance record should be enough.
Example 1: Creating a provenance record referencing things in Synapse (assuming already logged in and syn is a synapse object in Python)
Provenance records in Synapse can reference files stored in Synapse by either specifying the synapseIds or the entities stored in synapse#Lets create a file that was created using syn445865 and syn1446185 as input and store it in project syn123
| Code Block | ||||
|---|---|---|---|---|
| ||||
myFile = syn.store(File("/path/to/file", parentId="syn123"), used=['syn445865', 'syn1446185']) |
| Code Block | ||
|---|---|---|
| ||
myFile <- synStore(File("/path/to/file", parentId="syn123"), used=['syn445865', 'syn1446185']) |
#Lets store some code that was executed to generate another file and reference this code as being executed
| Code Block | ||||
|---|---|---|---|---|
| ||||
myCode = syn.store(File("/path/to/script.py", parentId="syn123"))
myFile = syn.store(File("/path/to/file", parentId="syn123"), used=['syn445865', 'syn1446185'], executed=myCode) |
| Code Block | ||
|---|---|---|
| ||
myCode <- syn.store(File("/path/to/script.R", parentId="syn123"))
myFile <- synStore(File("/path/to/file", parentId="syn123"), used=['syn445865', 'syn1446185'], executed=myCode) |
#To specify the name of the activity that was performed you can specify the name and description of the activity
| Code Block | ||||
|---|---|---|---|---|
| ||||
myFile = syn.store(File("/path/to/file", parentId="syn123"), used=['syn445865', 'syn1446185'], activityName="Manual editing of file", activityDescription="Corrected spelling of variable names") |
| Code Block | ||
|---|---|---|
| ||
myFile <- synStore(File("/path/to/file", parentId="syn123"), used=['syn445865', 'syn1446185'],
activityName="Manual editing of file", activityDescription="Corrected spelling of variable names")) |
Example 2: Creating a provenance record referencing things files stored on the web
The provenance record can also store references to datasets and information stored in external URLs not in Synapse. In this case the
| Code Block | ||||
|---|---|---|---|---|
| ||||
myFile = syn.store(File("/path/to/file", parentId="syn123"), used=['syn445865', 'http://www.google.com'], activityName="Updated dbgap ids") |
| Code Block | ||
|---|---|---|
| ||
myFile <- synStore(File("/path/to/file", parentId="syn123"),
used=['syn445865', 'http://www.google.com'],
activityName="Updated dbgap ids") |
Command Set
We conceptual divide the client commands into three levels (1) Common functions, (2) Advanced functions and (3) low-level Web API functions. The first collection of commands captures the majority of functionality of interest to users. The second collection rounds out the functionality with less frequently used functions. The third set comprises simple, low level wrappers around the Synapse web service interface. By including this third set users can access web services in advance of having specialized commands in the analytic clients.
...
| Command | comments | R Syntax | Python Syntax | Command Line Syntax |
|---|---|---|---|---|
| 1 – Common functions | ||||
| Create a Synapse file handle in memory, specifying the path to the file in the local file system, the name in Synapse, and the Folder in Synapse. This step 'stages' a file to be sent to Synapse. Additional parameters (...) are interpreted as properties or annotations, in the manner of synSet(), defined below. If 'synapseStore' is TRUE then file is uploaded to S3, else only the file location is saved. | The specified file doesn't move or get copied. | File(path, parentId, synapseStore=T, ...)
example: File(path="/path/to/file", parentId="syn101") | File(path ="/path/to/file", parentId, synapseStore=T, **kwargs)
example: File('/foo/baz/bar.txt', 'syn123') | NA |
| Create a Synapse file handle in memory which will be a serialized version of an in-memory object. Additional parameters (...) are interpreted as properties or annotations, in the manner of synSet(), defined below. If 'synapseStore' is TRUE then file is uploaded to S3, else only the file location is saved. | The object is not serialized at this time. (We are hoping people will like calling the object a File, even though it takes an in-memory object as a parameter.) | File(obj) Note: Due to limitations in R, properties and annotations must be added separately
example: file<-File(obj=dataObject) synAnnot(file, "parentId")<-"syn101"
| Will not be implemented in python. | NA |
| Create a Synapse Record in memory, specifying the name and the Folder in Synapse. This step 'stages' a Record to be sent to Synapse. Additional parameters (...) are interpreted as properties or annotations, in the manner of synSet(), defined below.
| Files aren't moved or copied. TODO: How do you specify file annotations (as distinct from Strings)? Shall we introduce in-memory wrappers around files and urls to help distinguish them? | Record(name=NULL, parentId="syn101", ...) example: | Record(name="foo", parentId="syn101", **kwargs) | |
| Create a Folder or Project in memory. Name and parentId are optional. | Folder(name=NULL, parentId=NULL, ...) Project(name=NULL, ...) example: | Folder(name="foo", parentId="syn101", **kwargs) Project(name="foo", **kwargs) | ||
| Set an entity's attribute (property or annotation) in memory. Client first checks properties, then goes to annotations; (setting to NULL deletes it in R, using DEL operator in python deletes it) | TODO: we want to include files and (for R) in memory objects | synAnnot(entity, name)<-value | entity.parentId="syn101" | synapse update id --parentId syn101 |
| Gets an entity's attribute value (property or annotation) from the object already in memory. | synAnnot(entity, name); returns NULL if undefined | entity.name; throws exception if value is undefined | ||
| Create or update an entity (File, Folder, etc.) in Synapse. May also specify (1) the list of entities 'used' to generate this one, (2) the list of entities 'executed' to generate this one, (3) the name of the generation activity, and (4) the description of the generation activity, (5) whether a name collision in an attempted 'create' should become an 'update', (6) whether to 'force' a new version to be created, and (7) whether the data is restricted (which will put a download 'lock' on the data and contact the Synapse Access and Compliance team for review. | TODO: Give some examples. | synStore(entity, used=NULL, executed=NULL, activityName=NULL, activityDescription=NULL, createOrUpdate=T, forceVersion=T, isRestricted=F)
synStore(entity, activity=NULL, createOrUpdate=T, forceVersion=T, isRestricted=F) | synapse.store(entity, used, executed, activityName=None, activityDescription=None, createOrUpdate=T, forceVersion=T, isRestricted=F)
synapse.store(entity, activity, createOrUpdate=T, forceVersion=T, isRestricted=F) | synapse create --name NAME --parentid PARENTID --description DESCRIPTION --type TYPE --file PATH --update=T/F --forceVersion=T/F
--annotations={foo=bar, bar=foo} |
| Get an entity (file, folder, etc.) from the Synapse server, with its attributes (properties, annotations) and, optionally, with its associated file(s). if.collision is one of "keep.both", "keep.local", or "overwrite.local", telling the system what to do if a different file is found at the given local file location. | 'download' and 'load' are ignored for objects lacking Files. OK for download=F and load=T, this means don't cache (a valid choice if the File lives on a network share). If a downloadLocation is not provided a default, read-only cache location is used. If a downloadLocation IS provided, then the client must handle collisions with existing files. Note, 'downloadLocation' must be a folder, i.e. it cannot be used to rename files. | synGet(id, version, downloadFile=T, downloadLocation=NULL, ifcollision="keep.both", load=F) | synapse.get(id, version, downloadFile=True, downloadLocation=None, ifcollision="keep.both", load=False) | synapse get ID -v NUMBER |
| Get the directly readable location of the file associated with a file object. | For downloaded files, this is the path on the local file system. For "linked" files (not uploaded into Synapse) that are not downloaded, this is the URL known to Synapse. For uploaded files which have not been retrieved, returns NULL. | getFileLocation(file) | TODO: Why not access it as a property of the entity the way it is currently done? | TODO |
| Trash an entity, and all of its children (move all Folders and Files within a Folder to the trash can). | synTrash(id) / synTrash(entity) | synapse.trash(id) | synapse trash id | |
| Open the web browser to the page for this entity. | onWeb(entityId) / onWeb(entity) | synapse.onweb(entityId) / synapse.onweb(entity) | synapse onweb id | |
| log-in | get API key and write to user's properties file | synapseLogin(<user>,<pw>) | synapse.login(<user>,<pw>, sessionToken=None) | synapse login -u USER -p PASSWORD |
| log-out | delete API key from properties file | synapseLogout() | synapse.logout() | synapse logout |
| 2 –Advanced functions | ||||
| Execute query | TODO: pagination, e.g. the function returns an iterator. Look at current implementation in R client. | synQuery(queryString) | synapse.query(queryString) | synapse query |
| we talked about this, but is it needed? | ||||
| we talked about this, but is it needed? | ||||
| Retrieve the wiki for an entity | TODO: Is it a requirement that we retrieve attachments? If not, do we retrieve file handles? Is this the id of the wiki or the wiki? | synGetWiki(id, version) / synGetWiki(entity) | synapse.getWiki(id, version) synapse.getWiki(entity) | |
| synStoreWiki() | synapse.storeWiki() | |||
| synGetAnnotations() | synapse.getAnnotations(entity/entityId) | |||
| synSetAnnotations() | synapse.setAnntotations(entity/entityId, annotations) | |||
| synGetProperties() | NA | NA | ||
| Access properties, throwing exception if property is not defined. | synSetProperties() | NA | NA | |
| synGetAnnotation() | ||||
| synSetAnnotation() | ||||
| Access property, throwing exception if property is not defined. | synGetProperty() | NA | NA | |
| Access property, throwing exception if property is not defined. Setting to NULL deletes. | synSetProperty() | NA | NA | |
| Create an Activity (provenance object) in memory. | Activity(name, description, used, executed) | Activity(name, description, used, exectuted) | NA | |
| Create or update the Activity in Synapse | synStoreActivity(activity) | Activity.store() | NA | |
| Get the Activity which generated the given entity. | synGetActivity(entity) / synGetActivity(entityId) | synapse.getActivity(entity/entityId) | NA | |
| Set the Activity which generated the given entity | synSetActivity(entity)<-activity | synapse.setActivity(entity/entityId, activity) | NA | |
| Empty trash can | ||||
| Restore from trash can | ||||
| Run code, capturing output, code and provenance relationship. | synapseExecute(executable, args, resultParentId, codeParentId, resultEntityProperties = NULL, resultEntityName=NULL, replChar=".") | synapse.exceute(executable, args, resultParentId, codeParentId, resultEntityProperties = None, resultEntityName=None, replChar=".") | NA | |
| Create evaluation object | Evaluation(name, description, status) | Evaluation(name, description, status) | NA | |
| Join evaluation | addParticipant(evaluation, principalId) | evaluation.addParticipant(principalId) | NA | |
| Submit for evaluation | submit(evaluation, entity) | evaluation.submit(entity)/ synapse.submitToEvaluation(entity, evaluation) | synapse submitEvaluation | |
| 3 – Web API Level functions | ||||
| Execute GET request | See details below. | synRestGET(endpoint, uri) | NA? (already only a line) | |
| Execute POST request | See details below. | synRestPOST(endpoint, uri, body) | NA? | |
| Execute PUT request | See details below. | synRestPUT(endpoint, uri, body) | NA? | |
| Execute DELETE request | See details below. | synRestDELETE(endpoint, uri) | NA? |
...