Data in Synapse can be downloaded using the programmatic clients (Python, R, and command line) as well as the web client.
Downloading a File
Every entity in Synapse has a unique synID associated with it. It can be found on every entity page next to Synapse ID:, starting with syn ending with numbers (i.e. syn00123). Files can be downloaded by using the get command. By default, the File downloaded will always be the most recent version. If the current version of the File has already been downloaded, it will not re-download the File.
When using the Python, R, or command line clients, files downloaded using the get command are stored and/or registered in a cache. By default, the cache location is in your home directory in a hidden folder named .synapseCache. Whenever the get function is invoked, the cache is checked to see if the same file is already present by checking its MD5 checksum. If it already exists, the file will not be downloaded again.
For the Python and R clients the default download location is the Synapse cache. The command line client downloads to your current working directory. On the web, your own browser settings determine the download location for files. The Synapse cache is not updated to reflect downloads through a web browser. In all cases you can specify the directory in which to download the file.
For example, to get the experimental protocol file on Adult Mouse Cardiac Myocyte Isolation (syn3158111) from the Progenitor Cell Biology Consortium (PCBC) you would run the following:
Command line
| Code Block |
|---|
synapse get syn3158111
|
Python
| Code Block |
|---|
import synapseclient
syn = synapseclient.Synapse()
syn.login()
entity = syn.get("syn3158111")
|
R
| Code Block |
|---|
library(synapser)
synLogin()
entity <- synGet("syn3158111")
|
Once a File has been downloaded, you can find the filepath using the following:
Command line
| Code Block |
|---|
# When downloading using the command line client, it will print the filepath of where the file was saved to.
|
Python
| Code Block |
|---|
filepath = entity.path
|
R
| Code Block |
|---|
filepath <- entity$path
|
Versions
If there are multiple versions of a File, a specific version can be downloaded by passing the version parameter.
In this example, there are multiple versions of an miRNA FASTQ file (syn3260973) from the Progenitor Cell Biology Consortium. To download the first version:
Command line
| Code Block |
|---|
synapse get syn3260973 -v 1
|
Python
| Code Block |
|---|
entity = syn.get("syn3260973", version=1)
|
R
| Code Block |
|---|
entity <- synGet("syn3260973", version=1)
|
See versioning for more details.
Links
When you click on a Link entity on the Synapse website, it will redirect you to the linked entity. The followLink parameter will have to be specified when using the programmatic clients or you will only retrieve the link itself without downloading the linked entity.
Command line
| Code Block |
|---|
synapse get syn1234 --followLink
|
Python
| Code Block |
|---|
import synapseclient
syn = synapseclient.login()
linkEnt = syn.get("syn1234")
entity = syn.get("syn1234", followLink=True)
|
R
| Code Block |
|---|
library(synapser)
synLogin()
linkEnt = synGet("syn1234")
entity = synGet("syn1234", followLink=TRUE)
|
Download Location
To override the default download location (to not download to the Synapse cache directory, for example), you can specify the downloadLocation parameter.
Command line
| Code Block |
|---|
synapse get syn00123 --downloadLocation /path/to/folder
|
Python
| Code Block |
|---|
entity = syn.get("syn00123", downloadLocation="/path/to/folder")
|
R
| Code Block |
|---|
entity <- synGet("syn00123", downloadLocation="/path/to/folder")
|
Finding and Downloading Files
Files can be annotated to facilitate finding them. In order to search the annotations, a File View must be created first. It is possible to query based on any of the annotations attached to the files.
For example, the PCBC Project has a table listing sequencing data files that have been annotated. To find all mRNA fastq files originating from CD34+ cells in the we can query by:
Command line
| Code Block |
|---|
synapse query 'select * from syn7511263 where dataType="mRNA" AND fileType="fastq" AND Cell_Type_of_Origin="CD34+ cells"'
|
Python
| Code Block |
|---|
results = syn.tableQuery('select * from syn7511263 where dataType="mRNA" AND fileType="fastq" AND Cell_Type_of_Origin="CD34+ cells"')
|
R
| Code Block |
|---|
results <- synTableQuery('select * from syn7511263 where dataType="mRNA" AND fileType="fastq" AND Cell_Type_of_Origin="CD34+ cells"')
df <- as.data.frame(results)
|
Once you’ve queried for the files of interest, they can be downloaded using the following:
Command line
| Code Block |
|---|
synapse get -q 'select * from syn7511263 where dataType="mRNA" AND fileType="fastq" AND Cell_Type_of_Origin="CD34+ cells"'
|
Python
| Code Block |
|---|
results = syn.tableQuery('select * from syn7511263 where dataType="mRNA" AND fileType="fastq" AND Cell_Type_of_Origin="CD34+ cells"')
entity = [syn.get(r['file.id']) for r in results]
|
R
| Code Block |
|---|
results <- synTableQuery('select * from syn7511263 where dataType="mRNA" AND fileType="fastq" AND Cell_Type_of_Origin="CD34+ cells"')
df <- as.data.frame(results)
entity <- lapply(df$file.id, function(x) synGet(x))
|
Recursive Downloads
The folder structure that is present on Synapse can be maintained by recursive downloading.
Command line
| Code Block |
|---|
synapse get -r syn2390898
|
Python
| Code Block |
|---|
import synapseutils
import synapseclient
syn = synapseclient.login()
files = synapseutils.syncFromSynapse(syn, 'syn2390898')
|
R
| Code Block |
|---|
# Unfortunately, this feature is not currently available in the R client
|
Download Tables
Please view here to learn how to use Tables.
Download Wikis
The structure of a project’s Wiki page can be extracted through the R and Python clients. The id, title and parent Wiki page of each sub-Wiki page is also determined through the same method.
Python
| Code Block |
|---|
wiki = syn.getWikiHeaders("syn00123")
|
R
| Code Block |
|---|
entity <- synGet("syn00123")
wiki <- synGetWikiHeaders(entity)
|
The Markdown and other information of a Project sub-Wiki page can be obtained by knowing the id of the Wiki. The Wiki page id can either be obtained through the above method or can be found in the URL “www.synapse.org/#!Synapse:syn00123/wiki/12345” where 12345 is the Wiki page id.
Python
| Code Block |
|---|
wiki = syn.getWiki("syn00123", 12345)
|
R
| Code Block |
|---|
entity <- synGet("syn00123")
wiki <- synGetWiki(entity, 12345)
|
Downloading in Bulk
Files can be downloaded in bulk using the syncFromSynapse function found in the synapseutils helper package. This function crawls all the subfolders of the project/folder that you specify and retrieves all the files that have not been downloaded. By default, the files will be downloaded into your synapseCache, but a different download location can be specified with the path parameter. If you do download to a location out side of synapseCache, this function will also create a tab-delimited manifest of all the files along with their metadata (path, provenance, annotations, etc).
Python
| Code Block |
|---|
# Load required libraries
import synapseclient
import synapseutils
# login to Synapse
syn = synapseclient.login(email='me@example.com', password='secret', rememberMe=True)
# download all the files in folder syn123 to a local folder called "myFolder"
all_files = synapseutils.syncFromSynapse(syn, entity='syn123', path='/path/to/myFolder')
|
R
| Code Block |
|---|
# Load required libraries
library(synapser)
library(synapserutils)
# login to Synapse
synLogin(email='me@example.com', password='secret', rememberMe=TRUE)
# download all the files in folder syn123 to a local folder called "myFolder"
all_files = syncFromSynapse(entity='syn123', path='/path/to/myFolder') |
...
| Info |
|---|
Remember, when downloading this way, the maximum size of a download is 5 GB, or a maximum of 100 files if using the download cart. |
Once you have /wiki/spaces/DOCS/pages/2003796231, here’s how to download that data. Reference the screenshots below for a visual representation of the instructions.
Within the project, you may see a series of folders. Any standalone files are downloadable using the down arrow icon in the Download column for that file (1). You can click the > arrow next to any folder to expand it and view its files within (2). To download any individual file from here, click the download icon in the Download column for that file (1), which will add that file to your download cart.
...
Alternatively, if you want to download every file within a folder, you can do so more efficiently. Click on the name of the folder (instead of just expanding it). On this new page with just that folder and its contents, click Download Options (3) followed by Add to Download Cart (4). You’ll be notified of the number and size of files, and asked if you wish to proceed—click Add (5).
...
Repeat this process for as many files (individual and within folders) that you want to download!
As you add items to your download cart, notice that this gets reflected in the Downloads icon of your Synapse toolbar on the left (6). Click this icon once you are ready to download all files in your cart (at this point, they are not downloaded to your computer yet).
In your download cart, review all the items in the list. From here, you can use the Action column to remove any files that you no longer wish to download (7).
When you’re ready to download, click Download As .Zip Packages (8). This will reveal a Create Your Download Package box below, which will prompt you to enter a package name (9). Enter a name that will be easy to find and follows protocols for your project. Then, click Download Package (10). The zipped package will now be available on your computer.
...