{"type":"doc","content":[{"type":"layoutSection","marks":[{"type":"breakout","attrs":{"mode":"wide"}}],"content":[{"type":"layoutColumn","attrs":{"width":66.66},"content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"Data Formats","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"9baad844-829f-4dc8-b7e9-60458d34f3fb"}}]}]},{"type":"paragraph","content":[{"text":"Digital health data is exported by Bridge to Synapse Projects in two formats: raw and parquet.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Raw Data","type":"text"}]},{"type":"paragraph","content":[{"text":"Raw data is the data exactly as it has been sent by the app to Bridge. Bridge exports this data to Synapse as Synapse File entities. The raw data can be found in the study’s Synapse project under the “Files” tab in a folder called “Bridge Raw Data”. Additionally, raw data and its metadata can be viewed in a ","type":"text"},{"text":"view","type":"text","marks":[{"type":"link","attrs":{"href":"https://help.synapse.org/docs/Views.2011070739.html"}}]},{"text":" under the “Tables” tab. The view will be named “Bridge Raw Data View”. We don’t recommend working directly with raw data, although you may find the view convenient for working with the metadata.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Parquet Data","type":"text"}]},{"type":"paragraph","content":[{"text":"Parquet data","type":"text","marks":[{"type":"link","attrs":{"href":"https://parquet.apache.org/"}}]},{"text":" is a normalized version of the raw data. It can be found in the study’s Synapse project under the “Files” tab in a folder named “parquet”. Don’t freak out if this folder is empty! This is by design. Underneath the folder name will be text similar to the following:","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center"},"content":[{"type":"media","attrs":{"width":1374,"id":"fd4eb047-cc7e-44ed-804f-df02ebbfee2f","collection":"contentId-2637856853","type":"file","height":72}}]},{"type":"paragraph","content":[{"text":"This tells us where the data ","type":"text"},{"text":"really","type":"text","marks":[{"type":"em"}]},{"text":" is. We use Synapse to control access to this data, although all the real action happens in S3.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Parquet Datasets","type":"text"}]},{"type":"paragraph","content":[{"text":"The Parquet data is stored in S3 as a Parquet dataset, a collection of Parquet files partitioned by a folder hierarchy. For an explanation of how this data is organized, please see ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://sagebionetworks.jira.com/wiki/spaces/BD/pages/2644115482"}},{"text":" .","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Accessing the Data","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"719b1cce-a6b1-416f-9fd5-2d0e5a29c32e"}}]}]},{"type":"paragraph","content":[{"text":"We assume that you already have access to the Synapse project containing your study data. If you don’t, please talk with your PI or whoever configured the study in the Bridge Study Manager. ","type":"text"},{"text":"The Bridge Downstream team cannot unilaterally grant you access to any digital health data.","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"To access the Parquet data, it’s necessary to authenticate with AWS so that their services can grant access to the data in S3. Synapse makes this authentication easy with STS tokens. ","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"68f72000-2564-43cf-a896-1eb2659b32b0"}}]},{"text":"STS tokens","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"68f72000-2564-43cf-a896-1eb2659b32b0"}},{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"ea85bfa1-7948-42dc-bf24-a3b1aaee5b18"}},{"type":"link","attrs":{"href":"https://help.synapse.org/docs/Compute-Directly-on-Data-in-Synapse-or-S3.2048426057.html"}}]},{"text":" c","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"68f72000-2564-43cf-a896-1eb2659b32b0"}},{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"ea85bfa1-7948-42dc-bf24-a3b1aaee5b18"}}]},{"text":"an be retrieved using the Python, R, Java, or command line Synapse clients. Instructions are provided below for Python and R.","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"68f72000-2564-43cf-a896-1eb2659b32b0"}}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"1. Install clients","type":"text"}]},{"type":"paragraph","content":[{"text":"Because of the way we will be interfacing with the data, we don’t need to install an AWS client. Instead, we will use the Synapse client to authenticate with Synapse and retrieve an STS token. We will then use an ","type":"text"},{"text":"Arrow","type":"text","marks":[{"type":"link","attrs":{"href":"https://arrow.apache.org/docs/index.html"}}]},{"text":" client to load the parquet data directly from S3. To work with the parquet data as a data frame in R we also need the ","type":"text"},{"text":"dplyr","type":"text","marks":[{"type":"code"}]},{"text":" dependency.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Python","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"synapseclient","type":"text","marks":[{"type":"link","attrs":{"href":"https://help.synapse.org/docs/Installing-Synapse-API-Clients.1985249668.html"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"PyArrow","type":"text","marks":[{"type":"link","attrs":{"href":"https://arrow.apache.org/docs/python/index.html"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"pandas","type":"text","marks":[{"type":"underline"},{"type":"link","attrs":{"href":"https://pandas.pydata.org/pandas-docs/stable/getting_started/install.html"}}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"R","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"synapser","type":"text","marks":[{"type":"link","attrs":{"href":"https://help.synapse.org/docs/Installing-Synapse-API-Clients.1985249668.html"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"arrow","type":"text","marks":[{"type":"link","attrs":{"href":"https://arrow.apache.org/docs/r/index.html"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"dplyr","type":"text","marks":[{"type":"link","attrs":{"href":"https://dplyr.tidyverse.org/"}}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"2. (optional) Configure Synapse credentials","type":"text"}]},{"type":"paragraph","content":[{"text":"There are multiple ways to cache Synapse credentials so that you don’t need to type your username and password each time you log in with the Synapse client. One of the simplest ways to do so, which works for both Python and R clients, is to edit the following section in the ","type":"text"},{"text":".synapseConfig","type":"text","marks":[{"type":"code"}]},{"text":" file, which is written to the home directory (","type":"text"},{"text":"~","type":"text","marks":[{"type":"code"}]},{"text":") when installing the Synapse client. Don’t forget to uncomment the ","type":"text"},{"text":"[authentication]","type":"text","marks":[{"type":"code"}]},{"text":" header and log in parameters.","type":"text"}]},{"type":"codeBlock","attrs":{"language":"plaintext"},"content":[{"text":"###########################\n# Login Credentials #\n###########################\n\n## Used for logging in to Synapse\n## Alternatively, you can use rememberMe=True in synapseclient.login or login subcommand of the commandline client.\n[authentication]\nusername = \nauthtoken = ","type":"text"}]},{"type":"paragraph","content":[{"text":"Complete documentation on how to configure the client can be found ","type":"text"},{"text":"here","type":"text","marks":[{"type":"link","attrs":{"href":"https://help.synapse.org/docs/Client-Configuration.1985446156.html"}}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"3. Authenticate with AWS using an STS token","type":"text"}]},{"type":"paragraph","content":[{"text":"The below code defines a global variable ","type":"text"},{"text":"PARQUET_FOLDER","type":"text","marks":[{"type":"code"}]},{"text":". This is the Synapse ID of the parquet folder described in the “Parquet Data” section above. Change this value to match the ID of the parquet folder specific to your project.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Python","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"c7f96374-ff09-4671-85df-8628044c86b5"}}]}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"text":"import synapseclient\nfrom pyarrow import fs, parquet\n\nPARQUET_FOLDER = \"syn00000000\"\nsyn = synapseclient.login() # Pass your credentials\n # or configure your .synapseConfig\n# Get STS credentials\ntoken = syn.get_sts_storage_token( \n entity=PARQUET_FOLDER,\n permission=\"read_only\",\n output_format=\"json\") \n# Pass STS credentials to Arrow filesystem interface\ns3 = fs.S3FileSystem(\n access_key=token['accessKeyId'],\n secret_key=token['secretAccessKey'],\n session_token=token['sessionToken'],\n region=\"us-east-1\")","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"R","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"2cfb879a-27fd-463a-b33d-d0b09b594479"}}]}]},{"type":"codeBlock","attrs":{"language":"r"},"content":[{"text":"# Optional, we use :: operators to make the namespace explicit\nlibrary(synapser)\nlibrary(arrow)\nlibrary(dplyr)\n\nPARQUET_FOLDER <- \"syn00000000\"\nsynapser::synLogin() # Pass your credentials\n # or configure your .synapseConfig\n# Get STS credentials\ntoken <- synapser::synGetStsStorageToken(\n entity = PARQUET_FOLDER,\n permission = \"read_only\",\n output_format = \"json\")\n# Pass STS credentials to Arrow filesystem interface\ns3 <- arrow::S3FileSystem$create(\n access_key = token$accessKeyId,\n secret_key = token$secretAccessKey,\n session_token = token$sessionToken,\n region=\"us-east-1\")","type":"text"}]},{"type":"paragraph","content":[{"text":"For those who are curious, full documentation on using STS with Synapse can be found ","type":"text"},{"text":"here","type":"text","marks":[{"type":"link","attrs":{"href":"https://help.synapse.org/docs/Compute-Directly-on-Data-in-Synapse-or-S3.2048426057.html"}}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"4. Read parquet dataset as a data frame","type":"text"}]},{"type":"paragraph","content":[{"text":"Now that we have an S3 file system interface, we can begin interfacing with the S3 bucket. An explanation of how the parquet datasets are organized within the S3 bucket can be found on the page ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://sagebionetworks.jira.com/wiki/spaces/BD/pages/2644115482"}},{"text":" . To view which parquet datasets are available to us:","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"List Parquet datasets","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Python","type":"text"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"text":"base_s3_uri = \"{}/{}\".format(token[\"bucket\"], token[\"baseKey\"])\nparquet_datasets = s3.get_file_info(\n fs.FileSelector(base_s3_uri, recursive=False))\nfor dataset in parquet_datasets:\n print(dataset.path)","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"R","type":"text"}]},{"type":"codeBlock","attrs":{"language":"r"},"content":[{"text":"base_s3_uri <- paste0(token$bucket, \"/\", token$baseKey)\nparquet_datasets <- s3$GetFileInfo(\n arrow::FileSelector$create(base_s3_uri, recursive=FALSE))\nfor (dataset in parquet_datasets) {\n print(dataset$path)\n}","type":"text"}]},{"type":"paragraph","content":[{"text":"Example output:","type":"text"}]},{"type":"codeBlock","attrs":{"language":"plaintext"},"content":[{"text":"bridge-downstream-parquet/bridge-downstream/app-id/study-id/parquet/dataset_audiolevelrecord_v1\nbridge-downstream-parquet/bridge-downstream/app-id/study-id/parquet/dataset_motionrecord_v1\nbridge-downstream-parquet/bridge-downstream/app-id/study-id/parquet/dataset_sharedschema_v1\nbridge-downstream-parquet/bridge-downstream/app-id/study-id/parquet/dataset_sharedschema_v1_userInteractions\nbridge-downstream-parquet/bridge-downstream/app-id/study-id/parquet/dataset_sharedschema_v1_userInteractions_controlEvent\nbridge-downstream-parquet/bridge-downstream/app-id/study-id/parquet/dataset_archivemetadata_v1\nbridge-downstream-parquet/bridge-downstream/app-id/study-id/parquet/dataset_archivemetadata_v1_files\nbridge-downstream-parquet/bridge-downstream/app-id/study-id/parquet/dataset_taskresultobject_v1\nbridge-downstream-parquet/bridge-downstream/app-id/study-id/parquet/dataset_weatherresult_v1\nbridge-downstream-parquet/bridge-downstream/app-id/study-id/parquet/owner.txt","type":"text"}]},{"type":"paragraph","content":[{"text":"Each of the above is a parquet dataset, with the exception of ","type":"text"},{"text":"owner.txt","type":"text","marks":[{"type":"code"}]},{"text":". We can read a parquet dataset into memory directly from S3.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Serialize Parquet dataset as data frame","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Python","type":"text"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"text":"metadata = parquet.read_table(\n \"{}/{}\".format(base_s3_uri, \"dataset_archivemetadata_v1\"),\n filesystem=s3)\nmetadata_df = metadata.to_pandas()","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"R","type":"text"}]},{"type":"codeBlock","attrs":{"language":"r"},"content":[{"text":"metadata <- arrow::open_dataset(\n s3$path(paste0(base_s3_uri, \"/\", \"dataset_archivemetadata_v1\")))\nmetadata_df <- dplyr::collect(metadata)","type":"text"}]},{"type":"paragraph","content":[{"text":"In the above example, we loaded the entire parquet dataset – all metadata across all assessments – into memory. In some cases this isn’t desirable. For example, we might only want to work with data from a specific assessment. To do so, we can take advantage of the partition field ","type":"text"},{"text":"assessmentid","type":"text","marks":[{"type":"code"}]},{"text":", and only read data from a specific assessment. Assuming there is an assessment called ","type":"text"},{"text":"memory-for-sequences","type":"text","marks":[{"type":"code"}]},{"text":" :","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Filter and serialize Parquet dataset as data frame","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Python","type":"text"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"text":"metadata_mfs = parquet.read_table(\n \"{}/{}\".format(base_s3_uri, \"dataset_archivemetadata_v1\"),\n filters = [(\"assessmentid\", \"=\", \"memory-for-sequences\")],\n filesystem=s3)\nmetadata_mfs_df = metadata_mfs.to_pandas()","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"R","type":"text"}]},{"type":"codeBlock","attrs":{"language":"r"},"content":[{"text":"# If you haven't loaded dplyr into your global namespace yet, do so now\n# library(dplyr)\n\nmetadata_mfs <- arrow::open_dataset(\n s3$path(paste0(base_s3_uri, \"/\", \"dataset_archivemetadata_v1\")))\nmetadata_mfs_df <- metadata %>%\n filter(assessmentid == \"memory-for-sequences\") %>%\n collect()","type":"text"}]},{"type":"paragraph","content":[{"text":"You can filter on non-partition columns as well, but this isn’t any faster than serializing the entire parquet dataset. The only advantage is that you won’t need to store the entire dataset in memory, which can be particularly useful for motion or other time-series data.","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Conclusion","type":"text"}]},{"type":"paragraph","content":[{"text":"You now understand the basics of serializing assessment data from a parquet dataset. For more information about interpreting and working with the parquet datasets, see ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://sagebionetworks.jira.com/wiki/spaces/BD/pages/2644115482"}},{"text":" .","type":"text"}]}]},{"type":"layoutColumn","attrs":{"width":33.33},"content":[{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{"minLevel":{"value":"1"},"maxLevel":{"value":"7"}},"macroMetadata":{"macroId":{"value":"65781826c771b60cfcf0a510ccefa322"},"schemaVersion":{"value":"1"},"title":"Table of Contents"}},"localId":"5b738c04-d83f-4c25-bca6-95f7d377dfc8"}}]}]},{"type":"paragraph"}],"version":1}