Versions Compared

Old Version 12

changes.mady.by.user Eric Wu

Saved on

compared with

New Version Current

changes.mady.by.user John Hill

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 5.3

...

Code Block
{
    "description":"The Status of the stack",
    "properties":{
        "status":{
            "type":"string",
            "description":"The status of this stack can be one of the following enumerations",
            "name":"StatusEnum",
            "id":"org.sagebionetworks.repo.model.status.StatusEnum",
            "enum":[
                "READ_WRITE",
                "READ_ONLY",
                "DOWN",
             
            ],
            
        },
        "currentMessage":{
            "type":"string",
            "description":"This message applies to the current state of the stack."
        },
        "pendingMaintenanceMessage":{
            "type":"string",
            "description":"This message is used to notify users of pending maintenance"
        },
        
    }
}

Creating a Repository Backup Snapshot

A backup copy of the Repository Services is created by starting a daemon process that will stream all of the repository to a file, and then upload the file to its S3 bucket. This daemon can be
launched and monitored from a set of HTTP Rest calls. Note: All Backup/Restore services will require an administrator's session token.

First, log-in as an administrator (use the platform@sagebase.org account) to get an administrator's session token:

...

Storage Usage

Check the (external) storage usage by the Synapse system.  You must be an administrator to perform these actions.

Get aggregated totals

Aggregated over any combination of dimensions listed in StorageUsageDimension (see StorageUsageDimension in Synapse Entity Types) except for USER_ID and ENTITY_ID which are provided via separate APIs.  For example, the following command lists storage usage by content type and storage provider:

Code Block
languagebash
curl -i -H sessionToken:<token> -H Accept:application/json

...

Once you have the admin token the backup daemon can be started with the following command:
Request

Code Block
curl -i -k -H sessionToken:<your admin token> -H Accept:application/json -H Content-Type:application/json -d '{
}' https://staging-reposervice.elasticbeanstalk.com/repo/v1/admin/daemon/backup

Response

Code Block
HTTP/1.1 201 Created
Content-Type: application/json
Date: Thu, 18 Aug 2011 22:38:56 GMT
Server: Apache-Coyote/1.1
Content-Length: 256
Connection: keep-alive
{
	"id":"6695",
	"type":"BACKUP",
	"status":"STARTED",
	"errorMessage":null,
	"progresssMessage":"Starting...",
	"progresssCurrent":0,
	"progresssTotal":0,
	"errorDetails":null,
	"backupUrl":null,
	"totalTimeMS":0,
	"startedBy":"platform@sagebase.org",
	"startedOn":1313707136615
}

...

Once the daemon is started, its progress can be monitored using its 'id' returned from the 'admin/daemon' call:
Request

Code Block
curl -i -k -H sessionToken:<your admin token> -H Accept:application/json -H Content-Type:application/json  https://staging-reposervice.elasticbeanstalk.com/repo/v1/admin/daemon/6695

Response

Code Block
HTTP/1.1 200 OK
Content-Type: application/json
Date: Thu, 18 Aug 2011 22:46:06 GMT
Server: Apache-Coyote/1.1
Content-Length: 1114
Connection: keep-alive

{
	"id":"6695",
	"type":"BACKUP",
	"status":"FAILED",
	"errorMessage":"Access Denied",
	"progresssMessage":"Starting to upload temp file: /opt/tomcat7/temp/BackupDaemonJob6695-762955157468269943.zip to S3...",
	"progresssCurrent":785,
	"progresssTotal":863,
	"errorDetails":"Status Code: 403, AWS Request ID: CF8A8149099FE5F9, AWS Error Code: AccessDenied,...",
	"backupUrl":null,
	"totalTimeMS":26145,
	"startedBy":"platform@sagebase.org",
	"startedOn":1313707136615
}

In this example, we are showing that the backup failed. When there is a failure, the status will show as 'FAILED', and 'errorMessage' and 'errorDetails' should show the message and stack trace of the failure. In this example, the service IAM user did not have permission to write the backup file to S3. After fixing the AWS permissions we can try running the backup again:
Response

Code Block
HTTP/1.1 200 OK
Content-Type: application/json
Date: Thu, 18 Aug 2011 23:00:10 GMT
Server: Apache-Coyote/1.1
transfer-encoding: chunked
Connection: keep-alive

{
	"id":"6696",
	"type":"BACKUP",
	"status":"COMPLETED",
	"errorMessage":null,
	"progresssMessage":"Finished: BACKUP",
	"progresssCurrent":863,
	"progresssTotal":863,
	"errorDetails":null,
	"backupUrl":"https://s3.amazonaws.com/stagingdata.sagebase.org/BackupDaemonJob6696-911306061719227050.zip",
	"totalTimeMS":24880,
	"startedBy":"platform@sagebase.org",
	"startedOn":1313708374613
}

This time we can see that the backup 'status'='COMPLETED', and that the 'backupUrl' is no longer null and that the entire backup ~25 seconds to complete. We can now use the file found at the 'backupUrl' to restore synapse.

Restore a Repository from a Backup Snapshot

Restoring a Repository Service from a backup is just the reverse of a backup. A restore daemon is started that will download the backup file from the service's S3 bucket, and then stream the data into repository.
Caution - The restoration process will start by clearing all data from the repository before applying the backup data.
Note: All Backup/Restore services will require an administrator's session token.

  1. Fist make sure the backup file that you want to restore is in the S3 bucket that belongs to that service. A service can only download files from its own bucket.

  2. Once the backup in place in S3 we are ready to authenticate and get an administrator's session token:

    Code Block
    curl -i -k -H sessionToken:YourSessionToken -H Accept:application/json -H Content-Type:application/json -d '{
  "email": "<admin username>",
  "password": "<admin password>"
}' http://localhost:8080/services-authentication-0.6-SNAPSHOT/auth/v1/session

  3. Now use the administrator's token to start the restore daemon. You must provide the file name of the backup file found on S3 to the daemon:
    Request

    Code Block
    curl -i -k -H sessionToken:YourSessionToken -H Accept:application/json -H Content-Type:application/json -d '{
  "url": "BackupDaemonJob6696-911306061719227050.zip"
}' http://localhost:8080/services-repository-0.6-SNAPSHOT/repo/v1/admin/daemon/restore

    Response

    Code Block
    HTTP/1.1 201 Created
Server: Apache-Coyote/1.1
Content-Type: application/json
Transfer-Encoding: chunked
Date: Thu, 18 Aug 2011 23:31:28 GMT

{
	"id":"4",
	"type":"RESTORE",
	"status":"STARTED",
	"progresssMessage":"Starting...",
	"progresssCurrent":0,
	"progresssTotal":0,
	"errorMessage":null,
	"errorDetails":null,
	"backupUrl":null,
	"totalTimeMS":0,
	"startedBy":"platform@sagebase.org",
	"startedOn":1313710288153
}

Once the daemon is started its progress can be monitored in the same way as we monitored the backup daemon, using the 'id' provided by the /admin/daemon:
Request

Code Block
curl -i -k -H sessionToken:YourSessionToken -H Accept:application/json -H Content-Type:application/json  http://localhost:8080/services-repository-0.6-SNAPSHOT/repo/v1/admin/daemon/4

Response

Code Block
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: application/json
Transfer-Encoding: chunked
Date: Thu, 18 Aug 2011 23:37:16 GMT

{
	"id":"4",
	"type":"RESTORE",
	"status":"FAILED",
	"progresssMessage":"Starting to download the file from S3...",
	"progresssCurrent":0,
	"progresssTotal":0,
	"errorMessage":"Access Denied",
	"errorDetails":"Status Code: 403, AWS Request ID: 5A6F56E7416E1203, AWS Error Code: AccessDenied, AWS Error Message: Access Denied, S3 Extended Request ID: ...",
	"totalTimeMS":781,
	"startedBy":"platform@sagebase.org",
	"startedOn":1313710288153
}

Just like in our backup example, the restore failed as indicated by the 'status'='FAILED'. Again it looks like the service did not have permission to download the S3 file. Upon closer inspection we find that we have a cut and paste error with the file name. Once our error is identified, we can start another restore daemon with the following results:
Response

Code Block
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: application/json
Transfer-Encoding: chunked
Date: Thu, 18 Aug 2011 23:59:13 GMT

{
	"id":"5",
	"type":"RESTORE",
	"status":"COMPLETED",
	"progresssMessage":"Finished: RESTORE",
	"progresssCurrent":1164611,
	"progresssTotal":1164611,
	"errorMessage":null,
	"errorDetails":null,
	"backupUrl":"https://s3.amazonaws.com/devdata.sagebase.org/BackupDaemonJob6696-5911306061719227050.zip",
	"totalTimeMS":46800,
	"startedBy":"platform@sagebase.org",
	"startedOn":1313711855784
}

This time we can see that the restore 'status'='COMPLETED', and that the entire restore took ~47 seconds to complete. We have successfully migrated all data from the staging to our local repository service.

Storage Usage

Check the (external) storage usage by the Synapse system.  You must be an administrator to perform these actions.

Get aggregated totals

Aggregate over any combination of dimensions listed in StorageUsageDimension except for USER_ID and ENTITY_ID which are provided via separate APIs.  For the list of aggregating dimensions, see Synapse Entity Types.  For example, the following command lists storage usage by content type and storage provider:

Code Block
languagebash
curl -i -H sessionToken:<token> -H Accept:application/json https://repo-staging.sagebase.org/repo/v1/admin/storageSummary?aggregation=CONTENT_TYPE,STORAGE_PROVIDER

Will get results like the following

Code Block
languagebash
{
    "summaryList": [https://repo-staging.sagebase.org/repo/v1/admin/storageSummary?aggregation=CONTENT_TYPE,STORAGE_PROVIDER

Will get results like the following

Code Block
languagebash
{
    "summaryList": [
        {
            "aggregatedCount": 181, 
            "aggregatedSize": 5409454583, 
            "dimensionList": [
                {
                    "dimension": "CONTENT_TYPE", 
                    "value": "application/binary"
                }, 
                {
                    "dimension": "STORAGE_PROVIDER", 
                    "value": "awss3"
                }
            ]
        }, 
        {
            "aggregatedCount": 9, 
            "aggregatedSize": 0, 
            "dimensionList": [
                {
                    "dimension": "CONTENT_TYPE", 
                    "value": "application/binary"
                }, 
                {
                    "dimension": "STORAGE_PROVIDER", 
                    "value": "external"
                }
            ]
        }, 
        {
            "aggregatedCount": 3, 
            "aggregatedSize": 195333619, 
            "dimensionList": [
                {
                    "dimension": "CONTENT_TYPE", 
                    "value": "application/octet-stream"
                }, 
                {
                    "dimension": "STORAGE_PROVIDER", 
                    "value": "awss3"
                }
            ]
        }, 
        {
            "aggregatedCount": 3590, 
            "aggregatedSize": 0, 
            "dimensionList": [
                {
                    "dimension": "CONTENT_TYPE", 
                    "value": "application/octet-stream"
                }, 
                {
                    "dimension": "STORAGE_PROVIDER", 
                    "value": "external"
                }
            ]
        }, 
        {
            "aggregatedCount": 19, 
            "aggregatedSize": 9927282, 
            "dimensionList": [
                {
                    "dimension": "CONTENT_TYPE", 
                    "value": "application/pdf"
                }, 
                {
                    "dimension": "STORAGE_PROVIDER", 
                    "value": "awss3"
                }
            ]
        }, 
        {
            "aggregatedCount": 1818370, 
            "aggregatedSize": 54094545830, 
            "dimensionList": [
                {
                    "dimension": "CONTENT_TYPE", 
                    "value": "application/binaryx-tar"
                }, 
                {
                    "dimension": "STORAGE_PROVIDER", 
                    "value": "awss3"
                }
   external"
        ]         }, 
        {
            "aggregatedCount": 9, 

           "aggregatedSize": 0,    ]
        },
"dimensionList": [   ] 
}

Without any aggregation, the same URL will get only the grand totals.

Code Block
languagebash
curl -i -H sessionToken:<token> -H       Accept:application/json https://repo-staging.sagebase.org/repo/v1/admin/storageSummary
Code Block
languagebash
{
        "totalCount":133827,
           "dimension": "CONTENT_TYPE""totalSize":1080865415607,
                     "value": "application/binary"
                }, 
      "summaryList":[]
}

Get usage by users

Lists the aggregated usage by users in descending order of 'aggregatedSize':

Code Block
languagebash
curl -i -H sessionToken:<token> -H Accept:application/json https://repo-staging.sagebase.org/repo/v1/admin/storageSummary/perUser

Example response body listing the top 3 users:

Code Block
languagebash
{
    "summaryList": [
         {
       
            "dimensionaggregatedCount": "STORAGE_PROVIDER"35204,  
                   "value"aggregatedSize": "external"283329626902, 
            "dimensionList": [
 }             ]  {
      },          {    "dimension": "USER_ID", 
      "aggregatedCount": 3,             "value": "aggregatedSize273975":
195333619,              "dimensionList": [ }
            ]
  {      }, 
        {
    "dimension": "CONTENT_TYPE",       "aggregatedCount": 458, 
            "valueaggregatedSize": "application/octet-stream"227459357704, 
               }, "dimensionList": [
                {
                    "dimension": "STORAGEUSER_PROVIDERID", 
                    "value": "awss3342024"
                }
            ]
        }, 
        {
            "aggregatedCount": 3590179, 
            "aggregatedSize": 066928683678, 
            "dimensionList": [
                {
                    "dimension": "CONTENTUSER_TYPEID", 
                    "value": "application/octet-stream274010"
                },
            ]
    {                     "dimension": "STORAGE_PROVIDER"},
                     "value": "external"
      ...
    ]
}

Get usage by entities

Lists the aggregated usage by entities in descending order of 'aggregatedSize':

Code Block
languagebash
curl -i -H sessionToken:<token> }
  -H Accept:application/json https://repo-staging.sagebase.org/repo/v1/admin/storageSummary/perEntity

Example response body listing the top 3 entities:

Code Block
languagebash
{
         ]
        }, "summaryList": [
        {
            "aggregatedCount": 191, 
            "aggregatedSize": 99272825210220350, 
            "dimensionList": [
                {
                    "dimension": "CONTENTNODE_TYPEID", 
                    "value": "application/pdf317475"
                },
            ]
    {    }, 
         {
     "dimension": "STORAGE_PROVIDER",       "aggregatedCount": 1, 
            "valueaggregatedSize": "awss3"4670860925, 
            "dimensionList": [
 }             ]   {
     },          {     "dimension": "NODE_ID", 
     "aggregatedCount": 8370,              "aggregatedSizevalue": 0,"1113803"
             "dimensionList": [  }
            ]
 {       }, 
        {
   "dimension": "CONTENT_TYPE",        "aggregatedCount": 2, 
           "value": "application/x-taraggregatedSize": 4235837739, 
              }, "dimensionList": [
                {
                    "dimension": "STORAGENODE_PROVIDERID", 
                    "value": "external4494"
                }
            ]
     
  },     ]  }

Without any aggregation, the same URL will get only the grand totals.

Code Block
languagebash
curl -i -H]
sessionToken:<token> -H Accept:application/json https://repo-staging.sagebase.org/repo/v1/admin/storageSummary
Code Block
languagebash
{     "totalCount":133827,}, 
        "totalSize":1080865415607,...
    "summaryList":[]
}