The current multi-part file upload to Synapse involves the following four steps:
The current clients (R, Python, Web, Java) all implements some level of robustness using the above four steps. For example, if a client fails to POST a part to its pre-signed URL the client will attempt to re-try that part. Also, if the job fails at step 4 the clients may restart the file upload starting back at step 1.
The most common problem that causes file upload to fails is the client receives a 201 from a POST of a part, but the part cannot be found in S3. When this occurs the "complete" upload job will fail after waiting 5 minutes for the part to appear. When this happens there is no way for the client to determine which part failed so the only option is to start the file upload from the beginning. Starting over from the beginning is simply unacceptable for large file uploads.
There are three other problems with the current chunked upload API:
We believe, all of these problems can be solved by changing our multi-part file upload API.
With the new file upload, the server will persist the state of a file's upload as each part is successfully added to the upload. This will allow clients to resume a file upload even after critical failures such as client crashes, power outages, and network outages. In order to start a multi-part file upload, the client will be expected to provide key pieces of information that will uniquely identify a file to be uploaded: <user_id>-<md5>-<part_size>. Each unique combination of <user_id>-<md5>-<part_size> will then be issued an ID from the server. All upload state will be persisted and will migrate from stack to stack.
When a client starts a multi-part upload, no assumption about the state of the files should be made. Instead, the server will return the state of the requested file upload which could be anywhere between not-started (no parts upload) to already complete (all parts uploads, and complete with an issued file handle id). If the file has already been upload and a file handle already issued, there will be nothing else for the client to do. If the upload is not completed, the server will return the state of each part (according to the provided part size). The client is then expected to only upload parts that have not yet been successfully added to the multi-part upload. In this way, once a part is successfully added to the multi-part, that part will never need to be re-uploaded even if the client crashes.
Here is an example of how file upload would work:
Here is another example:
Description | Response | URL | Request | Type |
---|---|---|---|---|
Start or resume a multipart upload of a file. By default this method is idempotent, so subsequent calls will simply return the current status of the file upload. If for some reason, the client must restart a file upload from the beginning then then following optional query parameters should be included: forceRestart=true (/file/multipart/upload?forceRestart=true) | MultipartUploadStatus | /file/multipart/upload | MultipartUploadRequest | POST |
Used to get a batch of pre-signed URLS that should be used to upload file parts. Each part will require a unique pre-signed URL. The client is expected to PUT the contents of each part to the corresponding pre-signed URL. Each per-signed URL will expire 15 minute after issued. If a URL has expired, the client will need to request a new URL for that part. | BatchPartUploadURLResponse | /file/multipart/{uploadId}/presignedurl/batch | BatchPartUploadURLRequest | POST |
After the contents of part have been upload (PUT to a pre-signed URL) this method is used to added the part to the multipart upload. If the upload part can be found, and the provided MD5 matches the MD5 of the part, the part will be accepted and added to the multipart upload. | AddMultipartResponse | /file/multipart/{uploadId}/add/{partNumber}?partMD5Hex={partMD5Hex} | PUT | |
After all of the parts have been upload and added successfully, this method is called to complete the upload resulting in the creation of a new file handle. | MultipartUploadStatus | /file/multipart/{uploadId}/complete | PUT |