BAM file upload failing at 96%

Description

A user on the TESLA project is submitting BAM files using the py command-line client. She reports the following behavior:

synapse add /mnt/tesla/<NFS location for BAM> --parentId=syn8594124
The response was a bunch of progress lines which stopped updating after:
de538d-indelreUploading [###################]96.67% 28.5GB/29.5GB (14.5GB/s) c16742443f11d2f94131598e0b169451a130f97936d00974a92cd2e248e3e8e1mark_dups-1-a5453de538d-indelreUploading [###################]96.70% 28.5GB/29.5GB (25.6GB/s) c16742443f11d2f94131598e0b169451a130f97936d00974a92cd2e248e3e8e1mark_dups-1-a5453de538d-indelreUploading [###################]96.73% 28.5GB/29.5GB (32.6GB/s) c16742443f11d2f94131598e0b169451a130f97936d00974a92cd2e248e3e8e1mark_dups-1-a5453de538d-indelreUploading [###################]96.75% 28.5GB/29.5GB (27.5GB/s) c16742443f11d2f94131598e0b169451a130f97936d00974a92cd2e248e3e8e1mark_dups-1-a5453de538d-indelreUploading [###################]96.78% 28.5GB/29.5GB (26.8GB/s) c16742443f11d2f94131598e0b169451a130f97936d00974a92cd2e248e3e8e1mark_dups-1-a5453de538d-indelreUploading [###################]96.81% 28.5GB/29.5GB (27.9GB/s) c16742443f11d2f94131598e0b169451a130f97936d00974a92cd2e248e3e8e1mark_dups-1-a5453de538d-indelreUploading [###################]96.82% 28.5GB/29.5GB (9.9GB/s) c16742443f11d2f94131598e0b169451a130f97936d00974a92cd2e248e3e8e1mark_dups-1-a5453de538d-indelreaUploading [###################]96.85% 28.5GB/29.5GB (23.5GB/s) c16742443f11d2f94131598e0b169451a130f97936d00974a92cd2e248e3e8e1mark_dups-1-a5453de538d-indelreal-mergeddefaultdefaultgatk_bqsr.bam

It hung at 96.85%. The synapse project syn8594124 is the private project I made to contain the TESLA files, with no folders.

Environment

None

Activity

Show:
Kristen Dang
June 2, 2017, 4:24 AM

Can't replicate, so closing.

Kristen Dang
June 1, 2017, 11:10 PM

I can't recreate this issue, so will allow Platform to close if deemed fixed.

Ziming Dong
April 28, 2017, 6:24 PM

should fix this

Ziming Dong
April 27, 2017, 11:59 PM
Edited

the HTTP 'content-length' header value should be larger than the actual file size only in the case where a very small file (the case i found was a 43 byte file) was encoded, causing the metadata from the compression(e.g. gzip) to actually increase the size of data transferred(43 bytes became 55 bytes)

-For these larger, files with several GBs of data, it should be the case that 'content-length' <= file size, "equals" being the case where its not compressed at all.
Also the 'content-length' bug occurred for us on downloads, not uploads-

That being said, I don't really know how I would reproduce this. It shouldn't matter what type of file we are uploading (BAM or not) because we just take the bytes from the file and pass them along to AWS. I also noticed that the user managed to upload files greater than 29.5GB to the same parent project so file size should not be the issue (and I'm sure there has been even larger files uploaded to synapse by other users). Perhaps something went wrong with AWS on that day

Larsson, I misread your comment yesterday and thought it was about uncompressed files being smaller than the compressed file size from the 'content-length' header.

Yes, this should be fixed by SYNPY-431.

Larsson Omberg
April 27, 2017, 11:32 PM

Ziming: I wonder if your retry for file sizes smaller than content-sizes fixes this. If, as I, you suspect this. Please close.

Duplicate

Assignee

Ziming Dong

Reporter

Kristen Dang

Labels

None

Validator

Kristen Dang

Development Area

None

Release Version History

None

Slack Channel

None

Fix versions

Priority

Critical