Improve error messages in syncToSynapse, especially by adding a summary report
I have a collaborator that uploaded >1000 files using syncToSynapse. However, 30 of the files failed to upload but she was unaware of it because there was no notification that informed her of the failure. She asked if there was a way to know when using the function that something failed, rather than doing a diff of the fileview versus her manifest. It would be great if an object was returned that contained the results of the successes/failures of uploading via syncToSynapse.
Discussed with and . We are going to improve syncToSynapse by printing out the results (how many files was processed, which file was successful upload, which file failed to upload, upload speed, ...) before terminating. This should be the same with what we will send users via email.
Notes for self: Another suggestions I have heard recently is to add an optional parameter to write the results to a log file (1). This is similar with a rotating log file idea (2). The difference is in (1), the format of the log file should be human readable but also programmatically readable. And the log file only contains results/ logs for syncToSynapse. While (2) suggests to have a log file that capture xxx lines of logs for all Synapse activities. In (2), users does not have to know about this log file. It comes in handy when a Synapse Python client developer needs more information for debugging a problem.
They should have recieved error messages on the console and via email unless they turned of email messages. We don't clearly indicate permanent failures though. Currently we inicate that we are retrying:
but don't summarize the failure rate at the end.
Same collaborator retried uploading the missing files and turns out two of them failed to upload this time around.