Single thread uploading fails in Lambda python3.6 environment

Description

If I try to `syn.store` a file object within a Lambda function, I get the following traceback:

I've created a repo that you can clone to reproduce the problem. To reproduce:

1. Create an EC2 instance using this AMI (the same AMI which Lambda uses)
2. Assign the instance an IAM role with a Lambda access policy
3. Create a Lambda function with the name `test_single_thread`. Change the handler to `main.handler`, and give it environment variables `synapseUsername` and `synapsePassword`.
4. Clone this repo onto the instance and run `main_builder.sh`.
4. Test the function with an arbitrary test event.

Environment

None

Activity

Show:
Kimyen Truong
January 8, 2019, 6:11 PM

Thanks ! I will reduce the priority of this issue and will be working on it as part of Python client version 2.0.

Kimyen Truong
February 8, 2019, 12:19 AM

I am evaluating all aspects of this issue that we discussed in previous conversations. So I will comment about my findings one at a time.

, I looked into features and multiprocessing. The problem they mentions are in concurrent.features.ProcessPoolExecutor. We are using multiprocessing.pool.ThreadPool which is 1. a different package, and 2. we are using threads instead of processes. So I think we are safe here.

Kimyen Truong
February 8, 2019, 12:42 AM

Per this post, multiprocessing.Value is used to shared the counter between multiple processes: https://eli.thegreenplace.net/2012/01/04/shared-counter-with-pythons-multiprocessing

In our code, Value is used to keep track of completed and whether the presigned URL has ben expired expired.

In a single thread case, nothing needs to be shared. completed and expired are only changed by one single thread. Their default value is defined:

Kimyen Truong
February 12, 2019, 9:59 PM

, would you have time to validate this ticket by Friday?

Phil Snyder
February 14, 2019, 12:06 AM

Yay!

Assignee

Kimyen Truong

Reporter

Phil Snyder

Labels

Validator

Phil Snyder

Development Area

None

Release Version History

None

Sprint

None

Fix versions

Affects versions

Priority

Major
Configure