Store Tables timeout error
Not really sure why this is happening, but...
Notes to self: from discussion with Jay regarding SWC-4248, if we separate the "upload" and "query", we still need to "do something" (initiate the query) for the backend to build the index for the updated table.
, here is the discussion about updating a table in the Python client. I still need to talk to more users before decide what to do here.
I will be talking to Python users to figure out what most users expect to happen in this case.
Here is what comes out of my discussion with . Kenny, please feel free to fill in what I miss:
For long running process, it would be nice to know that it's going to take a long time before user starts it. Even though warning and asking users to either "continue" or "cancel" is not very convenience for users who write scripts, we can add "force" like parameter for users to by pass the warning.
In the case of table upload, the operation store is different than storing a file. We can monitor the file upload process. And right after the file is uploaded, the file is available for download.
With table, the csv may be uploaded long before the table is available for download. The advantage of checking for table being available for download is that it's simplify the user's interaction with Synapse, and it's consistent with storing files.
However, the disadvantage of this implementation is the client doesn't know how long it should wait. And if we ask users to specify the wait time, we need to explain why it may take a long time. This is something we should already be doing via documentation. Even then, how would a user know how long they should wait? And what to do after the wait time, and table is still not available? If they restart the job, they worsen the problem.
We can limit the scope of store to upload for both file entity and table entity. When the upload is completed, store should returns success.
We can also add notifyMe parameter to notify user when the entity is available. I could see that in the file entity case, this parameter is close to useless because the file is available after the upload is done. But in the table entity case, this parameter can be used to notify users when their updated table is available.
Assigning to to help me think through options here.
I just read through all the comments and based on the current status, this doesn't sound like a "Blocker" – downgrading to "Critical" but I still think we don't yet know the right solution here. I like the idea mentioned re: a configurable timeout (and a sane default for the client overall), but I'm not sure if there are other considerations.