No DoubleList column type for tables or views
I noticed that annotations consisting of a list of double values are truncated when aggregated in an entity view. I've created an example Synapse file with two annotations: one list of double values and one list of integers. I also created an associated file view that presents these two annotation columns. The output of each using the command line client is shown below.
After some digging, it seems that this is probably caused by the fact that there is no DoubleList column type for tables (and views) broadly speaking. I couldn't find another ticket tracking this, so I thought I would create one.
While I don't have an urgent use case for adding support for a Double, I must admit that I was surprised when I saw the truncation. Specifically, I would have expected a warning. Could this perhaps be done in the interim?
Yep. Thanks for catching that. I've edited the comment and removed the redundant item
Thanks for the explanation. We'll have to prioritize the fix. Item 4, above, seems to be redundant (repeats item 3). Am I misunderstanding?
I initially attempted to support it, but then realized there needed to be additional logic to handle storing and querying NaN in the table indexing the list values. No one had a use case for DOUBLE_LIST at the time so after discussing w/ , we decided to not dedicate the extra effort towards building it.
The warning that is suggesting could be tricky because we don't modify user's schema on their behalf when the annotation list size increases. The same truncation situation could occur on STRING columns if:
Create entities w/ only single-value STRING "foo" annotation
Create a view of these entities using the suggested columns based on existing annotations, which would suggest "foo" as a STRING ColumnType
If we ever updated the Entities' Annotations to have multiple values under "foo", only the first value would appear on the Entity View
Updating Annotations via the Entity View again would then truncate the "foo" annotation back to single value
We could do some validation when replicating values into the user's Entity View and error out if there's more than a single value for all of the Entity's Annotations.
Do you know if there is an explicit reason we did not add a DoubleList column type?