No DoubleList column type for tables or views

Description

I noticed that annotations consisting of a list of double values are truncated when aggregated in an entity view. I've created an example Synapse file with two annotations: one list of double values and one list of integers. I also created an associated file view that presents these two annotation columns. The output of each using the command line client is shown below.

After some digging, it seems that this is probably caused by the fact that there is no DoubleList column type for tables (and views) broadly speaking. I couldn't find another ticket tracking this, so I thought I would create one.

While I don't have an urgent use case for adding support for a Double, I must admit that I was surprised when I saw the truncation. Specifically, I would have expected a warning. Could this perhaps be done in the interim?

Environment

None

Activity

Show:
Ziming Dong
March 2, 2021, 10:25 PM

Yep. Thanks for catching that. I've edited the comment and removed the redundant item

Bruce Hoff
March 2, 2021, 10:19 PM

Thanks for the explanation. We'll have to prioritize the fix. Item 4, above, seems to be redundant (repeats item 3). Am I misunderstanding?

Ziming Dong
March 2, 2021, 9:20 PM
Edited

:

I initially attempted to support it, but then realized there needed to be additional logic to handle storing and querying NaN in the table indexing the list values. No one had a use case for DOUBLE_LIST at the time so after discussing w/ , we decided to not dedicate the extra effort towards building it.

The warning that is suggesting could be tricky because we don't modify user's schema on their behalf when the annotation list size increases. The same truncation situation could occur on STRING columns if:

  1. Create entities w/ only single-value STRING "foo" annotation

  2. Create a view of these entities using the suggested columns based on existing annotations, which would suggest "foo" as a STRING ColumnType

  3. If we ever updated the Entities' Annotations to have multiple values under "foo", only the first value would appear on the Entity View

  4. Updating Annotations via the Entity View again would then truncate the "foo" annotation back to single value

We could do some validation when replicating values into the user's Entity View and error out if there's more than a single value for all of the Entity's Annotations.

Bruce Hoff
March 2, 2021, 5:07 PM

Do you know if there is an explicit reason we did not add a DoubleList column type?

Assignee

Unassigned

Reporter

Bruno Grande

Validator

Bruce Hoff

Priority

Major

Labels

Development Area

None

Sprint

None

Fix versions

None

Release Version History

None

Story Points

None

Epic Link

None

Slack Channel

None