LargeText columns are 3000 bytes wide

Description

While using Synapse tables, we hit the max length in a table that uses a lot of LargeText fields. While doing some testing, we discovered that Synapse considers LargeText to be 3000 bytes wide (same as a 1000-character string). Is this an oversight or is this by design?

We use a lot of LargeText fields because we write a lot of JSON data to our tables, and we don’t know ahead of time how large these JSON blobs will be. We’d also prefer not to use File Handles for this use case, because researchers have complained about having to download lots of little files when looking at table data.

Is there a way for us to write a lot of data columns of unknown length to Synapse tables without having to either download a bunch of small files or limiting ourselves to 20ish fields per table?

On a related note, is there any documentation on field sizes and max table row size, so we can enforce it in our table creation code on the Bridge side?

Environment

None

Assignee

John Hill

Reporter

Dwayne Jeng

Labels

None

Validator

Dwayne Jeng

Development Area

None

Release Version History

None

Components

Sprint

None

Fix versions

Priority

Minor
Configure