DOI API Update [WIP] (Proposal) (2018)

Background & Motivation

See Meredith Slota (Unlicensed)'s 1-pager and the epic issue:

  PLFM-4063 - Getting issue details... STATUS

In summary, we should aim to do these now:

1. Transition to a new DOI provider, DataCite (necessary, old provider is discontinuing service)
2. Overhaul how we handle asynchronous requests to manage a DOI to match how we handle other asynchronous requests
3. Overhaul how we handle metadata submission to comply with the new provider's standards
   1. At the moment, we submit a lot of "dummy' information with no way to change it, which has caused practical issues for users who utilize DOIs

And prepare to do the following in the future:

1. Be able to easily extend the metadata interface
   1. Submission of metadata can be extended to include optional fields
   2. DataCite occasionally updates their metadata schema. If we can easily adjust to newer schemas as they are released, then we can more easily avert the risk of using a deprecated schema.

Current API + Notes for change

We should consider deprecating the DOI creation calls. We could pseudo-maintain until we decide to deprecate by just making the new API calls when these are called.

In addition to the preexisting API:

Note: the Doi object has a DoiStatus field, we need to evaluate how that should be handled with asynchronous workers (we would probably just deprecate that field in favor of using AsynchronousJobStatus).

Internal design notes

This section contains notes about how we plan to interface with DataCite, and what goes on under-the-hood to register/update a DOI

API Choice

DataCite has two APIs that we can use. They have a standard "MDS" API that they recommend for users, and they have a new (but also seemingly temporary) EZ API that is designed for orgs like us who are transitioning from EZID. For the sake of not having to do more work later, we are opting to not use DataCite's temporary EZ API that is designed to mock the EZID API. Instead, we will be using their standard MDS API, as we would need to transition to it eventually anyways.

HTTP Client

The current EZID client interfaces with EZID using now-deprecated implementations of Apache's HTTP client. We will replace this client with a new client that will use our SimpleHttpClient to make requests to the DataCite MDS API.

CRUD Workflows

With the MDS API the basic workflow to create a DOI is to

1. (POST) Register metadata (including the DOI symbol e.g. 10.####/syn01234)
2. (PUT) Register the DOI symbol and tie it to a URL (synapse.org/#!Synapse:syn01234)

Simply updating the metadata requires just step 1. Both of the above calls are idempotent, so we can combine create and update calls and simply treat the implementation as a create. This would simplify the implementation, though an unnecessary outgoing PUT call would made when existing DOIs are updated.

Retrieval and Conversion of Metadata

DataCite requires that new DOIs have associated metadata that adheres to a schema that they revise occasionally. The current version of their schema is v4.1 (Oct 2017). Metadata that we have registered through EZID is adherent to v2.2 (June 2011), v3, and v4. It is unclear if/when DataCite will deprecate v2.2 and no longer accept it. In another attempt to future-proof our DOI minting service, we will only submit metadata adherent to v4.1.

As a result of this, we must be able to retrieve metadata adherent to schemas 2.2+ in order for the client to update it. We can create a translator tool to convert data from these schemas to an intermediate object (see DoiMetadata above) that can hold the appropriate metadata. This object can be translated into the most recent version of the schema. The client can retrieve and submit this object by interfacing with our API 

Required Involvement and Timeline

Outside of our control

DataCite has yet to approve us and give us a registration account. This should happen soon, at which point we have ~3 months to shift to the new provider

Platform

• Create, test, and implement a Datacite Java client that simplifies creating/updating DOIs and their metadata.
  • Should begin as soon as we agree upon the API
  • This can be done without coordination if we preserve existing behavior, but it would be much easier if we create this client intending to only support proposed and agreed-upon behavior.
  • Implementation can use test credentials until we are ready to switch to DataCite in prod
• Create and route new API changes

Clients

• Support asynchronous API + metadata submission
  • Can begin as soon as we agree upon the API
  • Can implement as soon as it is tested and implemented on backend

UX

• User-facing design of DOI minting process and metadata submission

Questions that need Input

• Should we permit creating DOIs for any object? Or just entities?
  • Shifting to support DOI non-entity objects is non-trivial but it would be easier to support them sooner than later
• Schema enforcement
  • Should we force users to provide required metadata to mint a DOI?
    • Permit and submit no metadata (this is currently the only way to mint a DOI in Synapse)
      • In Datacite, it is only possible to mint a DOI without metadata with the temporary EZID bridge API i.e. it will likely not be possible to mint without metadata in the near future
    • Should we allow users to not supply required metadata? We could fill required fields with "mock" data. (For example, permit submitting a blank author field, and then the backend can submit  "(Author not available)" to Datacite as we currently do)
    • One required metadata field is ResourceTypeGeneral, which has specified categories for the type of resource a DOI refers to. Should we omit categories of resources that are likely not in Synapse? Like "Audiovisual" or "Physical Object". There is no technical benefit of excluding these fields.
  • Future feature expansion: which recommended/optional metadata fields should we permit or require?
    
    • Synapse could theoretically support all metadata fields, but for scope/UX reasons, maybe we shouldn't. Input from UX, users, anyone would be helpful.
• Which fields should be immutable? 
  • DOI ID (this can actually be retrieved from the API call rather than the request body, so the client doesn't need to worry about this)
  • Publisher: "Synapse"
  • Publication Year?
  • Do we hide these from the client, or just automatically overwrite them if they try to change them?

Mockups

TBD

Misc. Notes

When we retrieve DOI data for existing Synapse entities (published to Datacite through EZID), the metadata is compliant with Datacite Schema 2.2 (the most recent version is 4.1). We can leave these alone, and force users to supply the metadata required to be compliant with 4.1 if they want to update the info. This way, we avoid running into issues if/when the old schema is deprecated.