Document toolboxDocument toolbox

Generate OpenAPI Spec

Owned by Leon Li (Unlicensed)
Last updated: Jun 30, 2023
7 min read

The purpose of this document is to specify the steps and effort needed to generate an OpenAPI specification for the Synapse API.

The research spike for this feature can be found here:

PLFM-7447 - Getting issue details... STATUS

The epic for this project can be found here (add Jira tickets to the epic moving forward):

PLFM-7768 - Getting issue details... STATUS

Note: This project is still in progress! To view next steps, see the “Work moving forward” section of this document.

Background

Currently, the synapse REST API can be found here and use mechanisms developed inside sage to generate documentation for each endpoint. This is an issue as we do not follow a common standard regarding how a client can be generated. This means that if an outside party wants to create a Python client for out service, they would need to manually define and maintain types for all of the endpoints and what they return.

This not only requires a lot of effort on all developers who want to use the Synapse API but is also a place where many bugs can be introduced. The goal of this project is to find a way to generate a specification for the endpoints that can be used to create clients dynamically.

Solution

The OpenAPI specification is a standard way to describe an API through a json/yaml file. By creating an OpenAPI specification for Synapse, we can leverage the Swagger tool to generate documentation for the API as well as dynamic clients in any language.

Helpful Terms

  • OpenAPI - Specification (what the dynamically generated api-docs should look like, etc..)

  • Swagger - Set of tools that actually implement the specification

    • Swagger UI - collection of assets that dynamically generate documentation that is readable and user’s can interact with

    • Swagger Codegen - allows the generation of API client libraries, server stubs, and documentation automatically when given OpenAPI spec

  • Springfox - Tool used to automatically generate JSON API documentation that is compatible with OpenAPI spec through annotations

    • Full documentation here

  • Springdoc - Another tool used to dynamically generate JSON API documentation through annotations that is compatible with OpenAPI v3 in spring-boot applications

    • Full documentation here

How to generate the OpenAPI specification

There are several different ways to generate an OpenAPI specification for an application:

  1. Use SpringFox

  2. Use SpringDoc

  3. Use Doclet to scan through Controllers then output OpenAPI specification ourselves.

Below is the discussion regarding which option to use:

Springfox

Pros

  • Compatible with non-spring boot applications

Cons

  • Not really being maintained anymore, this can be seen by the number of github issues that it has and that the last commit was in 2020.

    • Conversations on migrating from springfox to springboot here.)

  • For inheritance to work, annotations need to be added to parent class of all subclasses. See github conversation about this here.

  • If we do face an issue (which is sure to happen), it will be hard to create a ticket for it and get support to fix it as there is no active maintenance.

Springdoc

Pros

  • Actively being maintained and little issues have been reported

Cons

  • It is natively meant to integrate with spring-boot applications (which synapse is not). This means more work needs to be done in order to make it compatible with synapse. Here is a section on this in their FAQ.

    • Discussion on github regarding this subject here.

  • Very hard to tell if all of it will work with our non-spring boot application. What if we get 90 percent of the way there but the last 10 percent is just unfeasible?

Use Doclet and generate it ourselves (chosen)

Pros

  • We will have full control of what the generated spec will look like and will not need to depend on third party libraries

  • Going straight from annotations to the specification is much quicker than having to write a program to create the annotations and then move it to the OpenAPI specification.

Cons

  • Might be more potential work since we have to parse through all of the javadocs in the controllers ourselves

Specific Steps

The workflow for using Doclet to parse through the controllers to generating the OpenAPI specification can be seen below.

Therefore, in the current process the steps being taken are (all code can be found under the lib-openapi package of the Synapse-Repository-Services repository):

  1. Create an OpenAPI model that closely resembles what a valid OpenAPI specification should look like

    1. See here for the OpenAPI specification

    2. We decided to structure the OpenAPI model in such a way that when converted into a JsonObject through the gson libarary, it will conform to the structure of the openAPI spec.

  2. Create a ControllerModel that stores information regarding a controller and its methods.

    1. Use Doclet to read through a controller and gather information needed for the OpenAPI Model

    2. We decided to first start out by getting the functionality working for primitive parameters and responses (int, String, boolean)

  3. Create a translator that translates multiple of these ControllerModels into a single OpenAPIModel

    1. With this finished, we are able to translate controllers with primitive types into the correct OpenAPIModel

  4. Add onto ControllerModel so that it is able to handle complex types appropriately

    1. When a method returns a class or an interface, we need to define the concreteTypeof what is being returned.

      1. In the OpenAPI specification, this is done through a discrimintator in addition to adding the appropriate allOf or oneOf attributes. See the specification here for more.

  5. Make sure that current comments for controllers and methods are displayed correctly and can link to the right portion of the Swagger-UI documentation

    1. Currently the Synapse Rest API documentation has links that appear in the controller/method descriptions which link to other areas of the specification. We need to make sure that this feature works correctly in Swagger-UI.

  6. Test the client that is being generated/explore corner cases.

Work moving forward

As of 6/30/2023, work on generating the OpenAPI specification will stop temporarily as that is when Leon’s internship ends. While we have made significant progress in generating the specification, there is more work that needs to be done. The purpose of this section of the document is to outline the current state of the project and what needs to be done moving forward.

For the majority of the internship, we focused on generating a valid specification for a few example controllers that we manually created along with a few example objects (you can find those controllers here and the example objects here). While these controllers and classes do share many features with the actual controllers, there are a lot of cases that aren't covered. Below describes the current state of the components generation and endpoint generation of the project.

Currently we are able to go through all of the concrete classes that are used in the actual synapse endpoint and are thus populating the components section of the specification correctly (this is where all the schemas for the objects we use exist). While we are able to parse through all of these concrete classes, we won’t know if they are correctly represented until we actually test the client at the very end of the project.

The majority of the work moving forward in regards to generating a working client involves fixing all of the errors that occur when we parse through the controllers. Since there are around 50 controllers and ~500 endpoints in total, we don’t know the exact number of issues that might appear (tons of edge cases possible). Due to this, all we can do is solve one issue at a time as they come up when attempting to generate the OpenAPI specification.

To resume where we left off (fixing the errors one at a time when parsing through the controllers), use the below workflow to generate the OpenAPI specification.

Workflow

We parse through the controllers using the ControllerModelDoclet that we created, which is from the lib-openapi package. We added a plugin for this in the services-repository pom.xml file.

To generate the specification:

  1. Currently the ControllerModelDoclet is disabled by default since it will cause the build to break as there are unresolved errors. Therefore, the first step is to enable the doclet by setting the --should-run parameter to true in the services-repository pom.xml.

  2. In the services/repository directory, run mvn clean install -Dmaven.test.skip=true -e, which will build the package its dependencies (one of which is our doclet)

    1. When doing this, an error will pop up if there are still issues processing all of the controllers.

    2. To aid the process of finding which controller and which endpoint is causing the current issue, I have printed the controller and method that the translator is parsing through at the moment:

      1. In the above example, we can see the error is stemming from the getFileHandleURL method in the UploadController controller

  3. Repeat step 2 until no more errors are generated when running the build

  4. Create a client using swagger, follow the guide here.

Work beyond client generation

The steps mentioned above will allow us to successfully generate a client that is able to interact with the SynapseAPI correctly. However, OpenAPI can be used for much more than that, such as generating the synapse api documentation through swagger-ui. To accomplish this task, we need to make sure that comments for each controller and endpoint are being represented correctly. More specifically, there is a problem with linking to another section of the api documentation from a comment.

Please consult @John Hill with further questions regarding this topic.