Versions Compared

Version Old Version 80 New Version Current
Changes made by

Jay Hodgson

Nick Grosenbacher

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Clarify URL of app in development, add commented dev stack property to settings.xml example
Table of Contents
minLevel3
maxLevel6
outlinefalse
typelist
separatorbrackets
printabletrue
Child pages

UI Development

Prerequisite software to install

  • Java (Corretto JDK 11+ recommended)

  • Maven 3

  • Docker

  • pnpm v9

Set up

...

your

...

GitHub fork and a local repository

  1. Create a GitHub user account if you don't already have one.

  2. Set up your local git environment according to /wiki/spaces/IT/pages/802881544

  3. Fork the Sage-Bionetworks SynapseWebClient repository into your own GitHub account: https://help.github.com/articles/fork-a-repo

  4. Clone

...

  1. your fork of SynapseWebClient project

...

  1. to your

...

  1. local machine and enter the project root directory

    Code Block
    languagebash
    git clone https://github.com/[YOUR

...

  1.  GITHUB NAME]/SynapseWebClient.git

...

Set up the Eclipse Build

  1. Install Eclipse (Mars 4.5, Neon 4.6 GWT super dev mode does not start properly)
  2. Install Google Web Toolkit, Google App Engine SDK, and Google plug-in for Eclipse using Eclipse new software from http://www.gwtproject.org/download.html.  Note: if you were using an older version of this plugin you will need to uninstall it and then re-install it to ensure it has  GWT 2.8 beta1 support. Another note: Installing the Google App Engine Maven Integration software made Eclipse error when building the SynapseWebClient. It would not work until I uninstalled it.
  3. Install the Maven2Eclipse plugin: http://download.eclipse.org/technology/m2e/releases
  4. Create a GitHub user account
    1. Contributors should fork the repository and submit GitHub Pull Requests for code inclusion.

  5. Set up Git: https://help.github.com/articles/set-up-git

  6. Import the project as a maven project
    1. File → Import → Maven → Existing Maven Projects
    2. Next, in Root Directory enter the local path to your repository clone
    3. Next, Finish 
  7. You need to make sure the GWT xml files are on the classpath. All of these resources can be found in src/main/resources and src/test/resources directories. 
    1. Right-click on portal (project name) → Build Path → Configure Build Path...
    2. Remove the "Exclude" filter from the source tab on the src/main/resources and src/test/resources directories.
  8. We need to tell the GWT Eclipse-plugin that this is a GWT project. Do this by Right-Clicking on the project in the package explorer and select: Google → Web Toolkit Settings...
    1. From the resulting dialog make sure the "Use Google Web Tookit" check box is selected.
    2. Also make sure you are using GWT 2.8 beta1
  9. We need to tell the Google plugin where our web app directory can be found.  Right-click on the project in the package explorer and select Properties.
    1.  From the tree on the left of the dialog navigate to Google → Web Application
    2. Check the the check box: "This project has a WAR directory"
    3. With the "Browse" button, select "src/main/webapp"
    4. UN-CHECK the "Launch and deploy from this directory"  This is very important, if you keep this checked then Maven will not be able to generate a clean WAR file.  If you see "GWT needs to recompile" when you deploy your WAR then you probably have this box checked.
  10. Now make sure GWT can compile your code
    1. Right-Click on the project from the package explorer.
    2. Select Google→GWT Compile...
    3. Under the "Entry Point Modules" you should see "Portal - org.sagebionetworks.web",  if not, then add it with the add button.
    4. The first time you run this you will be asked to select the output directory where GWT will compile the code. You want this to match the Maven WAR output directory, so use "target/portal-<VERSION>-SNAPSHOT"
    5. If you get compilation errors from the JavaScript validator (i.e. NullPointerException), navigate to the project's properties → JavaScript → Include Path → Source, and exclude all files from the source. If an OutOfMemoryError was thrown while compiling, you can click the "Advanced" tab towards the bottom of the Google → GWT Compile window and add "-Xms512M -Xmx1524M" to the VM arguments to increase heap space.
  11. If the GWT Compile successfully compiled the 5 permutations, then you're ready to run the application.  If your local hardware resources are insufficient to build, try limiting the permutations by editing your Portal.gwt.xml (set <set-property name="user.agent" value="safari"/>)
    1. Option 1: start from within Eclipse.
      1. Right-Click on the project in the package explorer
      2. Select: Run As → Web Application (GWT Super Dev Mode)
      3. Set the run configuration VM Arguments to the following: "-XX:MaxPermSize=512m -Xms512m -Xmx2048m -XstartOnFirstThread".  Also look at your eclipse.ini file (if launching SDM from Eclipse).
      4. Double-click on the link provided in the Development Mode window to view the portal in your browser.  If you get asked what page to start on choose Portal.html.
    2. Option 2: build and run from the command line.
      1. Run 'mvn clean install'
      2. Run 'mvn gwt:run'

Eclipse Workspace Settings

Set file encoding to UTF-8 and the line-endings (for new files) to Unix, so that text files are saved in a format that is not specific to the Windows or Mac OS and most easily shared across heterogeneous developer desktops:

  • Open Preferences
  • Navigate to the Workspace preferences: General → Workspace
  • Change the Text File Encoding to UTF-8
  • Change the New Text File Line Delimiter to Other and select Unix from the drop-down

Troubleshooting

  • Check your exclusion filters through configure build path in Eclipse to assure nothing is excluded, then refresh and clean the SWC project.
  • "Could not deserialize" or similar problems in Super Dev Mode are most often fixed by re-building through the command line (running 'mvn clean install')
  • (git) Pull the latest changes from the upstream repository.
  • Try to rebuild your Eclipse's Portal project and keep your files in place.
  • Perform a Maven update.  
  1. Delete your project from Eclipse (leave "Delete project contents on disk" UNCHECKED)
  2. Go to the project root on your file system and delete .classpath, .project and the .settings/ folder
  3. Go to Step 6 in "UI Development above"

If this still doesn't work, or your dev mode is broken for another reason, there are some further steps you can take.
(Check out this site for more information: http://www.eclipsezone.com/eclipse/forums/t61566.html)

  1. Use the '-clean' argument.
    1. One way to do this is to add -clean on its own line as the first line on the eclipse.ini file, and restart Eclipse.
  2. Use a new workspace.
    1. I followed the instructions in the site linked above. The gist of it is:
      1. Export your preferences to somewhere outside your Eclipse environment environment (Export → General → Preferences. Make sure that 'Export All' is checked.)
      2. Switch to a new workspace. (Switch Workspace → Other, then name your new workspace.)
      3. Import your preferences into this new workspace (Import → General → Preferences, make sure 'Import All' is checked.)
      4. Import trunk as usual.

Technologies used

There are several technologies that we are currently utilizing in the portal. The following is a list of primers for each:

Cross-browser testing

We have a shared account to use BrowserStack to test the website for cross browser compatibility issues.
Alternatively, for testing IE on Mac I use VirtualBox with an image provided by Microsoft from modern.IE.  For this to work, in your Windows VM, you will need to map an ip to outer in order to access your local machine (note, Windows UAC requires you to do this as an admin).

How to limit GWT build permutations

To speed up build time, add the following property to your local maven settings.xml file:

<gwt.module.suffix>Debug</gwt.module.suffix>

How to point to the local stack

...

Build and launch a local backend stack.  From an up-to-date clone of Synapse-Repository-Services on your local box:

...

Build the local stack by running 'mvn clean install -Dorg.sagebionetworks.database.drop.schema=true'.

...

  1. cd integration-test
  2. mvn cargo:run

Verify the services are running correctly by visiting http://localhost:8080/services-repository-develop-SNAPSHOT/repo/v1/version you should see something like this:

Code Block
{"version":"develop-SNAPSHOT"}

...

<org.sagebionetworks.authenticationservice.privateendpoint>http://localhost:8080/services-repository-develop-SNAPSHOT/auth/v1</org.sagebionetworks.authenticationservice.privateendpoint>
<org.sagebionetworks.authenticationservice.publicendpoint>http://localhost:8080/services-repository-develop-SNAPSHOT/auth/v1</org.sagebionetworks.authenticationservice.publicendpoint>
<org.sagebionetworks.repositoryservice.endpoint>http://localhost:8080/services-repository-develop-SNAPSHOT/repo/v1</org.sagebionetworks.repositoryservice.endpoint>
<org.sagebionetworks.fileservice.endpoint>http://localhost:8080/services-repository-develop-SNAPSHOT/file/v1</org.sagebionetworks.fileservice.endpoint>
<org.sagebionetworks.searchservice.endpoint>http://localhost:8080/services-repository-develop-SNAPSHOT/</org.sagebionetworks.searchservice.endpoint>
<org.sagebionetworks.portal.endpoint>http://127.0.0.1:8888/Portal.html</org.sagebionetworks.portal.endpoint>

Note that emails will not be sent when pointing to a local repo, the SynapseEmailService will instead write files to the devdata.sagebase.org s3 bucket in format <to_email_address>.json

Restart the Portal app to load these properties from settings.xml.

How to point to staging

Set the repository endpoints in your maven settings.xml file by adding the following properties:

<org.sagebionetworks.authenticationservice.privateendpoint>https://auth-staging.prod.sagebase.org/auth/v1</org.sagebionetworks.authenticationservice.privateendpoint>
<org.sagebionetworks.authenticationservice.publicendpoint>https://auth-staging.prod.sagebase.org/auth/v1</org.sagebionetworks.authenticationservice.publicendpoint>
<org.sagebionetworks.repositoryservice.endpoint>https://repo-staging.prod.sagebase.org/repo/v1</org.sagebionetworks.repositoryservice.endpoint>
<org.sagebionetworks.fileservice.endpoint>https://file-staging.prod.sagebase.org/file/v1</org.sagebionetworks.fileservice.endpoint>
<org.sagebionetworks.searchservice.endpoint>https://search-staging.prod.sagebase.org/</org.sagebionetworks.searchservice.endpoint>
<org.sagebionetworks.portal.endpoint>http://127.0.0.1:8888/Portal.html</org.sagebionetworks.portal.endpoint>

Restart the Portal app to load these properties from settings.xml.

How to point to prod

Set the repository endpoints in your maven settings.xml file by adding the following properties:

...

  1. 
cd SynapseWebClient

  2. If you didn't set up the global pre-commit hook in step #2, set up a local pre-commit hook to detect secrets (do this for all repos that you clone in the future!):

    Code Block
    git secrets --install
git secrets --register-aws

  3. Set up upstream with

    Code Block
    git remote add upstream https://github.com/Sage-Bionetworks/SynapseWebClient

  4. Fetch and merge changes from the Sage Bionetworks repo, which was named upstream: 

    Code Block
    git fetch upstream

  5. Configure or update your Maven configuration settings.xml

    1. The file needs to placed in $HOME/.m2/settings.xml (e.g. jane/.m2/settings.xml)

    2. This should be the content of this file:

      settings.xml

      Code Block
      languagexml
      <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                      http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <localRepository/>
  <interactiveMode/>
  <usePluginRegistry/>
  <offline/>
  <pluginGroups/>
  <servers/>
  <mirrors/>
  <proxies/>
  <profiles>
    <profile>
      <id>dev-environment</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
      <properties>
		<org.sagebionetworks.portal.endpoint>http://127.0.0.1:8888/Portal.html</org.sagebionetworks.portal.endpoint>
        <org.sagebionetworks.repositoryservice.endpoint>https://repo-prod.prod.sagebase.org/repo/v1</org.sagebionetworks.repositoryservice.endpoint>
        <!-- insert properties described below here -->
        <!-- If you want to point to a different stack (staging, dev), comment out the above and uncomment one of the below -->
        <!-- <org.sagebionetworks.repositoryservice.endpoint>https://repo-staging.prod.sagebase.org/repo/v1</org.sagebionetworks.repositoryservice.endpoint> -->
        <!-- <org.sagebionetworks.repositoryservice.endpoint>https://repo-dev.dev.sagebase.org/repo/v1</org.sagebionetworks.repositoryservice.endpoint> -->
        <!-- Uncomment the property below to limit GWT permutations and speed up build time -->
        <!-- <gwt.module.suffix>Debug</gwt.module.suffix> -->
      </properties>
    </profile>
  </profiles>
  <activeProfiles/>
</settings>

  6. Verify you can build the project and run the unit tests:

    Code Block
    mvn clean install

Development

You may use an IDE of your choice. Instructions for setting up IntelliJ IDEA and Eclipse can be found below.

Running SynapseWebClient in development mode requires many different services, including the GWT Codeserver, a Vite codeserver, a Java servlet (Tomcat), and the Sage Bionetworks Account site (required to sign in via UI). For convenience, we maintain a few different ways to run SWC in development mode.

Expand
titleRunning all services in Docker containers (docker compose) [PREFERRED]

To run the app in development mode in a set of Docker containers, you can run pnpm start:<stack>, where stack is one of development,staging,production. This will invoke docker compose to instantiate a container for each service required to run the app in development mode. The latest version of the accounts site (SageAccountWeb) will also be launched on port 3000 as a convenience.

Expand
titleDevelopment mode using native Java/Maven installation

To run the app in development mode, run pnpm install then pnpm dev. This will launch three processes:

  • The GWT Codeserver, which compiles Java to JavaScript, and watches for source changes

  • The Vite codeserver, which bundles JavaScript assets and watches for source changes

  • A file watcher that monitors changes to static assets, and copies them to the deployed web application directory when they change

  • An Apache Tomcat Docker container that will host the web application

Note that this does NOT run the Sage account management application (SageAccountWeb), which is separate from SWC. If you want to log in through the UI, you must run SageAccountWeb on port 3000. You can either run SageAccountWeb in synapse-web-monorepo, or you can run the latest Docker image for SageAccountWeb.

Running the local version of SageAccountWeb

SynapseWebClient delegates authentication and account management to a separate web application. In order to sign in to a local instance of SynapseWebClient, you must also start a local instance of the account management application. Set up the synapse-web-monorepo project and run the following command.

Code Block
 pnpm nx run SageAccountWeb:start --port=3000 --host --mode=development

If your portal instance is configured to use to a different backend stack, you will need to configure SageAccountWeb to also use the same stack (i.e. --mode=production or --mode=staging).

Running the accounts site in a Docker image

You can run the latest version of the accounts site (synapse-web-monorepo/main) with the following command:

Code Block
docker run -it --rm -d -p 3000:80 --name SageAccountWeb ghcr.io/sage-bionetworks/sage-account-web:latest-development

If you want the site to point to staging/production, change development in the image tag to the stack you want to point to.

Alternative ways to run SynapseWebClient in development mode

Instead of running pnpm dev, you can alternatively directly use the GWT Maven plugin (e.g. mvn gwt:run) or an IDE integration. Please note that depending on your IDE/plugin configuration, your environment may be subtly different than the production system (e.g. many GWT plugins deploy a Jetty server, when we use Tomcat in production), which can result in subtle differences in behavior.

Once the service is started, you should be able to see the app at http://127.0.0.1:8888/ .

As you make changes to the source code and static assets, changes will be picked up in the app when you refresh the page.

JavaScript dependency development

If you have made changes to a JavaScript dependency (e.g. synapse-react-client), you can see those changes in your local development environment by “linking” the projects. As an example, we will link the synapse-react-client project.

  1. In the SynapseWebClient directory, run

    Code Block
    pnpm link ../path/to/synapse-web-monrepo/packages/synapse-react-client

    This will create a symlink on your machine so the node_modules/synapse-react-client directory points to your local synapse-react-client directory.

  2. Make desired changes in the synapse-react-client project

  3. Rebuild the JavaScript artifacts for synapse-react-client (in synapse-web-monorepo, run pnpm nx run synapse-react-client:build)

  4. If you’re running pnpm dev, the changed JavaScript file should automatically be picked up. Refresh your browser window to see the changes (you will likely need to open your browser dev tools → Network → Disable Cache, and leave the browser devtools open)

Info

If you’re using Docker Compose to run the development server, you must add your locally-linked package as a volume on the container. Instructions on how to do this are in compose-link.yml.

When you want to unlink synapse-react-client, in the SynapseWebClient directory you can run pnpm unlink synapse-react-client.

Note

Linking a package updates the lockfile to reference the local repository, so you must unlink the package before you commit changes to the lockfile.

IDE Setup Instructions

This section contains instructions for setting up various IDEs to develop SynapseWebClient.

Expand
titleSetup with Eclipse

  1. Install the latest Eclipse (for Java Development)

  2. Install the Google Plug-in for Eclipse.  Do not include GWT SDK in the install (GWT SDK will be installed by the maven install).  Do not install the Google App Engine Maven Integration.

  3. Import the project as a maven project

    1. File → Import → Maven → Existing Maven Projects

    2. Next, in Root Directory enter the local path to your repository clone

    3. Next, Finish 

  4. You need to make sure the GWT xml files are on the classpath. All of these resources can be found in src/main/resources and src/test/resources directories. 

    1. Right-click on portal (project name) → Build Path → Configure Build Path...

    2. Remove the "Exclude" filter from the source tab on the src/main/resources and src/test/resources directories.

  5. We need to tell the GWT Eclipse-plugin that this is a GWT project. Do this by Right-Clicking on the project in the package explorer and select: Google → Web Toolkit Settings...

    1. From the resulting dialog make sure the "Use Google Web Toolkit" check box is selected.

    2. Also make sure you are using the GWT SDK from your local maven repository (.m2/repository/com/google/gwt)

  6. Verify GWT can compile your code

    1. Right-Click on the project from the package explorer.

    2. Select Google→GWT Compile...

    3. Under the "Entry Point Modules" you should see "PortalDebug - org.sagebionetworks.web",  if not, then add it with the add button.

    4. Remove the "Portal - org.sagebionetworks.web" entry.

    5. The first time you run this you will be asked to select the output directory where GWT will compile the code. You want this to match the Maven WAR output directory, so use "target/portal-<VERSION>-SNAPSHOT"

    6. If you get compilation errors from the JavaScript validator (i.e. NullPointerException), navigate to the project's properties → JavaScript → Include Path → Source, and exclude all files from the source. If an OutOfMemoryError was thrown while compiling, you can click the "Advanced" tab towards the bottom of the Google → GWT Compile window and add "-Xms512M -Xmx1524M" to the VM arguments to increase heap space.

  7. If the GWT Compile successfully compiled, then you're ready to run the application.

    1. Right-Click on the project in the package explorer

    2. Select: Run As → GWT Development Mode with Jetty, and select Portal.html

    3. Double-click on the link provided in the Development Mode window to view the portal in your browser. 

  8. To run the unit tests:

    1. Right-Click on the project from the package explorer.

    2. Select: Run As → JUnit Test

Eclipse Workspace Settings verification:

Set file encoding to UTF-8 and the line-endings (for new files) to Unix, so that text files are saved in a format that is not specific to the Windows or Mac OS and most easily shared across heterogeneous developer desktops:

  • Open Preferences

  • Navigate to the Workspace preferences: General → Workspace

  • Change the Text File Encoding to UTF-8

  • Change the New Text File Line Delimiter to Other and select Unix from the drop-down

Troubleshooting

  • Check your exclusion filters through configure build path in Eclipse to assure nothing is excluded, then refresh and clean the SWC project.

  • (git) Pull the latest changes from the upstream repository.

  • Try to rebuild your Eclipse's Portal project and keep your files in place.

  • Perform a Maven update.  

  1. Delete your project from Eclipse (leave "Delete project contents on disk" UNCHECKED)

  2. Go to the project root on your file system and delete .classpath, .project and the .settings/ folder

  3. Go to "Set up with Eclipse"

Verify the web app directory is correct. Right-click on the project in the package explorer and select Properties:

  1.  From the tree on the left of the dialog navigate to Google → Web Application

  2. Check the the check box: "This project has a WAR directory"

  3. With the "Browse" button, select "src/main/webapp"

  4. UN-CHECK the "Launch and deploy from this directory"  This is very important, if you keep this checked then Maven will not be able to generate a clean WAR file.  If you see "GWT needs to recompile" when you deploy your WAR then you probably have this box checked

If this still doesn't work, or your dev mode is broken for another reason, there are some further steps you can take.

  1. Use the '-clean' argument.

    1. One way to do this is to add -clean on its own line as the first line on the eclipse.ini file, and restart Eclipse.

  2. Use a new workspace.

    1. I followed the instructions in the site linked above. The gist of it is:

      1. Export your preferences to somewhere outside your Eclipse environment environment (Export → General → Preferences. Make sure that 'Export All' is checked.)

      2. Switch to a new workspace. (Switch Workspace → Other, then name your new workspace.)

      3. Import your preferences into this new workspace (Import → General → Preferences, make sure 'Import All' is checked.)

      4. Import trunk as usual.

Expand
titleSetup with Intellij IDEA

  1. Download IntelliJ IDEA 

    1. The community edition is free

    2. The Ultimate edition requires purchasing a license. It includes enhanced support for GWT, but it’s not necessary for development. 

  2. Import the SynapseWebClient project into IntelliJ by File → New Project From Existing Sources... <Select Path to SWC Location>

  3. Set up IntelliJ to directly call pnpm dev

    1. Run → Edit Configurations…

    2. Click “Add New Configuration” → npm

    3. Set the following options

      • Command: run

      • Script: dev

      • Package manager: pnpm

Technologies used

There are several technologies that we are currently utilizing in the portal. The following is a list of primers for each:

End-to-end testing

Instructions on running end-to-end tests are included in this README.

How to point to a local instance of the Synapse backend

  1. Follow the instructions on the Platform Bootstrap to set up a local backend stack. 

  2. Build and launch a local backend stack.  From an up-to-date clone of Synapse-Repository-Services on your local box:

    1. Build the local stack by running mvn clean install -Dorg.sagebionetworks.database.drop.schema=true.

    2. Now launch it:

      Code Block
      cd integration-test
mvn cargo:run

    3. Verify the services are running correctly by visiting http://localhost:8080/services-repository-develop-SNAPSHOT/repo/v1/version. You should see something like this:

      Code Block
      {"version":"develop-SNAPSHOT"}

  3. In the SWC project root pom.xml, change the synapse.version tag to develop-SNAPSHOT, so that Maven utilizes artifacts that were built by running on your local stack.

  4. Update the repository endpoint property in your maven settings.xml file:

    Code Block
    <org.sagebionetworks.repositoryservice.endpoint>http://localhost:8080/services-repository-develop-SNAPSHOT/repo/v1</org.sagebionetworks.repositoryservice.endpoint>

  5. Restart the Portal app to load these properties from settings.xml.

Emails will not be sent when pointing to a local repo, the SynapseEmailService will instead write files to the devdata.sagebase.org s3 bucket in format <to_email_address>.json

How to point to a remote stack

Set the repository endpoint parameters in your maven settings.xml file. 

To point to the development stack set:

Code Block
<org.sagebionetworks.repositoryservice.endpoint>https://repo-dev.dev.sagebase.org/repo/v1</org.sagebionetworks.repositoryservice

...

.endpoint>

To point to staging set:

Code Block
<org.sagebionetworks.

...

repositoryservice.

...

endpoint>https://

...

repo-

...

staging.prod.sagebase.org/repo/

...

v1</org.sagebionetworks.

...

repositoryservice.endpoint>

To point to production set:

Code Block
<org.sagebionetworks.

...

repositoryservice.

...

endpoint>https:

...

/

...

/repo-prod.prod.sagebase.org/repo/v1</org.sagebionetworks.

...

repositoryservice.endpoint>

Restart the Portal app to load these properties this property from settings.xml.

Documenting complicated UI in the code

To get the big picture of widget relationships, sometimes it is helpful to have a picture.  We are testing the use of tools like httphttps://asciiflowmermaid.js.comorg/ to help in these situations.

...