iOS CI Setup

Overview

This is documentation on how the iOS continuous integration is setup.  The purpose of this is to keep a record of our specific setup and as a guide to debugging problems when things are broken.  It's provides as a nice guide to setting up CI for new iOS projects.  Note that references to secrets can be found in Lastpass (Secrets for iOS CI setup).

Repositories

These are the repositories that were created during the CI setup.  Travis treats public and private repos differently so both types were created to test builds on Travis.

• ios-certificates - repository that contains all the shared certificates and profiles for ios projects.  Since we have setup multiple teams on developer.apple.com we need to create equivalent branches in the repo to manage them.  Certs and profiles are kept in branches.
• sandboxpriv - a private sandbox repository for testing.  Test projects are in branches.
• sandboxpub - a public sandbox repository for testing.

References:

https://github.com/fastlane/fastlane/tree/master/match#important-use-one-git-branch-per-team

Requirements

These are some of the tools required to setup this CI.  You will need to install these.

• ruby (install with brew)
• rbenv (optional) - helps manage ruby versions
• Fastlane . (install with gem not brew)
• Travis client
• Amazon aws client
• openssl (cannot be LibreSSL). If your mac has LibreSSL I would recommend running a ubuntu linux docker instance and install openssl (instructions below)

Setup Code Signing

We use fastlane match to manage osx certificates and profiles.  The idea behind match is to setup shared certificates and profiles that an entire time can use to generate iOS builds.  Developers run fastlane to download the shared certs and profiles but match manages them and fetches them for the developers.  An private ios-certificates repo was created to store our shared certificates and profiles.  The credentials are in branches identified by the team id.

Note that fastlane match's "enterprise" type does not work correctly.  The only way to make it work is to create an enterprise profile but use the "appstore" type.  Look at dummyios project to see an example of how to do that.

General

Read Common build mistakes you've probably made and try the codesign doc tool on your project

Apple Portal

Login to apple portal and view the certs.  An alternative (and better) way to do that is to use the get_certs.sh script.  This script provides more relevant information than what can be seen from the web interface such as the Cert Id.

❯❯❯ ruby scripts/get_certs.rb appleservicer@sagebase.org
Multiple teams found on the Developer Portal, please enter the number of the team you want to use:
1) 4B822CZK9N "Sage Bionetworks" (In-House)
2) KA9Z8R6M6K "Sage Bionetworks, a Not-For-Profit Research Organization" (Company/Organization)
2
Cert id: LV2SM85KUC, name: iOS Development, expires: 2018-07-12, type: Development
Cert id: X857B7UB43, name: iOS Distribution, expires: 2018-08-15, type: Production
Cert id: 8U984RC44K, name: iOS Development, expires: 2018-10-18, type: Development
Cert id: 6YAV285S55, name: Apple Push Services, expires: 2018-12-27, type: ProductionPush
Cert id: 9XD88BXDQ2, name: iOS Development, expires: 2018-12-08, type: Development

Also the portal's web UI limits names to 50 chars when adding App ID and Provisioning profiles.  The limit only applies when adding from the web browser, not when using the API. To get around this use fastlane produce and sign.

 Create a Certificate:
   bundle exec fastlane cert --development -o ./CERT_OUT -u appleservicer@sagebase.org -b KA9Z8R6M6K --platform ios

 Create an App ID:
  bundle exec fastlane produce -a org.sagebase.bridge.JourneyPRO.watchkitapp.watchkitextension -u appleservicer@sagebase.org -q "XC org sagebase bridge JourneyPRO watchkitapp watckitextension" -b KA9Z8R6M6K -j ios -i

 Create a provisioning profile:
  bundle exec fastlane sigh -a org.sagebase.bridge.JourneyPRO.watchkitapp.watchkitextension -u appleservicer@sagebase.org -b KA9Z8R6M6K -n "match Distribution org.sagebase.bridge.JourneyPRO.watchkitapp.watchkitextension" -i 8U984RC44K -p ios

Important: The App IDs must conform to apple convention of "XC com abc xyz", Ref: https://stackoverflow.com/a/36693674/1094247

Xcode

Ensure the schemes of your target in xcode are Shared. This option makes a scheme visible from command line builds. To enable it, go to the menu: Product > Scheme > Manage Schemes > check the "shared" checkbox.  You can verify that's it's visible by running this on command line: "xcodebuild -list -project ./BridgeAppSDK.xcodeproj"

Enable agvtool in xcode so CI builds can automatically update verion and build numbers

    Reference: https://developer.apple.com/library/content/qa/qa1827/_index.html 

Fastlane

Installing fastlane. It is recommended to install using 'gem install fastlane' instead of using brew. I've noticed that fastlane plugins will not work if you install using brew.

Travis ENV

Setup the following ENV vars on Travis:

• FASTLANE_EXPLICIT_OPEN_SIMULATOR = 1
• CI_USER_TOKEN = <github access token for tcisagebio>
• MATCH_PASSWORD = <password used to create and unlock keychain and access ios-certificates repo>
• FASTLANE_PASSWORD = <password used to login to https://itunesconnect.apple.com>

**NOTE**
Encrypted environment variables (secrets) are not available from PR builds.  This means that we cannot run any builds or tests if it requires encrypted information.

References:
https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
https://github.com/travis-ci/travis-ci/issues/1946
https://groups.google.com/forum/#!topic/sonarqube/5U1h5ooq_GM
https://github.com/pockethub/PocketHub/issues/884

Travis SSH Key

Part of the deployment process is to commit a tag to the git repo.  In order for travis to do that a SSH key will need to be setup on travis.  I typically just upload the travis user "tcisagebio" ssh key which can be found in lastpass.  Copy/Paste the private key to a "id_travis_rsa" file and do the following:

# from https://github.com/travis-ci/travis-ci/issues/8680
# In your local terminal
> cd path/to/your/local/gitrepo
# login by your account --pro or --org
> travis login --pro
# add the ssh key to travis
> travis sshkey --upload id_travis_rsa --repo Sage-Bionetworks/BloodPressureApp-Android --description travis

For Development

Take a look at the match instructions on how to create new development profiles and certificates.

1. cd into project root folder
2. run `fastlane init` to create a fastlane project
3. run `fastlane development` to create development profiles and certificates
4. for the project.
5. add a fastlane/Matchfile which should look something like this:

git_url "https://github.com/Sage-Bionetworks/ios-certificates.git"

git_branch "KA9Z8R6M6K"

type "development" # The default type, can be: appstore, adhoc, enterprise or development

# The bundle identifier of your app
app_identifier ["org.sagebase.BridgeAppSDK", "org.sagebase.BridgeAppSDKSample", "org.sagebase.BridgeAppSDKSample.watchkitapp", "org.sagebase.BridgeAppSDKSample.watchkitapp.watchkitextension"]
username "appleservicer@sagebase.org" # Your Apple Developer Portal username

readonly true

NOTE - An archive build requires all 

For Distribution

To codesign for distribution you need to have all things coordinated. This means xcode settings, certificates, profiles and Travis.

Since we have existing certs and profiles we can't just create new ones so we need to import existing certs and profiles to ios-certificate repo. Note, when match attempts to create new distribution certificates apple portal complains that we have reached the max limit of certs and fails.

References:

https://github.com/fastlane/fastlane/issues/5765

Make Sure Everyone Will Use the Same Provisioning Profile.

NOTE: This is literally the most difficult part to understand and get correct.  IMPORTANT, an "In House" profile is the same as "Enterprise" profile.

Reference: https://stackoverflow.com/questions/14858266/what-is-the-difference-between-in-house-versus-ad-hoc-distribution-for-enterpris

The trick is to find the minimal set of certs and profiles that are need to build the project.

a. export all your osx/ios certificates from your keychain
b. save off all all your existing provisioning profiles
c. delete all ios certificates from your keychain
d. delete all provisioning profiles from your computer
e. open project in xcode
f. view all certificates with ruby script: http://macoscope.com/blog/simplify-your-life-with-fastlane-match/#migration
g. import only the certificates and provisioning profiles that are need to allow you to successfully archive and export
the build.
h. Migrate your cerfiticates and profiles to ios-certificates repo. This step imports the certificate in from your keychain to the the ios-certificates repo.  (full instructions in link on step f).

Start your Keychain app (it's keychain access on MAC)

Find the certificate and then export it, both Personal Information exchange (certificate.p12) and Certificate (certificate.cert) files.

Note: make sure to select both certificate and private key when exporting.

Now we need to prepare the cert and p12 files for our Sage-Bionetworks/ios-certificates repository.

openssl pkcs12 -nocerts -nodes -out key.pem -in certificates.p12 -password pass:$MATCH_PASSWORD

This step will extract the private key to a key.pem file.

*Note - some versions of openssl is incompatible with fastlane: https://github.com/fastlane/fastlane/issues/13242

Now encrypt the certificate files:

openssl aes-256-cbc -k $MATCH_PASSWORD -in key.pem -out $CERT_ID.p12 -a
openssl aes-256-cbc -k $MATCH_PASSWORD -in certificates.cer -out $CERT_ID.cer -a

*Note - The $CERT_ID can be found using the ruby script.  It should be something like QNY9FZ8GWL

Repeat if needed for the second type of certificate.  If you want to verify the certificates you created you can manually decrypt them and import them into a keychain.

copy $CERT_ID.cer and $CERT_ID.p12 file to ios-certificates/certs/distribution folder then commit the them to the repo.

i.  Migrate your profiles to ios-certificates repo.  Do this step to import profiles from the apple developer site to the ios-certificates repo.

Go into the provisioning profiles directory view the profiles:
/Users/$USER/Library/MobileDevice/Provisioning Profiles

grep -a -A1 "<key>Name</key>" *.mobileprovision | grep -v "<key>Name</key>" | grep -v "^--"

If you need more than a name, for example list of UDIDs, this command shows all content of particular provisioning profile:

security cms -D -i xxx.mobileprovision

https://stackoverflow.com/questions/922695/removing-provisioning-profile-from-xcode

To export and encrypt the profile do this:

openssl aes-256-cbc -k MATCH_PASSWORD \
-in "/Users/$USER/Library/MobileDevice/Provisioning Profiles/af2252dc-9e0e-4a9b-aa43-5ed3e8462d9c.mobileprovision" \
-out AppStore_org.sagebase.BridgeAppSDKSample.mobileprovision -a

*Note - fastlane will use the MATCH_PASSWORD to decrypt the profile.

Copy the "AppStore_org.sagebase.BridgeAppSDKSample.mobileprovision" to  ios-certificates/profiles/appstore

Now push the changes to ios-certificates repo.
*Note - all files must being with either 'Development_' or 'AppStore_'. I was not able to get the 'InHouse_' nor
'Enterprise_' to work.

References:

https://lyricsboy.github.io/2016/11/30/xcode-8-automatic-signing-ci/

Make sure the xcode project's option to 'automatic code signing' is disabled otherwise the build may fail with message:
❌ Code signing is required for product type 'Application' in SDK 'iOS 10.2'

References:

https://docs.fastlane.tools/codesigning/xcode-project/#setting-up-your-xcode-project
https://docs.fastlane.tools/codesigning/getting-started/#automatic-manual-signing
http://macoscope.com/blog/simplify-your-life-with-fastlane-match/#migration

https://github.com/fastlane/fastlane/issues/6519#issuecomment-253426674

If possible, setup fastlane to switch between manual and automatic code signing. I was not able to make this work
https://github.com/fastlane/fastlane/issues/8567#issuecomment-287299421
https://github.com/fastlane/fastlane/issues/6533#issuecomment-253746551
https://github.com/fastlane/fastlane/issues/6832#issuecomment-257883114

S3 Deployment

You can setup deployment to s3 buckets in .travis.yml or in fastlane. I tried using the fastlane plugin but it didn't work so I deployed in .travis.yml
https://docs.travis-ci.com/user/deployment/s3/
https://github.com/fsaragoca/fastlane-plugin-s3_actions

Run this command where .travis.yml is located:  travis encrypt --add deploy.secret_access_key

NOTE - The secret_access_key is unique to a repository, it cannot be reused across repos.

App Store Deployment

Before travis can automatically deploy to the app store you must manually add/create the app. You will also need to add the required images for your app and set the privacy policy.  Once you have your app setup on the app store travis can deploy new builds as updates. 

Workflow

The general workflow is to create stable branches for releases to testflight or the appstore.  Master is used as the development branch and stable-x branches are used for releases.  These are  the actions when a PR or commit occurs:

pre-merge (or PRs): build and run tests only

post-merge to master: generate an `enterprise` archive build and deploy to s3 bucket

post-merge to stable-x branch: bump build #, generate an `app-store` archive build, deploy to testflight, commit the new build number to the branch.   The build then may get promoted to the appstore.

Reference: https://github.com/fastlane/fastlane/issues/9052 

Setup Openssl on docker

1. run ubuntu docker instance (docker run -it --rm -v /tmp:/tmp ubuntu /bin/bash)
2. install openssl (sudo apt-get update -y && sudo apt-get install -y openssl)
3. run "openssl version".  It should be "OpenSSL 1.0.1f 6 Jan 2014".  Just make sure you don't get LibreSSL

Debugging

This is a collection of issues I ran into and wanted to keep track of.  This info may help in debugging problems with integration of xcode, fastlane and travis. 

Provisioning Profile location and how to view profiles
https://stackoverflow.com/questions/922695/removing-provisioning-profile-from-xcode

Hanging builds may be caused by the keystore prompting (with a dialgo) asking permission to allow codesign to access the keychain.  The message is something like "codesign wants to access key "key name" in your keychain. Do you want to allow access to this item?".  If build hangs, it's very likely that the keychain is locked. Travis CI is aware of this and has instructions on how to bypass the prompt.   In order to disable this dialog you need to run:

security set-key-partition-list -S apple-tool:,apple: -k <keychain password> ~/Library/Keychains/ios-build.keychain-db

Reference: https://openradar.appspot.com/28524119

Example of how to setup custom keychains:

security create-keychain -p <keychain password> ~/Library/Keychains/ios-build.keychain-db
security list-keychains -s ~/Library/Keychains/ios-build.keychain-db
security default-keychain -s ~/Library/Keychains/ios-build.keychain-db
security unlock-keychain -p <keychain password> ~/Library/Keychains/ios-build.keychain-db
security import Certificate.cer -k ~/Library/Keychains/ios-build.keychain-db -T /usr/bin/codesign -P <keychain password>
security import Certificate.p12 -k ~/Library/Keychains/ios-build.keychain-db -T /usr/bin/codesign -P <cert password>
security set-key-partition-list -S apple-tool:,apple: -k <keychain password> ~/Library/Keychains/ios-build.keychain-db

This also shows how to setup keychains without fastlane: https://stackoverflow.com/questions/16550594/jenkins-xcode-build-works-codesign-fails

People have used the gym "build legacy_build_api" option to fix things.  I've tried and it seems to work in places however it will create a ipa that is 4 times larger in size so so not a good option.

Issues with openssl: https://github.com/fastlane/fastlane/issues/6699#issuecomment-257743867

more info about how code signing works in xcode: http://devcenter.bitrise.io/ios/code-signing-technical-details/

create_keychain action overwrites existing login.keychain: https://github.com/fastlane/fastlane/issues/8790

use security command to view keychain info

security list-keychains
security find-identity -v -p codesigning

run ruby script to list all certificates from apple portal: http://macoscope.com/blog/simplify-your-life-with-fastlane-match/#migration

If you cannot upload to S3 bucket make sure policy is setup correctly.

https://docs.travis-ci.com/user/deployment/s3/#S3-ACL-with-bucket-policy

https://github.com/travis-ci/dpl/issues/237

Install multiple versions of xcode to test on local machine

http://iosdevelopertips.com/xcode/install-multiple-versions-of-xcode.html

https://github.com/schwa/xcode-toggle

Xcode 8.3 Error: "Unable to read diagnostics from file"

https://github.com/fastlane/fastlane/issues/8698

https://github.com/JohnEstropia/CoreStore/issues/113

Use aws command line to list files on s3 buckets

aws s3 ls s3://ios-apps.sagebridge.org/org.sagebase.BridgeSDK/Documentation/latest

Run fastlane from command line

fastlane match development --readonly --git_url https://github.com/Sage-Bionetworks/ios-certificates.git --git_branch "KA9Z8R6M6K"

fastlane gym --scheme "Lilly" --clean --output_directory ./build --configuration Debug --include_bitcode true --include_symbols false --export_method enterprise

Run this to show all xcode build settings

xcodebuild clean -showBuildSettings -scheme BridgeAppSDKSample -project ./BridgeAppSDK.xcodeproj

Initialize the simulator before running tests otherwise the tests may fail with message: ** TEST FAILED ** Exit status: 65

FASTLANE_EXPLICIT_OPEN_SIMULATOR=2  bundle exec fastlane test

Reference: https://github.com/fastlane/fastlane/issues/5375

Get this error:  undefined method `fetch' for nil:NilClass 

       Reference: https://github.com/fastlane/fastlane/issues/5643#issuecomment-238440668

See dummyios project to view .gitignore for fastlane and bundle.  Those need to be set otherwise versioning thru travis will not work.

You may get this error if you are using an invalid version of XCode to build the app:

  ❌ fatal error: unexpectedly found nil while unwrapping an Optional value

Fastlane match may fail to import the private key into the keychain, "Unknown format in import", which could be an OpenSSL problem.  Note: fastlane may silently fail when this happens.

     Reference: https://github.com/fastlane/fastlane/issues/13242

Related articles

• Page:
  Bridge Server CI Setup