iOS Testing Needs

iOS Testing Needs

Broadly speaking, the testing needs for iOS fall into three general categories: unit tests, end-to-end tests, and UI tests.

Unit Testing

Unit testing is used extensively in AzureCore and other key hand-crafted libraries such as AzureIdentity and AzureCommunication (which facilitates auth for communication libraries). Depending on the authoring language, the library may have unit tests in Swift and ObjectiveC, both of which utilize the XCTest framework. Unit testing is not heavily used for testing service client libraries because they are generated by Autorest, and Autorest code is generated to pass the Autorest test server test cases. Unit testing capability in iOS is currently sufficient to meet our needs.

End-to-End Testing

All services across languages do extensive end-to-end testing of their services through the creation of “scenario tests” which sequence a variety of service calls to test some scenario. Most languages make use of a recording-playback framework such that a scenario test can be run against a live service and the HTTP responses are saved. The test can then be replayed without a network connection as the saved HTTP responses essentially mock the network response. This type of testing has the following benefits:

• When run live, this tests the entire SDK pipeline as well as the underlying service, detecting errors anywhere in the chain. These tests also tend to mirror the way developers would manually test, either in an interactive REPL or test script. Many languages run scenario tests live on a nightly basis to detect problems with the service before customers do.
• When run in playback mode, scenario tests provide extensive coverage of the client code much more realistically and efficiently than would be possible by trying to unit test the client, without using actual networking or Azure resources. Scenario tests are typically run in playback mode with every pull request to ensure that no changes to the SDK unintentionally break the end-user contract. End-to-end testing capability in iOS is currently insufficient. There are no popular record-playback libraries available for Swift, which means we will need to write one. The kinds of features this library would need include:
  • ability to run tests in live, record, and playback modes
  • ability to scrub sensitive information from recordings and still play back successfully
  • ability to spin up Azure resources for purposes of running live and subsequently tear them down once the test has completed. In Python, this is accomplished using “Preparers”.

UI Testing

We anticipate that UI tests will be used sparingly by select services. It should only be needed by those services which release customized UI components that need to “just work” on a variety of devices and configurations. It should not be needed to test client API calls. This type of test requires a test or demo app as well as tests written using iOS’s XCUITests framework. Several services already have demo apps that they use as samples or for manual testing, and it should be fairly simple to extend these to facilitate UI testing on a range of devices with AppCenter. One complicating factor is that such tests must be registered with AppCenter and may need to be signed with a development certificate to run on actual devices. That investigation is on-going, but we evaluate our UI testing capability to insufficient. AppCenter does not provide network mocking, which means the efficacy of UI Testing will be limited by our lack of a record-playback framework as it would not be practical or efficient to run tests live on multiple devices concurrently. Running a single scenario test live usually involves the creation of a resource group and one or more resources. When large test suites run concurrently, we already risk hitting resource limits. Multiplying each test by the number of platforms being tested would exacerbate that problem.

Platforms

While our team is currently focused on Swift SDKs, because we rely on XCTest and XCUITest, these test frameworks work for both languages. ObjC-based libraries also lack record-playback mechanisms, so either writing the record-playback framework in ObjectiveC or with @objc should be strongly considered.