Skip to content
This repository has been archived by the owner on Feb 2, 2021. It is now read-only.

Wishlist

Jump to bottom
Aleksey Kononov edited this page Apr 7, 2015 · 6 revisions

Here are some things we'd love to have in xctool:

Annotations that control how tests are run.

Sometimes we write tests that are only applicable to a certain iOS SDK version, or even a certain device type (e.g. the 3.5" vs 4" simulator). There's currently no way to express these preferences or make the test runner handle these preferences.

We'd like to be able to annotate our test code with extra attributes. Before xctool runs a test bundle, it would first query the test bundle to discover these preferences, then do the test run while honoring those preferences. The querying already happens via otest-query - we fetch a list of all test cases in the bundle - so now we need to extend this to dump extra attributes about each test.

To start, we'd probably want attributes to control: SDK version, device type, timeout.

How do we add annotations to Obj-C which doesn't have them? I don't know. Here's one idea, but maybe there's something better we can do.

We could add a TEST macro that looks like the following...

TEST(testSomething, @{SDKS : @["6.0”, "7.0"],
                      DEVICES : @["iPhone Retina (4-inch)"],
                      TIMEOUT : 10.0,
                      })
{
  XCTFail("This test fails!");
}

... but expands to ...

- (NSDictionary *)annotations_testSomething
{
  // otest-query will call this method for each test case.
  return @{SDKS : @["6.0", "7.0"],
           DEVICES : @["iPhone Retina (4-inch)"],
           TIMEOUT : 10.0};
}

- (void)testSomething
{
  XCTFail("This test fails!");
}

otest-query will then be able to invoke this special annotations method to retrieve the attributes for a given test. Then, instead of otest-query returning just a JSON array of test names, it could return something like ...

[
{"name": "SomeTestClass/testSomething", "annotations": {"SDKS": ["6.0, "7.0"], "TIMEOUT": 10.0}},
{"name": "SomeTestClass/testSomethingElse", "annotations": {}}
]

Monitoring of infrastructure failures.

We need monitoring to help us identify problems in xctool across Facebook's and Travis-CI's build clusters. For example, we've seen instances where the simulator will fail to launch for application tests, but we don't have any idea how wide spread the problem is.

We should build a new xctool reporter that can log info about infrastructure failures (like the simulator failing to start). It might also be interesting to log stats about build / test times in the aggregate. Of course, it should be careful to not log anything specific to private projects.

If we made it log to Parse, we could possible re-use it between FB and Travis-CI.

Automatically identify flaky tests.

We occasionally get flaky tests. They might be too timing dependent (e.g. quick timeouts). They might be flaky when run concurrently with other tests (we’ve seen this with tests that use the same hard-coded temp file paths). They might only pass or fail when run in a certain order (e.g. an earlier test might tickle some global state and not clean up after itself, and this might cause problems for later tests run in the same process).

When tests fail, we should re-run them 1 or 2 times to see if it’s a constant failure.

flaky should also be a proper error state along with passed, failed, errored, etc.

Disabled test black-list.

We’d like to have a better way to disable specific tests in our continuous integration runs. Something like a black-list of specific tests to skip.

At FB, we typically disable iOS tests by renaming the method to something like DISABLED_testSomething. For us, this is better than unchecking a specific test in the Xcode scheme since we have many schemes / workspaces and we’d have to update them all. And, it’s better than commenting out the code since it’d be more likely to rot that way. The downside is this requires a commit.

We should add an option to the run-tests command that reads in a text file of test cases to skip. Once we have this, we can integrate it with FB’s web-based, database-backed tool that lets quickly enable/disable tests.

BDD-style output for the Kiwi + Specta test frameworks.

Right now, Kiwi test output looks like …

run-test KiwiTests-XCTest.xctest (iphonesimulator7.0, logic-test, GC OFF)
    -[KiwiTests_XCTest Foo_ItDoesSomething]
  ✓ -[KiwiTests_XCTest Foo_ItDoesSomething] (3 ms)
    -[KiwiTests_XCTest Foo_ItDoesAnotherthing]
  ✓ -[KiwiTests_XCTest Foo_ItDoesAnotherthing] (0 ms)

It’d be awesome if it could instead look like …

run-test KiwiTests-XCTest.xctest (iphonesimulator7.0, logic-test, GC OFF)

    Foo
      does something ✓ (3 ms)
      does another thing ✓ (0 ms)

Kiwi knows the full-text description for the describe and it blocks, but we’re not currently capturing that inside of otest-shim. The BDD community would probably love this.

Make code-coverage work out-of-the-box with xctool.

It seems pretty hard to setup iOS testing with code coverage:

Could we make testing with code-coverage work out-of-the-box with xctool? It’d be great if the run-tests action could automatically output HTML code coverage results without requiring any code or project modifications.

Clone this wiki locally