Decide policy on multidev mutability

[greg-1-anderson]

The example-drops-8-composer project uses the Behat Drupal Driver to run tests, so the multidev environment is not altered by the test procedure. This is not 100% reliable, though; it is possible that users may add their own tests that alter the multidev state.

The example-wordpress-composer repository explicitly backs up the environment before the test, and then restores it again. This has the advantage of keeping the multidev in a state where it could be used for other purposes, e.g. for an agency to review work-in-progress with a client.

Usually, instead of backup-alter-restore, a more stable design paradigm is copy-alter (the copy)-destroy.

Disadvantages of backup / restore over copy / delete:

• Complexity of implementation
  • Restore might not happen if the test fails, unless there is a when on_fail step to catch this.
  • If there is an on_fail step, it must be skipped if the step failed before the backup was made
• Potentially dangerous
  • Work might be lost if an agency makes a change after the backup, but before the restore. (Tests take minutes to run, and users are not necessarily aware of what stage the tests are in when they are using the multidev.)

The use-case for having a usable pr multidev might be better served by simply using a ci-BUILD_NUMBER multidev to run all tests, and make a pr-PULL_REQUEST_ID multidev only for agency use -- never for test running. The ci-* multidevs are cleaned up later (we could do this in a separate Circle CI 2.0 build step, in parallel with the other steps), so we'd have to compare the cost of multidev creation against the comparable cost of one backup + one restore. The time trade-offs may differ for small sites vs. large sites. It is likely that multidev creation is slower than backup + restore, especially for smaller sites. We need to weigh the performance cost against the disadvantages of backup / restore.

[stevector]

The use-case for having a usable pr multidev might be better served by simply using a ci-BUILD_NUMBER multidev to run all tests, and make a pr-PULL_REQUEST_ID multidev only for agency use -- never for test running.

I've often said that we're using these environments as both MultiDev and MultiTest. (Dev being an environment for working on a branch/PR over time and Test being an environment for reviewing tag/commit)

This question highlights how those overlapping usages don't always play well together. Making both pr-N and ci-N could help resolve that tension.

But it also has a high likelihood of increasing confusion for Pantheon customers overwhelmed by the number of moving pieces in this set up. I'd rather not cross that bridge right now if we don't have to. Such a decision feels to me like one that is best answered by a focus group of developers using these repos on real projects.

Though we do already have a precedent for a commit building to two environments. Commits on master build to ci-N and master on Pantheon.

---

For this specific issue, I'd like us to find a mechanism that can be configured per site. Yes, the starting point of the WordPress and Drupal repos should be the same. But if an individual team wants to restore after testing or not, it should be easy to switch.

One way we could do that would be by putting the mechanism for restoring/recloning into the running of ./vendor/bin/behat itself, perhaps with an exec() from an @afterSuite method in a `FeatureContext.php.

IMO a 30 or 70 line bash file wrapping ./vendor/bin/behat is a problem to be solved. When I do Jump Starts and walk people through running Behat tests for the first time it is a pain to explain which lines need to be copy/pasted out of those files so that ./vendor/bin/behat can be run from a local machine.

---

But maybe I'm being short sited by only thinking about this question of multidev mutability from the perspective of Behat.

I think there is a good chance that real projects will add more testing jobs that have the potential to change data (performance tests, load tests, visual regression tests). For instance, I can imagine that a project would want a Circle job to add 1000 pieces of content and make assertions about data coming from New Relic. The team would probably want those 1000 dummy posts gone before sending the multidev url to the client.

That team might ask us "So, when all of the build steps are done and every Circle job has run, will the Multidev be exactly how Dev would be if we click merge on the pull request?" I think the answer to the question probably should be "yes".

---

So all that to say, for teams that want it, a restore/reclone should run at the end of build_deploy_and_test or in a separate job following (not in the Behat bash wrapper). My hesitation at this point is that I'm concerned about the overall complexity of the build process and doing the same thing (setting up one multidev) twice in one run feels like a net increase in complexity.

[greg-1-anderson]

doing the same thing (setting up one multidev) twice in one run feels like a net increase in complexity.

Yeah, it's hard to cover all of these use-cases without some increase in complexity. Backup/restore has more hidden complexity (works fine until you don't notice the tests erased a db change).

Another thing we might consider is the previous idea of running the ci-* tests on Circle instead of Pantheon. The on-circle tests wouldn't have the same fidelity as the on-Pantheon tests, but they'd run faster. If a subset of the tests ran in parallel on Circle (for pr-* builds as well), you'd get an early warning for some failures.

Of course, this does not satisfy the needs of anyone who wants to run a destructive test that only runs on Pantheon. If we used a Lando docker image on Circle, that might help some.