Testing

The commonmark spec.txt file is an excellent format which provides the narrative description, source and generated HTML. It is a format against which to run parser compliance tests.

The format of this file was modified to add AST output to allow testing of the generated AST which is crucial for using this parser for syntax highlighting. All test util classes were modified to handle the original format and the extended format.

To allow testing of parser options and extensions, the format was extended to specify which options should be used in running the test. The test subclass must provide a mapping from the option string to a test specific options set. This allows a single test spec file to be used to test more than one parser configuration.

In all cases the original format is supported and the original spec.txt file is used to validate parser compliance.

ComboSpecTestCase class which combines the functionality of SpecTestCase and FullSpecTestCase. Only one test class per extension is needed if all the tests can be done via a spec.txt file.

FullSpecTestCase regenerates the spec text with the expected HTML and AST replaced by the parser generated results then asserting that this is equal to the original, in addition to running the individual tests. This allows comparing compliance to full spec in one place and in the case of running the tests in JetBrains IDEA, makes it easy to copy generated results to the expected inputs to make creating and updating expected results easier.....

If the spec file does not have an AST section then the expected AST will not be generated or validated, nor will it be present in the generated full file result.

The markdown headers in the spec are used to mark sections. Each test case has the following format:

• The last markdown header is used as the Section value of SpecExample instance

• all text in the start line of an example, between example and end of line is ignored

• if the start example line end in: options(....) then the text between () is used as the option set identifier, leading and trailing blanks of the identifier are ignored. If the resulting identifier is empty then default parser configuration will be used.

  The string in options is taken as a comma separated list, spaces are trimmed off. If more than one option is present then the combination of the DataHolder contents will be used to run the test case.

  If the same key is set in two options, the value assigned to the one coming later in the list will be used.

  ⚠️ Option ignore is processed by the base class and if present will result in an AssumptionViolatedException being thrown causing the test case to be ignored.

• FullSpecTestCase will add Section, example number and any options provided in the original spec file.

Sample spec examples with and without the options() clause

## Reference Repository Keep First tests

Test repository KEEP_FIRST behavior, meaning the first reference def is used

```````````````````````````````` example Reference Repository Keep First tests: 1
[ref]

[ref]: /url1
[ref]: /url2
[ref]: /url3
.
<p><a href="/url1">ref</a></p>
.
Document[0, 46]
  Paragraph[0, 6]
    LinkRef[0, 5] textOpen:[0, 0] text:[0, 0] textClose:[0, 0] referenceOpen:[0, 1, "["] reference:[1, 4, "ref"] referenceClose:[4, 5, "]"]
      Text[1, 4] chars:[1, 4, "ref"]
  Reference[7, 19] refOpen:[7, 8, "["] ref:[8, 11, "ref"] refClose:[11, 13, "]:"] urlOpen:[0, 0] url:[14, 19, "/url1"] urlClose:[0, 0] titleOpen:[0, 0] title:[0, 0] titleClose:[0, 0]
  Reference[20, 32] refOpen:[20, 21, "["] ref:[21, 24, "ref"] refClose:[24, 26, "]:"] urlOpen:[0, 0] url:[27, 32, "/url2"] urlClose:[0, 0] titleOpen:[0, 0] title:[0, 0] titleClose:[0, 0]
  Reference[33, 45] refOpen:[33, 34, "["] ref:[34, 37, "ref"] refClose:[37, 39, "]:"] urlOpen:[0, 0] url:[40, 45, "/url3"] urlClose:[0, 0] titleOpen:[0, 0] title:[0, 0] titleClose:[0, 0]
````````````````````````````````

## Reference Repository Keep Last tests

Test repository KEEP_LAST behavior, meaning the last reference def is used

```````````````````````````````` example(Reference Repository Keep Last tests: 1) options(keep-last)
[ref]

[ref]: /url1
[ref]: /url2
[ref]: /url3
.
<p><a href="/url3">ref</a></p>
.
Document[0, 46]
  Paragraph[0, 6]
    LinkRef[0, 5] textOpen:[0, 0] text:[0, 0] textClose:[0, 0] referenceOpen:[0, 1, "["] reference:[1, 4, "ref"] referenceClose:[4, 5, "]"]
      Text[1, 4] chars:[1, 4, "ref"]
  Reference[7, 19] refOpen:[7, 8, "["] ref:[8, 11, "ref"] refClose:[11, 13, "]:"] urlOpen:[0, 0] url:[14, 19, "/url1"] urlClose:[0, 0] titleOpen:[0, 0] title:[0, 0] titleClose:[0, 0]
  Reference[20, 32] refOpen:[20, 21, "["] ref:[21, 24, "ref"] refClose:[24, 26, "]:"] urlOpen:[0, 0] url:[27, 32, "/url2"] urlClose:[0, 0] titleOpen:[0, 0] title:[0, 0] titleClose:[0, 0]
  Reference[33, 45] refOpen:[33, 34, "["] ref:[34, 37, "ref"] refClose:[37, 39, "]:"] urlOpen:[0, 0] url:[40, 45, "/url3"] urlClose:[0, 0] titleOpen:[0, 0] title:[0, 0] titleClose:[0, 0]
````````````````````````````````

The first part is the markdown source, the expected HTML is separated by single . on the line. Expected AST is added as a third part to the original spec.txt, also separated by a single . on the line.

The best way to create a test for an extension is to start with a copy of an existing one and modify the markdown source for the extension, deleting the expected HTML and AST text but leaving the . separator lines. Running the FullSpecTestCase derived test will create a full spec file with all section filled in with actual results. These should be validated then copied to the spec file.

The options are passed to the test class options(String optionSet) method with a DataHolder return value which will be used to run the example for which this option was provided.

public class ComboCustomSpecTest extends ComboSpecTestCase {
    private static final Map<String, DataHolder> optionsMap = new HashMap<>();
    static {
        optionsMap.put("option", new MutableDataSet()
                .set(CustomExtension.USE_CUSTOM_OPTION, true)
        );
    }

    @Override
    protected DataHolder options(String optionSet) {
        return optionsSet(optionSet);
    }
}