Testing

The commonmark spec.txt file is an excellent format which provides the narrative description, source and generated HTML. It is a format against which to run parser compliance tests.

The format of this file was modified to add AST output to allow testing of the generated AST which is crucial for using this parser for syntax highlighting. All test util classes were modified to handle the original format and the extended format.

To allow testing of parser options and extensions, the format was extended to specify which options should be used in running the test. The test subclass must provide a mapping from the option string to a test specific options set. This allows a single test spec file to be used to test more than one parser configuration.

In all cases the original format is supported and the original spec.txt file is used to validate parser compliance.

ComboSpecTestCase class which combines the functionality of SpecTestCase and FullSpecTestCase. Only one test class per extension is needed if all the tests can be done via a spec.txt file.

FullSpecTestCase regenerates the spec text with the expected HTML and AST replaced by the parser generated results then asserting that this is equal to the original, in addition to running the individual tests. This allows comparing compliance to full spec in one place and in the case of running the tests in JetBrains IDEA, makes it easy to copy generated results to the expected inputs to make creating and updating expected results easier.....

If the spec file does not have an AST section then the expected AST will not be generated or validated, nor will it be present in the generated full file result.

The markdown headers in the spec are used to mark sections. Each test case has the following format:

• The last markdown header is used as the Section value of SpecExample instance

• all text in the start line of an example, between example and end of line is ignored

• if the start example line ends in: options(....) then the text between () is used as the option set identifier, leading and trailing blanks of the identifier are ignored. If the resulting identifier is empty then default parser configuration will be used.

  The string in options is taken as a comma separated list, spaces are trimmed off. If more than one option is present then the combination of the DataHolder contents will be used to run the test case.

  If the same key is set in two options, the value assigned to the one coming later in the list will be used.

  ⚠️ Option IGNORE is processed by the base class and if present will result in an AssumptionViolatedException being thrown causing the test case to be ignored.

  Option FAIL is processed by the base class and if present will result in an expectation of failed comparison to be thrown by the example.

  Option NO_FILE_EOL is the default and will remove the trailing EOL from example source if it is not preceded by a blank line. FILE_EOL can be used to disable this behaviour.

• FullSpecTestCase will add Section, example number and any options provided in the original spec file.

Sample spec examples with and without the options() clause

## Reference Repository Keep First tests

Test repository KEEP_FIRST behavior, meaning the first reference def is used

```````````````````````````````` example Reference Repository Keep First tests: 1
[ref]

[ref]: /url1
[ref]: /url2
[ref]: /url3
.
<p><a href="/url1">ref</a></p>
.
Document[0, 46]
  Paragraph[0, 6]
    LinkRef[0, 5] textOpen:[0, 0] text:[0, 0] textClose:[0, 0] referenceOpen:[0, 1, "["] reference:[1, 4, "ref"] referenceClose:[4, 5, "]"]
      Text[1, 4] chars:[1, 4, "ref"]
  Reference[7, 19] refOpen:[7, 8, "["] ref:[8, 11, "ref"] refClose:[11, 13, "]:"] urlOpen:[0, 0] url:[14, 19, "/url1"] urlClose:[0, 0] titleOpen:[0, 0] title:[0, 0] titleClose:[0, 0]
  Reference[20, 32] refOpen:[20, 21, "["] ref:[21, 24, "ref"] refClose:[24, 26, "]:"] urlOpen:[0, 0] url:[27, 32, "/url2"] urlClose:[0, 0] titleOpen:[0, 0] title:[0, 0] titleClose:[0, 0]
  Reference[33, 45] refOpen:[33, 34, "["] ref:[34, 37, "ref"] refClose:[37, 39, "]:"] urlOpen:[0, 0] url:[40, 45, "/url3"] urlClose:[0, 0] titleOpen:[0, 0] title:[0, 0] titleClose:[0, 0]
````````````````````````````````

## Reference Repository Keep Last tests

Test repository KEEP_LAST behavior, meaning the last reference def is used

```````````````````````````````` example(Reference Repository Keep Last tests: 1) options(keep-last)
[ref]

[ref]: /url1
[ref]: /url2
[ref]: /url3
.
<p><a href="/url3">ref</a></p>
.
Document[0, 46]
  Paragraph[0, 6]
    LinkRef[0, 5] textOpen:[0, 0] text:[0, 0] textClose:[0, 0] referenceOpen:[0, 1, "["] reference:[1, 4, "ref"] referenceClose:[4, 5, "]"]
      Text[1, 4] chars:[1, 4, "ref"]
  Reference[7, 19] refOpen:[7, 8, "["] ref:[8, 11, "ref"] refClose:[11, 13, "]:"] urlOpen:[0, 0] url:[14, 19, "/url1"] urlClose:[0, 0] titleOpen:[0, 0] title:[0, 0] titleClose:[0, 0]
  Reference[20, 32] refOpen:[20, 21, "["] ref:[21, 24, "ref"] refClose:[24, 26, "]:"] urlOpen:[0, 0] url:[27, 32, "/url2"] urlClose:[0, 0] titleOpen:[0, 0] title:[0, 0] titleClose:[0, 0]
  Reference[33, 45] refOpen:[33, 34, "["] ref:[34, 37, "ref"] refClose:[37, 39, "]:"] urlOpen:[0, 0] url:[40, 45, "/url3"] urlClose:[0, 0] titleOpen:[0, 0] title:[0, 0] titleClose:[0, 0]
````````````````````````````````

The first part is the markdown source, the expected HTML is separated by single . on the line. Expected AST is added as a third part to the original spec.txt, also separated by a single . on the line.

The best way to create a test for an extension is to start with a copy of an existing one and modify the markdown source for the extension, deleting the expected HTML and AST text but leaving the . separator lines. Running the RendererSpecTest derived test will create a full spec file with all section filled in with actual results. These should be validated then copied to the spec file.

import com.vladsch.flexmark.ext.typographic.TypographicExtension;
import com.vladsch.flexmark.parser.Parser;
import com.vladsch.flexmark.util.data.DataHolder;
import com.vladsch.flexmark.util.data.MutableDataSet;

import java.util.Arrays;
import java.util.Collections;

public class ComboCustomSpecTest extends RendererSpecTest {
    final private static String SPEC_RESOURCE = "/ext_typographic_ast_spec.md";
    final public static @NotNull ResourceLocation RESOURCE_LOCATION = ResourceLocation.of(COMBO_CUSTOM_SPEC_TEST.class, SPEC_RESOURCE);

    final private static DataHolder OPTIONS = new MutableDataSet()
            .set(Parser.EXTENSIONS, Arrays.asList(TypographicExtension.create()))
            .toImmutable();

    final private static Map<String, DataHolder> optionsMap = new HashMap<String, DataHolder>();
    static {
        optionsMap.put("option", new MutableDataSet()
                .set(CustomExtension.USE_CUSTOM_OPTION, true)
        );
    }
    public ComboCustomSpecTest() {
        super(example, optionsMap, OPTIONS);
    }

    @Parameterized.Parameters(name = "{0}")
    public static List<Object[]> data() {
        return getTestData(RESOURCE_LOCATION);
    }

    @Override
    public @NotNull
    ResourceLocation getSpecResourceLocation() {
        return RESOURCE_LOCATION;
    }
}