Programatically creating Documents

[dstovall]

I've got a data set that I would to render as Markdown.

My naive approach is to use Parser to get a Document instance, append some new Nodes to it, then use Formatter to render it as Markdown:

Document document = this.markdownParser.parse("# Parsed Heading");

Heading heading = new Heading(BasedSequence.of("New Heading"));
heading.setOpeningMarker(BasedSequence.of("#"));
heading.setClosingMarker(BasedSequence.of(""));
heading.setText(BasedSequence.of("Newer Heading"));
heading.setLevel(2);

document.appendChild(heading);
String markdown = markdownFormatter.render(document);

expecting something like:

# Parsed Heading

## New Heading

Instead, I get an exception. :/

After adding various permutations of setCharsFromContent, I can get formatted to render like this:

# Parsed Heading

#

So, three questions really:

1. What's wrong with the code above - why is my new heading blank?
2. Is there an example of programmatically building a Document with other types of nodes (lists, bold, etc)?
3. Should I even be trying to do this with Flexmark?

[dstovall]

I've discovered that I need to add a Text child node to get an actual heading to show up:

Text textNode = new Text("New Heading Text");
heading.appendChild(textNode);

So it seems that the answer to my first question is -> I just don't understand the AST model completely. That will come with time I suppose (unless the model is documented somewhere that I don't know about).

I would still love to see an example, even a partial or less-than-perfect example of building a Document with paragraphs, lists, headings, and bold text. Something with embedded HTML would be really awesome. (I'll try to develop this myself and post it when I get something working)

For now:

Document document = this.markdownParser.parse("# Heading 1\n\nParagraph 1");

Heading newHeading = new Heading(BasedSequence.of("New Heading"));
newHeading.setOpeningMarker(BasedSequence.of("##"));
newHeading.setClosingMarker(BasedSequence.of(""));

Text newHeadingText = new Text("New Heading Text");
newHeading.appendChild(newHeadingText);
document.appendChild(newHeading);

String markdown = markdownFormatter.render(document);

will produce

# Heading 1

Paragraph 1

## New Heading Text 

[vsch]

@dstovall, I don't think creating the AST to then create markdown is an efficient approach. The AST tracks the source position, so it is geared towards creating it from a single character sequence. It has a lot of intricacies in terms of child nodes, which the parser creates. Doing it by bit-banging on the classes is error prone and frankly too painful.

Why not just output your markdown to a StringBuilder or StringBuffer? Markdown is just text.

You can always take your generated markdown text and parse it, then render it using the Formatter. That will give you nicely formatted markdown without you having to worry about any indentations, other than needed for the syntax.

As an alternative, you can use MarkdownWriter a subclass of Appendable to have indentations inserted by setting a prefix, you can see how CoreNodeFormatter uses this class when rendering a document for formatting.

[dstovall]

Yes, you're right. This is basically what I ended up doing.

Perhaps the full use case will help clarify why I'm interested. I'm creating two sets of markdown documents.

1. technical documentation for a software product
2. a simple project status site

In both cases, using markdown makes it easy to add and update the content across the team (of software engineers) and I can store/share it all through git which produces a really nice history of all the changes.

Of course, the markdown itself is then passed off to another system (e.g. Material for MkDocs) for final rendering.

What I would like to do is periodically update some of the markdown content programmatically. For example:

1. pull code snippets from the production code base into the middle of various documentation pages.
2. pull "to do" and "done" items from another system and insert them into the project status dashboard.
3. copy portions of markdown from one page into another with (maybe?) some simple transformations or augmentations.

What I've found is that I can create temporary markdown in a StringBuffer, parse it, extract the Nodes from the resulting document, insert those Nodes into the "real" Document, then format the real Document to a String. As needed, I can reparse the formatted real document string to get a clean and consistent Document AST.

I toyed with the idea of inserting my temporary markdown directly into the real document's string, but had a hard time reliably finding the right offset. I could probably make this work with a bit more effort though.

This is working well enough - it just feels awkward.

-Drew

PS - If you do decide to allow direct manipulation of the AST, I'd be particularly interested in the extraction and insertion of lists and tables.

[vsch]

@dstovall, thank you for the clarification.

You are doing it the easiest way. Letting the parser handle the heavy lifting of creating the AST. You can parse the original document, then use the AST to find the offset(s) to insert/replace text, but keep in mind, the text may need to be properly indented before insertion, to make sure it will parse properly. This is a lot more complicated that would seem at first glance.

If inserting AST from one document into another works, then it is the easiest and most reliable way to do it. Formatting it to a string, should take care of all the indentations, bugs not withstanding.

If you have specific use cases, which I can use to test code, please let me know. The ability to swap AST sections is an interesting use case and once I have brought the library up to functional again, I will take a look at that. However, the only way to get a clean AST after swapping is to format to string and reparse the result, so all nodes are based on the same source text.

What do I mean by functional? Travis CI is now a paid service. With my old account gone, I have to set up GitHub workflows for CI builds. I also want to migrate the builds to Gradle from Maven, go through the PRs and merge ones that still apply, update to more recent CommonMark spec.