breaking(transformers): redesign notation transformers

[fuma-nama]

Description

This PR aims to redesign notation transformer utility and its usage to improve performance and make it more intuitive.

Changes

• Parse comments separately and cache it into the data property of code nodes.
• Notation transformers read and replace parsed comments.
• Update original token, remove empty comment tokens and lines if necessary.

New Syntax

console.log("hello") // [!code highlight]

console.log("highlighted") // [!code highlight:2]
console.log("highlighted")

// the comment below will be removed
// [!code highlight:2]
console.log("highlighted")
console.log("highlighted")

console.log("combine") // [!code focus] [!code highlight]
console.log("combine") // Comment Allowed [!code focus] [!code highlight]

JSX

<div>
    <p>Hello World</p>
    {/* [!code highlight] */}
    <p>Highlighted</p>
</div>

Breaking

It only looks for the last few tokens of line to scan comments:

console.log("// [!code highlight]"); // will not work

It doesn't need multiple // to chain notations:

console.log("test") // [!code highlight] // [!code ++]

// will cause weird result like:
console.log("test") // //

Edge cases

for usage like:

// some text [!code highlight:2]
console.log("highlighted")
console.log("highlighted")

line 2-3 will be highlighted, but the original comment is kept.

Backward Compatibility

Added a legacy option to force legacy behaviors:

transformerNotationFocus({ legacy: true })
transformerNotationHighlight({ legacy: true })
transformerNotationErrorLevel({ legacy: true })

Linked Issues

#619 #770

Additional context

this PR will introduce breaking changes, I currently made a separate package to avoid the problems. Backward compatibility is possible (edit: supported), but it takes extra care and further discussion.

code implementations:

• the createCommentNotationTransformer parses comment with parseComments and cache the result.

• each parsed comment object contains related hast nodes, and info:

      /* hello world */
   |----|-----------|-|
   prefix   content  suffix
  

• remove nodes when content became empty.

• transformers using createCommentNotationTransformer still provides a regex, but the regex no longer need to match the comment

[netlify]

✅ Deploy Preview for shiki-matsu ready!

Name	Link
🔨 Latest commit	d5696e0
🔍 Latest deploy log	https://app.netlify.com/sites/shiki-matsu/deploys/673b4b5b1260590008b9bd8f
😎 Deploy Preview	https://deploy-preview-835--shiki-matsu.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[netlify]

✅ Deploy Preview for shiki-next ready!

Name	Link
🔨 Latest commit	d5696e0
🔍 Latest deploy log	https://app.netlify.com/sites/shiki-next/deploys/673b4b5b5a05400007fb2c22
😎 Deploy Preview	https://deploy-preview-835--shiki-next.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[fuma-nama]

it indicates the line contains only the comment itself, like:

// [!code highlight]
console.log("test")

so we will highlight the next line, and remove the original line when content became empty.

[fuma-nama]

we no longer need to provide the regex for matching comments, it is better given that the name createCommentNotationTransformer should handles this too.

[fuma-nama]

The core changes are basically done.

I kept the failing unit tests for future experiments about backward compatibility, I'm thinking about a legacy option to support deprecated behaviours, going to make that this weekend