Tracking Issue for `frontmatter`

[epage]

This is a tracking issue for the RFC "frontmatter" (rust-lang/rfcs#3503).
The feature gate for the issue is #![feature(frontmatter)].

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.
Discussion comments will get marked as off-topic or deleted.
Repeated discussions on the tracking issue may lead to the tracking issue getting locked.

Steps

• Implement the RFC for rustc:
  • Add unstable frontmatter support #137193
  • Implement RFC 3503: frontmatters #140035
  • fix: Include frontmatter in -Zunpretty output  #143708
  • fix(lexer): Allow '-' in the frontmatter infostring continue set #145751
  • fix(lexer): Don't require frontmatters to be escaped with indented fences #145754
  • test(rustfmt): Verify frontmatter is preserved #145766
  • r-a
• Adjust documentation (see instructions on rustc-dev-guide)
  • docs(ref): Specify frontmatter reference#1974
• Formatting for new syntax has been added to the Style Guide (nightly-style-procedure) (Styling of Rust Frontmatter style-team#212)
  • docs(style): Specify the frontmatter style #145617
• Stabilization PR (see instructions on rustc-dev-guide)
  • Stabilize Frontmatter #148051

Non-blocking

• rustfmt support

Test coverage prior to stabilization

In the following, the RFC text in its accepted form will be referred to as the "accepted text".

• (Before stabilization) we should add a run-make test that exercises how cargo and rustc works together w.r.t. frontmatter, depending on what the final cooperation scheme we end up using.
• (Before stabilization) we should add a positive smoke test to check that rustfix is able to account for frontmatters (or properly ignore), i.e. //@ run-rustfix.

Unresolved Questions

• What level of quality of error messages is a blocker for stabilization? See also Add unstable frontmatter support #137193 (comment)
  • In Implement RFC 3503: frontmatters #140035, the initial (compiler) impl already tries to provide diagnostics for the most common erroneous usages.
  • However, it might be worth refining how the diagnostics responsibility of reporting invalid frontmatters get shared between $tools <-> rustc.
• How does frontmatter interact with -Zunpretty? See discussions in Add unstable frontmatter support #137193 (comment).
  • Implemented in fix: Include frontmatter in -Zunpretty output  #143708
• How will various tools (rust-analyzer, rustfmt, cargo, clippy) handle frontmatter (and interop w/ rustc)? Will parsing support (or ability to ignore frontmatter) be a stabilization blocker (re. toolchain UX)? For all of them, for a subset of the tools?
• Should frontmatter be outright banned in doctests? Consider these examples: Implement RFC 3503: frontmatters #140035 (comment).
• (Needs explicit design decision prior to stabilization) Should whitespace be permitted between the starting dashes and the infostring? E.g. would --- cargo be accepted? In Implement RFC 3503: frontmatters #140035 and the accepted text, whitespace between starting dashes and the infostring is permitted.
• feature(frontmatter) rejects frontmatters with dashes with leading spaces #141367
• Interaction with proc-macro and breakages like Introduction of frontmatter was a theoretical breaking change (via TokenStream::from_str) #145520

Implementation history

Initial implementations

• Add unstable frontmatter support #137193
• Implement RFC 3503: frontmatters #140035

(Significant) changes since the RFC was accepted

Frontmatter opener vs infostring starting with - character grammar ambiguity

The accepted text has a grammar ambiguity in the case where the infostring starts with a -.

• The same - character is used to delimit the frontmatter "opener" ---.
• The opener can have more than 3 - characters to support escaping nesting of ---s within the frontmatter.
• However, infostrings described in the accepted text can also begin with -, making it ambiguous as to whether it's a 3-- opener + --starting infostring, or if it is a 4-- opener + infostring.

Remedy: we can require the infostring to begin with Unicode XID_Start, then permit subsequently characters in the set { XID_Continue, . }. I.e. (illustrative)

- infostring = term *
- term       = { all characters - ( whitespace | ',' ) }
+ infostring = XID_Start , { (XID_Continue | '.') } * ;

See discussions at #140035 (comment).

• This can be further relaxed in the future (carefully) if it does not introduce more ambiguities.
• RFC initially required infostring to consist of identifier, but was relaxed to support . in e.g. file.ext (see RFC: Syntax for embedding cargo-script manifests rfcs#3503 (comment)).

Illustrative grammar

#140035 (comment) (with start and end dashes >= 3 - characters and must match in - count):

frontmatter_start = dashes , { whitespace } * , infostring , { whitespace } * , '\n' ;
frontmatter_end = dashes , { whitespace } * , '\n' ;
infostring = XID_Start , { (XID_Continue | '.') } * ;
dashes = "---" , { '-' } * ;

[epage]

Support in external parsers

• feature: frontmatter syntax support tree-sitter/tree-sitter-rust#253
• [Rust] Frontmatter syntax support sublimehq/Packages#4164
• Support Rust's frontmatter syntax in rust.vim rust.vim#522 (Support Rust's frontmatter syntax in rust.vim neovim/neovim#32534)
• Support Rust's frontmatter syntax dustypomerleau/rust-syntax#50 (used by VS Code)