chore: Create CI action that validates links in md files

[comphead]

DataFusion does not have a CI action to check broken links in markdown files.

Here's what the existing CI workflows do for docs/markdown:

• docs_pr.yaml — Builds the Sphinx docs (docs/build.sh) on PRs that touch docs/**, catching build errors/warnings, but not specifically checking links.
• docs.yaml — Deploys built docs to the asf-site branch on pushes to main.
• dev.yml — Runs prettier formatting checks on docs, spell checking with typos, and license header checks — but no link checking.

There's no use of a link checker tool like lychee, markdown-link-check, or linkchecker anywhere in the workflows

Originally posted by @comphead in #21679 (comment)

[rluvaton]

Not only markdown files is needed but in rust comments as well

[Geethapranay1]

hi @comphead i can take this on if it’s still available, since here @rluvaton mentions rust comments too, I’ll cover both markdown links and rust doc-comment links in the same pr unless you’d prefer to split them.

[comphead]

Thanks @Geethapranay1 feel free to take it

[Geethapranay1]

Hi @comphead, I have the first version working with lychee in CI.

One clarification before I finalize the pr: should the markdown link check validate only local/internal links in tracked markdown files, or should it also validate external HTTP/HTTPS links?

Right now I’m leaning toward checking all tracked *.md files for local links, while excluding external URLs to avoid flaky CI failures from network issues/rate limits. Rust doc-comment links are covered through the existing cargo doc job with RUSTDOCFLAGS=-D warnings.

Does that scope match what you expected for this issue, or would you prefer external links to be included too?

[comphead]

Lets focus on internal links for now, but we need to document that the tool currently checks only internal ones.

For external it might be expensive and flaky as you said