Add llm.txt files to the website.

[Daanvdplas]

Like: https://docs.moonbeam.network/ai-resources/ai-resources/

The Transformation Process

1. Content Extraction

Scrape or export content from existing documentation websites
Extract the raw text, removing HTML formatting, navigation elements, and other UI components
Preserve the core informational content while stripping away presentation layer

2. Metadata Addition

Add structured headers with title and description
Include source URLs for reference
Add categorization tags (like "Basics", "Ethereum Toolkit", "Precompiles", etc.)
Create topic classifications to help with search and organization

3. Content Restructuring

Convert to markdown or plain text format
Standardize formatting across all documents
Ensure consistent heading structures
Remove redundant navigation text and boilerplate

4. AI Optimization

Structure content for better LLM consumption
Add contextual tags and categories
Create clear document boundaries
Optimize for semantic search and retrieval

Tools and Approaches

Automated Solutions:
Web scraping tools (Scrapy, BeautifulSoup)
Documentation crawlers
CMS export functions
API-based content extraction

Manual Curation:
Content review and cleanup
Category assignment
Quality assurance
Link validation

[cmichi]

Note that we already generate an https://use.ink/llms.txt. There is probably room for improving the content in there based on the above list.

[niklasp]

What we have is probably whata called llms-full.txt in the example page. Using the exiating tooling multiple files with other entry points can be created

[Daanvdplas]

So recently the monsters ink! team has given us a analysis of AI what is missing in the llm.txt file of ink based off the ink docs. The ink docs uses repomix which generates the llm.txt file from scratch when a new PR is merged.

Tommy also has been playing around with this txt file and has made adjustments to it in terms of info that was incorrect or missing. This however lives locally on his machine or in our new pop mcp repo: https://github.com/r0gue-io/pop-mcp

The llm.txt file should live on the ink docs but based of the two above feedback the docs need to be improved.