diff --git a/.gitignore b/.gitignore index cf9381d..ccf99db 100644 --- a/.gitignore +++ b/.gitignore @@ -3,3 +3,4 @@ dist/ *.tsbuildinfo .worktrees/ .superpowers/ +coverage/ diff --git a/.prettierignore b/.prettierignore index 52af816..c45c1e5 100644 --- a/.prettierignore +++ b/.prettierignore @@ -2,3 +2,4 @@ dist/ node_modules/ pnpm-lock.yaml charts/ +coverage/ diff --git a/README.md b/README.md index bd60779..3bb6657 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # @copilotkit/llmock [![Unit Tests](https://github.com/CopilotKit/llmock/actions/workflows/test-unit.yml/badge.svg)](https://github.com/CopilotKit/llmock/actions/workflows/test-unit.yml) [![Drift Tests](https://github.com/CopilotKit/llmock/actions/workflows/test-drift.yml/badge.svg)](https://github.com/CopilotKit/llmock/actions/workflows/test-drift.yml) [![npm version](https://img.shields.io/npm/v/@copilotkit/llmock)](https://www.npmjs.com/package/@copilotkit/llmock) -Deterministic mock LLM server for testing. A real HTTP server on a real port — not an in-process interceptor — so every process in your stack (Playwright, Next.js, agent workers, microservices) can point at it via `OPENAI_BASE_URL` / `ANTHROPIC_BASE_URL` and get reproducible, instant responses. Streams SSE in real OpenAI, Claude, Gemini, Bedrock, Azure, Vertex AI, Ollama, and Cohere API formats, driven entirely by fixtures. Zero runtime dependencies. +Mock infrastructure for AI application testing — LLM APIs, MCP tools, A2A agents, vector databases, search, and more. Real HTTP server on a real port, fixture-driven, zero runtime dependencies. ## Quick Start @@ -23,72 +23,106 @@ const url = await mock.start(); await mock.stop(); ``` -## When to Use This vs MSW +## Usage Scenarios -[MSW (Mock Service Worker)](https://mswjs.io/) is a popular API mocking library, but it solves a different problem. +### In-process testing -**The key difference is architecture.** llmock runs a real HTTP server on a port. MSW patches `http`/`https`/`fetch` modules inside a single Node.js process. MSW can only intercept requests from the process that calls `server.listen()` — child processes, separate services, and workers are unaffected. +Use the programmatic API to start and stop the mock server in your test setup. Every test framework works — Vitest, Jest, Playwright, Mocha, anything. -This matters for E2E tests where multiple processes make LLM API calls: +```typescript +import { LLMock } from "@copilotkit/llmock"; + +const mock = new LLMock({ port: 5555 }); +mock.loadFixtureDir("./fixtures"); +const url = await mock.start(); +process.env.OPENAI_BASE_URL = `${url}/v1`; + +// ... run tests ... + +await mock.stop(); +``` + +### Running locally +Use the CLI with `--watch` to hot-reload fixtures as you edit them. Point your app at the mock and iterate without touching real APIs. + +```bash +llmock -p 4010 -f ./fixtures --watch ``` -Playwright test runner (Node) - └─ controls browser → Next.js app (separate process) - └─ OPENAI_BASE_URL → llmock :5555 - ├─ Mastra agent workers - ├─ LangGraph workers - └─ CopilotKit runtime + +### CI pipelines + +Use the Docker image with `--strict` mode and record-and-replay for deterministic, zero-cost CI runs. + +```yaml +# GitHub Actions example +- name: Start aimock + run: | + docker run -d --name aimock \ + -v ./fixtures:/fixtures \ + -p 4010:4010 \ + ghcr.io/copilotkit/aimock \ + llmock --strict -f /fixtures + +- name: Run tests + env: + OPENAI_BASE_URL: http://localhost:4010/v1 + run: pnpm test + +- name: Stop aimock + run: docker stop aimock ``` -MSW can't intercept any of those calls. llmock can — it's a real server on a real port. +### Cross-language testing -**Use llmock when:** +The Docker image runs as a standalone HTTP server — any language that speaks HTTP can use it. Python, Go, Rust, Ruby, Java, anything. -- Multiple processes need to hit the same mock (E2E tests, agent frameworks, microservices) -- You want multi-provider SSE format out of the box (OpenAI, Claude, Gemini, Bedrock, Azure, Vertex AI, Ollama, Cohere) -- You prefer defining fixtures as JSON files rather than code -- You need a standalone CLI server +```bash +docker run -d -p 4010:4010 ghcr.io/copilotkit/aimock llmock -f /fixtures -**Use MSW when:** +# Python +client = openai.OpenAI(base_url="http://localhost:4010/v1", api_key="mock") -- All API calls originate from a single Node.js process (unit tests, SDK client tests) -- You're mocking many different APIs, not just OpenAI -- You want in-process interception without running a server +# Go +client := openai.NewClient(option.WithBaseURL("http://localhost:4010/v1")) -| Capability | llmock | MSW | -| ---------------------------- | --------------------- | ------------------------------------------------------------------------- | -| Cross-process interception | **Yes** (real server) | **No** (in-process only) | -| OpenAI Chat Completions SSE | **Built-in** | Manual — build `data: {json}\n\n` + `[DONE]` yourself | -| OpenAI Responses API SSE | **Built-in** | Manual — MSW's `sse()` sends `data:` events, not OpenAI's `event:` format | -| Claude Messages API SSE | **Built-in** | Manual — build `event:`/`data:` SSE yourself | -| Gemini streaming | **Built-in** | Manual — build `data:` SSE yourself | -| WebSocket APIs | **Built-in** | **No** | -| Fixture file loading (JSON) | **Yes** | **No** — handlers are code-only | -| Request journal / inspection | **Yes** | **No** — track requests manually | -| Non-streaming responses | **Yes** | **Yes** | -| Error injection (one-shot) | **Yes** | **Yes** (via `server.use()`) | -| CLI for standalone use | **Yes** | **No** | -| Zero dependencies | **Yes** | **No** (~300KB) | +# Rust +let client = Client::new().with_base_url("http://localhost:4010/v1"); +``` ## Features -- **[Multi-provider support](https://llmock.copilotkit.dev/compatible-providers.html)** — [OpenAI Chat Completions](https://llmock.copilotkit.dev/chat-completions.html), [OpenAI Responses](https://llmock.copilotkit.dev/responses-api.html), [Anthropic Claude](https://llmock.copilotkit.dev/claude-messages.html), [Google Gemini](https://llmock.copilotkit.dev/gemini.html), [AWS Bedrock](https://llmock.copilotkit.dev/aws-bedrock.html) (streaming + Converse), [Azure OpenAI](https://llmock.copilotkit.dev/azure-openai.html), [Vertex AI](https://llmock.copilotkit.dev/vertex-ai.html), [Ollama](https://llmock.copilotkit.dev/ollama.html), [Cohere](https://llmock.copilotkit.dev/cohere.html) +- **[Record-and-replay](https://llmock.copilotkit.dev/record-replay.html)** — VCR-style proxy records real API responses as fixtures for deterministic replay +- **[Multi-provider support](https://llmock.copilotkit.dev/compatible-providers.html)** — [OpenAI Chat Completions](https://llmock.copilotkit.dev/chat-completions.html), [Responses API](https://llmock.copilotkit.dev/responses-api.html), [Anthropic Claude](https://llmock.copilotkit.dev/claude-messages.html), [Google Gemini](https://llmock.copilotkit.dev/gemini.html), [AWS Bedrock](https://llmock.copilotkit.dev/aws-bedrock.html), [Azure OpenAI](https://llmock.copilotkit.dev/azure-openai.html), [Vertex AI](https://llmock.copilotkit.dev/vertex-ai.html), [Ollama](https://llmock.copilotkit.dev/ollama.html), [Cohere](https://llmock.copilotkit.dev/cohere.html) +- **[MCPMock](https://llmock.copilotkit.dev/mcp-mock.html)** — Mock MCP server with tools, resources, prompts, and session management +- **[A2AMock](https://llmock.copilotkit.dev/a2a-mock.html)** — Mock A2A protocol server with agent cards, message routing, and streaming +- **[VectorMock](https://llmock.copilotkit.dev/vector-mock.html)** — Mock vector database with Pinecone, Qdrant, and ChromaDB endpoints +- **[Services](https://llmock.copilotkit.dev/services.html)** — Built-in search (Tavily), rerank (Cohere), and moderation (OpenAI) mocks +- **[Chaos testing](https://llmock.copilotkit.dev/chaos-testing.html)** — Probabilistic failure injection: 500 errors, malformed JSON, mid-stream disconnects +- **[Prometheus metrics](https://llmock.copilotkit.dev/metrics.html)** — Request counts, latencies, and fixture match rates at `/metrics` - **[Embeddings API](https://llmock.copilotkit.dev/embeddings.html)** — OpenAI-compatible embedding responses with configurable dimensions - **[Structured output / JSON mode](https://llmock.copilotkit.dev/structured-output.html)** — `response_format`, `json_schema`, and function calling - **[Sequential responses](https://llmock.copilotkit.dev/sequential-responses.html)** — Stateful multi-turn fixtures that return different responses on each call - **[Streaming physics](https://llmock.copilotkit.dev/streaming-physics.html)** — Configurable `ttft`, `tps`, and `jitter` for realistic timing - **[WebSocket APIs](https://llmock.copilotkit.dev/websocket.html)** — OpenAI Responses WS, Realtime API, and Gemini Live - **[Error injection](https://llmock.copilotkit.dev/error-injection.html)** — One-shot errors, rate limiting, and provider-specific error formats -- **[Chaos testing](https://llmock.copilotkit.dev/chaos-testing.html)** — Probabilistic failure injection: 500 errors, malformed JSON, mid-stream disconnects -- **[Prometheus metrics](https://llmock.copilotkit.dev/metrics.html)** — Request counts, latencies, and fixture match rates at `/metrics` - **[Request journal](https://llmock.copilotkit.dev/docs.html)** — Record, inspect, and assert on every request - **[Fixture validation](https://llmock.copilotkit.dev/fixtures.html)** — Schema validation at load time with `--validate-on-load` - **CLI with hot-reload** — Standalone server with `--watch` for live fixture editing - **[Docker + Helm](https://llmock.copilotkit.dev/docker.html)** — Container image and Helm chart for CI/CD pipelines -- **Record-and-replay** — VCR-style proxy-on-miss records real API responses as fixtures for deterministic replay - **[Drift detection](https://llmock.copilotkit.dev/drift-detection.html)** — Daily CI runs against real APIs to catch response format changes - **Claude Code integration** — `/write-fixtures` skill teaches your AI assistant how to write fixtures correctly +## aimock CLI (Full-Stack Mock) + +For projects that need more than LLM mocking, the `aimock` CLI reads a JSON config file and serves all mock services on one port: + +```bash +aimock --config aimock.json --port 4010 +``` + +See the [aimock documentation](https://llmock.copilotkit.dev/aimock-cli.html) for config file format and Docker usage. + ## CLI Quick Reference ```bash @@ -97,6 +131,7 @@ llmock [options] | Option | Short | Default | Description | | -------------------- | ----- | ------------ | ------------------------------------------- | +| `--config` | | | Config file for aimock CLI | | `--port` | `-p` | `4010` | Port to listen on | | `--host` | `-h` | `127.0.0.1` | Host to bind to | | `--fixtures` | `-f` | `./fixtures` | Path to fixtures directory or file | @@ -137,6 +172,19 @@ Full API reference, fixture format, E2E patterns, and provider-specific guides: **[https://llmock.copilotkit.dev/docs.html](https://llmock.copilotkit.dev/docs.html)** +## llmock vs MSW + +[MSW (Mock Service Worker)](https://mswjs.io/) patches `http`/`https`/`fetch` inside a single Node.js process. llmock runs a real HTTP server on a real port that any process can reach — child processes, microservices, agent workers, Docker containers. MSW can't intercept any of those; llmock can. For a detailed comparison including other tools, see the [full comparison on the docs site](https://llmock.copilotkit.dev/#comparison). + +| Capability | llmock | MSW | +| -------------------------- | ---------------------------- | ---------------------- | +| Cross-process interception | **Yes** (real server) | No (in-process only) | +| LLM SSE streaming | **Built-in** (13+ providers) | Manual for each format | +| Fixture files (JSON) | **Yes** | No (code-only) | +| Record & replay | **Yes** | No | +| WebSocket APIs | **Yes** | No | +| Zero dependencies | **Yes** | No (~300KB) | + ## Real-World Usage [CopilotKit](https://github.com/CopilotKit/CopilotKit) uses llmock across its test suite to verify AI agent behavior across multiple LLM providers without hitting real APIs. diff --git a/docs/a2a-mock.html b/docs/a2a-mock.html new file mode 100644 index 0000000..9346e3b --- /dev/null +++ b/docs/a2a-mock.html @@ -0,0 +1,279 @@ + + + + + + A2AMock — llmock + + + + + + + + +
+ + +
+

A2AMock

+

+ Mock A2A (Agent-to-Agent) protocol server for testing multi-agent systems. Implements the + A2A JSON-RPC protocol with agent card discovery, message routing, task management, and SSE + streaming. +

+ +

Quick Start

+
+
+ Standalone mode typescript +
+
import { A2AMock } from "@copilotkit/llmock";
+
+const a2a = new A2AMock();
+
+a2a.registerAgent({
+  name: "translator",
+  description: "Translates text between languages",
+  skills: [{ id: "translate", name: "Translate" }],
+});
+
+a2a.onMessage("translator", "translate", [{ text: "Translated text" }]);
+
+const url = await a2a.start();
+// Agent card at: ${url}/.well-known/agent-card.json
+// JSON-RPC at: ${url}/
+
+ +

Mounted Mode

+

+ Mount A2AMock onto an LLMock server to share a single port with LLM mocking and other + services: +

+
+
+ Mount on LLMock typescript +
+
import { LLMock, A2AMock } from "@copilotkit/llmock";
+
+const llm = new LLMock({ port: 5555 });
+const a2a = new A2AMock();
+
+a2a.registerAgent({ name: "assistant" });
+a2a.onMessage("assistant", "hello", [{ text: "Hi!" }]);
+
+llm.mount("/a2a", a2a);
+await llm.start();
+// A2A available at http://127.0.0.1:5555/a2a
+
+ +

Subpath Import

+

A2AMock is also available via a dedicated subpath import for tree-shaking:

+
+
+ Subpath import typescript +
+
import { A2AMock } from "@copilotkit/llmock/a2a";
+
+ +

Agent Registration

+

Register agents with skills and capabilities:

+
+
+ Register agents typescript +
+
a2a.registerAgent({
+  name: "researcher",
+  description: "Research assistant",
+  version: "1.0.0",
+  skills: [
+    { id: "search", name: "Web Search", tags: ["research"] },
+    { id: "summarize", name: "Summarize" },
+  ],
+  capabilities: { streaming: true },
+});
+
+ +

Message Patterns

+

Route messages to responses using string or RegExp patterns:

+
+
+ Message patterns typescript +
+
// String substring match
+a2a.onMessage("agent", "hello", [{ text: "Hi there!" }]);
+
+// RegExp match
+a2a.onMessage("agent", /^translate\s+(.+)/i, [{ text: "Translation result" }]);
+
+// Task with artifacts
+a2a.onTask("agent", "compute", [
+  { parts: [{ text: "42" }], name: "result" },
+]);
+
+ +

Streaming Tasks

+

Simulate streaming responses with SSE events:

+
+
Streaming typescript
+
a2a.onStreamingTask("agent", "long-task", [
+  { type: "status", state: "TASK_STATE_WORKING" },
+  { type: "artifact", parts: [{ text: "partial result" }], name: "output" },
+  { type: "artifact", parts: [{ text: "final result" }], lastChunk: true, name: "output" },
+], 50); // 50ms delay between events
+
+ +

Config File

+

A2AMock can be configured via the aimock JSON config file:

+
+
aimock.json json
+
{
+  "a2a": {
+    "path": "/a2a",
+    "agents": [
+      {
+        "name": "assistant",
+        "description": "A helpful assistant",
+        "skills": [{ "id": "chat", "name": "Chat" }],
+        "messages": [
+          { "pattern": "hello", "parts": [{ "text": "Hi there!" }] }
+        ]
+      }
+    ]
+  }
+}
+
+ +

JSON-RPC Methods

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MethodDescription
SendMessageSend a message, get a synchronous response
SendStreamingMessageSend a message, get an SSE stream of events
GetTaskRetrieve a task by ID
ListTasksList tasks, optionally filtered by contextId
CancelTaskCancel a non-terminal task
+ +

Agent Card

+

+ The agent card is served at GET /.well-known/agent-card.json and includes all + registered agents' skills and capabilities. The A2A-Version: 1.0 header is + included on all responses. +

+ +

Inspection

+
+
+ Inspection API typescript +
+
a2a.health();  // { status: "ok", agents: 2, tasks: 5 }
+a2a.reset();   // Clears all agents and tasks
+
+
+
+ + + diff --git a/docs/aimock-cli.html b/docs/aimock-cli.html new file mode 100644 index 0000000..e4a0b26 --- /dev/null +++ b/docs/aimock-cli.html @@ -0,0 +1,333 @@ + + + + + + aimock CLI — llmock + + + + + + + + +
+ + +
+

aimock CLI

+

+ aimock is the full-stack mock orchestrator. Where llmock serves + LLM endpoints only, aimock reads a JSON config file and serves LLM mocks + alongside additional mock services (MCP, A2A, vector stores) on a single port. +

+ +

aimock vs llmock

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Capabilityllmock CLIaimock CLI
LLM mock endpointsYesYes
Additional mock servicesNoYes (via mount)
Config fileCLI flags onlyJSON config file
Single-port routingLLM paths onlyAll services on one port
+ +

Quick Start

+ +
+
Run aimock bash
+
aimock --config aimock.json --port 4010
+
+ +

Config File Format

+

+ The config file is a JSON object describing which services to run and how to configure + them. The llm section configures the core LLMock server. Additional services + are mounted at path prefixes. +

+ +
+
aimock.json json
+
{
+  "llm": {
+    "fixtures": "./fixtures",
+    "latency": 0,
+    "chunkSize": 20,
+    "logLevel": "info",
+    "validateOnLoad": true,
+    "metrics": true,
+    "strict": false
+  },
+  "services": {
+    "/mcp": {
+      "type": "mcp",
+      "tools": "./mcp-tools.json"
+    },
+    "/a2a": {
+      "type": "a2a",
+      "agents": "./a2a-agents.json"
+    }
+  }
+}
+
+ +

Config Fields

+ + + + + + + + + + + + + + + + + + + + +
FieldTypeDescription
llmobject + LLMock configuration. Accepts fixtures, latency, + chunkSize, logLevel, validateOnLoad, + metrics, strict, chaos, + streamingProfile. +
servicesobject + Map of mount paths to service configs. Each key is a URL path prefix (e.g. + /mcp), each value describes the service type and its options. +
+ +

CLI Flags

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
OptionDefaultDescription
--configaimock.jsonPath to JSON config file
--port4010Port to listen on (overrides config)
--host127.0.0.1Host to bind to (overrides config)
--helpShow help
+ +

Single-Port Routing

+

+ All services share one port. Requests are routed by path prefix. LLM endpoints live at the + root, mounted services at their configured prefix: +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
PathService
/v1/chat/completionsLLMock (OpenAI Chat Completions)
/v1/messagesLLMock (Anthropic Claude)
/v1/embeddingsLLMock (Embeddings)
/mcp/*MCP mock service
/a2a/*A2A mock service
/healthUnified health check (all services)
/metricsPrometheus metrics (if enabled)
+ +

+ Path stripping is automatic — a request to /mcp/tools/list arrives at + the MCP service as /tools/list. +

+ +

Docker Usage

+ +
+
+ Docker run with config shell +
+
# Mount config and fixtures into the container
+docker run -p 4010:4010 \
+  -v $(pwd)/aimock.json:/app/aimock.json \
+  -v $(pwd)/fixtures:/app/fixtures \
+  ghcr.io/copilotkit/llmock aimock --config /app/aimock.json --host 0.0.0.0
+
+ +

Docker Compose

+ +
+
docker-compose.yml yaml
+
services:
+  aimock:
+    image: ghcr.io/copilotkit/llmock:latest
+    command: aimock --config /app/aimock.json --host 0.0.0.0
+    ports:
+      - "4010:4010"
+    volumes:
+      - ./aimock.json:/app/aimock.json:ro
+      - ./fixtures:/app/fixtures:ro
+
+  app:
+    build: .
+    environment:
+      OPENAI_BASE_URL: http://aimock:4010/v1
+      MCP_SERVER_URL: http://aimock:4010/mcp
+    depends_on:
+      - aimock
+
+
+
+ + + diff --git a/docs/aws-bedrock.html b/docs/aws-bedrock.html index 09cf238..d2886c7 100644 --- a/docs/aws-bedrock.html +++ b/docs/aws-bedrock.html @@ -69,11 +69,21 @@

Features

>WebSocket APIs Record & Replay Prometheus Metrics + Mount & Composition + + diff --git a/docs/azure-openai.html b/docs/azure-openai.html index c17a494..b86d677 100644 --- a/docs/azure-openai.html +++ b/docs/azure-openai.html @@ -69,11 +69,21 @@

Features

>WebSocket APIs Record & Replay Prometheus Metrics + Mount & Composition + + diff --git a/docs/chaos-testing.html b/docs/chaos-testing.html index e0dfc67..908cf90 100644 --- a/docs/chaos-testing.html +++ b/docs/chaos-testing.html @@ -69,11 +69,21 @@

Features

>WebSocket APIsRecord & ReplayPrometheus Metrics + Mount & Composition + + diff --git a/docs/chat-completions.html b/docs/chat-completions.html index 353d4f2..461eaa7 100644 --- a/docs/chat-completions.html +++ b/docs/chat-completions.html @@ -71,10 +71,20 @@

Features

WebSocket APIs Record & Replay Prometheus Metrics + Mount & Composition + + diff --git a/docs/claude-messages.html b/docs/claude-messages.html index d034278..4b26975 100644 --- a/docs/claude-messages.html +++ b/docs/claude-messages.html @@ -69,11 +69,21 @@

Features

>WebSocket APIs Record & Replay Prometheus Metrics + Mount & Composition + + diff --git a/docs/cohere.html b/docs/cohere.html index 162f738..718c3f0 100644 --- a/docs/cohere.html +++ b/docs/cohere.html @@ -69,11 +69,21 @@

Features

>WebSocket APIsRecord & ReplayPrometheus Metrics + Mount & Composition + + diff --git a/docs/compatible-providers.html b/docs/compatible-providers.html index 2bb2179..11b161e 100644 --- a/docs/compatible-providers.html +++ b/docs/compatible-providers.html @@ -69,11 +69,21 @@

Features

>WebSocket APIs Record & Replay Prometheus Metrics + Mount & Composition + + diff --git a/docs/docker.html b/docs/docker.html index 4f71441..19592cd 100644 --- a/docs/docker.html +++ b/docs/docker.html @@ -69,11 +69,21 @@

Features

>WebSocket APIs Record & Replay Prometheus Metrics + Mount & Composition + + diff --git a/docs/docs.html b/docs/docs.html index 7df43c9..2e300a0 100644 --- a/docs/docs.html +++ b/docs/docs.html @@ -77,10 +77,20 @@

Features

WebSocket APIs Record & Replay Prometheus Metrics + Mount & Composition + + diff --git a/docs/drift-detection.html b/docs/drift-detection.html index dc8f9d4..c7c7ec8 100644 --- a/docs/drift-detection.html +++ b/docs/drift-detection.html @@ -69,11 +69,21 @@

Features

>WebSocket APIs Record & Replay Prometheus Metrics + Mount & Composition + + diff --git a/docs/embeddings.html b/docs/embeddings.html index 00d06f0..9e72ee7 100644 --- a/docs/embeddings.html +++ b/docs/embeddings.html @@ -69,11 +69,21 @@

Features

>WebSocket APIs Record & Replay Prometheus Metrics + Mount & Composition + + diff --git a/docs/error-injection.html b/docs/error-injection.html index 80ac5ee..9e08369 100644 --- a/docs/error-injection.html +++ b/docs/error-injection.html @@ -70,11 +70,21 @@

Features

>WebSocket APIs Record & Replay Prometheus Metrics + Mount & Composition + + diff --git a/docs/fixtures.html b/docs/fixtures.html index 0a13382..f4006f2 100644 --- a/docs/fixtures.html +++ b/docs/fixtures.html @@ -70,11 +70,21 @@

Features

>WebSocket APIs Record & Replay Prometheus Metrics + Mount & Composition + + diff --git a/docs/gemini.html b/docs/gemini.html index b3beeb1..8ffbdc4 100644 --- a/docs/gemini.html +++ b/docs/gemini.html @@ -70,11 +70,21 @@

Features

>WebSocket APIs Record & Replay Prometheus Metrics + Mount & Composition + + diff --git a/docs/index.html b/docs/index.html index bc2e99e..11be12e 100644 --- a/docs/index.html +++ b/docs/index.html @@ -1034,6 +1034,7 @@
$ llmock
+ + @@ -303,6 +313,80 @@

Fixture Lifecycle

recordings +

Local Development Workflow

+

+ Record once against real APIs, then replay from fixtures for fast, offline development. +

+ +
+
Record then replay bash
+
# First run: record real API responses
+llmock --record --provider-openai https://api.openai.com -f ./fixtures
+
+# Subsequent runs: replay from recorded fixtures
+llmock -f ./fixtures
+
+ +

CI Pipeline Workflow

+

+ Use the Docker image in CI with --strict mode to ensure every request matches + a recorded fixture. No API keys needed, no flaky network calls. +

+ +
+
+ GitHub Actions example yaml +
+
- name: Start aimock
+  run: |
+    docker run -d --name aimock \
+      -v ./fixtures:/fixtures \
+      -p 4010:4010 \
+      ghcr.io/copilotkit/aimock \
+      llmock --strict -f /fixtures
+
+- name: Run tests
+  env:
+    OPENAI_BASE_URL: http://localhost:4010/v1
+  run: pnpm test
+
+- name: Stop aimock
+  run: docker stop aimock
+
+ +

Building Fixture Sets

+

A practical workflow for building and maintaining fixture sets:

+
    +
  1. Run with --record against real APIs during development
  2. +
  3. Review recorded fixtures in fixtures/recorded/
  4. +
  5. Move and rename to organized fixture directories
  6. +
  7. Switch to --strict mode in CI
  8. +
  9. Re-record when upstream APIs change (drift detection catches this)
  10. +
+ +

Cross-Language Testing

+

+ The Docker image serves any language that speaks HTTP. Point your client at the mock + server's URL instead of the real API. +

+ +
+
+ Any language, one server bash +
+
# Docker image serves all languages
+docker run -d -p 4010:4010 ghcr.io/copilotkit/aimock llmock -f /fixtures
+
+# Python
+import openai
+client = openai.OpenAI(base_url="http://localhost:4010/v1", api_key="mock")
+
+# Go
+client := openai.NewClient(option.WithBaseURL("http://localhost:4010/v1"))
+
+# Rust
+let client = Client::new().with_base_url("http://localhost:4010/v1");
+