feat(cli): add checkly init command with skill-first onboarding by thebiglabasky · Pull Request #1267 · checkly/checkly-cli

thebiglabasky · 2026-03-20T08:53:52Z

Summary

Adds npx checkly init — a new onboarding command that replaces npm create checkly@latest with a streamlined, AI-agent-aware flow. The skill installation is front and center: install the skill, get a tailored prompt copied to your clipboard, hand off to your agent.

npm create checkly@latest now shows a deprecation notice pointing to npx checkly init
npx checkly (no args) nudges toward init and skills install when no config is detected
Starter prompts are stored as .md files in src/ai-context/onboarding-prompts/ — easy to review and tweak without touching TypeScript
Clipboard integration uses native OS commands (pbcopy/xclip/xsel/clip) with silent fallback — no external dependency

Supported paths (all tested)

State	Skill?	Playwright?	Behavior
Pristine	Installs	No	Deps → base prompt → agent footer
Pristine	Installs	Yes	Deps → PW prompt → agent footer + pw-test/reporter hints
Pristine	Declines	No	Demo checks? → config → deps → manual footer
Pristine	Declines	Yes	Demo checks? → config → deps → PW hint → footer
Existing	Has skill	-	Refresh skill silently → concise next steps
Existing	Installs	-	Existing-project prompt → agent footer
Existing	Declines	-	Concise next steps
Agent mode	Auto	-	Clean JSON output, no prompts
CI mode	Skip	-	Config + deps silently

Reuses existing CLI utilities

findPlaywrightConfigPath from services/util.ts (extended with .mts/.mjs)
defaultFilenames from checkly-config-loader.ts
detectPackageManager for robust PM detection (lockfiles, config, runtime, user agent)
promptForPlatformTarget shared between skills install and init
logSymbols via shared successMessage() helper
PLATFORM_TARGETS — skill directories derived from it to prevent drift

Points of attention for reviewers

Prompts copy needs marketing review — all user-facing text is in messages.ts (shared blocks) and onboarding-prompts/*.md (agent prompts). Easy to change in one place.
Clipboard relies on OS commands — works on macOS/Linux (with xclip or xsel)/Windows. Falls back gracefully with a "copy the prompt below" message. No external npm dependency.
Boilerplate templates in src/ai-context/onboarding-boilerplate/ are excluded from tsconfig (not compiled) and vitest (not test files) — they're copied to dist at build time.
AI context references updated: AssertionBuilder method reference added, public URL requirement documented, plan-tier location restrictions noted, env var auth promoted as primary path, npm create checkly references updated to npx checkly init.

Test plan

Run through the 9 interactive paths in the table above
Test agent mode: CLAUDECODE=1 npx checkly init
Test npx checkly with no config — should show nudge
Test npm create checkly@latest — should show deprecation notice
Run npx vitest run — 740+ tests pass
Verify clipboard works (macOS at minimum)

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… installation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ages

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…tion

- Agent mode: clean JSON-only output with error envelope - CI mode: skip prompts, run non-interactively - Graceful Ctrl+C handling across all prompts - Error handling for file I/O, malformed package.json, missing templates - Better copy: skill install context, platform picker wording - Conditional clipboard label in expand prompt - Footer: add --record flag - Sanitize scoped package names in logicalId - Re-init warning when checkly config exists Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…tion

…ting without skill - Detect existing skill installation by scanning known platform directories - Existing Checkly projects: no boilerplate, no dep install, offer skill + AI prompt - Existing + skill already installed: refresh skill silently, show AI prompt directly - Brand new projects: separate prompts for boilerplate and dependency installation - Split boilerplate creation and dep install into independently callable functions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

… parsing Detects PM via lockfiles, config files, runtime, and executables — not just npm_config_user_agent which is empty when running checkly directly.

… the skill matters

…oy, upgrade

…lution

…ader

… projects Existing projects get contextual suggestions (test, deploy, skills) with a simple Enter to exit, instead of the new-user footer or a select menu.

… to npx checkly init

- Add AssertionBuilder method reference with all available methods (agents were guessing incorrect methods like .body()) - Add public URL requirement warnings to API checks, browser checks, and URL monitors (agents were generating localhost checks) - Add location/plan tier documentation to configure reference (agents were setting locations unavailable on free plans) - Promote env var auth as primary path in initialize reference (agents can't do browser OAuth) - Update npm create checkly references to npx checkly init Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…lled - Skill installed path: show prompt + agent footer with platform hint - No boilerplate or deps prompts when skill is installed - Existing projects: no deps install (they already have Checkly) - Manual fallback always shown in dimmed text Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…g detected

packages/cli/src/commands/skills/install.ts

github-actions · 2026-03-20T10:05:14Z

🎉 Experimental release successfully published on npm

npm install checkly@0.0.0-pr.1267.6c1ae81

MichaelHogers · 2026-03-20T10:15:27Z

packages/cli/src/ai-context/references/configure-api-checks.md

 - If endpoints require authentication ask the user which authentication method to use and then generate a setupScript to authenticate the given requests.
 - Referenced `setup-script.ts` and `teardown-script.ts` for API checks must be plain ts files and not export anything.
 - Check in the code if API endpoints require authentication.
+- **Important:** The target URL must be publicly accessible. Checks run on Checkly's cloud infrastructure, not locally. If the user is developing against localhost, suggest a tunneling tool (ngrok, cloudflare tunnel) or a preview/staging deployment.


works for me, just wondering if the dev workflow here was intentionally added

This comes from me testing locally and getting checks generated on localhost by default, failing, then asking about it. That part however is something I think is a bit too ad hoc. I wonder if we couldn't just make this a global-ish rule

Should we talk about the Checkly reporter instead? I always disliked the idea of bringing in a tunnel just for using the checkly stuff?

MichaelHogers · 2026-03-20T10:16:33Z

packages/cli/src/ai-context/references/configure-api-checks.md

+`AssertionBuilder` provides methods to assert on API responses. Available methods:
+
+- `AssertionBuilder.statusCode()` — Assert on HTTP status code. Example: `.equals(200)`, `.isGreaterThan(199)`
+- `AssertionBuilder.jsonBody(path)` — Assert on JSON body using JSONPath. Example: `AssertionBuilder.jsonBody('$.id').isNotNull()`
+- `AssertionBuilder.textBody()` — Assert on raw text body. Example: `.contains('OK')`
+- `AssertionBuilder.headers(name)` — Assert on response headers. Example: `AssertionBuilder.headers('content-type').contains('application/json')`
+- `AssertionBuilder.responseTime()` — Assert on response time in ms. Example: `.isLessThan(2000)`
+
+Each method returns a chainable builder with comparison methods: `.equals()`, `.notEquals()`, `.contains()`, `.notContains()`, `.isGreaterThan()`, `.isLessThan()`, `.isEmpty()`, `.isNotEmpty()`, `.isNull()`, `.isNotNull()`


unrelated to this PR @thebiglabasky but since you mentioned CLI support for agentic checks.

we'll be able to do something similar

MichaelHogers

LGTM

Looks quite alright, not a ton of time to look at details

…mode hint - npx checkly init --target claude: installs skill non-interactively - CI mode now hints about skills install at the end - Examples in --help show --target and CI=true usage Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…l package.json

thebiglabasky · 2026-03-20T14:47:29Z

packages/cli/src/commands/__tests__/init.spec.ts

+  successMessage: vi.fn((msg: string) => `OK ${msg}`),
+}))
+vi.mock('prompts', () => ({ default: vi.fn() }))
+vi.mock('fs', async importOriginal => {


critical to ensure we don't overwrite stuff during tests

stefanjudis · 2026-03-22T14:03:45Z

packages/cli/src/ai-context/onboarding-prompts/base.md

@@ -0,0 +1 @@
+Initialize Checkly in this project at {{projectPath}}. Set up monitoring checks tailored to this codebase — look at the project structure, existing tests, and endpoints to create relevant API checks, browser checks, or URL monitors.


Out f curiosity, why do we focus on browser checks and not playwright checks?

stefanjudis · 2026-03-22T14:07:56Z

packages/cli/src/ai-context/references/initialize.md


- **Option A: Interactive login** — The user runs `npx checkly login` themselves. This command opens a browser for OAuth authentication and cannot be completed by an AI agent. Tell the user to run the command, complete the browser flow, and let you know when they're done so you can re-run `npx checkly whoami` to verify.
- **Option B: Environment variables (recommended for agentic / CI use)** — The user sets `CHECKLY_API_KEY` and `CHECKLY_ACCOUNT_ID` as environment variables. They can create an API key in the Checkly dashboard under **User Settings > API Keys**. Once both variables are set, re-run `npx checkly whoami` to verify.
+- **Option A: Environment variables (recommended)** — The user sets `CHECKLY_API_KEY` and `CHECKLY_ACCOUNT_ID` as environment variables. They can create an API key in the Checkly dashboard under **User Settings > API Keys**. This is the recommended approach as it works in all contexts (agentic, CI/CD, and interactive). Once both variables are set, re-run `npx checkly whoami` to verify.


can we add a URL to API keys to prompt the user directly?

stefanjudis · 2026-03-22T14:12:59Z

packages/cli/src/ai-context/onboarding-boilerplate/__checks__/url.check.ts

@@ -0,0 +1,15 @@
+import { UrlAssertionBuilder, UrlMonitor } from 'checkly/constructs'
+


Nit: out of consistency would be nice to have a comment here too. 🫣

stefanjudis · 2026-03-22T14:19:03Z

packages/cli/src/helpers/onboarding/boilerplate.ts

+    || 'checkly-project'
+}
+
+async function detectPM (projectDir: string): Promise<{ name: string, installCmd: string }> {


hardcore nit: i'd appreciate detectPackageManager here. I see the variable name clash but would then rename the imported one and expose one with an expressive name. 🫣

stefanjudis · 2026-03-22T14:24:48Z

packages/cli/src/helpers/onboarding/messages.ts

+  return [
+    ...playwrightBlock(),
+    '',
+    chalk.cyan(`  An AI agent can help configure this — run ${chalk.bold('npx checkly skills install')}`),


If I get the context correctly, this message isn't great. An "empty this" is never great and I'm unsure how npx checkly skills install helps directly. Shouldn't we be more explicit here?

stefanjudis · 2026-03-22T14:26:24Z

packages/cli/src/helpers/onboarding/skill-install.ts

+  log: (msg: string) => void,
+): Promise<SkillInstallResult> {
+  log('')
+  log(`  Checkly is an ${chalk.bold('AI-native monitoring as code')} platform.`)


Suggested change

log(` Checkly is an ${chalk.bold('AI-native monitoring as code')} platform.`)

log(` Checkly is an ${chalk.bold('AI-native Monitoring as Code')} platform.`)

Here i actually care about the casing. :D

stefanjudis · 2026-03-22T14:28:39Z

packages/cli/package.json

    "clean": "npm run clean:dist && npm run clean:gen",
    "prepack": "npx oclif manifest",
-    "prepare:ai-context": "cross-env CHECKLY_SKIP_AUTH=1 CHECKLY_CLI_VERSION=99.0.0 ./bin/run import plan --root gen --debug-import-plan-input-file ./src/ai-context/context.fixtures.json && ts-node ./scripts/prepare-ai-context.ts",
+    "prepare:ai-context": "cross-env CHECKLY_SKIP_AUTH=1 CHECKLY_CLI_VERSION=99.0.0 ./bin/run import plan --root gen --debug-import-plan-input-file ./src/ai-context/context.fixtures.json && ts-node ./scripts/prepare-ai-context.ts && cp -r src/ai-context/onboarding-boilerplate dist/ai-context/onboarding-boilerplate && cp -r src/ai-context/onboarding-prompts dist/ai-context/onboarding-prompts",


Could we have the copy logic in prepare-ai-context? I think it already copies stuff around. :) Would also keep the package.json cleaner.

stefanjudis · 2026-03-22T14:30:13Z

packages/create-cli/e2e/__tests__/bootstrap.spec.ts

Generally, I think it's very bold to touch the "standard" init flow in one go. Personally, I'd split this into two stages because you might get more feedback from a released init command. But yeah, no strong feelings if you wanna go fast.

stefanjudis

I've spent 30min reviewing the code. I'll play around with it to see how it feels and approve if everything feels great.

stefanjudis · 2026-03-22T14:48:43Z

Nit: should we decide on the line length in the print? I could see this fighting back for folks running in a heavily split view?

I'm not sure how I feel about these new "lines" and "line dividers"...

I also had trouble understanding what I was looking at. I "think" I'd appreciate a manual step with some explanations to make things very clear. People only read the last line so I wonder how many people don't see it on first try.

#pseudo code
Use the provided example prompt to set up Checkly.
Wanna copy prompt to clipboard (Y/n)?

Great. Paste it and see the magic.

This footer feels a bit like a deadend to me.

I'm not really sure what to do with it. Having at least a checkly.config created or something. This feels like "use AI or we leave you hanging".

When running the provided install command, I'm seeing this.

This is a bit odd because I've just run npx checkly init. 🤔

It also runs install again.

It's not a huge deal, but we might want to add a guard for this?

FYI It hallucinated this.

Confirmation Protocol

Write commands (e.g. incidents create, deploy, destroy) return exit code 2 with a confirmation_required JSON envelope instead of executing. Always present the changes to the user and wait for approval before running the confirmCommand. Never auto-append --force. This applies to every write command individually — updates and resolutions need confirmation too, not just the initial create.

I'm unsure if this is expected but it happily deployed and destroyed my checkly stuff when I told it to. I'm unsure if this is expected. In my demos with similar safety guardrails, Claude usually goes "Are you sure I should deploy now"?

I see that we're branching out depending on the question, if a person wants to use the AI skills? I found this a bit surprising. In my mind, the question of whether I want to use an agentic setup flow should be called out very explicitly.

# Pseudo code
Do you want your agent to set up your Checkly project? (Y/n) Y
Your agent will perform best with an installed Checkly skills. Install? (Y/n)

Alrighty. :) I think this is a great step forward! 👏 But right now I'm not very ready to approve bc after playing with it for an hour, I think we can polish some more things. But great work!

And yeah, before deprecating the old way of installing things, I'd really ship this "secretly" and let solutions play with it...

thebiglabasky and others added 30 commits March 19, 2026 16:59

refactor: extract reusable skill install utilities for init command

a55900e

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

feat(init): add cross-platform clipboard utility

6be9c07

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(init): add project context detection utility

599e114

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(init): add markdown prompt templates with variable replacement

e6ff100

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(init): add starter prompt display with clipboard and preview/expand

0178e40

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(init): add skill installation step with interactive and agent modes

d0d623b

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

feat(init): bundle boilerplate config template and example checks

cd54098

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(init): add boilerplate setup with config, checks, and dependency…

978ad40

… installation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

feat(init): add onboarding greeting, footer, and playwright hint mess…

894cb65

…ages

feat(cli): add checkly init command with skill-first onboarding flow

c19aabb

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: exclude onboarding boilerplate templates from vitest test collec…

e80c174

…tion

fix: exclude onboarding boilerplate templates from TypeScript compila…

17f8175

…tion

fix(init): use CLI's detectPackageManager instead of naive user-agent…

f43f349

… parsing Detects PM via lockfiles, config files, runtime, and executables — not just npm_config_user_agent which is empty when running checkly directly.

feat(init): add raccoon logo banner with Checkly brand colors

a366d6d

feat(init): improve onboarding copy — explain what Checkly is and why…

0d99ee8

… the skill matters

feat(init): improve raccoon logo based on SVG reference

d7a8078

feat(init): add quick actions menu for existing projects — test, depl…

2281286

…oy, upgrade

fix(init): add .js extension to dynamic import for node16 module reso…

c9304bc

…lution

feat(init): simplify greeting — drop ASCII art, keep clean branded he…

438a7ff

…ader

feat(init): replace action menu with tailored next steps for existing…

a355d4f

… projects Existing projects get contextual suggestions (test, deploy, skills) with a simple Enter to exit, instead of the new-user footer or a select menu.

fix(init): align description columns in next steps output

b6c31a7

feat(init): strip prompt and starter prompt from existing project flow

a3c2eb3

test(init): update test for existing project flow — no prompt display

1361e1c

feat(init): ask before creating boilerplate in deterministic path

fa8dcfe

feat(init): ask before creating files, rename boilerplate to demo checks

d23ff61

feat(create-cli): replace onboarding with deprecation notice pointing…

fea5e63

… to npx checkly init

thebiglabasky added 3 commits March 20, 2026 09:48

feat(cli): nudge toward init and skills install when no checkly confi…

338526c

…g detected

fix: align help nudge formatting with oclif help style

f7f55cc

style(init): tweak copy styling — cyan URLs, spacing adjustments

1afd1d8

thebiglabasky commented Mar 20, 2026

View reviewed changes

packages/cli/src/commands/skills/install.ts Show resolved Hide resolved

fix: remove unused Bundler import from util.ts

6c1ae81

thebiglabasky added the build Issue regarding building and packaging label Mar 20, 2026

thebiglabasky requested review from MichaelHogers, sorccu and stefanjudis March 20, 2026 10:03

MichaelHogers reviewed Mar 20, 2026

View reviewed changes

MichaelHogers previously approved these changes Mar 20, 2026

View reviewed changes

test(create-cli): update e2e tests for deprecation notice

f2a7bd9

thebiglabasky dismissed MichaelHogers’s stale review via f2a7bd9 March 20, 2026 10:32

thebiglabasky and others added 4 commits March 20, 2026 11:57

test(cli): add init command to help e2e test expectations

75b926c

feat(init): offer to create package.json when missing instead of exiting

95931b2

fix(init): mock writeFileSync in init test to prevent overwriting rea…

b2e170d

…l package.json

thebiglabasky commented Mar 20, 2026

View reviewed changes

stefanjudis reviewed Mar 22, 2026

View reviewed changes

		@@ -0,0 +1 @@
		Initialize Checkly in this project at {{projectPath}}. Set up monitoring checks tailored to this codebase — look at the project structure, existing tests, and endpoints to create relevant API checks, browser checks, or URL monitors.

		@@ -0,0 +1,15 @@
		import { UrlAssertionBuilder, UrlMonitor } from 'checkly/constructs'

	log(` Checkly is an ${chalk.bold('AI-native monitoring as code')} platform.`)
	log(` Checkly is an ${chalk.bold('AI-native Monitoring as Code')} platform.`)

Conversation

thebiglabasky commented Mar 20, 2026

Summary

Supported paths (all tested)

Reuses existing CLI utilities

Points of attention for reviewers

Test plan

Uh oh!

Uh oh!

github-actions bot commented Mar 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MichaelHogers left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stefanjudis left a comment

Choose a reason for hiding this comment

Uh oh!

stefanjudis commented Mar 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confirmation Protocol

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

stefanjudis commented Mar 22, 2026 •

edited

Loading