diff --git a/.agents/skills/agent-testing/SKILL.md b/.agents/skills/agent-testing/SKILL.md index 29f0334e57..65862dc1b9 100644 --- a/.agents/skills/agent-testing/SKILL.md +++ b/.agents/skills/agent-testing/SKILL.md @@ -157,10 +157,12 @@ full-stack `dev` command so Next can proxy the SPA HTML from Vite: Useful subcommands: ```bash -./.agents/skills/agent-testing/scripts/init-dev-env.sh env # print exports -./.agents/skills/agent-testing/scripts/init-dev-env.sh write # write .records/env/agent-testing-dev.env -./.agents/skills/agent-testing/scripts/init-dev-env.sh migrate # migrations only -./.agents/skills/agent-testing/scripts/init-dev-env.sh clean-db # remove managed DB container +./.agents/skills/agent-testing/scripts/init-dev-env.sh env # print exports +./.agents/skills/agent-testing/scripts/init-dev-env.sh write # write .records/env/agent-testing-dev.env +./.agents/skills/agent-testing/scripts/init-dev-env.sh migrate # migrations only +./.agents/skills/agent-testing/scripts/init-dev-env.sh seed-user # seed user + CLI API key +./.agents/skills/agent-testing/scripts/init-dev-env.sh qstash # local QStash for workflow paths +./.agents/skills/agent-testing/scripts/init-dev-env.sh clean-db # remove managed DB container ``` Default script env: @@ -169,14 +171,18 @@ Default script env: - `DATABASE_URL=postgresql://postgres:postgres@localhost:5433/postgres` - `DATABASE_DRIVER=node` - `FEATURE_FLAGS=-agent_self_iteration` so local smoke does not require QStash +- Local QStash defaults (`QSTASH_URL`, `QSTASH_TOKEN`, signing keys) are exported; + run `init-dev-env.sh qstash` in a separate terminal when the path under test + triggers QStash/Workflow. - `KEY_VAULTS_SECRET`, `AUTH_SECRET`, auth verification off - S3 mock vars - Managed DB container: `lobehub-agent-testing-postgres` `seed-user` creates `agent-testing@lobehub.com` / `TestPassword123!` with -onboarding already completed for manual or agent-browser checks. When running -Cucumber against this dev server, pass the same script env into the test process -too; Cucumber has its own `BeforeAll` seed path and it must see `DATABASE_URL` +onboarding already completed, plus a local API key in +`.records/env/agent-testing-cli.env` for CLI automation. When running Cucumber +against this dev server, pass the same script env into the test process too; +Cucumber has its own `BeforeAll` seed path and it must see `DATABASE_URL` instead of silently skipping setup: ```bash @@ -199,22 +205,23 @@ Electron login state unless the test spans those surfaces. Use `status` with no `--surface` only for cross-surface test plans. -| Surface | Mechanism | One-key path | Standard check | -| -------- | ------------------------------------------------- | ------------------------------ | ----------------------------------------- | -| CLI | OIDC Device Code Flow (`apps/cli/.lobehub-dev`) | `setup-auth.sh cli` | `setup-auth.sh status --surface cli` | -| Web | better-auth cookie injection into `agent-browser` | `pbpaste \| setup-auth.sh web` | `setup-auth.sh status --surface web` | -| Electron | App's own persistent login state | Log in once in the app | `setup-auth.sh status --surface electron` | -| Bot | Native apps already logged in | — | per-platform screenshot | +| Surface | Mechanism | One-key path | Standard check | +| -------- | --------------------------------------------- | ------------------------ | ----------------------------------------- | +| CLI | Seeded API key, device-code fallback | `setup-auth.sh cli-seed` | `setup-auth.sh status --surface cli` | +| Web | Seeded better-auth login into `agent-browser` | `setup-auth.sh web-seed` | `setup-auth.sh status --surface web` | +| Electron | App's own persistent login state | Log in once in the app | `setup-auth.sh status --surface electron` | +| Bot | Native apps already logged in | — | per-platform screenshot | Login-state checks are standardized — do NOT hand-roll `window.__LOBE_STORES` eval snippets; use `scripts/app-probe.sh auth` (returns `{ isSignedIn, userId }`, works for Electron CDP and web sessions via `AB_TARGET`). For Web tests, the test surface is always `agent-browser --session lobehub-dev`. -The user's normal Chrome is only a source for copying the Cookie header when -`status --surface web` fails. If Chrome is already logged in, do not open a -login page; verify agent-browser first, then request the Network `Cookie:` -header only if that verification fails. Full background and failure modes: +Use `setup-auth.sh web-seed` first in the seeded local env. The user's normal +Chrome is only a source for copying the Cookie header when seed auth is not +available or `status --surface web` still fails. If Chrome is already logged in, +do not open a login page; verify agent-browser first, then request the Network +`Cookie:` header only if that verification fails. Full background and failure modes: [references/auth.md](./references/auth.md). ## Step 1 — Pick the surface by change scope diff --git a/.agents/skills/agent-testing/cli/index.md b/.agents/skills/agent-testing/cli/index.md index d829452b55..789f2d54ce 100644 --- a/.agents/skills/agent-testing/cli/index.md +++ b/.agents/skills/agent-testing/cli/index.md @@ -13,17 +13,18 @@ flakiness. ## Prerequisites -| Requirement | Details | -| ------------ | --------------------------------------------------------------------------------- | -| Dev server | `localhost:3010` — see [../references/dev-server.md](../references/dev-server.md) | +| Requirement | Details | +| ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- | +| Dev server | `localhost:3010` — see [../references/dev-server.md](../references/dev-server.md) | | CLI source | `apps/cli/` — runs from source, no rebuild; standalone `node_modules` — run `pnpm install` inside `apps/cli/` (root install does not cover it) | -| CLI dev mode | `LOBEHUB_CLI_HOME=.lobehub-dev` for isolated credentials | -| Auth | Device Code Flow login — see [../references/auth.md](../references/auth.md) | +| CLI dev mode | `LOBEHUB_CLI_HOME=.lobehub-dev` for isolated settings | +| Auth | Seeded API key first; Device Code Flow only as fallback — see [../references/auth.md](../references/auth.md) | All CLI dev commands run from `apps/cli/`. Subsequent examples use `$CLI`: ```bash -CLI="LOBEHUB_CLI_HOME=.lobehub-dev bun src/index.ts" +source ../../.records/env/agent-testing-cli.env +CLI="bun src/index.ts" ``` ## Workflow @@ -39,14 +40,23 @@ check, start, and restart commands. Server-side code changes require a restart. ./.agents/skills/agent-testing/scripts/setup-auth.sh status ``` -If the CLI is not logged in, **the user must run the login themselves** -(interactive browser authorization): +If the CLI is not ready in the seeded local environment: + +```bash +./.agents/skills/agent-testing/scripts/init-dev-env.sh seed-user +source .records/env/agent-testing-cli.env +./.agents/skills/agent-testing/scripts/setup-auth.sh cli-seed +``` + +If the target environment is not seeded, use the interactive fallback: ```bash cd apps/cli && LOBEHUB_CLI_HOME=.lobehub-dev bun src/index.ts login --server http://localhost:3010 ``` -Credentials persist in `apps/cli/.lobehub-dev/`. Details: +Seeded API-key auth does not store credentials. It writes local settings under +`$HOME/.lobehub-dev` and requires the generated env file to be sourced before +CLI commands. Details: [../references/auth.md](../references/auth.md). ### Step 3 — Test with CLI commands @@ -133,10 +143,10 @@ $CLI provider test ## Troubleshooting -| Issue | Solution | -| --------------------------- | ----------------------------------------------- | -| `No authentication found` | Run `login --server http://localhost:3010` | -| `UNAUTHORIZED` on API calls | Token expired; re-run login | -| `ECONNREFUSED` | Dev server not running — see dev-server.md | -| CLI shows old data/behavior | Server needs restart to pick up code changes | -| Login opens wrong server | Must use `--server` flag (env var doesn't work) | +| Issue | Solution | +| --------------------------- | ------------------------------------------------------------------------------------------------------ | +| `No authentication found` | Source `.records/env/agent-testing-cli.env`, or run device-code `login --server http://localhost:3010` | +| `UNAUTHORIZED` on API calls | Re-run `init-dev-env.sh seed-user` and re-source the env file; for device-code fallback, re-run login | +| `ECONNREFUSED` | Dev server not running — see dev-server.md | +| CLI shows old data/behavior | Server needs restart to pick up code changes | +| Login opens wrong server | Must use `--server` flag (env var doesn't work) | diff --git a/.agents/skills/agent-testing/references/auth.md b/.agents/skills/agent-testing/references/auth.md index f07f19a6ad..d262ad4723 100644 --- a/.agents/skills/agent-testing/references/auth.md +++ b/.agents/skills/agent-testing/references/auth.md @@ -19,8 +19,10 @@ Quick reference after initialization: | ------------------------------ | -------------------------------------------------- | | `$SCRIPT status` | Check all surfaces (server + CLI + web + Electron) | | `$SCRIPT status --surface web` | Check only the Web surface gate | +| `$SCRIPT cli-seed` | Configure CLI API-key auth from the seeded key | | `$SCRIPT cli` | Interactive CLI device-code login (user must run) | | `$SCRIPT open-chrome` | Open Chrome at `SERVER_URL` with DevTools | +| `$SCRIPT web-seed` | Sign in the seeded user and inject cookies | | `pbpaste \| $SCRIPT web` | Inject a copied Cookie header into agent-browser | | `$SCRIPT web-verify` | Live-check agent-browser session auth | @@ -29,20 +31,42 @@ not `127.0.0.1`. ## Per-surface overview -| Surface | Mechanism | Persistence | Human interaction | -| -------- | ---------------------------------------- | ----------------------------------------------------------------- | ----------------------------------------------- | -| CLI | OIDC Device Code Flow | `apps/cli/.lobehub-dev/settings.json` | Yes — browser authorization, every token expiry | -| Web | better-auth cookie injection | `~/.lobehub-agent-testing/web-state.json` + agent-browser session | Copy the Cookie header once per token rotation | -| Electron | App's own login state | Electron user-data dir | Log in once manually in the app | -| Bot | Native apps (Discord/WeChat/…) logged in | Each app's own session | Once per app | +| Surface | Mechanism | Persistence | Human interaction | +| -------- | ---------------------------------------- | ----------------------------------------------------------------- | ---------------------------------------------- | +| CLI | Seeded API key or OIDC Device Code Flow | `.records/env/agent-testing-cli.env` + `$HOME/.lobehub-dev` | No for seed path; yes for device-code fallback | +| Web | Seeded better-auth login or cookie copy | `~/.lobehub-agent-testing/web-state.json` + agent-browser session | No for seed path; copy cookie only as fallback | +| Electron | App's own login state | Electron user-data dir | Log in once manually in the app | +| Bot | Native apps (Discord/WeChat/…) logged in | Each app's own session | Once per app | -## CLI — Device Code Flow +## CLI — Seeded API key +For the self-contained no-root-`.env` dev environment, seed the baseline user +and API key once: + +```bash +./.agents/skills/agent-testing/scripts/init-dev-env.sh seed-user +source .records/env/agent-testing-cli.env +./.agents/skills/agent-testing/scripts/setup-auth.sh cli-seed +``` + +The seed step writes `LOBE_API_KEY` for humans and maps it to the CLI's current +auth variable, `LOBEHUB_CLI_API_KEY`. It also sets `LOBEHUB_SERVER` so CLI +commands hit the local server without needing a stored device-code token. + +Use this for automated CLI verification: + +```bash +cd apps/cli +source ../../.records/env/agent-testing-cli.env +bun src/index.ts +``` + +## CLI — Device Code Flow fallback + +Use device-code login only when testing against a non-seeded environment. Credentials are isolated from the user's real CLI config via -`LOBEHUB_CLI_HOME=.lobehub-dev` (kept inside `apps/cli/`, gitignored). - -Login requires interactive browser authorization, so **the user must run it -themselves** (e.g. via the `!` prefix in Claude Code): +`LOBEHUB_CLI_HOME=.lobehub-dev`, which the current CLI stores under +`$HOME/.lobehub-dev`. ```bash cd apps/cli && LOBEHUB_CLI_HOME=.lobehub-dev bun src/index.ts login --server http://localhost:3010 @@ -51,16 +75,31 @@ cd apps/cli && LOBEHUB_CLI_HOME=.lobehub-dev bun src/index.ts login --server htt - The `--server` flag is required — an env var does NOT work and login will hit the wrong server without it. - Check state without logging in: `setup-auth.sh status` (verifies - `settings.json` exists and `serverUrl` matches). + `LOBEHUB_CLI_API_KEY` when present, otherwise checks the stored server URL). - `UNAUTHORIZED` on API calls means the token expired — re-run login. -## Web — better-auth cookie injection (agent-browser) +## Web — seeded better-auth login The Web test surface is `agent-browser --session lobehub-dev`. The user's ordinary Chrome is only a cookie source; Chrome screenshots, Chrome Network records, and Chrome logged-in state do not prove the agent-browser test session is authenticated. +For the seeded local dev environment, use the automatic path: + +```bash +./.agents/skills/agent-testing/scripts/init-dev-env.sh seed-user +./.agents/skills/agent-testing/scripts/setup-auth.sh web-seed +``` + +`web-seed` posts the seeded email/password to +`/api/auth/sign-in/email`, stores the returned cookie jar under +`~/.lobehub-agent-testing/`, converts it to Playwright `storageState`, loads it +into the `agent-browser` session, and verifies the session does not land on +`/signin`. + +## Web — manual cookie injection fallback + `agent-browser --headed` on macOS often creates the Chromium window off-screen — the user can't see or interact with it, so manual login inside the agent-browser session fails. Instead, copy the **better-auth session cookie** out of the @@ -72,9 +111,10 @@ secret: don't paste it into shared logs, PRs, or commit it anywhere. ### Web — decision flow 1. `$SCRIPT status --surface web` — green? Start testing. Do not ask for a Cookie header. -2. Not green → `$SCRIPT open-chrome` opens Chrome at `SERVER_URL` with DevTools. -3. User copies the `Cookie:` header from Network tab → any same-origin request → Request Headers → right-click `Cookie:` → **Copy value**. Must be from Network, NOT `document.cookie` (HttpOnly cookies are invisible to `document.cookie`). -4. `pbpaste | $SCRIPT web` — filters to better-auth cookies (`session_token`, `session_data`, `state`), builds Playwright `storageState`, loads it into the `agent-browser` session (`lobehub-dev`), opens `SERVER_URL`, and asserts the URL is not `/signin`. +2. Not green and using the seeded local env → `$SCRIPT web-seed`. +3. Still not green or not using the seed env → `$SCRIPT open-chrome` opens Chrome at `SERVER_URL` with DevTools. +4. User copies the `Cookie:` header from Network tab → any same-origin request → Request Headers → right-click `Cookie:` → **Copy value**. Must be from Network, NOT `document.cookie` (HttpOnly cookies are invisible to `document.cookie`). +5. `pbpaste | $SCRIPT web` — filters to better-auth cookies (`session_token`, `session_data`, `state`), builds Playwright `storageState`, loads it into the `agent-browser` session (`lobehub-dev`), opens `SERVER_URL`, and asserts the URL is not `/signin`. ### Using the authenticated session diff --git a/.agents/skills/agent-testing/references/dev-server.md b/.agents/skills/agent-testing/references/dev-server.md index bcadb1f4d6..17514e36fb 100644 --- a/.agents/skills/agent-testing/references/dev-server.md +++ b/.agents/skills/agent-testing/references/dev-server.md @@ -60,6 +60,9 @@ bun run dev # Without root .env: ./.agents/skills/agent-testing/scripts/init-dev-env.sh dev +# Local QStash. Run in a separate terminal only when testing workflow paths. +./.agents/skills/agent-testing/scripts/init-dev-env.sh qstash + # Restart — required to pick up server-side code changes lsof -ti:"$PORT" | xargs kill pnpm run dev:next @@ -83,8 +86,13 @@ in doubt. ## Troubleshooting -| Issue | Solution | -| ------------------------- | ------------------------------------------------------- | -| `ECONNREFUSED` | Server not running — start it | -| `EADDRINUSE` on the port | Already running — `lsof -ti: \| xargs kill` first | -| Stale data / old behavior | Server needs a restart to pick up code changes | +| Issue | Solution | +| ------------------------- | --------------------------------------------------------------------------------------------- | +| `ECONNREFUSED` | Server not running — start it | +| `EADDRINUSE` on the port | Already running — `lsof -ti: \| xargs kill` first | +| Stale data / old behavior | Server needs a restart to pick up code changes | +| QStash workflow failures | Start `init-dev-env.sh qstash` and make sure dev server inherited the script's `QSTASH_*` env | + +Marketplace/community endpoints are not part of the local agent-testing auth +gate. Do not block local product-chain verification on marketplace API auth +unless the change explicitly targets marketplace behavior. diff --git a/.agents/skills/agent-testing/scripts/init-dev-env.sh b/.agents/skills/agent-testing/scripts/init-dev-env.sh index 915bb49bcd..ec90140d4a 100755 --- a/.agents/skills/agent-testing/scripts/init-dev-env.sh +++ b/.agents/skills/agent-testing/scripts/init-dev-env.sh @@ -14,13 +14,14 @@ # init-dev-env.sh write [file] # write a source-able env file # init-dev-env.sh setup-db # start local Postgres and run migrations # init-dev-env.sh migrate # run DB migrations against the configured DB -# init-dev-env.sh seed-user # seed the baseline test user +# init-dev-env.sh seed-user # seed the baseline test user + CLI API key +# init-dev-env.sh qstash # run local Upstash QStash dev server # init-dev-env.sh dev-next # exec `pnpm run dev:next` with this env # init-dev-env.sh dev # exec `bun run dev` with this env # init-dev-env.sh clean-db # remove the managed Postgres container # # Overrides: -# SERVER_PORT=3010 DB_PORT=5433 DB_CONTAINER=lobehub-agent-testing-postgres +# SERVER_PORT=3010 DB_PORT=5433 DB_CONTAINER=lobehub-agent-testing-postgres QSTASH_DEV_PORT=8080 set -euo pipefail @@ -32,6 +33,12 @@ DB_PORT="${DB_PORT:-5433}" DB_CONTAINER="${DB_CONTAINER:-lobehub-agent-testing-postgres}" DATABASE_URL="${DATABASE_URL:-postgresql://postgres:postgres@localhost:${DB_PORT}/postgres}" ENV_FILE_DEFAULT="$REPO_ROOT/.records/env/agent-testing-dev.env" +CLI_ENV_FILE_DEFAULT="$REPO_ROOT/.records/env/agent-testing-cli.env" +AGENT_TESTING_API_KEY="${AGENT_TESTING_API_KEY:-sk-lh-agenttesting0001}" +QSTASH_DEV_PORT="${QSTASH_DEV_PORT:-8080}" +QSTASH_LOCAL_TOKEN="${QSTASH_LOCAL_TOKEN:-eyJVc2VySUQiOiJkZWZhdWx0VXNlciIsIlBhc3N3b3JkIjoiZGVmYXVsdFBhc3N3b3JkIn0=}" +QSTASH_LOCAL_CURRENT_SIGNING_KEY="${QSTASH_LOCAL_CURRENT_SIGNING_KEY:-sig_7kYjw48mhY7kAjqNGcy6cr29RJ6r}" +QSTASH_LOCAL_NEXT_SIGNING_KEY="${QSTASH_LOCAL_NEXT_SIGNING_KEY:-sig_5ZB6DVzB1wjE8S6rZ7eenA8Pdnhs}" ok() { printf ' \033[32m✔\033[0m %s\n' "$1"; } bad() { printf ' \033[31m✘\033[0m %s\n' "$1"; } @@ -57,6 +64,11 @@ apply_env() { export NEXT_PUBLIC_AUTH_EMAIL_VERIFICATION="${NEXT_PUBLIC_AUTH_EMAIL_VERIFICATION:-0}" export NODE_OPTIONS="${NODE_OPTIONS:---max-old-space-size=6144}" export PORT="${PORT:-$SERVER_PORT}" + export QSTASH_CURRENT_SIGNING_KEY="${QSTASH_CURRENT_SIGNING_KEY:-$QSTASH_LOCAL_CURRENT_SIGNING_KEY}" + export QSTASH_DEV_PORT + export QSTASH_NEXT_SIGNING_KEY="${QSTASH_NEXT_SIGNING_KEY:-$QSTASH_LOCAL_NEXT_SIGNING_KEY}" + export QSTASH_TOKEN="${QSTASH_TOKEN:-$QSTASH_LOCAL_TOKEN}" + export QSTASH_URL="${QSTASH_URL:-http://127.0.0.1:${QSTASH_DEV_PORT}}" export S3_ACCESS_KEY_ID="${S3_ACCESS_KEY_ID:-agent-testing-access-key}" export S3_BUCKET="${S3_BUCKET:-agent-testing-bucket}" export S3_ENDPOINT="${S3_ENDPOINT:-https://agent-testing-s3.localhost}" @@ -75,6 +87,11 @@ env_keys() { NEXT_PUBLIC_AUTH_EMAIL_VERIFICATION \ NODE_OPTIONS \ PORT \ + QSTASH_CURRENT_SIGNING_KEY \ + QSTASH_DEV_PORT \ + QSTASH_NEXT_SIGNING_KEY \ + QSTASH_TOKEN \ + QSTASH_URL \ S3_ACCESS_KEY_ID \ S3_BUCKET \ S3_ENDPOINT \ @@ -148,9 +165,14 @@ migrate_db() { seed_user() { apply_env + export AGENT_TESTING_API_KEY + export AGENT_TESTING_CLI_ENV_FILE="${AGENT_TESTING_CLI_ENV_FILE:-$CLI_ENV_FILE_DEFAULT}" cd "$REPO_ROOT" node <<'NODE' const bcrypt = require('bcryptjs'); +const crypto = require('node:crypto'); +const fs = require('node:fs'); +const path = require('node:path'); const pg = require('pg'); const databaseUrl = process.env.DATABASE_URL; @@ -166,13 +188,72 @@ const TEST_USER = { username: 'agent_testing_user', }; +const TEST_API_KEY = { + id: 'api_key_agent_testing_001', + key: process.env.AGENT_TESTING_API_KEY || 'sk-lh-agenttesting0001', + name: 'Agent Testing CLI API Key', +}; + +const validateApiKeyFormat = (apiKey) => /^sk-lh-[\da-z]{16}$/.test(apiKey); + +const hashApiKey = (apiKey) => { + const secret = process.env.KEY_VAULTS_SECRET; + if (!secret) throw new Error('KEY_VAULTS_SECRET is required to seed the baseline API key.'); + + return crypto.createHmac('sha256', secret).update(apiKey).digest('hex'); +}; + +const encryptWithKeyVaultsSecret = (plaintext) => { + const secret = process.env.KEY_VAULTS_SECRET; + if (!secret) throw new Error('KEY_VAULTS_SECRET is required to seed the baseline API key.'); + + const rawKey = Buffer.from(secret, 'base64'); + if (![16, 24, 32].includes(rawKey.length)) { + throw new Error( + `KEY_VAULTS_SECRET must decode to 16, 24, or 32 bytes, got ${rawKey.length} bytes.`, + ); + } + + const iv = crypto.randomBytes(12); + const cipher = crypto.createCipheriv(`aes-${rawKey.length * 8}-gcm`, rawKey, iv); + const encrypted = Buffer.concat([cipher.update(plaintext, 'utf8'), cipher.final()]); + const authTag = cipher.getAuthTag(); + + return `${iv.toString('hex')}:${authTag.toString('hex')}:${encrypted.toString('hex')}`; +}; + +const writeCliEnvFile = () => { + const file = process.env.AGENT_TESTING_CLI_ENV_FILE || '.records/env/agent-testing-cli.env'; + fs.mkdirSync(path.dirname(file), { recursive: true }); + fs.writeFileSync( + file, + [ + '# Source this file before running LobeHub CLI agent tests.', + '# Generated by init-dev-env.sh seed-user', + `export LOBE_API_KEY=${TEST_API_KEY.key}`, + `export LOBEHUB_CLI_API_KEY="${'${LOBE_API_KEY}'}"`, + `export LOBEHUB_SERVER=${process.env.APP_URL}`, + 'export LOBEHUB_CLI_HOME=.lobehub-dev', + '', + ].join('\n'), + ); + + return file; +}; + const client = new pg.Client({ connectionString: databaseUrl }); (async () => { + if (!validateApiKeyFormat(TEST_API_KEY.key)) { + throw new Error(`Invalid AGENT_TESTING_API_KEY format: ${TEST_API_KEY.key}`); + } + await client.connect(); const now = new Date().toISOString(); const onboarding = JSON.stringify({ finishedAt: now, version: 1 }); const passwordHash = await bcrypt.hash(TEST_USER.password, 10); + const encryptedApiKey = encryptWithKeyVaultsSecret(TEST_API_KEY.key); + const apiKeyHash = hashApiKey(TEST_API_KEY.key); await client.query( `INSERT INTO users (id, email, normalized_email, username, full_name, email_verified, onboarding, created_at, updated_at, last_active_at) @@ -204,9 +285,35 @@ const client = new pg.Client({ connectionString: databaseUrl }); ], ); + await client.query( + `INSERT INTO api_keys (id, name, key, key_hash, enabled, expires_at, user_id, workspace_id, created_at, updated_at) + VALUES ($1, $2, $3, $4, $5, NULL, $6, NULL, $7, $7) + ON CONFLICT (id) DO UPDATE + SET name = EXCLUDED.name, + key = EXCLUDED.key, + key_hash = EXCLUDED.key_hash, + enabled = EXCLUDED.enabled, + expires_at = NULL, + updated_at = EXCLUDED.updated_at`, + [ + TEST_API_KEY.id, + TEST_API_KEY.name, + encryptedApiKey, + apiKeyHash, + true, + TEST_USER.id, + now, + ], + ); + + const cliEnvFile = writeCliEnvFile(); + console.log('seeded baseline user:'); console.log(` email: ${TEST_USER.email}`); console.log(` password: ${TEST_USER.password}`); + console.log('seeded baseline API key:'); + console.log(` LOBE_API_KEY: ${TEST_API_KEY.key}`); + console.log(` CLI env: ${cliEnvFile}`); })() .finally(() => client.end()) .catch((error) => { @@ -222,6 +329,7 @@ cmd_status() { note "APP_URL=$APP_URL" note "DATABASE_URL=$DATABASE_URL" note "PORT=$PORT" + note "QSTASH_URL=$QSTASH_URL" if command -v docker > /dev/null 2>&1; then ok "docker CLI available" if docker ps --format '{{.Names}}' | grep -Fxq "$DB_CONTAINER"; then @@ -234,6 +342,14 @@ cmd_status() { fi } +cmd_qstash() { + apply_env + cd "$REPO_ROOT" + note "starting local QStash dev server at $QSTASH_URL" + note "keep this process running while testing workflow paths" + exec pnpm run qstash -- -port "$QSTASH_DEV_PORT" +} + cmd_dev_next() { apply_env cd "$REPO_ROOT" @@ -279,6 +395,7 @@ case "$COMMAND" in ;; migrate) migrate_db ;; seed-user) seed_user ;; + qstash) cmd_qstash ;; dev-next) cmd_dev_next ;; dev) cmd_dev ;; clean-db) cmd_clean_db ;; diff --git a/.agents/skills/agent-testing/scripts/setup-auth.sh b/.agents/skills/agent-testing/scripts/setup-auth.sh index 3b2f6ea9ca..a068500a1c 100755 --- a/.agents/skills/agent-testing/scripts/setup-auth.sh +++ b/.agents/skills/agent-testing/scripts/setup-auth.sh @@ -7,8 +7,10 @@ # Usage: # setup-auth.sh status # check server + CLI + web + Electron readiness # setup-auth.sh status --surface web # check only the Web surface gate +# setup-auth.sh cli-seed # configure CLI API-key auth from seeded local env # setup-auth.sh cli # interactive CLI device-code login (run by a human) # setup-auth.sh open-chrome # open SERVER_URL in Chrome and show DevTools +# setup-auth.sh web-seed # sign in seeded user and inject cookies automatically # setup-auth.sh web # stdin = Cookie header -> inject into agent-browser session # setup-auth.sh web-verify # live-check the agent-browser session is authenticated # @@ -16,6 +18,7 @@ # SERVER_URL (default from test-env.sh) dev server under test # SESSION (default lobehub-dev) agent-browser session name # AUTH_DIR (default ~/.lobehub-agent-testing) where web state is persisted +# SEED_EMAIL / SEED_PASSWORD seeded better-auth login set -euo pipefail @@ -78,7 +81,13 @@ SERVER_URL="${SERVER_URL:-$(default_server_url)}" SESSION="${SESSION:-lobehub-dev}" AUTH_DIR="${AUTH_DIR:-$HOME/.lobehub-agent-testing}" STATE_FILE="$AUTH_DIR/web-state.json" -CLI_HOME="$REPO_ROOT/apps/cli/.lobehub-dev" +CLI_HOME_NAME="${LOBEHUB_CLI_HOME:-.lobehub-dev}" +CLI_HOME="$HOME/${CLI_HOME_NAME#/}" +CLI_CREDENTIALS_FILE="$CLI_HOME/credentials.json" +SEED_EMAIL="${SEED_EMAIL:-agent-testing@lobehub.com}" +SEED_PASSWORD="${SEED_PASSWORD:-TestPassword123!}" +SEED_API_KEY="${SEED_API_KEY:-${AGENT_TESTING_API_KEY:-sk-lh-agenttesting0001}}" +CLI_ENV_FILE="${CLI_ENV_FILE:-$REPO_ROOT/.records/env/agent-testing-cli.env}" ok() { printf ' \033[32m✔\033[0m %s\n' "$1"; } bad() { printf ' \033[31m✘\033[0m %s\n' "$1"; } @@ -88,8 +97,10 @@ usage() { cat << EOF Usage: $0 status [--surface all|cli|web|electron] + $0 cli-seed $0 cli $0 open-chrome [--dry-run] + $0 web-seed $0 web $0 web-verify @@ -97,6 +108,8 @@ Env: SERVER_URL=$SERVER_URL SESSION=$SESSION AUTH_DIR=$AUTH_DIR + SEED_EMAIL=$SEED_EMAIL + CLI_HOME=$CLI_HOME EOF } @@ -113,11 +126,35 @@ check_server() { } check_cli() { - if [[ -f "$CLI_HOME/settings.json" ]] && grep -q "$SERVER_URL" "$CLI_HOME/settings.json"; then - ok "CLI logged in to $SERVER_URL (creds: apps/cli/.lobehub-dev)" + local api_key="${LOBEHUB_CLI_API_KEY:-${LOBE_API_KEY:-}}" + if [[ -n "$api_key" ]]; then + local body_file code + body_file="$(mktemp)" + code=$(curl -sS -o "$body_file" -w '%{http_code}' \ + -H "Authorization: Bearer $api_key" \ + "$SERVER_URL/api/v1/users/me?includeCount=0" 2> /dev/null || true) + + if [[ "$code" =~ ^[23] ]]; then + rm -f "$body_file" + ok "CLI API-key auth valid for $SERVER_URL" + return 0 + fi + + bad "CLI API-key auth failed for $SERVER_URL (http_code='$code')" + note "seed the local API key first:" + note "./.agents/skills/agent-testing/scripts/init-dev-env.sh seed-user" + note "source $CLI_ENV_FILE" + rm -f "$body_file" + return 1 + fi + + if [[ -f "$CLI_HOME/settings.json" ]] && grep -q "$SERVER_URL" "$CLI_HOME/settings.json" && [[ -f "$CLI_CREDENTIALS_FILE" ]]; then + ok "CLI device-code credentials configured for $SERVER_URL (creds: $CLI_HOME)" else bad "CLI not logged in to $SERVER_URL" - note "ask the user to run:" + note "automated path:" + note "./.agents/skills/agent-testing/scripts/init-dev-env.sh seed-user && source $CLI_ENV_FILE && $0 cli-seed" + note "interactive fallback:" note "cd apps/cli && LOBEHUB_CLI_HOME=.lobehub-dev bun src/index.ts login --server $SERVER_URL" return 1 fi @@ -128,7 +165,8 @@ check_web() { ok "web auth state saved ($STATE_FILE)" else bad "no web auth state for agent-browser" - note "copy the Cookie header from Chrome DevTools (Network tab), then:" + note "for the seeded local user, run: $0 web-seed" + note "or copy the Cookie header from Chrome DevTools (Network tab), then:" note "pbpaste | $0 web (see references/auth.md)" return 1 fi @@ -246,6 +284,44 @@ cmd_cli() { LOBEHUB_CLI_HOME=.lobehub-dev bun src/index.ts login --server "$SERVER_URL" } +write_cli_seed_env() { + mkdir -p "$(dirname "$CLI_ENV_FILE")" + cat > "$CLI_ENV_FILE" << EOF +# Source this file before running LobeHub CLI agent tests. +# Generated by setup-auth.sh cli-seed +export LOBE_API_KEY=$SEED_API_KEY +export LOBEHUB_CLI_API_KEY="\${LOBE_API_KEY}" +export LOBEHUB_SERVER=$SERVER_URL +export LOBEHUB_CLI_HOME=.lobehub-dev +EOF +} + +write_cli_settings() { + mkdir -p "$CLI_HOME" + python3 - "$CLI_HOME/settings.json" "$SERVER_URL" << 'PY' +import json +import os +import sys + +path, server_url = sys.argv[1], sys.argv[2] +os.makedirs(os.path.dirname(path), exist_ok=True) +with open(path, "w") as f: + json.dump({"serverUrl": server_url}, f, indent=2) + f.write("\n") +os.chmod(path, 0o600) +PY +} + +cmd_cli_seed() { + check_server || return 1 + write_cli_seed_env + write_cli_settings + ok "wrote CLI seed env: $CLI_ENV_FILE" + note "source it before CLI commands: source $CLI_ENV_FILE" + note "settings saved at: $CLI_HOME/settings.json" + LOBE_API_KEY="$SEED_API_KEY" LOBEHUB_CLI_API_KEY="$SEED_API_KEY" check_cli +} + cmd_open_chrome() { local mode="${1:-}" if [[ "$mode" != "" && "$mode" != "--dry-run" ]]; then @@ -301,6 +377,26 @@ OSA ok "opened Chrome at $SERVER_URL/ and requested DevTools Network panel" } +cookie_header_from_jar() { + local jar="$1" + awk ' + BEGIN { first = 1 } + /^$/ { next } + /^#/ { + if ($0 !~ /^#HttpOnly_/) next + sub(/^#HttpOnly_/, "") + } + NF >= 7 { + if (!first) printf "; " + printf "%s=%s", $6, $7 + first = 0 + } + END { + if (!first) printf "\n" + } + ' "$jar" +} + # Build a Playwright storageState file from a raw Cookie header on stdin, # keeping only the better-auth cookies. See references/auth.md for why the # header must come from a Network request (HttpOnly) and why httpOnly=false. @@ -357,6 +453,49 @@ PY cmd_web_verify } +cmd_web_seed() { + check_server || return 1 + mkdir -p "$AUTH_DIR" + + local cookie_jar="$AUTH_DIR/web-seed-cookie.jar" + local response_body="$AUTH_DIR/web-seed-response.json" + local payload code + payload="$( + SEED_EMAIL="$SEED_EMAIL" SEED_PASSWORD="$SEED_PASSWORD" python3 - << 'PY' +import json +import os + +print(json.dumps({ + "callbackURL": "/", + "email": os.environ["SEED_EMAIL"], + "password": os.environ["SEED_PASSWORD"], +})) +PY + )" + + code=$(curl -sS -o "$response_body" -w '%{http_code}' \ + -c "$cookie_jar" \ + -H 'Content-Type: application/json' \ + -X POST "$SERVER_URL/api/auth/sign-in/email" \ + --data "$payload" 2> /dev/null || true) + + if [[ ! "$code" =~ ^[23] ]]; then + bad "seed user sign-in failed at $SERVER_URL/api/auth/sign-in/email (http_code='$code')" + note "make sure the seed user exists:" + note "./.agents/skills/agent-testing/scripts/init-dev-env.sh seed-user" + return 1 + fi + + local cookie_header + cookie_header="$(cookie_header_from_jar "$cookie_jar")" + if [[ -z "$cookie_header" ]]; then + bad "seed sign-in succeeded but no cookies were written to $cookie_jar" + return 1 + fi + + printf '%s\n' "$cookie_header" | cmd_web +} + cmd_web_verify() { local skip_server_check="${1:-}" if [[ "$skip_server_check" != "--skip-server-check" ]]; then @@ -364,7 +503,8 @@ cmd_web_verify() { fi if [[ ! -f "$STATE_FILE" ]]; then bad "no web auth state for agent-browser" - note "copy the Cookie header from Chrome DevTools (Network tab), then:" + note "for the seeded local user, run: $0 web-seed" + note "or copy the Cookie header from Chrome DevTools (Network tab), then:" note "pbpaste | $0 web" return 1 fi @@ -396,16 +536,18 @@ case "${1:-status}" in shift || true cmd_status "$@" ;; + cli-seed) cmd_cli_seed ;; cli) cmd_cli ;; open-chrome) shift || true cmd_open_chrome "$@" ;; + web-seed) cmd_web_seed ;; web) cmd_web ;; web-verify) cmd_web_verify ;; -h|--help) usage ;; *) - echo "Usage: $0 {status|cli|web|web-verify}" >&2 + echo "Usage: $0 {status|cli-seed|cli|open-chrome|web-seed|web|web-verify}" >&2 exit 2 ;; esac diff --git a/.agents/skills/agent-testing/scripts/setup-auth.test.sh b/.agents/skills/agent-testing/scripts/setup-auth.test.sh index 422d02516f..e2bf732ba9 100755 --- a/.agents/skills/agent-testing/scripts/setup-auth.test.sh +++ b/.agents/skills/agent-testing/scripts/setup-auth.test.sh @@ -29,6 +29,7 @@ cleanup() { rm -rf "$tmp_dir" } trap cleanup EXIT +export HOME="$tmp_dir/home" port="$(python3 - << 'PY' import socket @@ -40,8 +41,59 @@ sock.close() PY )" -python3 -m http.server "$port" --bind localhost --directory "$tmp_dir" \ - > "$tmp_dir/http.log" 2>&1 & +python3 - "$port" << 'PY' > "$tmp_dir/http.log" 2>&1 & +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +import sys + + +class Handler(BaseHTTPRequestHandler): + def do_GET(self): + if self.path.startswith("/api/v1/users/me"): + if self.headers.get("authorization") != "Bearer sk-lh-agenttesting0001": + self.send_response(401) + self.end_headers() + self.wfile.write(b'{"success":false}') + return + + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.end_headers() + self.wfile.write(b'{"success":true,"data":{"id":"user_agent_testing_001"}}') + return + + self.send_response(200) + self.end_headers() + self.wfile.write(b"ok") + + def do_POST(self): + length = int(self.headers.get("content-length") or "0") + if length: + self.rfile.read(length) + + if self.path != "/api/auth/sign-in/email": + self.send_response(404) + self.end_headers() + return + + self.send_response(200) + self.send_header( + "Set-Cookie", + "better-auth.session_token=seed.token; Path=/; HttpOnly; SameSite=Lax", + ) + self.send_header( + "Set-Cookie", + "better-auth.session_data=seed.data; Path=/; HttpOnly; SameSite=Lax", + ) + self.send_header("Content-Type", "application/json") + self.end_headers() + self.wfile.write(b'{"ok":true}') + + def log_message(self, format, *args): + return + + +ThreadingHTTPServer(("localhost", int(sys.argv[1])), Handler).serve_forever() +PY server_pid="$!" server_url="http://localhost:$port" @@ -103,10 +155,40 @@ if names != expected: raise SystemExit(f"unexpected cookies: {sorted(names)}") PY +"$SCRIPT" web-seed > "$tmp_dir/web-seed.out" + +python3 - "$AUTH_DIR/web-state.json" << 'PY' +import json, sys + +with open(sys.argv[1]) as f: + state = json.load(f) + +values = {cookie["name"]: cookie["value"] for cookie in state["cookies"]} +expected = { + "better-auth.session_token": "seed.token", + "better-auth.session_data": "seed.data", +} +if values != expected: + raise SystemExit(f"unexpected seeded cookies: {values}") +PY + "$SCRIPT" status --surface web > "$tmp_dir/status.out" assert_contains "$tmp_dir/status.out" "surface=web" assert_contains "$tmp_dir/status.out" "web auth green" +"$SCRIPT" cli-seed > "$tmp_dir/cli-seed.out" +assert_contains "$tmp_dir/cli-seed.out" "CLI API-key auth valid" +assert_contains "$tmp_dir/cli-seed.out" "settings saved at: $HOME/.lobehub-dev/settings.json" + +if "$SCRIPT" status --surface cli > "$tmp_dir/cli-no-env.out"; then + fail "cli status without API key unexpectedly passed" +fi +assert_contains "$tmp_dir/cli-no-env.out" "CLI not logged in" + +LOBEHUB_CLI_API_KEY=sk-lh-agenttesting0001 "$SCRIPT" status --surface cli > "$tmp_dir/cli-status.out" +assert_contains "$tmp_dir/cli-status.out" "CLI API-key auth valid" +assert_contains "$tmp_dir/cli-status.out" "cli auth green" + if printf 'foo=bar\n' | "$SCRIPT" web > "$tmp_dir/invalid.out" 2> "$tmp_dir/invalid.err"; then fail "invalid cookie unexpectedly passed" fi diff --git a/.agents/skills/agent-testing/ui/web.md b/.agents/skills/agent-testing/ui/web.md index 85a183546a..4a59d8946f 100644 --- a/.agents/skills/agent-testing/ui/web.md +++ b/.agents/skills/agent-testing/ui/web.md @@ -12,9 +12,16 @@ backend-only changes prefer [../cli/index.md](../cli/index.md). - Complete [Step 0.0](../SKILL.md#00-resolve-the-current-test-environment) (resolve ports) and [Step -1](../SKILL.md#step--1--plan-approval-for-non-trivial-tests) (plan approval) first. - Local dev server running — [../references/dev-server.md](../references/dev-server.md) -- Web auth verified in agent-browser — see [auth decision flow](../references/auth.md#web--decision-flow). +- Web auth verified in agent-browser — prefer `setup-auth.sh web-seed`, see [auth decision flow](../references/auth.md#web--decision-flow). -## Option A — agent-browser with injected auth (recommended) +## Option A — agent-browser with seeded auth (recommended) + +```bash +./.agents/skills/agent-testing/scripts/init-dev-env.sh seed-user +./.agents/skills/agent-testing/scripts/setup-auth.sh web-seed +``` + +Then drive the verified session: ```bash SESSION=lobehub-dev @@ -26,7 +33,8 @@ agent-browser --session $SESSION snapshot -i Use this session as the evidence source. Do not use ordinary Chrome screenshots or Chrome Network records as proof for Web tests; ordinary Chrome is only a -source for copying cookies into agent-browser. +fallback source for copying cookies into agent-browser when the seeded login is +not available. ### Watch the API while driving the UI