references/lobe-chat
1
0
Fork 0
mirror of https://github.com/lobehub/lobe-chat.git synced 2026-06-14 03:30:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

🐛 fix: surface model list fetch failures (#15753)

Browse Source
This commit is contained in:
YuTengjing
2026-06-13 23:05:44 +08:00
committed by GitHub
parent 55a969a3c1
commit 39bce329fd
49 changed files with 1668 additions and 732 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/skills/agent-testing/SKILL.md
+72 -12
View File
@@ -19,9 +19,23 @@ also run as full cloud automation. Every test session follows the same
four-step contract:
```
Step 0: Env + Auth → Step 1: Pick surface → Step 2: Run → Step 3: Structured report
Step -1: Plan approval → Step 0: Env + Auth → Step 1: Pick surface → Step 2: Run → Step 3: Structured report
```
## Step -1 — Plan approval for non-trivial tests
Skip directly to Step 0 if: the test is a single re-run after a fix, the plan
was already agreed on, or the user gave exact commands.
Otherwise, propose a test plan (surface, cases, expected evidence, assumptions)
and use the runtime structured question tool (`request_user_input` /
ask-user-question equivalent) with two fixed choices:
1. `开始执行 (Recommended)` — 测试方案没问题，开始执行
2. `先讨论下` — 方案有问题，先讨论下
Wait for the user's choice before proceeding.
## Step 0 — Environment setup + auth check (mandatory)
Step 0 is about getting the environment ready: **dependencies are healthy**
@@ -29,6 +43,36 @@ and **auth is green**. A test run that dies halfway on a missing dependency or
a login wall wastes the whole session — clear both gates BEFORE writing a
single test step.
### 0.0 Resolve the current test environment
Before starting a dev server, checking auth, opening agent-browser, or writing
test steps, print and confirm the current local test environment:
```bash
./.agents/skills/agent-testing/scripts/test-env.sh
```
This command is the source of truth for local test ports. It reads the current
shell plus `.env` files using the same precedence as `scripts/runWithEnv.mts`,
then prints:
- `APP_URL`
- `PORT`
- `SERVER_URL`
- `AUTH_TRUSTED_ORIGINS`
- `SPA_PORT`
- `MOBILE_SPA_PORT`
- `DESKTOP_PORT`
For commands that need these values, export them from the same resolver:
```bash
eval "$(./.agents/skills/agent-testing/scripts/test-env.sh --exports)"
```
Do not rely on hard-coded port tables. If the printed values do not match the
running dev server, fix/export the env first, then continue.
### 0.1 Dependencies are installed — root AND standalone apps
The root pnpm workspace does **NOT** cover every app: `pnpm-workspace.yaml`
@@ -142,27 +186,35 @@ eval "$(../.agents/skills/agent-testing/scripts/init-dev-env.sh env)"
BASE_URL=http://localhost:3010 HEADLESS=true bun run test:smoke
```
### 0.4 Auth is green
### 0.4 Auth is green for the selected surface
**Auth is the gate for all automated testing.**
**Auth is the gate for automated testing, but the gate is surface-scoped.**
Pick the intended surface first when it is already clear from the task, then
check only that surface. Do not block a Web test on CLI device-code auth or an
Electron login state unless the test spans those surfaces.
```bash
./.agents/skills/agent-testing/scripts/setup-auth.sh status
./.agents/skills/agent-testing/scripts/setup-auth.sh status --surface web
```
| Surface | Mechanism | One-key path | Standard check |
| -------- | ------------------------------------------------- | ------------------------------ | -------------------------- |
| CLI | OIDC Device Code Flow (`apps/cli/.lobehub-dev`) | `setup-auth.sh cli` | `setup-auth.sh status` |
| Web | better-auth cookie injection into `agent-browser` | `pbpaste \| setup-auth.sh web` | `setup-auth.sh web-verify` |
| Electron | App's own persistent login state | Log in once in the app | `app-probe.sh auth` |
| Bot | Native apps already logged in | — | per-platform screenshot |
Use `status` with no `--surface` only for cross-surface test plans.
| Surface | Mechanism | One-key path | Standard check |
| -------- | ------------------------------------------------- | ------------------------------ | ----------------------------------------- |
| CLI | OIDC Device Code Flow (`apps/cli/.lobehub-dev`) | `setup-auth.sh cli` | `setup-auth.sh status --surface cli` |
| Web | better-auth cookie injection into `agent-browser` | `pbpaste \| setup-auth.sh web` | `setup-auth.sh status --surface web` |
| Electron | App's own persistent login state | Log in once in the app | `setup-auth.sh status --surface electron` |
| Bot | Native apps already logged in | — | per-platform screenshot |
Login-state checks are standardized — do NOT hand-roll `window.__LOBE_STORES`
eval snippets; use `scripts/app-probe.sh auth` (returns `{ isSignedIn, userId }`,
works for Electron CDP and web sessions via `AB_TARGET`).
If `status` is not all green, fix auth first (the steps that need a human must be
requested from the user explicitly). Full background and failure modes:
For Web tests, the test surface is always `agent-browser --session lobehub-dev`.
The user's normal Chrome is only a source for copying the Cookie header when
`status --surface web` fails. If Chrome is already logged in, do not open a
login page; verify agent-browser first, then request the Network `Cookie:`
header only if that verification fails. Full background and failure modes:
[references/auth.md](./references/auth.md).
## Step 1 — Pick the surface by change scope
@@ -237,6 +289,7 @@ All under `.agents/skills/agent-testing/scripts/`:
| Script | Usage |
| ------------------------- | ---------------------------------------------------------------------------- |
| `test-env.sh` | Print/export the resolved local test env and ports |
| `setup-auth.sh` | One-stop auth setup & status check (`status` / `cli` / `web`) |
| `init-dev-env.sh` | Self-contained local dev env (`setup-db` / `seed-user` / `dev-next` / `dev`) |
| `app-probe.sh` | LobeHub app probes: `auth` / `route` / `ops` / `goto <path>` / `errors` |
@@ -284,6 +337,13 @@ Two hard rules worth front-loading:
must use Markdown image syntax like `![case 1](assets/case1.png)`. Do not
use bare file paths, Markdown links, or local file links as the primary
visual evidence; those make the report unreadable without opening each asset.
- **Final replies must include visual evidence links.** When a run includes UI
screenshots or GIFs, include the report directory and the most important
visual artifacts in the final chat response. Each item must include a stable
label, an evidence caption describing the observed UI outcome, and a
repo-relative path, for example:
`[Image #1 - error toast shows provider auth failure](<report-dir>/assets/foo.png)`.
Use repo-relative paths, not absolute paths.
- **Time-based behavior needs a GIF, not a screenshot.** If a case asserts
change over time (streaming output, a ticking timer, loading states,
animations), record it with `scripts/record-gif.sh` and embed the GIF —
.agents/skills/agent-testing/references/auth.md
+39 -36
View File
@@ -1,20 +1,31 @@
# Auth Setup for Local Agent Testing
**Auth is the gate for all automated testing.** Prepare and verify it before
writing any test step. The one-stop entry point is:
**Auth is the gate for all automated testing.** Complete
[Step 0.0](../SKILL.md#00-resolve-the-current-test-environment) first so
`SERVER_URL` and ports are resolved, then verify auth before writing any test
step.
Initialize helpers first:
```bash
SCRIPT=".agents/skills/agent-testing/scripts/setup-auth.sh"
$SCRIPT status # check server + CLI + web auth readiness
$SCRIPT cli # interactive CLI device-code login (must be run by the user)
pbpaste | $SCRIPT web # inject a copied Cookie header into the agent-browser session
$SCRIPT web-verify # live-check that the agent-browser session is authenticated
SCRIPT="./.agents/skills/agent-testing/scripts/setup-auth.sh"
TEST_ENV="./.agents/skills/agent-testing/scripts/test-env.sh"
eval "$($TEST_ENV --exports)"
```
`SERVER_URL` defaults to `http://localhost:3010` (this repo's `dev:next` port).
Override it when testing against another server (e.g. `SERVER_URL=http://localhost:3011`
in the cloud repo).
Quick reference after initialization:
| Command | Purpose |
| ------------------------------ | -------------------------------------------------- |
| `$SCRIPT status` | Check all surfaces (server + CLI + web + Electron) |
| `$SCRIPT status --surface web` | Check only the Web surface gate |
| `$SCRIPT cli` | Interactive CLI device-code login (user must run) |
| `$SCRIPT open-chrome` | Open Chrome at `SERVER_URL` with DevTools |
| `pbpaste \| $SCRIPT web` | Inject a copied Cookie header into agent-browser |
| `$SCRIPT web-verify` | Live-check agent-browser session auth |
Use `localhost` for Web auth; better-auth cookies are stored for `localhost`,
not `127.0.0.1`.
## Per-surface overview
@@ -45,6 +56,11 @@ cd apps/cli && LOBEHUB_CLI_HOME=.lobehub-dev bun src/index.ts login --server htt
## Web — better-auth cookie injection (agent-browser)
The Web test surface is `agent-browser --session lobehub-dev`. The user's
ordinary Chrome is only a cookie source; Chrome screenshots, Chrome Network
records, and Chrome logged-in state do not prove the agent-browser test session
is authenticated.
`agent-browser --headed` on macOS often creates the Chromium window off-screen —
the user can't see or interact with it, so manual login inside the agent-browser
session fails. Instead, copy the **better-auth session cookie** out of the
@@ -53,31 +69,18 @@ user's own logged-in Chrome and inject it as a Playwright-style state file.
Do **not** use this on production URLs — only local dev. Treat the cookie as a
secret: don't paste it into shared logs, PRs, or commit it anywhere.
### One-key path
### Web — decision flow
1. Ask the user to copy the Cookie header **from a Network request, NOT
`document.cookie`** (`document.cookie` cannot see HttpOnly cookies, which is
exactly where better-auth puts its session):
- Open the logged-in tab (`http://localhost:<port>/…`) in Chrome.
- `Cmd+Option+I` → **Network** tab → refresh → click any same-origin request.
- Under **Request Headers**, right-click the `Cookie:` line → **Copy value**.
2. Inject and verify in one shot:
```bash
pbpaste | ./.agents/skills/agent-testing/scripts/setup-auth.sh web
```
The script filters the header down to the better-auth cookies
(`better-auth.session_token`, `better-auth.state`), builds the Playwright
`storageState` JSON, loads it into the `agent-browser` session (default name
`lobehub-dev`), opens `SERVER_URL`, and asserts the URL is not `/signin`.
1. `$SCRIPT status --surface web` — green? Start testing. Do not ask for a Cookie header.
2. Not green → `$SCRIPT open-chrome` opens Chrome at `SERVER_URL` with DevTools.
3. User copies the `Cookie:` header from Network tab → any same-origin request → Request Headers → right-click `Cookie:` → **Copy value**. Must be from Network, NOT `document.cookie` (HttpOnly cookies are invisible to `document.cookie`).
4. `pbpaste | $SCRIPT web` — filters to better-auth cookies (`session_token`, `session_data`, `state`), builds Playwright `storageState`, loads it into the `agent-browser` session (`lobehub-dev`), opens `SERVER_URL`, and asserts the URL is not `/signin`.
### Using the authenticated session
```bash
agent-browser --session lobehub-dev open "http://localhost:3010/"
agent-browser --session lobehub-dev open "$SERVER_URL/"
agent-browser --session lobehub-dev snapshot -i | head -20
# Look for the user's avatar/name in the sidebar, or absence of the signin form.
```
### Notes
@@ -90,12 +93,12 @@ agent-browser --session lobehub-dev snapshot -i | head -20
### Common failure modes
| Symptom | Cause | Fix |
| --------------------------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------- |
| Still redirects to `/signin` after injection | User pasted from `document.cookie` → missed HttpOnly session | Re-pull from Network request Headers, not console |
| Script reports `no better-auth cookies found` | Separator wrong, or user pasted URL-decoded value | Keep the raw `Cookie:` header as-is |
| Login works briefly then expires | `better-auth.session_token` rotated (user logged out / signed in again) | Re-copy and re-inject |
| Domain mismatch | Cookie domain must be `localhost` literally, no leading dot for local dev | — |
| Symptom | Cause | Fix |
| --------------------------------------------- | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| Still redirects to `/signin` after injection | User pasted from `document.cookie` → missed HttpOnly session | Re-pull from Network request Headers, not console |
| Script reports `no better-auth cookies found` | User pasted the wrong value, or the cookie parser regressed | Keep the raw `Cookie:` header as-is; run `scripts/setup-auth.test.sh` if the input looks valid |
| Login works briefly then expires | `better-auth.session_token` rotated (user logged out / signed in again) | Re-copy and re-inject |
| Domain mismatch | Cookie domain must be `localhost` literally, no leading dot for local dev | — |
## Electron
.agents/skills/agent-testing/references/dev-server.md
+32 -10
View File
@@ -3,22 +3,44 @@
Single source of truth for starting / restarting the backend that all test
surfaces (CLI, Electron, Web) hit.
## Resolve ports first
Run `test-env.sh` as described in
[SKILL.md Step 0.0](../SKILL.md#00-resolve-the-current-test-environment)
before starting or probing any local test surface.
## Ports & modes
| Command | What it runs | Port |
| ------------------- | --------------------------------------------------------- | --------------------------------- |
| `pnpm run dev:next` | Next.js backend (API + auth) | `3010` |
| `bun run dev` | Full-stack (Next.js + Vite SPA, via `devStartupSequence`) | `3010` (API) + SPA on `9876` |
| `bun run dev:spa` | Vite SPA only, proxies API to `3010` | `9876` (prints a Debug Proxy URL) |
| Command | What it runs | Port source |
| ------------------- | --------------------------------------------------------- | ------------------- |
| `pnpm run dev:next` | Next.js backend (API + auth) | `PORT` |
| `bun run dev` | Full-stack (Next.js + Vite SPA, via `devStartupSequence`) | `PORT` + `SPA_PORT` |
| `bun run dev:spa` | Vite SPA only, proxies API to `PORT` | `SPA_PORT` |
In the **cloud repo** (where this repo is the `lobehub/` submodule) the dev
server conventionally runs on `3011` — set `SERVER_URL=http://localhost:3011`
for the scripts in this skill when testing there.
In the **cloud repo** (where this repo is the `lobehub/` submodule), local
worktree names map to fallback defaults only when `.env` and shell env do not
provide values:
| Workspace directory | Default `SERVER_URL` |
| ------------------- | -------------------------------- |
| `lobehub` | `http://localhost:3010` |
| `lobehub-cloud` | `http://localhost:3020` |
| `lobehub-cloud-1` | `http://localhost:3021` |
| `lobehub-cloud-N` | `http://localhost:$((3020 + N))` |
`test-env.sh` and `setup-auth.sh` both use the resolved env first and these
worktree defaults only as fallback. Treat the dev-server terminal output as the
final source of truth when testing a non-standard port, then export it for every
agent-testing command:
```bash
export SERVER_URL=http://localhost:<port-from-dev-output>
```
## Health check
```bash
curl -s -o /dev/null -w '%{http_code}' http://localhost:3010/
curl -s -o /dev/null -w '%{http_code}' "$SERVER_URL/"
```
## Start / restart
@@ -39,7 +61,7 @@ bun run dev
./.agents/skills/agent-testing/scripts/init-dev-env.sh dev
# Restart — required to pick up server-side code changes
lsof -ti:3010 | xargs kill
lsof -ti:"$PORT" | xargs kill
pnpm run dev:next
# or, when no root .env exists:
# ./.agents/skills/agent-testing/scripts/init-dev-env.sh dev-next
.agents/skills/agent-testing/references/report.md
+4 -1
View File
@@ -64,7 +64,10 @@ output):
not acceptable as primary visual evidence.
4. **Set the verdict** in both `report.md` and `result.json`, then link the
report directory in your final answer to the user.
report directory in your final answer to the user. If UI evidence exists,
list the key screenshot/GIF links in the final chat response. Use Markdown
link text as the evidence caption, for example:
`[Image #1 - observed outcome](<report-dir>/assets/case1.png)`.
## Report language (hard rule)
.agents/skills/agent-testing/scripts/setup-auth.sh
+260 -23
View File
@@ -5,29 +5,101 @@
# test step. Background and failure modes: ../references/auth.md
#
# Usage:
# setup-auth.sh status # check server + CLI + web auth readiness
# setup-auth.sh status # check server + CLI + web + Electron readiness
# setup-auth.sh status --surface web # check only the Web surface gate
# setup-auth.sh cli # interactive CLI device-code login (run by a human)
# setup-auth.sh open-chrome # open SERVER_URL in Chrome and show DevTools
# setup-auth.sh web # stdin = Cookie header -> inject into agent-browser session
# setup-auth.sh web-verify # live-check the agent-browser session is authenticated
#
# Env:
# SERVER_URL (default http://localhost:3010) dev server under test
# SERVER_URL (default from test-env.sh) dev server under test
# SESSION (default lobehub-dev) agent-browser session name
# AUTH_DIR (default ~/.lobehub-agent-testing) where web state is persisted
set -euo pipefail
SERVER_URL="${SERVER_URL:-http://localhost:3010}"
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../.." && pwd)"
workspace_root_for_port() {
local root="$REPO_ROOT"
local name
name="$(basename "$root")"
if [[ "$name" == "lobehub" ]]; then
local parent
parent="$(cd "$root/.." && pwd)"
local parent_name
parent_name="$(basename "$parent")"
if [[ "$parent_name" == lobehub-cloud* ]]; then
root="$parent"
fi
fi
printf '%s\n' "$root"
}
default_server_url() {
local env_resolver resolved
env_resolver="$(dirname "${BASH_SOURCE[0]}")/test-env.sh"
if [[ -x "$env_resolver" ]]; then
resolved="$("$env_resolver" --value SERVER_URL 2> /dev/null || true)"
if [[ -n "$resolved" ]]; then
printf '%s\n' "$resolved"
return 0
fi
fi
local root name suffix port
root="$(workspace_root_for_port)"
name="$(basename "$root")"
case "$name" in
lobehub-cloud)
port=3020
;;
lobehub-cloud-*)
suffix="${name#lobehub-cloud-}"
if [[ "$suffix" =~ ^[0-9]+$ ]]; then
port=$((3020 + 10#$suffix))
else
port=3010
fi
;;
*)
port=3010
;;
esac
printf 'http://localhost:%s\n' "$port"
}
SERVER_URL="${SERVER_URL:-$(default_server_url)}"
SESSION="${SESSION:-lobehub-dev}"
AUTH_DIR="${AUTH_DIR:-$HOME/.lobehub-agent-testing}"
STATE_FILE="$AUTH_DIR/web-state.json"
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../.." && pwd)"
CLI_HOME="$REPO_ROOT/apps/cli/.lobehub-dev"
ok() { printf ' \033[32m✔\033[0m %s\n' "$1"; }
bad() { printf ' \033[31m✘\033[0m %s\n' "$1"; }
note() { printf ' %s\n' "$1"; }
usage() {
cat << EOF
Usage:
$0 status [--surface all|cli|web|electron]
$0 cli
$0 open-chrome [--dry-run]
$0 web
$0 web-verify
Env:
SERVER_URL=$SERVER_URL
SESSION=$SESSION
AUTH_DIR=$AUTH_DIR
EOF
}
check_server() {
local code
code=$(curl -s -o /dev/null -w '%{http_code}' "$SERVER_URL/" 2> /dev/null || true)
@@ -54,13 +126,23 @@ check_cli() {
check_web() {
if [[ -f "$STATE_FILE" ]]; then
ok "web auth state saved ($STATE_FILE)"
note "live-verify: $0 web-verify"
else
bad "no web auth state for agent-browser"
note "copy the Cookie header from Chrome DevTools (Network tab), then:"
note "pbpaste | $0 web (see references/auth.md)"
return 1
fi
cmd_web_verify --skip-server-check
}
check_agent_browser() {
if command -v agent-browser > /dev/null 2>&1; then
ok "agent-browser available"
else
bad "agent-browser command not found"
note "install or expose agent-browser before Web/Electron UI testing"
return 1
fi
}
check_electron() {
@@ -84,16 +166,75 @@ check_electron() {
}
cmd_status() {
echo "agent-testing auth status (SERVER_URL=$SERVER_URL):"
local surface="all"
while [[ $# -gt 0 ]]; do
case "$1" in
--surface)
if [[ $# -lt 2 ]]; then
echo "--surface requires one of: all, cli, web, electron" >&2
return 2
fi
surface="${2:-}"
shift 2
;;
--surface=*)
surface="${1#*=}"
shift
;;
all|cli|web|electron)
surface="$1"
shift
;;
-h|--help)
usage
return 0
;;
*)
echo "unknown status option: $1" >&2
usage >&2
return 2
;;
esac
done
case "$surface" in
all|cli|web|electron) ;;
"")
echo "--surface requires one of: all, cli, web, electron" >&2
return 2
;;
*)
echo "unknown surface: $surface" >&2
usage >&2
return 2
;;
esac
echo "agent-testing auth status (surface=$surface, SERVER_URL=$SERVER_URL):"
local rc=0
check_server || rc=1
check_cli || rc=1
check_web || rc=1
check_electron || rc=1
case "$surface" in
all)
check_server || rc=1
check_cli || rc=1
check_web || rc=1
check_electron || rc=1
;;
cli)
check_server || rc=1
check_cli || rc=1
;;
web)
check_server || rc=1
check_web || rc=1
;;
electron)
check_electron || rc=1
;;
esac
if [[ $rc -eq 0 ]]; then
echo "all green — safe to start automated testing."
echo "$surface auth green — safe to start automated testing on this surface."
else
echo "auth NOT ready — fix the ✘ items before writing any test step."
echo "$surface auth NOT ready — fix the ✘ items before writing any test step."
fi
return $rc
}
@@ -105,23 +246,90 @@ cmd_cli() {
LOBEHUB_CLI_HOME=.lobehub-dev bun src/index.ts login --server "$SERVER_URL"
}
cmd_open_chrome() {
local mode="${1:-}"
if [[ "$mode" != "" && "$mode" != "--dry-run" ]]; then
echo "unknown open-chrome option: $mode" >&2
usage >&2
return 2
fi
if [[ "$mode" == "--dry-run" ]]; then
echo "would open Google Chrome at $SERVER_URL/"
echo "would press Cmd+Option+I to open DevTools"
echo "would open DevTools command menu and run 'Show Network'"
return 0
fi
if [[ "$(uname -s)" != "Darwin" ]]; then
bad "open-chrome is macOS-only"
note "open $SERVER_URL/ in your browser and open DevTools manually"
return 1
fi
if ! command -v osascript > /dev/null 2>&1; then
bad "osascript not found"
note "open $SERVER_URL/ in Chrome and press Cmd+Option+I manually"
return 1
fi
SERVER_URL="$SERVER_URL" osascript << 'OSA'
set targetUrl to (system attribute "SERVER_URL") & "/"
tell application "Google Chrome"
activate
if (count of windows) = 0 then
make new window
end if
tell front window to make new tab with properties {URL:targetUrl}
end tell
delay 1
tell application "System Events"
tell process "Google Chrome"
set frontmost to true
keystroke "i" using {command down, option down}
delay 1
keystroke "p" using {command down, shift down}
delay 0.2
keystroke "Show Network"
key code 36
end tell
end tell
OSA
ok "opened Chrome at $SERVER_URL/ and requested DevTools Network panel"
}
# Build a Playwright storageState file from a raw Cookie header on stdin,
# keeping only the better-auth cookies. See references/auth.md for why the
# header must come from a Network request (HttpOnly) and why httpOnly=false.
cmd_web() {
mkdir -p "$AUTH_DIR"
python3 - "$STATE_FILE" << 'PY'
import json, sys, time
local raw
raw="$(cat)"
COOKIE_INPUT="$raw" python3 - "$STATE_FILE" << 'PY'
import json, os, sys, time
raw = sys.stdin.read().strip()
if raw.lower().startswith("cookie:"):
raw = raw.split(":", 1)[1].strip()
raw = os.environ.get("COOKIE_INPUT", "").strip()
cookie_lines = []
for line in raw.splitlines():
stripped = line.strip()
if not stripped:
continue
if stripped.lower().startswith("cookie:"):
cookie_lines.append(stripped.split(":", 1)[1].strip())
else:
cookie_lines.append(stripped)
WANTED = {"better-auth.session_token", "better-auth.state"}
raw = "; ".join(cookie_lines)
WANTED = {"better-auth.session_token", "better-auth.session_data", "better-auth.state"}
exp = int(time.time()) + 30 * 24 * 3600 # 30 days
cookies = []
for pair in raw.split("; "):
for pair in raw.split(";"):
pair = pair.strip()
if "=" not in pair:
continue
name, _, value = pair.partition("=")
@@ -146,14 +354,35 @@ with open(sys.argv[1], "w") as f:
json.dump({"cookies": cookies, "origins": []}, f, indent=2)
print(f"wrote {len(cookies)} cookie(s) to {sys.argv[1]}")
PY
agent-browser --session "$SESSION" state load "$STATE_FILE"
cmd_web_verify
}
cmd_web_verify() {
agent-browser --session "$SESSION" open "$SERVER_URL/" > /dev/null
local skip_server_check="${1:-}"
if [[ "$skip_server_check" != "--skip-server-check" ]]; then
check_server || return 1
fi
if [[ ! -f "$STATE_FILE" ]]; then
bad "no web auth state for agent-browser"
note "copy the Cookie header from Chrome DevTools (Network tab), then:"
note "pbpaste | $0 web"
return 1
fi
check_agent_browser || return 1
if ! agent-browser --session "$SESSION" state load "$STATE_FILE" > /dev/null; then
bad "failed to load web auth state into agent-browser session '$SESSION'"
return 1
fi
if ! agent-browser --session "$SESSION" open "$SERVER_URL/" > /dev/null; then
bad "failed to open $SERVER_URL in agent-browser session '$SESSION'"
return 1
fi
local url
url=$(agent-browser --session "$SESSION" get url)
url=$(agent-browser --session "$SESSION" get url 2> /dev/null || true)
if [[ -z "$url" ]]; then
bad "agent-browser session '$SESSION' did not report a current URL"
return 1
fi
if [[ "$url" == *"/signin"* || "$url" == *"/login"* ]]; then
bad "agent-browser session '$SESSION' NOT authenticated (landed on $url)"
note "re-copy the Cookie header and re-run: pbpaste | $0 web"
@@ -163,10 +392,18 @@ cmd_web_verify() {
}
case "${1:-status}" in
status) cmd_status ;;
status)
shift || true
cmd_status "$@"
;;
cli) cmd_cli ;;
open-chrome)
shift || true
cmd_open_chrome "$@"
;;
web) cmd_web ;;
web-verify) cmd_web_verify ;;
-h|--help) usage ;;
*)
echo "Usage: $0 {status|cli|web|web-verify}" >&2
exit 2
.agents/skills/agent-testing/scripts/setup-auth.test.sh
Executable
+115
View File
@@ -0,0 +1,115 @@
#!/usr/bin/env bash
# Smoke tests for setup-auth.sh. Uses a temporary agent-browser stub and local
# HTTP server, so it does not need real browser auth.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SCRIPT="$SCRIPT_DIR/setup-auth.sh"
fail() {
echo "FAIL: $*" >&2
exit 1
}
assert_contains() {
local file="$1"
local text="$2"
grep -Fq "$text" "$file" || fail "expected '$text' in $file"
}
tmp_dir="$(mktemp -d)"
server_pid=""
cleanup() {
if [[ -n "$server_pid" ]]; then
kill "$server_pid" > /dev/null 2>&1 || true
wait "$server_pid" > /dev/null 2>&1 || true
fi
rm -rf "$tmp_dir"
}
trap cleanup EXIT
port="$(python3 - << 'PY'
import socket
sock = socket.socket()
sock.bind(("127.0.0.1", 0))
print(sock.getsockname()[1])
sock.close()
PY
)"
python3 -m http.server "$port" --bind localhost --directory "$tmp_dir" \
> "$tmp_dir/http.log" 2>&1 &
server_pid="$!"
server_url="http://localhost:$port"
for _ in {1..50}; do
if curl -s -o /dev/null "$server_url/"; then
break
fi
sleep 0.1
done
curl -s -o /dev/null "$server_url/" || fail "test HTTP server did not start"
mkdir -p "$tmp_dir/bin" "$tmp_dir/auth"
cat > "$tmp_dir/bin/agent-browser" << 'SH'
#!/usr/bin/env bash
set -euo pipefail
if [[ "${1:-}" == "--session" ]]; then
shift 2
fi
case "${1:-}" in
state)
[[ "${2:-}" == "load" ]] || exit 2
[[ -f "${3:-}" ]] || exit 1
;;
open)
printf '%s\n' "${2:-}" > "${AGENT_BROWSER_URL_FILE:?}"
;;
get)
[[ "${2:-}" == "url" ]] || exit 2
cat "${AGENT_BROWSER_URL_FILE:?}"
;;
*)
echo "unexpected agent-browser command: $*" >&2
exit 2
;;
esac
SH
chmod +x "$tmp_dir/bin/agent-browser"
export PATH="$tmp_dir/bin:$PATH"
export AUTH_DIR="$tmp_dir/auth"
export SESSION="setup-auth-test"
export SERVER_URL="$server_url"
export AGENT_BROWSER_URL_FILE="$tmp_dir/current-url"
cookie_header="Cookie: foo=bar; better-auth.session_token=test.token; better-auth.session_data=encoded%3D; theme=dark"
printf '%s\n' "$cookie_header" | "$SCRIPT" web > "$tmp_dir/web.out"
python3 - "$AUTH_DIR/web-state.json" << 'PY'
import json, sys
with open(sys.argv[1]) as f:
state = json.load(f)
names = {cookie["name"] for cookie in state["cookies"]}
expected = {"better-auth.session_token", "better-auth.session_data"}
if names != expected:
raise SystemExit(f"unexpected cookies: {sorted(names)}")
PY
"$SCRIPT" status --surface web > "$tmp_dir/status.out"
assert_contains "$tmp_dir/status.out" "surface=web"
assert_contains "$tmp_dir/status.out" "web auth green"
if printf 'foo=bar\n' | "$SCRIPT" web > "$tmp_dir/invalid.out" 2> "$tmp_dir/invalid.err"; then
fail "invalid cookie unexpectedly passed"
fi
assert_contains "$tmp_dir/invalid.err" "no better-auth cookies found"
echo "setup-auth tests passed"
.agents/skills/agent-testing/scripts/test-env.sh
Executable
+377
View File
@@ -0,0 +1,377 @@
#!/usr/bin/env bash
# Print the resolved local test environment for agent-testing.
#
# This is intentionally read-only. It mirrors scripts/runWithEnv.mts precedence:
# .env -> .env.$NODE_ENV -> .env.local -> .env.$NODE_ENV.local, then shell env.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../../../.." && pwd)"
NODE_ENV="${NODE_ENV:-development}"
VALUE_APP_URL=""
VALUE_PORT=""
VALUE_SERVER_URL=""
VALUE_AUTH_TRUSTED_ORIGINS=""
VALUE_SPA_PORT=""
VALUE_MOBILE_SPA_PORT=""
VALUE_DESKTOP_PORT=""
SOURCE_APP_URL=""
SOURCE_PORT=""
SOURCE_SERVER_URL=""
SOURCE_AUTH_TRUSTED_ORIGINS=""
SOURCE_SPA_PORT=""
SOURCE_MOBILE_SPA_PORT=""
SOURCE_DESKTOP_PORT=""
LOADED_ENV_FILES=""
keys() {
printf '%s\n' \
APP_URL \
PORT \
SERVER_URL \
AUTH_TRUSTED_ORIGINS \
SPA_PORT \
MOBILE_SPA_PORT \
DESKTOP_PORT
}
trim() {
local value="$1"
value="${value#"${value%%[![:space:]]*}"}"
value="${value%"${value##*[![:space:]]}"}"
printf '%s' "$value"
}
workspace_root() {
local root="$REPO_ROOT"
local name
name="$(basename "$root")"
if [[ "$name" == "lobehub" ]]; then
local parent parent_name
parent="$(cd "$root/.." && pwd)"
parent_name="$(basename "$parent")"
if [[ "$parent_name" == lobehub-cloud* ]]; then
root="$parent"
fi
fi
printf '%s\n' "$root"
}
workspace_offset() {
local name="$1"
case "$name" in
lobehub-cloud)
printf '0\n'
;;
lobehub-cloud-*)
local suffix="${name#lobehub-cloud-}"
if [[ "$suffix" =~ ^[0-9]+$ ]]; then
printf '%s\n' "$((10#$suffix))"
else
printf '\n'
fi
;;
*)
printf '\n'
;;
esac
}
default_port() {
local base="$1"
local fallback="$2"
local root name offset
root="$(workspace_root)"
name="$(basename "$root")"
offset="$(workspace_offset "$name")"
if [[ -n "$offset" ]]; then
printf '%s\n' "$((base + offset))"
else
printf '%s\n' "$fallback"
fi
}
url_port() {
local url="$1"
local hostport
hostport="${url#*://}"
hostport="${hostport%%/*}"
if [[ "$hostport" == *:* ]]; then
local port="${hostport##*:}"
if [[ "$port" =~ ^[0-9]+$ ]]; then
printf '%s\n' "$port"
return 0
fi
fi
return 1
}
url_origin() {
local url="$1"
local scheme rest hostport
if [[ "$url" == *"://"* ]]; then
scheme="${url%%://*}"
rest="${url#*://}"
hostport="${rest%%/*}"
printf '%s://%s\n' "$scheme" "$hostport"
else
printf '%s\n' "$url"
fi
}
set_value() {
local key="$1"
local value="$2"
local source="$3"
case "$key" in
APP_URL) VALUE_APP_URL="$value"; SOURCE_APP_URL="$source" ;;
PORT) VALUE_PORT="$value"; SOURCE_PORT="$source" ;;
SERVER_URL) VALUE_SERVER_URL="$value"; SOURCE_SERVER_URL="$source" ;;
AUTH_TRUSTED_ORIGINS) VALUE_AUTH_TRUSTED_ORIGINS="$value"; SOURCE_AUTH_TRUSTED_ORIGINS="$source" ;;
SPA_PORT) VALUE_SPA_PORT="$value"; SOURCE_SPA_PORT="$source" ;;
MOBILE_SPA_PORT) VALUE_MOBILE_SPA_PORT="$value"; SOURCE_MOBILE_SPA_PORT="$source" ;;
DESKTOP_PORT) VALUE_DESKTOP_PORT="$value"; SOURCE_DESKTOP_PORT="$source" ;;
esac
}
value_for() {
case "$1" in
APP_URL) printf '%s\n' "$VALUE_APP_URL" ;;
PORT) printf '%s\n' "$VALUE_PORT" ;;
SERVER_URL) printf '%s\n' "$VALUE_SERVER_URL" ;;
AUTH_TRUSTED_ORIGINS) printf '%s\n' "$VALUE_AUTH_TRUSTED_ORIGINS" ;;
SPA_PORT) printf '%s\n' "$VALUE_SPA_PORT" ;;
MOBILE_SPA_PORT) printf '%s\n' "$VALUE_MOBILE_SPA_PORT" ;;
DESKTOP_PORT) printf '%s\n' "$VALUE_DESKTOP_PORT" ;;
esac
}
source_for() {
case "$1" in
APP_URL) printf '%s\n' "$SOURCE_APP_URL" ;;
PORT) printf '%s\n' "$SOURCE_PORT" ;;
SERVER_URL) printf '%s\n' "$SOURCE_SERVER_URL" ;;
AUTH_TRUSTED_ORIGINS) printf '%s\n' "$SOURCE_AUTH_TRUSTED_ORIGINS" ;;
SPA_PORT) printf '%s\n' "$SOURCE_SPA_PORT" ;;
MOBILE_SPA_PORT) printf '%s\n' "$SOURCE_MOBILE_SPA_PORT" ;;
DESKTOP_PORT) printf '%s\n' "$SOURCE_DESKTOP_PORT" ;;
esac
}
is_tracked_key() {
case "$1" in
APP_URL|PORT|SERVER_URL|AUTH_TRUSTED_ORIGINS|SPA_PORT|MOBILE_SPA_PORT|DESKTOP_PORT) return 0 ;;
*) return 1 ;;
esac
}
parse_env_file() {
local file="$1"
local root="$2"
local label="${file#$root/}"
local line key value
[[ -f "$file" ]] || return 0
if [[ -z "$LOADED_ENV_FILES" ]]; then
LOADED_ENV_FILES="$label"
else
LOADED_ENV_FILES="$LOADED_ENV_FILES, $label"
fi
while IFS= read -r line || [[ -n "$line" ]]; do
line="$(trim "$line")"
[[ -z "$line" || "$line" == \#* ]] && continue
if [[ "$line" == export[[:space:]]* ]]; then
line="$(trim "${line#export}")"
fi
[[ "$line" == *=* ]] || continue
key="$(trim "${line%%=*}")"
value="$(trim "${line#*=}")"
is_tracked_key "$key" || continue
if [[ "$value" == \"*\" && "$value" == *\" && ${#value} -ge 2 ]]; then
value="${value:1:${#value}-2}"
elif [[ "$value" == \'* && "$value" == *\' && ${#value} -ge 2 ]]; then
value="${value:1:${#value}-2}"
fi
set_value "$key" "$value" "$label"
done < "$file"
}
apply_env_files() {
local root="$1"
parse_env_file "$root/.env" "$root"
parse_env_file "$root/.env.$NODE_ENV" "$root"
parse_env_file "$root/.env.local" "$root"
parse_env_file "$root/.env.$NODE_ENV.local" "$root"
}
apply_shell_overrides() {
local key value
while IFS= read -r key; do
if [[ -n "${!key+x}" ]]; then
value="${!key}"
set_value "$key" "$value" "shell"
fi
done < <(keys)
}
resolve_defaults() {
local app_port spa_port mobile_spa_port desktop_port
app_port="$(default_port 3020 3010)"
spa_port="$(default_port 9800 9876)"
mobile_spa_port="$(default_port 3810 3012)"
desktop_port="$(default_port 3030 3015)"
if [[ -z "$VALUE_APP_URL" ]]; then
set_value APP_URL "http://localhost:$app_port" "inferred"
fi
if [[ -z "$VALUE_PORT" ]]; then
if app_port="$(url_port "$VALUE_APP_URL")"; then
set_value PORT "$app_port" "inferred from APP_URL"
else
set_value PORT "$(default_port 3020 3010)" "inferred"
fi
fi
if [[ -z "$VALUE_SERVER_URL" ]]; then
set_value SERVER_URL "$VALUE_APP_URL" "from APP_URL"
fi
if [[ -z "$VALUE_SPA_PORT" ]]; then
set_value SPA_PORT "$spa_port" "inferred"
fi
if [[ -z "$VALUE_MOBILE_SPA_PORT" ]]; then
set_value MOBILE_SPA_PORT "$mobile_spa_port" "inferred"
fi
if [[ -z "$VALUE_DESKTOP_PORT" ]]; then
set_value DESKTOP_PORT "$desktop_port" "inferred"
fi
if [[ -z "$VALUE_AUTH_TRUSTED_ORIGINS" ]]; then
set_value AUTH_TRUSTED_ORIGINS "$(url_origin "$VALUE_APP_URL"),http://localhost:$VALUE_SPA_PORT" "inferred"
fi
}
contains_origin() {
local list="$1"
local expected="$2"
local item
IFS=',' read -r -a items <<< "$list"
for item in "${items[@]}"; do
item="$(trim "$item")"
[[ "$item" == "$expected" ]] && return 0
done
return 1
}
print_exports() {
local key value
while IFS= read -r key; do
value="$(value_for "$key")"
printf 'export %s=%q\n' "$key" "$value"
done < <(keys)
}
print_value() {
local key="$1"
if ! is_tracked_key "$key"; then
echo "unknown key: $key" >&2
exit 2
fi
value_for "$key"
}
print_human() {
local root="$1"
local key value source
echo "agent-testing test env:"
printf ' workspace: %s\n' "$root"
printf ' NODE_ENV: %s\n' "$NODE_ENV"
printf ' env files: %s\n' "${LOADED_ENV_FILES:-none}"
echo
echo "resolved values:"
while IFS= read -r key; do
value="$(value_for "$key")"
source="$(source_for "$key")"
printf ' %-22s %s (%s)\n' "$key=$value" "" "$source"
done < <(keys)
echo
echo "checks:"
local app_origin spa_origin app_port
app_origin="$(url_origin "$VALUE_APP_URL")"
spa_origin="http://localhost:$VALUE_SPA_PORT"
if app_port="$(url_port "$VALUE_APP_URL")" && [[ "$app_port" == "$VALUE_PORT" ]]; then
printf ' OK PORT matches APP_URL (%s)\n' "$VALUE_PORT"
else
printf ' WARN PORT (%s) does not match APP_URL (%s)\n' "$VALUE_PORT" "$VALUE_APP_URL"
fi
if contains_origin "$VALUE_AUTH_TRUSTED_ORIGINS" "$app_origin"; then
printf ' OK AUTH_TRUSTED_ORIGINS includes %s\n' "$app_origin"
else
printf ' WARN AUTH_TRUSTED_ORIGINS is missing %s\n' "$app_origin"
fi
if contains_origin "$VALUE_AUTH_TRUSTED_ORIGINS" "$spa_origin"; then
printf ' OK AUTH_TRUSTED_ORIGINS includes %s\n' "$spa_origin"
else
printf ' WARN AUTH_TRUSTED_ORIGINS is missing %s\n' "$spa_origin"
fi
}
usage() {
cat << EOF
Usage:
$0 # print resolved test environment
$0 --exports # print source-able export lines
$0 --value KEY # print one resolved value
Tracked keys:
APP_URL PORT SERVER_URL AUTH_TRUSTED_ORIGINS SPA_PORT MOBILE_SPA_PORT DESKTOP_PORT
EOF
}
ROOT="$(workspace_root)"
apply_env_files "$ROOT"
apply_shell_overrides
resolve_defaults
case "${1:-}" in
"")
print_human "$ROOT"
;;
--exports)
print_exports
;;
--value)
print_value "${2:-}"
;;
-h|--help)
usage
;;
*)
echo "unknown option: $1" >&2
usage >&2
exit 2
;;
esac
.agents/skills/agent-testing/scripts/test-env.test.sh
Executable
+57
View File
@@ -0,0 +1,57 @@
#!/usr/bin/env bash
# Smoke tests for test-env.sh.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
fail() {
echo "FAIL: $*" >&2
exit 1
}
assert_eq() {
local actual="$1"
local expected="$2"
[[ "$actual" == "$expected" ]] || fail "expected '$expected', got '$actual'"
}
assert_contains() {
local file="$1"
local text="$2"
grep -Fq "$text" "$file" || fail "expected '$text' in $file"
}
tmp_dir="$(mktemp -d)"
trap 'rm -rf "$tmp_dir"' EXIT
mkdir -p "$tmp_dir/lobehub-cloud-1/.agents/skills" "$tmp_dir/lobehub/.agents/skills"
ln -s "$SCRIPT_DIR/.." "$tmp_dir/lobehub-cloud-1/.agents/skills/agent-testing"
ln -s "$SCRIPT_DIR/.." "$tmp_dir/lobehub/.agents/skills/agent-testing"
cloud_script="$tmp_dir/lobehub-cloud-1/.agents/skills/agent-testing/scripts/test-env.sh"
oss_script="$tmp_dir/lobehub/.agents/skills/agent-testing/scripts/test-env.sh"
assert_eq "$("$cloud_script" --value SERVER_URL)" "http://localhost:3021"
assert_eq "$("$cloud_script" --value SPA_PORT)" "9801"
assert_eq "$("$cloud_script" --value MOBILE_SPA_PORT)" "3811"
assert_eq "$("$cloud_script" --value DESKTOP_PORT)" "3031"
assert_eq "$("$oss_script" --value SERVER_URL)" "http://localhost:3010"
cat > "$tmp_dir/lobehub-cloud-1/.env" << 'EOF'
APP_URL=http://localhost:4123
PORT=4123
AUTH_TRUSTED_ORIGINS=http://localhost:4123,http://localhost:9823
SPA_PORT=9823
MOBILE_SPA_PORT=3823
DESKTOP_PORT=3043
EOF
assert_eq "$("$cloud_script" --value SERVER_URL)" "http://localhost:4123"
assert_eq "$("$cloud_script" --value SPA_PORT)" "9823"
"$cloud_script" --exports > "$tmp_dir/exports.out"
assert_contains "$tmp_dir/exports.out" "export APP_URL=http://localhost:4123"
assert_contains "$tmp_dir/exports.out" "export SERVER_URL=http://localhost:4123"
assert_contains "$tmp_dir/exports.out" "export AUTH_TRUSTED_ORIGINS=http://localhost:4123\\,http://localhost:9823"
echo "test-env tests passed"
.agents/skills/agent-testing/ui/web.md
+7 -6
View File
@@ -10,23 +10,24 @@ backend-only changes prefer [../cli/index.md](../cli/index.md).
## Prerequisites
- Complete [Step 0.0](../SKILL.md#00-resolve-the-current-test-environment) (resolve ports) and [Step -1](../SKILL.md#step--1--plan-approval-for-non-trivial-tests) (plan approval) first.
- Local dev server running — [../references/dev-server.md](../references/dev-server.md)
- Web auth injected into agent-browser — [../references/auth.md](../references/auth.md):
```bash
pbpaste | ./.agents/skills/agent-testing/scripts/setup-auth.sh web # after copying the Cookie header
```
- Web auth verified in agent-browser — see [auth decision flow](../references/auth.md#web--decision-flow).
## Option A — agent-browser with injected auth (recommended)
```bash
SESSION=lobehub-dev
agent-browser --session $SESSION open "http://localhost:3010/"
agent-browser --session $SESSION open "$SERVER_URL/"
agent-browser --session $SESSION snapshot -i
# interact via refs — full command reference: ../references/agent-browser.md
```
Use this session as the evidence source. Do not use ordinary Chrome screenshots
or Chrome Network records as proof for Web tests; ordinary Chrome is only a
source for copying cookies into agent-browser.
### Watch the API while driving the UI
```bash