diff --git a/.agents/skills/agent-testing/SKILL.md b/.agents/skills/agent-testing/SKILL.md index c0ed383dca..29f0334e57 100644 --- a/.agents/skills/agent-testing/SKILL.md +++ b/.agents/skills/agent-testing/SKILL.md @@ -19,9 +19,23 @@ also run as full cloud automation. Every test session follows the same four-step contract: ``` -Step 0: Env + Auth → Step 1: Pick surface → Step 2: Run → Step 3: Structured report +Step -1: Plan approval → Step 0: Env + Auth → Step 1: Pick surface → Step 2: Run → Step 3: Structured report ``` +## Step -1 — Plan approval for non-trivial tests + +Skip directly to Step 0 if: the test is a single re-run after a fix, the plan +was already agreed on, or the user gave exact commands. + +Otherwise, propose a test plan (surface, cases, expected evidence, assumptions) +and use the runtime structured question tool (`request_user_input` / +ask-user-question equivalent) with two fixed choices: + +1. `开始执行 (Recommended)` — 测试方案没问题，开始执行 +2. `先讨论下` — 方案有问题，先讨论下 + +Wait for the user's choice before proceeding. + ## Step 0 — Environment setup + auth check (mandatory) Step 0 is about getting the environment ready: **dependencies are healthy** @@ -29,6 +43,36 @@ and **auth is green**. A test run that dies halfway on a missing dependency or a login wall wastes the whole session — clear both gates BEFORE writing a single test step. +### 0.0 Resolve the current test environment + +Before starting a dev server, checking auth, opening agent-browser, or writing +test steps, print and confirm the current local test environment: + +```bash +./.agents/skills/agent-testing/scripts/test-env.sh +``` + +This command is the source of truth for local test ports. It reads the current +shell plus `.env` files using the same precedence as `scripts/runWithEnv.mts`, +then prints: + +- `APP_URL` +- `PORT` +- `SERVER_URL` +- `AUTH_TRUSTED_ORIGINS` +- `SPA_PORT` +- `MOBILE_SPA_PORT` +- `DESKTOP_PORT` + +For commands that need these values, export them from the same resolver: + +```bash +eval "$(./.agents/skills/agent-testing/scripts/test-env.sh --exports)" +``` + +Do not rely on hard-coded port tables. If the printed values do not match the +running dev server, fix/export the env first, then continue. + ### 0.1 Dependencies are installed — root AND standalone apps The root pnpm workspace does **NOT** cover every app: `pnpm-workspace.yaml` @@ -142,27 +186,35 @@ eval "$(../.agents/skills/agent-testing/scripts/init-dev-env.sh env)" BASE_URL=http://localhost:3010 HEADLESS=true bun run test:smoke ``` -### 0.4 Auth is green +### 0.4 Auth is green for the selected surface -**Auth is the gate for all automated testing.** +**Auth is the gate for automated testing, but the gate is surface-scoped.** +Pick the intended surface first when it is already clear from the task, then +check only that surface. Do not block a Web test on CLI device-code auth or an +Electron login state unless the test spans those surfaces. ```bash -./.agents/skills/agent-testing/scripts/setup-auth.sh status +./.agents/skills/agent-testing/scripts/setup-auth.sh status --surface web ``` -| Surface | Mechanism | One-key path | Standard check | -| -------- | ------------------------------------------------- | ------------------------------ | -------------------------- | -| CLI | OIDC Device Code Flow (`apps/cli/.lobehub-dev`) | `setup-auth.sh cli` | `setup-auth.sh status` | -| Web | better-auth cookie injection into `agent-browser` | `pbpaste \| setup-auth.sh web` | `setup-auth.sh web-verify` | -| Electron | App's own persistent login state | Log in once in the app | `app-probe.sh auth` | -| Bot | Native apps already logged in | — | per-platform screenshot | +Use `status` with no `--surface` only for cross-surface test plans. + +| Surface | Mechanism | One-key path | Standard check | +| -------- | ------------------------------------------------- | ------------------------------ | ----------------------------------------- | +| CLI | OIDC Device Code Flow (`apps/cli/.lobehub-dev`) | `setup-auth.sh cli` | `setup-auth.sh status --surface cli` | +| Web | better-auth cookie injection into `agent-browser` | `pbpaste \| setup-auth.sh web` | `setup-auth.sh status --surface web` | +| Electron | App's own persistent login state | Log in once in the app | `setup-auth.sh status --surface electron` | +| Bot | Native apps already logged in | — | per-platform screenshot | Login-state checks are standardized — do NOT hand-roll `window.__LOBE_STORES` eval snippets; use `scripts/app-probe.sh auth` (returns `{ isSignedIn, userId }`, works for Electron CDP and web sessions via `AB_TARGET`). -If `status` is not all green, fix auth first (the steps that need a human must be -requested from the user explicitly). Full background and failure modes: +For Web tests, the test surface is always `agent-browser --session lobehub-dev`. +The user's normal Chrome is only a source for copying the Cookie header when +`status --surface web` fails. If Chrome is already logged in, do not open a +login page; verify agent-browser first, then request the Network `Cookie:` +header only if that verification fails. Full background and failure modes: [references/auth.md](./references/auth.md). ## Step 1 — Pick the surface by change scope @@ -237,6 +289,7 @@ All under `.agents/skills/agent-testing/scripts/`: | Script | Usage | | ------------------------- | ---------------------------------------------------------------------------- | +| `test-env.sh` | Print/export the resolved local test env and ports | | `setup-auth.sh` | One-stop auth setup & status check (`status` / `cli` / `web`) | | `init-dev-env.sh` | Self-contained local dev env (`setup-db` / `seed-user` / `dev-next` / `dev`) | | `app-probe.sh` | LobeHub app probes: `auth` / `route` / `ops` / `goto ` / `errors` | @@ -284,6 +337,13 @@ Two hard rules worth front-loading: must use Markdown image syntax like `![case 1](assets/case1.png)`. Do not use bare file paths, Markdown links, or local file links as the primary visual evidence; those make the report unreadable without opening each asset. +- **Final replies must include visual evidence links.** When a run includes UI + screenshots or GIFs, include the report directory and the most important + visual artifacts in the final chat response. Each item must include a stable + label, an evidence caption describing the observed UI outcome, and a + repo-relative path, for example: + `[Image #1 - error toast shows provider auth failure](/assets/foo.png)`. + Use repo-relative paths, not absolute paths. - **Time-based behavior needs a GIF, not a screenshot.** If a case asserts change over time (streaming output, a ticking timer, loading states, animations), record it with `scripts/record-gif.sh` and embed the GIF — diff --git a/.agents/skills/agent-testing/references/auth.md b/.agents/skills/agent-testing/references/auth.md index 90077aa62d..f07f19a6ad 100644 --- a/.agents/skills/agent-testing/references/auth.md +++ b/.agents/skills/agent-testing/references/auth.md @@ -1,20 +1,31 @@ # Auth Setup for Local Agent Testing -**Auth is the gate for all automated testing.** Prepare and verify it before -writing any test step. The one-stop entry point is: +**Auth is the gate for all automated testing.** Complete +[Step 0.0](../SKILL.md#00-resolve-the-current-test-environment) first so +`SERVER_URL` and ports are resolved, then verify auth before writing any test +step. + +Initialize helpers first: ```bash -SCRIPT=".agents/skills/agent-testing/scripts/setup-auth.sh" - -$SCRIPT status # check server + CLI + web auth readiness -$SCRIPT cli # interactive CLI device-code login (must be run by the user) -pbpaste | $SCRIPT web # inject a copied Cookie header into the agent-browser session -$SCRIPT web-verify # live-check that the agent-browser session is authenticated +SCRIPT="./.agents/skills/agent-testing/scripts/setup-auth.sh" +TEST_ENV="./.agents/skills/agent-testing/scripts/test-env.sh" +eval "$($TEST_ENV --exports)" ``` -`SERVER_URL` defaults to `http://localhost:3010` (this repo's `dev:next` port). -Override it when testing against another server (e.g. `SERVER_URL=http://localhost:3011` -in the cloud repo). +Quick reference after initialization: + +| Command | Purpose | +| ------------------------------ | -------------------------------------------------- | +| `$SCRIPT status` | Check all surfaces (server + CLI + web + Electron) | +| `$SCRIPT status --surface web` | Check only the Web surface gate | +| `$SCRIPT cli` | Interactive CLI device-code login (user must run) | +| `$SCRIPT open-chrome` | Open Chrome at `SERVER_URL` with DevTools | +| `pbpaste \| $SCRIPT web` | Inject a copied Cookie header into agent-browser | +| `$SCRIPT web-verify` | Live-check agent-browser session auth | + +Use `localhost` for Web auth; better-auth cookies are stored for `localhost`, +not `127.0.0.1`. ## Per-surface overview @@ -45,6 +56,11 @@ cd apps/cli && LOBEHUB_CLI_HOME=.lobehub-dev bun src/index.ts login --server htt ## Web — better-auth cookie injection (agent-browser) +The Web test surface is `agent-browser --session lobehub-dev`. The user's +ordinary Chrome is only a cookie source; Chrome screenshots, Chrome Network +records, and Chrome logged-in state do not prove the agent-browser test session +is authenticated. + `agent-browser --headed` on macOS often creates the Chromium window off-screen — the user can't see or interact with it, so manual login inside the agent-browser session fails. Instead, copy the **better-auth session cookie** out of the @@ -53,31 +69,18 @@ user's own logged-in Chrome and inject it as a Playwright-style state file. Do **not** use this on production URLs — only local dev. Treat the cookie as a secret: don't paste it into shared logs, PRs, or commit it anywhere. -### One-key path +### Web — decision flow -1. Ask the user to copy the Cookie header **from a Network request, NOT - `document.cookie`** (`document.cookie` cannot see HttpOnly cookies, which is - exactly where better-auth puts its session): - - Open the logged-in tab (`http://localhost:/…`) in Chrome. - - `Cmd+Option+I` → **Network** tab → refresh → click any same-origin request. - - Under **Request Headers**, right-click the `Cookie:` line → **Copy value**. -2. Inject and verify in one shot: - -```bash -pbpaste | ./.agents/skills/agent-testing/scripts/setup-auth.sh web -``` - -The script filters the header down to the better-auth cookies -(`better-auth.session_token`, `better-auth.state`), builds the Playwright -`storageState` JSON, loads it into the `agent-browser` session (default name -`lobehub-dev`), opens `SERVER_URL`, and asserts the URL is not `/signin`. +1. `$SCRIPT status --surface web` — green? Start testing. Do not ask for a Cookie header. +2. Not green → `$SCRIPT open-chrome` opens Chrome at `SERVER_URL` with DevTools. +3. User copies the `Cookie:` header from Network tab → any same-origin request → Request Headers → right-click `Cookie:` → **Copy value**. Must be from Network, NOT `document.cookie` (HttpOnly cookies are invisible to `document.cookie`). +4. `pbpaste | $SCRIPT web` — filters to better-auth cookies (`session_token`, `session_data`, `state`), builds Playwright `storageState`, loads it into the `agent-browser` session (`lobehub-dev`), opens `SERVER_URL`, and asserts the URL is not `/signin`. ### Using the authenticated session ```bash -agent-browser --session lobehub-dev open "http://localhost:3010/" +agent-browser --session lobehub-dev open "$SERVER_URL/" agent-browser --session lobehub-dev snapshot -i | head -20 -# Look for the user's avatar/name in the sidebar, or absence of the signin form. ``` ### Notes @@ -90,12 +93,12 @@ agent-browser --session lobehub-dev snapshot -i | head -20 ### Common failure modes -| Symptom | Cause | Fix | -| --------------------------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------- | -| Still redirects to `/signin` after injection | User pasted from `document.cookie` → missed HttpOnly session | Re-pull from Network request Headers, not console | -| Script reports `no better-auth cookies found` | Separator wrong, or user pasted URL-decoded value | Keep the raw `Cookie:` header as-is | -| Login works briefly then expires | `better-auth.session_token` rotated (user logged out / signed in again) | Re-copy and re-inject | -| Domain mismatch | Cookie domain must be `localhost` literally, no leading dot for local dev | — | +| Symptom | Cause | Fix | +| --------------------------------------------- | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | +| Still redirects to `/signin` after injection | User pasted from `document.cookie` → missed HttpOnly session | Re-pull from Network request Headers, not console | +| Script reports `no better-auth cookies found` | User pasted the wrong value, or the cookie parser regressed | Keep the raw `Cookie:` header as-is; run `scripts/setup-auth.test.sh` if the input looks valid | +| Login works briefly then expires | `better-auth.session_token` rotated (user logged out / signed in again) | Re-copy and re-inject | +| Domain mismatch | Cookie domain must be `localhost` literally, no leading dot for local dev | — | ## Electron diff --git a/.agents/skills/agent-testing/references/dev-server.md b/.agents/skills/agent-testing/references/dev-server.md index ff6177cf7c..bcadb1f4d6 100644 --- a/.agents/skills/agent-testing/references/dev-server.md +++ b/.agents/skills/agent-testing/references/dev-server.md @@ -3,22 +3,44 @@ Single source of truth for starting / restarting the backend that all test surfaces (CLI, Electron, Web) hit. +## Resolve ports first + +Run `test-env.sh` as described in +[SKILL.md Step 0.0](../SKILL.md#00-resolve-the-current-test-environment) +before starting or probing any local test surface. + ## Ports & modes -| Command | What it runs | Port | -| ------------------- | --------------------------------------------------------- | --------------------------------- | -| `pnpm run dev:next` | Next.js backend (API + auth) | `3010` | -| `bun run dev` | Full-stack (Next.js + Vite SPA, via `devStartupSequence`) | `3010` (API) + SPA on `9876` | -| `bun run dev:spa` | Vite SPA only, proxies API to `3010` | `9876` (prints a Debug Proxy URL) | +| Command | What it runs | Port source | +| ------------------- | --------------------------------------------------------- | ------------------- | +| `pnpm run dev:next` | Next.js backend (API + auth) | `PORT` | +| `bun run dev` | Full-stack (Next.js + Vite SPA, via `devStartupSequence`) | `PORT` + `SPA_PORT` | +| `bun run dev:spa` | Vite SPA only, proxies API to `PORT` | `SPA_PORT` | -In the **cloud repo** (where this repo is the `lobehub/` submodule) the dev -server conventionally runs on `3011` — set `SERVER_URL=http://localhost:3011` -for the scripts in this skill when testing there. +In the **cloud repo** (where this repo is the `lobehub/` submodule), local +worktree names map to fallback defaults only when `.env` and shell env do not +provide values: + +| Workspace directory | Default `SERVER_URL` | +| ------------------- | -------------------------------- | +| `lobehub` | `http://localhost:3010` | +| `lobehub-cloud` | `http://localhost:3020` | +| `lobehub-cloud-1` | `http://localhost:3021` | +| `lobehub-cloud-N` | `http://localhost:$((3020 + N))` | + +`test-env.sh` and `setup-auth.sh` both use the resolved env first and these +worktree defaults only as fallback. Treat the dev-server terminal output as the +final source of truth when testing a non-standard port, then export it for every +agent-testing command: + +```bash +export SERVER_URL=http://localhost: +``` ## Health check ```bash -curl -s -o /dev/null -w '%{http_code}' http://localhost:3010/ +curl -s -o /dev/null -w '%{http_code}' "$SERVER_URL/" ``` ## Start / restart @@ -39,7 +61,7 @@ bun run dev ./.agents/skills/agent-testing/scripts/init-dev-env.sh dev # Restart — required to pick up server-side code changes -lsof -ti:3010 | xargs kill +lsof -ti:"$PORT" | xargs kill pnpm run dev:next # or, when no root .env exists: # ./.agents/skills/agent-testing/scripts/init-dev-env.sh dev-next diff --git a/.agents/skills/agent-testing/references/report.md b/.agents/skills/agent-testing/references/report.md index 781b3dd0ed..8f2ad611a7 100644 --- a/.agents/skills/agent-testing/references/report.md +++ b/.agents/skills/agent-testing/references/report.md @@ -64,7 +64,10 @@ output): not acceptable as primary visual evidence. 4. **Set the verdict** in both `report.md` and `result.json`, then link the - report directory in your final answer to the user. + report directory in your final answer to the user. If UI evidence exists, + list the key screenshot/GIF links in the final chat response. Use Markdown + link text as the evidence caption, for example: + `[Image #1 - observed outcome](/assets/case1.png)`. ## Report language (hard rule) diff --git a/.agents/skills/agent-testing/scripts/setup-auth.sh b/.agents/skills/agent-testing/scripts/setup-auth.sh index 8ac6afab93..3b2f6ea9ca 100755 --- a/.agents/skills/agent-testing/scripts/setup-auth.sh +++ b/.agents/skills/agent-testing/scripts/setup-auth.sh @@ -5,29 +5,101 @@ # test step. Background and failure modes: ../references/auth.md # # Usage: -# setup-auth.sh status # check server + CLI + web auth readiness +# setup-auth.sh status # check server + CLI + web + Electron readiness +# setup-auth.sh status --surface web # check only the Web surface gate # setup-auth.sh cli # interactive CLI device-code login (run by a human) +# setup-auth.sh open-chrome # open SERVER_URL in Chrome and show DevTools # setup-auth.sh web # stdin = Cookie header -> inject into agent-browser session # setup-auth.sh web-verify # live-check the agent-browser session is authenticated # # Env: -# SERVER_URL (default http://localhost:3010) dev server under test +# SERVER_URL (default from test-env.sh) dev server under test # SESSION (default lobehub-dev) agent-browser session name # AUTH_DIR (default ~/.lobehub-agent-testing) where web state is persisted set -euo pipefail -SERVER_URL="${SERVER_URL:-http://localhost:3010}" +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../.." && pwd)" + +workspace_root_for_port() { + local root="$REPO_ROOT" + local name + name="$(basename "$root")" + + if [[ "$name" == "lobehub" ]]; then + local parent + parent="$(cd "$root/.." && pwd)" + local parent_name + parent_name="$(basename "$parent")" + if [[ "$parent_name" == lobehub-cloud* ]]; then + root="$parent" + fi + fi + + printf '%s\n' "$root" +} + +default_server_url() { + local env_resolver resolved + env_resolver="$(dirname "${BASH_SOURCE[0]}")/test-env.sh" + if [[ -x "$env_resolver" ]]; then + resolved="$("$env_resolver" --value SERVER_URL 2> /dev/null || true)" + if [[ -n "$resolved" ]]; then + printf '%s\n' "$resolved" + return 0 + fi + fi + + local root name suffix port + root="$(workspace_root_for_port)" + name="$(basename "$root")" + + case "$name" in + lobehub-cloud) + port=3020 + ;; + lobehub-cloud-*) + suffix="${name#lobehub-cloud-}" + if [[ "$suffix" =~ ^[0-9]+$ ]]; then + port=$((3020 + 10#$suffix)) + else + port=3010 + fi + ;; + *) + port=3010 + ;; + esac + + printf 'http://localhost:%s\n' "$port" +} + +SERVER_URL="${SERVER_URL:-$(default_server_url)}" SESSION="${SESSION:-lobehub-dev}" AUTH_DIR="${AUTH_DIR:-$HOME/.lobehub-agent-testing}" STATE_FILE="$AUTH_DIR/web-state.json" -REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../.." && pwd)" CLI_HOME="$REPO_ROOT/apps/cli/.lobehub-dev" ok() { printf ' \033[32m✔\033[0m %s\n' "$1"; } bad() { printf ' \033[31m✘\033[0m %s\n' "$1"; } note() { printf ' %s\n' "$1"; } +usage() { + cat << EOF +Usage: + $0 status [--surface all|cli|web|electron] + $0 cli + $0 open-chrome [--dry-run] + $0 web + $0 web-verify + +Env: + SERVER_URL=$SERVER_URL + SESSION=$SESSION + AUTH_DIR=$AUTH_DIR +EOF +} + check_server() { local code code=$(curl -s -o /dev/null -w '%{http_code}' "$SERVER_URL/" 2> /dev/null || true) @@ -54,13 +126,23 @@ check_cli() { check_web() { if [[ -f "$STATE_FILE" ]]; then ok "web auth state saved ($STATE_FILE)" - note "live-verify: $0 web-verify" else bad "no web auth state for agent-browser" note "copy the Cookie header from Chrome DevTools (Network tab), then:" note "pbpaste | $0 web (see references/auth.md)" return 1 fi + cmd_web_verify --skip-server-check +} + +check_agent_browser() { + if command -v agent-browser > /dev/null 2>&1; then + ok "agent-browser available" + else + bad "agent-browser command not found" + note "install or expose agent-browser before Web/Electron UI testing" + return 1 + fi } check_electron() { @@ -84,16 +166,75 @@ check_electron() { } cmd_status() { - echo "agent-testing auth status (SERVER_URL=$SERVER_URL):" + local surface="all" + while [[ $# -gt 0 ]]; do + case "$1" in + --surface) + if [[ $# -lt 2 ]]; then + echo "--surface requires one of: all, cli, web, electron" >&2 + return 2 + fi + surface="${2:-}" + shift 2 + ;; + --surface=*) + surface="${1#*=}" + shift + ;; + all|cli|web|electron) + surface="$1" + shift + ;; + -h|--help) + usage + return 0 + ;; + *) + echo "unknown status option: $1" >&2 + usage >&2 + return 2 + ;; + esac + done + + case "$surface" in + all|cli|web|electron) ;; + "") + echo "--surface requires one of: all, cli, web, electron" >&2 + return 2 + ;; + *) + echo "unknown surface: $surface" >&2 + usage >&2 + return 2 + ;; + esac + + echo "agent-testing auth status (surface=$surface, SERVER_URL=$SERVER_URL):" local rc=0 - check_server || rc=1 - check_cli || rc=1 - check_web || rc=1 - check_electron || rc=1 + case "$surface" in + all) + check_server || rc=1 + check_cli || rc=1 + check_web || rc=1 + check_electron || rc=1 + ;; + cli) + check_server || rc=1 + check_cli || rc=1 + ;; + web) + check_server || rc=1 + check_web || rc=1 + ;; + electron) + check_electron || rc=1 + ;; + esac if [[ $rc -eq 0 ]]; then - echo "all green — safe to start automated testing." + echo "$surface auth green — safe to start automated testing on this surface." else - echo "auth NOT ready — fix the ✘ items before writing any test step." + echo "$surface auth NOT ready — fix the ✘ items before writing any test step." fi return $rc } @@ -105,23 +246,90 @@ cmd_cli() { LOBEHUB_CLI_HOME=.lobehub-dev bun src/index.ts login --server "$SERVER_URL" } +cmd_open_chrome() { + local mode="${1:-}" + if [[ "$mode" != "" && "$mode" != "--dry-run" ]]; then + echo "unknown open-chrome option: $mode" >&2 + usage >&2 + return 2 + fi + + if [[ "$mode" == "--dry-run" ]]; then + echo "would open Google Chrome at $SERVER_URL/" + echo "would press Cmd+Option+I to open DevTools" + echo "would open DevTools command menu and run 'Show Network'" + return 0 + fi + + if [[ "$(uname -s)" != "Darwin" ]]; then + bad "open-chrome is macOS-only" + note "open $SERVER_URL/ in your browser and open DevTools manually" + return 1 + fi + + if ! command -v osascript > /dev/null 2>&1; then + bad "osascript not found" + note "open $SERVER_URL/ in Chrome and press Cmd+Option+I manually" + return 1 + fi + + SERVER_URL="$SERVER_URL" osascript << 'OSA' +set targetUrl to (system attribute "SERVER_URL") & "/" + +tell application "Google Chrome" + activate + if (count of windows) = 0 then + make new window + end if + tell front window to make new tab with properties {URL:targetUrl} +end tell + +delay 1 + +tell application "System Events" + tell process "Google Chrome" + set frontmost to true + keystroke "i" using {command down, option down} + delay 1 + keystroke "p" using {command down, shift down} + delay 0.2 + keystroke "Show Network" + key code 36 + end tell +end tell +OSA + ok "opened Chrome at $SERVER_URL/ and requested DevTools Network panel" +} + # Build a Playwright storageState file from a raw Cookie header on stdin, # keeping only the better-auth cookies. See references/auth.md for why the # header must come from a Network request (HttpOnly) and why httpOnly=false. cmd_web() { mkdir -p "$AUTH_DIR" - python3 - "$STATE_FILE" << 'PY' -import json, sys, time + local raw + raw="$(cat)" + COOKIE_INPUT="$raw" python3 - "$STATE_FILE" << 'PY' +import json, os, sys, time -raw = sys.stdin.read().strip() -if raw.lower().startswith("cookie:"): - raw = raw.split(":", 1)[1].strip() +raw = os.environ.get("COOKIE_INPUT", "").strip() +cookie_lines = [] +for line in raw.splitlines(): + stripped = line.strip() + if not stripped: + continue + if stripped.lower().startswith("cookie:"): + cookie_lines.append(stripped.split(":", 1)[1].strip()) + else: + cookie_lines.append(stripped) -WANTED = {"better-auth.session_token", "better-auth.state"} +raw = "; ".join(cookie_lines) + +WANTED = {"better-auth.session_token", "better-auth.session_data", "better-auth.state"} exp = int(time.time()) + 30 * 24 * 3600 # 30 days cookies = [] -for pair in raw.split("; "): +for pair in raw.split(";"): + pair = pair.strip() if "=" not in pair: continue name, _, value = pair.partition("=") @@ -146,14 +354,35 @@ with open(sys.argv[1], "w") as f: json.dump({"cookies": cookies, "origins": []}, f, indent=2) print(f"wrote {len(cookies)} cookie(s) to {sys.argv[1]}") PY - agent-browser --session "$SESSION" state load "$STATE_FILE" cmd_web_verify } cmd_web_verify() { - agent-browser --session "$SESSION" open "$SERVER_URL/" > /dev/null + local skip_server_check="${1:-}" + if [[ "$skip_server_check" != "--skip-server-check" ]]; then + check_server || return 1 + fi + if [[ ! -f "$STATE_FILE" ]]; then + bad "no web auth state for agent-browser" + note "copy the Cookie header from Chrome DevTools (Network tab), then:" + note "pbpaste | $0 web" + return 1 + fi + check_agent_browser || return 1 + if ! agent-browser --session "$SESSION" state load "$STATE_FILE" > /dev/null; then + bad "failed to load web auth state into agent-browser session '$SESSION'" + return 1 + fi + if ! agent-browser --session "$SESSION" open "$SERVER_URL/" > /dev/null; then + bad "failed to open $SERVER_URL in agent-browser session '$SESSION'" + return 1 + fi local url - url=$(agent-browser --session "$SESSION" get url) + url=$(agent-browser --session "$SESSION" get url 2> /dev/null || true) + if [[ -z "$url" ]]; then + bad "agent-browser session '$SESSION' did not report a current URL" + return 1 + fi if [[ "$url" == *"/signin"* || "$url" == *"/login"* ]]; then bad "agent-browser session '$SESSION' NOT authenticated (landed on $url)" note "re-copy the Cookie header and re-run: pbpaste | $0 web" @@ -163,10 +392,18 @@ cmd_web_verify() { } case "${1:-status}" in - status) cmd_status ;; + status) + shift || true + cmd_status "$@" + ;; cli) cmd_cli ;; + open-chrome) + shift || true + cmd_open_chrome "$@" + ;; web) cmd_web ;; web-verify) cmd_web_verify ;; + -h|--help) usage ;; *) echo "Usage: $0 {status|cli|web|web-verify}" >&2 exit 2 diff --git a/.agents/skills/agent-testing/scripts/setup-auth.test.sh b/.agents/skills/agent-testing/scripts/setup-auth.test.sh new file mode 100755 index 0000000000..422d02516f --- /dev/null +++ b/.agents/skills/agent-testing/scripts/setup-auth.test.sh @@ -0,0 +1,115 @@ +#!/usr/bin/env bash +# Smoke tests for setup-auth.sh. Uses a temporary agent-browser stub and local +# HTTP server, so it does not need real browser auth. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SCRIPT="$SCRIPT_DIR/setup-auth.sh" + +fail() { + echo "FAIL: $*" >&2 + exit 1 +} + +assert_contains() { + local file="$1" + local text="$2" + grep -Fq "$text" "$file" || fail "expected '$text' in $file" +} + +tmp_dir="$(mktemp -d)" +server_pid="" + +cleanup() { + if [[ -n "$server_pid" ]]; then + kill "$server_pid" > /dev/null 2>&1 || true + wait "$server_pid" > /dev/null 2>&1 || true + fi + rm -rf "$tmp_dir" +} +trap cleanup EXIT + +port="$(python3 - << 'PY' +import socket + +sock = socket.socket() +sock.bind(("127.0.0.1", 0)) +print(sock.getsockname()[1]) +sock.close() +PY +)" + +python3 -m http.server "$port" --bind localhost --directory "$tmp_dir" \ + > "$tmp_dir/http.log" 2>&1 & +server_pid="$!" + +server_url="http://localhost:$port" +for _ in {1..50}; do + if curl -s -o /dev/null "$server_url/"; then + break + fi + sleep 0.1 +done +curl -s -o /dev/null "$server_url/" || fail "test HTTP server did not start" + +mkdir -p "$tmp_dir/bin" "$tmp_dir/auth" +cat > "$tmp_dir/bin/agent-browser" << 'SH' +#!/usr/bin/env bash +set -euo pipefail + +if [[ "${1:-}" == "--session" ]]; then + shift 2 +fi + +case "${1:-}" in + state) + [[ "${2:-}" == "load" ]] || exit 2 + [[ -f "${3:-}" ]] || exit 1 + ;; + open) + printf '%s\n' "${2:-}" > "${AGENT_BROWSER_URL_FILE:?}" + ;; + get) + [[ "${2:-}" == "url" ]] || exit 2 + cat "${AGENT_BROWSER_URL_FILE:?}" + ;; + *) + echo "unexpected agent-browser command: $*" >&2 + exit 2 + ;; +esac +SH +chmod +x "$tmp_dir/bin/agent-browser" + +export PATH="$tmp_dir/bin:$PATH" +export AUTH_DIR="$tmp_dir/auth" +export SESSION="setup-auth-test" +export SERVER_URL="$server_url" +export AGENT_BROWSER_URL_FILE="$tmp_dir/current-url" + +cookie_header="Cookie: foo=bar; better-auth.session_token=test.token; better-auth.session_data=encoded%3D; theme=dark" +printf '%s\n' "$cookie_header" | "$SCRIPT" web > "$tmp_dir/web.out" + +python3 - "$AUTH_DIR/web-state.json" << 'PY' +import json, sys + +with open(sys.argv[1]) as f: + state = json.load(f) + +names = {cookie["name"] for cookie in state["cookies"]} +expected = {"better-auth.session_token", "better-auth.session_data"} +if names != expected: + raise SystemExit(f"unexpected cookies: {sorted(names)}") +PY + +"$SCRIPT" status --surface web > "$tmp_dir/status.out" +assert_contains "$tmp_dir/status.out" "surface=web" +assert_contains "$tmp_dir/status.out" "web auth green" + +if printf 'foo=bar\n' | "$SCRIPT" web > "$tmp_dir/invalid.out" 2> "$tmp_dir/invalid.err"; then + fail "invalid cookie unexpectedly passed" +fi +assert_contains "$tmp_dir/invalid.err" "no better-auth cookies found" + +echo "setup-auth tests passed" diff --git a/.agents/skills/agent-testing/scripts/test-env.sh b/.agents/skills/agent-testing/scripts/test-env.sh new file mode 100755 index 0000000000..f8eb77a744 --- /dev/null +++ b/.agents/skills/agent-testing/scripts/test-env.sh @@ -0,0 +1,377 @@ +#!/usr/bin/env bash +# Print the resolved local test environment for agent-testing. +# +# This is intentionally read-only. It mirrors scripts/runWithEnv.mts precedence: +# .env -> .env.$NODE_ENV -> .env.local -> .env.$NODE_ENV.local, then shell env. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../../../.." && pwd)" +NODE_ENV="${NODE_ENV:-development}" + +VALUE_APP_URL="" +VALUE_PORT="" +VALUE_SERVER_URL="" +VALUE_AUTH_TRUSTED_ORIGINS="" +VALUE_SPA_PORT="" +VALUE_MOBILE_SPA_PORT="" +VALUE_DESKTOP_PORT="" + +SOURCE_APP_URL="" +SOURCE_PORT="" +SOURCE_SERVER_URL="" +SOURCE_AUTH_TRUSTED_ORIGINS="" +SOURCE_SPA_PORT="" +SOURCE_MOBILE_SPA_PORT="" +SOURCE_DESKTOP_PORT="" + +LOADED_ENV_FILES="" + +keys() { + printf '%s\n' \ + APP_URL \ + PORT \ + SERVER_URL \ + AUTH_TRUSTED_ORIGINS \ + SPA_PORT \ + MOBILE_SPA_PORT \ + DESKTOP_PORT +} + +trim() { + local value="$1" + value="${value#"${value%%[![:space:]]*}"}" + value="${value%"${value##*[![:space:]]}"}" + printf '%s' "$value" +} + +workspace_root() { + local root="$REPO_ROOT" + local name + name="$(basename "$root")" + + if [[ "$name" == "lobehub" ]]; then + local parent parent_name + parent="$(cd "$root/.." && pwd)" + parent_name="$(basename "$parent")" + if [[ "$parent_name" == lobehub-cloud* ]]; then + root="$parent" + fi + fi + + printf '%s\n' "$root" +} + +workspace_offset() { + local name="$1" + + case "$name" in + lobehub-cloud) + printf '0\n' + ;; + lobehub-cloud-*) + local suffix="${name#lobehub-cloud-}" + if [[ "$suffix" =~ ^[0-9]+$ ]]; then + printf '%s\n' "$((10#$suffix))" + else + printf '\n' + fi + ;; + *) + printf '\n' + ;; + esac +} + +default_port() { + local base="$1" + local fallback="$2" + local root name offset + root="$(workspace_root)" + name="$(basename "$root")" + offset="$(workspace_offset "$name")" + + if [[ -n "$offset" ]]; then + printf '%s\n' "$((base + offset))" + else + printf '%s\n' "$fallback" + fi +} + +url_port() { + local url="$1" + local hostport + hostport="${url#*://}" + hostport="${hostport%%/*}" + + if [[ "$hostport" == *:* ]]; then + local port="${hostport##*:}" + if [[ "$port" =~ ^[0-9]+$ ]]; then + printf '%s\n' "$port" + return 0 + fi + fi + + return 1 +} + +url_origin() { + local url="$1" + local scheme rest hostport + if [[ "$url" == *"://"* ]]; then + scheme="${url%%://*}" + rest="${url#*://}" + hostport="${rest%%/*}" + printf '%s://%s\n' "$scheme" "$hostport" + else + printf '%s\n' "$url" + fi +} + +set_value() { + local key="$1" + local value="$2" + local source="$3" + + case "$key" in + APP_URL) VALUE_APP_URL="$value"; SOURCE_APP_URL="$source" ;; + PORT) VALUE_PORT="$value"; SOURCE_PORT="$source" ;; + SERVER_URL) VALUE_SERVER_URL="$value"; SOURCE_SERVER_URL="$source" ;; + AUTH_TRUSTED_ORIGINS) VALUE_AUTH_TRUSTED_ORIGINS="$value"; SOURCE_AUTH_TRUSTED_ORIGINS="$source" ;; + SPA_PORT) VALUE_SPA_PORT="$value"; SOURCE_SPA_PORT="$source" ;; + MOBILE_SPA_PORT) VALUE_MOBILE_SPA_PORT="$value"; SOURCE_MOBILE_SPA_PORT="$source" ;; + DESKTOP_PORT) VALUE_DESKTOP_PORT="$value"; SOURCE_DESKTOP_PORT="$source" ;; + esac +} + +value_for() { + case "$1" in + APP_URL) printf '%s\n' "$VALUE_APP_URL" ;; + PORT) printf '%s\n' "$VALUE_PORT" ;; + SERVER_URL) printf '%s\n' "$VALUE_SERVER_URL" ;; + AUTH_TRUSTED_ORIGINS) printf '%s\n' "$VALUE_AUTH_TRUSTED_ORIGINS" ;; + SPA_PORT) printf '%s\n' "$VALUE_SPA_PORT" ;; + MOBILE_SPA_PORT) printf '%s\n' "$VALUE_MOBILE_SPA_PORT" ;; + DESKTOP_PORT) printf '%s\n' "$VALUE_DESKTOP_PORT" ;; + esac +} + +source_for() { + case "$1" in + APP_URL) printf '%s\n' "$SOURCE_APP_URL" ;; + PORT) printf '%s\n' "$SOURCE_PORT" ;; + SERVER_URL) printf '%s\n' "$SOURCE_SERVER_URL" ;; + AUTH_TRUSTED_ORIGINS) printf '%s\n' "$SOURCE_AUTH_TRUSTED_ORIGINS" ;; + SPA_PORT) printf '%s\n' "$SOURCE_SPA_PORT" ;; + MOBILE_SPA_PORT) printf '%s\n' "$SOURCE_MOBILE_SPA_PORT" ;; + DESKTOP_PORT) printf '%s\n' "$SOURCE_DESKTOP_PORT" ;; + esac +} + +is_tracked_key() { + case "$1" in + APP_URL|PORT|SERVER_URL|AUTH_TRUSTED_ORIGINS|SPA_PORT|MOBILE_SPA_PORT|DESKTOP_PORT) return 0 ;; + *) return 1 ;; + esac +} + +parse_env_file() { + local file="$1" + local root="$2" + local label="${file#$root/}" + local line key value + + [[ -f "$file" ]] || return 0 + if [[ -z "$LOADED_ENV_FILES" ]]; then + LOADED_ENV_FILES="$label" + else + LOADED_ENV_FILES="$LOADED_ENV_FILES, $label" + fi + + while IFS= read -r line || [[ -n "$line" ]]; do + line="$(trim "$line")" + [[ -z "$line" || "$line" == \#* ]] && continue + + if [[ "$line" == export[[:space:]]* ]]; then + line="$(trim "${line#export}")" + fi + + [[ "$line" == *=* ]] || continue + key="$(trim "${line%%=*}")" + value="$(trim "${line#*=}")" + is_tracked_key "$key" || continue + + if [[ "$value" == \"*\" && "$value" == *\" && ${#value} -ge 2 ]]; then + value="${value:1:${#value}-2}" + elif [[ "$value" == \'* && "$value" == *\' && ${#value} -ge 2 ]]; then + value="${value:1:${#value}-2}" + fi + + set_value "$key" "$value" "$label" + done < "$file" +} + +apply_env_files() { + local root="$1" + parse_env_file "$root/.env" "$root" + parse_env_file "$root/.env.$NODE_ENV" "$root" + parse_env_file "$root/.env.local" "$root" + parse_env_file "$root/.env.$NODE_ENV.local" "$root" +} + +apply_shell_overrides() { + local key value + while IFS= read -r key; do + if [[ -n "${!key+x}" ]]; then + value="${!key}" + set_value "$key" "$value" "shell" + fi + done < <(keys) +} + +resolve_defaults() { + local app_port spa_port mobile_spa_port desktop_port + app_port="$(default_port 3020 3010)" + spa_port="$(default_port 9800 9876)" + mobile_spa_port="$(default_port 3810 3012)" + desktop_port="$(default_port 3030 3015)" + + if [[ -z "$VALUE_APP_URL" ]]; then + set_value APP_URL "http://localhost:$app_port" "inferred" + fi + + if [[ -z "$VALUE_PORT" ]]; then + if app_port="$(url_port "$VALUE_APP_URL")"; then + set_value PORT "$app_port" "inferred from APP_URL" + else + set_value PORT "$(default_port 3020 3010)" "inferred" + fi + fi + + if [[ -z "$VALUE_SERVER_URL" ]]; then + set_value SERVER_URL "$VALUE_APP_URL" "from APP_URL" + fi + + if [[ -z "$VALUE_SPA_PORT" ]]; then + set_value SPA_PORT "$spa_port" "inferred" + fi + + if [[ -z "$VALUE_MOBILE_SPA_PORT" ]]; then + set_value MOBILE_SPA_PORT "$mobile_spa_port" "inferred" + fi + + if [[ -z "$VALUE_DESKTOP_PORT" ]]; then + set_value DESKTOP_PORT "$desktop_port" "inferred" + fi + + if [[ -z "$VALUE_AUTH_TRUSTED_ORIGINS" ]]; then + set_value AUTH_TRUSTED_ORIGINS "$(url_origin "$VALUE_APP_URL"),http://localhost:$VALUE_SPA_PORT" "inferred" + fi +} + +contains_origin() { + local list="$1" + local expected="$2" + local item + IFS=',' read -r -a items <<< "$list" + for item in "${items[@]}"; do + item="$(trim "$item")" + [[ "$item" == "$expected" ]] && return 0 + done + return 1 +} + +print_exports() { + local key value + while IFS= read -r key; do + value="$(value_for "$key")" + printf 'export %s=%q\n' "$key" "$value" + done < <(keys) +} + +print_value() { + local key="$1" + if ! is_tracked_key "$key"; then + echo "unknown key: $key" >&2 + exit 2 + fi + value_for "$key" +} + +print_human() { + local root="$1" + local key value source + + echo "agent-testing test env:" + printf ' workspace: %s\n' "$root" + printf ' NODE_ENV: %s\n' "$NODE_ENV" + printf ' env files: %s\n' "${LOADED_ENV_FILES:-none}" + echo + echo "resolved values:" + while IFS= read -r key; do + value="$(value_for "$key")" + source="$(source_for "$key")" + printf ' %-22s %s (%s)\n' "$key=$value" "" "$source" + done < <(keys) + echo + echo "checks:" + + local app_origin spa_origin app_port + app_origin="$(url_origin "$VALUE_APP_URL")" + spa_origin="http://localhost:$VALUE_SPA_PORT" + if app_port="$(url_port "$VALUE_APP_URL")" && [[ "$app_port" == "$VALUE_PORT" ]]; then + printf ' OK PORT matches APP_URL (%s)\n' "$VALUE_PORT" + else + printf ' WARN PORT (%s) does not match APP_URL (%s)\n' "$VALUE_PORT" "$VALUE_APP_URL" + fi + + if contains_origin "$VALUE_AUTH_TRUSTED_ORIGINS" "$app_origin"; then + printf ' OK AUTH_TRUSTED_ORIGINS includes %s\n' "$app_origin" + else + printf ' WARN AUTH_TRUSTED_ORIGINS is missing %s\n' "$app_origin" + fi + + if contains_origin "$VALUE_AUTH_TRUSTED_ORIGINS" "$spa_origin"; then + printf ' OK AUTH_TRUSTED_ORIGINS includes %s\n' "$spa_origin" + else + printf ' WARN AUTH_TRUSTED_ORIGINS is missing %s\n' "$spa_origin" + fi +} + +usage() { + cat << EOF +Usage: + $0 # print resolved test environment + $0 --exports # print source-able export lines + $0 --value KEY # print one resolved value + +Tracked keys: + APP_URL PORT SERVER_URL AUTH_TRUSTED_ORIGINS SPA_PORT MOBILE_SPA_PORT DESKTOP_PORT +EOF +} + +ROOT="$(workspace_root)" +apply_env_files "$ROOT" +apply_shell_overrides +resolve_defaults + +case "${1:-}" in + "") + print_human "$ROOT" + ;; + --exports) + print_exports + ;; + --value) + print_value "${2:-}" + ;; + -h|--help) + usage + ;; + *) + echo "unknown option: $1" >&2 + usage >&2 + exit 2 + ;; +esac diff --git a/.agents/skills/agent-testing/scripts/test-env.test.sh b/.agents/skills/agent-testing/scripts/test-env.test.sh new file mode 100755 index 0000000000..f778ef0174 --- /dev/null +++ b/.agents/skills/agent-testing/scripts/test-env.test.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# Smoke tests for test-env.sh. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +fail() { + echo "FAIL: $*" >&2 + exit 1 +} + +assert_eq() { + local actual="$1" + local expected="$2" + [[ "$actual" == "$expected" ]] || fail "expected '$expected', got '$actual'" +} + +assert_contains() { + local file="$1" + local text="$2" + grep -Fq "$text" "$file" || fail "expected '$text' in $file" +} + +tmp_dir="$(mktemp -d)" +trap 'rm -rf "$tmp_dir"' EXIT + +mkdir -p "$tmp_dir/lobehub-cloud-1/.agents/skills" "$tmp_dir/lobehub/.agents/skills" +ln -s "$SCRIPT_DIR/.." "$tmp_dir/lobehub-cloud-1/.agents/skills/agent-testing" +ln -s "$SCRIPT_DIR/.." "$tmp_dir/lobehub/.agents/skills/agent-testing" + +cloud_script="$tmp_dir/lobehub-cloud-1/.agents/skills/agent-testing/scripts/test-env.sh" +oss_script="$tmp_dir/lobehub/.agents/skills/agent-testing/scripts/test-env.sh" + +assert_eq "$("$cloud_script" --value SERVER_URL)" "http://localhost:3021" +assert_eq "$("$cloud_script" --value SPA_PORT)" "9801" +assert_eq "$("$cloud_script" --value MOBILE_SPA_PORT)" "3811" +assert_eq "$("$cloud_script" --value DESKTOP_PORT)" "3031" +assert_eq "$("$oss_script" --value SERVER_URL)" "http://localhost:3010" + +cat > "$tmp_dir/lobehub-cloud-1/.env" << 'EOF' +APP_URL=http://localhost:4123 +PORT=4123 +AUTH_TRUSTED_ORIGINS=http://localhost:4123,http://localhost:9823 +SPA_PORT=9823 +MOBILE_SPA_PORT=3823 +DESKTOP_PORT=3043 +EOF + +assert_eq "$("$cloud_script" --value SERVER_URL)" "http://localhost:4123" +assert_eq "$("$cloud_script" --value SPA_PORT)" "9823" +"$cloud_script" --exports > "$tmp_dir/exports.out" +assert_contains "$tmp_dir/exports.out" "export APP_URL=http://localhost:4123" +assert_contains "$tmp_dir/exports.out" "export SERVER_URL=http://localhost:4123" +assert_contains "$tmp_dir/exports.out" "export AUTH_TRUSTED_ORIGINS=http://localhost:4123\\,http://localhost:9823" + +echo "test-env tests passed" diff --git a/.agents/skills/agent-testing/ui/web.md b/.agents/skills/agent-testing/ui/web.md index 78678b4ad6..85a183546a 100644 --- a/.agents/skills/agent-testing/ui/web.md +++ b/.agents/skills/agent-testing/ui/web.md @@ -10,23 +10,24 @@ backend-only changes prefer [../cli/index.md](../cli/index.md). ## Prerequisites +- Complete [Step 0.0](../SKILL.md#00-resolve-the-current-test-environment) (resolve ports) and [Step -1](../SKILL.md#step--1--plan-approval-for-non-trivial-tests) (plan approval) first. - Local dev server running — [../references/dev-server.md](../references/dev-server.md) -- Web auth injected into agent-browser — [../references/auth.md](../references/auth.md): - -```bash -pbpaste | ./.agents/skills/agent-testing/scripts/setup-auth.sh web # after copying the Cookie header -``` +- Web auth verified in agent-browser — see [auth decision flow](../references/auth.md#web--decision-flow). ## Option A — agent-browser with injected auth (recommended) ```bash SESSION=lobehub-dev -agent-browser --session $SESSION open "http://localhost:3010/" +agent-browser --session $SESSION open "$SERVER_URL/" agent-browser --session $SESSION snapshot -i # interact via refs — full command reference: ../references/agent-browser.md ``` +Use this session as the evidence source. Do not use ordinary Chrome screenshots +or Chrome Network records as proof for Web tests; ordinary Chrome is only a +source for copying cookies into agent-browser. + ### Watch the API while driving the UI ```bash diff --git a/locales/en-US/modelProvider.json b/locales/en-US/modelProvider.json index c286a0d69a..0fda42f6f3 100644 --- a/locales/en-US/modelProvider.json +++ b/locales/en-US/modelProvider.json @@ -307,6 +307,8 @@ "providerModels.list.enabledActions.sort": "Custom Model Sorting", "providerModels.list.enabledEmpty": "No enabled models available. Please enable your preferred models from the list below~", "providerModels.list.fetcher.clear": "Clear fetched models", + "providerModels.list.fetcher.error": "Failed to fetch models: {{message}}", + "providerModels.list.fetcher.errorFallback": "Unknown error", "providerModels.list.fetcher.fetch": "Fetch models", "providerModels.list.fetcher.fetching": "Fetching model list...", "providerModels.list.fetcher.latestTime": "Last updated: {{time}}", diff --git a/locales/zh-CN/modelProvider.json b/locales/zh-CN/modelProvider.json index eeaa5541fa..378c1b79ef 100644 --- a/locales/zh-CN/modelProvider.json +++ b/locales/zh-CN/modelProvider.json @@ -307,6 +307,8 @@ "providerModels.list.enabledActions.sort": "自定义模型排序", "providerModels.list.enabledEmpty": "还没有启用模型。先从下方列表挑一个启用，之后随时可以调整", "providerModels.list.fetcher.clear": "清除获取的模型", + "providerModels.list.fetcher.error": "获取模型列表失败：{{message}}", + "providerModels.list.fetcher.errorFallback": "未知错误", "providerModels.list.fetcher.fetch": "获取模型列表", "providerModels.list.fetcher.fetching": "正在获取模型列表…", "providerModels.list.fetcher.latestTime": "上次更新时间：{{time}}", diff --git a/packages/locales/src/default/modelProvider.ts b/packages/locales/src/default/modelProvider.ts index d7f09c3e63..c856469adb 100644 --- a/packages/locales/src/default/modelProvider.ts +++ b/packages/locales/src/default/modelProvider.ts @@ -384,6 +384,8 @@ export default { 'providerModels.list.enabledEmpty': 'No enabled models available. Please enable your preferred models from the list below~', 'providerModels.list.fetcher.clear': 'Clear fetched models', + 'providerModels.list.fetcher.error': 'Failed to fetch models: {{message}}', + 'providerModels.list.fetcher.errorFallback': 'Unknown error', 'providerModels.list.fetcher.fetch': 'Fetch models', 'providerModels.list.fetcher.fetching': 'Fetching model list...', 'providerModels.list.fetcher.latestTime': 'Last updated: {{time}}', diff --git a/packages/model-runtime/src/providers/aihubmix/index.test.ts b/packages/model-runtime/src/providers/aihubmix/index.test.ts index b7ddb5abb0..e7e24baa89 100644 --- a/packages/model-runtime/src/providers/aihubmix/index.test.ts +++ b/packages/model-runtime/src/providers/aihubmix/index.test.ts @@ -4,6 +4,12 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import * as modelParse from '../../utils/modelParse'; import { LobeAiHubMixAI, params } from './index'; +const loadModelsMock = vi.hoisted(() => vi.fn().mockResolvedValue([])); + +vi.mock('@lobechat/business-model-bank/model-config', () => ({ + loadModels: loadModelsMock, +})); + const mockFetch = vi.fn(); global.fetch = mockFetch; @@ -21,6 +27,7 @@ describe('LobeAiHubMixAI', () => { let instance: InstanceType; beforeEach(() => { + loadModelsMock.mockResolvedValue([]); instance = new LobeAiHubMixAI({ apiKey: 'test_api_key' }); }); @@ -190,27 +197,24 @@ describe('LobeAiHubMixAI', () => { expect(passedModels.find((m) => m.id === 'gpt-4o')).toBeDefined(); }); - it('should return empty array on non-ok HTTP response', async () => { + it('should throw on non-ok HTTP response', async () => { mockFetch.mockResolvedValueOnce(new Response('Unauthorized', { status: 401 })); - const list = await instance.models(); - expect(list).toEqual([]); + await expect(instance.models()).rejects.toThrow('HTTP 401: Unauthorized'); }); - it('should return empty array on network error', async () => { + it('should throw on network error', async () => { mockFetch.mockRejectedValueOnce(new Error('Network Error')); - const list = await instance.models(); - expect(list).toEqual([]); + await expect(instance.models()).rejects.toThrow('Network Error'); }); - it('should return empty array on timeout (AbortError)', async () => { + it('should throw on timeout (AbortError)', async () => { mockFetch.mockRejectedValueOnce( Object.assign(new Error('The operation was aborted'), { name: 'AbortError' }), ); - const list = await instance.models(); - expect(list).toEqual([]); + await expect(instance.models()).rejects.toThrow('The operation was aborted'); }); }); }); diff --git a/packages/model-runtime/src/providers/aihubmix/index.ts b/packages/model-runtime/src/providers/aihubmix/index.ts index b7cf9b4f49..b2c15743a7 100644 --- a/packages/model-runtime/src/providers/aihubmix/index.ts +++ b/packages/model-runtime/src/providers/aihubmix/index.ts @@ -162,6 +162,7 @@ export const params: CreateRouterRuntimeOptions = { // 'APP-Code' is an AiHubMix-required client identifier. const controller = new AbortController(); const timeoutId = setTimeout(() => controller.abort(), 10_000); + try { const response = await fetch('https://aihubmix.com/api/v1/models', { headers: { @@ -181,12 +182,6 @@ export const params: CreateRouterRuntimeOptions = { .filter((m: any) => !UNSUPPORTED_AIHUBMIX_TYPES.has(m.types ?? '')) .map((m: any) => mapAiHubMixModel(m)); return await processMultiProviderModelList(modelList, 'aihubmix'); - } catch (error) { - console.warn( - 'Failed to fetch AiHubMix models. Please ensure your AiHubMix API key is valid:', - error, - ); - return []; } finally { clearTimeout(timeoutId); } diff --git a/packages/model-runtime/src/providers/akashchat/index.test.ts b/packages/model-runtime/src/providers/akashchat/index.test.ts index 2061a551ff..f9c366fd1e 100644 --- a/packages/model-runtime/src/providers/akashchat/index.test.ts +++ b/packages/model-runtime/src/providers/akashchat/index.test.ts @@ -5,6 +5,12 @@ import { beforeEach, describe, expect, it, vi } from 'vitest'; import { testProvider } from '../../providerTestUtils'; import { LobeAkashChatAI, params } from './index'; +const loadModelsMock = vi.hoisted(() => vi.fn().mockResolvedValue([])); + +vi.mock('@lobechat/business-model-bank/model-config', () => ({ + loadModels: loadModelsMock, +})); + const provider = ModelProvider.AkashChat; const defaultBaseURL = 'https://chatapi.akash.network/api/v1'; @@ -313,8 +319,7 @@ describe('LobeAkashChatAI - custom features', () => { expect(models).toEqual([]); }); - it('should handle API errors gracefully and return empty array', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + it('should throw when model API fails', async () => { const mockClient = { apiKey: 'test', baseURL: 'https://chatapi.akash.network/api/v1', @@ -323,19 +328,10 @@ describe('LobeAkashChatAI - custom features', () => { }, }; - const models = await params.models({ client: mockClient as any }); - - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch AkashChat models. Please ensure your AkashChat API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); + await expect(params.models({ client: mockClient as any })).rejects.toThrow('API Error'); }); it('should handle network timeout errors', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); const mockClient = { apiKey: 'test', baseURL: 'https://chatapi.akash.network/api/v1', @@ -344,16 +340,10 @@ describe('LobeAkashChatAI - custom features', () => { }, }; - const models = await params.models({ client: mockClient as any }); - - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalled(); - - consoleWarnSpy.mockRestore(); + await expect(params.models({ client: mockClient as any })).rejects.toThrow('Network timeout'); }); it('should handle invalid API key errors', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); const mockClient = { apiKey: 'invalid', baseURL: 'https://chatapi.akash.network/api/v1', @@ -362,18 +352,10 @@ describe('LobeAkashChatAI - custom features', () => { }, }; - const models = await params.models({ client: mockClient as any }); - - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch AkashChat models. Please ensure your AkashChat API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); + await expect(params.models({ client: mockClient as any })).rejects.toThrow('Unauthorized'); }); - it('should handle malformed response data', async () => { + it('should throw on malformed response data', async () => { const mockClient = { apiKey: 'test', baseURL: 'https://chatapi.akash.network/api/v1', @@ -384,10 +366,9 @@ describe('LobeAkashChatAI - custom features', () => { }, }; - const models = await params.models({ client: mockClient as any }); - - expect(models).toBeDefined(); - expect(Array.isArray(models)).toBe(true); + await expect(params.models({ client: mockClient as any })).rejects.toThrow( + /Cannot destructure property 'created'/, + ); }); }); diff --git a/packages/model-runtime/src/providers/akashchat/index.ts b/packages/model-runtime/src/providers/akashchat/index.ts index 8b570b0d7d..42c8c1f038 100644 --- a/packages/model-runtime/src/providers/akashchat/index.ts +++ b/packages/model-runtime/src/providers/akashchat/index.ts @@ -36,22 +36,14 @@ export const params = { chatCompletion: () => process.env.DEBUG_AKASH_CHAT_COMPLETION === '1', }, models: async ({ client }) => { - try { - const modelsPage = (await client.models.list()) as any; - const rawList: any[] = modelsPage.data || []; + const modelsPage = (await client.models.list()) as any; + const rawList: any[] = modelsPage.data || []; - // Remove `created` field from each model item + // Remove `created` field from each model item - const modelList: AkashChatModelCard[] = rawList.map(({ created: _, ...rest }) => rest); + const modelList: AkashChatModelCard[] = rawList.map(({ created: _, ...rest }) => rest); - return await processMultiProviderModelList(modelList, 'akashchat'); - } catch (error) { - console.warn( - 'Failed to fetch AkashChat models. Please ensure your AkashChat API key is valid:', - error, - ); - return []; - } + return await processMultiProviderModelList(modelList, 'akashchat'); }, provider: ModelProvider.AkashChat, } satisfies OpenAICompatibleFactoryOptions; diff --git a/packages/model-runtime/src/providers/cerebras/index.test.ts b/packages/model-runtime/src/providers/cerebras/index.test.ts index b8880bfdaf..e40eb0f008 100644 --- a/packages/model-runtime/src/providers/cerebras/index.test.ts +++ b/packages/model-runtime/src/providers/cerebras/index.test.ts @@ -5,6 +5,12 @@ import { beforeEach, describe, expect, it, vi } from 'vitest'; import { testProvider } from '../../providerTestUtils'; import { LobeCerebrasAI, params } from './index'; +const loadModelsMock = vi.hoisted(() => vi.fn().mockResolvedValue([])); + +vi.mock('@lobechat/business-model-bank/model-config', () => ({ + loadModels: loadModelsMock, +})); + testProvider({ Runtime: LobeCerebrasAI, bizErrorType: 'ProviderBizError', @@ -330,8 +336,7 @@ describe('LobeCerebrasAI - custom features', () => { expect(models).toHaveLength(0); }); - it('should handle network error and return empty array', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + it('should throw when network error occurs', async () => { const mockClient = { apiKey: 'test_api_key', baseURL: 'https://api.cerebras.ai/v1', @@ -340,22 +345,12 @@ describe('LobeCerebrasAI - custom features', () => { }, } as any; - const models = await params.models!({ client: mockClient }); + await expect(params.models!({ client: mockClient })).rejects.toThrow('Network error'); expect(mockClient.models.list).toHaveBeenCalledTimes(1); - expect(models).toBeDefined(); - expect(Array.isArray(models)).toBe(true); - expect(models).toHaveLength(0); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch Cerebras models. Please ensure your Cerebras API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); }); - it('should handle API authentication error and return empty array', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + it('should throw when API authentication fails', async () => { const mockClient = { apiKey: 'invalid_key', baseURL: 'https://api.cerebras.ai/v1', @@ -364,20 +359,12 @@ describe('LobeCerebrasAI - custom features', () => { }, } as any; - const models = await params.models!({ client: mockClient }); + await expect(params.models!({ client: mockClient })).rejects.toThrow('401 Unauthorized'); expect(mockClient.models.list).toHaveBeenCalledTimes(1); - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch Cerebras models. Please ensure your Cerebras API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); }); - it('should handle API rate limit error and return empty array', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + it('should throw when API rate limit fails', async () => { const mockClient = { apiKey: 'test_api_key', baseURL: 'https://api.cerebras.ai/v1', @@ -386,20 +373,12 @@ describe('LobeCerebrasAI - custom features', () => { }, } as any; - const models = await params.models!({ client: mockClient }); + await expect(params.models!({ client: mockClient })).rejects.toThrow('429 Too Many Requests'); expect(mockClient.models.list).toHaveBeenCalledTimes(1); - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch Cerebras models. Please ensure your Cerebras API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); }); - it('should handle timeout error and return empty array', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + it('should throw when request times out', async () => { const mockClient = { apiKey: 'test_api_key', baseURL: 'https://api.cerebras.ai/v1', @@ -408,20 +387,12 @@ describe('LobeCerebrasAI - custom features', () => { }, } as any; - const models = await params.models!({ client: mockClient }); + await expect(params.models!({ client: mockClient })).rejects.toThrow('Request timeout'); expect(mockClient.models.list).toHaveBeenCalledTimes(1); - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch Cerebras models. Please ensure your Cerebras API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); }); it('should handle malformed JSON response', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); const mockClient = { apiKey: 'test_api_key', baseURL: 'https://api.cerebras.ai/v1', @@ -430,15 +401,9 @@ describe('LobeCerebrasAI - custom features', () => { }, } as any; - const models = await params.models!({ client: mockClient }); + await expect(params.models!({ client: mockClient })).rejects.toThrow('Invalid JSON'); - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch Cerebras models. Please ensure your Cerebras API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); + expect(mockClient.models.list).toHaveBeenCalledTimes(1); }); it('should pass correct client to processMultiProviderModelList', async () => { diff --git a/packages/model-runtime/src/providers/cerebras/index.ts b/packages/model-runtime/src/providers/cerebras/index.ts index c4de3f7fa8..dfb7d41134 100644 --- a/packages/model-runtime/src/providers/cerebras/index.ts +++ b/packages/model-runtime/src/providers/cerebras/index.ts @@ -21,22 +21,14 @@ export const params = { chatCompletion: () => process.env.DEBUG_CEREBRAS_CHAT_COMPLETION === '1', }, models: async ({ client }) => { - try { - const modelsPage = (await client.models.list()) as any; - const modelList = Array.isArray(modelsPage?.data) - ? modelsPage.data - : Array.isArray(modelsPage) - ? modelsPage - : []; + const modelsPage = (await client.models.list()) as any; + const modelList = Array.isArray(modelsPage?.data) + ? modelsPage.data + : Array.isArray(modelsPage) + ? modelsPage + : []; - return await processMultiProviderModelList(modelList, 'cerebras'); - } catch (error) { - console.warn( - 'Failed to fetch Cerebras models. Please ensure your Cerebras API key is valid:', - error, - ); - return []; - } + return await processMultiProviderModelList(modelList, 'cerebras'); }, provider: ModelProvider.Cerebras, } satisfies OpenAICompatibleFactoryOptions; diff --git a/packages/model-runtime/src/providers/cloudflare/index.test.ts b/packages/model-runtime/src/providers/cloudflare/index.test.ts index c4c3298d8e..babbe4ccc0 100644 --- a/packages/model-runtime/src/providers/cloudflare/index.test.ts +++ b/packages/model-runtime/src/providers/cloudflare/index.test.ts @@ -181,7 +181,7 @@ describe('LobeCloudflareAI', () => { expect(result).toBeInstanceOf(Response); }); - it('should call Cloudflare API with supported opions', async () => { + it('should call Cloudflare API with supported options', async () => { // Arrange const mockResponse = new Response( new ReadableStream({ @@ -558,5 +558,32 @@ describe('LobeCloudflareAI', () => { expect(result).toHaveLength(2); }); + + it('should throw regular Error when API returns null result', async () => { + const instance = new LobeCloudflareAI({ + apiKey: 'test_api_key', + baseURLOrAccountID: accountID, + }); + + vi.spyOn(globalThis, 'fetch').mockResolvedValue( + new Response( + JSON.stringify({ + errors: [{ code: 10000, message: 'Authentication error' }], + result: null, + success: false, + }), + { status: 401 }, + ), + ); + + await expect(instance.models()).rejects.toMatchObject({ + cause: { + errors: [{ code: 10000, message: 'Authentication error' }], + result: null, + success: false, + }, + message: 'Cloudflare models API returned an invalid response', + }); + }); }); }); diff --git a/packages/model-runtime/src/providers/cloudflare/index.ts b/packages/model-runtime/src/providers/cloudflare/index.ts index ad38098555..6fad25c804 100644 --- a/packages/model-runtime/src/providers/cloudflare/index.ts +++ b/packages/model-runtime/src/providers/cloudflare/index.ts @@ -162,7 +162,11 @@ export class LobeCloudflareAI implements LobeRuntimeAI { }); const json = await response.json(); - const modelList: CloudflareModelCard[] = json.result; + const modelList: CloudflareModelCard[] | undefined = json.result; + + if (!Array.isArray(modelList)) { + throw new Error('Cloudflare models API returned an invalid response', { cause: json }); + } return modelList .map((model) => { diff --git a/packages/model-runtime/src/providers/cometapi/index.test.ts b/packages/model-runtime/src/providers/cometapi/index.test.ts index d5758f100f..804035cc3f 100644 --- a/packages/model-runtime/src/providers/cometapi/index.test.ts +++ b/packages/model-runtime/src/providers/cometapi/index.test.ts @@ -5,6 +5,12 @@ import { beforeEach, describe, expect, it, vi } from 'vitest'; import { testProvider } from '../../providerTestUtils'; import { LobeCometAPIAI, params } from './index'; +const loadModelsMock = vi.hoisted(() => vi.fn().mockResolvedValue([])); + +vi.mock('@lobechat/business-model-bank/model-config', () => ({ + loadModels: loadModelsMock, +})); + // Basic provider tests testProvider({ Runtime: LobeCometAPIAI, @@ -295,16 +301,7 @@ describe('LobeCometAPIAI - custom features', () => { }, }; - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); - - const models = await params.models!({ client: mockClient as any }); - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch CometAPI models. Please ensure your CometAPI API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); + await expect(params.models!({ client: mockClient as any })).rejects.toThrow('API Error'); }); it('should handle network error', async () => { @@ -316,12 +313,7 @@ describe('LobeCometAPIAI - custom features', () => { }, }; - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); - - const models = await params.models!({ client: mockClient as any }); - expect(models).toEqual([]); - - consoleWarnSpy.mockRestore(); + await expect(params.models!({ client: mockClient as any })).rejects.toThrow('Network Error'); }); it('should handle unauthorized error', async () => { @@ -333,12 +325,7 @@ describe('LobeCometAPIAI - custom features', () => { }, }; - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); - - const models = await params.models!({ client: mockClient as any }); - expect(models).toEqual([]); - - consoleWarnSpy.mockRestore(); + await expect(params.models!({ client: mockClient as any })).rejects.toThrow('Unauthorized'); }); it('should process multi-provider model list', async () => { diff --git a/packages/model-runtime/src/providers/cometapi/index.ts b/packages/model-runtime/src/providers/cometapi/index.ts index 0aa4b044d4..6a9e84af64 100644 --- a/packages/model-runtime/src/providers/cometapi/index.ts +++ b/packages/model-runtime/src/providers/cometapi/index.ts @@ -27,25 +27,17 @@ export const params = { chatCompletion: () => process.env.DEBUG_COMETAPI_COMPLETION === '1', }, models: async ({ client }) => { - try { - const modelsPage = (await client.models.list()) as any; - const rawList: any[] = modelsPage.data || []; + const modelsPage = (await client.models.list()) as any; + const rawList: any[] = modelsPage.data || []; - // Process the model list and remove unnecessary fields - const modelList: CometAPIModelCard[] = rawList.map((model) => ({ - id: model.id, - object: model.object, - owned_by: model.owned_by, - })); + // Process the model list and remove unnecessary fields + const modelList: CometAPIModelCard[] = rawList.map((model) => ({ + id: model.id, + object: model.object, + owned_by: model.owned_by, + })); - return await processMultiProviderModelList(modelList, 'cometapi'); - } catch (error) { - console.warn( - 'Failed to fetch CometAPI models. Please ensure your CometAPI API key is valid:', - error, - ); - return []; - } + return await processMultiProviderModelList(modelList, 'cometapi'); }, provider: ModelProvider.CometAPI, } satisfies OpenAICompatibleFactoryOptions; diff --git a/packages/model-runtime/src/providers/giteeai/index.test.ts b/packages/model-runtime/src/providers/giteeai/index.test.ts index a7672f0e15..99f43e28bc 100644 --- a/packages/model-runtime/src/providers/giteeai/index.test.ts +++ b/packages/model-runtime/src/providers/giteeai/index.test.ts @@ -5,6 +5,12 @@ import { beforeEach, describe, expect, it, vi } from 'vitest'; import { testProvider } from '../../providerTestUtils'; import { LobeGiteeAI, params } from './index'; +const loadModelsMock = vi.hoisted(() => vi.fn().mockResolvedValue([])); + +vi.mock('@lobechat/business-model-bank/model-config', () => ({ + loadModels: loadModelsMock, +})); + testProvider({ Runtime: LobeGiteeAI, chatDebugEnv: 'DEBUG_GITEE_AI_CHAT_COMPLETION', @@ -200,8 +206,7 @@ describe('LobeGiteeAI - custom features', () => { expect(models).toHaveLength(0); }); - it('should handle network error and return empty array', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + it('should throw when network error occurs', async () => { const mockClient = { apiKey: 'test_api_key', baseURL: 'https://ai.gitee.com/v1', @@ -210,22 +215,12 @@ describe('LobeGiteeAI - custom features', () => { }, } as any; - const models = await params.models!({ client: mockClient }); + await expect(params.models!({ client: mockClient })).rejects.toThrow('Network error'); expect(mockClient.models.list).toHaveBeenCalledTimes(1); - expect(models).toBeDefined(); - expect(Array.isArray(models)).toBe(true); - expect(models).toHaveLength(0); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch GiteeAI models. Please ensure your GiteeAI API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); }); - it('should handle API authentication error and return empty array', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + it('should throw when API authentication fails', async () => { const mockClient = { apiKey: 'invalid_key', baseURL: 'https://ai.gitee.com/v1', @@ -234,20 +229,12 @@ describe('LobeGiteeAI - custom features', () => { }, } as any; - const models = await params.models!({ client: mockClient }); + await expect(params.models!({ client: mockClient })).rejects.toThrow('401 Unauthorized'); expect(mockClient.models.list).toHaveBeenCalledTimes(1); - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch GiteeAI models. Please ensure your GiteeAI API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); }); - it('should handle API rate limit error and return empty array', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + it('should throw when API rate limit fails', async () => { const mockClient = { apiKey: 'test_api_key', baseURL: 'https://ai.gitee.com/v1', @@ -256,20 +243,12 @@ describe('LobeGiteeAI - custom features', () => { }, } as any; - const models = await params.models!({ client: mockClient }); + await expect(params.models!({ client: mockClient })).rejects.toThrow('429 Too Many Requests'); expect(mockClient.models.list).toHaveBeenCalledTimes(1); - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch GiteeAI models. Please ensure your GiteeAI API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); }); - it('should handle timeout error and return empty array', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + it('should throw when request times out', async () => { const mockClient = { apiKey: 'test_api_key', baseURL: 'https://ai.gitee.com/v1', @@ -278,20 +257,12 @@ describe('LobeGiteeAI - custom features', () => { }, } as any; - const models = await params.models!({ client: mockClient }); + await expect(params.models!({ client: mockClient })).rejects.toThrow('Request timeout'); expect(mockClient.models.list).toHaveBeenCalledTimes(1); - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch GiteeAI models. Please ensure your GiteeAI API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); }); it('should handle malformed JSON response', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); const mockClient = { apiKey: 'test_api_key', baseURL: 'https://ai.gitee.com/v1', @@ -300,15 +271,9 @@ describe('LobeGiteeAI - custom features', () => { }, } as any; - const models = await params.models!({ client: mockClient }); + await expect(params.models!({ client: mockClient })).rejects.toThrow('Invalid JSON'); - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch GiteeAI models. Please ensure your GiteeAI API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); + expect(mockClient.models.list).toHaveBeenCalledTimes(1); }); it('should pass correct client to processMultiProviderModelList', async () => { diff --git a/packages/model-runtime/src/providers/giteeai/index.ts b/packages/model-runtime/src/providers/giteeai/index.ts index 0ab6b07d63..9592113230 100644 --- a/packages/model-runtime/src/providers/giteeai/index.ts +++ b/packages/model-runtime/src/providers/giteeai/index.ts @@ -14,22 +14,14 @@ export const params = { chatCompletion: () => process.env.DEBUG_GITEE_AI_CHAT_COMPLETION === '1', }, models: async ({ client }) => { - try { - const modelsPage = (await client.models.list()) as any; - const modelList: GiteeAIModelCard[] = Array.isArray(modelsPage?.data) - ? modelsPage.data - : Array.isArray(modelsPage) - ? modelsPage - : []; + const modelsPage = (await client.models.list()) as any; + const modelList: GiteeAIModelCard[] = Array.isArray(modelsPage?.data) + ? modelsPage.data + : Array.isArray(modelsPage) + ? modelsPage + : []; - return await processMultiProviderModelList(modelList, 'giteeai'); - } catch (error) { - console.warn( - 'Failed to fetch GiteeAI models. Please ensure your GiteeAI API key is valid:', - error, - ); - return []; - } + return await processMultiProviderModelList(modelList, 'giteeai'); }, provider: ModelProvider.GiteeAI, } satisfies OpenAICompatibleFactoryOptions; diff --git a/packages/model-runtime/src/providers/githubCopilot/index.test.ts b/packages/model-runtime/src/providers/githubCopilot/index.test.ts index b47890a977..1ce4091ce9 100644 --- a/packages/model-runtime/src/providers/githubCopilot/index.test.ts +++ b/packages/model-runtime/src/providers/githubCopilot/index.test.ts @@ -216,19 +216,61 @@ describe('LobeGithubCopilotAI', () => { expect(models).toEqual([]); }); + + it('should throw regular Error when models request fails', async () => { + mockFetch.mockResolvedValueOnce({ + json: () => Promise.resolve({ error: { message: 'Copilot access denied' } }), + ok: false, + status: 403, + }); + + const instance = new LobeGithubCopilotAI({ + bearerToken: 'cached-bearer-token', + bearerTokenExpiresAt: Date.now() + 60 * 60 * 1000, + }); + + try { + await instance.models(); + expect.fail('Expected models() to reject'); + } catch (error) { + expect(error).toBeInstanceOf(Error); + expect((error as Error).message).toBe('GitHub Copilot models API request failed'); + expect((error as Error).cause).toEqual({ status: 403 }); + expect((error as Error & { errorType?: string }).errorType).toBeUndefined(); + } + }); + + it('should throw runtime payload when token exchange fails before models request', async () => { + mockFetch.mockResolvedValueOnce({ + json: () => Promise.resolve({}), + ok: false, + status: 403, + }); + + const instance = new LobeGithubCopilotAI({ apiKey: 'ghp_models_denied' }); + + try { + await instance.models(); + expect.fail('Expected models() to reject'); + } catch (error) { + expect(error).toEqual({ + error: { message: 'No GitHub Copilot subscription or access denied' }, + errorType: 'PermissionDenied', + }); + } + }); }); describe('error handling in constructor', () => { - it('should throw InvalidGithubCopilotToken when no credentials provided', () => { - expect(() => new LobeGithubCopilotAI({})).toThrow(); - }); - - it('should throw with descriptive message', () => { + it('should throw runtime payload when no credentials provided', () => { try { new LobeGithubCopilotAI({}); expect.fail('Should have thrown'); } catch (error: any) { - expect(error.errorType).toBe('InvalidGithubCopilotToken'); + expect(error).toEqual({ + error: { message: 'GitHub Personal Access Token or OAuth token is required' }, + errorType: 'InvalidGithubCopilotToken', + }); } }); }); diff --git a/packages/model-runtime/src/providers/githubCopilot/index.ts b/packages/model-runtime/src/providers/githubCopilot/index.ts index 65b9b0c29f..6d20992246 100644 --- a/packages/model-runtime/src/providers/githubCopilot/index.ts +++ b/packages/model-runtime/src/providers/githubCopilot/index.ts @@ -398,34 +398,45 @@ export class LobeGithubCopilotAI implements LobeRuntimeAI { } async models(): Promise { - return this.executeWithRetry(async () => { - const bearerToken = this.cachedBearerToken || (await tokenManager.getToken(this.githubToken)); + return this.executeWithRetry( + async () => { + const bearerToken = + this.cachedBearerToken || (await tokenManager.getToken(this.githubToken)); - const response = await fetch(`${COPILOT_BASE_URL}/models`, { - headers: { - 'Accept': 'application/json', - 'Authorization': `Bearer ${bearerToken}`, - 'Copilot-Integration-Id': 'vscode-chat', - 'Editor-Plugin-Version': 'LobeChat/1.0', - 'Editor-Version': 'LobeChat/1.0', - }, - method: 'GET', - }); + const response = await fetch(`${COPILOT_BASE_URL}/models`, { + headers: { + 'Accept': 'application/json', + 'Authorization': `Bearer ${bearerToken}`, + 'Copilot-Integration-Id': 'vscode-chat', + 'Editor-Plugin-Version': 'LobeChat/1.0', + 'Editor-Version': 'LobeChat/1.0', + }, + method: 'GET', + }); - if (!response.ok) { - throw { status: response.status }; - } + if (!response.ok) { + throw Object.assign( + new Error('GitHub Copilot models API request failed', { + cause: { status: response.status }, + }), + { + status: response.status, + }, + ); + } - const data = await response.json(); + const data = await response.json(); - // Transform Copilot models to ChatModelCard format - return (data.models || data.data || []).map((model: any) => ({ - displayName: model.name || model.id, - enabled: true, - id: model.id || model.name, - type: 'chat', - })); - }); + // Transform Copilot models to ChatModelCard format + return (data.models || data.data || []).map((model: any) => ({ + displayName: model.name || model.id, + enabled: true, + id: model.id || model.name, + type: 'chat', + })); + }, + { mapError: false }, + ); } private handlePayload(payload: ChatStreamPayload) { @@ -443,7 +454,10 @@ export class LobeGithubCopilotAI implements LobeRuntimeAI { return { type: tool.type, ...tool.function } as any; }; - private async executeWithRetry(fn: () => Promise): Promise { + private async executeWithRetry( + fn: () => Promise, + options: { mapError?: boolean } = {}, + ): Promise { let totalAttempts = 0; let hasRefreshedAuth = false; let rateLimitAttempts = 0; @@ -471,6 +485,7 @@ export class LobeGithubCopilotAI implements LobeRuntimeAI { // If retry-after exceeds the quota exhaustion threshold, surface immediately if (retryAfter > QUOTA_EXHAUSTION_THRESHOLD_MS) { + if (options.mapError === false) throw error; throw this.mapError(error); } @@ -481,10 +496,17 @@ export class LobeGithubCopilotAI implements LobeRuntimeAI { } // Map and throw + if (options.mapError === false) throw error; throw this.mapError(error); } } + if (options.mapError === false) { + throw new Error('Max retry attempts exceeded', { + cause: { endpoint: this.baseURL }, + }); + } + throw AgentRuntimeError.chat({ endpoint: this.baseURL, error: { message: 'Max retry attempts exceeded' }, diff --git a/packages/model-runtime/src/providers/huggingface/index.test.ts b/packages/model-runtime/src/providers/huggingface/index.test.ts index 92e2b86317..e859b63c49 100644 --- a/packages/model-runtime/src/providers/huggingface/index.test.ts +++ b/packages/model-runtime/src/providers/huggingface/index.test.ts @@ -5,6 +5,12 @@ import { beforeEach, describe, expect, it, vi } from 'vitest'; import { AgentRuntimeErrorType } from '../../types/error'; import { LobeHuggingFaceAI, params } from './index'; +const loadModelsMock = vi.hoisted(() => vi.fn().mockResolvedValue([])); + +vi.mock('@lobechat/business-model-bank/model-config', () => ({ + loadModels: loadModelsMock, +})); + describe('LobeHuggingFaceAI', () => { let instance: any; @@ -355,6 +361,7 @@ describe('LobeHuggingFaceAI', () => { describe('Models Fetching', () => { beforeEach(() => { vi.restoreAllMocks(); + loadModelsMock.mockResolvedValue([]); }); it('should fetch and process models from HuggingFace API', async () => { @@ -676,8 +683,7 @@ describe('LobeHuggingFaceAI', () => { it('should handle API errors gracefully', async () => { global.fetch = vi.fn().mockRejectedValue(new Error('API Error')); - const result = await params.models!(); - expect(result).toEqual([]); + await expect(params.models!()).rejects.toThrow('API Error'); }); it('should handle invalid JSON response', async () => { @@ -688,8 +694,7 @@ describe('LobeHuggingFaceAI', () => { }, } as unknown as Response); - const result = await params.models!(); - expect(result).toEqual([]); + await expect(params.models!()).rejects.toThrow('Invalid JSON'); }); it('should preserve contextWindowTokens from provider info', async () => { diff --git a/packages/model-runtime/src/providers/huggingface/index.ts b/packages/model-runtime/src/providers/huggingface/index.ts index 2beff6c498..00d69f6ebd 100644 --- a/packages/model-runtime/src/providers/huggingface/index.ts +++ b/packages/model-runtime/src/providers/huggingface/index.ts @@ -8,7 +8,7 @@ import { createOpenAICompatibleRuntime } from '../../core/openaiCompatibleFactor import { convertIterableToStream } from '../../core/streams'; import { AgentRuntimeErrorType } from '../../types/error'; import { processMultiProviderModelList } from '../../utils/modelParse'; -import type { HuggingFaceRouterModelCard, HuggingFaceRouterResponse } from './type'; +import type { HuggingFaceRouterResponse } from './type'; export const params = { chatCompletion: { @@ -54,19 +54,14 @@ export const params = { chatCompletion: () => process.env.DEBUG_HUGGINGFACE_CHAT_COMPLETION === '1', }, models: async () => { - let modelList: HuggingFaceRouterModelCard[] = []; - - try { - const response = await fetch('https://router.huggingface.co/v1/models'); - if (response.ok) { - const data: HuggingFaceRouterResponse = await response.json(); - modelList = data.data; - } - } catch (error) { - console.error('Failed to fetch HuggingFace router models:', error); - return []; + const response = await fetch('https://router.huggingface.co/v1/models'); + if (!response.ok) { + throw new Error(`HuggingFace models API request failed with status ${response.status}`); } + const data: HuggingFaceRouterResponse = await response.json(); + const modelList = data.data; + const formattedModels = modelList .map((model) => { const { architecture, providers } = model; diff --git a/packages/model-runtime/src/providers/modelscope/index.test.ts b/packages/model-runtime/src/providers/modelscope/index.test.ts index dae3f92c8a..75b9c53714 100644 --- a/packages/model-runtime/src/providers/modelscope/index.test.ts +++ b/packages/model-runtime/src/providers/modelscope/index.test.ts @@ -6,6 +6,12 @@ import { testProvider } from '../../providerTestUtils'; import type { ModelScopeModelCard } from './index'; import { LobeModelScopeAI, params } from './index'; +const loadModelsMock = vi.hoisted(() => vi.fn().mockResolvedValue([])); + +vi.mock('@lobechat/business-model-bank/model-config', () => ({ + loadModels: loadModelsMock, +})); + const provider = ModelProvider.ModelScope; const defaultBaseURL = 'https://api-inference.modelscope.cn/v1'; @@ -55,7 +61,6 @@ describe('LobeModelScopeAI - custom features', () => { describe('models function', () => { let mockClient: any; - let consoleWarnSpy: any; beforeEach(() => { mockClient = { @@ -63,7 +68,6 @@ describe('LobeModelScopeAI - custom features', () => { list: vi.fn(), }, }; - consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); }); it('should fetch and process models successfully', async () => { @@ -91,7 +95,6 @@ describe('LobeModelScopeAI - custom features', () => { expect(mockClient.models.list).toHaveBeenCalledTimes(1); expect(models).toBeDefined(); expect(Array.isArray(models)).toBe(true); - expect(consoleWarnSpy).not.toHaveBeenCalled(); }); it('should handle empty model list', async () => { @@ -105,7 +108,6 @@ describe('LobeModelScopeAI - custom features', () => { expect(models).toBeDefined(); expect(Array.isArray(models)).toBe(true); expect(models).toHaveLength(0); - expect(consoleWarnSpy).not.toHaveBeenCalled(); }); it('should handle missing data field', async () => { @@ -117,47 +119,28 @@ describe('LobeModelScopeAI - custom features', () => { expect(models).toBeDefined(); expect(Array.isArray(models)).toBe(true); expect(models).toHaveLength(0); - expect(consoleWarnSpy).not.toHaveBeenCalled(); }); it('should handle API errors gracefully', async () => { const mockError = new Error('API Error'); mockClient.models.list.mockRejectedValue(mockError); - const models = await params.models({ client: mockClient }); - + await expect(params.models({ client: mockClient })).rejects.toThrow('API Error'); expect(mockClient.models.list).toHaveBeenCalledTimes(1); - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch ModelScope models. Please ensure your ModelScope API key is valid and your Alibaba Cloud account is properly bound:', - mockError, - ); }); it('should handle network errors', async () => { const networkError = new Error('Network timeout'); mockClient.models.list.mockRejectedValue(networkError); - const models = await params.models({ client: mockClient }); - - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch ModelScope models. Please ensure your ModelScope API key is valid and your Alibaba Cloud account is properly bound:', - networkError, - ); + await expect(params.models({ client: mockClient })).rejects.toThrow('Network timeout'); }); it('should handle invalid API key errors', async () => { const authError = new Error('Invalid API key'); mockClient.models.list.mockRejectedValue(authError); - const models = await params.models({ client: mockClient }); - - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch ModelScope models. Please ensure your ModelScope API key is valid and your Alibaba Cloud account is properly bound:', - authError, - ); + await expect(params.models({ client: mockClient })).rejects.toThrow('Invalid API key'); }); it('should process models with processMultiProviderModelList', async () => { diff --git a/packages/model-runtime/src/providers/modelscope/index.ts b/packages/model-runtime/src/providers/modelscope/index.ts index 06dfada862..281258fe46 100644 --- a/packages/model-runtime/src/providers/modelscope/index.ts +++ b/packages/model-runtime/src/providers/modelscope/index.ts @@ -17,18 +17,10 @@ export const params = { chatCompletion: () => process.env.DEBUG_MODELSCOPE_CHAT_COMPLETION === '1', }, models: async ({ client }) => { - try { - const modelsPage = (await client.models.list()) as any; - const modelList: ModelScopeModelCard[] = modelsPage.data || []; + const modelsPage = (await client.models.list()) as any; + const modelList: ModelScopeModelCard[] = modelsPage.data || []; - return await processMultiProviderModelList(modelList, 'modelscope'); - } catch (error) { - console.warn( - 'Failed to fetch ModelScope models. Please ensure your ModelScope API key is valid and your Alibaba Cloud account is properly bound:', - error, - ); - return []; - } + return await processMultiProviderModelList(modelList, 'modelscope'); }, provider: ModelProvider.ModelScope, } satisfies OpenAICompatibleFactoryOptions; diff --git a/packages/model-runtime/src/providers/moonshot/index.test.ts b/packages/model-runtime/src/providers/moonshot/index.test.ts index 24afc7ab8e..e540a545f6 100644 --- a/packages/model-runtime/src/providers/moonshot/index.test.ts +++ b/packages/model-runtime/src/providers/moonshot/index.test.ts @@ -873,15 +873,13 @@ describe('models', () => { expect(models).toEqual([]); }); - it('should handle fetch error gracefully', async () => { + it('should throw when model fetch fails', async () => { const mockClient = { models: { list: vi.fn().mockRejectedValue(new Error('Network error')), }, } as unknown as OpenAI; - const models = await fetchModels({ client: mockClient }); - - expect(models).toEqual([]); + await expect(fetchModels({ client: mockClient })).rejects.toThrow('Network error'); }); }); diff --git a/packages/model-runtime/src/providers/moonshot/index.ts b/packages/model-runtime/src/providers/moonshot/index.ts index cda2dfd310..1ade711c8c 100644 --- a/packages/model-runtime/src/providers/moonshot/index.ts +++ b/packages/model-runtime/src/providers/moonshot/index.ts @@ -213,21 +213,16 @@ const buildMoonshotOpenAIPayload = ( * Fetch Moonshot models from the API using OpenAI client */ const fetchMoonshotModels = async ({ client }: { client: OpenAI }): Promise => { - try { - const modelsPage = (await client.models.list()) as any; - const modelList: MoonshotModelCard[] = modelsPage.data || []; + const modelsPage = (await client.models.list()) as any; + const modelList: MoonshotModelCard[] = modelsPage.data || []; - const processedList = modelList.map((model) => ({ - contextWindowTokens: model.context_length, - id: model.id, - vision: model.supports_image_in, - })); + const processedList = modelList.map((model) => ({ + contextWindowTokens: model.context_length, + id: model.id, + vision: model.supports_image_in, + })); - return processModelList(processedList, MODEL_LIST_CONFIGS.moonshot, 'moonshot'); - } catch (error) { - console.warn('Failed to fetch Moonshot models:', error); - return []; - } + return processModelList(processedList, MODEL_LIST_CONFIGS.moonshot, 'moonshot'); }; /** diff --git a/packages/model-runtime/src/providers/ollamacloud/index.test.ts b/packages/model-runtime/src/providers/ollamacloud/index.test.ts index 9287588e06..a3cbabdf11 100644 --- a/packages/model-runtime/src/providers/ollamacloud/index.test.ts +++ b/packages/model-runtime/src/providers/ollamacloud/index.test.ts @@ -5,6 +5,12 @@ import { beforeEach, describe, expect, it, vi } from 'vitest'; import { testProvider } from '../../providerTestUtils'; import { LobeOllamaCloudAI, params } from './index'; +const loadModelsMock = vi.hoisted(() => vi.fn().mockResolvedValue([])); + +vi.mock('@lobechat/business-model-bank/model-config', () => ({ + loadModels: loadModelsMock, +})); + // Basic provider tests testProvider({ Runtime: LobeOllamaCloudAI, @@ -223,18 +229,9 @@ describe('LobeOllamaCloudAI - custom features', () => { }, }; - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); - - const result = await params.models({ client: mockClient as any }); + await expect(params.models({ client: mockClient as any })).rejects.toThrow('API Error'); expect(mockClient.models.list).toHaveBeenCalledTimes(1); - expect(result).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch Ollama Cloud models. Please ensure your Ollama Cloud API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); }); it('should handle network error', async () => { @@ -246,14 +243,7 @@ describe('LobeOllamaCloudAI - custom features', () => { }, }; - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); - - const result = await params.models({ client: mockClient as any }); - - expect(result).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalled(); - - consoleWarnSpy.mockRestore(); + await expect(params.models({ client: mockClient as any })).rejects.toThrow('Network Error'); }); it('should handle invalid API key error', async () => { @@ -265,17 +255,7 @@ describe('LobeOllamaCloudAI - custom features', () => { }, }; - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); - - const result = await params.models({ client: mockClient as any }); - - expect(result).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch Ollama Cloud models. Please ensure your Ollama Cloud API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); + await expect(params.models({ client: mockClient as any })).rejects.toThrow('Invalid API Key'); }); it('should handle null response', async () => { @@ -373,14 +353,7 @@ describe('LobeOllamaCloudAI - custom features', () => { }, }; - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); - - const result = await params.models({ client: mockClient as any }); - - expect(result).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalled(); - - consoleWarnSpy.mockRestore(); + await expect(params.models({ client: mockClient as any })).rejects.toThrow('Request timeout'); }); }); diff --git a/packages/model-runtime/src/providers/ollamacloud/index.ts b/packages/model-runtime/src/providers/ollamacloud/index.ts index 70d3fae5f2..9ca6305362 100644 --- a/packages/model-runtime/src/providers/ollamacloud/index.ts +++ b/packages/model-runtime/src/providers/ollamacloud/index.ts @@ -20,22 +20,14 @@ export const params = { chatCompletion: () => process.env.DEBUG_OLLAMA_CLOUD_CHAT_COMPLETION === '1', }, models: async ({ client }) => { - try { - const modelsPage = (await client.models.list()) as any; - const modelList = Array.isArray(modelsPage?.data) - ? modelsPage.data - : Array.isArray(modelsPage) - ? modelsPage - : []; + const modelsPage = (await client.models.list()) as any; + const modelList = Array.isArray(modelsPage?.data) + ? modelsPage.data + : Array.isArray(modelsPage) + ? modelsPage + : []; - return await processMultiProviderModelList(modelList, 'ollamacloud'); - } catch (error) { - console.warn( - 'Failed to fetch Ollama Cloud models. Please ensure your Ollama Cloud API key is valid:', - error, - ); - return []; - } + return await processMultiProviderModelList(modelList, 'ollamacloud'); }, provider: ModelProvider.OllamaCloud, } satisfies OpenAICompatibleFactoryOptions; diff --git a/packages/model-runtime/src/providers/openrouter/index.test.ts b/packages/model-runtime/src/providers/openrouter/index.test.ts index 0974a7be75..4693a868fc 100644 --- a/packages/model-runtime/src/providers/openrouter/index.test.ts +++ b/packages/model-runtime/src/providers/openrouter/index.test.ts @@ -5,6 +5,12 @@ import type { LobeOpenAICompatibleRuntime } from '../../core/BaseAI'; import { testProvider } from '../../providerTestUtils'; import { LobeOpenRouterAI, params } from './index'; +const loadModelsMock = vi.hoisted(() => vi.fn().mockResolvedValue([])); + +vi.mock('@lobechat/business-model-bank/model-config', () => ({ + loadModels: loadModelsMock, +})); + const provider = 'openrouter'; const defaultBaseURL = 'https://openrouter.ai/api/v1'; @@ -1347,29 +1353,24 @@ describe('LobeOpenRouterAI - custom features', () => { expect(models).toEqual([]); }); - it('should return empty array when fetch fails', async () => { + it('should throw when fetch fails', async () => { vi.stubGlobal( 'fetch', vi.fn().mockResolvedValue({ ok: false, + status: 401, }), ); - const models = await params.models(); - - expect(models).toEqual([]); + await expect(params.models()).rejects.toThrow( + 'OpenRouter models API request failed with status 401', + ); }); - it('should return empty array when fetch throws error', async () => { + it('should throw when fetch throws error', async () => { vi.stubGlobal('fetch', vi.fn().mockRejectedValue(new Error('Network error'))); - const models = await params.models(); - - expect(models).toEqual([]); - expect(console.error).toHaveBeenCalledWith( - 'Failed to fetch OpenRouter frontend models:', - expect.any(Error), - ); + await expect(params.models()).rejects.toThrow('Network error'); }); it('should handle models with missing optional fields', async () => { diff --git a/packages/model-runtime/src/providers/openrouter/index.ts b/packages/model-runtime/src/providers/openrouter/index.ts index ebac9b1d54..000ce6161d 100644 --- a/packages/model-runtime/src/providers/openrouter/index.ts +++ b/packages/model-runtime/src/providers/openrouter/index.ts @@ -93,19 +93,14 @@ export const params = { chatCompletion: () => process.env.DEBUG_OPENROUTER_CHAT_COMPLETION === '1', }, models: async () => { - let modelList: OpenRouterModelCard[] = []; - - try { - const response = await fetch('https://openrouter.ai/api/v1/models'); - if (response.ok) { - const data = await response.json(); - modelList = data['data']; - } - } catch (error) { - console.error('Failed to fetch OpenRouter frontend models:', error); - return []; + const response = await fetch('https://openrouter.ai/api/v1/models'); + if (!response.ok) { + throw new Error(`OpenRouter models API request failed with status ${response.status}`); } + const data = (await response.json()) as { data: OpenRouterModelCard[] }; + const modelList = data.data; + // Process the model info fetched from the frontend and convert to standard format const formattedModels = modelList.map((model) => { const { top_provider, architecture, pricing, supported_parameters } = model; diff --git a/packages/model-runtime/src/providers/straico/index.test.ts b/packages/model-runtime/src/providers/straico/index.test.ts new file mode 100644 index 0000000000..ed3b5e293a --- /dev/null +++ b/packages/model-runtime/src/providers/straico/index.test.ts @@ -0,0 +1,33 @@ +// @vitest-environment node +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +import { LobeStraicoAI } from './index'; + +const loadModelsMock = vi.hoisted(() => vi.fn().mockResolvedValue([])); + +vi.mock('@lobechat/business-model-bank/model-config', () => ({ + loadModels: loadModelsMock, +})); + +const mockFetch = vi.fn(); + +describe('LobeStraicoAI', () => { + beforeEach(() => { + mockFetch.mockReset(); + vi.stubGlobal('fetch', mockFetch); + }); + + afterEach(() => { + vi.unstubAllGlobals(); + }); + + describe('models', () => { + it('should throw a regular Error when the API request fails', async () => { + mockFetch.mockResolvedValueOnce(new Response('Unauthorized', { status: 401 })); + + const instance = new LobeStraicoAI({ apiKey: 'test-api-key' }); + + await expect(instance.models()).rejects.toThrow('HTTP 401'); + }); + }); +}); diff --git a/packages/model-runtime/src/providers/straico/index.ts b/packages/model-runtime/src/providers/straico/index.ts index d532cbdd7a..953927694c 100644 --- a/packages/model-runtime/src/providers/straico/index.ts +++ b/packages/model-runtime/src/providers/straico/index.ts @@ -27,7 +27,6 @@ export const LobeStraicoAI = createOpenAICompatibleRuntime({ baseURL: 'https://api.straico.com/v0', chatCompletion: { handlePayload: (payload) => { - const { model, ...rest } = payload; return { @@ -41,56 +40,48 @@ export const LobeStraicoAI = createOpenAICompatibleRuntime({ chatCompletion: () => process.env.DEBUG_STRAICO_CHAT_COMPLETION === '1', }, models: async ({ client }) => { - try { - const url = 'https://api.straico.com/v1/models'; - const response = await fetch(url, { - headers: { - Authorization: `Bearer ${client.apiKey}`, - }, - method: 'GET', - }); + const url = 'https://api.straico.com/v1/models'; + const response = await fetch(url, { + headers: { + Authorization: `Bearer ${client.apiKey}`, + }, + method: 'GET', + }); - if (!response.ok) { - throw new Error(`HTTP ${response.status}: ${response.statusText}`); - } - - const json: StraicoModelsResponse = await response.json(); - const chatModels: StraicoChatModel[] = json?.data?.chat || []; // There are also audio and image models to be adapted - - // Transform Straico models to standardized format - const formattedModels = chatModels.map((model) => { - const inputPrice = formatPrice(model.pricing); - const outputPrice = inputPrice; // Straico uses same price for input/output - - return { - contextWindowTokens: model.word_limit - ? Math.floor(model.word_limit * 1.33) // Convert words to tokens - : undefined, - description: model.metadata?.pros?.join('; ') || '', - displayName: cleanModelName(model.name), - enabled: model.enabled ?? false, - functionCall: false, - id: model.model, - maxOutput: model.max_output, - pricing: inputPrice - ? { - input: inputPrice, - output: outputPrice, - } - : undefined, - reasoning: model.metadata?.applications?.includes('Reasoning') ?? false, - vision: model.metadata?.features?.includes('Image input') ?? false, - }; - }); - - return await processMultiProviderModelList(formattedModels, 'straico'); - } catch (error) { - console.warn( - 'Failed to fetch Straico models. Please ensure your Straico API key is valid:', - error, - ); - return []; + if (!response.ok) { + throw new Error(`HTTP ${response.status}: ${response.statusText}`); } + + const json: StraicoModelsResponse = await response.json(); + const chatModels: StraicoChatModel[] = json?.data?.chat || []; // There are also audio and image models to be adapted + + // Transform Straico models to standardized format + const formattedModels = chatModels.map((model) => { + const inputPrice = formatPrice(model.pricing); + const outputPrice = inputPrice; // Straico uses same price for input/output + + return { + contextWindowTokens: model.word_limit + ? Math.floor(model.word_limit * 1.33) // Convert words to tokens + : undefined, + description: model.metadata?.pros?.join('; ') || '', + displayName: cleanModelName(model.name), + enabled: model.enabled ?? false, + functionCall: false, + id: model.model, + maxOutput: model.max_output, + pricing: inputPrice + ? { + input: inputPrice, + output: outputPrice, + } + : undefined, + reasoning: model.metadata?.applications?.includes('Reasoning') ?? false, + vision: model.metadata?.features?.includes('Image input') ?? false, + }; + }); + + return await processMultiProviderModelList(formattedModels, 'straico'); }, provider: ModelProvider.Straico, }); diff --git a/packages/model-runtime/src/providers/v0/index.test.ts b/packages/model-runtime/src/providers/v0/index.test.ts index f577137911..65f09c8250 100644 --- a/packages/model-runtime/src/providers/v0/index.test.ts +++ b/packages/model-runtime/src/providers/v0/index.test.ts @@ -5,6 +5,12 @@ import { beforeEach, describe, expect, it, vi } from 'vitest'; import { testProvider } from '../../providerTestUtils'; import { LobeV0AI, params } from './index'; +const loadModelsMock = vi.hoisted(() => vi.fn().mockResolvedValue([])); + +vi.mock('@lobechat/business-model-bank/model-config', () => ({ + loadModels: loadModelsMock, +})); + testProvider({ Runtime: LobeV0AI, bizErrorType: 'ProviderBizError', @@ -206,8 +212,7 @@ describe('LobeV0AI - custom features', () => { expect(models).toHaveLength(0); }); - it('should handle network error and return empty array', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + it('should throw when network error occurs', async () => { const mockClient = { apiKey: 'test_api_key', baseURL: 'https://api.v0.dev/v1', @@ -216,22 +221,12 @@ describe('LobeV0AI - custom features', () => { }, } as any; - const models = await params.models!({ client: mockClient }); + await expect(params.models!({ client: mockClient })).rejects.toThrow('Network error'); expect(mockClient.models.list).toHaveBeenCalledTimes(1); - expect(models).toBeDefined(); - expect(Array.isArray(models)).toBe(true); - expect(models).toHaveLength(0); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch V0 models. Please ensure your V0 API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); }); - it('should handle API authentication error and return empty array', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + it('should throw when API authentication fails', async () => { const mockClient = { apiKey: 'invalid_key', baseURL: 'https://api.v0.dev/v1', @@ -240,20 +235,12 @@ describe('LobeV0AI - custom features', () => { }, } as any; - const models = await params.models!({ client: mockClient }); + await expect(params.models!({ client: mockClient })).rejects.toThrow('401 Unauthorized'); expect(mockClient.models.list).toHaveBeenCalledTimes(1); - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch V0 models. Please ensure your V0 API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); }); - it('should handle API rate limit error and return empty array', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + it('should throw when API rate limit fails', async () => { const mockClient = { apiKey: 'test_api_key', baseURL: 'https://api.v0.dev/v1', @@ -262,20 +249,12 @@ describe('LobeV0AI - custom features', () => { }, } as any; - const models = await params.models!({ client: mockClient }); + await expect(params.models!({ client: mockClient })).rejects.toThrow('429 Too Many Requests'); expect(mockClient.models.list).toHaveBeenCalledTimes(1); - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch V0 models. Please ensure your V0 API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); }); - it('should handle timeout error and return empty array', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + it('should throw when request times out', async () => { const mockClient = { apiKey: 'test_api_key', baseURL: 'https://api.v0.dev/v1', @@ -284,20 +263,12 @@ describe('LobeV0AI - custom features', () => { }, } as any; - const models = await params.models!({ client: mockClient }); + await expect(params.models!({ client: mockClient })).rejects.toThrow('Request timeout'); expect(mockClient.models.list).toHaveBeenCalledTimes(1); - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch V0 models. Please ensure your V0 API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); }); it('should handle malformed JSON response', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); const mockClient = { apiKey: 'test_api_key', baseURL: 'https://api.v0.dev/v1', @@ -306,15 +277,9 @@ describe('LobeV0AI - custom features', () => { }, } as any; - const models = await params.models!({ client: mockClient }); + await expect(params.models!({ client: mockClient })).rejects.toThrow('Invalid JSON'); - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch V0 models. Please ensure your V0 API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); + expect(mockClient.models.list).toHaveBeenCalledTimes(1); }); it('should pass correct client to processModelList', async () => { diff --git a/packages/model-runtime/src/providers/v0/index.ts b/packages/model-runtime/src/providers/v0/index.ts index a15b4498fc..ae7919ec7b 100644 --- a/packages/model-runtime/src/providers/v0/index.ts +++ b/packages/model-runtime/src/providers/v0/index.ts @@ -14,19 +14,14 @@ export const params = { chatCompletion: () => process.env.DEBUG_V0_CHAT_COMPLETION === '1', }, models: async ({ client }) => { - try { - const modelsPage = (await client.models.list()) as any; - const modelList: V0ModelCard[] = Array.isArray(modelsPage?.data) - ? modelsPage.data - : Array.isArray(modelsPage) - ? modelsPage - : []; + const modelsPage = (await client.models.list()) as any; + const modelList: V0ModelCard[] = Array.isArray(modelsPage?.data) + ? modelsPage.data + : Array.isArray(modelsPage) + ? modelsPage + : []; - return processModelList(modelList, MODEL_LIST_CONFIGS.v0, 'v0'); - } catch (error) { - console.warn('Failed to fetch V0 models. Please ensure your V0 API key is valid:', error); - return []; - } + return processModelList(modelList, MODEL_LIST_CONFIGS.v0, 'v0'); }, provider: ModelProvider.V0, } satisfies OpenAICompatibleFactoryOptions; diff --git a/packages/model-runtime/src/providers/zeroone/index.test.ts b/packages/model-runtime/src/providers/zeroone/index.test.ts index ebc981cf15..667c508768 100644 --- a/packages/model-runtime/src/providers/zeroone/index.test.ts +++ b/packages/model-runtime/src/providers/zeroone/index.test.ts @@ -5,6 +5,12 @@ import { beforeEach, describe, expect, it, vi } from 'vitest'; import { testProvider } from '../../providerTestUtils'; import { LobeZeroOneAI, params } from './index'; +const loadModelsMock = vi.hoisted(() => vi.fn().mockResolvedValue([])); + +vi.mock('@lobechat/business-model-bank/model-config', () => ({ + loadModels: loadModelsMock, +})); + testProvider({ Runtime: LobeZeroOneAI, chatDebugEnv: 'DEBUG_ZEROONE_CHAT_COMPLETION', @@ -200,8 +206,7 @@ describe('LobeZeroOneAI - custom features', () => { expect(models).toHaveLength(0); }); - it('should handle network error and return empty array', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + it('should throw when network error occurs', async () => { const mockClient = { apiKey: 'test_api_key', baseURL: 'https://api.lingyiwanwu.com/v1', @@ -210,22 +215,12 @@ describe('LobeZeroOneAI - custom features', () => { }, } as any; - const models = await params.models!({ client: mockClient }); + await expect(params.models!({ client: mockClient })).rejects.toThrow('Network error'); expect(mockClient.models.list).toHaveBeenCalledTimes(1); - expect(models).toBeDefined(); - expect(Array.isArray(models)).toBe(true); - expect(models).toHaveLength(0); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch ZeroOne models. Please ensure your ZeroOne API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); }); - it('should handle API authentication error and return empty array', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + it('should throw when API authentication fails', async () => { const mockClient = { apiKey: 'invalid_key', baseURL: 'https://api.lingyiwanwu.com/v1', @@ -234,20 +229,12 @@ describe('LobeZeroOneAI - custom features', () => { }, } as any; - const models = await params.models!({ client: mockClient }); + await expect(params.models!({ client: mockClient })).rejects.toThrow('401 Unauthorized'); expect(mockClient.models.list).toHaveBeenCalledTimes(1); - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch ZeroOne models. Please ensure your ZeroOne API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); }); - it('should handle API rate limit error and return empty array', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + it('should throw when API rate limit fails', async () => { const mockClient = { apiKey: 'test_api_key', baseURL: 'https://api.lingyiwanwu.com/v1', @@ -256,20 +243,12 @@ describe('LobeZeroOneAI - custom features', () => { }, } as any; - const models = await params.models!({ client: mockClient }); + await expect(params.models!({ client: mockClient })).rejects.toThrow('429 Too Many Requests'); expect(mockClient.models.list).toHaveBeenCalledTimes(1); - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch ZeroOne models. Please ensure your ZeroOne API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); }); - it('should handle timeout error and return empty array', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + it('should throw when request times out', async () => { const mockClient = { apiKey: 'test_api_key', baseURL: 'https://api.lingyiwanwu.com/v1', @@ -278,20 +257,12 @@ describe('LobeZeroOneAI - custom features', () => { }, } as any; - const models = await params.models!({ client: mockClient }); + await expect(params.models!({ client: mockClient })).rejects.toThrow('Request timeout'); expect(mockClient.models.list).toHaveBeenCalledTimes(1); - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch ZeroOne models. Please ensure your ZeroOne API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); }); it('should handle malformed JSON response', async () => { - const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); const mockClient = { apiKey: 'test_api_key', baseURL: 'https://api.lingyiwanwu.com/v1', @@ -300,15 +271,9 @@ describe('LobeZeroOneAI - custom features', () => { }, } as any; - const models = await params.models!({ client: mockClient }); + await expect(params.models!({ client: mockClient })).rejects.toThrow('Invalid JSON'); - expect(models).toEqual([]); - expect(consoleWarnSpy).toHaveBeenCalledWith( - 'Failed to fetch ZeroOne models. Please ensure your ZeroOne API key is valid:', - expect.any(Error), - ); - - consoleWarnSpy.mockRestore(); + expect(mockClient.models.list).toHaveBeenCalledTimes(1); }); it('should pass correct client to processModelList', async () => { diff --git a/packages/model-runtime/src/providers/zeroone/index.ts b/packages/model-runtime/src/providers/zeroone/index.ts index 2fa14f5ca9..0cafa1bf10 100644 --- a/packages/model-runtime/src/providers/zeroone/index.ts +++ b/packages/model-runtime/src/providers/zeroone/index.ts @@ -14,22 +14,14 @@ export const params = { chatCompletion: () => process.env.DEBUG_ZEROONE_CHAT_COMPLETION === '1', }, models: async ({ client }) => { - try { - const modelsPage = (await client.models.list()) as any; - const modelList: ZeroOneModelCard[] = Array.isArray(modelsPage?.data) - ? modelsPage.data - : Array.isArray(modelsPage) - ? modelsPage - : []; + const modelsPage = (await client.models.list()) as any; + const modelList: ZeroOneModelCard[] = Array.isArray(modelsPage?.data) + ? modelsPage.data + : Array.isArray(modelsPage) + ? modelsPage + : []; - return processModelList(modelList, MODEL_LIST_CONFIGS.zeroone); - } catch (error) { - console.warn( - 'Failed to fetch ZeroOne models. Please ensure your ZeroOne API key is valid:', - error, - ); - return []; - } + return processModelList(modelList, MODEL_LIST_CONFIGS.zeroone); }, provider: ModelProvider.ZeroOne, } satisfies OpenAICompatibleFactoryOptions; diff --git a/src/app/(backend)/webapi/models/[provider]/route.test.ts b/src/app/(backend)/webapi/models/[provider]/route.test.ts index 716791d6f6..e02fde192c 100644 --- a/src/app/(backend)/webapi/models/[provider]/route.test.ts +++ b/src/app/(backend)/webapi/models/[provider]/route.test.ts @@ -1,6 +1,6 @@ // @vitest-environment node -import { type LobeRuntimeAI } from '@lobechat/model-runtime'; -import { ModelRuntime } from '@lobechat/model-runtime'; +import type { LobeRuntimeAI } from '@lobechat/model-runtime'; +import { AgentRuntimeErrorType, ModelRuntime } from '@lobechat/model-runtime'; import { ChatErrorType } from '@lobechat/types'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; @@ -28,6 +28,8 @@ vi.mock('@/server/modules/ModelRuntime', () => ({ let request: Request; beforeEach(() => { + vi.spyOn(console, 'error').mockImplementation(() => {}); + request = new Request(new URL('https://test.com'), { method: 'GET', }); @@ -41,11 +43,12 @@ beforeEach(() => { afterEach(() => { vi.clearAllMocks(); + vi.restoreAllMocks(); }); describe('GET handler', () => { describe('error handling', () => { - it('should not expose stack trace when an Error is thrown', async () => { + it('should return the thrown error message without exposing stack trace', async () => { const mockParams = Promise.resolve({ provider: 'google' }); const errorWithStack = new Error('Something went wrong'); @@ -62,16 +65,16 @@ describe('GET handler', () => { const response = await GET(request, { params: mockParams }); const responseBody = await response.json(); - expect(responseBody.body.error.name).toBe('Error'); - expect(responseBody.body.error.message).toBe('Something went wrong'); - expect(responseBody.body.error.stack).toBeUndefined(); + expect(response.status).toBe(500); + expect(responseBody.errorType).toBe(ChatErrorType.InternalServerError); + expect(responseBody.body.message).toBe('Something went wrong'); const responseText = JSON.stringify(responseBody); expect(responseText).not.toContain('/path/to/file.ts'); expect(responseText).not.toContain('at Object'); }); - it('should preserve error name for custom error types', async () => { + it('should return custom error messages', async () => { const mockParams = Promise.resolve({ provider: 'google' }); class CustomError extends Error { @@ -94,17 +97,17 @@ describe('GET handler', () => { const response = await GET(request, { params: mockParams }); const responseBody = await response.json(); - expect(responseBody.body.error.name).toBe('CustomError'); - expect(responseBody.body.error.message).toBe('Custom error occurred'); - expect(responseBody.body.error.stack).toBeUndefined(); + expect(response.status).toBe(500); + expect(responseBody.errorType).toBe(ChatErrorType.InternalServerError); + expect(responseBody.body.message).toBe('Custom error occurred'); }); - it('should pass through structured error objects as-is', async () => { + it('should preserve structured model fetch error context', async () => { const mockParams = Promise.resolve({ provider: 'google' }); const structuredError = { - errorType: ChatErrorType.InternalServerError, - error: { code: 'PROVIDER_ERROR', details: 'API limit exceeded' }, + errorType: AgentRuntimeErrorType.ProviderBizError, + error: { code: 'PROVIDER_ERROR', message: 'API limit exceeded' }, }; const mockRuntime: LobeRuntimeAI = { @@ -117,11 +120,15 @@ describe('GET handler', () => { const response = await GET(request, { params: mockParams }); const responseBody = await response.json(); + expect(response.status).toBe(471); + expect(responseBody.errorType).toBe(AgentRuntimeErrorType.ProviderBizError); expect(responseBody.body.error.code).toBe('PROVIDER_ERROR'); - expect(responseBody.body.error.details).toBe('API limit exceeded'); + expect(responseBody.body.error.message).toBe('API limit exceeded'); + expect(responseBody.body.message).toBe('API limit exceeded'); + expect(responseBody.body.provider).toBe('google'); }); - it('should return correct status code for errors', async () => { + it('should return generic status code for model fetch errors', async () => { const mockParams = Promise.resolve({ provider: 'google' }); const mockRuntime: LobeRuntimeAI = { @@ -132,8 +139,65 @@ describe('GET handler', () => { vi.mocked(initModelRuntimeFromDB).mockResolvedValue(new ModelRuntime(mockRuntime)); const response = await GET(request, { params: mockParams }); + const responseBody = await response.json(); expect(response.status).toBe(500); + expect(responseBody.errorType).toBe(ChatErrorType.InternalServerError); + expect(responseBody.body.message).toBe('Failed'); + }); + + it('should prefer wrapped cause message for model fetch errors', async () => { + const mockParams = Promise.resolve({ provider: 'openrouter' }); + + const cause = new Error('OpenRouter models API request failed with status 401'); + const wrappedError = new Error('Failed to fetch OpenRouter models', { cause }); + + const mockRuntime: LobeRuntimeAI = { + baseURL: 'abc', + chat: vi.fn(), + models: vi.fn().mockRejectedValue(wrappedError), + }; + vi.mocked(initModelRuntimeFromDB).mockResolvedValue(new ModelRuntime(mockRuntime)); + + const response = await GET(request, { params: mockParams }); + const responseBody = await response.json(); + + expect(response.status).toBe(500); + expect(responseBody.errorType).toBe(ChatErrorType.InternalServerError); + expect(responseBody.body.message).toBe( + 'OpenRouter models API request failed with status 401', + ); + }); + + it('should return generic status code for setup errors', async () => { + const mockParams = Promise.resolve({ provider: 'google' }); + + vi.mocked(initModelRuntimeFromDB).mockRejectedValue(new Error('Setup failed')); + + const response = await GET(request, { params: mockParams }); + const responseBody = await response.json(); + + expect(response.status).toBe(500); + expect(responseBody.errorType).toBe(ChatErrorType.InternalServerError); + expect(responseBody.body.message).toBe('Setup failed'); + }); + + it('should preserve structured setup error type and message', async () => { + const mockParams = Promise.resolve({ provider: 'githubcopilot' }); + + vi.mocked(initModelRuntimeFromDB).mockRejectedValue({ + error: { message: 'Invalid GitHub Copilot API key' }, + errorType: AgentRuntimeErrorType.InvalidProviderAPIKey, + }); + + const response = await GET(request, { params: mockParams }); + const responseBody = await response.json(); + + expect(response.status).toBe(401); + expect(responseBody.errorType).toBe(AgentRuntimeErrorType.InvalidProviderAPIKey); + expect(responseBody.body.message).toBe('Invalid GitHub Copilot API key'); + expect(responseBody.body.error.message).toBe('Invalid GitHub Copilot API key'); + expect(responseBody.body.provider).toBe('githubcopilot'); }); it('should include provider in error response', async () => { diff --git a/src/app/(backend)/webapi/models/[provider]/route.ts b/src/app/(backend)/webapi/models/[provider]/route.ts index 97e339977a..ed4d20f0ae 100644 --- a/src/app/(backend)/webapi/models/[provider]/route.ts +++ b/src/app/(backend)/webapi/models/[provider]/route.ts @@ -1,4 +1,4 @@ -import { type ChatCompletionErrorPayload } from '@lobechat/model-runtime'; +import type { ChatCompletionErrorPayload } from '@lobechat/model-runtime'; import { ChatErrorType } from '@lobechat/types'; import { NextResponse } from 'next/server'; @@ -8,6 +8,48 @@ import { createErrorResponse } from '@/utils/errorResponse'; import { resolveValidWorkspaceIdFromRequest } from '../../_utils/workspace'; +const getMessageFromError = (error: unknown): string | undefined => { + if (error === null || error === undefined) return; + if (typeof error === 'string') return error; + + if (error instanceof Error) { + if (error.cause instanceof Error && error.cause.message) return error.cause.message; + return error.message; + } + + if (typeof error !== 'object') return; + + const message = (error as { message?: unknown }).message; + return typeof message === 'string' ? message : undefined; +}; + +const createModelListErrorResponse = (provider: string, e: unknown) => { + let error = e; + let errorType: ChatCompletionErrorPayload['errorType'] = ChatErrorType.InternalServerError; + let rest: Partial = {}; + + if (e && typeof e === 'object') { + const { + error: errorContent, + errorType: payloadErrorType, + ...payloadRest + } = e as Partial; + + error = errorContent || e; + errorType = payloadErrorType || errorType; + rest = payloadRest; + } + + console.error(`Route: [${provider}] ${errorType}:`, error); + + return createErrorResponse(errorType, { + error, + ...rest, + message: getMessageFromError(error) || getMessageFromError(e) || rest.message, + provider, + }); +}; + export const GET = checkAuth(async (req, { params, userId, serverDB }) => { const provider = (await params)!.provider!; @@ -21,20 +63,6 @@ export const GET = checkAuth(async (req, { params, userId, serverDB }) => { return NextResponse.json(list); } catch (e) { - const { - errorType = ChatErrorType.InternalServerError, - error: errorContent, - ...res - } = e as ChatCompletionErrorPayload; - - const error = errorContent || e; - // track the error at server side - console.error(`Route: [${provider}] ${errorType}:`, error); - - // Sanitize error to avoid exposing stack traces to users - const sanitizedError = - error instanceof Error ? { message: error.message, name: error.name } : error; - - return createErrorResponse(errorType, { error: sanitizedError, ...res, provider }); + return createModelListErrorResponse(provider, e); } }); diff --git a/src/routes/(main)/settings/provider/features/ModelList/EmptyModels.tsx b/src/routes/(main)/settings/provider/features/ModelList/EmptyModels.tsx index 972b51d07d..614f02803c 100644 --- a/src/routes/(main)/settings/provider/features/ModelList/EmptyModels.tsx +++ b/src/routes/(main)/settings/provider/features/ModelList/EmptyModels.tsx @@ -1,4 +1,5 @@ import { Button, Center, Flexbox, Icon, Tooltip } from '@lobehub/ui'; +import { App } from 'antd'; import { createStaticStyles } from 'antd-style'; import { BrainIcon, LucideRefreshCcwDot, PlusIcon } from 'lucide-react'; import { memo, use, useState } from 'react'; @@ -48,6 +49,7 @@ const styles = createStaticStyles(({ css, cssVar }) => ({ const EmptyState = memo<{ provider: string }>(({ provider }) => { const { t } = useTranslation('modelProvider'); + const { message } = App.useApp(); const { allowed: canManageProvider, reason } = usePermission('manage_provider_key'); const [fetchRemoteModelList] = useAiInfraStore((s) => [s.fetchRemoteModelList]); @@ -89,10 +91,22 @@ const EmptyState = memo<{ provider: string }>(({ provider }) => { setFetchRemoteModelsLoading(true); try { await fetchRemoteModelList(provider); - } catch (e) { - console.error(e); + } catch (error) { + console.error(error); + + const errorMessage = + error instanceof Error + ? error.message + : t('providerModels.list.fetcher.errorFallback'); + + message.error( + t('providerModels.list.fetcher.error', { + message: errorMessage, + }), + ); + } finally { + setFetchRemoteModelsLoading(false); } - setFetchRemoteModelsLoading(false); }} > {fetchRemoteModelsLoading diff --git a/src/routes/(main)/settings/provider/features/ModelList/ModelTitle/index.tsx b/src/routes/(main)/settings/provider/features/ModelList/ModelTitle/index.tsx index 4d460b756c..2b583a6854 100644 --- a/src/routes/(main)/settings/provider/features/ModelList/ModelTitle/index.tsx +++ b/src/routes/(main)/settings/provider/features/ModelList/ModelTitle/index.tsx @@ -127,10 +127,22 @@ const ModelTitle = memo( setFetchRemoteModelsLoading(true); try { await fetchRemoteModelList(provider); - } catch (e) { - console.error(e); + } catch (error) { + console.error(error); + + const errorMessage = + error instanceof Error + ? error.message + : t('providerModels.list.fetcher.errorFallback'); + + message.error( + t('providerModels.list.fetcher.error', { + message: errorMessage, + }), + ); + } finally { + setFetchRemoteModelsLoading(false); } - setFetchRemoteModelsLoading(false); }} > {fetchRemoteModelsLoading diff --git a/src/services/__tests__/models.test.ts b/src/services/__tests__/models.test.ts index 3700b3fc1c..4164f6869c 100644 --- a/src/services/__tests__/models.test.ts +++ b/src/services/__tests__/models.test.ts @@ -1,3 +1,5 @@ +import type * as FetchSSE from '@lobechat/fetch-sse'; +import { getMessageError } from '@lobechat/fetch-sse'; import { type Mock } from 'vitest'; import { beforeEach, describe, expect, it, vi } from 'vitest'; @@ -10,6 +12,15 @@ import { ModelsService } from '../models'; vi.stubGlobal('fetch', vi.fn()); +vi.mock('@lobechat/fetch-sse', async () => { + const actual = (await vi.importActual('@lobechat/fetch-sse')) as typeof FetchSSE; + + return { + ...actual, + getMessageError: vi.fn(actual.getMessageError), + }; +}); + vi.mock('@/const/version', () => ({ isDesktop: false, })); @@ -49,6 +60,7 @@ vi.mock('@/store/user/selectors', () => ({ const modelsService = new ModelsService(); const mockedCreateHeaderWithAuth = vi.mocked(createHeaderWithAuth); +const mockedGetMessageError = vi.mocked(getMessageError); const mockedResolveRuntimeProvider = vi.mocked(resolveRuntimeProvider); const mockedInitializeWithClientStore = vi.mocked(initializeWithClientStore); @@ -56,6 +68,7 @@ describe('ModelsService', () => { beforeEach(() => { (fetch as Mock).mockClear(); mockedCreateHeaderWithAuth.mockClear(); + mockedGetMessageError.mockClear(); mockedResolveRuntimeProvider.mockReset(); mockedResolveRuntimeProvider.mockImplementation((provider: string) => provider); mockedInitializeWithClientStore.mockClear(); @@ -109,5 +122,71 @@ describe('ModelsService', () => { spyIsClient.mockRestore(); }); + + it('should throw model fetch error details when server response is not ok', async () => { + (fetch as Mock).mockResolvedValueOnce( + new Response( + JSON.stringify({ + body: { + error: { + message: 'Cloudflare models API returned an invalid response', + name: 'Error', + }, + message: 'Cloudflare models API returned an invalid response', + provider: 'cloudflare', + }, + errorType: 'ProviderBizError', + }), + { status: 471 }, + ), + ); + + await expect(modelsService.getModels('cloudflare')).rejects.toThrow( + 'Cloudflare models API returned an invalid response', + ); + }); + + it('should fall back to translated error message when server error body has no message', async () => { + mockedGetMessageError.mockResolvedValueOnce({ + body: { + provider: 'cloudflare', + }, + message: 'fallback model fetch failure', + type: 'ProviderBizError', + }); + (fetch as Mock).mockResolvedValueOnce( + new Response( + JSON.stringify({ + body: { + provider: 'cloudflare', + }, + errorType: 'ProviderBizError', + }), + { status: 471 }, + ), + ); + + await expect(modelsService.getModels('cloudflare')).rejects.toThrow( + 'fallback model fetch failure', + ); + }); + + it('should propagate server fetch network errors', async () => { + (fetch as Mock).mockRejectedValueOnce(new Error('network down')); + + await expect(modelsService.getModels('openai')).rejects.toThrow('network down'); + }); + + it('should propagate client runtime model fetch errors', async () => { + const spyIsClient = vi + .spyOn(aiProviderSelectors, 'isProviderFetchOnClient') + .mockReturnValue(() => true); + const mockModels = vi.fn().mockRejectedValue(new Error('client runtime failed')); + mockedInitializeWithClientStore.mockResolvedValue({ models: mockModels } as any); + + await expect(modelsService.getModels('openai')).rejects.toThrow('client runtime failed'); + + spyIsClient.mockRestore(); + }); }); }); diff --git a/src/services/models.ts b/src/services/models.ts index 32d22aaa61..b0d3f228f6 100644 --- a/src/services/models.ts +++ b/src/services/models.ts @@ -34,26 +34,30 @@ export class ModelsService { }); const runtimeProvider = resolveRuntimeProvider(provider); - try { - /** - * Use browser agent runtime - */ - const enableFetchOnClient = isEnableFetchOnClient(provider); - if (enableFetchOnClient) { - const agentRuntime = await initializeWithClientStore({ - provider, - runtimeProvider, - }); - return agentRuntime.models(); - } - - const res = await fetch(API_ENDPOINTS.models(provider), { headers }); - if (!res.ok) return; - - return res.json(); - } catch { - return; + /** + * Use browser agent runtime + */ + const enableFetchOnClient = isEnableFetchOnClient(provider); + if (enableFetchOnClient) { + const agentRuntime = await initializeWithClientStore({ + provider, + runtimeProvider, + }); + return agentRuntime.models(); } + + const res = await fetch(API_ENDPOINTS.models(provider), { headers }); + if (!res.ok) { + const error = await getMessageError(res); + const message = + typeof error.body?.message === 'string' && error.body.message + ? error.body.message + : error.message; + + throw new Error(message, { cause: error }); + } + + return res.json(); }; /** diff --git a/src/store/aiInfra/slices/aiModel/action.test.ts b/src/store/aiInfra/slices/aiModel/action.test.ts index 1a7878da72..103010767c 100644 --- a/src/store/aiInfra/slices/aiModel/action.test.ts +++ b/src/store/aiInfra/slices/aiModel/action.test.ts @@ -224,6 +224,7 @@ describe('AiModelAction', () => { .mockResolvedValue(undefined); // Mock dynamic import + vi.resetModules(); vi.doMock('@/services/models', () => ({ modelsService: { getModels: vi.fn().mockResolvedValue(mockRemoteModels), @@ -274,6 +275,7 @@ describe('AiModelAction', () => { .mockResolvedValue(undefined); // Mock dynamic import with null response + vi.resetModules(); vi.doMock('@/services/models', () => ({ modelsService: { getModels: vi.fn().mockResolvedValue(null), @@ -286,6 +288,28 @@ describe('AiModelAction', () => { expect(batchUpdateSpy).not.toHaveBeenCalled(); }); + + it('should propagate remote service errors', async () => { + const { result } = renderHook(() => useStore()); + const batchUpdateSpy = vi + .spyOn(result.current, 'batchUpdateAiModels') + .mockResolvedValue(undefined); + + vi.resetModules(); + vi.doMock('@/services/models', () => ({ + modelsService: { + getModels: vi.fn().mockRejectedValue(new Error('model fetch failed')), + }, + })); + + await expect(async () => { + await act(async () => { + await result.current.fetchRemoteModelList('test-provider'); + }); + }).rejects.toThrow('model fetch failed'); + + expect(batchUpdateSpy).not.toHaveBeenCalled(); + }); }); describe('internal_toggleAiModelLoading', () => {