Audit 2026-10-11 (v0.7.0)

View raw
On this page

This is a full end-to-end check before deploying v0.7.0 (User Chat, Dev Chat, Sandbox Factory, HF provider, quotas, devbox executor). We checked everything locally first (opennextjs-cloudflare preview with the remote AI binding and local D1), then on production after the deploy (the "Live" section).

Summary#

Area Result
Pages (home, templates + 10 templates, builder, design, docs + raw md + llms.txt, chat, chat/dev, factory, 404) 200 / 404 as expected, CSP on every response, no console errors
API /api/plan 415/413/400/429 + Retry-After, HF → offline chain, key never echoed
API chat 400 empty, 413 too_long, 400 invalid JSON, 404 foreign or cross-channel conversation, 404 foreign image, 405 on POST /api/chat, 429 burst
API attachments 413 too_large (11 MB), 415 unsupported_type, 503 uploads_disabled (no R2)
API factory 401 no key, 401 key_rejected for an invalid key, 200 config; self-hosted session with a placeholder-only command
User Chat end-to-end (local) Gemma 4, SEA-LION, Qwen Coder, DeepSeek V3 (HF) streamed; FLUX image 332 KB JPEG
Dev Chat end-to-end (local) Sandbox command python3 -c "print(sum(range(10)))" → 45, 70,369 in / 205 out tokens
Quotas unit-tested: 20 msg, 5 images, IP backstop ×3, agent global 30, Bangkok-midnight buckets, refund
a11y (axe wcag2a/aa, serious+critical) chat / dev / factory / templates / builder clean; older issues listed below
Mobile 390px + Thai + reduced motion no horizontal overflow, lang=th, reduced-motion respected
docs:check passed (51 pages)
Tests 180/180 vitest; tsc, eslint clean; cf:build OK
Leaked-secret scan no real secret values in committed files (scanned in-process against env; only fake test keys and redaction regexes)
devbox executor devboxctl status: supervisor running, 0 restarts, executor up since 13:13 (UTC+7)
D1 local at schema 5; remote at 2, with 0003–0005 pending (applied during the deploy below)
Secrets (names) production had OPENAI_API_KEY only → CHAT_COOKIE_SECRET and HF_TOKEN added at deploy (via stdin)
Git feat/d1-chat-schema and feat/chat-orchestrator were unmerged; both merged into main with --no-ff

Problems found#

Fixed in this release#

  1. Cloudflare context was imported dynamically in runtime.ts and rate-limit.ts. After the first request this failed with "Cannot perform I/O on behalf of a different request", so the Workers rate limiter for /api/plan silently fell back to in-memory counters in production. Fixed with a static import and getCloudflareContext({ async: true }).
  2. CSP blocked Cloudflare Web Analytics, which is injected on the zone, causing console errors on live. We now allow the https://static.cloudflareinsights.com script and https://cloudflareinsights.com connect.
  3. robots.txt and sitemap.xml returned 404. They are now generated by src/app/robots.ts and sitemap.ts (68 URLs; /api/ disallowed).
  4. The header overflowed at 1024–1279 px after adding the chat links (189 px). Dev Chat, Factory and Design now show from xl up, the hamburger menu covers up to xl, the palette trigger label only shows at 2xl (with an aria-label), and "Sandbox Factory" is shortened to "Factory" in the nav.
  5. The Dev Chat Sandbox panel showed User Chat quotas. It now shows the agent turns left.
  6. The home page code sample was a scroll region that couldn't take focus (axe scrollable-region-focusable). It now has tabIndex=0 and an aria-label.
  7. cloudflare-env.d.ts was regenerated with runtime types, which broke the type check. It is now generated with --include-runtime=false (the cf:typegen script).
  8. Plan tests depended on the box's HF_TOKEN. The tests now stub HF_TOKEN and OPENAI_FALLBACK, and a test for the HF-first chain was added.
  9. Orchestrator JSONL logs contained environment ids and remote URLs. They are now redacted (redact_ids) and existing logs were scrubbed. The logs are gitignored.

Open#

  1. R2 is not enabled on the account. Images are stored inline in D1 (attachment_blobs, ≤1.5 MB), uploads answer 503 uploads_disabled, and the attach button is disabled.
  2. Turnstile is not implemented on the chat pages. Abuse protection relies on the cookie, IP and global quotas plus the burst limiters.
  3. There is no hard max_output_tokens in the Agents API. Dev Chat cost is bounded by the global turn cap (30/day) and the $1/day budget estimate.
  4. The PDF spec contained a live OpenAI key and ids. We never used them; the key must be revoked by its owner.
  5. Docs prose links aren't underlined (axe link-in-text-block, /docs). This needs a design decision (visual change).
  6. One contrast hit on /design: a disabled-state hint (text-subtle on surface, 2.51:1). This needs a design token decision.
  7. Factory files created in a user's own sandbox can't be downloaded through us.
  8. The devbox reboot hook is best-effort (no systemd/cron on the box; devboxctl ensure runs from the shell profile). The dashboard self-hosted session may expire upstream.
  9. Planner output reports source: "openai" with model: "hf:…" when the plan came from HF. The schema only knows openai/fallback; this is cosmetic.
  10. One flaky test run: docs-check.test.ts failed once while cf:build was regenerating the docs bundle at the same time. It passes when run on its own (3/3).
  11. Workers AI in local preview always uses the account's real allowance (the binding is remote).

Costs (this audit)#

  • Workers AI: about 8 text replies plus 2 FLUX images, roughly 1–2k neurons, within the 10k/day free allowance.
  • HF: a few DeepSeek V3 calls (HF Pro credits).
  • OpenAI: 2 Dev Chat turns (about 70k input tokens each, mostly cached) at roughly $0.02–0.15 total. Factory test sessions with no turns cost nothing.

Live (production, https://devstack.bid)#

The deploy ran on 2026-10-11 at about 13:50 (UTC+7). Worker nvx-stack-builder, account 2d92bd5b25768fa9093d6adc0a8887fc, custom domains devstack.bid and www.devstack.bid (unchanged).

  • D1 remote: migrations 0003, 0004 and 0005 applied; schema_version = 5.
  • Secrets (names): OPENAI_API_KEY (kept), HF_TOKEN (new, via stdin), CHAT_COOKIE_SECRET (new, freshly generated, via stdin).
  • Bindings: AI (Workers AI), DB, NVX_PLAN_LIMITER, NVX_PLAN_AI_LIMITER, NVX_CHAT_LIMITER, NVX_FACTORY_LIMITER, ASSETS, WORKER_SELF_REFERENCE.
Check Result
All pages 200 (404 for unknown paths); CSP includes cloudflareinsights; www → 308 to apex
robots.txt / sitemap.xml 200
Console errors 0 on 20 page loads (desktop EN and mobile TH with reduced motion); the only error is the expected 404 resource on /nope
axe only the open items 5–6 remain (docs links, one /design contrast)
User Chat (Thai · SEA-LION) streamed Thai answer, 181 in / 160 out tokens, messagesLeft 19
User Chat image (FLUX) 453 KB JPEG, owner-only (404 without the cookie), imagesLeft 4
Dev Chat turn 2 docs lookups and a sandbox command → 3.12.12 plus the fastapi-ai template, 48,042 in / 203 out tokens, agentLeft 4
/factory loads; GET /api/factory lists models and scopes
/api/plan provider: huggingface, model: deepseek-ai/DeepSeek-V3

Screenshots: screenshots/v0.7/live-*.png.

Rollback#

Version What
450890b7-5432-4795-afbc-1c7ebd241bff v0.6.1 (D1 binding); the production version before this deploy
cfe6c97c-3b8d-40c4-8775-3f3f68b82076 v0.7.0, first deploy

Roll back with wrangler rollback <version-id> --name nvx-stack-builder. The D1 migrations are additive (new columns and tables), so old code keeps working with schema 5.

Source: docs/operations/audit-2026-10-11.md · /docs/operations/audit-2026-10-11.md