44 Commits

Author SHA1 Message Date
Fam Zheng
214d2a2338 feat: file sync from worker to server + report.md convention
- Add AgentUpdate::FileSync (base64-encoded) for file transfer
- Worker syncs all workspace files to server after workflow completes
  (skips .venv, __pycache__, .git, node_modules, files > 1MB)
- Server writes synced files to /app/data/workspaces/{project_id}/
- Remove report field from WorkflowComplete (use report.md convention)
- Update prompts: last step should generate report.md, remove KB/worker tool references
2026-04-06 19:41:17 +01:00
Fam Zheng
1958b5219b fix: remove serde flatten to avoid duplicate workflow_id, fix UTF-8 truncation panic 2026-04-06 16:35:41 +01:00
Fam Zheng
fc91716d83 fix: add WebSocket ping keepalive in worker (30s interval) 2026-04-06 15:56:21 +01:00
Fam Zheng
9e07880132 fix: add parse error logging in ws_worker, fix worker workdir path 2026-04-06 15:54:06 +01:00
Fam Zheng
54517b27f2 fix: make server/database config optional (worker only needs llm) 2026-04-06 14:03:11 +01:00
Fam Zheng
decabc0e8a refactor: server no longer runs agent loop or LLM
- Remove agent_loop from server (was ~400 lines) — server dispatches to workers
- AgentManager simplified to pure dispatcher (send_event → worker)
- Remove LLM config requirement from server (workers bring their own via config.yaml)
- Remove process_feedback, build_feedback_tools from server
- Remove chat API endpoint (LLM on workers only)
- Remove service proxy (services run on workers)
- Worker reads LLM config from its own config.yaml
- ws_worker.rs handles WorkerToServer::Update messages (DB + broadcast)
- Verified locally: tori server + tori worker connect and register
2026-04-06 13:18:21 +01:00
Fam Zheng
e4ba385112 refactor: worker mode — server offloads all LLM/exec to worker
- Split into `tori server` / `tori worker` subcommands (clap derive)
- Extract lib.rs for shared crate (agent, llm, exec, state, etc.)
- Introduce AgentUpdate channel to decouple agent loop from DB/broadcast
- New sink.rs: AgentUpdate enum + ServiceManager + handle_agent_updates
- New worker_runner.rs: connects to server WS, runs full agent loop
- Expand worker protocol: ServerToWorker (workflow_assign, comment)
  and WorkerToServer (register, result, update)
- Remove LLM from title generation (heuristic) and template selection
  (must be explicit)
- Remove KB tools (kb_search, kb_read) and remote worker tools
  (list_workers, execute_on_worker) from agent loop
- run_agent_loop/run_step_loop now take mpsc::Sender<AgentUpdate>
  instead of direct DB pool + broadcast sender
2026-04-06 12:54:57 +01:00
Fam Zheng
28a00dd2f3 feat: configurable OAuth (Google + TikTok SSO), project membership, inline file preview
- Auth: configurable OAuthProvider enum supporting Google OAuth and TikTok SSO
- Auth: /auth/provider endpoint for frontend to detect active provider
- Auth: user role system (admin via ADMIN_USERS env var sees all projects)
- Projects: project_members many-to-many table with role (owner/member)
- Projects: membership-based access control, auto-add creator as owner
- Projects: member management API (list/add/remove)
- Files: remove Content-Disposition attachment header, let browser decide
- Health: public /tori/api/health endpoint for k8s probes
2026-03-17 03:42:38 +00:00
Fam Zheng
63f0582f54 feat: add Google OAuth, remote worker system, and file browser
- Google OAuth login with JWT session cookies, per-user project isolation
- Remote worker registration via WebSocket, execute_on_worker/list_workers agent tools
- File browser UI in workflow view, file upload/download API
- Deploy script switched to local build, added tori.euphon.cloud ingress
2026-03-17 02:00:58 +00:00
Fam Zheng
186d882f35 feat: JWT token generation API + auto-inject TORI_JWT in executor
- POST /tori/api/token — sign ES256 JWT with configurable private key
- exec.rs auto-generates and injects TORI_JWT env var for all commands
- Config: jwt_private_key field for PEM file path
2026-03-16 09:44:58 +00:00
Fam Zheng
f2fa721ef0 refactor: rename wait_for_approval to ask_user
More general-purpose user intervention tool — not just approve/reject,
but any question or input request. Renames across Rust backend, Vue
frontend, prompts, and status strings.

Tool: wait_for_approval → ask_user (param: reason → question)
Status: WaitingApproval → WaitingUser, waiting_approval → waiting_user
Enum: NeedsApproval → NeedsInput
2026-03-16 08:50:24 +00:00
Fam Zheng
dae99d307a feat: support require_plan_approval in template config
Templates can now set "require_plan_approval": true in template.json
to require user approval after plan generation before execution begins.
On rejection, the LLM re-enters the planning loop with user feedback.
2026-03-12 19:25:35 +00:00
Fam Zheng
30e25f589b api: convert tool_input JSON to YAML in execution log response 2026-03-11 12:43:35 +00:00
Fam Zheng
63bbbae17c feat: show plan diff in execution log when revise_plan is called
- apply_plan_diff now returns a YAML unified diff string
- Pure Rust LCS diff implementation (no external dependency)
- revise_plan logs the diff to execution log with ```diff fencing
- Frontend renders diff with green/red syntax highlighting
2026-03-10 19:03:47 +00:00
Fam Zheng
978af45d5f fix: preserve step chat history when resuming after wait_for_approval
Previously, current_step_chat_history was cleared unconditionally on
resume, causing the agent to lose context and re-run the step from
scratch after approval. Now only clear when advancing to a new step.
2026-03-10 18:37:45 +00:00
Fam Zheng
5188027d8b fix: resume waiting_approval workflows directly instead of re-planning
When a user approves a wait_for_approval step, resume execution from that
step instead of calling process_feedback which may trigger revise_plan
and restart from step 1. Also check state snapshot for WaitingApproval
steps to handle pod restart scenarios where DB status may already be
'executing' but state still has the waiting step.
2026-03-10 13:55:47 +00:00
Fam Zheng
1105c909bd feat: force planning to follow template-defined steps verbatim 2026-03-09 20:09:22 +00:00
Fam Zheng
f214b67f92 feat: add template setup executable support
Templates can now include a `setup` file that runs in the workdir
before agent execution starts. Used for workspace initialization
like pulling tool binaries or installing dependencies.
2026-03-09 17:02:12 +00:00
Fam Zheng
c70fbc49f0 feat: auto-install Python deps from template pyproject.toml via uv
ExternalToolManager.discover() now accepts template root dir, detects
pyproject.toml and runs `uv sync` to create a venv. Tool invocation and
schema discovery inject the venv PATH/VIRTUAL_ENV so template tools can
import declared dependencies without manual installation.
2026-03-09 15:22:35 +00:00
Fam Zheng
fa800b1601 feat: step artifacts framework
- Add Artifact type to Step (name, path, artifact_type, description)
- step_done tool accepts optional artifacts parameter
- Save artifacts to step_artifacts DB table
- Display artifacts in frontend PlanSection (tag style)
- Show artifacts in step context for sub-agents and coordinator
- Add LLM client retry with exponential backoff
2026-03-09 12:01:29 +00:00
Fam Zheng
29f026e383 feat: real-time activity indicator in log panel
- New WsMessage::ActivityUpdate for live status broadcasting
- Shows current activity at bottom of log: LLM calls, tool execution, user approval
- Activity bar with spinner, auto-clears on workflow completion
- Status badge with pulse animation in log header
2026-03-09 10:26:42 +00:00
Fam Zheng
d4d9edeb78 feat: startup git clone for template repo + pass config through
- ensure_repo_ready() at startup: clone if missing, fetch if exists
- TemplateRepoConfig gains local_path field
- list_all_templates/select_template/extract_repo_template accept repo config
- Remove hardcoded repo_dir(), use config.local_path
2026-03-09 08:42:23 +00:00
Fam Zheng
815477a73b feat: template examples + recent templates quick buttons
- Add TemplateExample struct and examples scanning (local dir + git repo)
- Exclude examples/ from copy_dir_recursive
- Frontend: recent templates (localStorage), template-specific example buttons
2026-03-08 08:56:40 +00:00
Fam Zheng
feb2a08d97 feat: step isolation — each step runs in independent sub-loop
Main loop becomes a coordinator that reviews step summaries and may
revise the plan. Each step gets its own chat history and scratchpad,
preventing context pollution across steps.

- Add run_step_loop with 50-iteration limit and isolated context
- Replace advance_step with step_done (sub-loop only)
- Add coordinator review after each step completion
- Add scratchpad 8K capacity check
- Add 33 unit tests for state, tools, and message building
2026-03-08 08:35:41 +00:00
Fam Zheng
47546a9d15 feat: add approve/reject buttons for wait_for_approval
- CommentSection shows explicit approve/reject buttons when waiting
- Reject aborts the workflow, approve continues with optional feedback
- Backend parses approved:/rejected: prefixes from comment content
2026-03-07 16:41:15 +00:00
Fam Zheng
3500659e06 feat: add wait_for_approval tool for agent workflow pausing
Allow agent to pause execution at critical decision points and wait
for user confirmation via comments before continuing.
2026-03-07 16:37:17 +00:00
Fam Zheng
07f1f285b6 feat: multi-branch template scanning from git repo + manual template selection
- Rewrite template.rs to scan all remote branches via git commands
  (git fetch/branch -r/ls-tree/git show/git archive)
- Add manual template picker dropdown in CreateForm UI
- Remove sentence-transformers/embed.py from Dockerfile (separate container)
- Clean up Gitea API approach, use local git repo instead
- Add chat panel and sidebar layout improvements
2026-03-07 16:24:56 +00:00
Fam Zheng
69ad06ca5b improve: enhance KB search with better embedding and chunking 2026-03-04 11:47:03 +00:00
Fam Zheng
fe1370230f refactor: extract template and tools modules from agent
Split template selection and external tool management into dedicated
modules for better separation of concerns.
2026-03-04 11:47:01 +00:00
Fam Zheng
c0b681adc3 feat: add settings API with key-value store
GET/PUT endpoints for app settings backed by a settings table.
2026-03-04 11:46:58 +00:00
Fam Zheng
a3c9fbe8e5 feat: add object storage REST API
File-based object storage with GET/PUT/DELETE, directory listing,
path traversal protection, and streaming upload up to 2GB.
2026-03-04 11:46:55 +00:00
Fam Zheng
ae72e699f4 refactor: move routes under /tori/ prefix and add /api/obj mount
Routes now live at /tori/api, /ws/tori, and static files at /tori/.
Root / redirects to /tori/. Object storage mounted at /api/obj.
Dev proxy updated accordingly.
2026-03-04 11:46:52 +00:00
Fam Zheng
728ab2e8fd Pre-install common Python packages in workspace venv and warm Docker cache
Install httpx, fastapi, uvicorn, requests, flask, pydantic, numpy, pandas,
matplotlib, pillow, jinja2, pyyaml and more on workspace creation. Also
pre-warm uv cache in Dockerfile so first project setup is near-instant.
uv's global cache (~/.cache/uv/) is shared across all project venvs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 09:19:48 +00:00
Fam Zheng
0a8eee0285 LLM call logging, plan persistence API, quote-to-feedback UX, requirement input improvements
- Add llm_call_log table and per-call timing/token tracking in agent loop
- New GET /workflows/{id}/plan endpoint to restore plan from snapshots on page load
- New GET /workflows/{id}/llm-calls endpoint + WS LlmCallLog broadcast
- Parse Usage from LLM API response (prompt_tokens, completion_tokens)
- Detailed mode toggle in execution log showing LLM call cards with phase/tokens/latency
- Quote-to-feedback: hover quote buttons on plan steps and log entries, multi-quote chips in comment input
- Requirement input: larger textarea, multi-line display with pre-wrap and scroll

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 09:16:51 +00:00
Fam Zheng
46424cfbc4 Refactor agent runtime: state machine, feedback processing, execution log
- Add state.rs with AgentState/Step/StepStatus/AgentPhase as single source of truth
- Extract prompts to markdown files loaded via include_str!
- Replace plan_steps table with execution_log + agent_state_snapshots
- Implement user feedback processing with docker-build-cache plan diff:
  load snapshot → LLM revise_plan → diff (title, description) → invalidate from first mismatch → resume
- run_agent_loop accepts optional initial_state for mid-execution resume
- Broadcast plan step status (done/running/pending) to frontend on step transitions
- Rewrite frontend types/components to match new API (ExecutionLogEntry, PlanStepInfo with status)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 08:54:43 +00:00
Fam Zheng
ee4a5dfc95 App-templates: LLM auto-selects project template based on user requirement
- Add webapp template (FastAPI + SQLite) with INSTRUCTIONS.md and setup.sh
- select_template() scans templates, asks LLM to match; apply_template() copies to workspace
- ensure_workspace() runs setup.sh if present, otherwise falls back to default venv
- INSTRUCTIONS.md injected into planning and execution prompts
- Fix pre-existing clippy warning in kb.rs (filter_map → map)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 21:33:40 +00:00
Fam Zheng
837977cd17 Resume incomplete workflows on server startup
Query workflows with status pending/planning/executing after restart
and re-submit them to the agent manager for continuation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 09:02:01 +00:00
Fam Zheng
40f200db4f KB multi-article support: CRUD articles, per-article indexing, sidebar KB mode
- Replace singleton kb_content table with kb_articles (id, title, content)
- Add article_id to kb_chunks for per-article chunk tracking
- Auto-migrate old kb_content data on startup
- KbManager: index/delete per article, search across all with article_title
- API: full CRUD on /kb/articles, keep GET /kb for agent tool
- Agent: kb_search shows article labels, kb_read concatenates all articles
- Frontend: Sidebar KB mode with article list, KbEditor for single article

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 08:56:08 +00:00
Fam Zheng
ddbe61818b Use uv venv for embedding, pre-download model in Docker build
- Use /app/venv with uv instead of system python/pip
- Pre-download all-MiniLM-L6-v2 model during Docker build

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 08:33:21 +00:00
Fam Zheng
fbf636868c Switch from fastembed to Python sentence-transformers for embedding
ort (ONNX Runtime) has no prebuilt binaries for aarch64-musl.
Use a Python subprocess with sentence-transformers instead:
- scripts/embed.py: reads JSON stdin, outputs embeddings
- kb.rs: calls Python script via tokio subprocess
- Dockerfile: install python3 + sentence-transformers

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 08:31:31 +00:00
Fam Zheng
d9d3bc340c Add global knowledge base with RAG search
- KB module: fastembed (AllMiniLML6V2) for CPU embedding, SQLite for
  vector storage with brute-force cosine similarity search
- Chunking by ## headings, embeddings stored as BLOB in kb_chunks table
- API: GET/PUT /api/kb for full-text read/write with auto re-indexing
- Agent tools: kb_search (top-5 semantic search) and kb_read (full text)
  available in both planning and execution phases
- Frontend: Settings menu in sidebar footer, KB editor as independent
  view with markdown textarea and save button
- Also: extract shared db_err/ApiResult to api/mod.rs, add context
  management design doc

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 08:15:50 +00:00
Fam Zheng
1aa81896b5 Add project soft-delete with workspace archival
- Add delete button (×) to sidebar project list, shown on hover
- Soft-delete: mark projects as deleted in DB instead of hard delete
- Move workspace files to /app/data/deleted/ folder on deletion
- Filter deleted projects from list query
- Auto-select next project after deleting current one
- Also includes agent prompt improvements for reverse proxy paths

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 07:56:37 +00:00
Fam Zheng
2df4e12d30 Agent loop state machine refactor, unified LLM interface, and UI improvements
- Rewrite agent loop as Planning→Executing(N)→Completed state machine with
  per-step context isolation to prevent token explosion
- Split tools and prompts by phase (planning vs execution)
- Add advance_step/save_memo tools for step transitions and cross-step memory
- Unify LLM interface: remove duplicate types, single chat_with_tools path
- Add UTF-8 safe truncation (truncate_str) to prevent panics on Chinese text
- Extract CreateForm component, add auto-scroll to execution log
- Add report generation with app access URL, non-blocking title generation
- Add timer system, file serving, app proxy, exec module
- Update Dockerfile with uv, deployment config

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 22:35:33 +00:00
Fam Zheng
7edbbee471 Tori: AI agent workflow manager - initial implementation
Rust (Axum) + Vue 3 + SQLite. Features:
- Project CRUD REST API with proper error handling
- Per-project agent loop (mpsc + broadcast channels)
- LLM-driven plan generation and replan on user feedback
- SSH command execution with status streaming
- WebSocket real-time updates to frontend
- Four-zone UI: requirement, plan (left), execution (right), comment

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 10:36:50 +00:00