feat(db): add project personas and skills tables
CI / build-and-test (push) Failing after 1m52s

* Introduce project_personas table with foreign keys to projects and agent_personas
* Add project_skills table with foreign key to projects and agent_skills
* Include override boolean field in agent_persona_skills and project_skills
* Update schema and migration files to reflect new tables and fields
* Enhance CORS handling to reflect request origin
This commit is contained in:
2026-07-04 23:45:51 +02:00
parent 1adf50e3db
commit c179e014ad
69 changed files with 1329 additions and 184 deletions
+99
View File
@@ -1,5 +1,104 @@
# AMCS TODO
## Memory Consolidation and Retrieval Quality
No prior consolidation implementation notes were found in the project. This roadmap starts clean and follows AMCS's current Go, PostgreSQL/pgvector, thoughts, learnings, projects, and model-specific embeddings architecture.
### Current-shape decisions
- Use bigint thought IDs for foreign keys and queue references, matching the current schema. Keep thought GUIDs as external identifiers rather than introducing UUID foreign keys without a specific interoperability need.
- Embeddings are stored in the separate, model-specific `embeddings` table. Similarity queries and indexes must include the embedding model and project scope where applicable.
- Verify the existing pgvector index strategy before adding another HNSW index. Add or change an index only when the current generated schema and query plan show it is needed.
- All consolidation writes are soft, auditable, idempotent, and reversible.
### Phase 0 - Schema additions
Add to `thoughts`:
- `confidence smallint`
- `source text`
- `valid_until timestamptz`
- `superseded_by bigint` referencing `thoughts.id`
- `salience real`
- `outcome text` constrained to `confirmed`, `refuted`, or null
Add tables:
- `consolidation_queue(id, kind, thought_a, thought_b, status, created_at)`
- `consolidation_audit(id, action, before jsonb, after jsonb, actor, ts)`
The audit record must contain enough information to reverse every consolidation mutation. Define queue uniqueness and status transitions so retries cannot enqueue or process the same pair twice.
Confidence must be available through the existing admin thought resource. Return both the normalized numeric value and enough provenance to explain whether it was user-supplied, heuristic, outcome-derived, or consolidation-derived. Existing thoughts need a documented neutral default or backfill policy rather than silently appearing maximally confident.
### Phase 1 - Algorithmic candidate generation
Run nightly SQL candidate generation without an LLM:
- Near duplicates: self-join model-compatible embeddings and enqueue canonical thought pairs where `1 - (a.embedding <=> b.embedding) > 0.92`.
- Contradiction suspects: combine high cosine similarity with low `pg_trgm` `similarity()` and enqueue as `kind = 'contradiction'`.
- Decay: demote thoughts past `valid_until`; apply recency weighting during retrieval rather than rewriting content.
Candidate queries must de-duplicate ordered pairs, exclude already superseded thoughts, respect project boundaries, and avoid comparing embeddings from different models.
### Phase 2 - Retrieval ranking (ship first)
Change `recall_context` from cosine-only top-k ranking to a bounded composite score based on:
```text
cosine_similarity * exp(-lambda * age_days) * confidence_weight * outcome_weight
```
Exclude superseded and low-salience thoughts from default recall, with explicit options for diagnostic retrieval. Tune defaults conservatively and preserve the current full-text fallback when embeddings are unavailable.
This phase should ship first because it provides the largest recall-quality improvement without requiring consolidation writes or an adjudication model.
### Phase 3 - LLM adjudication worker
Add a resumable Go worker that claims queued pairs and calls a low-cost configured provider only for flagged candidates. Support two structured prompts:
- Merge: produce one normalized schema entry from genuine duplicates.
- Resolve conflict: verify a real contradiction and select or synthesize the supported result.
Writes remain soft: set `superseded_by`, never delete source thoughts, and append every mutation to `consolidation_audit`. Queue claiming, provider calls, and writes must be idempotent and safe across restarts.
### Phase 4 - Salience gate
Compute initial salience during `capture_thought` using explainable heuristics such as length, code presence, decision verbs, and thought type. Store low-salience thoughts but exclude them from default recall. Consider a logistic model only after measured retrieval outcomes show the heuristics have plateaued.
### Phase 5 - Outcome loop
Add a tool or update flag that marks a thought as `confirmed` or `refuted` when its decision plays out. Preserve the change in the consolidation audit and feed the outcome into retrieval ranking as the real-world error signal.
### Phase 6 - Admin confidence and SQL job observability
Extend the Svelte admin UI and its ResolveSpec-compatible backend resources with two read-oriented views.
#### Thought confidence
- Show confidence in thought list and detail views without requiring the operator to open raw metadata.
- Display the numeric value, a human-readable band, provenance, outcome, salience, validity window, and superseded state.
- Allow sorting and filtering by confidence, outcome, validity, and superseded state.
- Visually distinguish unknown/unscored confidence from low confidence; never coerce null into zero or one.
- Explain ranking inputs in the detail view so operators can understand why a thought was included or excluded from default recall.
- Keep confidence mutation behind the normal thought update/audit path rather than allowing unaudited inline table edits.
#### Consolidation SQL jobs
- Show configured `pg_cron` consolidation jobs and their schedule, enabled state, last start/end time, duration, status, and sanitized result/error message.
- Show currently running consolidation jobs separately from recent completed/failed runs using `cron.job` and `cron.job_run_details` through a permission-limited backend projection.
- Include queue counts by status and consolidation kind so a successful cron invocation cannot hide a stalled or growing queue.
- Add filters for job, status, and time range, plus clear empty/unavailable states when `pg_cron` is not installed or its views are inaccessible.
- Do not expose raw SQL command text, connection strings, credentials, or unrestricted `pg_stat_activity` data.
- Keep the first version read-only. Manual run, retry, pause, or cancel controls require separate authorization, audit logging, and explicit confirmation.
- Poll conservatively only while the jobs screen is visible; stop polling when hidden and avoid loading run history during normal admin startup.
Backend tests must cover `pg_cron` unavailable/permission-denied behavior, sanitized errors, running-versus-finished classification, pagination, and project-independent admin authorization. Frontend tests must cover confidence null handling, failed/running job states, queue backlog warnings, and polling cleanup.
### Delivery principle
SQL handles high-volume candidate recall and ranking; the LLM handles low-volume precision decisions for a bounded number of queued pairs. Implement Phase 2 first, then Phase 0/1 foundations needed for consolidation, followed by the worker, salience gate, outcome feedback loop, and admin observability. Confidence fields should appear in the admin UI as soon as Phase 0 lands; SQL job observability should land with the first scheduled Phase 1 jobs.
## Future Plugin: Lifestyle Tools (calendar, meals, household, CRM)
The following tool groups have been removed from the core server and are candidates for a separate optional plugin or extension server. The store/tool implementations remain in the codebase but are no longer registered.