Compare commits

...
59 Commits
Author SHA1 Message Date
Hein a4702161fb docs(readme): add breaking changes section for 2026-09-30 to 2026-10-01 2026-10-01 15:17:37 +02:00
Hein 640faeeeaf feat(security): full OAuth 2.1 / OpenID Connect server and OIDC relying-party client
Authorization server: consent and scopes, OIDC (nonce, auth_time, acr, sid,
at_hash, signed userinfo, RP-initiated and back-channel logout), managed
refresh tokens with rotation and reuse detection, RFC 9068 JWT access tokens,
DPoP, PAR, device grant, token exchange, private_key_jwt, RFC 7591/7592
registration, RFC 9207 iss, signing keyring with rotation.

State is DB-backed through a new lookup.OAuthGrantStore (procedure and direct
backends, four dialect DDLs, conformance cases).

Client side: WithOIDC discovery, PKCE, nonce, id_token validation, OAuth2LogoutURL.

PeekRefresh now returns already rotated tokens so RotateRefresh can detect reuse.

Docs: OAUTH2_SERVER.md, oauth2_full_example.go, breaking_changes.md step 8.
2026-10-01 14:42:12 +02:00
Hein f54b707040 feat(pgsql): add WhereGroup and a podman/docker hardening test
- PgSQLSelectQuery implements common.WhereGrouper so x-custom-sql-or
  is grouped with the client's own conditions on the pgx adapter too.
- Add a container test (opt-in via RESOLVESPEC_TEST_CONTAINERS=1) that
  starts PostgreSQL with podman or docker and checks the hardening
  against a real database: parenthesis escape, pg_sleep, catalog
  subquery, stacked statements, x-custom-sql-or grouping and the
  legacy behaviour when hardening is switched off.
2026-10-01 14:41:46 +02:00
Hein ca89cb8a73 fix(common): harden CORS, sort, raw-SQL WHERE and x-custom-sql-or
Add a `hardening` config section (RESOLVESPEC_HARDENING_*) so each
fix can be switched off to restore the previous behaviour:

- cors_strict_origins: only reflect origins listed in
  cors.allowed_origins / server URLs, with credentials; `*` never
  sends credentials; fix shared-slice append of expose headers.
- sort_strict: join aliases must match `alias.identifier` (empty alias
  no longer matches everything); sort expressions reject dangerous
  functions/catalogs; cql* columns must be identifier-safe.
- sql_strict: client raw-SQL fragments must have balanced parens and
  quotes, no comments/`;`/`$$`, DML keywords, dangerous functions or
  system catalogs; a rejected fragment now fails closed ("(1=0)")
  instead of dropping the filter. Subqueries stay allowed unless
  sql_block_subqueries is set.
- x-custom-sql-or is grouped together with the client's own
  conditions (new optional WhereGrouper, implemented for bun and gorm)
  so it can no longer OR past server-side filters.
2026-10-01 13:46:01 +02:00
Hein c1153522f2 docs(resolvemcp): rewrite README for meta tools, guard and limits; update plan status 2026-10-01 13:42:14 +02:00
Hein 155e04deea chore(resolvemcp): drop unused helpers, silence rangeValCopy 2026-10-01 13:40:21 +02:00
Hein e49c3a916e feat(resolvemcp): replace per-model tools with fixed meta tools, guarded filter writes and a function registry
Tools: list_tables, describe_table, select_table, insert_into_table, update_table,
delete_from_table, list_functions, call_function. Visibility follows the model rules.
Filter-based update/delete require filters (never dropped silently), cap the matched rows
(MaxWriteRows), support dry_run, and need a single-use confirm token bound to caller, table,
filters, data and the matched rows. RegisterFunction adds Go-callback and SQL-procedure
functions run in a transaction with BeforeCall/AfterCall hooks. Per-model tools and
resources are removed.

fix(pgsql): UPDATE with SET and a multi-placeholder WHERE renumbered the WHERE parameters
wrongly ($1, $2 became $3, $2); shift them in one pass.
2026-10-01 13:40:00 +02:00
Hein 276c3814d8 feat(resolvemcp): read/write limits, preload validation, query timeout, stable client error codes
Config gains DefaultLimit/MaxLimit/MaxOffset/MaxBatch/MaxPreloadDepth/MaxWriteRows/QueryTimeout/
ConfirmTTL. Reads are capped and the total COUNT is optional. Errors reach clients as
{code,message}; everything else is logged with a reference. Panics (handler and hooks) are
recovered without returning the panic value.
2026-10-01 13:35:00 +02:00
Hein 82f901a49c fix(resolvemcp): single transaction for create/update, hook registry mutex, uniform not-found, bounded SSE host cache 2026-10-01 13:33:11 +02:00
Hein ad2f54693f feat(resolvemcp): require authentication on MCP endpoints and enforce model rules on writes
Guard() rejects unauthenticated callers (no guest/optional mode); Setup*/New* helpers take a
SecurityList and have explicit *Unauthenticated variants. Model rules now reach the security
hooks, create checks CanCreate (security.CheckModelCreateAllowed), create/update validate keys
against the model's writable columns, update sets only given keys (NULL allowed), update and
delete go through row security via a new BeforeScan hook, and the annotation tool is opt-in
(Config.EnableAnnotations) and runs BeforeHandle.
2026-10-01 13:31:13 +02:00
Hein 7662d5055c test(security): seed expired key as UTC in direct auth test
Tests / Race Detector (push) Failing after 27s
Tests / Unit Tests (push) Failing after 29s
Tests / Integration Tests (push) Failing after 30s
Build , Vet Test, and Lint / Build (push) Successful in 1m14s
Build , Vet Test, and Lint / Lint Code (push) Successful in 1m32s
Build , Vet Test, and Lint / Run Vet Tests (1.23.x) (push) Successful in 1m42s
Build , Vet Test, and Lint / Run Vet Tests (1.24.x) (push) Successful in 1m40s
2026-10-01 13:24:38 +02:00
Hein ea6a2e705f test(security): add SQL Server container conformance test; bind timestamps as UTC in direct backend 2026-10-01 13:24:26 +02:00
Hein 2516fcb13d chore(security): apply golangci-lint fixes to lookup and security packages 2026-10-01 13:21:00 +02:00
Hein c9fa8c60f2 refactor(security): move all database access into pkg/security/lookup
pkg/security no longer contains SQL. Every provider calls a store interface
from lookup, implemented by a procedure backend (Postgres stored procedures,
the default there) and a direct backend (dialect-driven SQL for postgres,
sqlite, mysql and mssql with configurable table and column names).

- add sectypes, lookup, lookup/{dialect,procedure,direct,backends,ddl,conformance}
- split totp and providers sub packages out of the core package
- replace SQLNames/TableNames/QueryMode with lookup.Config (see breaking_changes.md)
- direct backend now covers column/row security and API-key login
- move txsettings SQL to lookup.ApplyTxSettings; remove password.go
- move schema scripts under lookup/, add reference DDL per dialect
- add a shared conformance suite; run it on sqlite, and on Postgres in a
  podman/docker container (RESOLVESPEC_TEST_CONTAINERS=1)
- fix procedure schema bugs found on real Postgres: duplicate p_data
  parameter, JSON null arrays, expires_at timezone casts, passkey list
  GROUP BY, missing resolvespec_passkey_login; accept zone-less timestamps
2026-10-01 13:19:44 +02:00
Hein 60bd0a6dd3 feat(security): add plan for pkg/security lookup sub package 2026-10-01 11:29:02 +02:00
Hein 982c90bfdd feat(websocketspec): fire BeforeDisconnect/AfterDisconnect hooks on close
Hooks fire from Connection.Close() once per connection. ConnectionManager
Shutdown now closes connections outside its lock so hooks can call back
into the manager. Update the single-transaction audit plan status.
2026-10-01 10:53:21 +02:00
Hein ae2b0a4ef4 fix(security): skip row security filter for insert queries
Insert queries have no Where clause and read no existing rows, so the
fail-closed check rejected every insert when a row security template
was defined.
2026-10-01 10:47:54 +02:00
Hein daeea241af fix(restheadspec): honour string URL ids on POST updates
POST with a non-numeric URL id (e.g. a string primary key) was treated as
having no id, so the body primary key was used to look up the existing row.
That broke primary key changes. Treat any non-empty, non-zero URL id as the
update target, and add an integration test through Handle for POST and PUT.
2026-10-01 10:23:23 +02:00
Hein 3bd3e46409 fix(restheadspec): handle primary key changes in updates
* Allow primary key changes when specified in the request body.
* Ensure correct record fetching after primary key updates.
* Add integration tests for primary key update scenarios.
2026-10-01 10:16:26 +02:00
Hein 247111c32e docs(README): update table of contents and feature descriptions 2026-10-01 09:46:17 +02:00
Hein Puth (Warkanum) ec8d4d2c77 Merge pull request #25 from bitechdev/fix/row-security-update-delete
Fix/row security update delete
2026-10-01 09:40:33 +02:00
Hein a3287f3b53 fix(security): reorder import statements for consistency 2026-10-01 09:33:57 +02:00
Hein 1e4a76643d fix(security): apply row security to update and delete queries
ApplyRowSecurity only accepted common.SelectQuery, so the BeforeScan hook
failed closed on update/delete when a row-security template existed.
Type-switch on SelectQuery, UpdateQuery and DeleteQuery; other types
still return an error.
2026-10-01 09:31:56 +02:00
Hein Puth (Warkanum) a81031b83d Merge pull request #23 from bitechdev/fix/db-connection-bursts
Tests / Race Detector (push) Failing after 23s
Tests / Unit Tests (push) Failing after 25s
Tests / Integration Tests (push) Failing after 26s
Build , Vet Test, and Lint / Build (push) Successful in 1m4s
Build , Vet Test, and Lint / Run Vet Tests (1.23.x) (push) Successful in 1m34s
Build , Vet Test, and Lint / Run Vet Tests (1.24.x) (push) Successful in 1m34s
Build , Vet Test, and Lint / Lint Code (push) Failing after 1m34s
Fix/db connection bursts
2026-09-30 23:53:17 +02:00
warkanum ac4cf9b4b6 Merge branch 'main' of github.com:bitechdev/ResolveSpec into fix/db-connection-bursts 2026-09-30 23:51:24 +02:00
warkanum b35399fdfa feat(security): exclude hidden/masked columns from create and update payloads 2026-09-30 23:48:09 +02:00
warkanum a65ca5f5ce fix: fire resolvespec AfterRead/AfterCreate/AfterDelete, key restheadspec total cache by record id 2026-09-30 23:40:32 +02:00
warkanum 5933637a88 docs(readme): update slogan placement in README 2026-09-30 23:35:08 +02:00
warkanum 7f84debdc5 test(tx): regression tests for per-request transactions across all specs 2026-09-30 23:19:58 +02:00
warkanum 2042205817 fix(pgsql): return subquery preload errors instead of logging and continuing 2026-09-30 23:11:30 +02:00
warkanum 6335bfe87e docs(readme): reference all pkg packages and clients 2026-09-30 23:04:56 +02:00
warkanum 129c1a043d docs(readme): document single transaction per request, OnTxBegin, test server 2026-09-30 23:03:50 +02:00
warkanum da0b1f5123 chore(testserver): host networking, ports 8123/8124, smoke read+update, dbtrace pooled=0 verified 2026-09-30 23:01:52 +02:00
warkanum ff76eb8e1f feat(security): stamp transaction-local settings on OnTxBegin in all specs 2026-09-30 22:55:26 +02:00
warkanum 3b93802a25 fix(tx): run restheadspec AfterRead in a second short transaction 2026-09-30 22:53:07 +02:00
warkanum 6b6f540ab0 feat(tx): run funcspec OnTxBegin and BeforeResponse in transactions 2026-09-30 22:50:41 +02:00
warkanum 4cbe4f597d feat(tx): run resolvemcp operations in per-operation transactions with OnTxBegin 2026-09-30 22:49:47 +02:00
warkanum ed457eb14a feat(tx): run websocketspec and mqttspec operations in per-message transactions
Reads and deletes run in one transaction; create and update write in one
and re-fetch plus After hooks in a second. OnTxBegin fires first in each.
2026-09-30 22:46:46 +02:00
warkanum 97bcb44fdc fix(clients): verify C# client, read Content-Range from content headers 2026-09-30 22:44:15 +02:00
warkanum 17bb6ea76d fix(clients): verify Dart client, case-insensitive Content-Range lookup 2026-09-30 22:42:55 +02:00
warkanum ce706bacda feat(tx): run update re-fetch and post-commit hooks in a second transaction
Re-fetch, BeforeScan and AfterUpdate/AfterCreate now run on a short
transaction that fires OnTxBegin. Existence selects inside the first
transaction use tx instead of the pool.
2026-09-30 22:42:33 +02:00
warkanum eb492d52aa feat(clients): add Go, Rust, C# and Dart clients for ResolveSpec and FunctionSpec 2026-09-30 22:40:33 +02:00
warkanum f2dbe2561c feat(hooks): add OnTxBegin and runInTx for resolvespec and restheadspec
Every transaction the handlers open now fires OnTxBegin first, with the
transaction in hookCtx.Tx, via common.RunRequestTx.
2026-09-30 22:40:06 +02:00
warkanum cd96404cdd fix(delete): run delete hooks and queries in one transaction
- resolvespec/restheadspec: single and batch delete use one transaction
- add sqlmock tests for delete transaction behaviour
- testmodels: serial integer ids; update tests accordingly
- add compose testserver, smoke script, podman-first Makefile targets
2026-09-30 22:33:25 +02:00
warkanum b2b815552f refactor: move JS and Python clients under clients/ 2026-09-30 22:31:43 +02:00
warkanum f6a9daa89e docs(audit): add plan for single transaction per request 2026-09-30 22:18:24 +02:00
warkanum 54e6a3b17c feat(resolvespec-python): add Python client for ResolveSpec, HeaderSpec, FunctionSpec and WebSocketSpec 2026-09-30 22:18:01 +02:00
warkanum ab3d2b5b04 docs(audit): add funcspec server-side audit 2026-09-30 22:18:01 +02:00
Hein Puth (Warkanum) 9c4d916490 Merge pull request #22 from bitechdev/fix/db-connection-bursts
Tests / Unit Tests (push) Failing after 28s
Tests / Race Detector (push) Failing after 29s
Tests / Integration Tests (push) Failing after 28s
Build , Vet Test, and Lint / Build (push) Successful in 1m7s
Build , Vet Test, and Lint / Run Vet Tests (1.24.x) (push) Successful in 1m38s
Build , Vet Test, and Lint / Lint Code (push) Failing after 1m39s
Build , Vet Test, and Lint / Run Vet Tests (1.23.x) (push) Successful in 1m39s
Fix/db connection bursts
2026-09-30 21:45:20 +02:00
warkanum 3e327d0c78 fix(db): reduce per-request connection bursts and add dbtrace
* Throttle async session-activity writes to once per token per minute
* Add singleflight to session lookups, keystore validation and
  column/row security loads to stop cold-cache stampedes
* Preload security rules in BeforeHandle (restheadspec, resolvespec) so
  they no longer need a second connection while the read tx is open
* Add pkg/dbtrace: opt-in per-request DB call counting and pool logging
  (db_trace.* config, RESOLVESPEC_DB_TRACE_* env), wired into testserver
* Add tests for load dedup, activity throttle and dbtrace
2026-09-30 21:44:28 +02:00
warkanum 62cc14c02a feat(resolvespec-js): support extended restheadspec headers
Add HeaderSpecOptions, vector_search, X-Preload-Where, X-Expand,
custom SQL joins/or, spatial/vector filters, response format, flags
and X-Files to buildHeaders, types and README.
2026-09-30 21:37:41 +02:00
Hein 4c5dffc3d1 test(mqttspec): add tests for update behavior with empty strings 2026-09-30 17:15:55 +02:00
Hein 2898b335f8 feat(db): add ApplicationName to connection configuration
* Introduced ApplicationName field to identify clients in DSN
* Set default ApplicationName to "ResolveSpec"
* Updated tests for ApplicationName handling in DSN
2026-09-30 17:12:04 +02:00
Hein 20ba8ed112 feat(update): allow clearing values with "" and null on update
Update handlers skipped empty strings and nulls, so a client could not
blank or null out a column. Every key present in the payload now
overwrites the stored value, including "" and null.

- add common.MergeUpdateValues and use it in resolvespec and restheadspec
- add Handler.SetDisallowNulls to skip null values (""still overwrites)
- websocketspec and mqttspec now write only the keys present in the
  payload via SetMap instead of updating the whole zeroed model, which
  clobbered absent fields
2026-09-30 17:03:42 +02:00
Hein 1214f69e0c feat(middleware): export clientqueue_enqueued_total counter
Counts every request placed in a wait queue regardless of outcome, so the
total ever queued no longer has to be summed from queued, timeout and
canceled.
2026-09-30 16:16:25 +02:00
Hein 89a58ab3a0 feat(middleware): export clientqueue_waiting_clients gauge
Counts clients with at least one queued request, updated whenever a
client's waiter count crosses zero (enqueue, grant, timeout, cancel).
2026-09-30 16:14:27 +02:00
Hein 48081b4aa4 docs: document client queue and dbmanager pool stats 2026-09-30 15:46:43 +02:00
Hein 467dbd66c8 feat(middleware): add per-client FIFO request queue with burst metrics
ClientQueue limits concurrent requests per client (X-Client-Id, then
Authorization, session, IP) and queues the rest first-in-first-out, with
bounded depth, max wait and idle eviction. Chain composes it with auth for
the Setup*Routes middleware slot. Exports burst, wait and depth metrics.
The test server uses it with a limit of 10.
2026-09-30 15:46:42 +02:00
Hein 178d40587d feat(dbmanager): report pool max size and total connections opened
Add MaxOpenConnections to provider and connection stats, exposed as the
max state of dbmanager_connection_pool_size. Count physical dials through a
counting driver connector and export dbmanager_connections_opened_total;
existing_db pools derive an approximate count from sql.DBStats.
2026-09-30 15:46:41 +02:00
346 changed files with 37990 additions and 8717 deletions
+1 -1
View File
@@ -28,5 +28,5 @@ test.db
/testserver
tests/data/
node_modules/
resolvespec-js/dist/
clients/resolvespec-js/dist/
.codex
+18 -4
View File
@@ -1,4 +1,7 @@
.PHONY: test test-unit test-race test-integration docker-up docker-down clean
# Container compose command: podman if installed, else docker
COMPOSE ?= $(shell command -v podman >/dev/null 2>&1 && echo "podman compose" || echo "docker compose")
.PHONY: testserver-up testserver-down testserver-smoke test test-unit test-race test-integration docker-up docker-down clean
GOLANGCI_LINT := $(shell go env GOPATH)/bin/golangci-lint
@@ -82,7 +85,7 @@ lintfix: ## Run linter
# Start PostgreSQL for integration tests
docker-up:
@echo "Starting PostgreSQL container..."
@podman compose up -d postgres-test
@$(COMPOSE) up -d postgres-test
@echo "Waiting for PostgreSQL to be ready..."
@sleep 5
@echo "PostgreSQL is ready!"
@@ -90,12 +93,23 @@ docker-up:
# Stop PostgreSQL container
docker-down:
@echo "Stopping PostgreSQL container..."
@podman compose down
@$(COMPOSE) down
# Test server + PostgreSQL in containers (dbtrace enabled)
testserver-up:
@$(COMPOSE) up -d --build postgres-test testserver
testserver-down:
@$(COMPOSE) down
testserver-smoke:
@COMPOSE="$(COMPOSE)" scripts/testserver-smoke.sh
# Clean up Docker volumes and test data
clean:
@echo "Cleaning up..."
@podman compose down -v
@$(COMPOSE) down -v
@echo "Cleanup complete!"
# Run integration tests with Docker (full workflow)
+327 -184
View File
@@ -13,70 +13,70 @@ ResolveSpec is a flexible and powerful REST API specification and implementation
All share the same core architecture and provide dynamic data querying, relationship preloading, and complex filtering.
![1.00](./generated_slogan.webp)
## Table of Contents
* [Features](#features)
* [Installation](#installation)
* [Quick Start](#quick-start)
* [ResolveSpec (Body-Based API)](#resolvespec---body-based-api)
* [RestHeadSpec (Header-Based API)](#restheadspec---header-based-api)
* [ResolveMCP (MCP Server)](#resolvemcp---mcp-server)
* [Architecture](#architecture)
* [API Structure](#api-structure)
* [RestHeadSpec Overview](#restheadspec-header-based-api)
* [Example Usage](#example-usage)
* [Testing](#testing)
* [Additional Packages](#additional-packages)
* [Security Considerations](#security-considerations)
* [What's New](#whats-new)
- [Features](#features)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [ResolveSpec (Body-Based API)](#resolvespec---body-based-api)
- [RestHeadSpec (Header-Based API)](#restheadspec---header-based-api)
- [ResolveMCP (MCP Server)](#resolvemcp---mcp-server)
- [Architecture](#architecture)
- [API Structure](#api-structure)
- [RestHeadSpec Overview](#restheadspec-header-based-api)
- [Example Usage](#example-usage)
- [Testing](#testing)
- [Additional Packages](#additional-packages)
- [Security Considerations](#security-considerations)
- [Breaking Changes](#breaking-changes)
- [What's New](#whats-new)
## Features
### Core Features
* **Dynamic Data Querying**: Select specific columns and relationships to return
* **Relationship Preloading**: Load related entities with custom column selection and filters
* **Complex Filtering**: Apply multiple filters with various operators
* **Sorting**: Multi-column sort support
* **Pagination**: Built-in limit/offset and cursor-based pagination (both ResolveSpec and RestHeadSpec)
* **Computed Columns**: Define virtual columns for complex calculations
* **Custom Operators**: Add custom SQL conditions when needed
* **🆕 Recursive CRUD Handler**: Automatically handle nested object graphs with foreign key resolution and per-record operation control via `_request` field
- **Dynamic Data Querying**: Select specific columns and relationships to return
- **Relationship Preloading**: Load related entities with custom column selection and filters
- **Complex Filtering**: Apply multiple filters with various operators
- **Sorting**: Multi-column sort support
- **Pagination**: Built-in limit/offset and cursor-based pagination (both ResolveSpec and RestHeadSpec)
- **Computed Columns**: Define virtual columns for complex calculations
- **Custom Operators**: Add custom SQL conditions when needed
- **🆕 One Transaction Per Request**: Every statement and DB-touching hook of a request runs on one transaction; `OnTxBegin` hook stamps transaction-local settings (RLS) first. See [pkg/common/TRANSACTIONS.md](pkg/common/TRANSACTIONS.md)
- **🆕 Recursive CRUD Handler**: Automatically handle nested object graphs with foreign key resolution and per-record operation control via `_request` field
### Architecture (v2.0+)
* **🆕 Database Agnostic**: Works with GORM, Bun, or any database layer through adapters
* **🆕 Router Flexible**: Integrates with Gorilla Mux, Gin, Echo, or custom routers
* **🆕 Backward Compatible**: Existing code works without changes
* **🆕 Better Testing**: Mockable interfaces for easy unit testing
- **🆕 Database Agnostic**: Works with GORM, Bun, or any database layer through adapters
- **🆕 Router Flexible**: Integrates with Gorilla Mux, Gin, Echo, or custom routers
- **🆕 Backward Compatible**: Existing code works without changes
- **🆕 Better Testing**: Mockable interfaces for easy unit testing
### ResolveMCP (v3.2+)
* **🆕 MCP Server**: Expose any registered database model as Model Context Protocol tools and resources
* **🆕 AI-Ready Descriptions**: Tool descriptions include the full column schema, primary key, nullable flags, and relations — giving AI models everything they need to query correctly without guessing
* **🆕 Four Tools Per Model**: `read_`, `create_`, `update_`, `delete_` tools auto-registered per model
* **🆕 Full Query Support**: Filters, sort, limit/offset, cursor pagination, column selection, and relation preloading all available as tool parameters
* **🆕 HTTP/SSE Transport**: Standards-compliant SSE transport for use with Claude Desktop, Cursor, and any MCP-compatible client
* **🆕 Lifecycle Hooks**: Same Before/After hook system as ResolveSpec for auth and side-effects
- **🆕 MCP Server**: Expose any registered database model as Model Context Protocol tools and resources
- **🆕 AI-Ready Descriptions**: Tool descriptions include the full column schema, primary key, nullable flags, and relations — giving AI models everything they need to query correctly without guessing
- **🆕 Four Tools Per Model**: `read_`, `create_`, `update_`, `delete_` tools auto-registered per model
- **🆕 Full Query Support**: Filters, sort, limit/offset, cursor pagination, column selection, and relation preloading all available as tool parameters
- **🆕 HTTP/SSE Transport**: Standards-compliant SSE transport for use with Claude Desktop, Cursor, and any MCP-compatible client
- **🆕 Lifecycle Hooks**: Same Before/After hook system as ResolveSpec for auth and side-effects
### RestHeadSpec (v2.1+)
* **🆕 Header-Based API**: All query options passed via HTTP headers instead of request body
* **🆕 Lifecycle Hooks**: Before/after hooks for create, read, update, and delete operations
* **🆕 Cursor Pagination**: Efficient cursor-based pagination with complex sort support
* **🆕 Multiple Response Formats**: Simple, detailed, and Syncfusion-compatible formats
* **🆕 Single Record as Object**: Automatically normalize single-element arrays to objects (enabled by default)
* **🆕 Advanced Filtering**: Field filters, search operators, AND/OR logic, and custom SQL
* **🆕 Base64 Encoding**: Support for base64-encoded header values
- **🆕 Header-Based API**: All query options passed via HTTP headers instead of request body
- **🆕 Lifecycle Hooks**: Before/after hooks for create, read, update, and delete operations
- **🆕 Cursor Pagination**: Efficient cursor-based pagination with complex sort support
- **🆕 Multiple Response Formats**: Simple, detailed, and Syncfusion-compatible formats
- **🆕 Single Record as Object**: Automatically normalize single-element arrays to objects (enabled by default)
- **🆕 Advanced Filtering**: Field filters, search operators, AND/OR logic, and custom SQL
- **🆕 Base64 Encoding**: Support for base64-encoded header values
### Routing & CORS (v3.0+)
* **🆕 Explicit Route Registration**: Routes created per registered model instead of dynamic lookups
* **🆕 OPTIONS Method Support**: Full OPTIONS method support returning model metadata
* **🆕 CORS Headers**: Comprehensive CORS support with all HeadSpec headers allowed
* **🆕 Better Route Control**: Customize routes per model with more flexibility
- **🆕 Explicit Route Registration**: Routes created per registered model instead of dynamic lookups
- **🆕 OPTIONS Method Support**: Full OPTIONS method support returning model metadata
- **🆕 CORS Headers**: Comprehensive CORS support with all HeadSpec headers allowed
- **🆕 Better Route Control**: Customize routes per model with more flexibility
## API Structure
@@ -130,7 +130,6 @@ X-DetailApi: true
For complete documentation including setup, headers, lifecycle hooks, cursor pagination, and more, see [pkg/restheadspec/README.md](pkg/restheadspec/README.md).
## Example Usage
For detailed examples of reading data, cursor pagination, recursive CRUD operations, filtering, sorting, and more, see [pkg/resolvespec/README.md](pkg/resolvespec/README.md).
@@ -141,14 +140,14 @@ First-class support for PostGIS geometry/geography and pgvector columns in `reso
### Column types (`pkg/spectypes`)
| Go type | SQL type | Wire / JSON |
|--------------------|--------------|--------------------------------------------------------|
| `SqlGeometry` | `geometry` | JSON in/out = **GeoJSON**; also accepts EWKT / hex-EWKB |
| `SqlGeography` | `geography` | same as `SqlGeometry` |
| `SqlVector` | `vector` | `[]float32` ⇄ `[1,2,3]` |
| `SqlHalfVector` | `halfvec` | `[]float32` ⇄ `[1,2,3]` |
| `SqlSparseVector` | `sparsevec` | `{"dim":8,"indices":[1,4],"values":[0.5,0.2]}` |
| `SqlBitVector` | `bit`/`varbit` | bool array or `"1011"` string |
| Go type | SQL type | Wire / JSON |
| ----------------- | -------------- | ------------------------------------------------------- |
| `SqlGeometry` | `geometry` | JSON in/out = **GeoJSON**; also accepts EWKT / hex-EWKB |
| `SqlGeography` | `geography` | same as `SqlGeometry` |
| `SqlVector` | `vector` | `[]float32` ⇄ `[1,2,3]` |
| `SqlHalfVector` | `halfvec` | `[]float32` ⇄ `[1,2,3]` |
| `SqlSparseVector` | `sparsevec` | `{"dim":8,"indices":[1,4],"values":[0.5,0.2]}` |
| `SqlBitVector` | `bit`/`varbit` | bool array or `"1011"` string |
- Geometry `Value()` emits `SRID=<n>;<WKT>` (PostGIS implicit text→geometry cast; no wrapper function needed).
- Declare dimensioned types with a tag: `gorm:"type:vector(1536)"` — the tag wins over the canonical name in metadata/OpenAPI.
@@ -158,32 +157,36 @@ First-class support for PostGIS geometry/geography and pgvector columns in `reso
`value` is a geometry (GeoJSON object, EWKT string, or hex-EWKB) unless noted.
| Operator | Value shape |
|----------|-------------|
| `st_intersects`, `st_contains`, `st_within`, `st_covers`, `st_coveredby`, `st_overlaps`, `st_touches`, `st_crosses`, `st_equals`, `st_disjoint` | geometry |
| `st_dwithin` | `{"geom": <geometry>, "distance": <meters>}` |
| `bbox` (alias `&&`) | geometry, or `{"bbox":[minx,miny,maxx,maxy],"srid":4326}` |
| Operator | Value shape |
| ----------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- |
| `st_intersects`, `st_contains`, `st_within`, `st_covers`, `st_coveredby`, `st_overlaps`, `st_touches`, `st_crosses`, `st_equals`, `st_disjoint` | geometry |
| `st_dwithin` | `{"geom": <geometry>, "distance": <meters>}` |
| `bbox` (alias `&&`) | geometry, or `{"bbox":[minx,miny,maxx,maxy],"srid":4326}` |
### Vector similarity filter operators
| Operator | pgvector op | Value shape |
|----------|-------------|-------------|
| `l2_within` / `euclidean_within` | `<->` | `{"vector":[...], "distance": <n>}` |
| `cosine_within` | `<=>` | same (also `"lt"`/`"lte"`/`"gt"`/`"gte"` instead of `"distance"`) |
| `ip_within` / `inner_within` | `<#>` | same |
| Operator | pgvector op | Value shape |
| -------------------------------- | ----------- | ----------------------------------------------------------------- |
| `l2_within` / `euclidean_within` | `<->` | `{"vector":[...], "distance": <n>}` |
| `cosine_within` | `<=>` | same (also `"lt"`/`"lte"`/`"gt"`/`"gte"` instead of `"distance"`) |
| `ip_within` / `inner_within` | `<#>` | same |
### KNN search (ordering + distance column)
**resolvespec** — `options.vector_search`:
```json
{ "options": { "vector_search": {
"column": "embedding",
"vector": [0.1, 0.2, 0.3],
"metric": "cosine",
"as": "_distance",
"direction": "asc"
}}}
{
"options": {
"vector_search": {
"column": "embedding",
"vector": [0.1, 0.2, 0.3],
"metric": "cosine",
"as": "_distance",
"direction": "asc"
}
}
}
```
Orders rows by distance; when `as` is set, returns the distance as an extra column (all model columns are auto-selected).
@@ -275,32 +278,28 @@ ResolveMCP exposes registered models as Model Context Protocol tools so AI model
```go
import "github.com/bitechdev/ResolveSpec/pkg/resolvemcp"
// Create handler
handler := resolvemcp.NewHandlerWithGORM(db)
handler := resolvemcp.NewHandlerWithGORM(db, resolvemcp.Config{BaseURL: "http://localhost:8080", BasePath: "/mcp"})
securityList, _ := security.NewSecurityList(provider)
resolvemcp.RegisterSecurityHooks(handler, securityList)
// Register models — must be done BEFORE Build()
handler.RegisterModel("public", "users", &User{})
handler.RegisterModel("public", "posts", &Post{})
// Finalize: registers MCP tools and resources
handler.Build()
// Mount SSE transport on your existing router
// Mount the guarded SSE transport (OAuth bearer, session token or API key required)
router := mux.NewRouter()
resolvemcp.SetupMuxRoutes(router, handler, "http://localhost:8080")
resolvemcp.SetupMuxRoutes(router, handler, securityList)
// MCP clients connect to:
// SSE stream: GET http://localhost:8080/mcp/sse
// Messages: POST http://localhost:8080/mcp/message
//
// Auto-registered tools per model:
// read_public_users — filter, sort, paginate, preload
// create_public_users — insert a new record
// update_public_users — update a record by ID
// delete_public_users — delete a record by ID
// Fixed meta tools (independent of the number of models):
// list_tables, describe_table, select_table, insert_into_table,
// update_table, delete_from_table, list_functions, call_function
```
For complete documentation, see [pkg/resolvemcp/README.md](pkg/resolvemcp/README.md) (if present) or the package source.
For complete documentation, see [pkg/resolvemcp/README.md](pkg/resolvemcp/README.md) .
## Architecture
@@ -342,25 +341,25 @@ Your Application Code
### Supported Database Layers
* **GORM** - Full support for PostgreSQL, SQLite, MSSQL
* **Bun** - Full support for PostgreSQL, SQLite, MSSQL
* **Native SQL** - Standard library `*sql.DB` with all supported databases
* **Custom ORMs** - Implement the `Database` interface
- **GORM** - Full support for PostgreSQL, SQLite, MSSQL
- **Bun** - Full support for PostgreSQL, SQLite, MSSQL
- **Native SQL** - Standard library `*sql.DB` with all supported databases
- **Custom ORMs** - Implement the `Database` interface
### Supported Databases
* **PostgreSQL** - Full schema support
* **SQLite** - Automatic schema.table to schema_table translation
* **Microsoft SQL Server** - Full schema support
* **MongoDB** - NoSQL document database (via MQTTSpec and custom handlers)
- **PostgreSQL** - Full schema support
- **SQLite** - Automatic schema.table to schema_table translation
- **Microsoft SQL Server** - Full schema support
- **MongoDB** - NoSQL document database (via MQTTSpec and custom handlers)
### Supported Routers
* **Gorilla Mux** (built-in support with `SetupRoutes()`)
* **BunRouter** (built-in support with `SetupBunRouterWithResolveSpec()`)
* **Gin** (manual integration, see examples above)
* **Echo** (manual integration, see examples above)
* **Custom Routers** (implement request/response adapters)
- **Gorilla Mux** (built-in support with `SetupRoutes()`)
- **BunRouter** (built-in support with `SetupBunRouterWithResolveSpec()`)
- **Gin** (manual integration, see examples above)
- **Echo** (manual integration, see examples above)
- **Custom Routers** (implement request/response adapters)
## Testing
@@ -370,6 +369,14 @@ ResolveSpec is designed for testability with mockable interfaces. For testing ex
- [RestHeadSpec Testing](pkg/restheadspec/README.md#testing)
- [WebSocketSpec Testing](pkg/websocketspec/README.md)
### Test Server (dbtrace, real PostgreSQL)
- `make testserver-up` / `make testserver-down`: testserver + PostgreSQL via compose (host networking)
- `make testserver-smoke`: create, read, update, delete, batch create/delete against the testserver
- Ports: testserver `8123`, PostgreSQL `8124`
- `dbtrace` logs per request `tx`, `tx_queries`, `pooled`, `raw`; `pooled=0` is the target
- Integration tests default to PostgreSQL on `localhost:8124`
## Continuous Integration
ResolveSpec uses GitHub Actions for automated testing and quality checks. The CI pipeline runs on every push and pull request.
@@ -378,10 +385,10 @@ ResolveSpec uses GitHub Actions for automated testing and quality checks. The CI
The project includes automated workflows that:
* **Test**: Run all tests with race detection and code coverage
* **Lint**: Check code quality with golangci-lint
* **Build**: Verify the project builds successfully
* **Multi-version**: Test against multiple Go versions (1.23.x, 1.24.x)
- **Test**: Run all tests with race detection and code coverage
- **Lint**: Check code quality with golangci-lint
- **Build**: Verify the project builds successfully
- **Multi-version**: Test against multiple Go versions (1.23.x, 1.24.x)
### Running Tests Locally
@@ -403,9 +410,9 @@ golangci-lint run
The project includes comprehensive test coverage:
* **Unit Tests**: Individual component testing
* **Integration Tests**: End-to-end API testing
* **CRUD Tests**: Standalone tests for both ResolveSpec and RestHeadSpec APIs
- **Unit Tests**: Individual component testing
- **Integration Tests**: End-to-end API testing
- **CRUD Tests**: Standalone tests for both ResolveSpec and RestHeadSpec APIs
To run only the CRUD standalone tests:
@@ -436,6 +443,7 @@ ResolveSpec includes several complementary packages that work together to provid
The core body-based REST API with GraphQL-like capabilities.
**Key Features**:
- JSON request body with operation and options
- Recursive CRUD with nested object support
- Cursor and offset pagination
@@ -449,6 +457,7 @@ For complete documentation, see [pkg/resolvespec/README.md](pkg/resolvespec/READ
Alternative REST API where query options are passed via HTTP headers.
**Key Features**:
- All query options via HTTP headers
- Same capabilities as ResolveSpec
- Cleaner separation of data and metadata
@@ -461,6 +470,7 @@ For complete documentation, see [pkg/restheadspec/README.md](pkg/restheadspec/RE
Expose any registered model as Model Context Protocol tools and resources consumable by AI models over HTTP/SSE.
**Key Features**:
- Four tools per model: `read_`, `create_`, `update_`, `delete_`
- Rich AI-readable descriptions: column names, types, primary key, nullable flags, and preloadable relations
- Full query support: filters, sort, limit/offset, cursor pagination, column selection, preloads
@@ -474,6 +484,7 @@ For complete documentation, see [pkg/resolvemcp/](pkg/resolvemcp/).
Execute SQL functions and queries through a simple HTTP API with header-based parameters.
**Key Features**:
- Direct SQL function invocation
- Header-based parameter passing
- Automatic pagination and counting
@@ -482,16 +493,30 @@ Execute SQL functions and queries through a simple HTTP API with header-based pa
For complete documentation, see [pkg/funcspec/](pkg/funcspec/).
#### Clients
All clients are under [clients/](clients/README.md); wire behaviour is identical across them.
| Client | Language | Specs | Docs |
| -------------------- | -------------- | ---------------------------------------------------- | ---------------------------------------------- |
| `resolvespec-js` | TypeScript | ResolveSpec, HeaderSpec, FunctionSpec, WebSocketSpec | [README](clients/resolvespec-js/README.md) |
| `resolvespec-python` | Python >= 3.11 | ResolveSpec, HeaderSpec, FunctionSpec, WebSocketSpec | [README](clients/resolvespec-python/README.md) |
| `resolvespec-go` | Go | ResolveSpec, FunctionSpec | [README](clients/resolvespec-go/README.md) |
| `resolvespec-rs` | Rust | ResolveSpec, FunctionSpec | [README](clients/resolvespec-rs/README.md) |
| `resolvespec-cs` | C# (.NET 8) | ResolveSpec, FunctionSpec | [README](clients/resolvespec-cs/README.md) |
| `resolvespec-dart` | Dart / Flutter | ResolveSpec, FunctionSpec | [README](clients/resolvespec-dart/README.md) |
#### ResolveSpec JS - TypeScript Client Library
TypeScript/JavaScript client library supporting all three REST and WebSocket protocols.
**Clients**:
- Body-based REST client (`read`, `create`, `update`, `deleteEntity`)
- Header-based REST client (`HeaderSpecClient`)
- WebSocket client (`WebSocketClient`) with CRUD, subscriptions, heartbeat, reconnect
For complete documentation, see [resolvespec-js/README.md](resolvespec-js/README.md).
For complete documentation, see [clients/resolvespec-js/README.md](clients/resolvespec-js/README.md).
### Real-Time Communication
@@ -500,6 +525,7 @@ For complete documentation, see [resolvespec-js/README.md](resolvespec-js/README
Real-time bidirectional communication with full CRUD operations and subscriptions.
**Key Features**:
- Persistent WebSocket connections
- Real-time subscriptions to entity changes
- Automatic push notifications
@@ -513,6 +539,7 @@ For complete documentation, see [pkg/websocketspec/README.md](pkg/websocketspec/
MQTT-based database operations ideal for IoT and mobile applications.
**Key Features**:
- Embedded or external MQTT broker support
- QoS 1 (at-least-once delivery)
- Real-time subscriptions
@@ -528,6 +555,7 @@ For complete documentation, see [pkg/mqttspec/README.md](pkg/mqttspec/README.md)
Flexible, interface-driven static file server.
**Key Features**:
- Router-agnostic with standard `http.Handler`
- Multiple filesystem backends (local, zip, embedded)
- Pluggable cache, MIME, and fallback policies
@@ -535,6 +563,7 @@ Flexible, interface-driven static file server.
- 140+ MIME types including modern formats
**Quick Example**:
```go
import "github.com/bitechdev/ResolveSpec/pkg/server/staticweb"
@@ -559,6 +588,7 @@ For complete documentation, see [pkg/server/staticweb/README.md](pkg/server/stat
Comprehensive event handling system for real-time event publishing and cross-instance communication.
**Key Features**:
- Multiple event sources (database, websockets, frontend, system)
- Multiple providers (in-memory, Redis Streams, NATS, PostgreSQL)
- Pattern-based subscriptions
@@ -573,13 +603,14 @@ For complete documentation, see [pkg/eventbroker/README.md](pkg/eventbroker/READ
Centralized management of multiple database connections with support for PostgreSQL, SQLite, MSSQL, and MongoDB.
**Key Features**:
- Multiple named database connections
- Multi-ORM access (Bun, GORM, Native SQL) sharing the same connection pool
- Automatic SQLite schema translation (`schema.table` → `schema_table`)
- Background health checks (report status; they never close the pool)
- Prometheus metrics for monitoring
- Configuration-driven via YAML
- Per-connection statistics and management
- Per-connection statistics and management, including pool limit (`max`) and total connections ever opened (`dbmanager_connection_pool_size{state="max"}`, `dbmanager_connections_opened_total`)
**How to use it correctly**:
@@ -612,13 +643,23 @@ For documentation, see [pkg/cache/README.md](pkg/cache/README.md).
#### Security
Authentication and authorization framework with hooks integration. Database-backed providers use PostgreSQL stored procedures by default, with a portable Direct mode (plain Go/SQL) for SQLite, MySQL, or Postgres without the procedures installed.
Authentication and authorization framework with hooks integration. Database-backed providers use PostgreSQL stored procedures by default, with a direct SQL backend (SQLite, MySQL, SQL Server, or Postgres without the procedures) selected through `lookup.Config`.
For documentation, see [pkg/security/README.md](pkg/security/README.md) (see "Direct Mode" for the SQLite/portable-SQL path).
For documentation, see [pkg/security/README.md](pkg/security/README.md) (see "Database access (lookup)" for the SQLite/portable-SQL path).
It includes a standards-based OAuth 2.1 / OpenID Connect authorization server (consent, rotating refresh tokens, JWT access tokens, DPoP, PAR, device grant, token exchange, logout) and an OIDC relying-party client; see [pkg/security/OAUTH2_SERVER.md](pkg/security/OAUTH2_SERVER.md).
#### Middleware
HTTP middleware collection for common tasks (CORS, logging, metrics, etc.).
HTTP middleware collection for common tasks (CORS, logging, metrics, rate limiting, etc.).
**Client request queue** (`middleware.ClientQueue`): limits how many requests each client runs concurrently and queues the rest first-in-first-out, smoothing bursts such as a page load that fires ~15 requests at once. Clients are identified by `X-Client-Id`, then `Authorization`, then the built-in session, then IP, so no client changes are required. Exposes burst, wait and queue-depth Prometheus metrics. Add it through the middleware slot of `SetupMuxRoutes` / `SetupBunRouterRoutes`:
```go
q := middleware.NewClientQueue(middleware.ClientQueueConfig{MaxConcurrent: 10})
defer q.Close()
restheadspec.SetupMuxRoutes(router, handler, middleware.Chain(authMiddleware, q.Middleware))
```
For documentation, see [pkg/middleware/README.md](pkg/middleware/README.md).
@@ -652,14 +693,31 @@ Configuration management with support for multiple formats and environments.
For documentation, see [pkg/config/README.md](pkg/config/README.md).
#### DB Trace
Per-request DB call counting (`tx`, `tx_queries`, `pooled`, `raw`) and pool logging. Off by default.
For documentation, see [pkg/dbtrace/README.md](pkg/dbtrace/README.md).
### Core Libraries
| Package | Purpose |
| ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`pkg/common`](pkg/common/) | Shared interfaces (database, request/response adapters), validation, recursive CRUD, request transactions ([TRANSACTIONS.md](pkg/common/TRANSACTIONS.md)) |
| [`pkg/modelregistry`](pkg/modelregistry/) | Model registration by schema/entity and per-model access rules |
| [`pkg/reflection`](pkg/reflection/) | Model/struct reflection helpers (primary keys, columns, relations) |
| [`pkg/spectypes`](pkg/spectypes/) | SQL-aware types (nullable, JSONB, PostGIS, vector) |
| [`pkg/logger`](pkg/logger/) | Logging used by all packages |
| [`pkg/testmodels`](pkg/testmodels/) | Shared test models and data for tests and the testserver |
## Security Considerations
* Implement proper authentication and authorization
* Validate all input parameters
* Use prepared statements (handled by GORM/Bun/your ORM)
* Implement rate limiting
* Control access at schema/entity level
* **New**: Database abstraction layer provides additional security through interface boundaries
- Implement proper authentication and authorization
- Validate all input parameters
- Use prepared statements (handled by GORM/Bun/your ORM)
- Implement rate limiting (`middleware.RateLimiter`) and per-client request queueing (`middleware.ClientQueue`)
- Control access at schema/entity level
- **New**: Database abstraction layer provides additional security through interface boundaries
## Contributing
@@ -673,124 +731,209 @@ For documentation, see [pkg/config/README.md](pkg/config/README.md).
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
## Breaking Changes
Changes from 2026-09-30 to 2026-10-01 that require action when upgrading. The full `pkg/security` migration tables are in [pkg/security/breaking_changes.md](pkg/security/breaking_changes.md).
### Security package (`pkg/security`)
- **No SQL in `pkg/security`**: all database access moved to `pkg/security/lookup`. Removed `SQLNames`, `TableNames`, `KeyStoreSQLNames`, `KeyStoreTableNames`, `QueryMode` (`ModeAuto`/`ModeProcedure`/`ModeDirect`), `ErrDirectModeUnsupported` and the `SQLNames`/`TableNames`/`QueryMode` option fields and `WithQueryMode`/`WithTableNames` builders. Use `lookup.Config` (`Dialect`, `Mode`, `Overrides`, `Procs`, `Schema`) via the `Lookup` option field or `WithLookup`/`WithLookupProvider`. The variadic `names ...*SQLNames` argument was dropped from `NewJWTAuthenticator`, `NewDatabaseColumnSecurityProvider`, `NewDatabaseRowSecurityProvider` and `NewDatabaseTwoFactorProvider`.
- **Default lookup mode is per dialect**: stored procedures on Postgres, direct SQL elsewhere. `ModeAuto` is opt-in.
- **Packages moved (no aliases)**: TOTP types to `pkg/security/totp`; header, config key store and config column/row providers to `pkg/security/providers`.
- **SQL schema files moved** from `pkg/security/` to `pkg/security/lookup/`.
- **Password handling**: bcrypt is verified in Direct mode and in the shipped procedures; passwords are hashed on register and reset; client-supplied roles/level are ignored at registration; legacy cleartext passwords need an explicit opt-in to upgrade. Row security templates bind the user as a parameter, and a filter that cannot be attached now fails the request.
- **OAuth2 / OIDC server**: existing databases need new columns (`oauth_clients.metadata`, `oauth_codes.extra`) and new tables (`oauth_consents`, `oauth_refresh_tokens`, `oauth_device_codes`, `oauth_par_requests`, `oauth_jti`); Postgres procedure mode needs the schema reapplied. `/oauth/introspect` and `/oauth/revoke` now require client authentication (`AllowAnonymousIntrospection` restores the old behaviour). Only PKCE `S256` is accepted. Authorization errors redirect to the client after `redirect_uri` validation.
- **Row security** now applies to update and delete queries (skipped for inserts); hidden/masked columns are excluded from create and update payloads.
### Request hardening (`pkg/common`)
Enabled by default under the new `hardening` config section (`RESOLVESPEC_HARDENING_*`); each can be switched off to restore the old behaviour.
- **`cors_strict_origins`**: only origins listed in `cors.allowed_origins` or the server URLs are reflected; `*` never sends credentials.
- **`sort_strict`**: sort expressions and join aliases must be identifier-safe; dangerous functions and catalogs are rejected.
- **`sql_strict`**: client raw-SQL fragments with unbalanced quotes/parens, comments, `;`, DML keywords or system catalogs are rejected, and a rejected fragment now fails closed (`1=0`) instead of dropping the filter.
- **`x-custom-sql-or`** is grouped with the client's own conditions so it can no longer OR past server-side filters. `common.Database` query builders gain an optional `WhereGrouper` (implemented for bun and gorm).
### ResolveMCP (`pkg/resolvemcp`)
- **Authentication required**: `SetupMuxRoutes`, `SetupBunRouterRoutes`, `SetupMuxStreamableHTTPRoutes`, `SetupBunRouterStreamableHTTPRoutes`, `NewSSEServer` and `NewStreamableHTTPHandler` now take a `*security.SecurityList`. Use the `*Unauthenticated` variants to keep the old open behaviour.
- **Per-model tools removed**: the `read_`/`create_`/`update_`/`delete_` tools and per-model resources are replaced by fixed meta tools: `list_tables`, `describe_table`, `select_table`, `insert_into_table`, `update_table`, `delete_from_table`, `list_functions`, `call_function`.
- **Guarded writes**: filter-based update/delete require filters, are capped by `MaxWriteRows`, and need a single-use confirm token (or `dry_run`).
- **Limits and errors**: reads are capped (`DefaultLimit`, `MaxLimit`, `MaxOffset`, `MaxBatch`, `MaxPreloadDepth`, `QueryTimeout`); the total `COUNT` is optional; errors reach clients as `{code, message}` only.
- **Writes**: create checks `CanCreate`; create/update reject keys outside the model's writable columns; update sets only the given keys; the annotation tool is opt-in via `Config.EnableAnnotations`.
### Update semantics
- **`""` and `null` now overwrite** stored values on update in resolvespec and restheadspec (previously skipped). Use `Handler.SetDisallowNulls` to skip nulls.
- **websocketspec / mqttspec** update only the keys present in the payload (`SetMap`) instead of writing the whole zeroed model.
### Transactions and hooks
- **One transaction per request** in every spec. Hooks must use `hookCtx.Tx`, not the pool; `BeforeHandle` runs before any transaction and must not touch the DB. `OnTxBegin` fires first in every transaction.
- **`AfterDelete` failure now rolls the delete back.**
- **Create/update re-fetch, `BeforeScan` and post-commit hooks** (`AfterCreate`, `AfterUpdate`, restheadspec `AfterRead`, funcspec `BeforeResponse`) run on a second, short transaction after the first commits.
- **websocketspec / mqttspec / funcspec** begin or commit failures answer `transaction_error`.
- **`BeforeDisconnect` / `AfterDisconnect`** hooks in websocketspec now fire on close.
### Other
- **Clients moved**: the JS and Python clients now live under `clients/`.
- **Test server ports**: `8123` (testserver) and `8124` (PostgreSQL), previously `8080` and `5434`.
- **pgsql**: subquery preload errors are returned instead of being logged and skipped; a `SET` plus multi-placeholder `WHERE` update previously renumbered `WHERE` parameters wrongly (fixed).
- **Cache**: `Clear()` on the Redis and Memcache providers requires `AllowFlush`; a missing key returns `ErrNotFound`; Memcache keys are hashed and namespaced, so existing entries are not found.
- **Config**: `NewManager` no longer replaces the global manager (use `SetConfigManager`); saved configs are written `0600`; `PathsConfig.Join` is confined to its base path.
## What's New
### Unreleased
**Single transaction per request**:
- **One tx per request**: hooks get the transaction in `hookCtx.Tx`, never the pool (`BeforeHandle` runs before any tx and must not touch the DB)
- **`OnTxBegin` hook**: all specs (mqttspec re-exports websocketspec's); fires once, first, in every tx; error or abort rolls back with no detail to the client
- **Second short tx**: create/update re-fetch, `BeforeScan` and post-commit hooks (`AfterCreate`, `AfterUpdate`, restheadspec `AfterRead`, funcspec `BeforeResponse`) run on a new tx after the first commits
- **Delete**: single and batch delete, hooks included, in one tx
- **websocketspec / mqttspec**: one tx per message; begin/commit failures answer `transaction_error`
- **resolvemcp**: read, create, update, delete transactional
- **RLS stamping**: `SecurityList.SetTxSettings(fn)`; `RegisterSecurityHooks` of every spec stamps `set_config(name, value, true)` on `OnTxBegin`; fails closed
- **New**: `common.RunRequestTx`, `common.TxContext`, `common.TxHookName`
- **Behavior changes**: `AfterDelete` failure now rolls the delete back; funcspec begin/commit failure answers 500 `transaction_error`
**Clients**: Go, Rust, C# and Dart clients for ResolveSpec and FunctionSpec under `clients/`.
**Test server**: compose uses host networking; ports `8123` (testserver) and `8124` (PostgreSQL), previously `8080` and `5434`.
### v3.2 (Latest - March 2026)
**ResolveMCP - Model Context Protocol Server (🆕)**:
* **MCP Tools**: Four tools auto-registered per model (`read_`, `create_`, `update_`, `delete_`) over HTTP/SSE transport
* **AI-Ready Descriptions**: Full column schema, primary key, nullable flags, and relation names surfaced in tool descriptions so AI models can query without guessing
* **Full Query Support**: Filters, sort, limit/offset, cursor pagination, column selection, and relation preloading all available as tool parameters
* **HTTP/SSE Transport**: Standards-compliant transport compatible with Claude Desktop, Cursor, and any MCP 2024-11-05 client
* **Lifecycle Hooks**: Same Before/After hook system as ResolveSpec for auth, auditing, and side-effects
* **MCP Resources**: Each model also exposed as a named resource for direct data access by AI clients
- **MCP Tools**: Four tools auto-registered per model (`read_`, `create_`, `update_`, `delete_`) over HTTP/SSE transport
- **AI-Ready Descriptions**: Full column schema, primary key, nullable flags, and relation names surfaced in tool descriptions so AI models can query without guessing
- **Full Query Support**: Filters, sort, limit/offset, cursor pagination, column selection, and relation preloading all available as tool parameters
- **HTTP/SSE Transport**: Standards-compliant transport compatible with Claude Desktop, Cursor, and any MCP 2024-11-05 client
- **Lifecycle Hooks**: Same Before/After hook system as ResolveSpec for auth, auditing, and side-effects
- **MCP Resources**: Each model also exposed as a named resource for direct data access by AI clients
### v3.1 (February 2026)
**SQLite Schema Translation (🆕)**:
* **Automatic Schema Translation**: SQLite support with automatic `schema.table` to `schema_table` conversion
* **Database Agnostic Models**: Write models once, use across PostgreSQL, SQLite, and MSSQL
* **Transparent Handling**: Translation occurs automatically in all operations (SELECT, INSERT, UPDATE, DELETE, preloads)
* **All ORMs Supported**: Works with Bun, GORM, and Native SQL adapters
- **Automatic Schema Translation**: SQLite support with automatic `schema.table` to `schema_table` conversion
- **Database Agnostic Models**: Write models once, use across PostgreSQL, SQLite, and MSSQL
- **Transparent Handling**: Translation occurs automatically in all operations (SELECT, INSERT, UPDATE, DELETE, preloads)
- **All ORMs Supported**: Works with Bun, GORM, and Native SQL adapters
### v3.0 (December 2025)
**Explicit Route Registration (🆕)**:
* **Breaking Change**: Routes are now created explicitly for each registered model
* **Better Control**: Customize routes per model with more flexibility
* **Registration Order**: Models must be registered BEFORE calling SetupMuxRoutes/SetupBunRouterRoutes
* **Benefits**: More flexible routing, easier to add custom routes per model, better performance
- **Breaking Change**: Routes are now created explicitly for each registered model
- **Better Control**: Customize routes per model with more flexibility
- **Registration Order**: Models must be registered BEFORE calling SetupMuxRoutes/SetupBunRouterRoutes
- **Benefits**: More flexible routing, easier to add custom routes per model, better performance
**OPTIONS Method & CORS Support (🆕)**:
* **OPTIONS Endpoint**: Full OPTIONS method support for CORS preflight requests
* **Metadata Response**: OPTIONS returns model metadata (same as GET /metadata)
* **CORS Headers**: Comprehensive CORS headers on all responses
* **Header Support**: All HeadSpec custom headers (`X-Select-Fields`, `X-FieldFilter-*`, etc.) allowed
* **No Auth on OPTIONS**: CORS preflight requests don't require authentication
* **Configurable**: Customize CORS settings via `common.CORSConfig`
- **OPTIONS Endpoint**: Full OPTIONS method support for CORS preflight requests
- **Metadata Response**: OPTIONS returns model metadata (same as GET /metadata)
- **CORS Headers**: Comprehensive CORS headers on all responses
- **Header Support**: All HeadSpec custom headers (`X-Select-Fields`, `X-FieldFilter-*`, etc.) allowed
- **No Auth on OPTIONS**: CORS preflight requests don't require authentication
- **Configurable**: Customize CORS settings via `common.CORSConfig`
### v2.1
**Cursor Pagination for ResolveSpec (🆕 Dec 9, 2025)**:
* **Cursor-Based Pagination**: Efficient cursor pagination now available in ResolveSpec (body-based API)
* **Consistent with RestHeadSpec**: Both APIs now support cursor pagination for feature parity
* **Multi-Column Sort Support**: Works seamlessly with complex sorting requirements
* **Better Performance**: Improved performance for large datasets compared to offset pagination
* **SQL Safety**: Proper SQL sanitization for cursor values
- **Cursor-Based Pagination**: Efficient cursor pagination now available in ResolveSpec (body-based API)
- **Consistent with RestHeadSpec**: Both APIs now support cursor pagination for feature parity
- **Multi-Column Sort Support**: Works seamlessly with complex sorting requirements
- **Better Performance**: Improved performance for large datasets compared to offset pagination
- **SQL Safety**: Proper SQL sanitization for cursor values
**Recursive CRUD Handler (🆕 Nov 11, 2025)**:
* **Nested Object Graphs**: Automatically handle complex object hierarchies with parent-child relationships
* **Foreign Key Resolution**: Automatic propagation of parent IDs to child records
* **Per-Record Operations**: Control create/update/delete operations per record via `_request` field
* **Transaction Safety**: All nested operations execute atomically within database transactions
* **Relationship Detection**: Automatic detection of belongsTo, hasMany, hasOne, and many2many relationships
* **Deep Nesting Support**: Handle relationships at any depth level
* **Mixed Operations**: Combine insert, update, and delete operations in a single request
- **Nested Object Graphs**: Automatically handle complex object hierarchies with parent-child relationships
- **Foreign Key Resolution**: Automatic propagation of parent IDs to child records
- **Per-Record Operations**: Control create/update/delete operations per record via `_request` field
- **Transaction Safety**: All nested operations execute atomically within database transactions
- **Relationship Detection**: Automatic detection of belongsTo, hasMany, hasOne, and many2many relationships
- **Deep Nesting Support**: Handle relationships at any depth level
- **Mixed Operations**: Combine insert, update, and delete operations in a single request
**Primary Key Improvements (Nov 11, 2025)**:
* **GetPrimaryKeyName**: Enhanced primary key detection for better preload and ID field handling
* **Better GORM/Bun Support**: Improved compatibility with both ORMs for primary key operations
* **Computed Column Support**: Fixed computed columns functionality across handlers
- **GetPrimaryKeyName**: Enhanced primary key detection for better preload and ID field handling
- **Better GORM/Bun Support**: Improved compatibility with both ORMs for primary key operations
- **Computed Column Support**: Fixed computed columns functionality across handlers
**Database Adapter Enhancements (Nov 11, 2025)**:
* **Bun ORM Relations**: Using Scan model method for better has-many and many-to-many relationship handling
* **Model Method Support**: Enhanced query building with proper model registration
* **Improved Type Safety**: Better handling of relationship queries with type-aware scanning
- **Bun ORM Relations**: Using Scan model method for better has-many and many-to-many relationship handling
- **Model Method Support**: Enhanced query building with proper model registration
- **Improved Type Safety**: Better handling of relationship queries with type-aware scanning
**RestHeadSpec - Header-Based REST API**:
* **Header-Based Querying**: All query options via HTTP headers instead of request body
* **Lifecycle Hooks**: Before/after hooks for create, read, update, delete operations
* **Cursor Pagination**: Efficient cursor-based pagination with complex sorting
* **Advanced Filtering**: Field filters, search operators, AND/OR logic
* **Multiple Response Formats**: Simple, detailed, and Syncfusion-compatible responses
* **Single Record as Object**: Automatically return single-element arrays as objects (default, toggleable via header)
* **Base64 Support**: Base64-encoded header values for complex queries
* **Type-Aware Filtering**: Automatic type detection and conversion for filters
- **Header-Based Querying**: All query options via HTTP headers instead of request body
- **Lifecycle Hooks**: Before/after hooks for create, read, update, delete operations
- **Cursor Pagination**: Efficient cursor-based pagination with complex sorting
- **Advanced Filtering**: Field filters, search operators, AND/OR logic
- **Multiple Response Formats**: Simple, detailed, and Syncfusion-compatible responses
- **Single Record as Object**: Automatically return single-element arrays as objects (default, toggleable via header)
- **Base64 Support**: Base64-encoded header values for complex queries
- **Type-Aware Filtering**: Automatic type detection and conversion for filters
**Core Improvements**:
* Better model registry with schema.table format support
* Enhanced validation and error handling
* Improved reflection safety
* Fixed COUNT query issues with table aliasing
* Better pointer handling throughout the codebase
* **Comprehensive Test Coverage**: Added standalone CRUD tests for both ResolveSpec and RestHeadSpec
- Better model registry with schema.table format support
- Enhanced validation and error handling
- Improved reflection safety
- Fixed COUNT query issues with table aliasing
- Better pointer handling throughout the codebase
- **Comprehensive Test Coverage**: Added standalone CRUD tests for both ResolveSpec and RestHeadSpec
### v2.0
**Breaking Changes**:
* **None!** Full backward compatibility maintained
- **None!** Full backward compatibility maintained
**New Features**:
* **Database Abstraction**: Support for GORM, Bun, and custom ORMs
* **Router Flexibility**: Works with any HTTP router through adapters
* **BunRouter Integration**: Built-in support for uptrace/bunrouter
* **Better Architecture**: Clean separation of concerns with interfaces
* **Enhanced Testing**: Mockable interfaces for comprehensive testing
- **Database Abstraction**: Support for GORM, Bun, and custom ORMs
- **Router Flexibility**: Works with any HTTP router through adapters
- **BunRouter Integration**: Built-in support for uptrace/bunrouter
- **Better Architecture**: Clean separation of concerns with interfaces
- **Enhanced Testing**: Mockable interfaces for comprehensive testing
**Performance Improvements**:
* More efficient query building through interface design
* Reduced coupling between components
* Better memory management with interface boundaries
- More efficient query building through interface design
- Reduced coupling between components
- Better memory management with interface boundaries
# Security Policy
## Reporting a vulnerability
Please do not open a public issue for security problems.
Report privately through GitHub: Security → Report a vulnerability
(https://github.com/bitechdev/ResolveSpec/security/advisories/new),
or email hein@bitechsystems.co.za / hein@warky.dev
You'll get an acknowledgement within 7 days. We aim to release a fix within
90 days and will credit reporters in the advisory unless they prefer otherwise.
## Acknowledgments
* Inspired by REST, OData, and GraphQL's flexibility
* **Header-based approach**: Inspired by REST best practices and clean API design
* **Database Support**: [GORM](https://gorm.io) and [Bun](https://bun.uptrace.dev/)
* **Router Support**: Gorilla Mux (built-in), BunRouter, Gin, Echo, and others through adapters
* Slogan generated using DALL-E
* AI used for documentation checking and correction
* Community feedback and contributions that made v2.0 and v2.1 possible
- Inspired by REST, OData, and GraphQL's flexibility
- **Header-based approach**: Inspired by REST best practices and clean API design
- **Database Support**: [GORM](https://gorm.io) and [Bun](https://bun.uptrace.dev/)
- **Router Support**: Gorilla Mux (built-in), BunRouter, Gin, Echo, and others through adapters
- Slogan generated using DALL-E
- AI used for documentation checking and correction
- Community feedback and contributions that made v2.0 and v2.1 possible
![1.00](./generated_slogan.webp)
+145
View File
@@ -0,0 +1,145 @@
# resolvemcp rewrite plan
Source: `audit/pkg/resolvemcp.audit.md`. Status: items 1-8 and 10 implemented; item 9 (tests) mostly done, see git log.
## Goal
Replace 4 tools + 1 resource per model with a fixed set of meta tools.
Endpoint guarded by OAuth / session token / API key; tools run as the authenticated caller.
Same rules as resolvespec CRUD, plus guardrails.
## Decisions
| Topic | Decision |
|---|---|
| Tools | Fixed meta tools; per-model tools/resources removed (breaking) |
| Functions | Explicit registry `Handler.RegisterFunction`; two kinds: Go callback (`func(ctx, tx, args)` + JSON-schema params) and SQL procedure by name (declared params); both behind `call_function`, run in tx with hooks |
| Create | `insert_into_table` included |
| Writes | update/delete by id **or** filters |
| Guardrails | require id or filters, max rows, `dry_run`, confirm token |
| Token scope | filter writes only; id writes = single row, no token |
| Confirm token store | in-memory, TTL, bound to user/table/filter hash; lost on restart, single instance |
| Read limits | server caps: limit, offset, batch, preload depth, timeout |
| Model exposure | all registered models visible; rules only restrict operations |
| Visibility | list tools show only what the caller may do (rules) |
| Identity | the authenticated caller's `UserContext`; **no fixed MCP user, no `SetUsername`, no service session, no background refresh** |
| Guard | endpoint always requires one of: OAuth bearer, session token, API key; no guest/optional mode |
| API key login | new `DatabaseAuthenticator.LoginWithAPIKey(ctx, rawKey)` + procedure `resolvespec_login_api_key`; validates key via keystore, creates session, returns `LoginResponse` |
| Session SQL | procedure mode + direct-SQL fallback (`ShouldUseProcedure`), same as `Login` |
| OAuth routes | `oauth2.go`/`oauth2_server.go` kept, part of the guard |
| Annotations | opt-in `Config.EnableAnnotations`, via `BeforeHandle` |
## Open
- None.
## Tools
| Tool | Purpose |
|---|---|
| `list_tables` | visible `schema.entity` + allowed ops |
| `describe_table` | columns, PK, relations, writable columns, rules, limits |
| `select_table` | filters, sort, columns, preloads, cursor; capped |
| `insert_into_table` | one or batch (capped); column allowlist |
| `update_table` | validated keys; id or filters; guardrails |
| `delete_from_table` | id or filters; guardrails |
| `list_functions` | registered functions + parameter schemas |
| `call_function` | validated args; tx + hooks + rules |
## Config additions
| Field | Purpose |
|---|---|
| `DefaultLimit`, `MaxLimit`, `MaxOffset` | read paging caps |
| `MaxBatch` | insert batch cap |
| `MaxPreloadDepth` | preload cap |
| `MaxWriteRows` | filter-write row cap |
| `QueryTimeout` | per-call context timeout |
| `ConfirmTTL` | confirm token lifetime |
| `EnableAnnotations` | opt-in annotate tool |
## Guardrail rules
| Rule | Behaviour |
|---|---|
| Target required | update/delete with neither id nor filters rejected |
| Max rows | count matches inside tx; abort above `MaxWriteRows` |
| `dry_run` | returns match count + preview, no write |
| Confirm token | filter write: first call returns token + preview; second call with token executes; bound to user, table, filter hash; expires at `ConfirmTTL` |
| Id write | single row, no token |
## Work items
### 1. API key login (`pkg/security`)
- Existing: keystore has `ValidateKey` and `KeyStoreAuthenticator`; `Login` needs a password; no key-to-session path.
- Add `resolvespec_login_api_key` to `lookup.ProcNames` (default + override; was `SQLNames` before the lookup refactor) and a SQL script beside the existing procedures. Contract: `p_success, p_error, p_data`, input raw key; hashes, validates active/non-expired key, creates session for the key's user.
- Add `DatabaseAuthenticator.LoginWithAPIKey(ctx, rawKey)`; procedure first, direct-SQL fallback via `ShouldUseProcedure`.
- Hashed lookup; same generic error for unknown, expired or inactive key; no key material in logs.
- Expose through the chain/composite authenticators so the middleware can accept it.
### 2. Endpoint guard
- Wire `security.NewAuthMiddleware` with a chain of OAuth bearer, session token (header/cookie) and API key.
- `SetupMux*`/`SetupBunRouter*` helpers require the guard; unauthenticated serving only when explicitly constructed without it, logged loudly. Remove `OptionalAuth*` from the MCP path.
- Caller `UserContext` flows to every tool call context; rules, RLS and `OnTxBegin` apply to that user.
### 3. Security fixes (audit #1-6)
- Put model rules in request context (`withRequestData`) and/or `AddRegistry` on construction.
- Add `BeforeCreate` -> `CheckModelCreateAllowed` (new in `pkg/security`).
- Call `BeforeHandle` first in `executeUpdate`.
- Validate create/update keys against `ColumnValidator`; reject unknown; resolve column names from model, not json tags.
- Apply row security to update/delete pre-read; fail if row not visible.
- Annotate tool: opt-in + `BeforeHandle`.
### 4. Limits (audit #7, #13)
- Apply default/max limit, max offset, batch cap, preload depth cap, timeout.
- Validate preload names against model relations.
- Count only when requested.
### 5. Error and panic surface (audit #9, #17)
- Stable error codes + short message to client.
- Details and stack logged server-side.
- Recover hook panics.
### 6. Update/create semantics (audit #10-12)
- `SET` from validated incoming keys only; explicit null supported.
- Lock row (`FOR UPDATE`) on update.
- Refetch and `After*` hooks inside the same tx, or report committed write with warning if not possible (see `audit/single_tran.md`).
### 7. Smaller fixes (audit #8, #14, #15)
- SSE pool: require `BaseURL` or cap/evict; allowlist Host.
- Uniform not-found vs hook error text.
- Add mutex to `HookRegistry`.
### 8. Meta tools
- New file for meta tools; reuse parse helpers and `buildModelInfo` for `describe_table`.
- Remove per-model register functions and resources.
- `RegisterModel` only registers to registry.
- Function registry (Go callback kind + SQL procedure kind) + validation of args against declared schema.
### 9. Tests
- Update `tx_test.go` (calls `executeRead/Create/Update/Delete`) and `tools_test.go`.
- New: rule enforcement, unknown keys, limits, guardrails (cap, dry_run, token expiry/binding), guard rejects unauthenticated, API key login (valid, expired, inactive, unknown), visibility filtering, `-race`.
- Check for existing test data first; ask before generating any.
### 10. Docs
- Rewrite `pkg/resolvemcp/README.md` cheatsheet style.
- Document `resolvespec_login_api_key` in `pkg/security` docs.
- Update root README references.
- Update audit file when findings are closed.
## Order
1. `LoginWithAPIKey` + procedure in `pkg/security` (1)
2. Guard + security fixes (2-3)
3. Limits, errors, update/create semantics (4-6)
4. Meta tools + function registry (8)
5. Smaller fixes (7)
6. Tests (9), docs (10)
## Breaking changes
- Per-model tools and resources gone.
- MCP endpoint requires authentication.
- Annotate tool off by default.
- Update/create reject unknown keys.
- Reads capped by default.
+228
View File
@@ -0,0 +1,228 @@
# Audit: `pkg/funcspec`
| | |
|---|---|
| **Package** | `github.com/bitechdev/ResolveSpec/pkg/funcspec` |
| **Files** | `function_api.go` (1251), `parameters.go` (411), `hooks.go` (179), `hooks_example.go`, `security_adapter.go` (117) |
| **Tests** | `function_api_test.go` (1278), `hooks_test.go` (589), `parameters_test.go` (549) — 2 416 lines; `go test` and `go test -race` pass, 76.1 % statement coverage |
| **Audit date** | 2026-09-30 |
| **Axes** | thread locking/waiting, slowness, security, panic handling & logging |
| **Threat model** | hostile internet client; query string, headers and body are attacker-controlled |
| **Depth** | targeted (server-side request path; verified against source) |
## Summary
`funcspec` exposes app-defined SQL templates as endpoints. The template is
trusted; everything the client adds to it is not. The package builds SQL by
string manipulation and has two kinds of client-controlled SQL fragments
(`X-Custom-SQL-W`, `X-Custom-SQL-Or`, `sort`) that are guarded only by a keyword
denylist (`ValidSQL(..., "select")`, `function_api.go:951-980`). That is not an
injection boundary: the fragment lands inside a query that may already carry
tenant or auth predicates, and the OR path produces wrong precedence that
widens results (findings 1-3).
The auth integration is weaker than it looks. `RegisterSecurityHooks` is opt-in,
the anonymous default is `UserID 0`, and the auth hooks return an error *and*
set `Abort`, so `Execute` returns the error first and the handler answers
**400 `hook_error`**, not 401 (finding 6).
Error handling leaks: `sendError` returns the DB error text and the full SQL to
the client, and the panic recovery writes the panic value into the 500 body
(finding 5).
Resource limits are absent: default limit 100 000, no cap on `X-Limit`, no
LIMIT at all when counting is skipped, unbounded `[post_body]` read, a 15-minute
timeout, and a `COUNT(1)` over the full query on every list request (finding 8).
Positives: there are no data races under the existing tests; the `[variable]`
substitution is quote-context aware; `Content-Type` and transaction handling are
consistent; hooks run inside the transaction.
## Findings
| # | Severity | Axis | Finding |
|---|---|---|---|
| 1 | **High** | security | `X-Custom-SQL-W`, `X-Custom-SQL-Or` and `sort` are appended as raw SQL, protected only by a keyword denylist (`ValidSQL "select"`) |
| 2 | **High** | security | `sqlQryWhereOr` emits `a AND b OR (c)`; OR conditions escape the AND-ed predicates (auth/tenant filters) |
| 3 | **Medium** | security | `sqlQryWhere`/`sqlQryWhereOr` locate WHERE/GROUP BY/ORDER BY/LIMIT by substring on the lower-cased query, including inside literals, subqueries and CTEs |
| 4 | **Medium** | security | Unquoted string `X-FieldFilter` value in `ApplyFilters`; `SearchOps` keyed per column (one op per column, random order) |
| 5 | **High** | security / logging | `sendError` returns `err.Error()` and the full SQL; panic recovery writes the panic value to the 500 body |
| 6 | **High** | security / correctness | Auth hooks return an error plus `Abort`; handler replies 400 `hook_error` instead of 401; hooks are opt-in; anonymous = `UserID 0` |
| 7 | **Medium** | security | `DecodeParam` (`ZIP_`/`__`) is applied to every header/param, ignores errors and recurses without a depth limit; headers matched with `HasPrefix` |
| 8 | **High** | slowness | Default limit 100 000, no `X-Limit` cap, no LIMIT with `NoCount`/`skipcount`, unbounded `io.ReadAll` of `[post_body]`, 15-minute timeout, full `COUNT(1)` per list request |
| 9 | **Medium** | locking | `HookRegistry` map and `variablesCallback` are unsynchronized; `Register`/`Clear*` race with `Execute` |
| 10 | **Medium** | security | Dollar-quote substitution (`[post_body]`, `[user]`, `[method]`, …) skips backslash escaping; `[id_session]` substituted unquoted |
| 11 | **Low** | correctness | `Content-Range` offset comes from the `offset` query param only; header offset ignored |
| 12 | **Low** | correctness | `X-Select-Fields`/`X-Not-Select-Fields` accepted but no-ops; `sort` `-col` negates instead of DESC |
| 13 | **Low** | panic / logging | Recovery is handler-level only; `Serving: Records` logged at Info per request; hook/filter strings logged unscrubbed (X8) |
| 14 | **Low** | correctness | `BeforeResponse` runs post-commit on the pool, not the tx (see `audit/single_tran.md`) |
| 15 | **Low** | security | Security adapter hard-codes schema `public` and entity `sql_query`; per-entity rules cannot be applied |
| 16 | **Info** | testing | Regexes compiled per call (`ValidSQL`, `sqlStripStringLiterals`); no `-race` in CI (X1); no hostile-input tests for findings 1-4 |
## 1. Raw SQL fragments behind a keyword denylist — High
`ApplyFilters` (`parameters.go:283-297`) passes `X-Custom-SQL-W` and
`X-Custom-SQL-Or` through `ValidSQL(..., "select")` and splices the result into
the query. `sort` goes the same way into `ORDER BY` (`function_api.go:~226`).
The denylist (`function_api.go:964-979`) removes `;`, `--`, `/*`, `*/`, `xp_`,
`sp_` and a few keywords **followed by a space**. It is not a parser:
- Subqueries, function calls (`pg_sleep`, `pg_read_*` where permitted),
`SELECT` itself and `)` are not blocked; a `)` can close the
`COUNT(1) FROM (%s) cnts` wrapper (`function_api.go:~241`).
- Keywords are removed rather than rejected, so input can be shaped so that
removal assembles a different token.
- Whitespace variants (tab, newline) bypass the `keyword␠` patterns.
Whether the raw fragments are reachable is decided by the handler; they are
parsed whenever `ParseParameters` runs, i.e. always. Fix: drop the two headers
from the wire contract, or accept only a column/operator/value structure built
by the server; validate `sort` against `^[A-Za-z0-9_.]+( (ASC|DESC))?(,…)*$`
and ideally an allowlist of columns.
## 2. OR precedence widens results — High
`sqlQryWhereOr` (`parameters.go:381-411`) rewrites `WHERE a AND b` into
`WHERE a AND b OR (c)`. SQL evaluates `AND` first, so the result is
`(a AND b) OR c`: any row satisfying `c` is returned regardless of `a`/`b`.
Where `a` is a tenant or ownership predicate in the template, a client-supplied
OR condition (`X-SearchOr`, `X-Custom-SQL-Or`, search operator with logic OR)
returns other tenants' rows. Verified with `ParseParameters` + `ApplyFilters`
on generated headers. Fix: wrap the existing WHERE body in parentheses before
appending `OR (...)`, or build a predicate tree.
## 3. Substring-based clause location — Medium
Both helpers use `strings.Index` on `" where "`, `" group by"`, `" order by"`,
`" limit "` over the whole lower-cased query. A match inside a string literal,
a subquery, a CTE or a column alias selects the wrong insertion point, and
`wherePos > 0` decides AND-append vs. new WHERE on the first match anywhere.
`ApplyDistinct` (`parameters.go:363-378`) similarly inserts after the first
`SELECT` substring, and the ORDER BY test (`function_api.go:~224`) compares the
first `order by` to the first `from `. `sqlStripStringLiterals` exists
(`function_api.go:858`) but is not used by these helpers.
## 4. Filter handling inconsistencies — Medium
- `ApplyFilters` builds `col = value` for `X-FieldFilter` without quoting the
value (`parameters.go:248-250`), so a string value becomes a column reference
(`status = active`). `mergeHeaderParams` quotes the same filter, so the
`SqlQuery` path applies it twice with different semantics.
- `RequestParameters.SearchOps` is a map keyed by column; two operators on one
column overwrite each other and map iteration order makes the generated WHERE
non-deterministic.
## 5. Information disclosure in errors — High
- `sendError` (`function_api.go:1150-1172`) sets `Detail = err.Error()` and,
for `*common.SQLError`, `SQL` = the final statement, including the template,
substituted values and any injected fragment. Used by every failure path
(`query_failed`, `count_failed`, `hook_error`).
- Panic recovery in `SqlQueryList` (`:80-86`) and `SqlQuery` (`:433-439`) calls
`http.Error(w, fmt.Sprintf("Internal server error: %v", err), 500)`; the
panic value reaches the client. Same class as `middleware` finding 4.
Fix: log server-side, return a generic message plus a request id.
## 6. Auth hook abort returns 400, hooks opt-in — High
`RegisterSecurityHooks` (`security_adapter.go:14-55`) sets `Abort`,
`AbortCode=401` **and returns an error**. `HookRegistry.Execute`
(`hooks.go:113-137`) returns the error before it evaluates `Abort`, and the
handler maps that to `sendError(400, "hook_error", …)`
(`function_api.go:~202`). The 401 branch in the handler is only reachable for
hooks that set `Abort` without returning an error. Clients therefore see 400
with `Detail: "hook execution failed: authentication required"`.
Also: without `RegisterSecurityHooks` there is no authentication at all; a
missing user context is replaced with `UserID 0, "anonymous"`
(`function_api.go:~103`) and the request proceeds. Fix: return nil after
setting `Abort` in the auth hooks, or have the handler honour `AbortCode` when
the error wraps an abort; consider fail-closed by default.
## 7. Header/param decoding — Medium
`decodeValue` (`parameters.go:203`) calls `restheadspec.DecodeParam` and drops
the error. `DecodeParam` replaces all `ZIP_`/`__` occurrences and decodes
recursively with no depth limit, so one value can force repeated base64/gzip
work (decompression amplification, since size is not capped). Header keys are
matched with `HasPrefix`, so `X-SearchOp-<anything>` variants and unrelated
headers with the same prefix are interpreted.
## 8. Unbounded resource use — High
- `parameters.go:54` default `Limit: 100000`; `X-Limit` and `limit` accept any
positive integer.
- In `SqlQueryList` the `LIMIT`/`OFFSET` clause is added **only inside
`if !options.NoCount`** (`function_api.go:~232-251`); `NoCount` or
`X-SkipCount` returns the whole result set.
- `COUNT(1) FROM (<full query>)` runs on every list request (double execution
cost).
- `[post_body]` uses `io.ReadAll(r.Body)` (`function_api.go:913`) with no
`http.MaxBytesReader`; the body is also embedded into the SQL text.
- `context.WithTimeout(…, 15*time.Minute)` (`:91`, `:444`) holds a transaction
and pooled connection for up to 15 minutes per request.
- `ValidSQL` and `sqlStripStringLiterals` compile regexes on each call.
Fix: hard cap on limit, always apply LIMIT, cap body size, configurable timeout.
## 9. Unsynchronized registry — Medium
`HookRegistry.hooks` (`hooks.go`) is a plain map; `Register`, `Clear`,
`ClearAll` mutate it while `Execute` reads it from request goroutines. Safe
only if all registration completes before serving. `Handler.variablesCallback`
(`function_api.go:60-68`) has the same property. Fix: `sync.RWMutex` and copy-on-
read, or document and enforce "register before serve".
## 10. Dollar-quote substitution — Medium
`safeSubstituteVar` returns the raw value when the placeholder is adjacent to
`$` (`function_api.go:1044-1049`), so neither backslash nor quote escaping
applies. The tag is neutralised only for `$M$`, `$PBODY$` and the equivalents
in `replaceMetaVariables`; a caller-supplied value in a template that uses a
different tag (or `$$`) is not. `isInsideDollarQuote` inspects only the first
occurrence of the placeholder. `[id_session]` is replaced without any quoting
(`function_api.go:~900`); its source is the auth layer, but it becomes an
injection point if a session token format allows quotes.
## 11-12. Behavioural defects — Low
- `Content-Range` offset uses only `r.URL.Query().Get("offset")`
(`function_api.go:~319`) while the applied offset can come from
`X-Offset`; the reported range is wrong for header-driven paging.
- `ApplyFieldSelection` (`parameters.go:226-241`) only logs; the headers have
no effect. `sort=-col` is not converted to DESC; it is emitted as `ORDER BY
-col`, which negates the column value.
## 13. Panic handling and logging — Low
Recovery exists per handler only (no middleware-level recovery for hooks run
outside), and the stack is logged via `logger.Error`, which forwards to Sentry
unscrubbed (X8). `logger.Info("Serving: Records …")` runs on every list request.
`logger.Debug` lines include the generated filter SQL and attacker-supplied
values. Hook failures log `err` with attacker-influenced text.
## 14. `BeforeResponse` outside the transaction — Low
`BeforeResponse` executes after `RunInTransaction` returns, with
`hookCtx.Tx = h.db` (`function_api.go:~336-343`, `:~640`). A hook that writes
cannot be rolled back with the query, and a failure returns 500 after the work
committed. Tracked in `audit/single_tran.md`.
## 15. Security adapter — Low
`funcSpecSecurityContext.GetSchema()` returns `"public"` and `GetEntity()`
returns `"sql_query"` for every endpoint (`security_adapter.go:84-92`), so
column/row security rules keyed by entity cannot distinguish funcspec endpoints.
`GetModel`, `GetQuery`, `SetQuery` are stubs.
## 16. Testing — Info
Tests cover handler flow, hooks and parameter parsing. No test exercises the
hostile inputs of findings 1-4 or the 401-vs-400 outcome. There is no `-race`
job in CI (X1). The earlier note about a failing
`TestReplaceMetaVariables/Replace_[user]` no longer reproduces: the package
passes today.
## Cross-references
X1 (no `-race`), X7 (inconsistent panic handling), X8 (logger forwards to
Sentry unscrubbed), `middleware` finding 4 (panic value in body),
`audit/single_tran.md` (post-commit hooks).
+122
View File
@@ -0,0 +1,122 @@
# Audit: `pkg/resolvemcp`
| | |
|---|---|
| **Package** | `github.com/bitechdev/ResolveSpec/pkg/resolvemcp` |
| **Files** | `handler.go` (901), `tools.go` (720), `cursor.go`, `oauth2.go`, `oauth2_server.go`, `annotation.go`, `hooks.go`, `security_hooks.go`, `context.go`, `resolvemcp.go` |
| **Tests** | `tools_test.go` (34), `tx_test.go` (207); `go test` passes. No hostile-input tests, no `-race` |
| **Audit date** | 2026-09-30 |
| **Status** | Rewrite implemented (meta tools, guard, limits, guardrails, function registry); see `audit/mcp_plan.md` |
| **Axes** | thread locking/waiting, slowness, security, panic handling & logging, agent usability |
| **Threat model** | hostile or confused MCP client (LLM agent, possibly prompt-injected); tool arguments are attacker-controlled |
| **Depth** | targeted (request path, security wiring; verified against source) |
## Summary
Every model registers 4 tools + 1 resource (`read_/create_/update_/delete_<schema>_<entity>`), each with an
inlined column list, relation list and schema doc. Tool list grows 4N; context cost is
paid on every session whether or not the table is used. Replace with fixed meta tools
(see Rewrite).
Security wiring fails open in several places: model rules never reach the hooks,
`create` has no rule check, `update` skips `BeforeHandle`, update/delete skip row-level
security, and create/update write client-chosen column names. Reads have no size cap.
`resolvespec_annotate` is an unauthenticated write channel into agent-visible text.
## Findings
| # | Severity | Axis | Finding |
|---|---|---|---|
| 1 | **High** | security | Model rules set via `RegisterModelWithRules` never reach `security.Check*`: handler uses a private registry that is not `modelregistry.AddRegistry`'d; hooks look up the global list |
| 2 | **High** | security | `create` has no rule check: `CheckModelAuthAllowed` only tests `CanPublicCreate`/auth; no `BeforeCreate` hook registered, `CanCreate` never read |
| 3 | **High** | security | `executeUpdate` never fires `BeforeHandle` (create/read/delete do); auth + public-rule check skipped, only `BeforeUpdate` (`CanUpdate`) runs |
| 4 | **High** | security | Create/update data keys are not validated against model columns (`q.Value(key,…)`, `SetMap(existingMap)`): mass assignment of any column, arbitrary identifiers |
| 5 | **High** | security | Row-level security (`ApplyRowSecurity`) is wired to `BeforeRead` only; update/delete by id bypass app-level RLS (DB-level RLS via `OnTxBegin` still applies) |
| 6 | **High** | security | `resolvespec_annotate` has no auth/rule check, writes through `h.db` (outside tx, no `OnTxBegin`), any `tool_name` key; annotations are agent-facing text, so it is a prompt-injection store |
| 7 | **High** | slowness | No default/max `limit`, no max `offset`, `COUNT(*)` on every read, `[]` batch create unbounded, no statement timeout |
| 8 | **Medium** | security | `dynamicSSEHandler.pool` keyed by `Host` + `X-Forwarded-Proto` (attacker-controlled): unbounded map growth and poisoned `message` endpoint URL sent to the client |
| 9 | **Medium** | security / logging | Raw `err.Error()` (DB errors, hook errors, panic value `"internal error: %s"`) returned as tool text; `logger.Error` of the same forwards to Sentry (X8) |
| 10 | **Medium** | correctness | Update reads row, merges **json-tag keys** into `SetMap` as column names, writes every column back; breaks when json tag ≠ db column, clobbers concurrent edits (no `FOR UPDATE`) |
| 11 | **Medium** | correctness | Update ignores `nil` and `""` values: a column cannot be set to NULL or empty |
| 12 | **Medium** | correctness | Create/update commit tx 1, then run tx 2 (refetch + `AfterCreate`). Tx 2 failure returns an error for a committed write; an agent retry duplicates the insert |
| 13 | **Medium** | security | Preload relation names are passed straight to `PreloadRelation` without checking the model's relations; no depth/breadth cap |
| 14 | **Low** | security | Update/delete distinguish `record not found` from hook errors, so ids can be enumerated by error text |
| 15 | **Medium** | locking | `HookRegistry.hooks` map unsynchronized; `Register`/`Clear*` race with `Execute` (same as funcspec #9) |
| 16 | **Low** | security | Filter columns are validated by `ColumnValidator` for reads only; sort/column values are interpolated unquoted after validation (relies on validator being exact); `CustomOperators`/`ComputedColumns` unreachable from tools today, keep it that way |
| 17 | **Low** | panic | `recoverPanic` returns the panic value to the client and loses the stack; hook panics in `Execute` are not recovered before the handler-level recover |
| 18 | **Low** | agent usability | Tool names embed schema+entity (`read_public_users`); no discovery tool, so clients cannot list tables without loading every tool schema |
| 19 | **Info** | testing | No tests for auth/rule enforcement, hostile filters, key validation, limits, or `-race` |
## Details
### 1. Rules invisible to hooks (High)
`NewHandlerWithGORM/Bun/DB` call `modelregistry.NewModelRegistry()`. `security` resolves rules
via `GetModelRulesFromContext` then `modelregistry.GetModelRulesByName`, which walks the
**global** list (`registries`). The handler registry is never added, so
`ErrModelNotFound` → `CheckModelUpdate/DeleteAllowed` return `nil` (allow) and
`CheckModelAuthAllowed` falls back to "auth required, public flags ignored".
`CanUpdate=false`, `CanDelete=false` are not enforced. Fix: put rules into the
request context in `withRequestData` (`security.ModelRulesKey`) and/or `AddRegistry` on
construction.
### 2-3. Create/update gating (High)
`CheckModelAuthAllowed(op)` handles public flags only. Add `BeforeCreate` →
`CheckModelCreateAllowed` (new, mirrors update/delete), and call `BeforeHandle` at the
top of `executeUpdate`.
### 4. Column allowlist (High)
Validate every key in create/update `data` against `common.NewColumnValidator(model)`;
reject unknown keys with an error (do not silently drop on writes). Also consider a
per-model writable-column set (excluding PK, `CanPublic*`-guarded columns) for agents.
### 5. RLS on writes (High)
Run `LoadSecurityRules` + a row predicate on the update/delete pre-read query; fail the
write when the row is not visible to the user.
### 6. Annotation tool (High)
Remove from default registration or gate behind `BeforeHandle` + explicit rule. Values
returned to the agent must be treated as data, not instructions.
### 7. Limits (High)
Server config: `DefaultLimit` (e.g. 50), `MaxLimit`, `MaxOffset`, `MaxBatch`, `MaxPreloadDepth`,
per-call `context.WithTimeout`. Skip `COUNT(*)` unless requested (`with_count`).
### 8. SSE pool (Medium)
Require `Config.BaseURL` for SSE, or cap/evict `pool`, and validate `Host` against an
allowlist.
### 9/17. Error surface (Medium/Low)
Map errors to stable codes + short message; log details server-side with stack
(`logger.HandlePanic`).
### 10-12. Update/create semantics (Medium)
Build the `SET` only from validated incoming keys (column names resolved from model,
not json tags); use `NULL` for explicit null; single tx including refetch and `After*`
hooks (see `audit/single_tran.md`), or return success + warning when tx 2 fails.
## Rewrite (agreed design)
Replace per-model tools with fixed meta tools. Decisions recorded 2026-09-30:
| Decision | Choice |
|---|---|
| Functions source | Explicit registry: `Handler.RegisterFunction(name, meta, fn)`; only registered functions visible/callable |
| Old tools/resources | Removed (breaking) |
| Discovery | `list_tables`, `describe_table`, `list_functions` |
| Create | `insert_into_table` added |
| Write scope | update/delete by id **or** filters; max-rows cap, `dry_run` and confirm token apply to filter writes; id writes are single-row, no token |
| Guardrails | require id/filter, max rows affected, `dry_run`, confirm token |
| Read limits / ACL | server caps (limit, offset, preload depth); list tools filtered per caller rules |
| Identity | Authenticated caller's `UserContext`; no fixed MCP user. Endpoint guarded by OAuth / session token / API key (new `resolvespec_login_api_key`); no guest mode |
| Annotations | `resolvespec_annotate` becomes opt-in (`Config.EnableAnnotations`) and goes through `BeforeHandle` |
| Tool | Purpose |
|---|---|
| `list_tables` | registered `schema.entity` visible to caller, with allowed ops |
| `describe_table` | columns, PK, relations, writable columns, rules, limits for one table |
| `select_table` | filters/sort/columns/preloads/cursor; capped |
| `insert_into_table` | one or batch (capped); column allowlist |
| `update_table` | validated keys; guardrails |
| `delete_from_table` | guardrails |
| `list_functions` | registered functions + parameter schemas |
| `call_function` | validated args, tx + hooks |
+1
View File
@@ -9,6 +9,7 @@
| **Docs** | `README.md`, `SECURITY_FEATURES.md`, `QUICK_REFERENCE.md`, `OAUTH2.md`, `OAUTH2_REFRESH_*.md`, `PASSKEY_QUICK_REFERENCE.md`, `KEYSTORE.md` |
| **Tests** | 6 359 lines across 13 `_test.go` files |
| **Audit date** | 2026-09-29 |
| **Note** | Point-in-time snapshot. File names and line numbers refer to the code as audited. Since then all SQL moved out of `pkg/security` into `pkg/security/lookup`: `providers_direct.go`, `sql_names.go`, `table_names.go`, `query_mode.go` and `password.go` are gone, `SQLNames` / `TableNames` / `QueryMode` became `lookup.Config`, and the SQL files moved to `pkg/security/lookup/`. See `pkg/security/breaking_changes.md` for the mapping. |
| **Axes** | thread locking/waiting, slowness, security, panic handling & logging |
| **Threat model** | hostile internet client; request bodies, headers, query params, schema/table/column names and filter expressions all attacker-controlled |
| **Depth** | deep |
+346
View File
@@ -0,0 +1,346 @@
# pkg/security lookup sub package plan
Status: implemented (steps 0-7). What shipped and every API change is recorded in `pkg/security/breaking_changes.md`;
usage is documented in `pkg/security/README.md` ("Database access (lookup)"). This file is kept as the design record.
Deviations from the plan below: only `totp` and `providers` were split out of `pkg/security` (no `oauth` package, the
OAuth server and passkey provider stay in `security`), the `Database*` constructors stay in `security`, and
`ddl/postgres.sql` is tables only and cannot be combined with the procedure schema.
Related: `audit/mcp_plan.md` (work item 1, API key login).
## Problem
- `pkg/security` mixes two data-access styles: stored procedures (`resolvespec_*`) and ~1,800 lines of hand-written
"direct" SQL (`*_direct.go`) selected per call by `QueryMode` / `ShouldUseProcedure`.
- Direct SQL is written once with `?` placeholders and only rewritten for Postgres. It assumes one fixed schema
(table and column names, JSON stored as TEXT, bool/time handling), and is only really exercised on SQLite.
- Table names are configurable (`TableNames`), column names are not. Procedure names are configurable (`SQLNames`).
- Postgres cannot be run "tables only" in a first-class way, and other databases have no defined support.
- Rule going forward: **`pkg/security` itself contains no SQL.** All lookups go through one sub package.
## Goal
New sub package `pkg/security/lookup` that owns every database read/write the security package needs.
| Requirement | Decision |
|---|---|
| Postgres default | Existing stored procedures, existing names, unchanged behaviour out of the box |
| Postgres direct | Optional: work on tables directly with no procs installed |
| SQLite | First-class: configurable tables and columns |
| Other DBs | MySQL/MariaDB and MSSQL via dialects; adding more = adding a dialect |
| Config | Procedure names, table names, column names, mode (procedure / direct / auto), per backend |
| `pkg/security` | Calls lookup interfaces only; no `SELECT`/`INSERT`/`UPDATE`/`DELETE`, no `pg_proc` probing |
## Design
### Package layout
```
pkg/security/ # core: behaviour interfaces, SecurityList, middleware, chain, composite, hooks, write security, tx settings, type aliases
pkg/security/sectypes/ # shared data types (no deps, no SQL, no logic beyond small helpers)
pkg/security/providers/ # concrete authenticators + security providers (see Package split)
pkg/security/oauth/ # OAuth2 client login + OAuth2 authorization server
pkg/security/totp/ # two-factor: generator, providers, TwoFactorAuthenticator
pkg/security/passkey/ # WebAuthn passkey provider + passkey login flow
pkg/security/lookup/
lookup.go # store interfaces + record types + Config + New(db, cfg)
schema.go # Schema: table + column names per entity, defaults, merge, validate
mode.go # Mode (Auto/Procedure/Direct), per-operation resolution, proc probe (pg only)
dialect/ # Dialect interface + postgres, sqlite, mysql, mssql
procedure/ # procedure backend (current SQLNames, p_success/p_error/p_data contract)
direct/ # dialect-driven SQL backend (no fixed SQL strings per dialect)
ddl/ # reference schemas per dialect (replaces database_schema*.sql variants)
```
### Shared types: `pkg/security/sectypes`
`lookup` cannot import `pkg/security` (cycle: security -> lookup -> security), so the plain data types move to a
dependency-free sub package that both import.
- Moves to `sectypes`: `UserContext`, `LoginRequest`, `LoginResponse`, `RegisterRequest`, `LogoutRequest`,
`PasswordResetRequest/Response/CompleteRequest`, `KeyType`, `UserKey`, `CreateKeyRequest/Response`,
`PasskeyCredential` (+ passkey request/option structs the stores return), `TwoFactorSecret`, OAuth server
client/code/token-info structs (`OAuthServerClient`, `OAuthCode`, `OAuthTokenInfo`), `ColumnSecurity`, `RowSecurity`.
- Stays in `pkg/security`: all behaviour interfaces (`Authenticator`, `SecurityProvider`, `Registrable`, ...),
authenticators, middleware, `OAuthServer`, TOTP generator, hooks. They reference `sectypes` types.
- Compatibility: `pkg/security` re-exports each moved type as an alias (`type UserContext = sectypes.UserContext`) and
the `KeyType*` constants, so `security.UserContext` etc. keep compiling and are identical types. In-repo users
(eventbroker, funcspec, mqttspec, resolvemcp, resolvespec, restheadspec, websocketspec, ~17 files) need no edits.
- `lookup` stores take and return `sectypes` types directly; no separate record types and no conversion layer.
- Rules: `sectypes` imports only the standard library (and `oauth2` types only if unavoidable, else a local struct);
JSON tags unchanged so wire formats and procedure `p_data` payloads stay identical.
### Package split
Dependency direction (no cycles): `sectypes` <- `lookup` <- `providers`/`oauth`/`totp`/`passkey` -> `security` (core) -> `sectypes`.
Core `security` never imports the sub packages. Sub packages do not import each other (see Dependency rules).
### Dependency rules (how cycles are avoided)
1. **Layers, imports only point down.** L0 `sectypes` (stdlib only) -> L1 `lookup`, core `security` interfaces ->
L2 `providers`, `oauth`, `totp`, `passkey`. A package may import lower layers, never its own layer or above.
2. **Define interfaces where they are consumed, not where they are implemented** (Go idiom). E.g. `oauth` declares
the small `SessionCreator` it needs; `providers.DatabaseAuthenticator` satisfies it without `oauth` importing `providers`.
3. **Shared data goes down, not sideways.** If two L2 packages need the same struct, it moves to `sectypes`
(or a tiny `internal/` package), never "A imports B for one type".
4. **No L2 -> L2 imports.** Composition happens in the application (or an optional top-level `security/setup`
package that imports everything and is imported by nobody in `pkg/security`).
5. **Dependency injection by constructor**, passing interfaces/stores (`lookup.Provider`, `security.Authenticator`);
no package-level registries that need a back-import; use functional options for optional collaborators.
6. **Core never imports concrete implementations**; where core needs behaviour it calls an interface it owns
(hooks, `SecurityContext`, `Authenticator`).
7. **Tests:** external test packages (`package foo_test`) for cross-package integration tests, so test-only
imports cannot create cycles; shared fixtures in an `internal/testutil` package.
8. **Guard in CI:** `go list -deps` / a small test that asserts the layer rules (e.g. `sectypes` imports only stdlib,
`lookup` does not import `security`, no L2 package imports another L2 package). `go build` already rejects true cycles.
| Package | Contents (from today's files) |
|---|---|
| `security` (core) | `Authenticator`, `SecurityProvider`, `Registrable`, `Refreshable`, `APIKeyLoginable`, ... interfaces; `SecurityList`; `SecurityContext`; middleware + cookie options; `ChainAuthenticator`; `CompositeSecurityProvider`; hooks; `WriteDataContext`; `TxSettings`; type aliases to `sectypes` |
| `providers` | `DatabaseAuthenticator`, `JWTAuthenticator`, `HeaderAuthenticator`, `KeyStoreAuthenticator`, `ConfigKeyStore`, `DatabaseKeyStore`, `DatabaseColumnSecurityProvider`, `DatabaseRowSecurityProvider`, `Config*SecurityProvider` |
| `oauth` | `OAuth2Config`, `OAuth2Provider`, Google/GitHub/Microsoft/Facebook/multi-provider constructors, OAuth2 refresh, `OAuthServer` + `OAuthServerConfig`, oauth server persistence (via `lookup.OAuthClientStore`) |
| `passkey` | `PasskeyProvider` impl (`DatabasePasskeyProvider`), registration/authentication flows, passkey request/option types that are not shared (shared ones stay in `sectypes`) |
| `totp` | `TwoFactorAuthProvider`, `TwoFactorConfig`, `TOTPGenerator`, `MemoryTwoFactorProvider`, `DatabaseTwoFactorProvider`, `TwoFactorAuthenticator` |
Consequences to design for:
- **Methods cannot span packages.** Today OAuth2 and passkey logic are methods on `DatabaseAuthenticator`
(`oauth2_methods*.go`, `oauth_server_db*.go`, passkey methods) and `NewOAuthServer` takes `*DatabaseAuthenticator`.
They become standalone types in `oauth` / `providers` that depend on `lookup` stores and on small interfaces
(e.g. `oauth.SessionCreator`) instead of the concrete authenticator. `NewGoogleAuthenticator(...)` etc. return a
`providers.DatabaseAuthenticator` configured with an `oauth.Provider`, or an `oauth.Authenticator` that implements
`security.Authenticator`; pick one in step 5 (see Open).
- **Constructors cannot be re-exported from core `security`** (it would import the sub packages = cycle). Types that
move to `sectypes` keep aliases; constructors and concrete types do not. This is a breaking import change.
In-repo callers affected (outside `pkg/security`): `pkg/resolvemcp` (`oauth2.go`, `oauth2_server.go`, `handler.go`),
`pkg/middleware/clientqueue.go`, docs and examples. Provide a mechanical migration table
(`security.NewDatabaseAuthenticator` -> `providers.NewDatabaseAuthenticator`, `security.OAuthServer` -> `oauth.Server`, ...).
- **Interfaces core needs from sub packages** (e.g. 2FA hook points) are defined in core or `sectypes`, implemented in
`totp`; core never imports `totp`.
- `examples*.go` / `oauth2_examples.go` / `passkey_examples.go` move next to the package they exemplify (or to
`_example_test.go` files) so core has no dependency on them.
- Tests move with their code; shared helpers (sqlite test DB, `authenticatedRequest`) go to an internal test helper package.
### Store interfaces (one per domain, mirrors current procs)
| Store | Operations (current proc in brackets) |
|---|---|
| `AuthStore` | `Login` [login], `Register` [register], `Logout` [logout], `Session` [session], `TouchSession` [session_update], `Refresh` [refresh_token], `LoginAPIKey` [login_api_key], `JWTLogin`, `JWTLogout`, `ResetRequest`, `ResetComplete` |
| `KeyStore` | `Create`, `List`, `Delete`, `Validate` [keystore_*] |
| `OAuthClientStore` | register client, get client, save code, exchange code, introspect, revoke |
| `OAuthUserStore` | get-or-create user, create session, get/update refresh token, get user |
| `PasskeyStore` | store, get, update counter, list, delete, rename, get by username, login |
| `TOTPStore` | enable, disable, status, secret, regenerate backup codes, validate backup code |
| `PolicyStore` | column security, row security: procedure backend (default) + direct backend over the `sec_*` table layout below |
Each store has a procedure implementation and a direct implementation. A `Provider` bundles them; `security`
constructors take a `lookup.Provider` (or build one from `db` + `lookup.Config`, so existing constructors keep working).
### Config
```go
type Config struct {
Dialect string // "postgres" | "sqlite" | "mysql" | "mssql"; empty = detect from driver
Mode Mode // Auto | Procedure | Direct; default: Procedure for postgres, Direct otherwise
Overrides map[Op]Mode // optional per-operation mode, e.g. direct for Session, procedure for Login
Procs ProcNames // = today's SQLNames (+ LoginAPIKey), defaults unchanged
Schema Schema // tables + columns, see below
}
```
- `Schema` = per entity `{Table string; Columns map[Column]string}` with typed column keys covering every column
(`users.id`, `users.username`, `users.password`, `users.is_active`, ...). Defaults reproduce the current schema,
so zero config behaves as today. Optional `Schema` name per entity for `schema.table` qualification.
- Merge + validation as today: non-empty override wins; every identifier checked against `^[a-zA-Z_][a-zA-Z0-9_]*$`
(plus optional single `schema.` prefix). Identifiers are quoted by the dialect, never interpolated raw.
- No back-compat for config: `SQLNames`, `KeyStoreSQLNames`, `TableNames`, `KeyStoreTableNames` and `QueryMode` are
removed from `pkg/security`; `lookup.Config` replaces them (decision 8).
### Dialect interface (the per-database adaptor)
One adaptor per database type (`dialect/postgres`, `sqlite`, `mysql`, `mssql`), selected by `Config.Dialect` or
detected from the driver, registered through `dialect.Register(name, factory)` so more databases can be added later
without touching core code. Each adaptor supplies only the things that differ; the direct backend builds queries from it.
| Concern | Dialect method |
|---|---|
| Placeholders | `Placeholder(n)` (`$n`, `?`, `@pn`) |
| Identifier quoting | `Quote(ident)` (`"x"`, `` `x` ``, `[x]`) |
| Booleans | `Bool(v)` / scan helper (bool vs 0/1) |
| Time | `Now()` expr or Go-side `time.Now()`; scan helper for drivers returning strings |
| Insert returning id | `InsertReturningID(table, cols, idCol)` returns the SQL + scan strategy: postgres `... RETURNING id` (QueryRow), sqlite/mysql `LastInsertId`, mssql `... OUTPUT INSERTED.id` (QueryRow). The only dialect-specific write construct (decision 12 / confirmed) |
| Get-or-create | none; standard SQL select-then-insert inside a tx (no upsert) |
| Limit/top | only if a query needs it |
| JSON columns (scopes/meta/roles) | `EncodeJSON` / `DecodeJSON` (native jsonb vs TEXT) |
| Random / hashing | done in Go (token generation, SHA-256 key hash, bcrypt) so direct mode needs no `pgcrypto` and no DB functions |
| Driver detection | `Detect(*sql.DB)` from driver type (replaces `driverIsPostgres` / `driverIsPortableOnly`) |
Queries are assembled by a small internal builder (select/insert/update/delete with named columns from `Schema`),
not string-concatenated per dialect and not via an ORM, to keep `pkg/security` free of bun/gorm.
### PolicyStore table layout (column / row security, direct backend)
Approved layout. Both the procedure backend (`resolvespec_column_security` / `resolvespec_row_security`, rewritten
in `database_schema.sql`) and the direct backend read these tables; the former external schema is no longer
referenced anywhere in the repo. All table and column names are configurable via `Schema`, defaults shown.
`sec_group_members` (optional; omit to use direct user rules only)
| Column | Type | Notes |
|---|---|---|
| `group_id` | int, not null | group a user belongs to |
| `user_id` | int, not null | FK users.id; PK (`group_id`, `user_id`) |
`sec_column_rules`
| Column | Type | Notes |
|---|---|---|
| `id` | int PK | |
| `user_id` | int null | rule for one user |
| `group_id` | int null | rule for every member of the group; exactly one of `user_id` / `group_id` set (check constraint) |
| `schema_name` | text, not null | matched case-insensitively |
| `table_name` | text, not null | matched case-insensitively |
| `column_path` | text, not null | dot path under the table (`col` or `col.sub.field`) = `ColumnSecurity.Path` joined by `.` |
| `access_type` | text, not null | `ColumnSecurity.Accesstype` (e.g. `mask`, `hide`, `read`) |
| `mask_start`, `mask_end` | int null | default 0 |
| `mask_invert` | bool null | default false |
| `mask_char` | text null | default `*` |
| `extra_filters` | text/JSON null | `ExtraFilters` map, JSON-encoded via dialect `EncodeJSON` |
| `is_active` | bool, not null | default true |
`sec_row_rules`
| Column | Type | Notes |
|---|---|---|
| `id` | int PK | |
| `user_id` | int null / `group_id` int null | as above, exactly one set |
| `schema_name`, `table_name` | text, not null | case-insensitive match |
| `template` | text null | SQL fragment with the existing placeholders (`RowSecurity.Template`) |
| `has_block` | bool, not null | default false; true = no rows visible (`RowSecurity.HasBlock`) |
| `is_active` | bool, not null | default true |
Resolution rules (direct backend; also the contract the conformance tests assert):
- Applicable rules = active rules where `user_id` = caller, plus rules of every group the caller belongs to.
- Column security: all applicable rules for the exact schema + table, returned as `[]ColumnSecurity` (union).
Exact table match, not a prefix match (a prefix would match `users_archive` for `users`).
- Row security: any applicable `has_block` wins; otherwise templates of all applicable rules are combined with
`AND` (each wrapped in parentheses); no rule = `RowSecurity{}` with `ErrNoRowSecurity` semantics unchanged.
- Templates are still validated/substituted by the existing safe-identifier code in core; the store only loads text.
- Both loaders keep the guarantees of today: user reference reduced to a scalar, failures are errors (fail closed),
no rule is "no rules".
### Mode resolution
- `Procedure`: always call the proc; missing proc = error (no silent fallback).
- `Direct`: always use tables via the dialect builder.
- `Auto`: Postgres probes `pg_proc` once per proc (cached, as today); other dialects resolve to `Direct`.
Probe lives in `lookup` and is the only place that queries the catalog.
- Postgres default stays `Procedure` so current installs do not change behaviour.
- Roles/user-level safety rules already enforced in direct mode (Register ignores client-supplied level/roles, bcrypt
hash, opt-in password upgrade) become backend-agnostic tests that both backends must pass.
## Work items
### 0. Extract shared types (prerequisite, no behaviour change)
- Create `pkg/security/sectypes`, move the types listed above, add aliases in `pkg/security`.
- Verify `go build ./...` and `go test ./pkg/...` unchanged; `go vet` for alias/import cycles.
- Do this first and on its own so the diff is a pure move.
### 0b. Package split (after 0, before `lookup` wiring)
- Create `providers`, `oauth`, `totp`; move files per the Package split table, one package per commit:
`totp` and `passkey` (self-contained) -> `providers` (key stores, authenticators, policy providers) -> `oauth` (needs de-methoding
from `DatabaseAuthenticator`).
- Break the method-on-`DatabaseAuthenticator` coupling for OAuth2 and passkey first (extract interfaces), then move.
- Update in-repo callers and docs; add migration table to `pkg/security/README.md`.
- Behaviour unchanged; at this point stores are still the old direct/proc code, only relocated.
### 1. Skeleton and contracts
- Create `lookup` package: records, store interfaces, `Config`, `Schema` (defaults, merge, validate), `Mode`.
- No behaviour change yet; compile-only.
### 2. Dialects
- Implement `postgres`, `sqlite`, `mysql`, `mssql` against the dialect interface; driver detection.
- Unit tests per dialect: placeholders, quoting, bool/time round trip, insert-returning-id.
### 3. Procedure backend
- Move existing proc calls out of `pkg/security` into `lookup/procedure` using `ProcNames` (current defaults).
- Keep the `p_success, p_error, p_data` contracts and reconnect-on-closed-DB helper.
- Include `resolvespec_login_api_key` (added in mcp_plan item 1) with the generic error behaviour.
### 4. Direct backend
- Port each `*_direct.go` to `lookup/direct` using `Schema` + dialect builder, one store at a time:
AuthStore -> KeyStore -> OAuth stores -> Passkey -> TOTP -> PolicyStore (column/row security tables).
- Add direct `LoginAPIKey` here (select by key hash, active, unexpired, key type in header_api/api, user active;
one generic error), since SQL is now allowed only inside `lookup`.
- Transactions: multi-step writes (login = session insert + last_login; register; reset complete) run in one tx.
### 5. Wire `pkg/security`
- Constructors accept `lookup.Provider` / `lookup.Config`; old options map onto it (deprecated).
- Replace every `*_direct.go` call and `ShouldUseProcedure` branch with a store call.
- Delete `*_direct.go`, `query_mode.go` probe/placeholder code, direct `TableNames` use; keep only aliases.
- Check no non-test code in `pkg/security` contains SQL keywords (CI grep guard).
### 5b. `pkg/security/breaking_changes.md`
- Create at step 0 and append as each step lands: moved types (aliased, no action), moved constructors/types
with rename table (old -> new import path and symbol), removed config types (`SQLNames`, `TableNames`,
`KeyStore*Names`, `QueryMode`) with the `lookup.Config` replacement, removed `database_schema_sqlite.sql`,
`ModeAuto` behaviour change, API key login procedure.
### 6. Schemas and docs
- `lookup/ddl`: reference DDL for postgres (tables only, no procs), sqlite, mysql, mssql; existing proc scripts
stay beside the procedure backend. Replace `database_schema_sqlite.sql`.
- Document: default (procs), Postgres tables-only, SQLite, custom column mapping, adding a dialect.
- Update `pkg/security` README/QUICK_REFERENCE; note API key procedure in security docs (mcp_plan item 10).
### 7. Tests
- Shared conformance suite run against every backend/dialect: login/register/logout/session/refresh, reset,
API keys (valid/expired/inactive/unknown/wrong type), keystore, OAuth server, passkey, TOTP, privilege rules.
- Backends covered: sqlite (in-memory, direct), postgres direct and postgres procedure (needs a Postgres instance,
skipped without `RESOLVESPEC_TEST_PG_DSN`), mysql/mssql behind env DSNs. Dialect unit tests need no DB.
- Procedure backend unit tests with sqlmock for the call/contract shape.
- Check for existing test data before creating any; ask before generating.
- Run with `-race`; migrate current `direct_mode_test.go` / `query_mode_test.go` cases into the suite.
## Order
0. Extract `sectypes` types + aliases (0)
0b. Package split: `totp`, `passkey` -> `providers` -> `oauth` (0b), callers + docs updated
1. Skeleton + Schema/Config (1)
2. Dialects (2)
3. Procedure backend extraction, `pkg/security` wired to it for procs only (3, part of 5) - zero behaviour change
4. Direct backend per store, then remove old `*_direct.go` as each store lands (4, 5)
5. DDL + docs + conformance suite (6, 7)
6. Resume `audit/mcp_plan.md` step 2 (guard) on top of `lookup`
## Breaking changes
- `QueryMode`, `SQLNames`, `TableNames`, `KeyStoreSQLNames`, `KeyStoreTableNames` removed; replaced by `lookup.Config`.
- `ModeAuto` no longer silently falls back from procedure to SQL on non-Postgres drivers without telling: resolution
is explicit and logged once per op.
- `database_schema_sqlite.sql` replaced by `lookup/ddl`.
- Concrete types and constructors move to `providers`, `oauth`, `totp` (e.g. `security.NewDatabaseAuthenticator`
-> `providers.NewDatabaseAuthenticator`, `security.NewOAuthServer` -> `oauth.NewServer`,
`security.NewTOTPGenerator` -> `totp.NewGenerator`). No aliases possible (import cycle); import paths must change.
- Type identity is preserved via aliases; code that used reflection on the package path of these types
(`security.UserContext` -> `sectypes.UserContext`) would see the new path (none found in-repo; re-check).
- Anything outside `lookup` that relied on SQL living in `pkg/security` (none found in-repo) must use the stores.
## Decisions
| # | Topic | Decision |
|---|---|---|
| 1 | Shared package name | `sectypes` |
| 2 | Type aliases in `security` | Kept permanently (public API) |
| 3 | `ColumnSecurity` / `RowSecurity` | Move to `sectypes` (types and their helper logic that has no outside deps) |
| 4 | Moved type names | Renamed for new paths (`oauth.Server`, `totp.Generator`, ...) |
| 5 | OAuth2 client login | Dedicated `oauth.Authenticator` type; `New{Google,GitHub,Microsoft,Facebook}Authenticator` return it |
| 6 | Migration | Rename table only; recorded in new `pkg/security/breaking_changes.md` |
| 7 | Dialects v1 | postgres, sqlite, mysql/mariadb, mssql; more later via the dialect interface |
| 8 | Deprecated config (`SQLNames`, `TableNames`, `KeyStore*Names`, `QueryMode`) | Removed, no aliases; recorded in `breaking_changes.md` |
| 9 | Column mapping | Every column of every entity configurable |
| 10 | DB handle | `*sql.DB` |
| 11 | Column / row security | Procedures (default) **and** a table layout for direct mode |
| 12 | Postgres direct SQL | Standard SQL only: no `ON CONFLICT` / `MERGE`; get-or-create = select then insert in a tx |
| 13 | Token format | Keep `sess_<hex>_<unix>`, generated in Go for direct mode |
## Open
- None.
+132
View File
@@ -0,0 +1,132 @@
# Single transaction per request — plan
## Goal
- Every DB statement and every hook that touches the DB in one request runs on **one transaction / one connection**.
- Hooks never receive the raw pool (`h.db`).
- Fixes: RLS GUCs (`set_config(..., true)`) lost on reads/creates/deletes; extra pool connections; select-then-write races.
## Why
- `set_config(..., true)` is transaction-local. A hook on the pool, or a query on another pool connection, never sees it → RLS returns 0 rows / 42501.
- Each un-transacted call takes its own pool connection → bursts with a small pool (see `dbtrace`).
- Already fixed: read/create hooks in `resolvespec` + `restheadspec` (commit `47708fc`, tag >= v1.1.28). Consumers on older tags still show the bug.
## Current state — BEFORE this work (historical baseline; everything below is now fixed, see Progress and Status)
| Spec | Read | Create | Update | Delete |
|---|---|---|---|---|
| restheadspec | tx; `AfterRead` post-commit on pool | tx; `AfterCreate` post-commit on pool | tx; re-fetch + `BeforeScan` post-commit on pool (`:1667-1674`) | **single: no tx, hook + select + delete on pool (`:1945-1994`)**; batch: tx, per-item `BeforeDelete` inside |
| resolvespec | tx | tx | tx; re-fetch on pool (`:1297, 1449, 1602`) | **single: hook + select + delete on pool (`:1654, 1794, 1806`)**; batch: one `BeforeDelete` before tx, none per item |
| websocketspec | **pool** (`:563-672`) | **pool** (`:708`) | **pool** (`:744`) | **pool** (`:757`) |
| mqttspec | **pool** (`:674-789`) | **pool** (`:838`) | **pool** (`:875`) | **pool** (`:889`) |
| resolvemcp | **pool** (`:253`) | single: **pool** (`:445`); batch: tx | tx | tx |
| funcspec | tx; `BeforeResponse` post-commit on pool (`:337, 640`) | — | — | — |
- Correction to earlier note: "no transactions" in mqttspec/websocketspec/resolvemcp-read is a gap for this problem, not a non-issue.
- `BeforeHandle` runs before any tx by design (auth + model checks, `PreloadSecurityRules`). Keep it DB-free except security preload (own connection, cached).
## In-tx hook coverage today (verified)
- Already in tx with `Tx: tx`: `BeforeRead`, `BeforeCreate`, `BeforeUpdate`, `BeforeScan` (read/create/update, both specs); restheadspec batch delete `BeforeDelete` + `AfterDelete`.
- **Not in tx:** single `BeforeDelete`/`AfterDelete` (both specs), resolvespec batch delete (no per-item hook), all `After*` post-commit, all websocketspec/mqttspec hooks, resolvemcp read/single create.
- Gap beyond coverage: no single guaranteed "tx opened" point. User/RLS stamping would have to be repeated in each `Before*` hook and is missed by any path without one (e.g. resolvespec batch delete). `OnTxBegin` closes this: fires once per tx, first, for read/insert/update/delete and for the second short tx.
## Scope
- `OnTxBegin` + `runInTx` apply to **all six**: resolvespec, restheadspec, websocketspec, mqttspec, resolvemcp, funcspec.
- Each spec has its own `HookType` (resolvespec, restheadspec, websocketspec, resolvemcp, funcspec); mqttspec aliases websocketspec, so it inherits the constant but needs its own handler wiring.
- Same semantics everywhere: fires once per tx, first, for read/insert/update/delete and the second short tx; failure aborts + rolls back, nothing leaked.
- Each spec's `security_hooks.go` registers the user/RLS stamping on `OnTxBegin`.
- Shared helper preferred over six copies: one small function in `pkg/common` (begin tx, set `Tx`, call spec-supplied begin callback), each spec passes its own hook executor.
## funcspec (different)
- Custom SQL handlers (`SqlQuery`, `SqlQueryList`), no CRUD, no model registry; one tx per request already (`:195`, `:561`).
- Already in tx: `BeforeQuery`/`BeforeQueryList`, `BeforeSQLExec`, `AfterSQLExec`, `AfterQuery`/`AfterQueryList`, plus `BeforeOp`.
- `BeforeOp` = generic pre-hook via `ExecuteBeforeOp`, fires before every `Before*` in the tx; but it fires **twice** per tx (query hook + `BeforeSQLExec`), so it is not a once-per-tx point.
- Only gap: `BeforeResponse` runs post-commit with `Tx = h.db` (`:337`, `:640`).
- Applies from this plan: once-per-tx `OnTxBegin` (stamping user/RLS before any SQL, incl. hook-mutated SQL), `BeforeResponse` on a tx, fail-closed abort.
- Does not apply: delete/insert/update phases, second re-fetch tx (no re-fetch; SQL is user-defined), `BeforeHandle` preload.
- Decided: add a real `OnTxBegin`. `BeforeOp` unchanged.
## Common interface (`pkg/common`, new `txhook.go`)
- Precedent: `security.SecurityContext` + per-spec `newSecurityContext(hookCtx)` adapter. Same pattern here.
- `common.TxHookName` = `"on_tx_begin"`: one shared string; each spec declares `OnTxBegin HookType = common.TxHookName` (HookTypes are per-spec types, so the constant value is shared, not the type).
- `common.TxContext` interface, implemented by each spec's `HookContext` via a small adapter: `GetContext()`, `GetTx()`, `SetTx(common.Database)`, plus `Abort` accessors for the abort path.
- `common.RunRequestTx(ctx, db, tc TxContext, onBegin func() error, body func(tx common.Database) error) error`: `RunInTransaction` -> `tc.SetTx(tx)` -> `onBegin()` (spec passes `registry.Execute(OnTxBegin, hookCtx)`) -> `body(tx)`. `onBegin` error or abort = return error = rollback.
- Shared stamping: one function in `pkg/security` taking `SecurityContext` + `common.Database` (sets tx-local user/RLS); each spec's `RegisterSecurityHooks` registers it on `OnTxBegin`. No per-spec copies of the logic.
- Each spec keeps its own registry/`HookContext`; only the tx lifecycle and stamping are shared.
- Out of scope: unifying the six `HookContext` / registry types.
## Design
1. **`OnTxBegin` hook** (new `HookType`, all specs). Runs first inside every tx the handler opens; gets `tx` in `hookCtx.Tx`. RLS stamping is registered once there; reads owner/tenant from request context.
2. **`runInTx` helper per handler**: wraps `RunInTransaction`, sets `hookCtx.Tx = tx`, fires `OnTxBegin`, runs the body. All handler paths use it; no path passes `h.db` to a hook.
3. **Post-commit work** (`After*`, update re-fetch, `BeforeResponse`): run in a second short `runInTx` (so `OnTxBegin` re-applies). Not inside the main tx.
4. **Delete**: hook → select → delete in one tx; 404 on no row; cache invalidation after commit.
5. **Backward compat**: hooks keep the same names/order; only `hookCtx.Tx` changes from pool to tx. `OnTxBegin` is additive.
## Decisions (settled)
- Insert/update: re-fetch + `AfterCreate`/`AfterUpdate`-style post-commit work run in a **second short tx** (must see trigger changes). Only insert/update; read and delete have no second tx.
- Update re-fetch is a plain SELECT in that second tx. No `RETURNING`.
- `OnTxBegin` failure aborts the whole request, rolls back, returns an error with no detail leaked to the client.
## Status summary
**Done**
- P0-P7 all DONE (baseline, delete in one tx, `OnTxBegin` + `runInTx`, second short tx, websocketspec/mqttspec, resolvemcp, funcspec, security stamping).
- Regression tests in all six specs, plus source guard `pkg/common/tx_guard_test.go`.
- `AfterRead` decided and done (restheadspec: second short tx; websocketspec/mqttspec: inside the read tx).
- websocketspec `BeforeDisconnect`/`AfterDisconnect` wired (see Progress); `unwiredHooks` allowlist is now empty.
**Not done**
- Real-Postgres `dbtrace` measurement for websocketspec, mqttspec, resolvemcp, restheadspec, funcspec (only resolvespec measured: `pooled=0` on every op). "Done when" bullet 1 is proven for resolvespec only.
- resolvespec batch delete: per-item `BeforeDelete` (one hook per request today). Deferred on purpose: behavior change.
- Confirm the consumer's ResolveSpec version is >= v1.1.28 (read/create already in tx). Not blocking; needs the consumer.
- Known, pre-existing, not ours: `pkg/security` `TestDatabaseAuthenticator` fails with `-count=2` (use `-count=1`); mqttspec integration tests need a DB.
## Phases
| # | Status | Change | Files | Notes |
|---|---|---|---|---|
| 0 | DONE | Baseline: enable `dbtrace` on testserver, record `tx/tx_queries/pooled/raw` per op | `cmd/testserver`, `pkg/dbtrace` | pooled > 0 on write ops = the gaps above |
| 1 | DONE | Delete in one tx (single + batch, per-item hooks inside tx) | `resolvespec/handler.go`, `restheadspec/handler.go` | fixes 2 pool connections + race + RLS |
| 2 | DONE | `OnTxBegin` hook type + `runInTx` helper | `common/txhook.go`, `*/hooks.go`, `*/handler.go` | resolvespec + restheadspec; other specs in P4-6 |
| 3 | DONE | Insert/update post-commit hooks + re-fetch in second short `runInTx` (select only) | restheadspec `:1005, 1467, 1667-1674`; resolvespec `:1297, 1449, 1602` | per decision above |
| 4 | DONE | websocketspec + mqttspec: wrap read/create/update/delete in `runInTx` | `websocketspec/handler.go`, `mqttspec/handler.go` | mqttspec aliases websocketspec hooks; confirm `OnTxBegin` alias |
| 5 | DONE | resolvemcp: read + single create in tx | `resolvemcp/handler.go:253, 445` | verify batch/update/delete hooks run inside tx |
| 6 | DONE | funcspec: `OnTxBegin` (or once-per-tx `BeforeOp`), `BeforeResponse` via `runInTx` | `funcspec/function_api.go:337, 640` | |
| 7 | DONE | Security hooks: register RLS stamping on `OnTxBegin`; document | `pkg/security/*`, README | |
## Progress
- DONE P0: baseline via `dbtrace` on real Postgres (commit `cd96404`): create/read/delete `pooled=0`; update `pooled=1` (re-fetch) = P3 target. websocketspec/mqttspec/resolvemcp not measured.
- DONE P1: single + batch delete in one tx (resolvespec, restheadspec). Not done: per-item `BeforeDelete` in resolvespec batch (behavior change, deferred).
- DONE infra: `sqlmock` delete tx tests (both specs); compose test server + `scripts/testserver-smoke.sh` (podman first); testmodels ids now serial.
- NOTE: restheadspec single delete still does the lookup before `BeforeDelete`; safe once `OnTxBegin` (P2) exists. An `AfterDelete` failure now rolls the delete back.
- DONE P2 (resolvespec + restheadspec): `common.TxHookName`, `common.TxContext` (`SetTx` only; no abort/context accessors needed since `Execute` already returns an error on abort), `common.RunRequestTx`; per-spec `OnTxBegin`, `HookContext.SetTx`, `Handler.runInTx`. Every `RunInTransaction` in both handlers now goes through it. Tests: `pkg/*/on_tx_begin_test.go` (once, first, on tx, failure rolls back). Not yet: the post-commit second tx (P3) and the security stamping registration (P7).
- DONE P3: restheadspec update re-fetch + `BeforeScan` + `AfterUpdate` and `AfterCreate` run in a second short `runInTx`; resolvespec update re-fetches (single, both batch paths) run in a second short `runInTx`. Fixed the pool reads inside the first tx (resolvespec single/batch update existing-record select, restheadspec update existence select) to use `tx`. Tests: `pkg/*/update_tx_test.go` (restheadspec uses the bun adapter; the pgsql adapter cannot build model-based updates).
- NOTE: resolvespec fires no `AfterCreate`/`AfterRead`/`AfterUpdate`-post-commit hooks other than `AfterUpdate` inside the tx; nothing more to move there.
- RESOLVED: restheadspec `AfterRead` question (see DONE AfterRead below).
- DONE P4: websocketspec + mqttspec. `OnTxBegin` (mqttspec re-exports the websocketspec constant), `HookContext.SetTx`, per-handler `runInTx`/`sendTxError`. Per message: read = 1 tx (Before/After hooks + queries); delete = 1 tx (Before, delete, After); create/update = tx 1 (Before + write) then tx 2 (re-fetch + `BeforeScan` + After). `create()`/`update()` no longer re-fetch; `read*`/`create`/`update`/`delete` use `hookCtx.Tx`. websocketspec `FetchRowNumber` keeps its public signature and delegates to a new tx-aware `fetchRowNumber`. A failure in begin/`OnTxBegin`/commit answers `transaction_error` with no detail. Tests: `pkg/websocketspec/tx_test.go` (sqlmock), `pkg/mqttspec/tx_test.go` (sqlite); mqttspec `update` tests now pass `Tx`.
- DECIDED in P4 (follow `AfterRead` question above): websocketspec/mqttspec run `AfterRead` inside the read tx (keeps "read has no second tx").
- DONE P5: resolvemcp. `OnTxBegin`, `HookContext.SetTx`, `Handler.runInTx`. Read = 1 tx (`BeforeRead`, count, scan, `AfterRead`; `readInTx`). Delete = 1 tx (`BeforeDelete` moved inside, after `OnTxBegin`). Create (single and batch, unified) = tx 1 (`BeforeCreate` + inserts) then tx 2 (re-fetch + `AfterCreate`); the old single-record pool insert/re-fetch is gone. Update = tx 1 (select, `BeforeUpdate`, update, `AfterUpdate`) then tx 2 (re-fetch). `BeforeHandle` still runs before any tx with `Tx = h.db`. Tests: `pkg/resolvemcp/tx_test.go` (sqlmock).
- DONE P6: funcspec. `OnTxBegin`, `HookContext.SetTx`, `Handler.runInTx` for `SqlQuery` and `SqlQueryList`. `BeforeResponse` now runs in a second short tx (`Tx` is no longer the pool). `BeforeOp` is unchanged (still per statement). A begin/`OnTxBegin`/commit failure answers 500 `transaction_error` / "Transaction failed" (before, it returned with no response); body failures still answer via `sendError`. Tests: `pkg/funcspec/tx_test.go`.
- DONE (AfterRead, decided by user): restheadspec `AfterRead` now runs in a second short tx. Test: `pkg/restheadspec/read_tx_test.go`.
- DONE P7: `pkg/security/txsettings.go`: `SecurityList.SetTxSettings(fn)`, `StampTxSettings`, `ApplyTxSettings` (configurable map, decided by user; `set_config(name, value, true)`, value hex-encoded, name validated, Postgres only, fail closed). Every spec's `RegisterSecurityHooks` registers it on `OnTxBegin`. Tests: `pkg/security/txsettings_test.go`, `pkg/resolvespec/tx_settings_test.go`. Docs: `pkg/common/TRANSACTIONS.md`.
- DONE real-Postgres check (resolvespec, testserver via compose): create `tx=1 pooled=0`, read `tx=1 pooled=0`, update `tx=2 pooled=0` (was `pooled=1`), single delete `tx=1 pooled=0`, batch create/delete `tx=1 pooled=0`. Compose now uses host networking (bridge fails here): testserver on 8123, Postgres on 8124 (was 8080/5434); integration test DSNs updated. Smoke script covers read and update. websocketspec/mqttspec/resolvemcp/restheadspec/funcspec not measured on real Postgres.
- DONE regression tests: per-spec read/create/update/delete hook-on-tx, failure-rollback (Before*/After*/`OnTxBegin`) and second-tx tests in all six specs (`ops_tx_test.go`, `tx_test.go`, `read_tx_test.go`); stamping tests for resolvespec, resolvemcp, funcspec; pgsql adapter preload tests (same connection, error returned); source guard `pkg/common/tx_guard_test.go` (no direct `RunInTransaction`/`BeginTx`, no `Tx = h.db` beyond the allowlisted BeforeHandle placeholders, no pool statements in spec handlers).
- FIXED: resolvespec now fires `AfterRead` (in the read tx; `Result` = scanned slice for single and list) `AfterCreate` (in the create tx, per record, all four create paths) and `AfterDelete` (in the delete tx, once per request; a failure rolls the delete back). Before, neither fired, so `AfterRead` column-level security masking (registered by `RegisterSecurityHooks`) was silently skipped on resolvespec reads. Failing `AfterRead`/`AfterCreate` fails the request and rolls back. Tests: `pkg/resolvespec/ops_tx_test.go` (incl. end-to-end column hiding).
- FIXED: restheadspec total-count cache key now includes the record id; a read by id (total 1) no longer poisons the list total for the 2-minute TTL. Tests: `pkg/restheadspec/cache_key_test.go`. resolvespec is unaffected (its count runs before the id filter, so the total is the list total by design). The cache is still process-wide, so read tests call `resetTotalCache`.
- NOTE: `pkg/security` `TestDatabaseAuthenticator` fails with `-count=2` (also on `cd96404`, before this work); use `-count=1`.
## Tests
- Existing: per-spec `handler_test.go`, `hooks_test.go`, `integration_test.go`; models in `pkg/testmodels/business.go`; `dbtrace` unit tests.
- Done: delete tx tests (`pkg/*/delete_tx_test.go`, sqlmock, 1-conn pool detects pool use), and read/create/update/`OnTxBegin`/other-spec tests (see "DONE regression tests" in Progress). Still missing: `dbtrace` `pooled == 0` on real Postgres for all specs except resolvespec.
- Add per spec/op: hook `Tx` is not the pool; `OnTxBegin` fires once per tx, before other hooks; single-ID delete = 1 tx; `dbtrace` `pooled == 0` on the request path.
- Test data: reuse `pkg/testmodels`; **ask before generating new data** (per project rule).
- Regression: full `go test -race` for security, dbmanager, common, restheadspec, resolvespec, websocketspec, mqttspec, resolvemcp, funcspec. Known pre-existing failures: mqttspec integration (no DB).
## Risks
- Long tx if a hook does slow work inside it → hold connection longer; keep hooks fast.
- Pool of 1: nothing inside a tx may take a second pool connection (auth/security loads are outside; keep it so).
- Behavior change: After hooks no longer get the pool handle; hooks that relied on an independent connection break.
- websocket/mqtt long-lived connections: tx must be per message, never per connection.
## Done when
- `dbtrace` shows `pooled=0` for every handler op on a hooked model.
- RLS GUC set in `OnTxBegin` is visible to read, create, update, delete queries and hooks.
- No `Tx: h.db` / `hookCtx.Tx = h.db` left in spec handlers.
- DONE: websocketspec `BeforeDisconnect`/`AfterDisconnect` fire from `Connection.Close()` (single close path, `closedOnce`, so exactly once per registered connection however it closes: read error, write error, slow-consumer eviction, shutdown). Connection lifecycle, not DB: `Tx` is not set. The hook context is detached from the connection cancel (`context.WithoutCancel`) so `AfterDisconnect` still has a live context. Errors are logged and never block the close. A connection rejected by `BeforeConnect` gets no disconnect hooks. `ConnectionManager.Shutdown` now closes connections outside its lock (a hook calling `Count()` would have deadlocked). Allowlist in `TestEveryDefinedHookHasACallSite` is now empty. Tests: `pkg/websocketspec/connection_test.go`. mqttspec already fired these in `Handler.Shutdown`; its per-client disconnect is unchanged.
- DONE: column-level hide/mask columns are dropped from create/update payloads (`security.ApplyWriteColumnSecurity`); rules preloaded in `BeforeHandle` for create/update. resolvemcp update now runs `BeforeHandle`.
+12
View File
@@ -0,0 +1,12 @@
# Clients
| Dir | Language | Specs | Verified |
|---|---|---|---|
| `resolvespec-js` | TypeScript | ResolveSpec, HeaderSpec, FunctionSpec, WebSocketSpec | yes |
| `resolvespec-python` | Python >= 3.11 | ResolveSpec, HeaderSpec, FunctionSpec, WebSocketSpec | yes (61 tests) |
| `resolvespec-go` | Go | ResolveSpec, FunctionSpec | yes (`go test`) |
| `resolvespec-rs` | Rust | ResolveSpec, FunctionSpec | yes (`cargo test`) |
| `resolvespec-cs` | C# (.NET 8) | ResolveSpec, FunctionSpec | yes (`dotnet test`) |
| `resolvespec-dart` | Dart / Flutter | ResolveSpec, FunctionSpec | yes (`dart test`) |
Wire behaviour is identical across clients; FunctionSpec server quirks are listed in each README.
+2
View File
@@ -0,0 +1,2 @@
bin/
obj/
+43
View File
@@ -0,0 +1,43 @@
# ResolveSpec.Client (C#)
.NET 8 client for ResolveSpec (JSON body) and FunctionSpec. `System.Text.Json`, no other dependencies.
> Tests run with `DOTNET_ROLL_FORWARD=Major` when only a newer runtime than 8.0 is installed.
## Clients
| Type | Constructor | Methods |
|---|---|---|
| `ResolveSpecClient` | `(baseUrl, ClientOptions?)` | `GetMetadataAsync` `ReadAsync` `CreateAsync` `UpdateAsync` `DeleteAsync` |
| `FuncSpecClient` | `(baseUrl, ClientOptions?)` | `QueryAsync` `QueryListAsync` |
`ClientOptions`: `Token`, `Headers`, `Timeout`, `HttpClient`. Precedence: Content-Type < custom headers < bearer token.
## ResolveSpec
- `id`: int/long/string → URL, `IEnumerable<string>` → body.
- `Options` with nullable properties; wire names via `JsonPropertyName`.
- Result: `Response{Success, Data (JsonElement), Metadata}`; `resp.Decode<T>()`.
## FunctionSpec
- Routes are server-defined: pass the `path`.
- Params (`IDictionary<string, object?>`) → query string (enumerable → repeated keys, bool → `true`/`false`, null skipped).
- `FuncSpecOptions` → `X-*` headers: `Filters`, `SearchFilters`, `CustomSqlWhere`, `CustomSqlOr`, `Sort`, `Limit`, `Offset`, `Distinct`, `SkipCount`, `SkipCache`, `ResponseFormat`.
- `QueryListAsync` fills `Metadata` from `Content-Range`; 206 is success.
- Static helpers: `BuildHeaders`, `BuildQuery`, `EncodeHeaderValue`, `DecodeHeaderValue`.
## Server quirks
- `Sort` is raw SQL in ORDER BY (client sends `col ASC|DESC`).
- One search operator per column.
- Values starting `ZIP_` / `__` are base64-decoded by the server.
- Non-ASCII, control chars and edge spaces are auto-encoded (`ZIP_`).
## Errors
`ResolveSpecException{StatusCode, Message, Error{Code, Detail, Sql}}`.
## Test
`dotnet test tests/`
@@ -0,0 +1,187 @@
using System.Globalization;
using System.Text;
using System.Text.Json;
using System.Text.RegularExpressions;
namespace ResolveSpec;
/// <summary>
/// Options sent to funcspec endpoints as X-* headers.
/// Server behaviour (pkg/funcspec): Sort is inserted raw into ORDER BY (so it is sent as SQL terms);
/// only one search operator per column is kept; values starting with "ZIP_" or "__" are
/// base64-decoded by the server, so such plaintext values cannot be sent faithfully.
/// </summary>
public sealed class FuncSpecOptions
{
/// <summary>eq+AND -> X-FieldFilter; others X-SearchOp / X-SearchOr.</summary>
public List<FilterOption>? Filters { get; set; }
/// <summary>X-SearchFilter-{col}: text ILIKE.</summary>
public Dictionary<string, string>? SearchFilters { get; set; }
public string? CustomSqlWhere { get; set; }
public string? CustomSqlOr { get; set; }
public List<SortOption>? Sort { get; set; }
public int? Limit { get; set; }
public int? Offset { get; set; }
public bool? Distinct { get; set; }
public bool? SkipCount { get; set; }
public bool? SkipCache { get; set; }
/// <summary>simple | detail | syncfusion</summary>
public string? ResponseFormat { get; set; }
}
/// <summary>Client for user-defined SQL endpoints. Routes are defined by the server application.</summary>
public sealed class FuncSpecClient
{
readonly Transport _t;
public FuncSpecClient(string baseUrl, ClientOptions? options = null) => _t = new Transport(baseUrl, options);
static readonly Dictionary<string, string> OperatorMap = new()
{
["eq"] = "equals", ["neq"] = "notequals", ["gt"] = "greaterthan", ["gte"] = "greaterthanorequal",
["lt"] = "lessthan", ["lte"] = "lessthanorequal", ["like"] = "contains", ["ilike"] = "contains",
["contains"] = "contains", ["startswith"] = "beginswith", ["endswith"] = "endswith", ["in"] = "in",
["between"] = "between", ["between_inclusive"] = "betweeninclusive",
["is_null"] = "empty", ["is_not_null"] = "notempty",
};
static string Scalar(object? v) => v switch
{
null => "",
string s => s,
bool b => b ? "true" : "false",
JsonElement { ValueKind: JsonValueKind.Null } => "",
JsonElement e => e.ValueKind == JsonValueKind.String ? e.GetString() ?? "" : e.ToString(),
IFormattable f => f.ToString(null, CultureInfo.InvariantCulture),
_ => v.ToString() ?? "",
};
static string FilterValue(object? v) =>
v is System.Collections.IEnumerable list and not string
? string.Join(",", list.Cast<object?>().Select(Scalar))
: Scalar(v);
/// <summary>Base64 (UTF-8) with the ZIP_ prefix.</summary>
public static string EncodeHeaderValue(string v) => "ZIP_" + Convert.ToBase64String(Encoding.UTF8.GetBytes(v));
/// <summary>Decode a value that may carry a ZIP_ or __ prefix (nested allowed).</summary>
public static string DecodeHeaderValue(string v)
{
foreach (var p in new[] { "ZIP_", "__" })
{
if (!v.StartsWith(p, StringComparison.Ordinal)) continue;
var b64 = Regex.Replace(v[p.Length..], "[\n\r ]", "");
b64 = b64.PadRight(b64.Length + (4 - b64.Length % 4) % 4, '=');
try { return DecodeHeaderValue(Encoding.UTF8.GetString(Convert.FromBase64String(b64))); }
catch (FormatException) { return v; }
}
return v;
}
/// <summary>Encode values that are unsafe as raw header/query text (non-ASCII, control chars, edge spaces).</summary>
static string Safe(string v) =>
v != v.Trim() || v.Any(c => c > 127 || char.IsControl(c)) ? EncodeHeaderValue(v) : v;
/// <summary>Build the X-* headers understood by funcspec.ParseParameters.</summary>
public static Dictionary<string, string> BuildHeaders(FuncSpecOptions? o)
{
var h = new Dictionary<string, string>();
if (o == null) return h;
foreach (var f in o.Filters ?? new())
{
var logic = string.IsNullOrEmpty(f.LogicOperator) ? "AND" : f.LogicOperator;
var v = Safe(FilterValue(f.Value));
if (f.Operator == "eq" && logic == "AND") { h[$"X-FieldFilter-{f.Column}"] = v; continue; }
var op = OperatorMap.TryGetValue(f.Operator, out var m) ? m : f.Operator;
h[$"{(logic == "OR" ? "X-SearchOr" : "X-SearchOp")}-{op}-{f.Column}"] = v;
}
foreach (var (col, text) in o.SearchFilters ?? new()) h[$"X-SearchFilter-{col}"] = Safe(text);
if (!string.IsNullOrEmpty(o.CustomSqlWhere)) h["X-Custom-SQL-W"] = Safe(o.CustomSqlWhere);
if (!string.IsNullOrEmpty(o.CustomSqlOr)) h["X-Custom-SQL-Or"] = Safe(o.CustomSqlOr);
if (o.Sort is { Count: > 0 })
{
// funcspec puts this verbatim into ORDER BY
h["X-Sort"] = Safe(string.Join(",", o.Sort.Select(s =>
$"{s.Column} {(string.Equals(s.Direction, "desc", StringComparison.OrdinalIgnoreCase) ? "DESC" : "ASC")}")));
}
if (o.Limit != null) h["X-Limit"] = o.Limit.Value.ToString(CultureInfo.InvariantCulture);
if (o.Offset != null) h["X-Offset"] = o.Offset.Value.ToString(CultureInfo.InvariantCulture);
if (o.Distinct != null) h["X-Distinct"] = Bool(o.Distinct.Value);
if (o.SkipCount != null) h["X-SkipCount"] = Bool(o.SkipCount.Value);
if (o.SkipCache != null) h["X-SkipCache"] = Bool(o.SkipCache.Value);
switch (o.ResponseFormat)
{
case "simple": h["X-SimpleApi"] = "true"; break;
case "detail": h["X-DetailApi"] = "true"; break;
case "syncfusion": h["X-Syncfusion"] = "true"; break;
}
return h;
}
static string Bool(bool b) => b ? "true" : "false";
/// <summary>Build query-string pairs: bools -> true/false, lists -> repeated keys, null skipped.</summary>
public static List<KeyValuePair<string, string>> BuildQuery(IDictionary<string, object?>? p)
{
var o = new List<KeyValuePair<string, string>>();
foreach (var (k, v) in p ?? new Dictionary<string, object?>())
{
if (v == null) continue;
if (v is System.Collections.IEnumerable list and not string)
foreach (var e in list) o.Add(new(k, Safe(Scalar(e))));
else o.Add(new(k, Safe(Scalar(v))));
}
return o;
}
static readonly Regex ContentRange = new(@"(\d+)-(\d+)/(\d+)");
static Metadata MetadataFrom(string? contentRange, FuncSpecOptions? o)
{
var m = new Metadata { Limit = o?.Limit ?? 0 };
var g = ContentRange.Match(contentRange ?? "");
if (g.Success)
{
var start = long.Parse(g.Groups[1].Value, CultureInfo.InvariantCulture);
var end = long.Parse(g.Groups[2].Value, CultureInfo.InvariantCulture);
var total = long.Parse(g.Groups[3].Value, CultureInfo.InvariantCulture);
m.Total = total; m.Filtered = total; m.Count = end - start; m.Offset = start;
}
return m;
}
async Task<Response> CallAsync(HttpMethod method, string path, IDictionary<string, object?>? p, FuncSpecOptions? o, bool list, CancellationToken ct)
{
var url = $"{_t.BaseUrl}/{path.TrimStart('/')}";
var q = BuildQuery(p);
if (q.Count > 0)
url += "?" + string.Join("&", q.Select(kv => $"{Uri.EscapeDataString(kv.Key)}={Uri.EscapeDataString(kv.Value)}"));
var (resp, text) = await _t.SendAsync(method, url, null, BuildHeaders(o), ct).ConfigureAwait(false);
var status = (int)resp.StatusCode;
if (!resp.IsSuccessStatusCode) throw Transport.ErrorFrom(status, text, resp.ReasonPhrase); // 206 is success
var r = new Response
{
Success = true,
Data = string.IsNullOrWhiteSpace(text) ? JsonDocument.Parse("null").RootElement.Clone() : JsonDocument.Parse(text).RootElement.Clone(),
};
if (list)
{
// Content-Range is a content header in HttpClient; fall back to response headers.
IEnumerable<string>? cr = null;
if (!resp.Content.Headers.TryGetValues("Content-Range", out cr)) resp.Headers.TryGetValues("Content-Range", out cr);
r.Metadata = MetadataFrom(cr?.FirstOrDefault(), o);
}
return r;
}
/// <summary>Single-record endpoint (SqlQuery). Data is the row object.</summary>
public Task<Response> QueryAsync(string path, IDictionary<string, object?>? p = null, FuncSpecOptions? o = null, HttpMethod? method = null, CancellationToken ct = default) =>
CallAsync(method ?? HttpMethod.Get, path, p, o, false, ct);
/// <summary>List endpoint (SqlQueryList). Metadata comes from Content-Range.</summary>
public Task<Response> QueryListAsync(string path, IDictionary<string, object?>? p = null, FuncSpecOptions? o = null, HttpMethod? method = null, CancellationToken ct = default) =>
CallAsync(method ?? HttpMethod.Get, path, p, o, true, ct);
}
+78
View File
@@ -0,0 +1,78 @@
using System.Net.Http.Headers;
using System.Text;
using System.Text.Json;
namespace ResolveSpec;
/// <summary>Shared HTTP configuration for both clients.</summary>
public sealed class ClientOptions
{
public string? Token { get; set; }
public Dictionary<string, string> Headers { get; } = new(StringComparer.OrdinalIgnoreCase);
public TimeSpan Timeout { get; set; } = TimeSpan.FromSeconds(30);
/// <summary>Supply your own HttpClient (tests, pooling). Its BaseAddress is ignored.</summary>
public HttpClient? HttpClient { get; set; }
}
internal sealed class Transport
{
public readonly string BaseUrl;
readonly ClientOptions _o;
readonly HttpClient _http;
public Transport(string baseUrl, ClientOptions? o)
{
BaseUrl = baseUrl.TrimEnd('/');
_o = o ?? new ClientOptions();
_http = _o.HttpClient ?? new HttpClient { Timeout = _o.Timeout };
}
/// <summary>Content-Type &lt; custom headers &lt; per-call headers &lt; bearer token.</summary>
public async Task<(HttpResponseMessage resp, string body)> SendAsync(
HttpMethod method, string url, string? json, IDictionary<string, string>? extra, CancellationToken ct)
{
using var req = new HttpRequestMessage(method, url);
if (json != null) req.Content = new StringContent(json, Encoding.UTF8, "application/json");
foreach (var (k, v) in _o.Headers) Set(req, k, v);
if (extra != null) foreach (var (k, v) in extra) Set(req, k, v);
if (!string.IsNullOrEmpty(_o.Token)) req.Headers.Authorization = new AuthenticationHeaderValue("Bearer", _o.Token);
var resp = await _http.SendAsync(req, ct).ConfigureAwait(false);
var body = await resp.Content.ReadAsStringAsync(ct).ConfigureAwait(false);
return (resp, body);
}
static void Set(HttpRequestMessage req, string name, string value)
{
req.Headers.Remove(name);
if (!req.Headers.TryAddWithoutValidation(name, value) && req.Content != null)
{
req.Content.Headers.Remove(name);
req.Content.Headers.TryAddWithoutValidation(name, value);
}
}
public static ResolveSpecException ErrorFrom(int status, string body, string? reason)
{
ApiError? err = null;
var isJson = false;
try
{
using var doc = JsonDocument.Parse(body);
isJson = true;
if (doc.RootElement.ValueKind == JsonValueKind.Object && doc.RootElement.TryGetProperty("error", out var e) && e.ValueKind == JsonValueKind.Object)
err = e.Deserialize<ApiError>();
}
catch (JsonException) { }
var message = err?.Message;
if (string.IsNullOrEmpty(message))
{
var text = isJson ? "" : body.Trim();
if (text.Length > 200) text = text[..200];
message = text.Length > 0 ? text : $"{reason ?? "Error"} ({status})";
}
return new ResolveSpecException(message, status, err);
}
public static string Segment(string s) => Uri.EscapeDataString(s);
}
@@ -0,0 +1,11 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<RootNamespace>ResolveSpec</RootNamespace>
<PackageId>ResolveSpec.Client</PackageId>
<Version>0.1.0</Version>
<Description>Client for ResolveSpec (JSON body) and FunctionSpec endpoints</Description>
</PropertyGroup>
</Project>
@@ -0,0 +1,65 @@
using System.Text.Json;
using System.Text.Json.Serialization;
namespace ResolveSpec;
/// <summary>Client for the ResolveSpec JSON body protocol: POST {operation, data, options}.</summary>
public sealed class ResolveSpecClient
{
static readonly JsonSerializerOptions Json = new() { DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull };
readonly Transport _t;
public ResolveSpecClient(string baseUrl, ClientOptions? options = null) => _t = new Transport(baseUrl, options);
sealed class Request
{
[JsonPropertyName("operation")] public string Operation { get; set; } = "";
[JsonPropertyName("id")] public string[]? Id { get; set; }
[JsonPropertyName("data")] public object? Data { get; set; }
[JsonPropertyName("options")] public Options? Options { get; set; }
}
// A single id (int/long/string) goes in the URL; string[] / IEnumerable<string> goes in the body.
static string? UrlId(object? id) => id switch
{
null => null,
string s => s,
IEnumerable<string> => null,
_ => Convert.ToString(id, System.Globalization.CultureInfo.InvariantCulture),
};
static string[]? BodyId(object? id) => id is IEnumerable<string> e ? e.ToArray() : null;
string Url(string schema, string entity, string? id)
{
var u = $"{_t.BaseUrl}/{Transport.Segment(schema)}/{Transport.Segment(entity)}";
return string.IsNullOrEmpty(id) ? u : $"{u}/{Transport.Segment(id)}";
}
async Task<Response> SendAsync(HttpMethod method, string url, Request? body, CancellationToken ct)
{
var json = body == null ? null : JsonSerializer.Serialize(body, Json);
var (resp, text) = await _t.SendAsync(method, url, json, null, ct).ConfigureAwait(false);
var status = (int)resp.StatusCode;
if (!resp.IsSuccessStatusCode) throw Transport.ErrorFrom(status, text, resp.ReasonPhrase);
var r = JsonSerializer.Deserialize<Response>(text, Json) ?? new Response();
if (!r.Success && r.Error != null) throw new ResolveSpecException(r.Error.Message, status, r.Error);
return r;
}
/// <summary>GET /{schema}/{entity}</summary>
public Task<Response> GetMetadataAsync(string schema, string entity, CancellationToken ct = default) =>
SendAsync(HttpMethod.Get, Url(schema, entity, null), null, ct);
public Task<Response> ReadAsync(string schema, string entity, object? id = null, Options? options = null, CancellationToken ct = default) =>
SendAsync(HttpMethod.Post, Url(schema, entity, UrlId(id)), new Request { Operation = "read", Id = BodyId(id), Options = options }, ct);
public Task<Response> CreateAsync(string schema, string entity, object data, Options? options = null, CancellationToken ct = default) =>
SendAsync(HttpMethod.Post, Url(schema, entity, null), new Request { Operation = "create", Data = data, Options = options }, ct);
public Task<Response> UpdateAsync(string schema, string entity, object data, object? id = null, Options? options = null, CancellationToken ct = default) =>
SendAsync(HttpMethod.Post, Url(schema, entity, UrlId(id)), new Request { Operation = "update", Id = BodyId(id), Data = data, Options = options }, ct);
public Task<Response> DeleteAsync(string schema, string entity, object id, CancellationToken ct = default) =>
SendAsync(HttpMethod.Post, Url(schema, entity, UrlId(id)), new Request { Operation = "delete" }, ct);
}
+137
View File
@@ -0,0 +1,137 @@
using System.Text.Json;
using System.Text.Json.Serialization;
namespace ResolveSpec;
// Types aligned with Go pkg/common/types.go. JsonPropertyName values are the wire names.
public sealed class FilterOption
{
[JsonPropertyName("column")] public string Column { get; set; } = "";
/// <summary>eq neq gt gte lt lte like ilike in contains startswith endswith between between_inclusive is_null is_not_null</summary>
[JsonPropertyName("operator")] public string Operator { get; set; } = "eq";
[JsonPropertyName("value")] public object? Value { get; set; }
/// <summary>AND | OR</summary>
[JsonPropertyName("logic_operator")] public string? LogicOperator { get; set; }
}
public sealed class SortOption
{
[JsonPropertyName("column")] public string Column { get; set; } = "";
/// <summary>asc | desc</summary>
[JsonPropertyName("direction")] public string Direction { get; set; } = "asc";
}
public sealed class Parameter
{
[JsonPropertyName("name")] public string Name { get; set; } = "";
[JsonPropertyName("value")] public string Value { get; set; } = "";
[JsonPropertyName("sequence")] public int? Sequence { get; set; }
}
public sealed class CustomOperator
{
[JsonPropertyName("name")] public string Name { get; set; } = "";
[JsonPropertyName("sql")] public string Sql { get; set; } = "";
}
public sealed class ComputedColumn
{
[JsonPropertyName("name")] public string Name { get; set; } = "";
[JsonPropertyName("expression")] public string Expression { get; set; } = "";
}
public sealed class PreloadOption
{
[JsonPropertyName("relation")] public string? Relation { get; set; }
[JsonPropertyName("table_name")] public string? TableName { get; set; }
[JsonPropertyName("columns")] public List<string>? Columns { get; set; }
[JsonPropertyName("omit_columns")] public List<string>? OmitColumns { get; set; }
[JsonPropertyName("sort")] public List<SortOption>? Sort { get; set; }
[JsonPropertyName("filters")] public List<FilterOption>? Filters { get; set; }
[JsonPropertyName("where")] public string? Where { get; set; }
[JsonPropertyName("limit")] public int? Limit { get; set; }
[JsonPropertyName("offset")] public int? Offset { get; set; }
[JsonPropertyName("updateable")] public bool? Updateable { get; set; }
[JsonPropertyName("computed_ql")] public Dictionary<string, string>? ComputedQl { get; set; }
[JsonPropertyName("recursive")] public bool? Recursive { get; set; }
[JsonPropertyName("primary_key")] public string? PrimaryKey { get; set; }
[JsonPropertyName("related_key")] public string? RelatedKey { get; set; }
[JsonPropertyName("foreign_key")] public string? ForeignKey { get; set; }
[JsonPropertyName("recursive_child_key")] public string? RecursiveChildKey { get; set; }
[JsonPropertyName("sql_joins")] public List<string>? SqlJoins { get; set; }
[JsonPropertyName("join_aliases")] public List<string>? JoinAliases { get; set; }
}
public sealed class VectorSearchOption
{
[JsonPropertyName("column")] public string Column { get; set; } = "";
[JsonPropertyName("vector")] public List<double> Vector { get; set; } = new();
/// <summary>l2 (default) | cosine | ip</summary>
[JsonPropertyName("metric")] public string? Metric { get; set; }
/// <summary>Distance column alias, default _distance.</summary>
[JsonPropertyName("as")] public string? As { get; set; }
[JsonPropertyName("direction")] public string? Direction { get; set; }
}
/// <summary>ResolveSpec request options object.</summary>
public sealed class Options
{
[JsonPropertyName("preload")] public List<PreloadOption>? Preload { get; set; }
[JsonPropertyName("columns")] public List<string>? Columns { get; set; }
[JsonPropertyName("omit_columns")] public List<string>? OmitColumns { get; set; }
[JsonPropertyName("filters")] public List<FilterOption>? Filters { get; set; }
[JsonPropertyName("sort")] public List<SortOption>? Sort { get; set; }
[JsonPropertyName("limit")] public int? Limit { get; set; }
[JsonPropertyName("offset")] public int? Offset { get; set; }
[JsonPropertyName("customOperators")] public List<CustomOperator>? CustomOperators { get; set; }
[JsonPropertyName("computedColumns")] public List<ComputedColumn>? ComputedColumns { get; set; }
[JsonPropertyName("parameters")] public List<Parameter>? Parameters { get; set; }
[JsonPropertyName("cursor_forward")] public string? CursorForward { get; set; }
[JsonPropertyName("cursor_backward")] public string? CursorBackward { get; set; }
[JsonPropertyName("fetch_row_number")] public string? FetchRowNumber { get; set; }
[JsonPropertyName("vector_search")] public VectorSearchOption? VectorSearch { get; set; }
}
public sealed class Metadata
{
[JsonPropertyName("total")] public long Total { get; set; }
[JsonPropertyName("count")] public long Count { get; set; }
[JsonPropertyName("filtered")] public long Filtered { get; set; }
[JsonPropertyName("limit")] public long Limit { get; set; }
[JsonPropertyName("offset")] public long Offset { get; set; }
}
public sealed class ApiError
{
[JsonPropertyName("code")] public string Code { get; set; } = "";
[JsonPropertyName("message")] public string Message { get; set; } = "";
[JsonPropertyName("details")] public JsonElement? Details { get; set; }
/// <summary>Server-side reason (funcspec / restheadspec).</summary>
[JsonPropertyName("detail")] public string? Detail { get; set; }
[JsonPropertyName("sql")] public string? Sql { get; set; }
}
/// <summary>ResolveSpec envelope. <see cref="Data"/> is raw JSON; use <see cref="Decode{T}"/>.</summary>
public sealed class Response
{
[JsonPropertyName("success")] public bool Success { get; set; }
[JsonPropertyName("data")] public JsonElement Data { get; set; }
[JsonPropertyName("metadata")] public Metadata? Metadata { get; set; }
[JsonPropertyName("error")] public ApiError? Error { get; set; }
public T? Decode<T>() => Data.ValueKind == JsonValueKind.Undefined ? default : Data.Deserialize<T>();
}
/// <summary>Thrown on a non-2xx response or an unsuccessful API result.</summary>
public sealed class ResolveSpecException : Exception
{
public int StatusCode { get; }
public ApiError Error { get; }
public ResolveSpecException(string message, int statusCode, ApiError? error = null) : base(message)
{
StatusCode = statusCode;
Error = error ?? new ApiError { Message = message };
}
}
+179
View File
@@ -0,0 +1,179 @@
using System.Net;
using System.Text;
using System.Text.Json;
using ResolveSpec;
using Xunit;
public class Stub : HttpMessageHandler
{
public HttpRequestMessage? Request;
public string Body = "";
readonly HttpStatusCode _status;
readonly string _json;
readonly Dictionary<string, string> _headers;
public Stub(HttpStatusCode status, string json, Dictionary<string, string>? headers = null)
{
_status = status; _json = json; _headers = headers ?? new();
}
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken ct)
{
Request = request;
Body = request.Content == null ? "" : await request.Content.ReadAsStringAsync(ct);
var r = new HttpResponseMessage(_status) { Content = new StringContent(_json, Encoding.UTF8, "application/json") };
foreach (var (k, v) in _headers)
if (!r.Headers.TryAddWithoutValidation(k, v)) r.Content.Headers.TryAddWithoutValidation(k, v);
return r;
}
}
public class ResolveSpecTests
{
static (ResolveSpecClient, Stub) Make(HttpStatusCode s, string json)
{
var stub = new Stub(s, json);
var o = new ClientOptions { Token = "tok", HttpClient = new HttpClient(stub) };
o.Headers["X-Tenant"] = "a";
return (new ResolveSpecClient("http://localhost:3000/", o), stub);
}
[Fact]
public async Task ReadPostsBody()
{
var (c, s) = Make(HttpStatusCode.OK, """{"success":true,"data":[{"id":1}]}""");
var r = await c.ReadAsync("public", "users", null, new Options { Limit = 5, Filters = new() { new FilterOption { Column = "a", Operator = "eq", Value = 1 } } });
Assert.Equal(HttpMethod.Post, s.Request!.Method);
Assert.Equal("/public/users", s.Request.RequestUri!.AbsolutePath);
Assert.Equal("Bearer tok", s.Request.Headers.Authorization!.ToString());
Assert.Equal("a", s.Request.Headers.GetValues("X-Tenant").Single());
using var body = JsonDocument.Parse(s.Body);
Assert.Equal("read", body.RootElement.GetProperty("operation").GetString());
Assert.Equal(5, body.RootElement.GetProperty("options").GetProperty("limit").GetInt32());
Assert.False(body.RootElement.TryGetProperty("id", out _));
Assert.Single(r.Decode<List<Dictionary<string, int>>>()!);
}
[Fact]
public async Task IdPlacement()
{
var (c, s) = Make(HttpStatusCode.OK, """{"success":true,"data":{}}""");
await c.ReadAsync("s", "e", 7);
Assert.Equal("/s/e/7", s.Request!.RequestUri!.AbsolutePath);
await c.UpdateAsync("s", "e", new { a = 1 }, new[] { "1", "2" });
Assert.Equal("/s/e", s.Request!.RequestUri!.AbsolutePath);
using (var b = JsonDocument.Parse(s.Body))
{
Assert.Equal(2, b.RootElement.GetProperty("id").GetArrayLength());
Assert.Equal("update", b.RootElement.GetProperty("operation").GetString());
}
await c.DeleteAsync("s", "e", "a/b");
Assert.Equal("/s/e/a%2Fb", s.Request!.RequestUri!.AbsoluteUri[(s.Request.RequestUri.AbsoluteUri.IndexOf("/s/e", StringComparison.Ordinal))..]);
Assert.Contains("\"delete\"", s.Body);
}
[Fact]
public async Task Errors()
{
var (c, _) = Make(HttpStatusCode.BadRequest, """{"success":false,"error":{"code":"x","message":"bad","detail":"why"}}""");
var e = await Assert.ThrowsAsync<ResolveSpecException>(() => c.ReadAsync("s", "e"));
Assert.Equal((400, "x", "bad", "why"), (e.StatusCode, e.Error.Code, e.Message, e.Error.Detail));
var (c2, _) = Make(HttpStatusCode.BadGateway, "bad gateway");
var e2 = await Assert.ThrowsAsync<ResolveSpecException>(() => c2.ReadAsync("s", "e"));
Assert.Equal((502, "bad gateway"), (e2.StatusCode, e2.Message));
var (c3, _) = Make(HttpStatusCode.OK, """{"success":false,"error":{"code":"c","message":"nope"}}""");
var e3 = await Assert.ThrowsAsync<ResolveSpecException>(() => c3.ReadAsync("s", "e"));
Assert.Equal("nope", e3.Message);
}
}
public class FuncSpecTests
{
[Fact]
public void HeaderFilters()
{
var h = FuncSpecClient.BuildHeaders(new FuncSpecOptions
{
Filters = new()
{
new() { Column = "status", Operator = "eq", Value = "active" },
new() { Column = "age", Operator = "gte", Value = 18 },
new() { Column = "name", Operator = "contains", Value = "x", LogicOperator = "OR" },
new() { Column = "deleted", Operator = "is_null" },
new() { Column = "id", Operator = "in", Value = new[] { 1, 2 } },
new() { Column = "p", Operator = "between_inclusive", Value = new[] { 1, 5 } },
},
});
Assert.Equal(new Dictionary<string, string>
{
["X-FieldFilter-status"] = "active",
["X-SearchOp-greaterthanorequal-age"] = "18",
["X-SearchOr-contains-name"] = "x",
["X-SearchOp-empty-deleted"] = "",
["X-SearchOp-in-id"] = "1,2",
["X-SearchOp-betweeninclusive-p"] = "1,5",
}, h);
}
[Fact]
public void HeaderMiscAndEncoding()
{
var h = FuncSpecClient.BuildHeaders(new FuncSpecOptions
{
SearchFilters = new() { ["name"] = "bob" }, CustomSqlWhere = "a = 1", CustomSqlOr = "b = 2",
Sort = new() { new() { Column = "name", Direction = "asc" }, new() { Column = "created_at", Direction = "DESC" } },
Limit = 5, Offset = 10, Distinct = true, SkipCount = true, SkipCache = false, ResponseFormat = "syncfusion",
});
Assert.Equal("name ASC,created_at DESC", h["X-Sort"]);
Assert.Equal("bob", h["X-SearchFilter-name"]);
Assert.Equal("a = 1", h["X-Custom-SQL-W"]);
Assert.Equal("false", h["X-SkipCache"]);
Assert.Equal("true", h["X-Syncfusion"]);
h = FuncSpecClient.BuildHeaders(new FuncSpecOptions { Filters = new()
{
new() { Column = "n", Operator = "eq", Value = "héllo" },
new() { Column = "m", Operator = "eq", Value = " pad" },
} });
Assert.StartsWith("ZIP_", h["X-FieldFilter-n"]);
Assert.Equal("héllo", FuncSpecClient.DecodeHeaderValue(h["X-FieldFilter-n"]));
Assert.Equal(" pad", FuncSpecClient.DecodeHeaderValue(h["X-FieldFilter-m"]));
}
[Fact]
public void QueryBuilding()
{
var q = FuncSpecClient.BuildQuery(new Dictionary<string, object?> { ["a"] = true, ["b"] = new[] { "x", "y" }, ["c"] = null, ["d"] = 3 });
Assert.Equal(new[] { "a=true", "b=x", "b=y", "d=3" }, q.Select(kv => $"{kv.Key}={kv.Value}"));
}
[Fact]
public async Task QueryListMetadata()
{
var stub = new Stub((HttpStatusCode)206, """[{"id":1},{"id":2}]""", new() { ["Content-Range"] = "items 10-12/50" });
var c = new FuncSpecClient("http://x", new ClientOptions { Token = "tok", HttpClient = new HttpClient(stub) });
var r = await c.QueryListAsync("/api/users", new Dictionary<string, object?> { ["org"] = 1 }, new FuncSpecOptions { Limit = 2 });
Assert.Equal("GET", stub.Request!.Method.Method);
Assert.Equal("/api/users", stub.Request.RequestUri!.AbsolutePath);
Assert.Equal("?org=1", stub.Request.RequestUri.Query);
Assert.Equal("2", stub.Request.Headers.GetValues("X-Limit").Single());
Assert.Equal((50L, 2L, 50L, 2L, 10L), (r.Metadata!.Total, r.Metadata.Count, r.Metadata.Filtered, r.Metadata.Limit, r.Metadata.Offset));
Assert.Equal(2, r.Data.GetArrayLength());
}
[Fact]
public async Task QuerySingleAndError()
{
var ok = new FuncSpecClient("http://x", new ClientOptions { HttpClient = new HttpClient(new Stub(HttpStatusCode.OK, """{"id":1}""")) });
var r = await ok.QueryAsync("api/u");
Assert.Null(r.Metadata);
Assert.Equal(1, r.Data.GetProperty("id").GetInt32());
var bad = new FuncSpecClient("http://x", new ClientOptions { HttpClient = new HttpClient(new Stub(HttpStatusCode.BadRequest,
"""{"success":false,"error":{"code":"hook_error","message":"Hook execution failed","detail":"authentication required"}}""")) });
var e = await Assert.ThrowsAsync<ResolveSpecException>(() => bad.QueryAsync("api/u"));
Assert.Equal(("hook_error", "authentication required"), (e.Error.Code, e.Error.Detail));
}
}
@@ -0,0 +1,16 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<IsPackable>false</IsPackable>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.11.1" />
<PackageReference Include="xunit" Version="2.9.2" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.8.2" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="../src/ResolveSpec.csproj" />
</ItemGroup>
</Project>
+3
View File
@@ -0,0 +1,3 @@
.dart_tool/
pubspec.lock
build/
+41
View File
@@ -0,0 +1,41 @@
# resolvespec (Dart)
Dart / Flutter client for ResolveSpec (JSON body) and FunctionSpec. Depends on `package:http`. Dart >= 3.3.
## Clients
| Type | Constructor | Methods |
|---|---|---|
| `ResolveSpecClient` | `(baseUrl, [ClientOptions])` | `getMetadata` `read` `create` `update` `delete` `close` |
| `FuncSpecClient` | `(baseUrl, [ClientOptions])` | `query` `queryList` `close` |
`ClientOptions(token:, headers:, timeout:, httpClient:)`. Precedence: Content-Type < custom headers < bearer token.
## ResolveSpec
- `id`: `int`/`String` → URL, `List<String>` → body. Named args: `id:`, `options:`.
- `Options`, `FilterOption(column, operator, [value, logic])`, `SortOption(column, [direction])`.
- Result: `Response{success, data (decoded JSON), metadata}`.
## FunctionSpec
- Routes are server-defined: pass the `path`.
- `params:` map → query string (list → repeated keys, null skipped).
- `FuncSpecOptions` → `X-*` headers: `filters`, `searchFilters`, `customSqlWhere`, `customSqlOr`, `sort`, `limit`, `offset`, `distinct`, `skipCount`, `skipCache`, `responseFormat`.
- `queryList` fills `metadata` from `Content-Range`; 206 is success.
- Helpers: `buildHeaders`, `buildQuery`, `encodeHeaderValue`, `decodeHeaderValue`.
## Server quirks
- `sort` is raw SQL in ORDER BY (client sends `col ASC|DESC`).
- One search operator per column.
- Values starting `ZIP_` / `__` are base64-decoded by the server.
- Non-ASCII, control chars and edge spaces are auto-encoded (`ZIP_`).
## Errors
`ResolveSpecException{statusCode, message, error: ApiError{code, detail, sql}}`.
## Test
`dart test`
@@ -0,0 +1 @@
include: package:lints/recommended.yaml
@@ -0,0 +1,7 @@
/// Client for ResolveSpec (JSON body) and FunctionSpec endpoints.
library;
export 'src/client.dart' show ClientOptions, ResolveSpecException;
export 'src/funcspec.dart';
export 'src/resolvespec.dart';
export 'src/types.dart';
@@ -0,0 +1,98 @@
import 'dart:convert';
import 'package:http/http.dart' as http;
import 'types.dart';
/// Thrown on a non-2xx response or an unsuccessful API result.
class ResolveSpecException implements Exception {
final int statusCode;
final String message;
final ApiError error;
ResolveSpecException(this.message, this.statusCode, [ApiError? error])
: error = error ?? ApiError(message: message);
@override
String toString() => 'ResolveSpecException($statusCode): $message';
}
/// Shared HTTP configuration for both clients.
class ClientOptions {
final String? token;
final Map<String, String> headers;
final Duration timeout;
/// Supply your own client (tests, pooling).
final http.Client? httpClient;
const ClientOptions(
{this.token,
this.headers = const {},
this.timeout = const Duration(seconds: 30),
this.httpClient});
}
class Transport {
final String baseUrl;
final ClientOptions options;
final http.Client _http;
Transport(String baseUrl, ClientOptions? options)
: baseUrl = baseUrl.replaceAll(RegExp(r'/+$'), ''),
options = options ?? const ClientOptions(),
_http = options?.httpClient ?? http.Client();
/// Content-Type < custom headers < per-call headers < bearer token.
Future<http.Response> send(String method, Uri uri,
{String? body, Map<String, String>? extra}) {
final headers = <String, String>{'Content-Type': 'application/json'};
void merge(Map<String, String> src) {
for (final e in src.entries) {
headers.removeWhere((k, _) => k.toLowerCase() == e.key.toLowerCase());
headers[e.key] = e.value;
}
}
merge(options.headers);
if (extra != null) merge(extra);
final token = options.token;
if (token != null && token.isNotEmpty) {
merge({'Authorization': 'Bearer $token'});
}
final req = http.Request(method, uri)..headers.addAll(headers);
if (body != null) req.body = body;
return _http
.send(req)
.timeout(options.timeout)
.then(http.Response.fromStream);
}
void close() => _http.close();
static ResolveSpecException errorFrom(http.Response resp) {
final body = utf8.decode(resp.bodyBytes, allowMalformed: true);
ApiError? err;
var isJson = false;
try {
final parsed = jsonDecode(body);
isJson = true;
if (parsed is Map<String, dynamic> &&
parsed['error'] is Map<String, dynamic>) {
err = ApiError.fromJson(parsed['error'] as Map<String, dynamic>);
}
} on FormatException {
// not JSON
}
var message = err?.message ?? '';
if (message.isEmpty) {
var text = isJson ? '' : body.trim();
if (text.length > 200) text = text.substring(0, 200);
message = text.isNotEmpty
? text
: '${resp.reasonPhrase ?? 'Error'} (${resp.statusCode})';
}
return ResolveSpecException(message, resp.statusCode, err);
}
}
@@ -0,0 +1,223 @@
import 'dart:convert';
import 'client.dart';
import 'types.dart';
/// Options sent to funcspec endpoints as X-* headers.
///
/// Server behaviour (pkg/funcspec): [sort] is inserted raw into ORDER BY (so it is sent as SQL
/// terms); only one search operator per column is kept; values starting with `ZIP_` or `__`
/// are base64-decoded by the server, so such plaintext values cannot be sent faithfully.
class FuncSpecOptions {
/// eq+AND -> X-FieldFilter; others X-SearchOp / X-SearchOr.
final List<FilterOption>? filters;
/// X-SearchFilter-{col}: text ILIKE.
final Map<String, String>? searchFilters;
final String? customSqlWhere;
final String? customSqlOr;
final List<SortOption>? sort;
final int? limit;
final int? offset;
final bool? distinct;
final bool? skipCount;
final bool? skipCache;
/// simple | detail | syncfusion
final String? responseFormat;
const FuncSpecOptions({
this.filters,
this.searchFilters,
this.customSqlWhere,
this.customSqlOr,
this.sort,
this.limit,
this.offset,
this.distinct,
this.skipCount,
this.skipCache,
this.responseFormat,
});
}
const _operatorMap = {
'eq': 'equals',
'neq': 'notequals',
'gt': 'greaterthan',
'gte': 'greaterthanorequal',
'lt': 'lessthan',
'lte': 'lessthanorequal',
'like': 'contains',
'ilike': 'contains',
'contains': 'contains',
'startswith': 'beginswith',
'endswith': 'endswith',
'in': 'in',
'between': 'between',
'between_inclusive': 'betweeninclusive',
'is_null': 'empty',
'is_not_null': 'notempty',
};
String _scalar(Object? v) => v == null ? '' : v.toString();
String _filterValue(Object? v) =>
v is Iterable ? v.map(_scalar).join(',') : _scalar(v);
/// Base64 (UTF-8) with the `ZIP_` prefix.
String encodeHeaderValue(String v) => 'ZIP_${base64.encode(utf8.encode(v))}';
/// Decode a value that may carry a `ZIP_` or `__` prefix (nested allowed).
String decodeHeaderValue(String v) {
for (final p in const ['ZIP_', '__']) {
if (v.startsWith(p)) {
var b64 = v.substring(p.length).replaceAll(RegExp(r'[\n\r ]'), '');
b64 = b64.padRight(b64.length + (4 - b64.length % 4) % 4, '=');
try {
return decodeHeaderValue(utf8.decode(base64.decode(b64)));
} on FormatException {
return v;
}
}
}
return v;
}
/// Encode values that are unsafe as raw header/query text (non-ASCII, control chars, edge spaces).
String _safe(String v) {
final unsafe =
v != v.trim() || v.runes.any((c) => c > 127 || c < 32 || c == 127);
return unsafe ? encodeHeaderValue(v) : v;
}
/// Build the X-* headers understood by funcspec.ParseParameters.
Map<String, String> buildHeaders(FuncSpecOptions? o) {
final h = <String, String>{};
if (o == null) return h;
for (final f in o.filters ?? const <FilterOption>[]) {
final logic = f.logicOperator ?? 'AND';
final v = _safe(_filterValue(f.value));
if (f.operator == 'eq' && logic == 'AND') {
h['X-FieldFilter-${f.column}'] = v;
} else {
final kind = logic == 'OR' ? 'X-SearchOr' : 'X-SearchOp';
h['$kind-${_operatorMap[f.operator] ?? f.operator}-${f.column}'] = v;
}
}
o.searchFilters
?.forEach((col, text) => h['X-SearchFilter-$col'] = _safe(text));
if (o.customSqlWhere != null && o.customSqlWhere!.isNotEmpty) {
h['X-Custom-SQL-W'] = _safe(o.customSqlWhere!);
}
if (o.customSqlOr != null && o.customSqlOr!.isNotEmpty) {
h['X-Custom-SQL-Or'] = _safe(o.customSqlOr!);
}
if (o.sort != null && o.sort!.isNotEmpty) {
// funcspec puts this verbatim into ORDER BY
h['X-Sort'] = _safe(o.sort!
.map((s) =>
'${s.column} ${s.direction.toLowerCase() == 'desc' ? 'DESC' : 'ASC'}')
.join(','));
}
if (o.limit != null) h['X-Limit'] = '${o.limit}';
if (o.offset != null) h['X-Offset'] = '${o.offset}';
if (o.distinct != null) h['X-Distinct'] = '${o.distinct}';
if (o.skipCount != null) h['X-SkipCount'] = '${o.skipCount}';
if (o.skipCache != null) h['X-SkipCache'] = '${o.skipCache}';
switch (o.responseFormat) {
case 'simple':
h['X-SimpleApi'] = 'true';
case 'detail':
h['X-DetailApi'] = 'true';
case 'syncfusion':
h['X-Syncfusion'] = 'true';
}
return h;
}
/// Build query-string pairs: lists -> repeated keys, null skipped, bools -> true/false.
Map<String, List<String>> buildQuery(Map<String, Object?>? params) {
final out = <String, List<String>>{};
params?.forEach((k, v) {
if (v == null) return;
out[k] = v is Iterable
? v.map((e) => _safe(_scalar(e))).toList()
: [_safe(_scalar(v))];
});
return out;
}
String? _header(Map<String, String> headers, String name) {
for (final e in headers.entries) {
if (e.key.toLowerCase() == name) return e.value;
}
return null;
}
final _contentRange = RegExp(r'(\d+)-(\d+)/(\d+)');
Metadata _metadata(String? contentRange, FuncSpecOptions? o) {
final m = _contentRange.firstMatch(contentRange ?? '');
if (m == null) return Metadata(limit: o?.limit ?? 0);
final start = int.parse(m.group(1)!);
final end = int.parse(m.group(2)!);
final total = int.parse(m.group(3)!);
return Metadata(
total: total,
count: end - start,
filtered: total,
limit: o?.limit ?? 0,
offset: start);
}
/// Client for user-defined SQL endpoints. Routes are defined by the server application.
class FuncSpecClient {
final Transport _t;
FuncSpecClient(String baseUrl, [ClientOptions? options])
: _t = Transport(baseUrl, options);
void close() => _t.close();
Future<Response> _call(String method, String path,
Map<String, Object?>? params, FuncSpecOptions? o, bool list) async {
final base =
Uri.parse('${_t.baseUrl}/${path.replaceAll(RegExp(r'^/+'), '')}');
final pairs = <String>[];
buildQuery(params).forEach((k, vs) {
for (final v in vs) {
pairs.add(
'${Uri.encodeQueryComponent(k)}=${Uri.encodeQueryComponent(v)}');
}
});
final uri = pairs.isEmpty ? base : base.replace(query: pairs.join('&'));
final resp = await _t.send(method, uri, extra: buildHeaders(o));
if (resp.statusCode < 200 || resp.statusCode > 299) {
throw Transport.errorFrom(resp); // 206 is success
}
final text = utf8.decode(resp.bodyBytes);
return Response(
success: true,
data: text.trim().isEmpty ? null : jsonDecode(text),
metadata:
list ? _metadata(_header(resp.headers, 'content-range'), o) : null,
);
}
/// Single-record endpoint (SqlQuery). `data` is the row object.
Future<Response> query(String path,
{Map<String, Object?>? params,
FuncSpecOptions? options,
String method = 'GET'}) =>
_call(method.toUpperCase(), path, params, options, false);
/// List endpoint (SqlQueryList). Metadata comes from Content-Range.
Future<Response> queryList(String path,
{Map<String, Object?>? params,
FuncSpecOptions? options,
String method = 'GET'}) =>
_call(method.toUpperCase(), path, params, options, true);
}
@@ -0,0 +1,88 @@
import 'dart:convert';
import 'client.dart';
import 'types.dart';
/// Client for the ResolveSpec JSON body protocol: POST {operation, data, options}.
///
/// A record `id` of type `int` or `String` goes in the URL; a `List<String>` goes in the body.
class ResolveSpecClient {
final Transport _t;
ResolveSpecClient(String baseUrl, [ClientOptions? options])
: _t = Transport(baseUrl, options);
void close() => _t.close();
static String? _urlId(Object? id) =>
id == null || id is List ? null : id.toString();
static List<String>? _bodyId(Object? id) =>
id is List ? id.map((e) => e.toString()).toList() : null;
Uri _url(String schema, String entity, String? id) {
var u =
'${_t.baseUrl}/${Uri.encodeComponent(schema)}/${Uri.encodeComponent(entity)}';
if (id != null && id.isNotEmpty) u += '/${Uri.encodeComponent(id)}';
return Uri.parse(u);
}
Future<Response> _send(
String method, Uri url, Map<String, dynamic>? body) async {
final resp = await _t.send(method, url,
body: body == null ? null : jsonEncode(body));
if (resp.statusCode < 200 || resp.statusCode > 299) {
throw Transport.errorFrom(resp);
}
final decoded = jsonDecode(utf8.decode(resp.bodyBytes));
final r = Response.fromJson(decoded as Map<String, dynamic>);
if (!r.success && r.error != null) {
throw ResolveSpecException(r.error!.message, resp.statusCode, r.error);
}
return r;
}
/// GET /{schema}/{entity}
Future<Response> getMetadata(String schema, String entity) =>
_send('GET', _url(schema, entity, null), null);
Future<Response> read(String schema, String entity,
{Object? id, Options? options}) =>
_send(
'POST',
_url(schema, entity, _urlId(id)),
{
'operation': 'read',
if (_bodyId(id) != null) 'id': _bodyId(id),
if (options != null) 'options': options.toJson()
},
);
Future<Response> create(String schema, String entity, Object data,
{Options? options}) =>
_send(
'POST',
_url(schema, entity, null),
{
'operation': 'create',
'data': data,
if (options != null) 'options': options.toJson()
},
);
Future<Response> update(String schema, String entity, Object data,
{Object? id, Options? options}) =>
_send(
'POST',
_url(schema, entity, _urlId(id)),
{
'operation': 'update',
if (_bodyId(id) != null) 'id': _bodyId(id),
'data': data,
if (options != null) 'options': options.toJson(),
},
);
Future<Response> delete(String schema, String entity, Object id) =>
_send('POST', _url(schema, entity, _urlId(id)), {'operation': 'delete'});
}
+287
View File
@@ -0,0 +1,287 @@
// Types aligned with Go pkg/common/types.go. toJson() emits the wire names.
Map<String, dynamic> _compact(Map<String, dynamic> m) {
m.removeWhere((_, v) => v == null);
return m;
}
class FilterOption {
final String column;
/// eq neq gt gte lt lte like ilike in contains startswith endswith between
/// between_inclusive is_null is_not_null
final String operator;
final Object? value;
/// AND | OR
final String? logicOperator;
const FilterOption(this.column, this.operator,
[this.value, this.logicOperator]);
Map<String, dynamic> toJson() => _compact({
'column': column,
'operator': operator,
'value': value,
'logic_operator': logicOperator,
});
}
class SortOption {
final String column;
/// asc | desc
final String direction;
const SortOption(this.column, [this.direction = 'asc']);
Map<String, dynamic> toJson() => {'column': column, 'direction': direction};
}
class Parameter {
final String name;
final String value;
final int? sequence;
const Parameter(this.name, this.value, [this.sequence]);
Map<String, dynamic> toJson() =>
_compact({'name': name, 'value': value, 'sequence': sequence});
}
class CustomOperator {
final String name;
final String sql;
const CustomOperator(this.name, this.sql);
Map<String, dynamic> toJson() => {'name': name, 'sql': sql};
}
class ComputedColumn {
final String name;
final String expression;
const ComputedColumn(this.name, this.expression);
Map<String, dynamic> toJson() => {'name': name, 'expression': expression};
}
class PreloadOption {
final String? relation;
final String? tableName;
final List<String>? columns;
final List<String>? omitColumns;
final List<SortOption>? sort;
final List<FilterOption>? filters;
final String? where;
final int? limit;
final int? offset;
final bool? updateable;
final Map<String, String>? computedQl;
final bool? recursive;
final String? primaryKey;
final String? relatedKey;
final String? foreignKey;
final String? recursiveChildKey;
final List<String>? sqlJoins;
final List<String>? joinAliases;
const PreloadOption({
this.relation,
this.tableName,
this.columns,
this.omitColumns,
this.sort,
this.filters,
this.where,
this.limit,
this.offset,
this.updateable,
this.computedQl,
this.recursive,
this.primaryKey,
this.relatedKey,
this.foreignKey,
this.recursiveChildKey,
this.sqlJoins,
this.joinAliases,
});
Map<String, dynamic> toJson() => _compact({
'relation': relation,
'table_name': tableName,
'columns': columns,
'omit_columns': omitColumns,
'sort': sort?.map((e) => e.toJson()).toList(),
'filters': filters?.map((e) => e.toJson()).toList(),
'where': where,
'limit': limit,
'offset': offset,
'updateable': updateable,
'computed_ql': computedQl,
'recursive': recursive,
'primary_key': primaryKey,
'related_key': relatedKey,
'foreign_key': foreignKey,
'recursive_child_key': recursiveChildKey,
'sql_joins': sqlJoins,
'join_aliases': joinAliases,
});
}
class VectorSearchOption {
final String column;
final List<double> vector;
/// l2 (default) | cosine | ip
final String? metric;
/// Distance column alias, default _distance.
final String? as;
final String? direction;
const VectorSearchOption(this.column, this.vector,
{this.metric, this.as, this.direction});
Map<String, dynamic> toJson() => _compact({
'column': column,
'vector': vector,
'metric': metric,
'as': as,
'direction': direction
});
}
/// ResolveSpec request options object.
class Options {
final List<PreloadOption>? preload;
final List<String>? columns;
final List<String>? omitColumns;
final List<FilterOption>? filters;
final List<SortOption>? sort;
final int? limit;
final int? offset;
final List<CustomOperator>? customOperators;
final List<ComputedColumn>? computedColumns;
final List<Parameter>? parameters;
final String? cursorForward;
final String? cursorBackward;
final String? fetchRowNumber;
final VectorSearchOption? vectorSearch;
const Options({
this.preload,
this.columns,
this.omitColumns,
this.filters,
this.sort,
this.limit,
this.offset,
this.customOperators,
this.computedColumns,
this.parameters,
this.cursorForward,
this.cursorBackward,
this.fetchRowNumber,
this.vectorSearch,
});
Map<String, dynamic> toJson() => _compact({
'preload': preload?.map((e) => e.toJson()).toList(),
'columns': columns,
'omit_columns': omitColumns,
'filters': filters?.map((e) => e.toJson()).toList(),
'sort': sort?.map((e) => e.toJson()).toList(),
'limit': limit,
'offset': offset,
'customOperators': customOperators?.map((e) => e.toJson()).toList(),
'computedColumns': computedColumns?.map((e) => e.toJson()).toList(),
'parameters': parameters?.map((e) => e.toJson()).toList(),
'cursor_forward': cursorForward,
'cursor_backward': cursorBackward,
'fetch_row_number': fetchRowNumber,
'vector_search': vectorSearch?.toJson(),
});
}
class Metadata {
final int total;
final int count;
final int filtered;
final int limit;
final int offset;
const Metadata(
{this.total = 0,
this.count = 0,
this.filtered = 0,
this.limit = 0,
this.offset = 0});
factory Metadata.fromJson(Map<String, dynamic> j) => Metadata(
total: (j['total'] as num?)?.toInt() ?? 0,
count: (j['count'] as num?)?.toInt() ?? 0,
filtered: (j['filtered'] as num?)?.toInt() ?? 0,
limit: (j['limit'] as num?)?.toInt() ?? 0,
offset: (j['offset'] as num?)?.toInt() ?? 0,
);
@override
bool operator ==(Object other) =>
other is Metadata &&
other.total == total &&
other.count == count &&
other.filtered == filtered &&
other.limit == limit &&
other.offset == offset;
@override
int get hashCode => Object.hash(total, count, filtered, limit, offset);
@override
String toString() =>
'Metadata(total: $total, count: $count, filtered: $filtered, limit: $limit, offset: $offset)';
}
class ApiError {
final String code;
final String message;
final Object? details;
/// Server-side reason (funcspec / restheadspec).
final String? detail;
final String? sql;
const ApiError(
{this.code = '', this.message = '', this.details, this.detail, this.sql});
factory ApiError.fromJson(Map<String, dynamic> j) => ApiError(
code: (j['code'] as String?) ?? '',
message: (j['message'] as String?) ?? '',
details: j['details'],
detail: j['detail'] as String?,
sql: j['sql'] as String?,
);
}
/// ResolveSpec envelope. [data] is the decoded JSON value (Map, List or scalar).
class Response {
final bool success;
final Object? data;
final Metadata? metadata;
final ApiError? error;
const Response({required this.success, this.data, this.metadata, this.error});
factory Response.fromJson(Map<String, dynamic> j) => Response(
success: j['success'] == true,
data: j['data'],
metadata: j['metadata'] is Map<String, dynamic>
? Metadata.fromJson(j['metadata'] as Map<String, dynamic>)
: null,
error: j['error'] is Map<String, dynamic>
? ApiError.fromJson(j['error'] as Map<String, dynamic>)
: null,
);
}
+14
View File
@@ -0,0 +1,14 @@
name: resolvespec
description: Client for ResolveSpec (JSON body) and FunctionSpec endpoints.
version: 0.1.0
publish_to: none
environment:
sdk: ">=3.3.0 <4.0.0"
dependencies:
http: ^1.2.0
dev_dependencies:
lints: ^4.0.0
test: ^1.25.0
@@ -0,0 +1,216 @@
import 'dart:convert';
import 'package:http/http.dart' as http;
import 'package:http/testing.dart';
import 'package:resolvespec/resolvespec.dart';
import 'package:test/test.dart';
(http.Client, List<http.Request>) stub(int status, Object body,
{Map<String, String> headers = const {}}) {
final seen = <http.Request>[];
final client = MockClient((req) async {
seen.add(req);
final text = body is String ? body : jsonEncode(body);
return http.Response(text, status,
headers: {'content-type': 'application/json', ...headers});
});
return (client, seen);
}
void main() {
group('resolvespec', () {
test('read posts body with headers', () async {
final (c, seen) = stub(200, {
'success': true,
'data': [
{'id': 1}
]
});
final client = ResolveSpecClient(
'http://localhost:3000/',
ClientOptions(token: 'tok', headers: {'X-Tenant': 'a'}, httpClient: c),
);
final r = await client.read('public', 'users',
options:
const Options(limit: 5, filters: [FilterOption('a', 'eq', 1)]));
final req = seen.single;
expect(req.method, 'POST');
expect(req.url.path, '/public/users');
expect(req.headers['authorization'], 'Bearer tok');
expect(req.headers['x-tenant'], 'a');
final body = jsonDecode(req.body) as Map<String, dynamic>;
expect(body['operation'], 'read');
expect(body['options']['limit'], 5);
expect(body.containsKey('id'), isFalse);
expect((r.data as List).length, 1);
});
test('id placement', () async {
final (c, seen) = stub(200, {'success': true, 'data': {}});
final client =
ResolveSpecClient('http://x', ClientOptions(httpClient: c));
await client.read('s', 'e', id: 7);
expect(seen.last.url.path, '/s/e/7');
await client.update('s', 'e', {'a': 1}, id: ['1', '2']);
expect(seen.last.url.path, '/s/e');
final b = jsonDecode(seen.last.body) as Map<String, dynamic>;
expect(b['id'], ['1', '2']);
expect(b['operation'], 'update');
await client.delete('s', 'e', 'a/b');
expect(seen.last.url.toString(), 'http://x/s/e/a%2Fb');
expect(jsonDecode(seen.last.body)['operation'], 'delete');
});
test('errors', () async {
final client = ResolveSpecClient(
'http://x',
ClientOptions(
httpClient: stub(400, {
'success': false,
'error': {'code': 'x', 'message': 'bad', 'detail': 'why'}
}).$1),
);
await expectLater(
client.read('s', 'e'),
throwsA(isA<ResolveSpecException>()
.having((e) => e.statusCode, 'status', 400)
.having((e) => e.error.code, 'code', 'x')
.having((e) => e.message, 'message', 'bad')
.having((e) => e.error.detail, 'detail', 'why')),
);
final plain = ResolveSpecClient(
'http://x', ClientOptions(httpClient: stub(502, 'bad gateway').$1));
await expectLater(
plain.read('s', 'e'),
throwsA(isA<ResolveSpecException>()
.having((e) => e.message, 'message', 'bad gateway')),
);
final soft = ResolveSpecClient(
'http://x',
ClientOptions(
httpClient: stub(200, {
'success': false,
'error': {'code': 'c', 'message': 'nope'}
}).$1),
);
await expectLater(
soft.read('s', 'e'),
throwsA(isA<ResolveSpecException>()
.having((e) => e.message, 'message', 'nope')));
});
});
group('funcspec', () {
test('header filters', () {
final h = buildHeaders(const FuncSpecOptions(filters: [
FilterOption('status', 'eq', 'active'),
FilterOption('age', 'gte', 18),
FilterOption('name', 'contains', 'x', 'OR'),
FilterOption('deleted', 'is_null'),
FilterOption('id', 'in', [1, 2]),
FilterOption('p', 'between_inclusive', [1, 5]),
]));
expect(h, {
'X-FieldFilter-status': 'active',
'X-SearchOp-greaterthanorequal-age': '18',
'X-SearchOr-contains-name': 'x',
'X-SearchOp-empty-deleted': '',
'X-SearchOp-in-id': '1,2',
'X-SearchOp-betweeninclusive-p': '1,5',
});
});
test('misc headers and encoding', () {
var h = buildHeaders(const FuncSpecOptions(
searchFilters: {'name': 'bob'},
customSqlWhere: 'a = 1',
customSqlOr: 'b = 2',
sort: [SortOption('name', 'asc'), SortOption('created_at', 'DESC')],
limit: 5,
offset: 10,
distinct: true,
skipCount: true,
skipCache: false,
responseFormat: 'syncfusion',
));
expect(h['X-Sort'], 'name ASC,created_at DESC');
expect(h['X-SearchFilter-name'], 'bob');
expect(h['X-Custom-SQL-W'], 'a = 1');
expect(h['X-SkipCache'], 'false');
expect(h['X-Syncfusion'], 'true');
h = buildHeaders(const FuncSpecOptions(filters: [
FilterOption('n', 'eq', 'héllo'),
FilterOption('m', 'eq', ' pad'),
]));
expect(h['X-FieldFilter-n'], startsWith('ZIP_'));
expect(decodeHeaderValue(h['X-FieldFilter-n']!), 'héllo');
expect(decodeHeaderValue(h['X-FieldFilter-m']!), ' pad');
});
test('query building', () {
expect(
buildQuery({
'a': true,
'b': ['x', 'y'],
'c': null,
'd': 3
}),
{
'a': ['true'],
'b': ['x', 'y'],
'd': ['3'],
});
});
test('queryList metadata', () async {
final (c, seen) = stub(206, [
{'id': 1},
{'id': 2}
], headers: {
'Content-Range': 'items 10-12/50'
});
final client = FuncSpecClient(
'http://x', ClientOptions(token: 'tok', httpClient: c));
final r = await client.queryList('/api/users',
params: {'org': 1}, options: const FuncSpecOptions(limit: 2));
expect(seen.single.method, 'GET');
expect(seen.single.url.path, '/api/users');
expect(seen.single.url.query, 'org=1');
expect(seen.single.headers['x-limit'], '2');
expect(
r.metadata,
const Metadata(
total: 50, count: 2, filtered: 50, limit: 2, offset: 10));
expect((r.data as List).length, 2);
});
test('query single and error', () async {
final ok = FuncSpecClient(
'http://x', ClientOptions(httpClient: stub(200, {'id': 1}).$1));
final r = await ok.query('api/u');
expect(r.metadata, isNull);
expect((r.data as Map)['id'], 1);
final bad = FuncSpecClient(
'http://x',
ClientOptions(
httpClient: stub(400, {
'success': false,
'error': {
'code': 'hook_error',
'message': 'Hook execution failed',
'detail': 'authentication required'
}
}).$1),
);
await expectLater(
bad.query('api/u'),
throwsA(isA<ResolveSpecException>()
.having((e) => e.error.code, 'code', 'hook_error')
.having(
(e) => e.error.detail, 'detail', 'authentication required')),
);
});
});
}
+40
View File
@@ -0,0 +1,40 @@
# resolvespec-go
Go client for ResolveSpec (JSON body) and FunctionSpec. Module: `github.com/bitechdev/ResolveSpec/clients/resolvespec-go`. Stdlib only.
## Clients
| Type | Constructor | Methods |
|---|---|---|
| `Client` | `NewClient(baseURL, opts...)` | `GetMetadata` `Read` `Create` `Update` `Delete` |
| `FuncSpecClient` | `NewFuncSpecClient(baseURL, opts...)` | `Query` `QueryList` `Do` |
Client options: `WithToken`, `WithHeader`, `WithHTTPClient`. Precedence: Content-Type < custom headers < bearer token.
## ResolveSpec
- All methods take `ctx`; `Read`/`Update`/`Delete` take `RecordID` (`nil`, int/string → URL, `[]string` → body).
- `Options` fields use pointers for optional ints/bools (`Int(n)`, `Bool(b)`).
- Result: `*Response{Success, Data (raw JSON), Metadata}`; `resp.Decode(&v)`.
## FunctionSpec
- Routes are server-defined: pass the `path`.
- `Params` → query string (slice → repeated keys, bool → `true`/`false`).
- `FuncSpecOptions` → `X-*` headers: `Filters`, `SearchFilters`, `CustomSQLWhere`, `CustomSQLOr`, `Sort`, `Limit`, `Offset`, `Distinct`, `SkipCount`, `SkipCache`, `ResponseFormat`.
- `QueryList` fills `Metadata` from `Content-Range` (`items a-b/total`); 206 is success.
## Server quirks
- `Sort` is raw SQL in ORDER BY (client sends `col ASC|DESC`).
- One search operator per column.
- Values starting `ZIP_` / `__` are base64-decoded by the server.
- Non-ASCII, control chars and edge spaces are auto-encoded (`ZIP_`).
## Errors
`*Error{StatusCode, APIError{Code, Message, Detail, SQL}}`.
## Test
`go test ./...`
+121
View File
@@ -0,0 +1,121 @@
package resolvespec
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"strings"
"time"
)
// APIError is the server error object.
type APIError struct {
Code string `json:"code"`
Message string `json:"message"`
Details any `json:"details,omitempty"`
Detail string `json:"detail,omitempty"` // server-side reason (funcspec / restheadspec)
SQL string `json:"sql,omitempty"`
}
// Error is returned on a non-2xx response or an unsuccessful result.
type Error struct {
StatusCode int
APIError
}
func (e *Error) Error() string {
if e.Message != "" {
return e.Message
}
return fmt.Sprintf("http %d", e.StatusCode)
}
type config struct {
baseURL string
token string
headers http.Header
http *http.Client
}
// Option configures a client.
type Option func(*config)
func WithToken(token string) Option { return func(c *config) { c.token = token } }
func WithHTTPClient(h *http.Client) Option { return func(c *config) { c.http = h } }
func WithHeader(name, value string) Option {
return func(c *config) { c.headers.Set(name, value) }
}
func newConfig(baseURL string, opts []Option) config {
c := config{baseURL: strings.TrimRight(baseURL, "/"), headers: http.Header{}, http: &http.Client{Timeout: 30 * time.Second}}
for _, o := range opts {
o(&c)
}
return c
}
// headers: Content-Type < custom headers < bearer token.
func (c *config) newRequest(ctx context.Context, method, u string, body []byte) (*http.Request, error) {
var r io.Reader
if body != nil {
r = bytes.NewReader(body)
}
req, err := http.NewRequestWithContext(ctx, method, u, r)
if err != nil {
return nil, err
}
req.Header.Set("Content-Type", "application/json")
for k, vs := range c.headers {
req.Header[k] = append([]string(nil), vs...)
}
if c.token != "" {
req.Header.Set("Authorization", "Bearer "+c.token)
}
return req, nil
}
func (c *config) do(req *http.Request) (*http.Response, []byte, error) {
resp, err := c.http.Do(req)
if err != nil {
return nil, nil, err
}
defer resp.Body.Close()
b, err := io.ReadAll(resp.Body)
return resp, b, err
}
func errorFrom(status int, body []byte) *Error {
e := &Error{StatusCode: status}
var env struct {
Error *APIError `json:"error"`
}
if json.Unmarshal(body, &env) == nil && env.Error != nil {
e.APIError = *env.Error
}
if e.Message == "" {
text := ""
if !json.Valid(body) {
text = strings.TrimSpace(string(body))
if len(text) > 200 {
text = text[:200]
}
}
if text == "" {
text = fmt.Sprintf("%s (%d)", http.StatusText(status), status)
}
e.Message = text
}
return e
}
func buildURL(base, schema, entity string, id string) string {
u := base + "/" + url.PathEscape(schema) + "/" + url.PathEscape(entity)
if id != "" {
u += "/" + url.PathEscape(id)
}
return u
}
+274
View File
@@ -0,0 +1,274 @@
package resolvespec
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"net/http"
"net/url"
"regexp"
"strconv"
"strings"
"unicode"
)
// FuncSpecOptions are sent to funcspec endpoints as X-* headers.
//
// Server behaviour (pkg/funcspec): Sort is inserted raw into ORDER BY (so it is sent as SQL
// terms); only one search operator per column is kept; values starting with "ZIP_" or "__"
// are base64-decoded by the server, so such plaintext values cannot be sent faithfully.
type FuncSpecOptions struct {
Filters []FilterOption // eq+AND -> X-FieldFilter; others X-SearchOp / X-SearchOr
SearchFilters map[string]string // X-SearchFilter-{col}: text ILIKE
CustomSQLWhere string // X-Custom-SQL-W
CustomSQLOr string // X-Custom-SQL-Or
Sort []SortOption
Limit *int
Offset *int
Distinct *bool
SkipCount *bool
SkipCache *bool
ResponseFormat string // simple | detail | syncfusion
}
// Params are query-string values. Slice values are sent as repeated keys (server: IN filter).
type Params map[string]any
// FuncSpecClient calls user-defined SQL endpoints. Routes are defined by the server app.
type FuncSpecClient struct{ cfg config }
func NewFuncSpecClient(baseURL string, opts ...Option) *FuncSpecClient {
return &FuncSpecClient{cfg: newConfig(baseURL, opts)}
}
var operatorMap = map[string]string{
"eq": "equals", "neq": "notequals", "gt": "greaterthan", "gte": "greaterthanorequal",
"lt": "lessthan", "lte": "lessthanorequal", "like": "contains", "ilike": "contains",
"contains": "contains", "startswith": "beginswith", "endswith": "endswith", "in": "in",
"between": "between", "between_inclusive": "betweeninclusive",
"is_null": "empty", "is_not_null": "notempty",
}
func scalar(v any) string {
switch x := v.(type) {
case nil:
return ""
case bool:
return strconv.FormatBool(x)
case string:
return x
case fmt.Stringer:
return x.String()
}
return fmt.Sprint(v)
}
func filterValue(v any) string {
switch x := v.(type) {
case nil:
return ""
case []string:
return strings.Join(x, ",")
case []int:
parts := make([]string, len(x))
for i, n := range x {
parts[i] = strconv.Itoa(n)
}
return strings.Join(parts, ",")
case []any:
parts := make([]string, len(x))
for i, n := range x {
parts[i] = scalar(n)
}
return strings.Join(parts, ",")
}
return scalar(v)
}
// EncodeHeaderValue base64-encodes (UTF-8) with the ZIP_ prefix.
func EncodeHeaderValue(v string) string { return "ZIP_" + base64.StdEncoding.EncodeToString([]byte(v)) }
// DecodeHeaderValue decodes a value that may carry a ZIP_ or __ prefix (nested allowed).
func DecodeHeaderValue(v string) string {
for _, p := range []string{"ZIP_", "__"} {
if strings.HasPrefix(v, p) {
b64 := strings.NewReplacer("\n", "", "\r", "", " ", "").Replace(v[len(p):])
for len(b64)%4 != 0 {
b64 += "="
}
raw, err := base64.StdEncoding.DecodeString(b64)
if err != nil {
return v
}
return DecodeHeaderValue(string(raw))
}
}
return v
}
// safe encodes values that are unsafe as raw header/query text (non-ASCII, control chars, edge spaces).
func safe(v string) string {
if v != strings.TrimSpace(v) {
return EncodeHeaderValue(v)
}
for _, r := range v {
if r > unicode.MaxASCII || !unicode.IsPrint(r) {
return EncodeHeaderValue(v)
}
}
return v
}
// BuildHeaders builds the X-* headers understood by funcspec.ParseParameters.
func BuildHeaders(o *FuncSpecOptions) map[string]string {
h := map[string]string{}
if o == nil {
return h
}
for _, f := range o.Filters {
logic := f.LogicOperator
if logic == "" {
logic = "AND"
}
v := safe(filterValue(f.Value))
if f.Operator == "eq" && logic == "AND" {
h["X-FieldFilter-"+f.Column] = v
continue
}
op := operatorMap[f.Operator]
if op == "" {
op = f.Operator
}
kind := "X-SearchOp"
if logic == "OR" {
kind = "X-SearchOr"
}
h[kind+"-"+op+"-"+f.Column] = v
}
for col, text := range o.SearchFilters {
h["X-SearchFilter-"+col] = safe(text)
}
if o.CustomSQLWhere != "" {
h["X-Custom-SQL-W"] = safe(o.CustomSQLWhere)
}
if o.CustomSQLOr != "" {
h["X-Custom-SQL-Or"] = safe(o.CustomSQLOr)
}
if len(o.Sort) > 0 {
terms := make([]string, len(o.Sort))
for i, s := range o.Sort {
dir := "ASC"
if strings.EqualFold(s.Direction, "desc") {
dir = "DESC"
}
terms[i] = s.Column + " " + dir // funcspec puts this verbatim into ORDER BY
}
h["X-Sort"] = safe(strings.Join(terms, ","))
}
if o.Limit != nil {
h["X-Limit"] = strconv.Itoa(*o.Limit)
}
if o.Offset != nil {
h["X-Offset"] = strconv.Itoa(*o.Offset)
}
for name, v := range map[string]*bool{"X-Distinct": o.Distinct, "X-SkipCount": o.SkipCount, "X-SkipCache": o.SkipCache} {
if v != nil {
h[name] = strconv.FormatBool(*v)
}
}
switch o.ResponseFormat {
case "simple":
h["X-SimpleApi"] = "true"
case "detail":
h["X-DetailApi"] = "true"
case "syncfusion":
h["X-Syncfusion"] = "true"
}
return h
}
// BuildQuery builds query-string values: bools -> true/false, slices -> repeated keys, nil skipped.
func BuildQuery(p Params) url.Values {
q := url.Values{}
for k, v := range p {
switch x := v.(type) {
case nil:
case []string:
for _, e := range x {
q.Add(k, safe(e))
}
case []int:
for _, e := range x {
q.Add(k, strconv.Itoa(e))
}
case []any:
for _, e := range x {
q.Add(k, safe(scalar(e)))
}
default:
q.Add(k, safe(scalar(v)))
}
}
return q
}
var contentRange = regexp.MustCompile(`(\d+)-(\d+)/(\d+)`)
func metadata(h http.Header, o *FuncSpecOptions) *Metadata {
m := &Metadata{}
if g := contentRange.FindStringSubmatch(h.Get("Content-Range")); g != nil {
start, _ := strconv.ParseInt(g[1], 10, 64)
end, _ := strconv.ParseInt(g[2], 10, 64)
total, _ := strconv.ParseInt(g[3], 10, 64)
m.Total, m.Count, m.Filtered, m.Offset = total, end-start, total, int(start)
}
if o != nil && o.Limit != nil {
m.Limit = *o.Limit
}
return m
}
func (c *FuncSpecClient) call(ctx context.Context, method, path string, p Params, o *FuncSpecOptions, withMeta bool) (*Response, error) {
u := c.cfg.baseURL + "/" + strings.TrimLeft(path, "/")
if q := BuildQuery(p); len(q) > 0 {
u += "?" + q.Encode()
}
req, err := c.cfg.newRequest(ctx, strings.ToUpper(method), u, nil)
if err != nil {
return nil, err
}
for k, v := range BuildHeaders(o) {
req.Header.Set(k, v)
}
resp, b, err := c.cfg.do(req)
if err != nil {
return nil, err
}
if resp.StatusCode < 200 || resp.StatusCode > 299 { // 206 is success
return nil, errorFrom(resp.StatusCode, b)
}
out := &Response{Success: true, Data: json.RawMessage(b)}
if len(b) == 0 {
out.Data = json.RawMessage("null")
}
if withMeta {
out.Metadata = metadata(resp.Header, o)
}
return out, nil
}
// Query calls a single-record endpoint (SqlQuery). Data is the row object.
func (c *FuncSpecClient) Query(ctx context.Context, path string, p Params, o *FuncSpecOptions) (*Response, error) {
return c.call(ctx, http.MethodGet, path, p, o, false)
}
// QueryList calls a list endpoint (SqlQueryList). Metadata comes from Content-Range.
func (c *FuncSpecClient) QueryList(ctx context.Context, path string, p Params, o *FuncSpecOptions) (*Response, error) {
return c.call(ctx, http.MethodGet, path, p, o, true)
}
// Do is like Query/QueryList with an explicit HTTP method (routes are app-defined).
func (c *FuncSpecClient) Do(ctx context.Context, method, path string, p Params, o *FuncSpecOptions, list bool) (*Response, error) {
return c.call(ctx, method, path, p, o, list)
}
+98
View File
@@ -0,0 +1,98 @@
package resolvespec
import (
"context"
"reflect"
"testing"
)
func TestBuildHeadersFilters(t *testing.T) {
got := BuildHeaders(&FuncSpecOptions{Filters: []FilterOption{
{Column: "status", Operator: "eq", Value: "active"},
{Column: "age", Operator: "gte", Value: 18},
{Column: "name", Operator: "contains", Value: "x", LogicOperator: "OR"},
{Column: "deleted", Operator: "is_null"},
{Column: "id", Operator: "in", Value: []int{1, 2}},
{Column: "p", Operator: "between_inclusive", Value: []any{1, 5}},
}})
want := map[string]string{
"X-FieldFilter-status": "active",
"X-SearchOp-greaterthanorequal-age": "18",
"X-SearchOr-contains-name": "x",
"X-SearchOp-empty-deleted": "",
"X-SearchOp-in-id": "1,2",
"X-SearchOp-betweeninclusive-p": "1,5",
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("%v", got)
}
}
func TestBuildHeadersMisc(t *testing.T) {
got := BuildHeaders(&FuncSpecOptions{
SearchFilters: map[string]string{"name": "bob"}, CustomSQLWhere: "a = 1", CustomSQLOr: "b = 2",
Sort: []SortOption{{"name", "asc"}, {"created_at", "DESC"}},
Limit: Int(5), Offset: Int(10), Distinct: Bool(true), SkipCount: Bool(true), SkipCache: Bool(false),
ResponseFormat: "syncfusion",
})
want := map[string]string{
"X-SearchFilter-name": "bob", "X-Custom-SQL-W": "a = 1", "X-Custom-SQL-Or": "b = 2",
"X-Sort": "name ASC,created_at DESC", "X-Limit": "5", "X-Offset": "10", "X-Distinct": "true",
"X-SkipCount": "true", "X-SkipCache": "false", "X-Syncfusion": "true",
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("%v", got)
}
}
func TestEncodeUnsafe(t *testing.T) {
h := BuildHeaders(&FuncSpecOptions{Filters: []FilterOption{{Column: "n", Operator: "eq", Value: "héllo"}, {Column: "m", Operator: "eq", Value: " pad"}}})
for _, k := range []string{"X-FieldFilter-n", "X-FieldFilter-m"} {
if len(h[k]) < 4 || h[k][:4] != "ZIP_" {
t.Fatalf("%s=%q", k, h[k])
}
}
if DecodeHeaderValue(h["X-FieldFilter-n"]) != "héllo" || DecodeHeaderValue(h["X-FieldFilter-m"]) != " pad" {
t.Fatal("roundtrip")
}
}
func TestBuildQuery(t *testing.T) {
q := BuildQuery(Params{"a": true, "b": []string{"x", "y"}, "c": nil, "d": 3})
if q.Get("a") != "true" || !reflect.DeepEqual(q["b"], []string{"x", "y"}) || q.Has("c") || q.Get("d") != "3" {
t.Fatalf("%v", q)
}
}
func TestQueryListMetadata(t *testing.T) {
srv, s := server(t, 206, `[{"id":1},{"id":2}]`, map[string]string{"Content-Range": "items 10-12/50"})
c := NewFuncSpecClient(srv.URL, WithToken("tok"))
resp, err := c.QueryList(context.Background(), "/api/users", Params{"org": 1}, &FuncSpecOptions{Limit: Int(2)})
if err != nil {
t.Fatal(err)
}
if s.method != "GET" || s.path != "/api/users?org=1" || s.header.Get("X-Limit") != "2" {
t.Fatalf("%s %v", s.path, s.header)
}
m := resp.Metadata
if m.Total != 50 || m.Count != 2 || m.Offset != 10 || m.Limit != 2 || m.Filtered != 50 {
t.Fatalf("%+v", m)
}
var rows []map[string]any
if err := resp.Decode(&rows); err != nil || len(rows) != 2 {
t.Fatal(err)
}
}
func TestQuerySingleNoMetadataAndError(t *testing.T) {
srv, _ := server(t, 200, `{"id":1}`, nil)
resp, err := NewFuncSpecClient(srv.URL).Query(context.Background(), "api/u", nil, nil)
if err != nil || resp.Metadata != nil {
t.Fatalf("%v %v", resp, err)
}
srv2, _ := server(t, 400, `{"success":false,"error":{"code":"hook_error","message":"Hook execution failed","detail":"authentication required"}}`, nil)
_, err = NewFuncSpecClient(srv2.URL).Query(context.Background(), "api/u", nil, nil)
if e := err.(*Error); e.Code != "hook_error" || e.Detail != "authentication required" {
t.Fatalf("%#v", e)
}
}
+3
View File
@@ -0,0 +1,3 @@
module github.com/bitechdev/ResolveSpec/clients/resolvespec-go
go 1.22
+94
View File
@@ -0,0 +1,94 @@
package resolvespec
import (
"context"
"encoding/json"
"fmt"
"net/http"
)
// Client speaks the ResolveSpec JSON body protocol: POST {operation, data, options}.
type Client struct{ cfg config }
func NewClient(baseURL string, opts ...Option) *Client {
return &Client{cfg: newConfig(baseURL, opts)}
}
// RecordID is a single id (int or string, sent in the URL) or a []string (sent in the body).
type RecordID any
func urlID(id RecordID) string {
switch v := id.(type) {
case nil:
return ""
case []string:
return ""
case string:
return v
default:
return fmt.Sprint(v)
}
}
func bodyID(id RecordID) []string {
ids, _ := id.([]string)
return ids
}
type request struct {
Operation string `json:"operation"`
ID []string `json:"id,omitempty"`
Data any `json:"data,omitempty"`
Options *Options `json:"options,omitempty"`
}
func (c *Client) send(ctx context.Context, method, schema, entity, id string, body any) (*Response, error) {
var payload []byte
if body != nil {
var err error
if payload, err = json.Marshal(body); err != nil {
return nil, err
}
}
req, err := c.cfg.newRequest(ctx, method, buildURL(c.cfg.baseURL, schema, entity, id), payload)
if err != nil {
return nil, err
}
resp, b, err := c.cfg.do(req)
if err != nil {
return nil, err
}
if resp.StatusCode < 200 || resp.StatusCode > 299 {
return nil, errorFrom(resp.StatusCode, b)
}
var out Response
if err := json.Unmarshal(b, &out); err != nil {
return nil, err
}
if !out.Success && out.Error != nil {
return nil, &Error{StatusCode: resp.StatusCode, APIError: *out.Error}
}
return &out, nil
}
// GetMetadata returns table metadata (GET /{schema}/{entity}).
func (c *Client) GetMetadata(ctx context.Context, schema, entity string) (*Response, error) {
return c.send(ctx, http.MethodGet, schema, entity, "", nil)
}
// Read reads records; id may be nil, an int/string (URL) or []string (body).
func (c *Client) Read(ctx context.Context, schema, entity string, id RecordID, opts *Options) (*Response, error) {
return c.send(ctx, http.MethodPost, schema, entity, urlID(id), request{Operation: "read", ID: bodyID(id), Options: opts})
}
func (c *Client) Create(ctx context.Context, schema, entity string, data any, opts *Options) (*Response, error) {
return c.send(ctx, http.MethodPost, schema, entity, "", request{Operation: "create", Data: data, Options: opts})
}
func (c *Client) Update(ctx context.Context, schema, entity string, data any, id RecordID, opts *Options) (*Response, error) {
return c.send(ctx, http.MethodPost, schema, entity, urlID(id), request{Operation: "update", ID: bodyID(id), Data: data, Options: opts})
}
func (c *Client) Delete(ctx context.Context, schema, entity string, id RecordID) (*Response, error) {
return c.send(ctx, http.MethodPost, schema, entity, urlID(id), request{Operation: "delete"})
}
@@ -0,0 +1,99 @@
package resolvespec
import (
"context"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"reflect"
"testing"
)
type seen struct {
method, path string
header http.Header
body map[string]any
}
func server(t *testing.T, status int, body string, hdr map[string]string) (*httptest.Server, *seen) {
t.Helper()
s := &seen{}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
s.method, s.path, s.header = r.Method, r.URL.EscapedPath()+"?"+r.URL.RawQuery, r.Header
b, _ := io.ReadAll(r.Body)
if len(b) > 0 {
_ = json.Unmarshal(b, &s.body)
}
for k, v := range hdr {
w.Header().Set(k, v)
}
w.WriteHeader(status)
_, _ = w.Write([]byte(body))
}))
t.Cleanup(srv.Close)
return srv, s
}
func TestReadBody(t *testing.T) {
srv, s := server(t, 200, `{"success":true,"data":[{"id":1}]}`, nil)
c := NewClient(srv.URL+"/", WithToken("tok"), WithHeader("X-Tenant", "a"))
resp, err := c.Read(context.Background(), "public", "users", nil, &Options{Limit: Int(5), Filters: []FilterOption{{Column: "a", Operator: "eq", Value: 1}}})
if err != nil {
t.Fatal(err)
}
if s.method != "POST" || s.path != "/public/users?" {
t.Fatalf("got %s %s", s.method, s.path)
}
if s.header.Get("Authorization") != "Bearer tok" || s.header.Get("X-Tenant") != "a" {
t.Fatalf("headers %v", s.header)
}
if s.body["operation"] != "read" || s.body["options"].(map[string]any)["limit"] != float64(5) {
t.Fatalf("body %v", s.body)
}
var rows []map[string]any
if err := resp.Decode(&rows); err != nil || len(rows) != 1 {
t.Fatalf("decode %v %v", rows, err)
}
}
func TestIDPlacement(t *testing.T) {
srv, s := server(t, 200, `{"success":true,"data":{}}`, nil)
c := NewClient(srv.URL)
ctx := context.Background()
_, _ = c.Read(ctx, "s", "e", 7, nil)
if s.path != "/s/e/7?" || s.body["id"] != nil {
t.Fatalf("%s %v", s.path, s.body)
}
_, _ = c.Update(ctx, "s", "e", map[string]any{"a": 1}, []string{"1", "2"}, nil)
if s.path != "/s/e?" || !reflect.DeepEqual(s.body["id"], []any{"1", "2"}) || s.body["operation"] != "update" {
t.Fatalf("%s %v", s.path, s.body)
}
_, _ = c.Delete(ctx, "s", "e", "a/b")
if s.path != "/s/e/a%2Fb?" || s.body["operation"] != "delete" {
t.Fatalf("%s %v", s.path, s.body)
}
_, _ = c.GetMetadata(ctx, "s", "e")
if s.method != "GET" {
t.Fatal(s.method)
}
}
func TestErrors(t *testing.T) {
srv, _ := server(t, 400, `{"success":false,"error":{"code":"x","message":"bad","detail":"why"}}`, nil)
_, err := NewClient(srv.URL).Read(context.Background(), "s", "e", nil, nil)
e, ok := err.(*Error)
if !ok || e.StatusCode != 400 || e.Code != "x" || e.Message != "bad" || e.Detail != "why" {
t.Fatalf("%#v", err)
}
srv2, _ := server(t, 502, "bad gateway", nil)
_, err = NewClient(srv2.URL).Read(context.Background(), "s", "e", nil, nil)
if e := err.(*Error); e.StatusCode != 502 || e.Message != "bad gateway" {
t.Fatalf("%#v", e)
}
srv3, _ := server(t, 200, `{"success":false,"error":{"code":"c","message":"nope"}}`, nil)
_, err = NewClient(srv3.URL).Read(context.Background(), "s", "e", nil, nil)
if e := err.(*Error); e.Message != "nope" {
t.Fatalf("%#v", e)
}
}
+107
View File
@@ -0,0 +1,107 @@
// Package resolvespec is a client for ResolveSpec (JSON body) and FunctionSpec endpoints.
package resolvespec
import "encoding/json"
// FilterOption mirrors common.FilterOption. Operator: eq neq gt gte lt lte like ilike in
// contains startswith endswith between between_inclusive is_null is_not_null.
type FilterOption struct {
Column string `json:"column"`
Operator string `json:"operator"`
Value any `json:"value"`
LogicOperator string `json:"logic_operator,omitempty"` // AND | OR
}
type SortOption struct {
Column string `json:"column"`
Direction string `json:"direction"` // asc | desc
}
type Parameter struct {
Name string `json:"name"`
Value string `json:"value"`
Sequence int `json:"sequence,omitempty"`
}
type CustomOperator struct {
Name string `json:"name"`
SQL string `json:"sql"`
}
type ComputedColumn struct {
Name string `json:"name"`
Expression string `json:"expression"`
}
type PreloadOption struct {
Relation string `json:"relation,omitempty"`
TableName string `json:"table_name,omitempty"`
Columns []string `json:"columns,omitempty"`
OmitColumns []string `json:"omit_columns,omitempty"`
Sort []SortOption `json:"sort,omitempty"`
Filters []FilterOption `json:"filters,omitempty"`
Where string `json:"where,omitempty"`
Limit *int `json:"limit,omitempty"`
Offset *int `json:"offset,omitempty"`
Updateable *bool `json:"updateable,omitempty"`
ComputedQL map[string]string `json:"computed_ql,omitempty"`
Recursive bool `json:"recursive,omitempty"`
PrimaryKey string `json:"primary_key,omitempty"`
RelatedKey string `json:"related_key,omitempty"`
ForeignKey string `json:"foreign_key,omitempty"`
RecursiveChildKey string `json:"recursive_child_key,omitempty"`
SQLJoins []string `json:"sql_joins,omitempty"`
JoinAliases []string `json:"join_aliases,omitempty"`
}
type VectorSearchOption struct {
Column string `json:"column"`
Vector []float64 `json:"vector"`
Metric string `json:"metric,omitempty"` // l2 (default) | cosine | ip
As string `json:"as,omitempty"` // distance alias, default _distance
Direction string `json:"direction,omitempty"`
}
// Options is the ResolveSpec request options object.
type Options struct {
Preload []PreloadOption `json:"preload,omitempty"`
Columns []string `json:"columns,omitempty"`
OmitColumns []string `json:"omit_columns,omitempty"`
Filters []FilterOption `json:"filters,omitempty"`
Sort []SortOption `json:"sort,omitempty"`
Limit *int `json:"limit,omitempty"`
Offset *int `json:"offset,omitempty"`
CustomOperators []CustomOperator `json:"customOperators,omitempty"`
ComputedColumns []ComputedColumn `json:"computedColumns,omitempty"`
Parameters []Parameter `json:"parameters,omitempty"`
CursorForward string `json:"cursor_forward,omitempty"`
CursorBackward string `json:"cursor_backward,omitempty"`
FetchRowNumber string `json:"fetch_row_number,omitempty"`
VectorSearch *VectorSearchOption `json:"vector_search,omitempty"`
}
// Metadata of a list response.
type Metadata struct {
Total int64 `json:"total"`
Count int64 `json:"count"`
Filtered int64 `json:"filtered"`
Limit int `json:"limit"`
Offset int `json:"offset"`
}
// Response is the ResolveSpec envelope. Data is left raw for the caller to decode.
type Response struct {
Success bool `json:"success"`
Data json.RawMessage `json:"data"`
Metadata *Metadata `json:"metadata,omitempty"`
Error *APIError `json:"error,omitempty"`
}
// Decode unmarshals Data into v.
func (r *Response) Decode(v any) error { return json.Unmarshal(r.Data, v) }
// Int returns a pointer to n, for optional Options fields.
func Int(n int) *int { return &n }
// Bool returns a pointer to b.
func Bool(b bool) *bool { return &b }
@@ -106,6 +106,25 @@ await client.delete('public', 'users', '42');
| `X-Fetch-RowNumber` | `fetch_row_number` | string |
| `X-CQL-SEL-{col}` | `computedColumns` | expression |
| `X-Custom-SQL-W` | `customOperators` | SQL AND-joined |
| `X-Preload-Where` | `preload[].where` | applies to all preloads in `X-Preload`; differing wheres go to `X-Preload-{n}` + `X-Preload-{n}-Where` |
| `X-Expand` | `expand` | `Rel:col1,col2` pipe-separated (LEFT JOIN) |
| `X-Custom-SQL-Join` | `custom_sql_joins` | JOIN clauses, pipe-separated |
| `X-Custom-SQL-Or` | `custom_sql_or` | SQL OR-joined |
| `X-SearchCols` | `search_columns` | comma-separated |
| `X-AdvSQL-{col}` | `advanced_sql` | column -> SQL |
| `X-SpatialFilter-{col}` | `filters` (`st_dwithin`, `st_*`, `bbox`) | JSON `{op,value,logic}` |
| `X-VectorFilter-{col}` | `filters` (`l2_within`, `cosine_within`, `ip_within`) | JSON `{op,value,logic}` |
| `X-Vector-Search-{col}` / `-Vector` / `-As` / `-Dir` | `vector_search` | metric / JSON array / alias / asc\|desc |
| `X-Clean-JSON` | `clean_json` | bool |
| `X-Distinct` | `distinct` | bool |
| `X-SkipCount` / `X-SkipCache` | `skip_count` / `skip_cache` | bool |
| `X-PKRow` | `pk_row` | string |
| `X-SimpleApi` / `X-DetailApi` / `X-Syncfusion` | `response_format` | `simple` \| `detail` \| `syncfusion` |
| `X-Single-Record-As-Object` | `single_record_as_object` | bool (server default true) |
| `X-Transaction-Atomic` | `atomic_transaction` | bool |
| `X-Files` | `xfiles` | JSON, sent as `ZIP_` base64 |
Extended fields live on `HeaderSpecOptions` (extends `Options`); `vector_search` is on `Options`.
### Utility Functions
@@ -2,6 +2,101 @@ import { describe, it, expect, vi, beforeEach } from 'vitest';
import { buildHeaders, encodeHeaderValue, decodeHeaderValue, HeaderSpecClient, getHeaderSpecClient } from '../headerspec/client';
import type { Options, ClientConfig, APIResponse } from '../common/types';
describe('buildHeaders (extended restheadspec options)', () => {
it('should set X-Preload-Where when all preloads share one where', () => {
const h = buildHeaders({
preload: [
{ relation: 'Items', columns: ['id'], where: 'active = true' },
{ relation: 'Tags', where: 'active = true' },
],
});
expect(h['X-Preload']).toBe('Items:id|Tags');
expect(h['X-Preload-Where']).toBe('active = true');
});
it('should use numbered headers for mixed where clauses', () => {
const h = buildHeaders({
preload: [
{ relation: 'Items', where: 'a = 1' },
{ relation: 'Category' },
{ relation: 'Tags', where: 'b = 2' },
],
});
expect(h['X-Preload']).toBe('Category');
expect(h['X-Preload-Where']).toBeUndefined();
expect(h['X-Preload-1']).toBe('Items');
expect(h['X-Preload-1-Where']).toBe('a = 1');
expect(h['X-Preload-2']).toBe('Tags');
expect(h['X-Preload-2-Where']).toBe('b = 2');
});
it('should set expand, joins, or-sql, search cols, advsql', () => {
const h = buildHeaders({
expand: [{ relation: 'Dept', columns: ['id', 'name'] }, { relation: 'Role' }],
custom_sql_joins: ['LEFT JOIN a ON a.id = b.id', 'INNER JOIN c ON c.id = b.cid'],
custom_sql_or: ['x = 1', 'y = 2'],
search_columns: ['name', 'email'],
advanced_sql: { total: 'a + b' },
});
expect(h['X-Expand']).toBe('Dept:id,name|Role');
expect(h['X-Custom-SQL-Join']).toBe('LEFT JOIN a ON a.id = b.id|INNER JOIN c ON c.id = b.cid');
expect(h['X-Custom-SQL-Or']).toBe('x = 1 OR y = 2');
expect(h['X-SearchCols']).toBe('name,email');
expect(h['X-AdvSQL-total']).toBe('a + b');
});
it('should set boolean flags, pk row and response format', () => {
const h = buildHeaders({
clean_json: true,
distinct: true,
skip_count: true,
skip_cache: false,
atomic_transaction: true,
single_record_as_object: false,
pk_row: '42',
response_format: 'detail',
});
expect(h['X-Clean-JSON']).toBe('true');
expect(h['X-Distinct']).toBe('true');
expect(h['X-SkipCount']).toBe('true');
expect(h['X-SkipCache']).toBe('false');
expect(h['X-Transaction-Atomic']).toBe('true');
expect(h['X-Single-Record-As-Object']).toBe('false');
expect(h['X-PKRow']).toBe('42');
expect(h['X-DetailApi']).toBe('true');
});
it('should set spatial and vector filters as JSON', () => {
const h = buildHeaders({
filters: [
{ column: 'geom', operator: 'st_dwithin', value: { geom: 'POINT(0 0)', distance: 5 }, logic_operator: 'OR' },
{ column: 'emb', operator: 'cosine_within', value: { vector: [1, 2], distance: 0.3 } },
],
});
expect(JSON.parse(h['X-SpatialFilter-geom'])).toEqual({
op: 'st_dwithin', value: { geom: 'POINT(0 0)', distance: 5 }, logic: 'or',
});
expect(JSON.parse(h['X-VectorFilter-emb']).op).toBe('cosine_within');
});
it('should set vector search headers', () => {
const h = buildHeaders({
vector_search: { column: 'emb', vector: [0.1, 0.2], metric: 'cosine', as: 'dist', direction: 'desc' },
});
expect(h['X-Vector-Search-emb']).toBe('cosine');
expect(h['X-Vector-Search-Vector']).toBe('[0.1,0.2]');
expect(h['X-Vector-Search-As']).toBe('dist');
expect(h['X-Vector-Search-Dir']).toBe('desc');
});
it('should encode X-Files as ZIP_ JSON', () => {
const xf = { tablename: 'users', prefix: 'USR', limit: 10 };
const h = buildHeaders({ xfiles: xf });
expect(h['X-Files'].startsWith('ZIP_')).toBe(true);
expect(JSON.parse(decodeHeaderValue(h['X-Files']))).toEqual(xf);
});
});
describe('buildHeaders', () => {
it('should set X-Select-Fields for columns', () => {
const h = buildHeaders({ columns: ['id', 'name', 'email'] });
@@ -5,7 +5,11 @@ export type Operator =
| 'like' | 'ilike' | 'in'
| 'contains' | 'startswith' | 'endswith'
| 'between' | 'between_inclusive'
| 'is_null' | 'is_not_null';
| 'is_null' | 'is_not_null'
// PostGIS spatial (sent via X-SpatialFilter-{col})
| 'st_dwithin' | 'bbox'
// pgvector similarity (sent via X-VectorFilter-{col})
| 'l2_within' | 'cosine_within' | 'ip_within';
export type Operation = 'read' | 'create' | 'update' | 'delete';
export type SortDirection = 'asc' | 'desc' | 'ASC' | 'DESC';
@@ -61,6 +65,54 @@ export interface ComputedColumn {
expression: string;
}
export type VectorMetric = 'l2' | 'cosine' | 'ip';
export type ResponseFormat = 'simple' | 'detail' | 'syncfusion';
/** pgvector KNN search: order by distance between `column` and `vector`. */
export interface VectorSearchOption {
column: string;
vector: number[];
metric?: VectorMetric;
/** Distance column alias. Default `_distance` */
as?: string;
direction?: 'asc' | 'desc';
}
/** LEFT JOIN expansion of a relation (X-Expand). */
export interface ExpandOption {
relation: string;
columns?: string[];
}
/** X-Files configuration (Go restheadspec XFiles). Sent as a single JSON header. */
export interface XFiles {
tablename?: string;
schema?: string;
primarykey?: string;
foreignkey?: string;
relatedkey?: string;
sort?: string[];
prefix?: string;
editable?: boolean;
recursive?: boolean;
expand?: boolean;
rownumber?: boolean;
skipcount?: boolean;
offset?: number;
limit?: number;
columns?: string[];
omit_columns?: string[];
cql_columns?: string[];
sql_joins?: string[];
sql_or?: string[];
sql_and?: string[];
parenttables?: XFiles[];
childtables?: XFiles[];
filter_fields?: { field: string; value: string; operator: string }[];
cursor_forward?: string;
cursor_backward?: string;
}
export interface Options {
preload?: PreloadOption[];
columns?: string[];
@@ -75,6 +127,39 @@ export interface Options {
cursor_forward?: string;
cursor_backward?: string;
fetch_row_number?: string;
vector_search?: VectorSearchOption;
}
/** Options only available to the header-based (restheadspec) protocol. */
export interface HeaderSpecOptions extends Options {
/** X-Expand: LEFT JOIN relations */
expand?: ExpandOption[];
/** X-Custom-SQL-Join: raw JOIN clauses */
custom_sql_joins?: string[];
/** X-Custom-SQL-Or: raw SQL, OR-combined */
custom_sql_or?: string[];
/** X-SearchCols: columns for multi-column search */
search_columns?: string[];
/** X-AdvSQL-{col}: column -> SQL expression */
advanced_sql?: Record<string, string>;
/** X-Clean-JSON */
clean_json?: boolean;
/** X-Distinct */
distinct?: boolean;
/** X-SkipCount: skip total count query */
skip_count?: boolean;
/** X-SkipCache */
skip_cache?: boolean;
/** X-PKRow: primary key value of a row to fetch */
pk_row?: string;
/** X-SimpleApi / X-DetailApi / X-Syncfusion */
response_format?: ResponseFormat;
/** X-Single-Record-As-Object (server default true) */
single_record_as_object?: boolean;
/** X-Transaction-Atomic */
atomic_transaction?: boolean;
/** X-Files: single JSON configuration */
xfiles?: XFiles;
}
export interface RequestBody {
@@ -5,7 +5,7 @@ import type {
ClientConfig,
CustomOperator,
FilterOption,
Options,
HeaderSpecOptions,
PreloadOption,
SortOption,
} from "../common/types";
@@ -59,8 +59,15 @@ function decodeBase64(str: string): string {
* - X-Fetch-RowNumber: row number fetch
* - X-CQL-SEL-{col}: computed columns
* - X-Custom-SQL-W: custom operators (AND)
* - X-Preload-Where: where for X-Preload (extra where groups use X-Preload-{n}[-Where])
* - X-SpatialFilter-{col} / X-VectorFilter-{col}: JSON {op,value,logic}
* - X-Vector-Search-{col|vector|as|dir}: pgvector KNN
* - X-Expand, X-Custom-SQL-Join, X-Custom-SQL-Or, X-SearchCols, X-AdvSQL-{col}
* - X-Clean-JSON, X-Distinct, X-SkipCount, X-SkipCache, X-PKRow
* - X-SimpleApi / X-DetailApi / X-Syncfusion, X-Single-Record-As-Object
* - X-Transaction-Atomic, X-Files
*/
export function buildHeaders(options: Options): Record<string, string> {
export function buildHeaders(options: HeaderSpecOptions): Record<string, string> {
const headers: Record<string, string> = {};
// Column selection
@@ -79,6 +86,17 @@ export function buildHeaders(options: Options): Record<string, string> {
const op = mapOperatorToHeaderOp(filter.operator);
const valueStr = formatFilterValue(filter);
const geoPrefix = geoFilterHeader(filter.operator);
if (geoPrefix) {
const payload: Record<string, unknown> = {
op: filter.operator,
value: filter.value,
};
if (logicOp === "OR") payload.logic = "or";
headers[`${geoPrefix}${filter.column}`] = JSON.stringify(payload);
continue;
}
if (filter.operator === "eq" && logicOp === "AND") {
// Simple field filter shorthand
headers[`X-FieldFilter-${filter.column}`] = valueStr;
@@ -117,13 +135,94 @@ export function buildHeaders(options: Options): Record<string, string> {
// Preload
if (options.preload?.length) {
const parts = options.preload.map((p: PreloadOption) => {
if (p.columns?.length) {
return `${p.relation}:${p.columns.join(",")}`;
// Go applies X-Preload-Where to every preload in the matching X-Preload header,
// so preloads are grouped by where clause.
const groups = new Map<string, string[]>();
for (const p of options.preload) {
const spec = p.columns?.length
? `${p.relation}:${p.columns.join(",")}`
: p.relation;
const where = p.where ?? "";
groups.set(where, [...(groups.get(where) ?? []), spec]);
}
let n = 0;
for (const [where, specs] of groups) {
if (!where) {
headers["X-Preload"] = specs.join("|");
} else if (!groups.has("") && n === 0) {
// X-Preload-Where would also apply to a where-less X-Preload, so only use it alone
headers["X-Preload"] = specs.join("|");
headers["X-Preload-Where"] = where;
n++;
} else {
n++;
headers[`X-Preload-${n}`] = specs.join("|");
headers[`X-Preload-${n}-Where`] = where;
}
return p.relation;
});
headers["X-Preload"] = parts.join("|");
}
}
// Expand (LEFT JOIN)
if (options.expand?.length) {
headers["X-Expand"] = options.expand
.map((e) =>
e.columns?.length ? `${e.relation}:${e.columns.join(",")}` : e.relation,
)
.join("|");
}
if (options.custom_sql_joins?.length) {
headers["X-Custom-SQL-Join"] = options.custom_sql_joins.join("|");
}
if (options.custom_sql_or?.length) {
headers["X-Custom-SQL-Or"] = options.custom_sql_or.join(" OR ");
}
if (options.search_columns?.length) {
headers["X-SearchCols"] = options.search_columns.join(",");
}
if (options.advanced_sql) {
for (const [col, sql] of Object.entries(options.advanced_sql)) {
headers[`X-AdvSQL-${col}`] = sql;
}
}
// pgvector KNN search
if (options.vector_search) {
const vs = options.vector_search;
headers[`X-Vector-Search-${vs.column}`] = vs.metric ?? "l2";
headers["X-Vector-Search-Vector"] = JSON.stringify(vs.vector);
if (vs.as) headers["X-Vector-Search-As"] = vs.as;
if (vs.direction) headers["X-Vector-Search-Dir"] = vs.direction;
}
// Flags
const flags: [string, boolean | undefined][] = [
["X-Clean-JSON", options.clean_json],
["X-Distinct", options.distinct],
["X-SkipCount", options.skip_count],
["X-SkipCache", options.skip_cache],
["X-Transaction-Atomic", options.atomic_transaction],
["X-Single-Record-As-Object", options.single_record_as_object],
];
for (const [name, val] of flags) {
if (val !== undefined) headers[name] = String(val);
}
if (options.pk_row) {
headers["X-PKRow"] = options.pk_row;
}
if (options.response_format) {
const formatHeaders = {
simple: "X-SimpleApi",
detail: "X-DetailApi",
syncfusion: "X-Syncfusion",
} as const;
headers[formatHeaders[options.response_format]] = "true";
}
if (options.xfiles) {
headers["X-Files"] = encodeHeaderValue(JSON.stringify(options.xfiles));
}
// Fetch row number
@@ -149,6 +248,17 @@ export function buildHeaders(options: Options): Record<string, string> {
return headers;
}
const VECTOR_OPS = new Set(["l2_within", "cosine_within", "ip_within"]);
function geoFilterHeader(operator: string): string | null {
const op = operator.toLowerCase();
if (VECTOR_OPS.has(op) || op.endsWith("_within")) return "X-VectorFilter-";
if (op.startsWith("st_") || op === "bbox" || op === "&&") {
return "X-SpatialFilter-";
}
return null;
}
function mapOperatorToHeaderOp(operator: string): string {
switch (operator) {
case "eq":
@@ -280,7 +390,7 @@ export class HeaderSpecClient {
schema: string,
entity: string,
id?: string,
options?: Options,
options?: HeaderSpecOptions,
): Promise<APIResponse<T>> {
const url = this.buildUrl(schema, entity, id);
const optHeaders = options ? buildHeaders(options) : {};
@@ -294,7 +404,7 @@ export class HeaderSpecClient {
schema: string,
entity: string,
data: any,
options?: Options,
options?: HeaderSpecOptions,
): Promise<APIResponse<T>> {
const url = this.buildUrl(schema, entity);
const optHeaders = options ? buildHeaders(options) : {};
@@ -310,7 +420,7 @@ export class HeaderSpecClient {
entity: string,
id: string,
data: any,
options?: Options,
options?: HeaderSpecOptions,
): Promise<APIResponse<T>> {
const url = this.buildUrl(schema, entity, id);
const optHeaders = options ? buildHeaders(options) : {};
+6
View File
@@ -0,0 +1,6 @@
__pycache__/
*.egg-info/
.venv/
dist/
.pytest_cache/
.coverage
+142
View File
@@ -0,0 +1,142 @@
# resolvespec (Python)
Python client for ResolveSpec REST, HeaderSpec (restheadspec), FunctionSpec and WebSocketSpec. Port of `resolvespec-js`.
- Python >= 3.11, `httpx` (REST, sync + async), `websockets` (WS, async)
- Options/filters/sorts are plain dicts using the wire key names (`TypedDict` hints in `resolvespec.types`)
```
pip install resolvespec
```
## Clients
| Protocol | Sync | Async | Transport |
|---|---|---|---|
| ResolveSpec | `ResolveSpecClient` | `AsyncResolveSpecClient` | POST + JSON body `{operation, id, data, options}` |
| HeaderSpec | `HeaderSpecClient` | `AsyncHeaderSpecClient` | GET/POST/PUT/DELETE, options as `X-*` headers |
| FunctionSpec | `FuncSpecClient` | `AsyncFuncSpecClient` | user-defined SQL endpoints; params via query string + `X-*` headers |
| WebSocketSpec | - | `WebSocketClient` | WebSocket JSON messages |
Constructor (REST): `Client(base_url, token=None, headers=None, timeout=30.0)`
- `token` -> `Authorization: Bearer`; wins over `headers`
- `headers`: custom headers, merged case-insensitively; snapshot at construction
- Sync: context manager / `close()`. Async: `async with` / `await aclose()`
- Cached sync factories: `get_resolvespec_client()`, `get_headerspec_client()` (same args -> same instance)
## ResolveSpec
URL: `{base}/{schema}/{entity}[/{id}]`
| Method | Signature |
|---|---|
| `get_metadata` | `(schema, entity)` (GET) |
| `read` | `(schema, entity, id=None, options=None)` |
| `create` | `(schema, entity, data, options=None)` |
| `update` | `(schema, entity, data, id=None, options=None)` |
| `delete` | `(schema, entity, id)` |
`id`: int/str -> URL path; `list[str]` -> body `id`.
Returns `{"success", "data", "metadata"?, "error"?}`.
## HeaderSpec
| Method | HTTP | Signature |
|---|---|---|
| `read` | GET | `(schema, entity, id=None, options=None)` |
| `create` | POST | `(schema, entity, data, options=None)` |
| `update` | PUT | `(schema, entity, id, data, options=None)` |
| `delete` | DELETE | `(schema, entity, id)` |
Response metadata derived from `Content-Range` (`offset-end/total`) and `X-Limit`.
`build_headers(options)`, `encode_header_value()` / `decode_header_value()` (`ZIP_` / `__` base64) are exported.
### Option -> header
| Option | Header |
|---|---|
| `columns` / `omit_columns` | `X-Select-Fields` / `X-Not-Select-Fields` |
| filter `eq` + AND | `X-FieldFilter-{col}` |
| filter AND / OR | `X-SearchOp-{op}-{col}` / `X-SearchOr-{op}-{col}` |
| spatial (`st_*`, `bbox`) / vector (`*_within`) filter | `X-SpatialFilter-{col}` / `X-VectorFilter-{col}` (JSON) |
| `sort` | `X-Sort` (`+col,-col`) |
| `limit` / `offset` | `X-Limit` / `X-Offset` |
| `cursor_forward` / `cursor_backward` | `X-Cursor-Forward` / `X-Cursor-Backward` |
| `preload` | `X-Preload` (`Rel:c1,c2\|Rel2`), `X-Preload-Where`, `X-Preload-{n}[-Where]` |
| `expand` | `X-Expand` |
| `custom_sql_joins` / `custom_sql_or` | `X-Custom-SQL-Join` / `X-Custom-SQL-Or` |
| `search_columns` | `X-SearchCols` |
| `advanced_sql` | `X-AdvSQL-{col}` |
| `computedColumns` | `X-CQL-SEL-{name}` |
| `customOperators` | `X-Custom-SQL-W` (AND-joined) |
| `vector_search` | `X-Vector-Search-{col}`, `-Vector`, `-As`, `-Dir` |
| `fetch_row_number` | `X-Fetch-RowNumber` |
| `clean_json`, `distinct`, `skip_count`, `skip_cache`, `atomic_transaction`, `single_record_as_object` | `X-Clean-JSON`, `X-Distinct`, `X-SkipCount`, `X-SkipCache`, `X-Transaction-Atomic`, `X-Single-Record-As-Object` |
| `pk_row` | `X-PKRow` |
| `response_format` (`simple`/`detail`/`syncfusion`) | `X-SimpleApi` / `X-DetailApi` / `X-Syncfusion` |
| `xfiles` | `X-Files` (`ZIP_` base64 JSON) |
Filter operator -> header op: `eq equals`, `neq notequals`, `gt greaterthan`, `gte greaterthanorequal`, `lt lessthan`, `lte lessthanorequal`, `like/ilike/contains contains`, `startswith beginswith`, `endswith`, `in`, `between`, `between_inclusive betweeninclusive`, `is_null empty`, `is_not_null notempty`.
## FunctionSpec
Routes are defined by the server app, so calls take a `path`. The server never reads a request body.
| Method | Server handler | Result |
|---|---|---|
| `query(path, params=None, options=None, *, method="GET")` | `SqlQuery` (single record) | `{success, data}` |
| `query_list(path, params=None, options=None, *, method="GET")` | `SqlQueryList` | `{success, data, metadata}` (from `Content-Range: items a-b/total`) |
- `params` -> query string. `bool` -> `true/false`, `None` skipped, `list` -> repeated key (server: `IN` filter). `p-` prefixed names are substituted into the SQL.
- `options` -> `X-*` headers. Query values override headers of the same name.
- 206 Partial Content (more rows than returned) is treated as success.
| Option | Header |
|---|---|
| `filters` (`eq`+AND) | `X-FieldFilter-{col}` |
| `filters` (other) | `X-SearchOp-{op}-{col}` / `X-SearchOr-{op}-{col}` |
| `search_filters` `{col: text}` | `X-SearchFilter-{col}` (ILIKE) |
| `custom_sql_where` / `custom_sql_or` | `X-Custom-SQL-W` / `X-Custom-SQL-Or` |
| `sort` | `X-Sort` as SQL terms: `col ASC,col DESC` |
| `limit` / `offset` | `X-Limit` / `X-Offset` |
| `distinct`, `skip_count`, `skip_cache` | `X-Distinct`, `X-SkipCount`, `X-SkipCache` |
| `response_format` | `X-SimpleApi` / `X-DetailApi` / `X-Syncfusion` (`data` shape changes: array / `{items,...}` / `{result,count}`) |
Server limits:
- `sort` goes verbatim into `ORDER BY`; `-col` (restheadspec style) does **not** mean DESC.
- `X-Select-Fields` / `X-Not-Select-Fields` are no-ops server-side, so not exposed.
- One search operator per column; same column twice keeps the last.
- Values starting with `ZIP_` / `__` are base64-decoded by the server; such plaintext cannot be sent.
- Non-ASCII / control-char values are sent `ZIP_`-encoded automatically.
## WebSocketSpec
`WebSocketClient(url, *, reconnect=True, reconnect_interval=3.0, max_reconnect_attempts=10, heartbeat_interval=30.0, request_timeout=30.0, subscribe_timeout=10.0, headers=None)`
| Method | Notes |
|---|---|
| `connect()` / `close()` | also `async with` |
| `request(operation, entity, *, schema, record_id, data, options)` | returns response `data` |
| `read(entity, *, schema, record_id, filters, columns, sort, preload, limit, offset)` | |
| `create(entity, data, *, schema)` | |
| `update(entity, id, data, *, schema)` | |
| `delete(entity, id, *, schema)` | |
| `meta(entity, *, schema)` | |
| `subscribe(entity, callback, *, schema, filters)` | returns subscription id; callback gets notification dict (sync or async) |
| `unsubscribe(subscription_id)` | |
| `on(event, cb)` / `off(event)` | events: `connect`, `disconnect`, `error`, `message`, `state_change` |
| `state`, `is_connected()`, `get_subscriptions()` | |
Auto-reconnect does not restore subscriptions; re-subscribe on `connect`.
## Errors
`ResolveSpecError(message, status_code, code, details)` on non-2xx (REST) or failed response / timeout / not connected (WS).
## Dev
```
pip install -e '.[dev]'
pytest
```
+24
View File
@@ -0,0 +1,24 @@
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "resolvespec"
version = "1.0.0"
description = "Python client for ResolveSpec REST, HeaderSpec and WebSocket APIs"
readme = "README.md"
requires-python = ">=3.11"
license = { text = "MIT" }
authors = [{ name = "Hein (Warkanum) Puth" }]
keywords = ["resolvespec", "headerspec", "websocket", "rest-client", "api-client"]
dependencies = ["httpx>=0.27", "websockets>=13"]
[project.optional-dependencies]
dev = ["pytest>=8", "pytest-asyncio>=0.23", "pytest-cov"]
[tool.hatch.build.targets.wheel]
packages = ["src/resolvespec"]
[tool.pytest.ini_options]
testpaths = ["tests"]
asyncio_mode = "auto"
@@ -0,0 +1,43 @@
"""ResolveSpec Python client: REST (ResolveSpec), HeaderSpec and WebSocketSpec."""
from typing import Mapping, Optional
from .headerspec import (
AsyncHeaderSpecClient,
HeaderSpecClient,
build_headers,
decode_header_value,
encode_header_value,
)
from .funcspec import AsyncFuncSpecClient, FuncSpecClient
from .http import ResolveSpecError, merge_headers
from .resolvespec import AsyncResolveSpecClient, ResolveSpecClient
from .types import * # noqa: F401,F403
from .websocket import Subscription, WebSocketClient
def _cache_key(base_url: str, token: Optional[str], headers: Optional[Mapping[str, str]]):
return (
base_url,
token,
tuple(sorted((k.lower(), v) for k, v in (headers or {}).items())),
)
_resolvespec: dict = {}
_headerspec: dict = {}
def get_resolvespec_client(base_url: str, token: Optional[str] = None, headers: Optional[Mapping[str, str]] = None) -> ResolveSpecClient:
"""Cached sync client, keyed by base_url + token + headers (case-insensitive names)."""
key = _cache_key(base_url, token, headers)
if key not in _resolvespec:
_resolvespec[key] = ResolveSpecClient(base_url, token, headers)
return _resolvespec[key]
def get_headerspec_client(base_url: str, token: Optional[str] = None, headers: Optional[Mapping[str, str]] = None) -> HeaderSpecClient:
"""Cached sync client, keyed by base_url + token + headers (case-insensitive names)."""
key = _cache_key(base_url, token, headers)
if key not in _headerspec:
_headerspec[key] = HeaderSpecClient(base_url, token, headers)
return _headerspec[key]
@@ -0,0 +1,197 @@
"""FunctionSpec client: calls user-defined SQL endpoints (Go pkg/funcspec).
Routes are defined by the server application, so calls take a `path`.
Parameters are sent as query string values and/or `X-*` headers; the server never
reads a request body. Query-string values override headers of the same name.
Server behaviour worth knowing (pkg/funcspec):
- `sort` is inserted raw into ORDER BY, so it must be SQL (`col DESC`), not `-col`.
- Field selection (`X-Select-Fields`) is a no-op server-side, so it is not exposed.
- Only one search operator per column is kept.
- Values starting with `ZIP_` or `__` are base64-decoded by the server (even after our
own encoding), so such plaintext values cannot be sent faithfully.
"""
from __future__ import annotations
import re
from typing import Any, Dict, List, Mapping, Optional
import httpx
from .headerspec import _OPERATOR_MAP, _bool, _filter_value, encode_header_value
from .http import client_headers, error_from, merge_headers, parse_json
from .types import APIResponse, FuncSpecOptions
Params = Mapping[str, Any]
_CONTENT_RANGE = re.compile(r"(\d+)-(\d+)/(\d+)")
def _safe(value: str) -> str:
"""Encode values that are unsafe as raw header/query text (non-ASCII, control chars, edge spaces)."""
if not value.isascii() or not value.isprintable() or value != value.strip():
return encode_header_value(value)
return value
def build_headers(options: Mapping[str, Any]) -> Dict[str, str]:
"""Build the X-* headers understood by funcspec.ParseParameters."""
h: Dict[str, str] = {}
o = options
for f in o.get("filters") or []:
operator = f["operator"]
logic = f.get("logic_operator") or "AND"
value = _safe(_filter_value(f))
if operator == "eq" and logic == "AND":
h[f"X-FieldFilter-{f['column']}"] = value
else:
kind = "X-SearchOr" if logic == "OR" else "X-SearchOp"
h[f"{kind}-{_OPERATOR_MAP.get(operator, operator)}-{f['column']}"] = value
for col, text in (o.get("search_filters") or {}).items():
h[f"X-SearchFilter-{col}"] = _safe(str(text)) # CAST(col AS TEXT) ILIKE %text%
if o.get("custom_sql_where"):
h["X-Custom-SQL-W"] = _safe(o["custom_sql_where"])
if o.get("custom_sql_or"):
h["X-Custom-SQL-Or"] = _safe(o["custom_sql_or"])
if o.get("sort"):
h["X-Sort"] = _safe(",".join(_sort_term(s) for s in o["sort"]))
if o.get("limit") is not None:
h["X-Limit"] = str(o["limit"])
if o.get("offset") is not None:
h["X-Offset"] = str(o["offset"])
for name, key in (("X-Distinct", "distinct"), ("X-SkipCount", "skip_count"), ("X-SkipCache", "skip_cache")):
if o.get(key) is not None:
h[name] = _bool(o[key])
fmt = o.get("response_format")
if fmt:
h[{"simple": "X-SimpleApi", "detail": "X-DetailApi", "syncfusion": "X-Syncfusion"}[fmt]] = "true"
return h
def _sort_term(s: Mapping[str, str]) -> str:
# funcspec puts this verbatim into ORDER BY
return f"{s['column']} {'DESC' if s.get('direction', 'asc').upper() == 'DESC' else 'ASC'}"
def build_query(params: Optional[Params]) -> Dict[str, Any]:
"""Query-string values: bools -> true/false, lists -> repeated keys (server: IN filter)."""
out: Dict[str, Any] = {}
for k, v in (params or {}).items():
if v is None:
continue
if isinstance(v, (list, tuple)):
out[k] = [_safe(_q(x)) for x in v]
else:
out[k] = _safe(_q(v))
return out
def _q(v: Any) -> str:
return _bool(v) if isinstance(v, bool) else str(v)
def _metadata(response: httpx.Response, options: Optional[Mapping[str, Any]]) -> Dict[str, int]:
"""Content-Range is `items {offset}-{offset+len}/{total}`."""
m = _CONTENT_RANGE.search(response.headers.get("content-range", ""))
start, end, total = (int(x) for x in m.groups()) if m else (0, 0, 0)
return {
"total": total,
"count": end - start,
"filtered": total,
"offset": start,
"limit": int((options or {}).get("limit") or 0),
}
def _wrap(response: httpx.Response, options: Optional[Mapping[str, Any]], with_metadata: bool) -> APIResponse:
data = parse_json(response)
if not response.is_success: # 206 Partial Content is success
raise error_from(response, data)
result: APIResponse = {"success": True, "data": data}
if with_metadata:
result["metadata"] = _metadata(response, options)
return result
class _Base:
def __init__(
self,
base_url: str,
token: Optional[str] = None,
headers: Optional[Mapping[str, str]] = None,
timeout: Optional[float] = 30.0,
):
self.base_url = base_url
self.token = token
self.headers = dict(headers or {}) # snapshot
self.timeout = timeout
def _req(self, method: str, path: str, params: Optional[Params], options: Optional[Mapping[str, Any]]):
url = f"{self.base_url.rstrip('/')}/{path.lstrip('/')}"
headers = merge_headers(
client_headers(self.token, self.headers),
build_headers(options) if options else {},
)
return method.upper(), url, headers, build_query(params)
class FuncSpecClient(_Base):
"""Synchronous client. Use as a context manager or call close()."""
def __init__(self, *args: Any, transport: Optional[httpx.BaseTransport] = None, **kwargs: Any):
super().__init__(*args, **kwargs)
self._http = httpx.Client(timeout=self.timeout, transport=transport)
def close(self) -> None:
self._http.close()
def __enter__(self) -> "FuncSpecClient":
return self
def __exit__(self, *exc: Any) -> None:
self.close()
def _send(self, req, options, with_metadata) -> APIResponse:
method, url, headers, query = req
return _wrap(self._http.request(method, url, headers=headers, params=query), options, with_metadata)
def query(self, path: str, params: Optional[Params] = None, options: Optional[FuncSpecOptions] = None, *, method: str = "GET") -> APIResponse:
"""Single-record endpoint (Handler.SqlQuery). `data` is the row object."""
return self._send(self._req(method, path, params, options), options, False)
def query_list(self, path: str, params: Optional[Params] = None, options: Optional[FuncSpecOptions] = None, *, method: str = "GET") -> APIResponse:
"""List endpoint (Handler.SqlQueryList). Adds `metadata` from Content-Range."""
return self._send(self._req(method, path, params, options), options, True)
class AsyncFuncSpecClient(_Base):
"""Asyncio client. Use as an async context manager or await aclose()."""
def __init__(self, *args: Any, transport: Optional[httpx.AsyncBaseTransport] = None, **kwargs: Any):
super().__init__(*args, **kwargs)
self._http = httpx.AsyncClient(timeout=self.timeout, transport=transport)
async def aclose(self) -> None:
await self._http.aclose()
async def __aenter__(self) -> "AsyncFuncSpecClient":
return self
async def __aexit__(self, *exc: Any) -> None:
await self.aclose()
async def _send(self, req, options, with_metadata) -> APIResponse:
method, url, headers, query = req
return _wrap(await self._http.request(method, url, headers=headers, params=query), options, with_metadata)
async def query(self, path: str, params: Optional[Params] = None, options: Optional[FuncSpecOptions] = None, *, method: str = "GET") -> APIResponse:
return await self._send(self._req(method, path, params, options), options, False)
async def query_list(self, path: str, params: Optional[Params] = None, options: Optional[FuncSpecOptions] = None, *, method: str = "GET") -> APIResponse:
return await self._send(self._req(method, path, params, options), options, True)
@@ -0,0 +1,336 @@
"""HeaderSpec client: query options sent as HTTP headers (Go restheadspec).
Methods: GET=read, POST=create, PUT=update, DELETE=delete.
"""
from __future__ import annotations
import base64
import json
import re
from typing import Any, Dict, Mapping, Optional
import httpx
from .http import build_url, client_headers, error_from, merge_headers, parse_json
from .types import APIResponse, FilterOption, HeaderSpecOptions
_PREFIXES = ("ZIP_", "__")
_OPERATOR_MAP = {
"eq": "equals",
"neq": "notequals",
"gt": "greaterthan",
"gte": "greaterthanorequal",
"lt": "lessthan",
"lte": "lessthanorequal",
"like": "contains",
"ilike": "contains",
"contains": "contains",
"startswith": "beginswith",
"endswith": "endswith",
"in": "in",
"between": "between",
"between_inclusive": "betweeninclusive",
"is_null": "empty",
"is_not_null": "notempty",
}
def encode_header_value(value: str) -> str:
"""Base64 (UTF-8) with ZIP_ prefix, for complex header values."""
return "ZIP_" + base64.b64encode(value.encode("utf-8")).decode("ascii")
def decode_header_value(value: str) -> str:
"""Decode a value that may carry a ZIP_ or __ base64 prefix (nested allowed)."""
code = value
for prefix in _PREFIXES:
if code.startswith(prefix):
b64 = re.sub(r"[\n\r ]", "", code[len(prefix):])
b64 += "=" * (-len(b64) % 4)
code = base64.b64decode(b64).decode("utf-8")
break
if code.startswith(_PREFIXES):
code = decode_header_value(code)
return code
def _geo_header(operator: str) -> Optional[str]:
op = operator.lower()
if op.endswith("_within"):
return "X-VectorFilter-"
if op.startswith("st_") or op in ("bbox", "&&"):
return "X-SpatialFilter-"
return None
def _filter_value(f: FilterOption) -> str:
v = f.get("value")
if v is None:
return ""
if isinstance(v, (list, tuple)):
return ",".join(_scalar(x) for x in v)
return _scalar(v)
def _scalar(v: Any) -> str:
if isinstance(v, bool): # match JS String(true)
return "true" if v else "false"
return str(v)
def _bool(v: bool) -> str:
return "true" if v else "false"
def _preload_spec(p: Mapping[str, Any]) -> str:
cols = p.get("columns")
return f"{p['relation']}:{','.join(cols)}" if cols else p["relation"]
def build_headers(options: HeaderSpecOptions) -> Dict[str, str]:
"""Build restheadspec HTTP headers from options. See README for the mapping."""
h: Dict[str, str] = {}
o = options
if o.get("columns"):
h["X-Select-Fields"] = ",".join(o["columns"])
if o.get("omit_columns"):
h["X-Not-Select-Fields"] = ",".join(o["omit_columns"])
for f in o.get("filters") or []:
logic = f.get("logic_operator") or "AND"
operator = f["operator"]
op = _OPERATOR_MAP.get(operator, operator)
value = _filter_value(f)
geo = _geo_header(operator)
if geo:
payload: Dict[str, Any] = {"op": operator, "value": f.get("value")}
if logic == "OR":
payload["logic"] = "or"
h[f"{geo}{f['column']}"] = json.dumps(payload, separators=(",", ":"))
elif operator == "eq" and logic == "AND":
h[f"X-FieldFilter-{f['column']}"] = value
elif logic == "OR":
h[f"X-SearchOr-{op}-{f['column']}"] = value
else:
h[f"X-SearchOp-{op}-{f['column']}"] = value
if o.get("sort"):
h["X-Sort"] = ",".join(
("-" if s["direction"].upper() == "DESC" else "+") + s["column"] for s in o["sort"]
)
if o.get("limit") is not None:
h["X-Limit"] = str(o["limit"])
if o.get("offset") is not None:
h["X-Offset"] = str(o["offset"])
if o.get("cursor_forward"):
h["X-Cursor-Forward"] = o["cursor_forward"]
if o.get("cursor_backward"):
h["X-Cursor-Backward"] = o["cursor_backward"]
if o.get("preload"):
# Go applies X-Preload-Where to every preload in the matching X-Preload header,
# so preloads are grouped by where clause.
groups: Dict[str, list] = {}
for p in o["preload"]:
groups.setdefault(p.get("where") or "", []).append(_preload_spec(p))
n = 0
for where, specs in groups.items():
if not where:
h["X-Preload"] = "|".join(specs)
elif "" not in groups and n == 0:
# X-Preload-Where would also apply to a where-less X-Preload, so only use it alone
h["X-Preload"] = "|".join(specs)
h["X-Preload-Where"] = where
n += 1
else:
n += 1
h[f"X-Preload-{n}"] = "|".join(specs)
h[f"X-Preload-{n}-Where"] = where
if o.get("expand"):
h["X-Expand"] = "|".join(_preload_spec(e) for e in o["expand"])
if o.get("custom_sql_joins"):
h["X-Custom-SQL-Join"] = "|".join(o["custom_sql_joins"])
if o.get("custom_sql_or"):
h["X-Custom-SQL-Or"] = " OR ".join(o["custom_sql_or"])
if o.get("search_columns"):
h["X-SearchCols"] = ",".join(o["search_columns"])
for col, sql in (o.get("advanced_sql") or {}).items():
h[f"X-AdvSQL-{col}"] = sql
vs = o.get("vector_search")
if vs:
h[f"X-Vector-Search-{vs['column']}"] = vs.get("metric") or "l2"
h["X-Vector-Search-Vector"] = json.dumps(vs["vector"], separators=(",", ":"))
if vs.get("as"):
h["X-Vector-Search-As"] = vs["as"]
if vs.get("direction"):
h["X-Vector-Search-Dir"] = vs["direction"]
for name, key in (
("X-Clean-JSON", "clean_json"),
("X-Distinct", "distinct"),
("X-SkipCount", "skip_count"),
("X-SkipCache", "skip_cache"),
("X-Transaction-Atomic", "atomic_transaction"),
("X-Single-Record-As-Object", "single_record_as_object"),
):
if o.get(key) is not None:
h[name] = _bool(o[key])
if o.get("pk_row"):
h["X-PKRow"] = o["pk_row"]
fmt = o.get("response_format")
if fmt:
h[{"simple": "X-SimpleApi", "detail": "X-DetailApi", "syncfusion": "X-Syncfusion"}[fmt]] = "true"
if o.get("xfiles"):
h["X-Files"] = encode_header_value(json.dumps(o["xfiles"], separators=(",", ":")))
if o.get("fetch_row_number"):
h["X-Fetch-RowNumber"] = o["fetch_row_number"]
for cc in o.get("computedColumns") or []:
h[f"X-CQL-SEL-{cc['name']}"] = cc["expression"]
if o.get("customOperators"):
h["X-Custom-SQL-W"] = " AND ".join(co["sql"] for co in o["customOperators"])
return h
def _int(s: Optional[str]) -> int:
try:
return int(s) # type: ignore[arg-type]
except (TypeError, ValueError):
return 0
def _wrap(response: httpx.Response) -> APIResponse:
"""Wrap a raw restheadspec body, deriving metadata from Content-Range / X-Limit."""
data = parse_json(response)
if not response.is_success:
raise error_from(response, data)
cr = response.headers.get("content-range")
total = _int(cr.split("/")[-1]) if cr else 0
offset = _int(cr.split("/")[0].split("-")[0].split(" ")[-1]) if cr else 0
return {
"data": data,
"success": True,
"error": data.get("error") if isinstance(data, dict) else None,
"metadata": {
"count": total,
"total": total,
"filtered": total,
"offset": offset,
"limit": _int(response.headers.get("x-limit")),
},
}
class _Base:
def __init__(
self,
base_url: str,
token: Optional[str] = None,
headers: Optional[Mapping[str, str]] = None,
timeout: Optional[float] = 30.0,
):
self.base_url = base_url
self.token = token
self.headers = dict(headers or {}) # snapshot
self.timeout = timeout
def _base_headers(self) -> Dict[str, str]:
return client_headers(self.token, self.headers)
def _req(self, method, schema, entity, id, options=None, body=None):
opt = build_headers(options) if options else {}
return (
method,
build_url(self.base_url, schema, entity, id),
merge_headers(self._base_headers(), opt),
body,
)
def _read_req(self, schema, entity, id, options):
return self._req("GET", schema, entity, id, options)
def _create_req(self, schema, entity, data, options):
return self._req("POST", schema, entity, None, options, data)
def _update_req(self, schema, entity, id, data, options):
return self._req("PUT", schema, entity, id, options, data)
def _delete_req(self, schema, entity, id):
return self._req("DELETE", schema, entity, id)
class HeaderSpecClient(_Base):
"""Synchronous client. Use as a context manager or call close()."""
def __init__(self, *args: Any, transport: Optional[httpx.BaseTransport] = None, **kwargs: Any):
super().__init__(*args, **kwargs)
self._http = httpx.Client(timeout=self.timeout, transport=transport)
def close(self) -> None:
self._http.close()
def __enter__(self) -> "HeaderSpecClient":
return self
def __exit__(self, *exc: Any) -> None:
self.close()
def _send(self, req) -> APIResponse:
method, url, headers, body = req
return _wrap(self._http.request(method, url, headers=headers, json=body))
def read(self, schema: str, entity: str, id: Optional[str] = None, options: Optional[HeaderSpecOptions] = None) -> APIResponse:
return self._send(self._read_req(schema, entity, id, options))
def create(self, schema: str, entity: str, data: Any, options: Optional[HeaderSpecOptions] = None) -> APIResponse:
return self._send(self._create_req(schema, entity, data, options))
def update(self, schema: str, entity: str, id: str, data: Any, options: Optional[HeaderSpecOptions] = None) -> APIResponse:
return self._send(self._update_req(schema, entity, id, data, options))
def delete(self, schema: str, entity: str, id: str) -> APIResponse:
return self._send(self._delete_req(schema, entity, id))
class AsyncHeaderSpecClient(_Base):
"""Asyncio client. Use as an async context manager or await aclose()."""
def __init__(self, *args: Any, transport: Optional[httpx.AsyncBaseTransport] = None, **kwargs: Any):
super().__init__(*args, **kwargs)
self._http = httpx.AsyncClient(timeout=self.timeout, transport=transport)
async def aclose(self) -> None:
await self._http.aclose()
async def __aenter__(self) -> "AsyncHeaderSpecClient":
return self
async def __aexit__(self, *exc: Any) -> None:
await self.aclose()
async def _send(self, req) -> APIResponse:
method, url, headers, body = req
return _wrap(await self._http.request(method, url, headers=headers, json=body))
async def read(self, schema: str, entity: str, id: Optional[str] = None, options: Optional[HeaderSpecOptions] = None) -> APIResponse:
return await self._send(self._read_req(schema, entity, id, options))
async def create(self, schema: str, entity: str, data: Any, options: Optional[HeaderSpecOptions] = None) -> APIResponse:
return await self._send(self._create_req(schema, entity, data, options))
async def update(self, schema: str, entity: str, id: str, data: Any, options: Optional[HeaderSpecOptions] = None) -> APIResponse:
return await self._send(self._update_req(schema, entity, id, data, options))
async def delete(self, schema: str, entity: str, id: str) -> APIResponse:
return await self._send(self._delete_req(schema, entity, id))
@@ -0,0 +1,76 @@
"""Shared HTTP helpers for the REST clients."""
from __future__ import annotations
from typing import Any, Dict, Mapping, Optional
from urllib.parse import quote
class ResolveSpecError(Exception):
"""Raised on a non-2xx response or an unsuccessful API result."""
def __init__(
self,
message: str,
status_code: Optional[int] = None,
code: Optional[str] = None,
details: Any = None,
detail: Optional[str] = None,
):
super().__init__(message)
self.message = message
self.status_code = status_code
self.code = code
self.details = details
self.detail = detail # server-side reason (funcspec / restheadspec errors)
def merge_headers(*sources: Mapping[str, str]) -> Dict[str, str]:
"""Merge HTTP headers case-insensitively; the last source wins and keeps its spelling."""
result: Dict[str, str] = {}
for source in sources:
for name, value in source.items():
for existing in [k for k in result if k.lower() == name.lower()]:
del result[existing]
result[name] = value
return result
def client_headers(token: Optional[str], headers: Optional[Mapping[str, str]]) -> Dict[str, str]:
"""Content-Type < custom headers < bearer token."""
return merge_headers(
{"Content-Type": "application/json"},
headers or {},
{"Authorization": f"Bearer {token}"} if token else {},
)
def build_url(base_url: str, schema: str, entity: str, id: Optional[Any] = None) -> str:
url = f"{base_url.rstrip('/')}/{quote(schema, safe='')}/{quote(entity, safe='')}"
if id is not None and id != "":
url += f"/{quote(str(id), safe='')}"
return url
def drop_none(d: Mapping[str, Any]) -> Dict[str, Any]:
return {k: v for k, v in d.items() if v is not None}
def parse_json(response: Any) -> Any:
try:
return response.json()
except ValueError:
return None
def error_from(response: Any, data: Any) -> ResolveSpecError:
err = data.get("error") if isinstance(data, dict) else None
err = err if isinstance(err, dict) else {}
text = (response.text or "").strip() if data is None else ""
fallback = text[:200] or f"{response.reason_phrase} ({response.status_code})"
return ResolveSpecError(
err.get("message") or fallback,
status_code=response.status_code,
code=err.get("code"),
details=err.get("details"),
detail=err.get("detail"),
)
@@ -0,0 +1,137 @@
"""ResolveSpec client: JSON body protocol (POST {operation, data, options})."""
from __future__ import annotations
from typing import Any, Dict, List, Mapping, Optional, Tuple
import httpx
from .http import build_url, client_headers, drop_none, error_from, parse_json
from .types import APIResponse, Options, RecordId
def _url_id(id: Optional[RecordId]) -> Optional[str]:
return str(id) if isinstance(id, (int, str)) else None
def _body_id(id: Optional[RecordId]) -> Optional[List[str]]:
return id if isinstance(id, list) else None
class _Base:
def __init__(
self,
base_url: str,
token: Optional[str] = None,
headers: Optional[Mapping[str, str]] = None,
timeout: Optional[float] = 30.0,
):
self.base_url = base_url
self.token = token
self.headers = dict(headers or {}) # snapshot
self.timeout = timeout
def _headers(self) -> Dict[str, str]:
return client_headers(self.token, self.headers)
def _request(
self, method: str, schema: str, entity: str, id: Optional[str], body: Optional[Dict[str, Any]]
) -> Tuple[str, str, Dict[str, str], Optional[Dict[str, Any]]]:
return method, build_url(self.base_url, schema, entity, id), self._headers(), body
@staticmethod
def _result(response: httpx.Response) -> APIResponse:
data = parse_json(response)
if not response.is_success:
raise error_from(response, data)
return data
# request builders (shared by sync and async)
def _metadata_req(self, schema, entity):
return self._request("GET", schema, entity, None, None)
def _read_req(self, schema, entity, id, options):
body = drop_none({"operation": "read", "id": _body_id(id), "options": options})
return self._request("POST", schema, entity, _url_id(id), body)
def _create_req(self, schema, entity, data, options):
body = drop_none({"operation": "create", "data": data, "options": options})
return self._request("POST", schema, entity, None, body)
def _update_req(self, schema, entity, data, id, options):
body = drop_none({"operation": "update", "id": _body_id(id), "data": data, "options": options})
return self._request("POST", schema, entity, _url_id(id), body)
def _delete_req(self, schema, entity, id):
return self._request("POST", schema, entity, str(id), {"operation": "delete"})
class ResolveSpecClient(_Base):
"""Synchronous client. Use as a context manager or call close()."""
def __init__(self, *args: Any, transport: Optional[httpx.BaseTransport] = None, **kwargs: Any):
super().__init__(*args, **kwargs)
self._http = httpx.Client(timeout=self.timeout, transport=transport)
def close(self) -> None:
self._http.close()
def __enter__(self) -> "ResolveSpecClient":
return self
def __exit__(self, *exc: Any) -> None:
self.close()
def _send(self, req) -> APIResponse:
method, url, headers, body = req
return self._result(self._http.request(method, url, headers=headers, json=body))
def get_metadata(self, schema: str, entity: str) -> APIResponse:
return self._send(self._metadata_req(schema, entity))
def read(self, schema: str, entity: str, id: Optional[RecordId] = None, options: Optional[Options] = None) -> APIResponse:
return self._send(self._read_req(schema, entity, id, options))
def create(self, schema: str, entity: str, data: Any, options: Optional[Options] = None) -> APIResponse:
return self._send(self._create_req(schema, entity, data, options))
def update(self, schema: str, entity: str, data: Any, id: Optional[RecordId] = None, options: Optional[Options] = None) -> APIResponse:
return self._send(self._update_req(schema, entity, data, id, options))
def delete(self, schema: str, entity: str, id: Any) -> APIResponse:
return self._send(self._delete_req(schema, entity, id))
class AsyncResolveSpecClient(_Base):
"""Asyncio client. Use as an async context manager or await aclose()."""
def __init__(self, *args: Any, transport: Optional[httpx.AsyncBaseTransport] = None, **kwargs: Any):
super().__init__(*args, **kwargs)
self._http = httpx.AsyncClient(timeout=self.timeout, transport=transport)
async def aclose(self) -> None:
await self._http.aclose()
async def __aenter__(self) -> "AsyncResolveSpecClient":
return self
async def __aexit__(self, *exc: Any) -> None:
await self.aclose()
async def _send(self, req) -> APIResponse:
method, url, headers, body = req
return self._result(await self._http.request(method, url, headers=headers, json=body))
async def get_metadata(self, schema: str, entity: str) -> APIResponse:
return await self._send(self._metadata_req(schema, entity))
async def read(self, schema: str, entity: str, id: Optional[RecordId] = None, options: Optional[Options] = None) -> APIResponse:
return await self._send(self._read_req(schema, entity, id, options))
async def create(self, schema: str, entity: str, data: Any, options: Optional[Options] = None) -> APIResponse:
return await self._send(self._create_req(schema, entity, data, options))
async def update(self, schema: str, entity: str, data: Any, id: Optional[RecordId] = None, options: Optional[Options] = None) -> APIResponse:
return await self._send(self._update_req(schema, entity, data, id, options))
async def delete(self, schema: str, entity: str, id: Any) -> APIResponse:
return await self._send(self._delete_req(schema, entity, id))
@@ -0,0 +1,166 @@
"""Types aligned with Go pkg/common/types.go. Dict keys are the wire names."""
from __future__ import annotations
from typing import Any, Dict, List, NotRequired, TypedDict, Union
Operator = str # eq neq gt gte lt lte like ilike in contains startswith endswith
# between between_inclusive is_null is_not_null
# st_dwithin bbox (spatial) | l2_within cosine_within ip_within (vector)
Operation = str # read | create | update | delete
SortDirection = str # asc | desc | ASC | DESC
VectorMetric = str # l2 | cosine | ip
ResponseFormat = str # simple | detail | syncfusion
RecordId = Union[int, str, List[str]]
class Parameter(TypedDict):
name: str
value: str
sequence: NotRequired[int]
class FilterOption(TypedDict):
column: str
operator: str
value: Any
logic_operator: NotRequired[str] # "AND" | "OR"
class SortOption(TypedDict):
column: str
direction: str
class CustomOperator(TypedDict):
name: str
sql: str
class ComputedColumn(TypedDict):
name: str
expression: str
class PreloadOption(TypedDict, total=False):
relation: str
table_name: str
columns: List[str]
omit_columns: List[str]
sort: List[SortOption]
filters: List[FilterOption]
where: str
limit: int
offset: int
updateable: bool
computed_ql: Dict[str, str]
recursive: bool
primary_key: str
related_key: str
foreign_key: str
recursive_child_key: str
sql_joins: List[str]
join_aliases: List[str]
# `as` is a keyword, so the functional syntax is required.
VectorSearchOption = TypedDict(
"VectorSearchOption",
{
"column": str,
"vector": List[float],
"metric": str, # l2 (default) | cosine | ip
"as": str, # distance column alias, default _distance
"direction": str, # asc (default) | desc
},
total=False,
)
class ExpandOption(TypedDict, total=False):
relation: str
columns: List[str]
class XFiles(TypedDict, total=False):
tablename: str
schema: str
primarykey: str
foreignkey: str
relatedkey: str
sort: List[str]
prefix: str
editable: bool
recursive: bool
expand: bool
rownumber: bool
skipcount: bool
offset: int
limit: int
columns: List[str]
omit_columns: List[str]
cql_columns: List[str]
sql_joins: List[str]
sql_or: List[str]
sql_and: List[str]
parenttables: List["XFiles"]
childtables: List["XFiles"]
filter_fields: List[Dict[str, str]]
cursor_forward: str
cursor_backward: str
class Options(TypedDict, total=False):
preload: List[PreloadOption]
columns: List[str]
omit_columns: List[str]
filters: List[FilterOption]
sort: List[SortOption]
limit: int
offset: int
customOperators: List[CustomOperator]
computedColumns: List[ComputedColumn]
parameters: List[Parameter]
cursor_forward: str
cursor_backward: str
fetch_row_number: str
vector_search: VectorSearchOption
class HeaderSpecOptions(Options, total=False):
"""Options only available to the header-based (restheadspec) protocol."""
expand: List[ExpandOption] # X-Expand
custom_sql_joins: List[str] # X-Custom-SQL-Join
custom_sql_or: List[str] # X-Custom-SQL-Or
search_columns: List[str] # X-SearchCols
advanced_sql: Dict[str, str] # X-AdvSQL-{col}
clean_json: bool # X-Clean-JSON
distinct: bool # X-Distinct
skip_count: bool # X-SkipCount
skip_cache: bool # X-SkipCache
pk_row: str # X-PKRow
response_format: str # X-SimpleApi / X-DetailApi / X-Syncfusion
single_record_as_object: bool # X-Single-Record-As-Object
atomic_transaction: bool # X-Transaction-Atomic
xfiles: XFiles # X-Files
class FuncSpecOptions(TypedDict, total=False):
"""Options understood by funcspec endpoints (sent as X-* headers)."""
filters: List[FilterOption] # eq+AND -> X-FieldFilter; others X-SearchOp / X-SearchOr (one per column)
search_filters: Dict[str, str] # X-SearchFilter-{col}: text ILIKE
custom_sql_where: str # X-Custom-SQL-W
custom_sql_or: str # X-Custom-SQL-Or
sort: List[SortOption] # sent as SQL ORDER BY terms ("col DESC")
limit: int
offset: int
distinct: bool
skip_count: bool
skip_cache: bool
response_format: str # simple | detail | syncfusion
# Responses are plain dicts: {"success", "data", "metadata"?, "error"?}
APIResponse = Dict[str, Any]
@@ -0,0 +1,335 @@
"""WebSocketSpec client (asyncio). Mirrors the Go websocketspec message protocol."""
from __future__ import annotations
import asyncio
import json
import logging
import uuid
from dataclasses import dataclass, field
from typing import Any, Awaitable, Callable, Dict, List, Optional, Union
from websockets.asyncio.client import ClientConnection, connect
from .http import ResolveSpecError
from .types import FilterOption, PreloadOption, SortOption
log = logging.getLogger("resolvespec.websocket")
# Connection states
DISCONNECTED = "disconnected"
CONNECTING = "connecting"
CONNECTED = "connected"
DISCONNECTING = "disconnecting"
RECONNECTING = "reconnecting"
Notification = Dict[str, Any]
Callback = Callable[[Any], Union[None, Awaitable[None]]]
EVENTS = ("connect", "disconnect", "error", "message", "state_change")
@dataclass
class Subscription:
id: str
entity: str
schema: Optional[str] = None
options: Optional[Dict[str, Any]] = None
callback: Optional[Callback] = field(default=None, repr=False)
def _drop_none(d: Dict[str, Any]) -> Dict[str, Any]:
return {k: v for k, v in d.items() if v is not None}
class WebSocketClient:
"""
Usage:
async with WebSocketClient("ws://localhost:8080/ws") as ws:
rows = await ws.read("users", schema="public", limit=10)
Events (`on(event, callback)`): connect, disconnect, error, message, state_change.
Callbacks may be sync or async.
"""
def __init__(
self,
url: str,
*,
reconnect: bool = True,
reconnect_interval: float = 3.0,
max_reconnect_attempts: int = 10,
heartbeat_interval: float = 30.0,
request_timeout: float = 30.0,
subscribe_timeout: float = 10.0,
headers: Optional[Dict[str, str]] = None,
):
self.url = url
self.reconnect = reconnect
self.reconnect_interval = reconnect_interval
self.max_reconnect_attempts = max_reconnect_attempts
self.heartbeat_interval = heartbeat_interval
self.request_timeout = request_timeout
self.subscribe_timeout = subscribe_timeout
self.headers = dict(headers or {})
self._ws: Optional[ClientConnection] = None
self._state = DISCONNECTED
self._pending: Dict[str, "asyncio.Future[Dict[str, Any]]"] = {}
self._subscriptions: Dict[str, Subscription] = {}
self._listeners: Dict[str, Callback] = {}
self._tasks: List["asyncio.Task[Any]"] = []
self._reader: Optional["asyncio.Task[Any]"] = None
self._manual_close = False
# ---- lifecycle -------------------------------------------------------
async def __aenter__(self) -> "WebSocketClient":
await self.connect()
return self
async def __aexit__(self, *exc: Any) -> None:
await self.close()
async def connect(self) -> None:
if self.is_connected():
return
self._manual_close = False
self._set_state(CONNECTING)
try:
self._ws = await connect(self.url, additional_headers=self.headers or None)
except Exception as e:
self._set_state(DISCONNECTED)
await self._emit("error", e)
raise
self._set_state(CONNECTED)
self._reader = asyncio.create_task(self._read_loop(self._ws))
self._heartbeat = asyncio.create_task(self._heartbeat_loop())
await self._emit("connect")
async def close(self) -> None:
self._manual_close = True
self._set_state(DISCONNECTING)
for t in (self._reader, getattr(self, "_heartbeat", None), getattr(self, "_reconnect_task", None)):
if t and t is not asyncio.current_task():
t.cancel()
if self._ws:
await self._ws.close()
self._ws = None
self._fail_pending(ResolveSpecError("WebSocket closed"))
self._set_state(DISCONNECTED)
def is_connected(self) -> bool:
return self._ws is not None and self._state == CONNECTED
@property
def state(self) -> str:
return self._state
def on(self, event: str, callback: Callback) -> None:
if event not in EVENTS:
raise ValueError(f"unknown event {event!r}; expected one of {EVENTS}")
self._listeners[event] = callback
def off(self, event: str) -> None:
self._listeners.pop(event, None)
def get_subscriptions(self) -> List[Subscription]:
return list(self._subscriptions.values())
# ---- operations ------------------------------------------------------
async def request(
self,
operation: str,
entity: str,
*,
schema: Optional[str] = None,
record_id: Optional[str] = None,
data: Any = None,
options: Optional[Dict[str, Any]] = None,
) -> Any:
message = _drop_none({
"type": "request",
"operation": operation,
"entity": entity,
"schema": schema,
"record_id": record_id,
"data": data,
"options": options,
})
response = await self._call(message, self.request_timeout, "Request")
return response.get("data")
async def read(
self,
entity: str,
*,
schema: Optional[str] = None,
record_id: Optional[str] = None,
filters: Optional[List[FilterOption]] = None,
columns: Optional[List[str]] = None,
sort: Optional[List[SortOption]] = None,
preload: Optional[List[PreloadOption]] = None,
limit: Optional[int] = None,
offset: Optional[int] = None,
) -> Any:
options = _drop_none({
"filters": filters, "columns": columns, "sort": sort,
"preload": preload, "limit": limit, "offset": offset,
})
return await self.request("read", entity, schema=schema, record_id=record_id, options=options)
async def create(self, entity: str, data: Any, *, schema: Optional[str] = None) -> Any:
return await self.request("create", entity, schema=schema, data=data)
async def update(self, entity: str, id: str, data: Any, *, schema: Optional[str] = None) -> Any:
return await self.request("update", entity, schema=schema, record_id=id, data=data)
async def delete(self, entity: str, id: str, *, schema: Optional[str] = None) -> None:
await self.request("delete", entity, schema=schema, record_id=id)
async def meta(self, entity: str, *, schema: Optional[str] = None) -> Any:
return await self.request("meta", entity, schema=schema)
async def subscribe(
self,
entity: str,
callback: Callback,
*,
schema: Optional[str] = None,
filters: Optional[List[FilterOption]] = None,
) -> str:
message = _drop_none({
"type": "subscription",
"operation": "subscribe",
"entity": entity,
"schema": schema,
"options": _drop_none({"filters": filters}),
})
response = await self._call(message, self.subscribe_timeout, "Subscription")
sub_id = (response.get("data") or {}).get("subscription_id")
if not sub_id:
raise ResolveSpecError("Subscription failed")
self._subscriptions[sub_id] = Subscription(
sub_id, entity, schema, _drop_none({"filters": filters}) or None, callback
)
return sub_id
async def unsubscribe(self, subscription_id: str) -> None:
message = {"type": "subscription", "operation": "unsubscribe", "subscription_id": subscription_id}
await self._call(message, self.subscribe_timeout, "Unsubscribe")
self._subscriptions.pop(subscription_id, None)
# ---- internals -------------------------------------------------------
async def _call(self, message: Dict[str, Any], timeout: float, what: str) -> Dict[str, Any]:
self._ensure_connected()
mid = str(uuid.uuid4())
message["id"] = mid
fut: "asyncio.Future[Dict[str, Any]]" = asyncio.get_running_loop().create_future()
self._pending[mid] = fut
try:
await self._ws.send(json.dumps(message)) # type: ignore[union-attr]
response = await asyncio.wait_for(fut, timeout)
except asyncio.TimeoutError:
raise ResolveSpecError(f"{what} timeout") from None
finally:
self._pending.pop(mid, None)
if not response.get("success"):
err = response.get("error") or {}
raise ResolveSpecError(
err.get("message") or f"{what} failed", code=err.get("code"), details=err.get("details")
)
return response
def _ensure_connected(self) -> None:
if not self.is_connected():
raise ResolveSpecError("WebSocket is not connected. Call connect() first.")
def _fail_pending(self, exc: Exception) -> None:
for fut in self._pending.values():
if not fut.done():
fut.set_exception(exc)
self._pending.clear()
async def _read_loop(self, ws: ClientConnection) -> None:
try:
async for raw in ws:
await self._handle_message(raw)
except asyncio.CancelledError:
raise
except Exception as e: # connection error
await self._emit("error", e)
# connection ended
if ws is not self._ws:
return
self._ws = None
if hb := getattr(self, "_heartbeat", None):
hb.cancel()
self._fail_pending(ResolveSpecError("WebSocket disconnected"))
self._set_state(DISCONNECTED)
await self._emit("disconnect", ws.close_code, ws.close_reason)
if self.reconnect and not self._manual_close:
self._reconnect_task = asyncio.create_task(self._reconnect())
async def _reconnect(self) -> None:
for attempt in range(1, self.max_reconnect_attempts + 1):
if self._manual_close:
return
log.debug("Reconnection attempt %d/%d", attempt, self.max_reconnect_attempts)
self._set_state(RECONNECTING)
await asyncio.sleep(self.reconnect_interval)
try:
await self.connect()
return
except Exception as e:
log.debug("Reconnection failed: %s", e)
self._set_state(DISCONNECTED)
async def _handle_message(self, raw: Union[str, bytes]) -> None:
try:
message = json.loads(raw)
except ValueError as e:
log.debug("Error parsing message: %s", e)
return
await self._emit("message", message)
kind = message.get("type")
if kind == "response":
fut = self._pending.get(message.get("id"))
if fut and not fut.done():
fut.set_result(message)
elif kind == "notification":
sub = self._subscriptions.get(message.get("subscription_id"))
if sub and sub.callback:
await _maybe_await(sub.callback(message))
elif kind != "pong":
log.debug("Unknown message type: %s", kind)
async def _heartbeat_loop(self) -> None:
try:
while True:
await asyncio.sleep(self.heartbeat_interval)
if self.is_connected():
await self._ws.send(json.dumps({"id": str(uuid.uuid4()), "type": "ping"})) # type: ignore[union-attr]
except asyncio.CancelledError:
raise
except Exception as e:
log.debug("Heartbeat failed: %s", e)
def _set_state(self, state: str) -> None:
if self._state != state:
self._state = state
cb = self._listeners.get("state_change")
if cb:
res = cb(state)
if asyncio.iscoroutine(res):
asyncio.ensure_future(res)
async def _emit(self, event: str, *args: Any) -> None:
cb = self._listeners.get(event)
if cb:
await _maybe_await(cb(*args))
async def _maybe_await(result: Any) -> None:
if asyncio.iscoroutine(result) or isinstance(result, asyncio.Future):
await result
@@ -0,0 +1,132 @@
import httpx
import pytest
from resolvespec import AsyncFuncSpecClient, FuncSpecClient, ResolveSpecError
from resolvespec.funcspec import build_headers, build_query
from resolvespec.headerspec import decode_header_value
def make(handler, **kw):
return FuncSpecClient("http://localhost:3000", "tok", transport=httpx.MockTransport(handler), **kw)
def capture(status=200, body=None, headers=None):
seen = []
def handler(req):
seen.append(req)
return httpx.Response(status, json=body if body is not None else [], headers=headers)
return seen, handler
def test_filters():
h = build_headers({"filters": [
{"column": "status", "operator": "eq", "value": "active"},
{"column": "age", "operator": "gte", "value": 18},
{"column": "name", "operator": "contains", "value": "x", "logic_operator": "OR"},
{"column": "deleted", "operator": "is_null", "value": None},
{"column": "id", "operator": "in", "value": [1, 2]},
{"column": "p", "operator": "between_inclusive", "value": [1, 5]},
]})
assert h == {
"X-FieldFilter-status": "active",
"X-SearchOp-greaterthanorequal-age": "18",
"X-SearchOr-contains-name": "x",
"X-SearchOp-empty-deleted": "",
"X-SearchOp-in-id": "1,2",
"X-SearchOp-betweeninclusive-p": "1,5",
}
def test_sort_is_sql_not_prefixed():
# server inserts sort verbatim into ORDER BY; "-col" would negate the column
h = build_headers({"sort": [{"column": "name", "direction": "asc"}, {"column": "created_at", "direction": "DESC"}]})
assert h["X-Sort"] == "name ASC,created_at DESC"
def test_misc_options():
h = build_headers({
"search_filters": {"name": "bob"}, "custom_sql_where": "a = 1", "custom_sql_or": "b = 2",
"limit": 5, "offset": 10, "distinct": True, "skip_count": True, "skip_cache": False,
"response_format": "syncfusion",
})
assert h == {
"X-SearchFilter-name": "bob", "X-Custom-SQL-W": "a = 1", "X-Custom-SQL-Or": "b = 2",
"X-Limit": "5", "X-Offset": "10", "X-Distinct": "true", "X-SkipCount": "true",
"X-SkipCache": "false", "X-Syncfusion": "true",
}
def test_ambiguous_values_are_encoded():
h = build_headers({"custom_sql_where": "name = 'café'", "filters": [{"column": "c", "operator": "eq", "value": " pad "}]})
assert h["X-Custom-SQL-W"].startswith("ZIP_")
assert decode_header_value(h["X-Custom-SQL-W"]) == "name = 'café'"
assert decode_header_value(h["X-FieldFilter-c"]) == " pad "
def test_build_query():
q = build_query({"p-id": 5, "flag": True, "ids": [1, 2], "skip": None, "m": "match=ab"})
assert q == {"p-id": "5", "flag": "true", "ids": ["1", "2"], "m": "match=ab"}
def test_query_list_request_and_metadata():
seen, h = capture(206, [{"id": 1}, {"id": 2}], {"content-range": "items 10-12/50"})
with make(h) as c:
res = c.query_list("/api/orders", {"p-status": "open", "id": [1, 2]}, {"limit": 2, "offset": 10})
r = seen[0]
assert r.method == "GET"
assert r.url.path == "/api/orders"
assert r.url.params.multi_items() == [("p-status", "open"), ("id", "1"), ("id", "2")]
assert r.headers["x-limit"] == "2" and r.headers["authorization"] == "Bearer tok"
assert res == {
"success": True,
"data": [{"id": 1}, {"id": 2}],
"metadata": {"total": 50, "count": 2, "filtered": 50, "offset": 10, "limit": 2},
}
def test_query_list_empty_result():
seen, h = capture(200, [], {"content-range": "items 0-0/0"})
with make(h) as c:
assert c.query_list("orders")["metadata"]["total"] == 0
assert seen[0].url.path == "/orders"
def test_query_single_has_no_metadata_and_method():
seen, h = capture(200, {"id": 1})
with make(h) as c:
res = c.query("api/order", method="post")
assert seen[0].method == "POST"
assert res == {"success": True, "data": {"id": 1}}
def test_detail_format_data_passthrough():
body = {"items": [{"a": 1}], "count": "1", "total": "1", "tablename": "/x", "tableprefix": "gsql"}
_, h = capture(200, body, {"content-range": "items 0-1/1"})
with make(h) as c:
assert c.query_list("x", options={"response_format": "detail"})["data"] == body
def test_server_error_shape():
err = {"success": False, "error": {"code": "query_failed", "message": "Failed to retrieve records", "detail": "no such column", "sql": "SELECT"}}
_, h = capture(400, err)
with make(h) as c:
with pytest.raises(ResolveSpecError, match="Failed to retrieve") as ei:
c.query_list("x")
assert ei.value.code == "query_failed" and ei.value.detail == "no such column" and ei.value.status_code == 400
def test_plain_text_panic_error():
with make(lambda r: httpx.Response(500, text="Internal server error: boom")) as c:
with pytest.raises(ResolveSpecError, match="boom"):
c.query("x")
async def test_async():
async def handler(req):
return httpx.Response(200, json=[{"id": 1}], headers={"content-range": "items 0-1/1"})
async with AsyncFuncSpecClient("http://localhost:3000", transport=httpx.MockTransport(handler)) as c:
assert (await c.query_list("x"))["metadata"]["total"] == 1
assert (await c.query("x"))["data"] == [{"id": 1}]
@@ -0,0 +1,236 @@
import json
import httpx
import pytest
from resolvespec import (
AsyncHeaderSpecClient,
HeaderSpecClient,
ResolveSpecError,
build_headers,
decode_header_value,
encode_header_value,
get_headerspec_client,
)
import base64
CFG = dict(base_url="http://localhost:3000", token="tok")
# ---- build_headers (ported from headerspec.test.ts) ----
def test_preload_shared_where():
h = build_headers({"preload": [
{"relation": "Items", "columns": ["id"], "where": "active = true"},
{"relation": "Tags", "where": "active = true"},
]})
assert h["X-Preload"] == "Items:id|Tags"
assert h["X-Preload-Where"] == "active = true"
def test_preload_mixed_where_numbered():
h = build_headers({"preload": [
{"relation": "Items", "where": "a = 1"},
{"relation": "Category"},
{"relation": "Tags", "where": "b = 2"},
]})
assert h["X-Preload"] == "Category"
assert "X-Preload-Where" not in h
assert h["X-Preload-1"] == "Items" and h["X-Preload-1-Where"] == "a = 1"
assert h["X-Preload-2"] == "Tags" and h["X-Preload-2-Where"] == "b = 2"
def test_expand_joins_or_searchcols_advsql():
h = build_headers({
"expand": [{"relation": "Dept", "columns": ["id", "name"]}, {"relation": "Role"}],
"custom_sql_joins": ["LEFT JOIN a ON a.id = b.id", "INNER JOIN c ON c.id = b.cid"],
"custom_sql_or": ["x = 1", "y = 2"],
"search_columns": ["name", "email"],
"advanced_sql": {"total": "a + b"},
})
assert h["X-Expand"] == "Dept:id,name|Role"
assert h["X-Custom-SQL-Join"] == "LEFT JOIN a ON a.id = b.id|INNER JOIN c ON c.id = b.cid"
assert h["X-Custom-SQL-Or"] == "x = 1 OR y = 2"
assert h["X-SearchCols"] == "name,email"
assert h["X-AdvSQL-total"] == "a + b"
def test_flags_pkrow_format():
h = build_headers({
"clean_json": True, "distinct": True, "skip_count": True, "skip_cache": False,
"atomic_transaction": True, "single_record_as_object": False,
"pk_row": "42", "response_format": "detail",
})
assert h["X-Clean-JSON"] == "true"
assert h["X-Distinct"] == "true"
assert h["X-SkipCount"] == "true"
assert h["X-SkipCache"] == "false"
assert h["X-Transaction-Atomic"] == "true"
assert h["X-Single-Record-As-Object"] == "false"
assert h["X-PKRow"] == "42"
assert h["X-DetailApi"] == "true"
def test_spatial_and_vector_filters():
h = build_headers({"filters": [
{"column": "geom", "operator": "st_dwithin", "value": {"geom": "POINT(0 0)", "distance": 5}, "logic_operator": "OR"},
{"column": "emb", "operator": "cosine_within", "value": {"vector": [1, 2], "distance": 0.3}},
]})
assert json.loads(h["X-SpatialFilter-geom"]) == {
"op": "st_dwithin", "value": {"geom": "POINT(0 0)", "distance": 5}, "logic": "or"}
assert json.loads(h["X-VectorFilter-emb"])["op"] == "cosine_within"
def test_vector_search():
h = build_headers({"vector_search": {"column": "emb", "vector": [0.1, 0.2], "metric": "cosine", "as": "dist", "direction": "desc"}})
assert h["X-Vector-Search-emb"] == "cosine"
assert h["X-Vector-Search-Vector"] == "[0.1,0.2]"
assert h["X-Vector-Search-As"] == "dist"
assert h["X-Vector-Search-Dir"] == "desc"
def test_xfiles_zip():
xf = {"tablename": "users", "prefix": "USR", "limit": 10}
h = build_headers({"xfiles": xf})
assert h["X-Files"].startswith("ZIP_")
assert json.loads(decode_header_value(h["X-Files"])) == xf
def test_columns_and_omit():
assert build_headers({"columns": ["id", "name", "email"]})["X-Select-Fields"] == "id,name,email"
assert build_headers({"omit_columns": ["secret", "internal"]})["X-Not-Select-Fields"] == "secret,internal"
def test_filters():
assert build_headers({"filters": [{"column": "status", "operator": "eq", "value": "active"}]})["X-FieldFilter-status"] == "active"
assert build_headers({"filters": [{"column": "age", "operator": "gte", "value": 18}]})["X-SearchOp-greaterthanorequal-age"] == "18"
assert build_headers({"filters": [{"column": "name", "operator": "contains", "value": "test", "logic_operator": "OR"}]})["X-SearchOr-contains-name"] == "test"
assert build_headers({"filters": [{"column": "price", "operator": "between", "value": [10, 100]}]})["X-SearchOp-between-price"] == "10,100"
assert build_headers({"filters": [{"column": "deleted_at", "operator": "is_null", "value": None}]})["X-SearchOp-empty-deleted_at"] == ""
assert build_headers({"filters": [{"column": "id", "operator": "in", "value": [1, 2, 3]}]})["X-SearchOp-in-id"] == "1,2,3"
assert build_headers({"filters": [{"column": "a", "operator": "eq", "value": True}]})["X-FieldFilter-a"] == "true"
def test_sort_pagination_cursor():
h = build_headers({
"sort": [{"column": "name", "direction": "asc"}, {"column": "created_at", "direction": "DESC"}],
"limit": 25, "offset": 0, "cursor_forward": "abc", "cursor_backward": "xyz",
})
assert h["X-Sort"] == "+name,-created_at"
assert h["X-Limit"] == "25" and h["X-Offset"] == "0"
assert h["X-Cursor-Forward"] == "abc" and h["X-Cursor-Backward"] == "xyz"
def test_preload_basic_rownumber_computed_custom():
h = build_headers({
"preload": [{"relation": "Items", "columns": ["id", "name"]}, {"relation": "Category"}],
"fetch_row_number": "42",
"computedColumns": [{"name": "total", "expression": "price * qty"}],
"customOperators": [{"name": "a", "sql": "status = 'active'"}, {"name": "v", "sql": "verified = true"}],
})
assert h["X-Preload"] == "Items:id,name|Category"
assert h["X-Fetch-RowNumber"] == "42"
assert h["X-CQL-SEL-total"] == "price * qty"
assert h["X-Custom-SQL-W"] == "status = 'active' AND verified = true"
def test_empty_options():
assert build_headers({}) == {}
# ---- encode / decode ----
def test_roundtrip():
for s in ("some complex value with spaces & symbols!", "café ☕ 你好"):
enc = encode_header_value(s)
assert enc.startswith("ZIP_")
assert decode_header_value(enc) == s
def test_decode_double_underscore_and_plain():
assert decode_header_value("__" + base64.b64encode(b"hello").decode()) == "hello"
assert decode_header_value("__" + base64.b64encode("café ☕".encode()).decode()) == "café ☕"
assert decode_header_value("plain") == "plain"
def test_decode_nested():
assert decode_header_value(encode_header_value(encode_header_value("x"))) == "x"
# ---- client ----
def make(handler, cls=HeaderSpecClient, **kw):
return cls(**{**CFG, **kw}, transport=httpx.MockTransport(handler))
def test_read_sends_get_with_headers():
seen = []
def handler(req):
seen.append(req)
return httpx.Response(200, json=[{"id": 1}], headers={"content-range": "0-9/100", "x-limit": "10"})
with make(handler) as c:
res = c.read("public", "users", options={"columns": ["id", "name"], "limit": 10})
r = seen[0]
assert str(r.url) == "http://localhost:3000/public/users"
assert r.method == "GET"
assert r.headers["x-select-fields"] == "id,name"
assert r.headers["x-limit"] == "10"
assert r.headers["authorization"] == "Bearer tok"
assert res["success"] is True
assert res["data"] == [{"id": 1}]
assert res["metadata"] == {"count": 100, "total": 100, "filtered": 100, "offset": 0, "limit": 10}
def test_metadata_defaults_without_content_range():
with make(lambda r: httpx.Response(200, json=[])) as c:
assert c.read("public", "users")["metadata"]["total"] == 0
def test_read_with_id_create_update_delete():
seen = []
def handler(req):
seen.append(req)
return httpx.Response(200, json={})
with make(handler) as c:
c.read("public", "users", "42")
c.create("public", "users", {"name": "Test"})
c.update("public", "users", "1", {"name": "Updated"}, {"filters": [{"column": "active", "operator": "eq", "value": True}]})
c.delete("public", "users", "1")
assert str(seen[0].url) == "http://localhost:3000/public/users/42"
assert seen[1].method == "POST" and json.loads(seen[1].content) == {"name": "Test"}
assert seen[2].method == "PUT" and str(seen[2].url).endswith("/public/users/1")
assert seen[2].headers["x-fieldfilter-active"] == "true"
assert seen[3].method == "DELETE"
def test_error_response():
with make(lambda r: httpx.Response(400, json={"error": {"code": "err", "message": "fail"}})) as c:
with pytest.raises(ResolveSpecError, match="fail") as ei:
c.read("public", "users")
assert ei.value.status_code == 400 and ei.value.code == "err"
def test_error_non_json():
with make(lambda r: httpx.Response(502, text="bad gateway")) as c:
with pytest.raises(ResolveSpecError, match="bad gateway") as ei:
c.read("public", "users")
assert ei.value.status_code == 502
async def test_async_client():
async def handler(req):
return httpx.Response(200, json=[{"id": 1}])
async with AsyncHeaderSpecClient(**CFG, transport=httpx.MockTransport(handler)) as c:
res = await c.read("public", "users", options={"limit": 1})
assert res["data"] == [{"id": 1}]
def test_singleton():
a = get_headerspec_client("http://hs-singleton:3000")
assert a is get_headerspec_client("http://hs-singleton:3000")
assert a is not get_headerspec_client("http://hs-singleton-b:3000")
@@ -0,0 +1,210 @@
import json
import httpx
import pytest
from resolvespec import (
AsyncHeaderSpecClient,
AsyncResolveSpecClient,
HeaderSpecClient,
ResolveSpecClient,
ResolveSpecError,
get_headerspec_client,
get_resolvespec_client,
)
CFG = dict(base_url="http://localhost:3000", token="test-token")
def make(handler, **kw):
return ResolveSpecClient(**{**CFG, **kw}, transport=httpx.MockTransport(handler))
def ok(_req):
return httpx.Response(200, json={"success": True, "data": [{"id": 1}]})
def capture():
seen = []
def handler(req):
seen.append(req)
return httpx.Response(200, json={"success": True, "data": {"id": 1, "name": "Test"}})
return seen, handler
def body(req):
return json.loads(req.content)
def test_read_with_numeric_id():
seen, h = capture()
with make(h) as c:
assert c.read("public", "users", 1)["success"] is True
r = seen[0]
assert str(r.url) == "http://localhost:3000/public/users/1"
assert r.method == "POST"
assert r.headers["authorization"] == "Bearer test-token"
assert r.headers["content-type"] == "application/json"
assert body(r) == {"operation": "read"}
def test_read_array_id_goes_in_body():
seen, h = capture()
with make(h) as c:
c.read("public", "users", ["1", "2"])
assert str(seen[0].url) == "http://localhost:3000/public/users"
assert body(seen[0])["id"] == ["1", "2"]
def test_read_options_passthrough():
seen, h = capture()
opts = {
"columns": ["id", "name"], "omit_columns": ["secret"],
"filters": [{"column": "active", "operator": "eq", "value": True}],
"sort": [{"column": "name", "direction": "asc"}],
"limit": 10, "offset": 0, "cursor_forward": "cursor1", "fetch_row_number": "5",
"customOperators": [{"name": "x", "sql": "a = 1"}],
}
with make(h) as c:
c.read("public", "users", options=opts)
assert body(seen[0])["options"] == opts
def test_create():
seen, h = capture()
with make(h) as c:
res = c.create("public", "users", {"name": "Test"})
assert res["data"]["name"] == "Test"
assert body(seen[0]) == {"operation": "create", "data": {"name": "Test"}}
def test_create_batch():
seen, h = capture()
with make(h) as c:
c.create("public", "users", [{"a": 1}, {"a": 2}])
assert body(seen[0])["data"] == [{"a": 1}, {"a": 2}]
def test_update_with_id_in_url_and_array():
seen, h = capture()
with make(h) as c:
c.update("public", "users", {"name": "X"}, 5)
c.update("public", "users", {"name": "X"}, ["1", "2"])
assert str(seen[0].url).endswith("/public/users/5")
assert body(seen[0]) == {"operation": "update", "data": {"name": "X"}}
assert str(seen[1].url).endswith("/public/users")
assert body(seen[1])["id"] == ["1", "2"]
def test_update_preserves_empty_string_and_null():
seen, h = capture()
with make(h) as c:
c.update("public", "users", {"a": "", "b": None}, 1)
assert body(seen[0])["data"] == {"a": "", "b": None}
def test_delete():
seen, h = capture()
with make(h) as c:
c.delete("public", "users", 1)
assert str(seen[0].url).endswith("/public/users/1")
assert body(seen[0]) == {"operation": "delete"}
def test_get_metadata():
seen, h = capture()
with make(h) as c:
c.get_metadata("public", "users")
assert seen[0].method == "GET"
assert str(seen[0].url) == "http://localhost:3000/public/users"
assert not seen[0].content
def test_error_uses_server_message():
with make(lambda r: httpx.Response(404, json={"success": False, "error": {"code": "not_found", "message": "nope"}})) as c:
with pytest.raises(ResolveSpecError, match="nope") as ei:
c.read("public", "users", 1)
assert ei.value.status_code == 404 and ei.value.code == "not_found"
def test_id_is_url_quoted():
seen, h = capture()
with make(h) as c:
c.read("public", "users", "a/b")
assert str(seen[0].url).endswith("/public/users/a%2Fb")
def test_trailing_slash_base_url():
seen, h = capture()
with make(h, base_url="http://localhost:3000/") as c:
c.read("public", "users")
assert str(seen[0].url) == "http://localhost:3000/public/users"
async def test_async_client():
async def handler(req):
return httpx.Response(200, json={"success": True, "data": [1]})
async with AsyncResolveSpecClient(**CFG, transport=httpx.MockTransport(handler)) as c:
assert (await c.read("public", "users"))["data"] == [1]
assert (await c.create("public", "users", {}))["success"]
assert (await c.update("public", "users", {}, 1))["success"]
assert (await c.delete("public", "users", 1))["success"]
assert (await c.get_metadata("public", "users"))["success"]
# ---- custom headers (ported from custom-headers.test.ts) ----
@pytest.mark.parametrize("cls", [ResolveSpecClient, HeaderSpecClient])
def test_custom_headers_on_every_op_case_insensitive(cls):
seen = []
def handler(req):
seen.append(req)
return httpx.Response(200, json={"success": True, "data": []})
headers = {"X-Tenant": "acme", "authorization": "Basic ignored",
"content-type": "application/custom+json", "x-limit": "99"}
with cls("http://localhost:3000", "tok", headers, transport=httpx.MockTransport(handler)) as c:
c.read("public", "users", options={"limit": 10})
c.create("public", "users", {})
if cls is ResolveSpecClient:
c.update("public", "users", {}, "1")
c.get_metadata("public", "users")
else:
c.update("public", "users", "1", {})
c.delete("public", "users", "1")
for r in seen:
assert r.headers["x-tenant"] == "acme"
assert r.headers["authorization"] == "Bearer tok"
assert r.headers["content-type"] == "application/custom+json"
if cls is HeaderSpecClient:
assert seen[0].headers["x-limit"] == "10"
assert headers["authorization"] == "Basic ignored"
assert headers["x-limit"] == "99"
@pytest.mark.parametrize("cls", [ResolveSpecClient, HeaderSpecClient])
def test_custom_auth_without_token(cls):
seen = []
def handler(req):
seen.append(req)
return httpx.Response(200, json={"success": True, "data": []})
with cls("http://localhost:3000", headers={"Authorization": "Basic custom"}, transport=httpx.MockTransport(handler)) as c:
c.read("public", "users")
assert seen[0].headers["authorization"] == "Basic custom"
@pytest.mark.parametrize("factory", [get_resolvespec_client, get_headerspec_client])
def test_cache_isolation_and_snapshot(factory):
headers = {"X-Tenant": "acme", "X-App": "grid"}
first = factory("http://tenant-cache", "one", headers)
assert factory("http://tenant-cache", "one", {"x-app": "grid", "x-tenant": "acme"}) is first
assert factory("http://tenant-cache", "two", headers) is not first
headers["X-Tenant"] = "other"
assert factory("http://tenant-cache", "one", headers) is not first
assert first.headers["X-Tenant"] == "acme"
@@ -0,0 +1,152 @@
import asyncio
import json
import pytest
from websockets.asyncio.server import serve
from resolvespec import ResolveSpecError, WebSocketClient
class Server:
"""Minimal in-process WebSocketSpec server."""
def __init__(self):
self.received = []
self.conns = set()
self.respond = True
async def handler(self, ws):
self.conns.add(ws)
try:
async for raw in ws:
msg = json.loads(raw)
self.received.append(msg)
if msg["type"] == "ping":
await ws.send(json.dumps({"type": "pong"}))
continue
if not self.respond:
continue
await ws.send(json.dumps(self.reply(msg)))
finally:
self.conns.discard(ws)
def reply(self, msg):
base = {"id": msg["id"], "type": "response", "success": True, "timestamp": "t"}
if msg["type"] == "subscription" and msg["operation"] == "subscribe":
return {**base, "data": {"subscription_id": "sub-1"}}
if msg.get("entity") == "fail":
return {**base, "success": False, "error": {"code": "bad", "message": "boom"}}
return {**base, "data": {"echo": msg.get("operation"), "record_id": msg.get("record_id")}}
@pytest.fixture
async def server():
s = Server()
async with serve(s.handler, "127.0.0.1", 0) as srv:
s.url = "ws://127.0.0.1:%d" % srv.sockets[0].getsockname()[1]
yield s
async def test_operations_and_message_shape(server):
async with WebSocketClient(server.url, reconnect=False) as c:
assert c.state == "connected"
assert await c.read("users", schema="public", record_id="1", limit=5, filters=[{"column": "a", "operator": "eq", "value": 1}]) == {"echo": "read", "record_id": "1"}
await c.create("users", {"n": 1}, schema="public")
await c.update("users", "2", {"n": 2})
await c.delete("users", "3")
await c.meta("users")
m = server.received
assert m[0]["type"] == "request" and m[0]["operation"] == "read"
assert m[0]["schema"] == "public" and m[0]["record_id"] == "1"
assert m[0]["options"] == {"filters": [{"column": "a", "operator": "eq", "value": 1}], "limit": 5}
assert m[1]["data"] == {"n": 1}
assert m[2]["record_id"] == "2"
assert [x["operation"] for x in m] == ["read", "create", "update", "delete", "meta"]
assert "schema" not in m[2]
assert len({x["id"] for x in m}) == 5
async def test_error_response_raises(server):
async with WebSocketClient(server.url, reconnect=False) as c:
with pytest.raises(ResolveSpecError, match="boom") as ei:
await c.read("fail")
assert ei.value.code == "bad"
async def test_request_timeout(server):
server.respond = False
async with WebSocketClient(server.url, reconnect=False, request_timeout=0.1) as c:
with pytest.raises(ResolveSpecError, match="timeout"):
await c.read("users")
assert not c._pending
async def test_not_connected_raises():
c = WebSocketClient("ws://127.0.0.1:1")
with pytest.raises(ResolveSpecError, match="not connected"):
await c.read("users")
async def test_subscribe_notify_unsubscribe(server):
got = asyncio.Queue()
async with WebSocketClient(server.url, reconnect=False) as c:
sid = await c.subscribe("users", got.put, schema="public", filters=[{"column": "a", "operator": "eq", "value": 1}])
assert sid == "sub-1"
assert [s.id for s in c.get_subscriptions()] == ["sub-1"]
assert server.received[0]["operation"] == "subscribe"
assert server.received[0]["options"] == {"filters": [{"column": "a", "operator": "eq", "value": 1}]}
for ws in server.conns:
await ws.send(json.dumps({"type": "notification", "operation": "create", "subscription_id": "sub-1",
"entity": "users", "data": {"id": 9}, "timestamp": "t"}))
n = await asyncio.wait_for(got.get(), 2)
assert n["data"] == {"id": 9}
await c.unsubscribe("sub-1")
assert c.get_subscriptions() == []
assert server.received[-1] == {**server.received[-1], "operation": "unsubscribe", "subscription_id": "sub-1"}
async def test_events_and_heartbeat(server):
events = []
c = WebSocketClient(server.url, reconnect=False, heartbeat_interval=0.05)
c.on("connect", lambda: events.append("connect"))
c.on("state_change", lambda s: events.append(s))
c.on("message", lambda m: events.append(("msg", m["type"])))
await c.connect()
await asyncio.sleep(0.2)
await c.close()
assert events[:3] == ["connecting", "connected", "connect"]
assert ("msg", "pong") in events
assert events[-1] == "disconnected"
assert any(m["type"] == "ping" for m in server.received)
with pytest.raises(ValueError):
c.on("bogus", lambda: None)
async def test_reconnect_after_server_drop(server):
states = []
c = WebSocketClient(server.url, reconnect=True, reconnect_interval=0.05)
c.on("state_change", states.append)
await c.connect()
for ws in list(server.conns):
await ws.close()
for _ in range(100):
if states.count("connected") >= 2:
break
await asyncio.sleep(0.05)
assert "reconnecting" in states
assert c.is_connected()
assert (await c.read("users"))["echo"] == "read"
await c.close()
async def test_pending_requests_fail_on_disconnect(server):
server.respond = False
c = WebSocketClient(server.url, reconnect=False)
await c.connect()
task = asyncio.create_task(c.read("users"))
await asyncio.sleep(0.05)
for ws in list(server.conns):
await ws.close()
with pytest.raises(ResolveSpecError, match="disconnected"):
await asyncio.wait_for(task, 2)
await c.close()
@@ -4,34 +4,34 @@
### 1. ResolveSpec Client API
- [ ] Core API implementation (read, create, update, delete, get_metadata)
- [ ] Unit tests for API functions
- [x] Core API implementation (read, create, update, delete, get_metadata)
- [x] Unit tests for API functions
- [ ] Integration tests with server
- [ ] Error handling and edge cases
- [x] Error handling and edge cases
### 2. HeaderSpec Client API
- [ ] Client API implementation
- [ ] Unit tests
- [x] Client API implementation
- [x] Unit tests
- [ ] Integration tests with server
### 3. FunctionSpec Client API
- [ ] Client API implementation
- [ ] Unit tests
- [x] Client API implementation
- [x] Unit tests
- [ ] Integration tests with server
### 4. WebSocketSpec Client API
- [ ] WebSocketClient class implementation (read, create, update, delete, meta, subscribe, unsubscribe)
- [ ] Unit tests for WebSocketClient
- [ ] Connection handling tests
- [ ] Subscription tests
- [x] WebSocketClient class implementation (read, create, update, delete, meta, subscribe, unsubscribe)
- [x] Unit tests for WebSocketClient
- [x] Connection handling tests
- [x] Subscription tests
- [ ] Integration tests with server
### 5. Testing Infrastructure
- [ ] Set up test framework (pytest)
- [x] Set up test framework (pytest)
- [ ] Configure test coverage reporting (pytest-cov)
- [ ] Add test utilities and fixtures
- [ ] Create test documentation
@@ -43,8 +43,8 @@
- [ ] Usage examples for each client API
- [ ] Installation guide
- [ ] Contributing guidelines
- [ ] README with quick start
- [x] README (cheatsheet)
---
**Last Updated:** 2026-02-07
**Last Updated:** 2026-09-30
+2
View File
@@ -0,0 +1,2 @@
target/
Cargo.lock
+18
View File
@@ -0,0 +1,18 @@
[package]
name = "resolvespec"
version = "0.1.0"
edition = "2021"
rust-version = "1.80"
description = "Client for ResolveSpec (JSON body) and FunctionSpec endpoints"
license = "MIT"
[dependencies]
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
base64 = "0.22"
thiserror = "1"
[dev-dependencies]
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
wiremock = "0.6"
+40
View File
@@ -0,0 +1,40 @@
# resolvespec (Rust)
Rust client for ResolveSpec (JSON body) and FunctionSpec. Async (`reqwest` + `tokio`). MSRV 1.80.
## Clients
| Type | Constructor | Methods |
|---|---|---|
| `ResolveSpecClient` | `new(base_url)` / `from_builder(ClientBuilder)` | `get_metadata` `read` `create` `update` `delete` |
| `FuncSpecClient` | `new(base_url)` / `from_builder(ClientBuilder)` | `query` `query_list` `request` |
`ClientBuilder::new(url).token().header().timeout().http_client()`. Precedence: Content-Type < custom headers < bearer token.
## ResolveSpec
- `RecordId`: `Int`/`Str` → URL, `Many(Vec<String>)` → body (`From` impls provided).
- `Options` (`Default` + struct update), optional fields are `Option`/empty `Vec`.
- Result: `Response{success, data: serde_json::Value, metadata}`; `resp.decode::<T>()`.
## FunctionSpec
- Routes are server-defined: pass the `path`.
- `Params = BTreeMap<String, Param>` → query string (`Param::List` → repeated keys).
- `FuncSpecOptions` → `X-*` headers: `filters`, `search_filters`, `custom_sql_where`, `custom_sql_or`, `sort`, `limit`, `offset`, `distinct`, `skip_count`, `skip_cache`, `response_format`.
- `query_list` fills `metadata` from `Content-Range`; 206 is success.
## Server quirks
- `sort` is raw SQL in ORDER BY (client sends `col ASC|DESC`).
- One search operator per column.
- Values starting `ZIP_` / `__` are base64-decoded by the server.
- Non-ASCII, control chars and edge spaces are auto-encoded (`ZIP_`).
## Errors
`Error::Api { status, message, error: ApiError{code, message, detail, sql} }`, `Error::Http`, `Error::Json`.
## Test
`cargo test`
+90
View File
@@ -0,0 +1,90 @@
use std::collections::HashMap;
use std::time::Duration;
use reqwest::header::{HeaderMap, HeaderName, HeaderValue, AUTHORIZATION, CONTENT_TYPE};
use crate::error::Result;
/// Shared HTTP configuration.
#[derive(Clone)]
pub(crate) struct Config {
pub base_url: String,
pub token: Option<String>,
pub headers: HashMap<String, String>,
pub http: reqwest::Client,
}
/// Builder options shared by both clients.
#[derive(Default, Clone)]
pub struct ClientBuilder {
base_url: String,
token: Option<String>,
headers: HashMap<String, String>,
timeout: Option<Duration>,
http: Option<reqwest::Client>,
}
impl ClientBuilder {
pub fn new(base_url: &str) -> Self {
Self { base_url: base_url.trim_end_matches('/').into(), timeout: Some(Duration::from_secs(30)), ..Default::default() }
}
pub fn token(mut self, token: &str) -> Self {
self.token = Some(token.into());
self
}
pub fn header(mut self, name: &str, value: &str) -> Self {
self.headers.insert(name.into(), value.into());
self
}
pub fn timeout(mut self, t: Duration) -> Self {
self.timeout = Some(t);
self
}
pub fn http_client(mut self, c: reqwest::Client) -> Self {
self.http = Some(c);
self
}
pub(crate) fn config(self) -> Result<Config> {
let http = match self.http {
Some(c) => c,
None => {
let mut b = reqwest::Client::builder();
if let Some(t) = self.timeout {
b = b.timeout(t);
}
b.build()?
}
};
Ok(Config { base_url: self.base_url, token: self.token, headers: self.headers, http })
}
}
impl Config {
/// Content-Type < custom headers < extra (per-call) < bearer token.
pub fn headers(&self, extra: &HashMap<String, String>) -> HeaderMap {
let mut m = HeaderMap::new();
m.insert(CONTENT_TYPE, HeaderValue::from_static("application/json"));
for (k, v) in self.headers.iter().chain(extra.iter()) {
if let (Ok(n), Ok(v)) = (HeaderName::try_from(k.as_str()), HeaderValue::from_str(v)) {
m.insert(n, v);
}
}
if let Some(t) = &self.token {
if let Ok(v) = HeaderValue::from_str(&format!("Bearer {t}")) {
m.insert(AUTHORIZATION, v);
}
}
m
}
}
pub(crate) fn path_segment(s: &str) -> String {
let mut out = String::new();
for b in s.bytes() {
match b {
b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'.' | b'_' | b'~' => out.push(b as char),
_ => out.push_str(&format!("%{b:02X}")),
}
}
out
}
+34
View File
@@ -0,0 +1,34 @@
use crate::types::ApiError;
/// Returned on transport failure, a non-2xx response or an unsuccessful API result.
#[derive(Debug, thiserror::Error)]
pub enum Error {
#[error("{message}")]
Api { status: u16, message: String, error: ApiError },
#[error(transparent)]
Http(#[from] reqwest::Error),
#[error(transparent)]
Json(#[from] serde_json::Error),
}
pub type Result<T> = std::result::Result<T, Error>;
pub(crate) fn error_from(status: u16, body: &str) -> Error {
let parsed: Option<serde_json::Value> = serde_json::from_str(body).ok();
let err: ApiError = parsed
.as_ref()
.and_then(|v| v.get("error"))
.and_then(|e| serde_json::from_value(e.clone()).ok())
.unwrap_or_default();
let message = if !err.message.is_empty() {
err.message.clone()
} else {
let text = if parsed.is_none() { body.trim().chars().take(200).collect::<String>() } else { String::new() };
if text.is_empty() {
format!("{} ({})", reqwest::StatusCode::from_u16(status).ok().and_then(|s| s.canonical_reason()).unwrap_or("Error"), status)
} else {
text
}
};
Error::Api { status, message, error: err }
}
+263
View File
@@ -0,0 +1,263 @@
use std::collections::{BTreeMap, HashMap};
use base64::{engine::general_purpose::STANDARD, Engine};
use reqwest::Method;
use serde_json::Value;
use crate::client::{ClientBuilder, Config};
use crate::error::{error_from, Result};
use crate::types::{FilterOption, Metadata, Response, SortOption};
/// Options sent to funcspec endpoints as `X-*` headers.
///
/// Server behaviour (`pkg/funcspec`): `sort` is inserted raw into ORDER BY (so it is sent as SQL
/// terms); only one search operator per column is kept; values starting with `ZIP_` or `__`
/// are base64-decoded by the server, so such plaintext values cannot be sent faithfully.
#[derive(Debug, Clone, Default)]
pub struct FuncSpecOptions {
/// eq+AND -> X-FieldFilter; others X-SearchOp / X-SearchOr.
pub filters: Vec<FilterOption>,
/// X-SearchFilter-{col}: text ILIKE.
pub search_filters: BTreeMap<String, String>,
pub custom_sql_where: Option<String>,
pub custom_sql_or: Option<String>,
pub sort: Vec<SortOption>,
pub limit: Option<i64>,
pub offset: Option<i64>,
pub distinct: Option<bool>,
pub skip_count: Option<bool>,
pub skip_cache: Option<bool>,
/// simple | detail | syncfusion
pub response_format: Option<String>,
}
/// Query-string parameter value. `List` is sent as repeated keys (server: IN filter).
#[derive(Debug, Clone)]
pub enum Param {
Str(String),
Int(i64),
Bool(bool),
List(Vec<String>),
}
impl From<&str> for Param {
fn from(v: &str) -> Self {
Self::Str(v.into())
}
}
impl From<String> for Param {
fn from(v: String) -> Self {
Self::Str(v)
}
}
impl From<i64> for Param {
fn from(v: i64) -> Self {
Self::Int(v)
}
}
impl From<bool> for Param {
fn from(v: bool) -> Self {
Self::Bool(v)
}
}
impl From<Vec<String>> for Param {
fn from(v: Vec<String>) -> Self {
Self::List(v)
}
}
pub type Params = BTreeMap<String, Param>;
fn operator(op: &str) -> &str {
match op {
"eq" => "equals",
"neq" => "notequals",
"gt" => "greaterthan",
"gte" => "greaterthanorequal",
"lt" => "lessthan",
"lte" => "lessthanorequal",
"like" | "ilike" | "contains" => "contains",
"startswith" => "beginswith",
"endswith" => "endswith",
"in" => "in",
"between" => "between",
"between_inclusive" => "betweeninclusive",
"is_null" => "empty",
"is_not_null" => "notempty",
other => other,
}
}
fn scalar(v: &Value) -> String {
match v {
Value::Null => String::new(),
Value::String(s) => s.clone(),
Value::Array(a) => a.iter().map(scalar).collect::<Vec<_>>().join(","),
other => other.to_string(),
}
}
/// Base64 (UTF-8) with the `ZIP_` prefix.
pub fn encode_header_value(v: &str) -> String {
format!("ZIP_{}", STANDARD.encode(v.as_bytes()))
}
/// Decode a value that may carry a `ZIP_` or `__` prefix (nested allowed).
pub fn decode_header_value(v: &str) -> String {
for p in ["ZIP_", "__"] {
if let Some(rest) = v.strip_prefix(p) {
let mut b64: String = rest.chars().filter(|c| !matches!(c, '\n' | '\r' | ' ')).collect();
while b64.len() % 4 != 0 {
b64.push('=');
}
return match STANDARD.decode(b64).ok().and_then(|b| String::from_utf8(b).ok()) {
Some(s) => decode_header_value(&s),
None => v.to_string(),
};
}
}
v.to_string()
}
/// Encode values that are unsafe as raw header/query text (non-ASCII, control chars, edge spaces).
fn safe(v: &str) -> String {
if v != v.trim() || v.chars().any(|c| !c.is_ascii() || c.is_ascii_control()) {
encode_header_value(v)
} else {
v.to_string()
}
}
/// Build the `X-*` headers understood by `funcspec.ParseParameters`.
pub fn build_headers(o: &FuncSpecOptions) -> BTreeMap<String, String> {
let mut h = BTreeMap::new();
for f in &o.filters {
let logic = f.logic_operator.as_deref().unwrap_or("AND");
let v = safe(&scalar(&f.value));
if f.operator == "eq" && logic == "AND" {
h.insert(format!("X-FieldFilter-{}", f.column), v);
} else {
let kind = if logic == "OR" { "X-SearchOr" } else { "X-SearchOp" };
h.insert(format!("{kind}-{}-{}", operator(&f.operator), f.column), v);
}
}
for (col, text) in &o.search_filters {
h.insert(format!("X-SearchFilter-{col}"), safe(text));
}
if let Some(v) = o.custom_sql_where.as_deref().filter(|s| !s.is_empty()) {
h.insert("X-Custom-SQL-W".into(), safe(v));
}
if let Some(v) = o.custom_sql_or.as_deref().filter(|s| !s.is_empty()) {
h.insert("X-Custom-SQL-Or".into(), safe(v));
}
if !o.sort.is_empty() {
let terms: Vec<String> = o
.sort
.iter()
.map(|s| format!("{} {}", s.column, if s.direction.eq_ignore_ascii_case("desc") { "DESC" } else { "ASC" }))
.collect();
h.insert("X-Sort".into(), safe(&terms.join(","))); // funcspec puts this verbatim into ORDER BY
}
if let Some(n) = o.limit {
h.insert("X-Limit".into(), n.to_string());
}
if let Some(n) = o.offset {
h.insert("X-Offset".into(), n.to_string());
}
for (name, v) in [("X-Distinct", o.distinct), ("X-SkipCount", o.skip_count), ("X-SkipCache", o.skip_cache)] {
if let Some(b) = v {
h.insert(name.into(), b.to_string());
}
}
match o.response_format.as_deref() {
Some("simple") => h.insert("X-SimpleApi".into(), "true".into()),
Some("detail") => h.insert("X-DetailApi".into(), "true".into()),
Some("syncfusion") => h.insert("X-Syncfusion".into(), "true".into()),
_ => None,
};
h
}
/// Build query-string pairs: bools -> true/false, lists -> repeated keys.
pub fn build_query(p: &Params) -> Vec<(String, String)> {
let mut out = Vec::new();
for (k, v) in p {
match v {
Param::Str(s) => out.push((k.clone(), safe(s))),
Param::Int(n) => out.push((k.clone(), n.to_string())),
Param::Bool(b) => out.push((k.clone(), b.to_string())),
Param::List(l) => out.extend(l.iter().map(|s| (k.clone(), safe(s)))),
}
}
out
}
fn metadata(content_range: Option<&str>, limit: Option<i64>) -> Metadata {
let mut m = Metadata { limit: limit.unwrap_or(0), ..Default::default() };
if let Some(cr) = content_range {
// "items {start}-{end}/{total}"
let rest = cr.rsplit(' ').next().unwrap_or("");
if let Some((range, total)) = rest.split_once('/') {
if let (Some((s, e)), Ok(t)) = (range.split_once('-'), total.parse::<i64>()) {
if let (Ok(s), Ok(e)) = (s.parse::<i64>(), e.parse::<i64>()) {
m.total = t;
m.filtered = t;
m.count = e - s;
m.offset = s;
}
}
}
}
m
}
/// Client for user-defined SQL endpoints. Routes are defined by the server application.
#[derive(Clone)]
pub struct FuncSpecClient {
cfg: Config,
}
impl FuncSpecClient {
pub fn new(base_url: &str) -> Result<Self> {
Self::from_builder(ClientBuilder::new(base_url))
}
pub fn from_builder(b: ClientBuilder) -> Result<Self> {
Ok(Self { cfg: b.config()? })
}
async fn call(&self, method: Method, path: &str, params: &Params, options: Option<&FuncSpecOptions>, list: bool) -> Result<Response> {
let url = format!("{}/{}", self.cfg.base_url, path.trim_start_matches('/'));
let extra: HashMap<String, String> = options.map(|o| build_headers(o).into_iter().collect()).unwrap_or_default();
let resp = self.cfg.http.request(method, url).headers(self.cfg.headers(&extra)).query(&build_query(params)).send().await?;
let status = resp.status();
let cr = resp.headers().get("content-range").and_then(|v| v.to_str().ok()).map(str::to_owned);
let text = resp.text().await?;
if !status.is_success() {
// 206 Partial Content is success
return Err(error_from(status.as_u16(), &text));
}
let data = if text.trim().is_empty() { Value::Null } else { serde_json::from_str(&text)? };
Ok(Response {
success: true,
data,
metadata: list.then(|| metadata(cr.as_deref(), options.and_then(|o| o.limit))),
error: None,
})
}
/// Single-record endpoint (`SqlQuery`). `data` is the row object.
pub async fn query(&self, path: &str, params: &Params, options: Option<&FuncSpecOptions>) -> Result<Response> {
self.call(Method::GET, path, params, options, false).await
}
/// List endpoint (`SqlQueryList`). Metadata comes from Content-Range.
pub async fn query_list(&self, path: &str, params: &Params, options: Option<&FuncSpecOptions>) -> Result<Response> {
self.call(Method::GET, path, params, options, true).await
}
/// Like `query` / `query_list` with an explicit HTTP method (routes are app-defined).
pub async fn request(&self, method: Method, path: &str, params: &Params, options: Option<&FuncSpecOptions>, list: bool) -> Result<Response> {
self.call(method, path, params, options, list).await
}
}
+12
View File
@@ -0,0 +1,12 @@
//! Client for ResolveSpec (JSON body) and FunctionSpec endpoints.
mod client;
mod error;
mod funcspec;
mod resolvespec;
pub mod types;
pub use client::ClientBuilder;
pub use error::{Error, Result};
pub use funcspec::{build_headers, build_query, decode_header_value, encode_header_value, FuncSpecClient, FuncSpecOptions, Param, Params};
pub use resolvespec::{RecordId, ResolveSpecClient};
pub use types::*;
+130
View File
@@ -0,0 +1,130 @@
use std::collections::HashMap;
use reqwest::Method;
use serde::Serialize;
use serde_json::Value;
use crate::client::{path_segment, ClientBuilder, Config};
use crate::error::{error_from, Error, Result};
use crate::types::{Options, Response};
/// A record id: a single value goes in the URL, a list goes in the body.
#[derive(Debug, Clone)]
pub enum RecordId {
Int(i64),
Str(String),
Many(Vec<String>),
}
impl From<i64> for RecordId {
fn from(v: i64) -> Self {
Self::Int(v)
}
}
impl From<&str> for RecordId {
fn from(v: &str) -> Self {
Self::Str(v.into())
}
}
impl From<Vec<String>> for RecordId {
fn from(v: Vec<String>) -> Self {
Self::Many(v)
}
}
fn url_id(id: &Option<RecordId>) -> Option<String> {
match id {
Some(RecordId::Int(n)) => Some(n.to_string()),
Some(RecordId::Str(s)) => Some(s.clone()),
_ => None,
}
}
#[derive(Serialize)]
struct Request<'a> {
operation: &'a str,
#[serde(skip_serializing_if = "Option::is_none")]
id: Option<Vec<String>>,
#[serde(skip_serializing_if = "Option::is_none")]
data: Option<Value>,
#[serde(skip_serializing_if = "Option::is_none")]
options: Option<&'a Options>,
}
/// Client for the ResolveSpec JSON body protocol.
#[derive(Clone)]
pub struct ResolveSpecClient {
cfg: Config,
}
impl ResolveSpecClient {
pub fn new(base_url: &str) -> Result<Self> {
Self::from_builder(ClientBuilder::new(base_url))
}
pub fn from_builder(b: ClientBuilder) -> Result<Self> {
Ok(Self { cfg: b.config()? })
}
fn url(&self, schema: &str, entity: &str, id: Option<String>) -> String {
let mut u = format!("{}/{}/{}", self.cfg.base_url, path_segment(schema), path_segment(entity));
if let Some(id) = id.filter(|i| !i.is_empty()) {
u.push('/');
u.push_str(&path_segment(&id));
}
u
}
async fn send(&self, method: Method, url: String, body: Option<Request<'_>>) -> Result<Response> {
let mut req = self.cfg.http.request(method, url).headers(self.cfg.headers(&HashMap::new()));
if let Some(b) = body {
req = req.body(serde_json::to_vec(&b)?);
}
let resp = req.send().await?;
let status = resp.status();
let text = resp.text().await?;
if !status.is_success() {
return Err(error_from(status.as_u16(), &text));
}
let out: Response = serde_json::from_str(&text)?;
if !out.success {
if let Some(e) = out.error.clone() {
return Err(Error::Api { status: status.as_u16(), message: e.message.clone(), error: e });
}
}
Ok(out)
}
/// GET /{schema}/{entity}
pub async fn get_metadata(&self, schema: &str, entity: &str) -> Result<Response> {
self.send(Method::GET, self.url(schema, entity, None), None).await
}
pub async fn read(&self, schema: &str, entity: &str, id: Option<RecordId>, options: Option<&Options>) -> Result<Response> {
let body = Request { operation: "read", id: many(&id), data: None, options };
self.send(Method::POST, self.url(schema, entity, url_id(&id)), Some(body)).await
}
pub async fn create(&self, schema: &str, entity: &str, data: Value, options: Option<&Options>) -> Result<Response> {
let body = Request { operation: "create", id: None, data: Some(data), options };
self.send(Method::POST, self.url(schema, entity, None), Some(body)).await
}
pub async fn update(&self, schema: &str, entity: &str, data: Value, id: Option<RecordId>, options: Option<&Options>) -> Result<Response> {
let body = Request { operation: "update", id: many(&id), data: Some(data), options };
self.send(Method::POST, self.url(schema, entity, url_id(&id)), Some(body)).await
}
pub async fn delete(&self, schema: &str, entity: &str, id: impl Into<RecordId>) -> Result<Response> {
let id = Some(id.into());
let body = Request { operation: "delete", id: None, data: None, options: None };
self.send(Method::POST, self.url(schema, entity, url_id(&id)), Some(body)).await
}
}
fn many(id: &Option<RecordId>) -> Option<Vec<String>> {
match id {
Some(RecordId::Many(v)) => Some(v.clone()),
_ => None,
}
}
+192
View File
@@ -0,0 +1,192 @@
//! Types aligned with Go `pkg/common/types.go`. Field names are the wire names.
use serde::{Deserialize, Serialize};
use serde_json::Value;
use std::collections::HashMap;
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct FilterOption {
pub column: String,
/// eq neq gt gte lt lte like ilike in contains startswith endswith between
/// between_inclusive is_null is_not_null
pub operator: String,
#[serde(default)]
pub value: Value,
#[serde(skip_serializing_if = "Option::is_none")]
pub logic_operator: Option<String>, // AND | OR
}
impl FilterOption {
pub fn new(column: &str, operator: &str, value: impl Into<Value>) -> Self {
Self { column: column.into(), operator: operator.into(), value: value.into(), logic_operator: None }
}
pub fn or(mut self) -> Self {
self.logic_operator = Some("OR".into());
self
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SortOption {
pub column: String,
pub direction: String, // asc | desc
}
impl SortOption {
pub fn new(column: &str, direction: &str) -> Self {
Self { column: column.into(), direction: direction.into() }
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Parameter {
pub name: String,
pub value: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub sequence: Option<i32>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CustomOperator {
pub name: String,
pub sql: String,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ComputedColumn {
pub name: String,
pub expression: String,
}
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct PreloadOption {
#[serde(skip_serializing_if = "Option::is_none")]
pub relation: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub table_name: Option<String>,
#[serde(skip_serializing_if = "Vec::is_empty", default)]
pub columns: Vec<String>,
#[serde(skip_serializing_if = "Vec::is_empty", default)]
pub omit_columns: Vec<String>,
#[serde(skip_serializing_if = "Vec::is_empty", default)]
pub sort: Vec<SortOption>,
#[serde(skip_serializing_if = "Vec::is_empty", default)]
pub filters: Vec<FilterOption>,
#[serde(skip_serializing_if = "Option::is_none")]
pub r#where: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub limit: Option<i64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub offset: Option<i64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub updateable: Option<bool>,
#[serde(skip_serializing_if = "HashMap::is_empty", default)]
pub computed_ql: HashMap<String, String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub recursive: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub primary_key: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub related_key: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub foreign_key: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub recursive_child_key: Option<String>,
#[serde(skip_serializing_if = "Vec::is_empty", default)]
pub sql_joins: Vec<String>,
#[serde(skip_serializing_if = "Vec::is_empty", default)]
pub join_aliases: Vec<String>,
}
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct VectorSearchOption {
pub column: String,
pub vector: Vec<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub metric: Option<String>, // l2 (default) | cosine | ip
#[serde(rename = "as", skip_serializing_if = "Option::is_none")]
pub alias: Option<String>, // distance alias, default _distance
#[serde(skip_serializing_if = "Option::is_none")]
pub direction: Option<String>,
}
/// ResolveSpec request options object.
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct Options {
#[serde(skip_serializing_if = "Vec::is_empty", default)]
pub preload: Vec<PreloadOption>,
#[serde(skip_serializing_if = "Vec::is_empty", default)]
pub columns: Vec<String>,
#[serde(skip_serializing_if = "Vec::is_empty", default)]
pub omit_columns: Vec<String>,
#[serde(skip_serializing_if = "Vec::is_empty", default)]
pub filters: Vec<FilterOption>,
#[serde(skip_serializing_if = "Vec::is_empty", default)]
pub sort: Vec<SortOption>,
#[serde(skip_serializing_if = "Option::is_none")]
pub limit: Option<i64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub offset: Option<i64>,
#[serde(rename = "customOperators", skip_serializing_if = "Vec::is_empty", default)]
pub custom_operators: Vec<CustomOperator>,
#[serde(rename = "computedColumns", skip_serializing_if = "Vec::is_empty", default)]
pub computed_columns: Vec<ComputedColumn>,
#[serde(skip_serializing_if = "Vec::is_empty", default)]
pub parameters: Vec<Parameter>,
#[serde(skip_serializing_if = "Option::is_none")]
pub cursor_forward: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub cursor_backward: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub fetch_row_number: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub vector_search: Option<VectorSearchOption>,
}
#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq, Eq)]
pub struct Metadata {
#[serde(default)]
pub total: i64,
#[serde(default)]
pub count: i64,
#[serde(default)]
pub filtered: i64,
#[serde(default)]
pub limit: i64,
#[serde(default)]
pub offset: i64,
}
/// ResolveSpec envelope. `data` is left as JSON for the caller to decode.
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct Response {
#[serde(default)]
pub success: bool,
#[serde(default)]
pub data: Value,
#[serde(skip_serializing_if = "Option::is_none")]
pub metadata: Option<Metadata>,
#[serde(skip_serializing_if = "Option::is_none")]
pub error: Option<ApiError>,
}
impl Response {
/// Decode `data` into `T`.
pub fn decode<T: serde::de::DeserializeOwned>(&self) -> Result<T, serde_json::Error> {
serde_json::from_value(self.data.clone())
}
}
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct ApiError {
#[serde(default)]
pub code: String,
#[serde(default)]
pub message: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub details: Option<Value>,
/// Server-side reason (funcspec / restheadspec).
#[serde(skip_serializing_if = "Option::is_none")]
pub detail: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub sql: Option<String>,
}
+164
View File
@@ -0,0 +1,164 @@
use resolvespec::*;
use serde_json::json;
use wiremock::matchers::{header, method, path, query_param};
use wiremock::{Mock, MockServer, ResponseTemplate};
#[tokio::test]
async fn read_posts_body_with_headers() {
let srv = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/public/users"))
.and(header("authorization", "Bearer tok"))
.and(header("x-tenant", "a"))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({"success": true, "data": [{"id": 1}]})))
.expect(1)
.mount(&srv)
.await;
let c = ResolveSpecClient::from_builder(ClientBuilder::new(&format!("{}/", srv.uri())).token("tok").header("X-Tenant", "a")).unwrap();
let opts = Options { limit: Some(5), filters: vec![FilterOption::new("a", "eq", 1)], ..Default::default() };
let r = c.read("public", "users", None, Some(&opts)).await.unwrap();
let rows: Vec<serde_json::Value> = r.decode().unwrap();
assert_eq!(rows.len(), 1);
let body: serde_json::Value = serde_json::from_slice(&srv.received_requests().await.unwrap()[0].body).unwrap();
assert_eq!(body["operation"], "read");
assert_eq!(body["options"]["limit"], 5);
assert!(body.get("id").is_none());
}
#[tokio::test]
async fn id_placement() {
let srv = MockServer::start().await;
Mock::given(method("POST")).respond_with(ResponseTemplate::new(200).set_body_json(json!({"success": true, "data": {}}))).mount(&srv).await;
let c = ResolveSpecClient::new(&srv.uri()).unwrap();
c.read("s", "e", Some(7.into()), None).await.unwrap();
c.update("s", "e", json!({"a": 1}), Some(vec!["1".to_string(), "2".to_string()].into()), None).await.unwrap();
c.delete("s", "e", "a/b").await.unwrap();
let reqs = srv.received_requests().await.unwrap();
assert_eq!(reqs[0].url.path(), "/s/e/7");
assert_eq!(reqs[1].url.path(), "/s/e");
let b: serde_json::Value = serde_json::from_slice(&reqs[1].body).unwrap();
assert_eq!(b["id"], json!(["1", "2"]));
assert_eq!(b["operation"], "update");
assert_eq!(reqs[2].url.path(), "/s/e/a%2Fb");
}
#[tokio::test]
async fn errors() {
let srv = MockServer::start().await;
Mock::given(path("/s/a")).respond_with(ResponseTemplate::new(400).set_body_json(json!({"success": false, "error": {"code": "x", "message": "bad", "detail": "why"}}))).mount(&srv).await;
Mock::given(path("/s/b")).respond_with(ResponseTemplate::new(502).set_body_string("bad gateway")).mount(&srv).await;
Mock::given(path("/s/c")).respond_with(ResponseTemplate::new(200).set_body_json(json!({"success": false, "error": {"code": "c", "message": "nope"}}))).mount(&srv).await;
let c = ResolveSpecClient::new(&srv.uri()).unwrap();
match c.read("s", "a", None, None).await.unwrap_err() {
Error::Api { status, message, error } => {
assert_eq!((status, message.as_str(), error.code.as_str(), error.detail.as_deref()), (400, "bad", "x", Some("why")))
}
e => panic!("{e:?}"),
}
match c.read("s", "b", None, None).await.unwrap_err() {
Error::Api { status, message, .. } => assert_eq!((status, message.as_str()), (502, "bad gateway")),
e => panic!("{e:?}"),
}
assert_eq!(c.read("s", "c", None, None).await.unwrap_err().to_string(), "nope");
}
#[test]
fn headers_filters() {
let o = FuncSpecOptions {
filters: vec![
FilterOption::new("status", "eq", "active"),
FilterOption::new("age", "gte", 18),
FilterOption::new("name", "contains", "x").or(),
FilterOption::new("deleted", "is_null", serde_json::Value::Null),
FilterOption::new("id", "in", json!([1, 2])),
FilterOption::new("p", "between_inclusive", json!([1, 5])),
],
..Default::default()
};
let h = build_headers(&o);
let want: std::collections::BTreeMap<String, String> = [
("X-FieldFilter-status", "active"),
("X-SearchOp-greaterthanorequal-age", "18"),
("X-SearchOr-contains-name", "x"),
("X-SearchOp-empty-deleted", ""),
("X-SearchOp-in-id", "1,2"),
("X-SearchOp-betweeninclusive-p", "1,5"),
]
.into_iter()
.map(|(k, v)| (k.to_string(), v.to_string()))
.collect();
assert_eq!(h, want);
}
#[test]
fn headers_misc_and_encoding() {
let o = FuncSpecOptions {
search_filters: [("name".to_string(), "bob".to_string())].into(),
custom_sql_where: Some("a = 1".into()),
custom_sql_or: Some("b = 2".into()),
sort: vec![SortOption::new("name", "asc"), SortOption::new("created_at", "DESC")],
limit: Some(5),
offset: Some(10),
distinct: Some(true),
skip_count: Some(true),
skip_cache: Some(false),
response_format: Some("syncfusion".into()),
..Default::default()
};
let h = build_headers(&o);
assert_eq!(h["X-Sort"], "name ASC,created_at DESC");
assert_eq!(h["X-SearchFilter-name"], "bob");
assert_eq!(h["X-Custom-SQL-W"], "a = 1");
assert_eq!(h["X-Limit"], "5");
assert_eq!(h["X-SkipCache"], "false");
assert_eq!(h["X-Syncfusion"], "true");
let o = FuncSpecOptions { filters: vec![FilterOption::new("n", "eq", "héllo"), FilterOption::new("m", "eq", " pad")], ..Default::default() };
let h = build_headers(&o);
assert!(h["X-FieldFilter-n"].starts_with("ZIP_"));
assert_eq!(decode_header_value(&h["X-FieldFilter-n"]), "héllo");
assert_eq!(decode_header_value(&h["X-FieldFilter-m"]), " pad");
}
#[test]
fn query_building() {
let mut p = Params::new();
p.insert("a".into(), true.into());
p.insert("b".into(), vec!["x".to_string(), "y".to_string()].into());
p.insert("d".into(), 3i64.into());
assert_eq!(build_query(&p), vec![("a".into(), "true".into()), ("b".into(), "x".into()), ("b".into(), "y".into()), ("d".into(), "3".into())]);
}
#[tokio::test]
async fn query_list_metadata() {
let srv = MockServer::start().await;
Mock::given(method("GET"))
.and(path("/api/users"))
.and(query_param("org", "1"))
.and(header("x-limit", "2"))
.respond_with(ResponseTemplate::new(206).insert_header("Content-Range", "items 10-12/50").set_body_json(json!([{"id": 1}, {"id": 2}])))
.expect(1)
.mount(&srv)
.await;
let c = FuncSpecClient::from_builder(ClientBuilder::new(&srv.uri()).token("tok")).unwrap();
let mut p = Params::new();
p.insert("org".into(), 1i64.into());
let r = c.query_list("/api/users", &p, Some(&FuncSpecOptions { limit: Some(2), ..Default::default() })).await.unwrap();
assert_eq!(r.metadata.unwrap(), Metadata { total: 50, count: 2, filtered: 50, limit: 2, offset: 10 });
assert_eq!(r.data.as_array().unwrap().len(), 2);
}
#[tokio::test]
async fn query_single_and_error() {
let srv = MockServer::start().await;
Mock::given(path("/api/ok")).respond_with(ResponseTemplate::new(200).set_body_json(json!({"id": 1}))).mount(&srv).await;
Mock::given(path("/api/bad")).respond_with(ResponseTemplate::new(400).set_body_json(json!({"success": false, "error": {"code": "hook_error", "message": "Hook execution failed", "detail": "authentication required"}}))).mount(&srv).await;
let c = FuncSpecClient::new(&srv.uri()).unwrap();
let r = c.query("api/ok", &Params::new(), None).await.unwrap();
assert!(r.metadata.is_none());
assert_eq!(r.data["id"], 1);
match c.query("api/bad", &Params::new(), None).await.unwrap_err() {
Error::Api { error, .. } => assert_eq!((error.code.as_str(), error.detail.as_deref()), ("hook_error", Some("authentication required"))),
e => panic!("{e:?}"),
}
}
+12 -1
View File
@@ -9,7 +9,9 @@ import (
"github.com/bitechdev/ResolveSpec/pkg/config"
"github.com/bitechdev/ResolveSpec/pkg/dbmanager"
"github.com/bitechdev/ResolveSpec/pkg/dbtrace"
"github.com/bitechdev/ResolveSpec/pkg/logger"
"github.com/bitechdev/ResolveSpec/pkg/middleware"
"github.com/bitechdev/ResolveSpec/pkg/modelregistry"
"github.com/bitechdev/ResolveSpec/pkg/server"
"github.com/bitechdev/ResolveSpec/pkg/testmodels"
@@ -67,8 +69,17 @@ func main() {
handler.RegisterModel("public", modelNames[i], model)
}
// Queue requests per client (X-Client-Id, Authorization, session, then IP)
// so a burst such as a page load cannot flood the database pool.
queue := middleware.NewClientQueue(middleware.ClientQueueConfig{MaxConcurrent: 10})
defer queue.Close()
// DB usage logging (off unless db_trace.enabled / RESOLVESPEC_DB_TRACE_ENABLED).
// Inside the queue so queue wait is not counted in request duration.
dbtrace.Configure(dbtrace.FromConfig(cfg.DBTrace))
// Setup routes using new SetupMuxRoutes function (without authentication)
resolvespec.SetupMuxRoutes(r, handler, nil)
resolvespec.SetupMuxRoutes(r, handler, middleware.Chain(queue.Middleware, dbtrace.Middleware))
// Create server manager
mgr := server.NewManager()
+20 -9
View File
@@ -6,22 +6,33 @@ services:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: postgres
ports:
- "5434:5432"
# Host networking (bridge networks are unavailable in some environments):
# postgres listens directly on host port 8124.
network_mode: host
command: ["postgres", "-p", "8124"]
volumes:
- postgres-test-data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
test: ["CMD-SHELL", "pg_isready -U postgres -p 8124"]
interval: 5s
timeout: 5s
retries: 5
networks:
- resolvespec-test
testserver:
build:
context: .
dockerfile: docker/Dockerfile.testserver
container_name: resolvespec-testserver
environment:
RESOLVESPEC_DB_TRACE_ENABLED: "true"
RESOLVESPEC_DB_TRACE_MIN_CALLS: "1"
RESOLVESPEC_DB_TRACE_POOL_LOG: "true"
# Serves on host port 8123 (docker/testserver.config.yaml).
network_mode: host
depends_on:
postgres-test:
condition: service_healthy
volumes:
postgres-test-data:
driver: local
networks:
resolvespec-test:
driver: bridge
+13
View File
@@ -0,0 +1,13 @@
FROM golang:1.25-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /out/testserver ./cmd/testserver
FROM alpine:3.20
RUN apk add --no-cache ca-certificates
COPY --from=build /out/testserver /usr/local/bin/testserver
COPY docker/testserver.config.yaml /etc/resolvespec/config.yaml
EXPOSE 8123
ENTRYPOINT ["testserver"]
+95
View File
@@ -0,0 +1,95 @@
# ResolveSpec Test Server Configuration (docker compose, PostgreSQL)
# This is a minimal configuration for the test server
servers:
default_server: "main"
shutdown_timeout: 30s
drain_timeout: 25s
read_timeout: 10s
write_timeout: 10s
idle_timeout: 120s
instances:
main:
name: "main"
host: "0.0.0.0"
port: 8123
description: "Main server instance"
gzip: true
tags:
env: "test"
logger:
dev: true
path: ""
cache:
provider: "memory"
middleware:
rate_limit_rps: 100.0
rate_limit_burst: 200
max_request_size: 10485760
cors:
allowed_origins:
- "*"
allowed_methods:
- "GET"
- "POST"
- "PUT"
- "DELETE"
- "OPTIONS"
allowed_headers:
- "*"
max_age: 3600
tracing:
enabled: false
service_name: "resolvespec"
service_version: "1.0.0"
endpoint: ""
error_tracking:
enabled: false
provider: "noop"
environment: "development"
sample_rate: 1.0
traces_sample_rate: 0.1
event_broker:
enabled: false
provider: "memory"
mode: "sync"
worker_count: 1
buffer_size: 100
instance_id: ""
dbmanager:
default_connection: "default"
max_open_conns: 25
max_idle_conns: 5
conn_max_lifetime: 30m
conn_max_idle_time: 5m
retry_attempts: 3
retry_delay: 1s
health_check_interval: 30s
enable_auto_reconnect: true
connections:
# "default" overrides the built-in default connection (all connections are connected at start)
default:
name: "default"
type: "postgres"
host: "localhost"
port: 8124
user: "postgres"
password: "postgres"
database: "postgres"
sslmode: "disable"
application_name: "resolvespec-testserver"
default_orm: "gorm"
enable_logging: true
enable_metrics: false
connect_timeout: 10s
query_timeout: 30s
paths: {}
+1 -1
View File
@@ -148,7 +148,7 @@ require (
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/mod v0.38.0 // indirect
golang.org/x/net v0.58.0 // indirect
golang.org/x/sync v0.22.0 // indirect
golang.org/x/sync v0.22.0
golang.org/x/text v0.41.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect
+47
View File
@@ -0,0 +1,47 @@
# Request Transactions (cheatsheet)
Every DB statement and every DB-touching hook of one request runs on one transaction. Hooks never get the pool.
## Rules
- `hookCtx.Tx` is always the open transaction (never `h.db`), except `BeforeHandle`, which runs before any tx and must not touch the DB.
- `OnTxBegin` fires once, first, in every tx the handler opens (incl. the second short tx).
- `OnTxBegin` error or abort: rollback, client gets a generic error, nothing leaked.
- Begin or commit failure: generic error (`transaction_error` / "Transaction failed" in websocketspec, mqttspec, funcspec).
- Transaction-local state (`set_config(..., true)`, RLS GUCs) is only visible on that tx. Set it in `OnTxBegin`.
## Transactions per operation
| Operation | Tx 1 | Tx 2 (short, after commit) |
|---|---|---|
| read | `BeforeRead`, count, scan, `AfterRead`* | restheadspec: `AfterRead` |
| create | `Before*`, insert | re-fetch, `BeforeScan`, `AfterCreate` |
| update | `Before*`, select, update, `AfterUpdate`† | re-fetch (+ `AfterUpdate` where noted) |
| delete (single/batch) | `BeforeDelete`, select, delete, `AfterDelete` | none |
| funcspec query | `BeforeQuery*`, `BeforeSQLExec`, SQL, `After*` | `BeforeResponse` |
\* resolvespec, websocketspec, mqttspec, resolvemcp. restheadspec runs `AfterRead` in tx 2.
† restheadspec, websocketspec, mqttspec run `AfterUpdate` in tx 2. resolvespec, resolvemcp run it in tx 1.
resolvespec has no tx 2 for create: `AfterCreate` runs in tx 1, once per record (per item in a batch), after the insert and re-fetch. Its `AfterRead` gets the scanned slice (single and list reads) and may mask in place; a failing `AfterRead` fails the read (fail closed).
Tx 2 exists so the re-fetch sees trigger changes from the committed write.
## Per spec
| Spec | `OnTxBegin` | Helper |
|---|---|---|
| resolvespec, restheadspec, websocketspec, resolvemcp, funcspec | own `HookType` = `common.TxHookName` | `Handler.runInTx` |
| mqttspec | re-exports `websocketspec.OnTxBegin` | `Handler.runInTx` |
- `common.RunRequestTx(ctx, db, TxContext, onBegin, body)`: open tx, `SetTx`, `onBegin`, `body`.
- `common.TxContext`: `SetTx(tx)`; implemented by each spec's `HookContext`.
## RLS / transaction settings (pkg/security)
- `SecurityList.SetTxSettings(fn)`: `fn(SecurityContext) (map[string]string, error)`; nil disables.
- Every spec's `RegisterSecurityHooks` registers `OnTxBegin` → `security.StampTxSettings`. `fn` is read per call, so set order does not matter.
- Stamps via `set_config(name, value, true)` in name order, before any other SQL.
- Fail closed: `fn` error, invalid name, or non-Postgres driver with a non-empty map aborts the tx.
- Name: dotted identifier (`ns.name`). Value is hex-encoded in SQL, never inlined.
- Low level: `security.ApplyTxSettings(secCtx, tx, map)`.
## Test notes
- sqlmock + `SetMaxOpenConns(1)`: any pool use inside an open tx blocks and fails.
- restheadspec model-based updates/reads need the bun adapter.
+35 -12
View File
@@ -12,6 +12,7 @@ import (
"github.com/uptrace/bun"
"github.com/bitechdev/ResolveSpec/pkg/common"
"github.com/bitechdev/ResolveSpec/pkg/dbtrace"
"github.com/bitechdev/ResolveSpec/pkg/logger"
"github.com/bitechdev/ResolveSpec/pkg/modelregistry"
"github.com/bitechdev/ResolveSpec/pkg/reflection"
@@ -201,7 +202,7 @@ func (b *BunAdapter) Exec(ctx context.Context, query string, args ...interface{}
err = run()
}
}
recordQueryMetrics(b.metricsEnabled, operation, schema, entity, table, startedAt, err)
recordQueryMetrics(ctx, b.metricsEnabled, operation, schema, entity, table, startedAt, err)
return &BunResult{result: result}, err
}
@@ -219,7 +220,7 @@ func (b *BunAdapter) Query(ctx context.Context, dest interface{}, query string,
err = b.getDB().NewRaw(query, args...).Scan(ctx, dest)
}
}
recordQueryMetrics(b.metricsEnabled, operation, schema, entity, table, startedAt, err)
recordQueryMetrics(ctx, b.metricsEnabled, operation, schema, entity, table, startedAt, err)
return err
}
@@ -254,6 +255,7 @@ func (b *BunAdapter) RunInTransaction(ctx context.Context, fn func(common.Databa
err = logger.HandlePanic("BunAdapter.RunInTransaction", r)
}
}()
defer dbtrace.TxBegin(ctx)()
run := func() error {
return b.getDB().RunInTx(ctx, &sql.TxOptions{}, func(ctx context.Context, tx bun.Tx) error {
adapter := &BunTxAdapter{tx: tx, driverName: b.driverName, metricsEnabled: b.metricsEnabled}
@@ -273,6 +275,14 @@ func (b *BunAdapter) GetUnderlyingDB() interface{} {
return b.getDB()
}
// SQLDB implements common.SQLDBProvider.
func (b *BunAdapter) SQLDB() *sql.DB {
if db := b.getDB(); db != nil {
return db.DB
}
return nil
}
func (b *BunAdapter) DriverName() string {
// Normalize Bun's dialect name to match the project's canonical vocabulary.
// Bun returns "pg" for PostgreSQL; the rest of the project uses "postgres".
@@ -527,6 +537,19 @@ func (b *BunSelectQuery) WhereOr(query string, args ...interface{}) common.Selec
return b
}
// WhereGroup wraps the conditions added by fn in one parenthesised group ANDed with the rest.
func (b *BunSelectQuery) WhereGroup(fn func(common.SelectQuery) common.SelectQuery) common.SelectQuery {
b.query = b.query.WhereGroup(" AND ", func(q *bun.SelectQuery) *bun.SelectQuery {
inner := *b
inner.query = q
if res, ok := fn(&inner).(*BunSelectQuery); ok {
return res.query
}
return q
})
return b
}
func (b *BunSelectQuery) Join(query string, args ...interface{}) common.SelectQuery {
// Extract optional prefix from args
// If the last arg is a string that looks like a table prefix, use it
@@ -1301,7 +1324,7 @@ func (b *BunSelectQuery) Scan(ctx context.Context, dest interface{}) (err error)
if r := recover(); r != nil {
err = logger.HandlePanic("BunSelectQuery.Scan", r)
}
recordQueryMetrics(b.metricsEnabled, "SELECT", b.schema, b.entity, b.tableName, startedAt, err)
recordQueryMetrics(ctx, b.metricsEnabled, "SELECT", b.schema, b.entity, b.tableName, startedAt, err)
}()
if dest == nil {
err = fmt.Errorf("destination cannot be nil")
@@ -1347,7 +1370,7 @@ func (b *BunSelectQuery) ScanModel(ctx context.Context) (err error) {
logger.Error("Panic in BunSelectQuery.ScanModel: %v. %s. SQL: %s", r, modelInfo, sqlStr)
err = logger.HandlePanic("BunSelectQuery.ScanModel", r)
}
recordQueryMetrics(b.metricsEnabled, "SELECT", b.schema, b.entity, b.tableName, startedAt, err)
recordQueryMetrics(ctx, b.metricsEnabled, "SELECT", b.schema, b.entity, b.tableName, startedAt, err)
}()
if b.query.GetModel() == nil {
err = fmt.Errorf("model is nil")
@@ -1391,7 +1414,7 @@ func (b *BunSelectQuery) Count(ctx context.Context) (count int, err error) {
err = logger.HandlePanic("BunSelectQuery.Count", r)
count = 0
}
recordQueryMetrics(b.metricsEnabled, "COUNT", b.schema, b.entity, b.tableName, startedAt, err)
recordQueryMetrics(ctx, b.metricsEnabled, "COUNT", b.schema, b.entity, b.tableName, startedAt, err)
}()
// If Model() was set, use bun's native Count() which works properly
if b.hasModel {
@@ -1425,7 +1448,7 @@ func (b *BunSelectQuery) Exists(ctx context.Context) (exists bool, err error) {
err = logger.HandlePanic("BunSelectQuery.Exists", r)
exists = false
}
recordQueryMetrics(b.metricsEnabled, "EXISTS", b.schema, b.entity, b.tableName, startedAt, err)
recordQueryMetrics(ctx, b.metricsEnabled, "EXISTS", b.schema, b.entity, b.tableName, startedAt, err)
}()
exists, err = b.query.Exists(ctx)
if err != nil {
@@ -1512,7 +1535,7 @@ func (b *BunInsertQuery) Exec(ctx context.Context) (res common.Result, err error
startedAt := time.Now()
b.prepareValues()
result, err := b.query.Exec(ctx)
recordQueryMetrics(b.metricsEnabled, "INSERT", b.schema, b.entity, b.tableName, startedAt, err)
recordQueryMetrics(ctx, b.metricsEnabled, "INSERT", b.schema, b.entity, b.tableName, startedAt, err)
return &BunResult{result: result}, err
}
@@ -1525,7 +1548,7 @@ func (b *BunInsertQuery) Scan(ctx context.Context, dest interface{}) (err error)
startedAt := time.Now()
b.prepareValues()
err = b.query.Scan(ctx, dest)
recordQueryMetrics(b.metricsEnabled, "INSERT", b.schema, b.entity, b.tableName, startedAt, err)
recordQueryMetrics(ctx, b.metricsEnabled, "INSERT", b.schema, b.entity, b.tableName, startedAt, err)
return err
}
@@ -1622,7 +1645,7 @@ func (b *BunUpdateQuery) Exec(ctx context.Context) (res common.Result, err error
logger.Error("BunUpdateQuery.Exec failed. SQL: %s. Error: %v", sqlStr, err)
err = common.WrapSQLError(err, sqlStr)
}
recordQueryMetrics(b.metricsEnabled, "UPDATE", b.schema, b.entity, b.tableName, startedAt, err)
recordQueryMetrics(ctx, b.metricsEnabled, "UPDATE", b.schema, b.entity, b.tableName, startedAt, err)
return &BunResult{result: result}, err
}
@@ -1674,7 +1697,7 @@ func (b *BunDeleteQuery) Exec(ctx context.Context) (res common.Result, err error
logger.Error("BunDeleteQuery.Exec failed. SQL: %s. Error: %v", sqlStr, err)
err = common.WrapSQLError(err, sqlStr)
}
recordQueryMetrics(b.metricsEnabled, "DELETE", b.schema, b.entity, b.tableName, startedAt, err)
recordQueryMetrics(ctx, b.metricsEnabled, "DELETE", b.schema, b.entity, b.tableName, startedAt, err)
return &BunResult{result: result}, err
}
@@ -1730,7 +1753,7 @@ func (b *BunTxAdapter) Exec(ctx context.Context, query string, args ...interface
startedAt := time.Now()
operation, schema, entity, table := metricTargetFromRawQuery(query, b.driverName)
result, err := b.tx.ExecContext(ctx, query, args...)
recordQueryMetrics(b.metricsEnabled, operation, schema, entity, table, startedAt, err)
recordQueryMetrics(ctx, b.metricsEnabled, operation, schema, entity, table, startedAt, err)
return &BunResult{result: result}, err
}
@@ -1738,7 +1761,7 @@ func (b *BunTxAdapter) Query(ctx context.Context, dest interface{}, query string
startedAt := time.Now()
operation, schema, entity, table := metricTargetFromRawQuery(query, b.driverName)
err := b.tx.NewRaw(query, args...).Scan(ctx, dest)
recordQueryMetrics(b.metricsEnabled, operation, schema, entity, table, startedAt, err)
recordQueryMetrics(ctx, b.metricsEnabled, operation, schema, entity, table, startedAt, err)
return err
}
+37 -10
View File
@@ -2,6 +2,7 @@ package database
import (
"context"
"database/sql"
"fmt"
"reflect"
"strings"
@@ -12,6 +13,7 @@ import (
"gorm.io/gorm/clause"
"github.com/bitechdev/ResolveSpec/pkg/common"
"github.com/bitechdev/ResolveSpec/pkg/dbtrace"
"github.com/bitechdev/ResolveSpec/pkg/logger"
"github.com/bitechdev/ResolveSpec/pkg/modelregistry"
"github.com/bitechdev/ResolveSpec/pkg/reflection"
@@ -151,7 +153,7 @@ func (g *GormAdapter) Exec(ctx context.Context, query string, args ...interface{
result = run()
}
}
recordQueryMetrics(g.metricsEnabled, operation, schema, entity, table, startedAt, result.Error)
recordQueryMetrics(ctx, g.metricsEnabled, operation, schema, entity, table, startedAt, result.Error)
return &GormResult{result: result}, result.Error
}
@@ -172,7 +174,7 @@ func (g *GormAdapter) Query(ctx context.Context, dest interface{}, query string,
err = run()
}
}
recordQueryMetrics(g.metricsEnabled, operation, schema, entity, table, startedAt, err)
recordQueryMetrics(ctx, g.metricsEnabled, operation, schema, entity, table, startedAt, err)
return err
}
@@ -206,6 +208,7 @@ func (g *GormAdapter) RunInTransaction(ctx context.Context, fn func(common.Datab
err = logger.HandlePanic("GormAdapter.RunInTransaction", r)
}
}()
defer dbtrace.TxBegin(ctx)()
run := func() error {
return g.getDB().WithContext(ctx).Transaction(func(tx *gorm.DB) error {
adapter := &GormAdapter{db: tx, dbFactory: g.dbFactory, driverName: g.driverName, metricsEnabled: g.metricsEnabled}
@@ -225,6 +228,20 @@ func (g *GormAdapter) GetUnderlyingDB() interface{} {
return g.getDB()
}
// SQLDB implements common.SQLDBProvider. It returns nil when GORM has no *sql.DB
// (for example a ConnPool that is not database/sql).
func (g *GormAdapter) SQLDB() *sql.DB {
db := g.getDB()
if db == nil {
return nil
}
sqlDB, err := db.DB()
if err != nil {
return nil
}
return sqlDB
}
func (g *GormAdapter) DriverName() string {
return normalizeGormDriverName(g.getDB())
}
@@ -360,6 +377,16 @@ func (g *GormSelectQuery) WhereOr(query string, args ...interface{}) common.Sele
return g
}
// WhereGroup wraps the conditions added by fn in one parenthesised group ANDed with the rest.
func (g *GormSelectQuery) WhereGroup(fn func(common.SelectQuery) common.SelectQuery) common.SelectQuery {
inner := *g
inner.db = g.db.Session(&gorm.Session{NewDB: true})
if res, ok := fn(&inner).(*GormSelectQuery); ok {
g.db = g.db.Where(res.db)
}
return g
}
func (g *GormSelectQuery) Join(query string, args ...interface{}) common.SelectQuery {
// Extract optional prefix from args
// If the last arg is a string that looks like a table prefix, use it
@@ -585,7 +612,7 @@ func (g *GormSelectQuery) Scan(ctx context.Context, dest interface{}) (err error
logger.Error("GormSelectQuery.Scan failed. SQL: %s. Error: %v", sqlStr, err)
err = common.WrapSQLError(err, sqlStr)
}
recordQueryMetrics(g.metricsEnabled, "SELECT", g.schema, g.entity, g.tableName, startedAt, err)
recordQueryMetrics(ctx, g.metricsEnabled, "SELECT", g.schema, g.entity, g.tableName, startedAt, err)
return err
}
@@ -616,7 +643,7 @@ func (g *GormSelectQuery) ScanModel(ctx context.Context) (err error) {
logger.Error("GormSelectQuery.ScanModel failed. SQL: %s. Error: %v", sqlStr, err)
err = common.WrapSQLError(err, sqlStr)
}
recordQueryMetrics(g.metricsEnabled, "SELECT", g.schema, g.entity, g.tableName, startedAt, err)
recordQueryMetrics(ctx, g.metricsEnabled, "SELECT", g.schema, g.entity, g.tableName, startedAt, err)
return err
}
@@ -646,7 +673,7 @@ func (g *GormSelectQuery) Count(ctx context.Context) (count int, err error) {
logger.Error("GormSelectQuery.Count failed. SQL: %s. Error: %v", sqlStr, err)
err = common.WrapSQLError(err, sqlStr)
}
recordQueryMetrics(g.metricsEnabled, "COUNT", g.schema, g.entity, g.tableName, startedAt, err)
recordQueryMetrics(ctx, g.metricsEnabled, "COUNT", g.schema, g.entity, g.tableName, startedAt, err)
return int(count64), err
}
@@ -676,7 +703,7 @@ func (g *GormSelectQuery) Exists(ctx context.Context) (exists bool, err error) {
logger.Error("GormSelectQuery.Exists failed. SQL: %s. Error: %v", sqlStr, err)
err = common.WrapSQLError(err, sqlStr)
}
recordQueryMetrics(g.metricsEnabled, "EXISTS", g.schema, g.entity, g.tableName, startedAt, err)
recordQueryMetrics(ctx, g.metricsEnabled, "EXISTS", g.schema, g.entity, g.tableName, startedAt, err)
return count > 0, err
}
@@ -752,7 +779,7 @@ func (g *GormInsertQuery) Exec(ctx context.Context) (res common.Result, err erro
result = run()
}
}
recordQueryMetrics(g.metricsEnabled, "INSERT", g.schema, g.entity, g.tableName, startedAt, result.Error)
recordQueryMetrics(ctx, g.metricsEnabled, "INSERT", g.schema, g.entity, g.tableName, startedAt, result.Error)
return &GormResult{result: result}, result.Error
}
@@ -790,7 +817,7 @@ func (g *GormInsertQuery) Scan(ctx context.Context, dest interface{}) (err error
}
}
recordQueryMetrics(g.metricsEnabled, "INSERT", g.schema, g.entity, g.tableName, startedAt, result.Error)
recordQueryMetrics(ctx, g.metricsEnabled, "INSERT", g.schema, g.entity, g.tableName, startedAt, result.Error)
if result.Error != nil {
return result.Error
}
@@ -937,7 +964,7 @@ func (g *GormUpdateQuery) Exec(ctx context.Context) (res common.Result, err erro
logger.Error("GormUpdateQuery.Exec failed. SQL: %s. Error: %v", sqlStr, result.Error)
return &GormResult{result: result}, common.WrapSQLError(result.Error, sqlStr)
}
recordQueryMetrics(g.metricsEnabled, "UPDATE", g.schema, g.entity, g.tableName, startedAt, result.Error)
recordQueryMetrics(ctx, g.metricsEnabled, "UPDATE", g.schema, g.entity, g.tableName, startedAt, result.Error)
return &GormResult{result: result}, result.Error
}
@@ -999,7 +1026,7 @@ func (g *GormDeleteQuery) Exec(ctx context.Context) (res common.Result, err erro
logger.Error("GormDeleteQuery.Exec failed. SQL: %s. Error: %v", sqlStr, result.Error)
return &GormResult{result: result}, common.WrapSQLError(result.Error, sqlStr)
}
recordQueryMetrics(g.metricsEnabled, "DELETE", g.schema, g.entity, g.tableName, startedAt, result.Error)
recordQueryMetrics(ctx, g.metricsEnabled, "DELETE", g.schema, g.entity, g.tableName, startedAt, result.Error)
return &GormResult{result: result}, result.Error
}
@@ -0,0 +1,230 @@
package database
import (
"bytes"
"context"
"database/sql"
"fmt"
"net"
"os"
"os/exec"
"strings"
"testing"
"time"
_ "github.com/jackc/pgx/v5/stdlib"
"github.com/uptrace/bun"
"github.com/uptrace/bun/dialect/pgdialect"
"github.com/bitechdev/ResolveSpec/pkg/common"
"github.com/bitechdev/ResolveSpec/pkg/config"
)
// These tests start a throwaway PostgreSQL server with podman or docker (whichever is
// installed, podman first) and run the client-SQL hardening against a real database. They
// pull an image, so they only run when RESOLVESPEC_TEST_CONTAINERS=1 and not with -short.
// The container is removed when the test ends.
const hardeningPGPassword = "Resolve_Spec_1"
type hardeningItem struct {
bun.BaseModel `bun:"table:items,alias:items"`
ID int `bun:"id"`
Tenant int `bun:"tenant"`
Name string `bun:"name"`
}
func hardeningRuntime(t *testing.T) string {
t.Helper()
if testing.Short() {
t.Skip("container tests are skipped with -short")
}
if os.Getenv("RESOLVESPEC_TEST_CONTAINERS") != "1" {
t.Skip("set RESOLVESPEC_TEST_CONTAINERS=1 to run tests that start a podman/docker container")
}
for _, rt := range []string{"podman", "docker"} {
if p, err := exec.LookPath(rt); err == nil {
return p
}
}
t.Skip("neither podman nor docker found in PATH")
return ""
}
func hardeningRun(t *testing.T, timeout time.Duration, name string, args ...string) string {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
var out, errb bytes.Buffer
cmd := exec.CommandContext(ctx, name, args...)
cmd.Stdout, cmd.Stderr = &out, &errb
if err := cmd.Run(); err != nil {
t.Fatalf("%s %s: %v\n%s", name, strings.Join(args, " "), err, errb.String())
}
return strings.TrimSpace(out.String())
}
// startHardeningPostgres runs postgres on a random localhost port and returns a ready *sql.DB.
func startHardeningPostgres(t *testing.T, rt string) *sql.DB {
t.Helper()
id := hardeningRun(t, 10*time.Minute, rt, "run", "-d", "--rm", "-p", "127.0.0.1::5432",
"-e", "POSTGRES_PASSWORD="+hardeningPGPassword, "docker.io/library/postgres:16-alpine") // first run may pull
t.Cleanup(func() { _ = exec.Command(rt, "rm", "-f", id).Run() })
out := hardeningRun(t, 30*time.Second, rt, "port", id, "5432")
line := strings.Fields(out)[len(strings.Fields(out))-1]
for _, l := range strings.Split(out, "\n") {
if strings.Contains(l, "127.0.0.1:") {
line = l[strings.LastIndex(l, " ")+1:]
break
}
}
_, port, err := net.SplitHostPort(line)
if err != nil {
t.Fatalf("cannot parse published port %q: %v", out, err)
}
dsn := fmt.Sprintf("postgres://postgres:%s@127.0.0.1:%s/postgres?sslmode=disable", hardeningPGPassword, port)
// The official image restarts once during init: wait, pause, wait again.
wait := func(d time.Duration) *sql.DB {
deadline := time.Now().Add(d)
var last error
for time.Now().Before(deadline) {
db, err := sql.Open("pgx", dsn)
if err == nil {
if last = db.Ping(); last == nil {
return db
}
_ = db.Close()
} else {
last = err
}
time.Sleep(time.Second)
}
t.Fatalf("postgres not ready within %s: %v", d, last)
return nil
}
_ = wait(90 * time.Second).Close()
time.Sleep(2 * time.Second)
db := wait(60 * time.Second)
t.Cleanup(func() { _ = db.Close() })
return db
}
// setHardeningConfig overrides the global hardening switches for the test.
func setHardeningConfig(t *testing.T, h config.HardeningConfig) {
t.Helper()
m := config.GetConfigManager()
cfg, err := m.GetConfig()
if err != nil {
t.Fatal(err)
}
old := cfg.Hardening
cfg.Hardening = h
if err := m.SetConfig(cfg); err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
cfg.Hardening = old
_ = m.SetConfig(cfg)
})
}
// clientWhere mirrors the restheadspec handler pipeline for x-custom-sql-w.
func clientWhere(raw string) string {
w := common.AddTablePrefixToColumns(raw, "items")
w = common.SanitizeWhereClause(w, "items")
return common.EnsureOuterParentheses(w)
}
func TestHardeningAgainstPostgresContainer(t *testing.T) {
rt := hardeningRuntime(t)
sqldb := startHardeningPostgres(t, rt)
if _, err := sqldb.Exec(`
CREATE TABLE items (id int PRIMARY KEY, tenant int, name text);
INSERT INTO items VALUES (1,5,'mine-a'),(2,5,'mine-b'),(3,6,'other-a'),(4,6,'awaiting update approval');`); err != nil {
t.Fatal(err)
}
bdb := bun.NewDB(sqldb, pgdialect.New())
adapter := NewBunAdapter(bdb)
ctx := context.Background()
// list runs the handler-shaped query: client x-custom-sql-w, then the server tenant filter.
list := func(t *testing.T, where string) []hardeningItem {
t.Helper()
var rows []hardeningItem
q := adapter.NewSelect().Model(&rows)
if w := clientWhere(where); w != "" {
q = q.Where(w)
}
q = q.Where("items.tenant = ?", 5)
if err := q.Scan(ctx, &rows); err != nil {
t.Fatalf("query failed for %q: %v", where, err)
}
return rows
}
t.Run("strict", func(t *testing.T) {
setHardeningConfig(t, config.HardeningConfig{CORSStrictOrigins: true, SortStrict: true, SQLStrict: true})
t.Run("legitimate filters keep working", func(t *testing.T) {
if got := list(t, "name = 'mine-a'"); len(got) != 1 || got[0].ID != 1 {
t.Errorf("simple filter: %v", got)
}
if got := list(t, "id in (select id from items where tenant = 5)"); len(got) != 2 {
t.Errorf("subquery filter: %v", got)
}
})
t.Run("parenthesis escape cannot leave the tenant", func(t *testing.T) {
if got := list(t, "1=1)) OR ((1=1"); len(got) != 0 {
t.Errorf("escape not rejected closed, got rows: %v", got)
}
})
t.Run("hostile fragments fail closed", func(t *testing.T) {
for _, w := range []string{
"id = 1 and pg_sleep(10) is not null",
"id = 1 or (select count(*) from pg_shadow) > 0",
"id = 1; delete/**/from items",
} {
start := time.Now()
if got := list(t, w); len(got) != 0 {
t.Errorf("%q returned rows: %v", w, got)
}
if time.Since(start) > 5*time.Second {
t.Errorf("%q was executed (took %s)", w, time.Since(start))
}
}
var n int
if err := sqldb.QueryRow("SELECT count(*) FROM items").Scan(&n); err != nil || n != 4 {
t.Errorf("items table modified: count=%d err=%v", n, err)
}
})
t.Run("x-custom-sql-or stays inside the tenant", func(t *testing.T) {
var rows []hardeningItem
q := adapter.NewSelect().Model(&rows)
orClause := common.EnsureOuterParentheses(common.SanitizeWhereClause("items.name = 'other-a'", "items"))
q = q.(common.WhereGrouper).WhereGroup(func(g common.SelectQuery) common.SelectQuery {
return g.Where("items.name = ?", "mine-a").WhereOr(orClause)
})
q = q.Where("items.tenant = ?", 5)
if err := q.Scan(ctx, &rows); err != nil {
t.Fatal(err)
}
if len(rows) != 1 || rows[0].ID != 1 {
t.Errorf("OR clause leaked outside tenant filter: %v", rows)
}
})
})
t.Run("switch off restores legacy behaviour", func(t *testing.T) {
setHardeningConfig(t, config.HardeningConfig{})
// Proves the strict assertions above are meaningful: without hardening the same
// escape returns the other tenant's rows.
if got := list(t, "1=1)) OR ((1=1"); len(got) < 3 {
t.Errorf("expected the legacy escape to leak rows, got %v", got)
}
})
}

Some files were not shown because too many files have changed in this diff Show More