wdevs/pgsql-broker
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

feat(broker): migrations-based install, roles, RLS, and job dependency groups

Browse Source 
Replace the ad-hoc tables/procedures install layout with versioned,
ordered SQL migrations tracked in broker_schema_migrations. Add
optional least-privilege role provisioning (--with-roles), multi-tenant
row-level security, lease-based job claiming with stale-lease recovery,
and job dependencies -- both by job id and by fan-in job group. Add
Docker/Compose support for running the broker and its test suite.
This commit is contained in:
warkanum
2026-09-17 22:09:41 +02:00
parent 602997bcdb
commit 4c8e1066d4
53 changed files with 2911 additions and 1016 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+156 -38
View File
@@ -1,6 +1,6 @@
# PostgreSQL Broker
A robust, event-driven job processing system for PostgreSQL that uses LISTEN/NOTIFY for real-time job execution. It supports multiple queues, priority-based scheduling, and can be used both as a standalone service or as a Go library.
A robust, event-driven job processing system for PostgreSQL that uses LISTEN/NOTIFY for real-time job execution. It supports multiple queues, priority-based scheduling, multi-tenant row-level security, and can be used both as a standalone service or as a Go library.
## Features
@@ -8,20 +8,23 @@ A robust, event-driven job processing system for PostgreSQL that uses LISTEN/NOT
- **Event-Driven**: Uses PostgreSQL LISTEN/NOTIFY for instant job notifications
- **Multiple Queues**: Support for concurrent job processing across multiple queues per database
- **Priority Scheduling**: Jobs can be prioritized for execution order
- **Job Dependencies**: Jobs can depend on other jobs being completed first
- **Job Dependencies**: Jobs can depend on other jobs completing first (cycle-checked, gated in `broker_get`)
- **Multi-Tenant Row-Level Security**: Jobs are isolated per `tenant_id` via FORCE RLS policies (`broker.broker_set_tenant`)
- **Lease-Based Claiming**: Jobs are claimed with an expiring lease and recovered automatically if a worker dies mid-job
- **Least-Privilege Roles**: Optional `broker_admin` / `broker_runtime` / `broker_enqueue` role separation
- **Versioned Migrations**: Embedded, ordered SQL migrations with an applied-versions tracking table
- **Adapter Pattern**: Clean interfaces for database and logging (easy to extend)
- **Standalone or Library**: Use as a CLI tool or integrate into your Go application
- **Configuration Management**: Viper-based config with support for YAML, JSON, and environment variables
- **Graceful Shutdown**: Proper cleanup and job completion on shutdown
- **Instance Tracking**: Monitor active broker instances through the database
- **Single Instance Per Database**: Enforces one broker instance per database to prevent conflicts
- **Embedded SQL Installer**: Database schema embedded in binary with built-in install command
- **Docker Support**: Production `Dockerfile`/`docker-compose.yml` and a test-runner `Dockerfile.test`/`docker-compose.test.yml`
## Architecture
The broker supports multi-database architecture where a single broker process can manage multiple database connections. Each database has its own instance with dedicated queues, but only ONE broker instance is allowed per database.
```
```text
┌─────────────────────────────────────────────────────────────────┐
│ Broker Process │
├─────────────────────────────────────────────────────────────────┤
@@ -46,17 +49,20 @@ The broker supports multi-database architecture where a single broker process ca
┌──────────────────────┐ ┌──────────────────────┐
│ PostgreSQL (DB1) │ │ PostgreSQL (DB2) │
│ - broker_jobs │ │ - broker_jobs │
│ - broker_job_dependency│ │ - broker_job_dependency│
│ - broker_queueinstance│ │ - broker_queueinstance│
│ - broker_schedule │ │ - broker_schedule │
└──────────────────────┘ └──────────────────────┘
```
**Key Points**:
- One broker process can manage multiple databases
- Each database has exactly ONE active broker instance
- Each database instance has its own queues and workers
- Validation prevents multiple broker processes from connecting to the same database
- Different databases can have different queue counts
- Jobs are isolated by `tenant_id` via FORCE RLS; `broker_get` only claims jobs with no incomplete dependency
## Installation
@@ -76,27 +82,57 @@ The binary will be available in `bin/pgsql-broker`.
go get git.warky.dev/wdevs/pgsql-broker
```
### With Docker
See [Docker](#docker) below for the production `Dockerfile`/`docker-compose.yml` and the `Dockerfile.test`/`docker-compose.test.yml` test runner.
## Quick Start
### 1. Setup Database
Install the required tables and stored procedures:
Migrations are embedded in the binary and applied in order, tracked in `broker.broker_schema_migrations`.
```bash
# Using the CLI (recommended)
# Apply all pending migrations, then verify
./bin/pgsql-broker install --config broker.yaml
# Verify only, no changes
./bin/pgsql-broker install --verify-only --config broker.yaml
# Or with make
make sql-install
# Verify installation
./bin/pgsql-broker install --verify-only --config broker.yaml
# Or manually with psql:
psql -f pkg/broker/install/sql/tables/00_install.sql
psql -f pkg/broker/install/sql/procedures/00_install.sql
```
#### Optional: least-privilege roles
`install --with-roles` additionally creates/rotates three roles (idempotent — safe to re-run):
| Role | Purpose |
| ---- | ------- |
| `broker_admin` | Schema owner, `BYPASSRLS`, runs migrations |
| `broker_runtime` | The role the running broker connects as; table/sequence/function privileges only, no `BYPASSRLS` |
| `broker_enqueue` | Narrow role for enqueuing jobs only (`broker_add_job`, `broker_set_tenant`, insert on `broker_jobs`/`broker_job_dependency`) |
```bash
./bin/pgsql-broker install --with-roles \
--admin-user postgres \
--broker-admin-password ... \
--broker-runtime-password ... \
--broker-enqueue-password ...
```
Credential resolution order for each value: CLI flag → environment variable → interactive masked prompt (TTY only) → error.
| Value | Flag | Env fallback(s) |
| ----- | ---- | ---------------- |
| Admin user | `--admin-user` | `PGUSER`, `PG_USER` |
| Admin password | `--admin-password` | `PGPASSWORD`, `PG_PASS` |
| `broker_admin` password | `--broker-admin-password` | `BROKER_ADMIN_PASSWORD` |
| `broker_runtime` password | `--broker-runtime-password` | `BROKER_RUNTIME_PASSWORD` |
| `broker_enqueue` password | `--broker-enqueue-password` | `BROKER_ENQUEUE_PASSWORD` |
`--with-roles` always connects as the admin login (not the config file's own user) since `broker_admin` must own the schema; it cannot be combined with `--verify-only`.
### 2. Configure
Create a configuration file `broker.yaml`:
@@ -111,6 +147,7 @@ databases:
password: your_password
sslmode: disable
queue_count: 4
tenant_id: default # optional; sets broker.tenant_id for RLS
# Optional: add more databases
- name: db2
@@ -125,13 +162,15 @@ databases:
broker:
name: pgsql-broker
enable_debug: false
lease_seconds: 60 # job lease duration before it's reclaimable
stale_job_recovery_sec: 30 # how often expired leases are recovered
logging:
level: info
format: json
```
**Note**: Each database requires a unique `name` identifier and can have its own `queue_count` configuration.
**Note**: Each database requires a unique `name` identifier and can have its own `queue_count` configuration. See [broker.example.yaml](./broker.example.yaml) for the full field set.
### 3. Run the Broker
@@ -149,18 +188,47 @@ make run
### 4. Add a Job
```sql
SELECT broker_add_job(
'My Job', -- job_name
'SELECT do_something()', -- execute_str
1, -- job_queue (default: 1)
0, -- job_priority (default: 0)
'sql', -- job_language (default: 'sql')
NULL, -- run_as
NULL, -- user_login
NULL -- schedule_id
SELECT * FROM broker.broker_add_job(
'My Job', -- p_job_name
'SELECT do_something()', -- p_execute_str
1, -- p_job_queue (default: 1)
0, -- p_job_priority (default: 0)
'sql', -- p_job_language (default: 'sql')
NULL, -- p_run_as
NULL, -- p_schedule_id
ARRAY[123]::BIGINT[], -- p_depends_on_job_ids (NULL = no deps)
NULL, -- p_idempotency_key
1, -- p_max_attempts (default: 1)
NULL, -- p_job_group (default: p_job_name)
ARRAY['extract'] -- p_depends_on_groups (NULL = no deps)
);
```
Or, for the common case (name, execute string, priority, dependencies), use the shortcut:
```sql
SELECT * FROM broker.broker_add_job_simple('transform', 'SELECT do_something()', 5, ARRAY['extract']);
```
#### Dependencies: job groups vs. job ids
Every job belongs to a **job group**, defaulting to its own `job_name` when `p_job_group` isn't given. A dependency can target either a whole group (`p_depends_on_groups`) or a single job id (`p_depends_on_job_ids`):
- **Group dependency** (fan-in): the dependent job is only claimable once **every** job currently tagged with that group has `complete_status = 2`. Multiple jobs can share a group (e.g. several parallel `extract` jobs feeding one `transform`); already-completed group members simply drop out of the check.
- **Id dependency**: waits on one specific job row, as before.
Both `broker_add_job` and `broker_add_job_simple` reject a job depending on its own group/id, and a one-hop cycle (A depends on B, B already depends on A).
When calling as `broker_enqueue` (or any RLS-scoped role), set the tenant first and wrap in a single transaction so `SET LOCAL` persists across statements:
```sql
BEGIN;
SET ROLE broker_enqueue;
SELECT broker.broker_set_tenant('my_tenant');
SELECT * FROM broker.broker_add_job_simple('extract', 'SELECT 1');
COMMIT;
```
## Usage as a Library
```go
@@ -194,34 +262,65 @@ func main() {
See the [examples](./examples/) directory for complete examples.
## Docker
### Production: `Dockerfile` + `docker-compose.yml`
Multi-stage build (`golang:1.26-alpine` → `alpine:3.22`, static binary, non-root user). The compose stack runs Postgres, a one-shot `migrate` service (`install --with-roles`), then the `broker` service connecting as `broker_runtime`.
```bash
cp broker.docker.example.yaml broker.docker.yaml # set broker_runtime password
cp .env.example .env # set POSTGRES_PASSWORD + 3 BROKER_*_PASSWORD
docker-compose up --build -d
```
`broker.docker.yaml` and `.env` are gitignored — never commit real credentials. `BROKER_RUNTIME_PASSWORD` in `.env` must match the `password:` field in `broker.docker.yaml`.
### Test runner: `Dockerfile.test` + `docker-compose.test.yml`
Builds the module and runs the full Go integration suite (`tests/integration/...`) against a disposable Postgres container:
```bash
docker-compose -f docker-compose.test.yml up --build --abort-on-container-exit --exit-code-from tests
docker-compose -f docker-compose.test.yml down -v
```
## Database Schema
Schema objects live under `broker.*` and are applied via ordered, embedded migrations (`pkg/broker/install/sql/migrations/`), tracked in `broker.broker_schema_migrations`.
### Tables
- **broker_queueinstance**: Tracks active broker queue instances (one per database)
- **broker_jobs**: Job queue with status tracking
- **broker_jobs**: Job queue with status, lease, attempt, tenant, and `job_group` tracking
- **broker_job_dependency**: Dependency edges, each targeting either a `depends_on_job_id` or a `depends_on_group`, enforced at claim time
- **broker_schedule**: Scheduled jobs (cron-like functionality)
### Stored Procedures
- **broker_get**: Fetch the next job from a queue
- **broker_get**: Fetch and lease the next claimable job from a queue (skips jobs with an incomplete id- or group-dependency)
- **broker_run**: Execute a job
- **broker_set**: Set runtime options (user, application_name, etc.)
- **broker_add_job**: Add a new job to the queue
- **broker_register_instance**: Register a broker instance
- **broker_ping_instance**: Update instance heartbeat
- **broker_shutdown_instance**: Mark instance as shutdown
- **broker_set_tenant**: Set the `broker.tenant_id` GUC used by RLS policies
- **broker_add_job**: Add a new job to the queue, with an optional `job_group` and dependencies by id and/or group
- **broker_add_job_simple**: Shortcut for `broker_add_job` — name, execute string, priority, and dependencies by group name
- **broker_register_instance / broker_ping_instance / broker_shutdown_instance**: Instance lifecycle tracking
- **broker_recover_stale_jobs**: Reclaim jobs whose lease has expired
### Roles (optional, via `install --with-roles`)
See `pkg/broker/install/sql/roles/0001_roles.sql` — `broker_admin`, `broker_runtime`, `broker_enqueue` (see table above).
## Configuration Reference
See [broker.example.yaml](./broker.example.yaml) for a complete configuration example.
See [broker.example.yaml](./broker.example.yaml) for a complete configuration example, or [broker.docker.example.yaml](./broker.docker.example.yaml) for the Docker Compose variant.
### Database Settings
The `databases` array can contain multiple database configurations. Each entry supports:
| Setting | Description | Default |
|---------|-------------|---------|
| ------- | ----------- | ------- |
| `name` | Unique identifier for this database | **Required** |
| `host` | PostgreSQL host | `localhost` |
| `port` | PostgreSQL port | `5432` |
@@ -234,13 +333,14 @@ The `databases` array can contain multiple database configurations. Each entry s
| `conn_max_lifetime` | Connection max lifetime | `5m` |
| `conn_max_idle_time` | Connection max idle time | `10m` |
| `queue_count` | Number of queues for this database | `4` |
| `tenant_id` | Tenant id set for this connection (RLS) | `default` |
### Broker Settings
Global settings applied to all database instances:
| Setting | Description | Default |
|---------|-------------|---------|
| ------- | ----------- | ------- |
| `name` | Broker instance name | `pgsql-broker` |
| `fetch_query_que_size` | Jobs per fetch cycle | `100` |
| `queue_timer_sec` | Seconds between polls | `10` |
@@ -248,6 +348,8 @@ Global settings applied to all database instances:
| `worker_idle_timeout_sec` | Worker idle timeout | `10` |
| `notify_retry_seconds` | NOTIFY retry interval | `30s` |
| `enable_debug` | Enable debug logging | `false` |
| `lease_seconds` | Job lease duration before reclaimable | - |
| `stale_job_recovery_sec` | Interval for reclaiming expired leases | - |
## Development
@@ -262,9 +364,19 @@ make vet # Run go vet
make test # Run tests
```
### Testing
```bash
go test ./... # unit tests
go test -v ./tests/integration/... # integration tests (needs local Postgres, see below)
docker-compose -f docker-compose.test.yml up --build --abort-on-container-exit --exit-code-from tests
```
Integration tests expect Postgres reachable at `localhost:5433` (see `tests/integration/`), including `rls_test.go` (multi-tenant isolation) and `stage5_test.go`.
### Project Structure
```
```text
pgsql-broker/
├── cmd/broker/ # CLI application
├── pkg/broker/ # Core broker package
@@ -274,20 +386,26 @@ pgsql-broker/
│ ├── queue/ # Queue management
│ ├── worker/ # Worker implementation
│ └── install/ # Database installer with embedded SQL
│ └── sql/ # SQL schema (embedded in binary)
│ ├── tables/ # Table definitions
│ └── procedures/ # Stored procedures
│ └── sql/
│ ├── migrations/ # Ordered, versioned schema migrations (embedded in binary)
│ └── roles/ # Optional least-privilege role DDL (--with-roles)
├── tests/integration/ # Go integration test suite
├── examples/ # Usage examples
├── Dockerfile # Production broker image
├── Dockerfile.test # Integration test runner image
├── docker-compose.yml # Postgres + migrate + broker
├── docker-compose.test.yml # Postgres + test runner
└── Makefile # Build automation
```
## Contributing
Contributions are welcome! Please ensure:
- Code is formatted with `go fmt`
- Tests pass with `go test ./...`
- Documentation is updated
## License
See [LICENSE](./LICENSE) file for details.
See [LICENSE](./LICENSE) file for details.