11 KiB
11 KiB
PostgreSQL Broker
A robust, event-driven job processing system for PostgreSQL that uses LISTEN/NOTIFY for real-time job execution. It supports multiple queues, priority-based scheduling, and can be used both as a standalone service or as a Go library.
Features
- Multi-Database Support: Single broker process can manage multiple database connections
- Event-Driven: Uses PostgreSQL LISTEN/NOTIFY for instant job notifications
- Multiple Queues: Support for concurrent job processing across multiple queues per database
- Priority Scheduling: Jobs can be prioritized for execution order
- Job Dependencies: Jobs can depend on other jobs being completed first
- Adapter Pattern: Clean interfaces for database and logging (easy to extend)
- Standalone or Library: Use as a CLI tool or integrate into your Go application
- Configuration Management: Viper-based config with support for YAML, JSON, and environment variables
- Graceful Shutdown: Proper cleanup and job completion on shutdown
- Instance Tracking: Monitor active broker instances through the database
- Single Instance Per Database: Enforces one broker instance per database to prevent conflicts
- Embedded SQL Installer: Database schema embedded in binary with built-in install command
Architecture
The broker supports multi-database architecture where a single broker process can manage multiple database connections. Each database has its own instance with dedicated queues, but only ONE broker instance is allowed per database.
┌─────────────────────────────────────────────────────────────────┐
│ Broker Process │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌────────────────────────────┐ ┌───────────────────────────┐ │
│ │ Database Instance (DB1) │ │ Database Instance (DB2) │ │
│ ├────────────────────────────┤ ├───────────────────────────┤ │
│ │ ┌────┐ ┌────┐ ┌────┐ │ │ ┌────┐ ┌────┐ │ │
│ │ │ Q1 │ │ Q2 │ │ QN │ │ │ │ Q1 │ │ Q2 │ │ │
│ │ └────┘ └────┘ └────┘ │ │ └────┘ └────┘ │ │
│ │ │ │ │ │
│ │ ┌────────────────────────┐│ │ ┌────────────────────────┐│ │
│ │ │ PostgreSQL Adapter ││ │ │ PostgreSQL Adapter ││ │
│ │ │ - Connection Pool ││ │ │ - Connection Pool ││ │
│ │ │ - LISTEN/NOTIFY ││ │ │ - LISTEN/NOTIFY ││ │
│ │ └────────────────────────┘│ │ └────────────────────────┘│ │
│ └────────────┬───────────────┘ └──────────┬────────────────┘ │
│ │ │ │
└───────────────┼──────────────────────────────┼───────────────────┘
│ │
▼ ▼
┌──────────────────────┐ ┌──────────────────────┐
│ PostgreSQL (DB1) │ │ PostgreSQL (DB2) │
│ - broker_jobs │ │ - broker_jobs │
│ - broker_queueinstance│ │ - broker_queueinstance│
│ - broker_schedule │ │ - broker_schedule │
└──────────────────────┘ └──────────────────────┘
Key Points:
- One broker process can manage multiple databases
- Each database has exactly ONE active broker instance
- Each database instance has its own queues and workers
- Validation prevents multiple broker processes from connecting to the same database
- Different databases can have different queue counts
Installation
From Source
git clone git.warky.dev/wdevs/pgsql-broker
cd pgsql-broker
make build
The binary will be available in bin/pgsql-broker.
As a Library
go get git.warky.dev/wdevs/pgsql-broker
Quick Start
1. Setup Database
Install the required tables and stored procedures:
# Using the CLI (recommended)
./bin/pgsql-broker install --config broker.yaml
# Or with make
make sql-install
# Verify installation
./bin/pgsql-broker install --verify-only --config broker.yaml
# Or manually with psql:
psql -f pkg/broker/install/sql/tables/00_install.sql
psql -f pkg/broker/install/sql/procedures/00_install.sql
2. Configure
Create a configuration file broker.yaml:
databases:
- name: db1
host: localhost
port: 5432
database: broker_db1
user: postgres
password: your_password
sslmode: disable
queue_count: 4
# Optional: add more databases
- name: db2
host: localhost
port: 5432
database: broker_db2
user: postgres
password: your_password
sslmode: disable
queue_count: 2
broker:
name: pgsql-broker
enable_debug: false
logging:
level: info
format: json
Note: Each database requires a unique name identifier and can have its own queue_count configuration.
3. Run the Broker
# Using the binary
./bin/pgsql-broker start --config broker.yaml
# Or with make
make run
# Or with custom log level
./bin/pgsql-broker start --log-level debug
4. Add a Job
SELECT broker_add_job(
'My Job', -- job_name
'SELECT do_something()', -- execute_str
1, -- job_queue (default: 1)
0, -- job_priority (default: 0)
'sql', -- job_language (default: 'sql')
NULL, -- run_as
NULL, -- user_login
NULL -- schedule_id
);
Usage as a Library
package main
import (
"git.warky.dev/wdevs/pgsql-broker/pkg/broker"
"git.warky.dev/wdevs/pgsql-broker/pkg/broker/adapter"
"git.warky.dev/wdevs/pgsql-broker/pkg/broker/config"
)
func main() {
// Load config
cfg, _ := config.LoadConfig("broker.yaml")
// Create logger
logger := adapter.NewSlogLogger(slog.LevelInfo)
// Create database adapter
dbAdapter := adapter.NewPostgresAdapter(cfg.Database.ToPostgresConfig(), logger)
// Create and start broker
instance, _ := broker.New(cfg, dbAdapter, logger, "1.0.0")
instance.Start()
// ... wait for shutdown signal ...
instance.Stop()
}
See the examples directory for complete examples.
Database Schema
Tables
- broker_queueinstance: Tracks active broker queue instances (one per database)
- broker_jobs: Job queue with status tracking
- broker_schedule: Scheduled jobs (cron-like functionality)
Stored Procedures
- broker_get: Fetch the next job from a queue
- broker_run: Execute a job
- broker_set: Set runtime options (user, application_name, etc.)
- broker_add_job: Add a new job to the queue
- broker_register_instance: Register a broker instance
- broker_ping_instance: Update instance heartbeat
- broker_shutdown_instance: Mark instance as shutdown
Configuration Reference
See broker.example.yaml for a complete configuration example.
Database Settings
The databases array can contain multiple database configurations. Each entry supports:
| Setting | Description | Default |
|---|---|---|
name |
Unique identifier for this database | Required |
host |
PostgreSQL host | localhost |
port |
PostgreSQL port | 5432 |
database |
Database name | Required |
user |
Database user | Required |
password |
Database password | - |
sslmode |
SSL mode | disable |
max_open_conns |
Max open connections | 25 |
max_idle_conns |
Max idle connections | 5 |
conn_max_lifetime |
Connection max lifetime | 5m |
conn_max_idle_time |
Connection max idle time | 10m |
queue_count |
Number of queues for this database | 4 |
Broker Settings
Global settings applied to all database instances:
| Setting | Description | Default |
|---|---|---|
name |
Broker instance name | pgsql-broker |
fetch_query_que_size |
Jobs per fetch cycle | 100 |
queue_timer_sec |
Seconds between polls | 10 |
queue_buffer_size |
Job buffer size | 50 |
worker_idle_timeout_sec |
Worker idle timeout | 10 |
notify_retry_seconds |
NOTIFY retry interval | 30s |
enable_debug |
Enable debug logging | false |
Development
Building
make build # Build the binary
make clean # Clean build artifacts
make deps # Download dependencies
make fmt # Format code
make vet # Run go vet
make test # Run tests
Project Structure
pgsql-broker/
├── cmd/broker/ # CLI application
├── pkg/broker/ # Core broker package
│ ├── adapter/ # Database & logger interfaces
│ ├── config/ # Configuration management
│ ├── models/ # Data models
│ ├── queue/ # Queue management
│ ├── worker/ # Worker implementation
│ └── install/ # Database installer with embedded SQL
│ └── sql/ # SQL schema (embedded in binary)
│ ├── tables/ # Table definitions
│ └── procedures/ # Stored procedures
├── examples/ # Usage examples
└── Makefile # Build automation
Contributing
Contributions are welcome! Please ensure:
- Code is formatted with
go fmt - Tests pass with
go test ./... - Documentation is updated
License
See LICENSE file for details.