PostgreSQL Broker

A robust, event-driven job processing system for PostgreSQL that uses LISTEN/NOTIFY for real-time job execution. It supports multiple queues, priority-based scheduling, and can be used both as a standalone service or as a Go library.

Features

• Multi-Database Support: Single broker process can manage multiple database connections
• Event-Driven: Uses PostgreSQL LISTEN/NOTIFY for instant job notifications
• Multiple Queues: Support for concurrent job processing across multiple queues per database
• Priority Scheduling: Jobs can be prioritized for execution order
• Job Dependencies: Jobs can depend on other jobs being completed first
• Adapter Pattern: Clean interfaces for database and logging (easy to extend)
• Standalone or Library: Use as a CLI tool or integrate into your Go application
• Configuration Management: Viper-based config with support for YAML, JSON, and environment variables
• Graceful Shutdown: Proper cleanup and job completion on shutdown
• Instance Tracking: Monitor active broker instances through the database
• Single Instance Per Database: Enforces one broker instance per database to prevent conflicts
• Embedded SQL Installer: Database schema embedded in binary with built-in install command

Architecture

The broker supports multi-database architecture where a single broker process can manage multiple database connections. Each database has its own instance with dedicated queues, but only ONE broker instance is allowed per database.

┌─────────────────────────────────────────────────────────────────┐
│                      Broker Process                              │
├─────────────────────────────────────────────────────────────────┤
│                                                                   │
│  ┌────────────────────────────┐  ┌───────────────────────────┐ │
│  │  Database Instance (DB1)   │  │  Database Instance (DB2)  │ │
│  ├────────────────────────────┤  ├───────────────────────────┤ │
│  │ ┌────┐ ┌────┐ ┌────┐      │  │ ┌────┐ ┌────┐            │ │
│  │ │ Q1 │ │ Q2 │ │ QN │      │  │ │ Q1 │ │ Q2 │            │ │
│  │ └────┘ └────┘ └────┘      │  │ └────┘ └────┘            │ │
│  │                            │  │                           │ │
│  │ ┌────────────────────────┐│  │ ┌────────────────────────┐│ │
│  │ │  PostgreSQL Adapter    ││  │ │  PostgreSQL Adapter    ││ │
│  │ │  - Connection Pool     ││  │ │  - Connection Pool     ││ │
│  │ │  - LISTEN/NOTIFY       ││  │ │  - LISTEN/NOTIFY       ││ │
│  │ └────────────────────────┘│  │ └────────────────────────┘│ │
│  └────────────┬───────────────┘  └──────────┬────────────────┘ │
│               │                              │                   │
└───────────────┼──────────────────────────────┼───────────────────┘
                │                              │
                ▼                              ▼
    ┌──────────────────────┐      ┌──────────────────────┐
    │  PostgreSQL (DB1)    │      │  PostgreSQL (DB2)    │
    │  - broker_jobs       │      │  - broker_jobs       │
    │  - broker_queueinstance│    │  - broker_queueinstance│
    │  - broker_schedule   │      │  - broker_schedule   │
    └──────────────────────┘      └──────────────────────┘

Key Points:

• One broker process can manage multiple databases
• Each database has exactly ONE active broker instance
• Each database instance has its own queues and workers
• Validation prevents multiple broker processes from connecting to the same database
• Different databases can have different queue counts

Installation

From Source

git clone git.warky.dev/wdevs/pgsql-broker
cd pgsql-broker
make build

The binary will be available in bin/pgsql-broker.

As a Library

go get git.warky.dev/wdevs/pgsql-broker

Quick Start

1. Setup Database

Install the required tables and stored procedures:

# Using the CLI (recommended)
./bin/pgsql-broker install --config broker.yaml

# Or with make
make sql-install

# Verify installation
./bin/pgsql-broker install --verify-only --config broker.yaml

# Or manually with psql:
psql -f pkg/broker/install/sql/tables/00_install.sql
psql -f pkg/broker/install/sql/procedures/00_install.sql

2. Configure

Create a configuration file broker.yaml:

databases:
  - name: db1
    host: localhost
    port: 5432
    database: broker_db1
    user: postgres
    password: your_password
    sslmode: disable
    queue_count: 4

  # Optional: add more databases
  - name: db2
    host: localhost
    port: 5432
    database: broker_db2
    user: postgres
    password: your_password
    sslmode: disable
    queue_count: 2

broker:
  name: pgsql-broker
  enable_debug: false

logging:
  level: info
  format: json

Note: Each database requires a unique name identifier and can have its own queue_count configuration.

3. Run the Broker

# Using the binary
./bin/pgsql-broker start --config broker.yaml

# Or with make
make run

# Or with custom log level
./bin/pgsql-broker start --log-level debug

4. Add a Job

SELECT broker_add_job(
    'My Job',                    -- job_name
    'SELECT do_something()',     -- execute_str
    1,                           -- job_queue (default: 1)
    0,                           -- job_priority (default: 0)
    'sql',                       -- job_language (default: 'sql')
    NULL,                        -- run_as
    NULL,                        -- user_login
    NULL                         -- schedule_id
);

Usage as a Library

package main

import (
    "git.warky.dev/wdevs/pgsql-broker/pkg/broker"
    "git.warky.dev/wdevs/pgsql-broker/pkg/broker/adapter"
    "git.warky.dev/wdevs/pgsql-broker/pkg/broker/config"
)

func main() {
    // Load config
    cfg, _ := config.LoadConfig("broker.yaml")

    // Create logger
    logger := adapter.NewSlogLogger(slog.LevelInfo)

    // Create database adapter
    dbAdapter := adapter.NewPostgresAdapter(cfg.Database.ToPostgresConfig(), logger)

    // Create and start broker
    instance, _ := broker.New(cfg, dbAdapter, logger, "1.0.0")
    instance.Start()

    // ... wait for shutdown signal ...

    instance.Stop()
}

See the examples directory for complete examples.

Database Schema

Tables

• broker_queueinstance: Tracks active broker queue instances (one per database)
• broker_jobs: Job queue with status tracking
• broker_schedule: Scheduled jobs (cron-like functionality)

Stored Procedures

• broker_get: Fetch the next job from a queue
• broker_run: Execute a job
• broker_set: Set runtime options (user, application_name, etc.)
• broker_add_job: Add a new job to the queue
• broker_register_instance: Register a broker instance
• broker_ping_instance: Update instance heartbeat
• broker_shutdown_instance: Mark instance as shutdown

Configuration Reference

See broker.example.yaml for a complete configuration example.

Database Settings

The databases array can contain multiple database configurations. Each entry supports:

Setting	Description	Default
name	Unique identifier for this database	Required
host	PostgreSQL host	localhost
port	PostgreSQL port	5432
database	Database name	Required
user	Database user	Required
password	Database password	-
sslmode	SSL mode	disable
max_open_conns	Max open connections	25
max_idle_conns	Max idle connections	5
conn_max_lifetime	Connection max lifetime	5m
conn_max_idle_time	Connection max idle time	10m
queue_count	Number of queues for this database	4

Broker Settings

Global settings applied to all database instances:

Setting	Description	Default
name	Broker instance name	pgsql-broker
fetch_query_que_size	Jobs per fetch cycle	100
queue_timer_sec	Seconds between polls	10
queue_buffer_size	Job buffer size	50
worker_idle_timeout_sec	Worker idle timeout	10
notify_retry_seconds	NOTIFY retry interval	30s
enable_debug	Enable debug logging	false

Development

Building

make build       # Build the binary
make clean       # Clean build artifacts
make deps        # Download dependencies
make fmt         # Format code
make vet         # Run go vet
make test        # Run tests

Project Structure

pgsql-broker/
├── cmd/broker/          # CLI application
├── pkg/broker/          # Core broker package
│   ├── adapter/         # Database & logger interfaces
│   ├── config/          # Configuration management
│   ├── models/          # Data models
│   ├── queue/           # Queue management
│   ├── worker/          # Worker implementation
│   └── install/         # Database installer with embedded SQL
│       └── sql/         # SQL schema (embedded in binary)
│           ├── tables/  # Table definitions
│           └── procedures/ # Stored procedures
├── examples/            # Usage examples
└── Makefile             # Build automation

Contributing

Contributions are welcome! Please ensure:

• Code is formatted with go fmt
• Tests pass with go test ./...
• Documentation is updated

License

See LICENSE file for details.