feat: 🎉 postgresql broker first commit of forked prototype from my original code
This commit is contained in:
294
README.md
294
README.md
@@ -1,3 +1,293 @@
|
||||
# pgsql-broker
|
||||
# PostgreSQL Broker
|
||||
|
||||
PostgreSQL Broker
|
||||
A robust, event-driven job processing system for PostgreSQL that uses LISTEN/NOTIFY for real-time job execution. It supports multiple queues, priority-based scheduling, and can be used both as a standalone service or as a Go library.
|
||||
|
||||
## Features
|
||||
|
||||
- **Multi-Database Support**: Single broker process can manage multiple database connections
|
||||
- **Event-Driven**: Uses PostgreSQL LISTEN/NOTIFY for instant job notifications
|
||||
- **Multiple Queues**: Support for concurrent job processing across multiple queues per database
|
||||
- **Priority Scheduling**: Jobs can be prioritized for execution order
|
||||
- **Job Dependencies**: Jobs can depend on other jobs being completed first
|
||||
- **Adapter Pattern**: Clean interfaces for database and logging (easy to extend)
|
||||
- **Standalone or Library**: Use as a CLI tool or integrate into your Go application
|
||||
- **Configuration Management**: Viper-based config with support for YAML, JSON, and environment variables
|
||||
- **Graceful Shutdown**: Proper cleanup and job completion on shutdown
|
||||
- **Instance Tracking**: Monitor active broker instances through the database
|
||||
- **Single Instance Per Database**: Enforces one broker instance per database to prevent conflicts
|
||||
- **Embedded SQL Installer**: Database schema embedded in binary with built-in install command
|
||||
|
||||
## Architecture
|
||||
|
||||
The broker supports multi-database architecture where a single broker process can manage multiple database connections. Each database has its own instance with dedicated queues, but only ONE broker instance is allowed per database.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Broker Process │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌────────────────────────────┐ ┌───────────────────────────┐ │
|
||||
│ │ Database Instance (DB1) │ │ Database Instance (DB2) │ │
|
||||
│ ├────────────────────────────┤ ├───────────────────────────┤ │
|
||||
│ │ ┌────┐ ┌────┐ ┌────┐ │ │ ┌────┐ ┌────┐ │ │
|
||||
│ │ │ Q1 │ │ Q2 │ │ QN │ │ │ │ Q1 │ │ Q2 │ │ │
|
||||
│ │ └────┘ └────┘ └────┘ │ │ └────┘ └────┘ │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ ┌────────────────────────┐│ │ ┌────────────────────────┐│ │
|
||||
│ │ │ PostgreSQL Adapter ││ │ │ PostgreSQL Adapter ││ │
|
||||
│ │ │ - Connection Pool ││ │ │ - Connection Pool ││ │
|
||||
│ │ │ - LISTEN/NOTIFY ││ │ │ - LISTEN/NOTIFY ││ │
|
||||
│ │ └────────────────────────┘│ │ └────────────────────────┘│ │
|
||||
│ └────────────┬───────────────┘ └──────────┬────────────────┘ │
|
||||
│ │ │ │
|
||||
└───────────────┼──────────────────────────────┼───────────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌──────────────────────┐ ┌──────────────────────┐
|
||||
│ PostgreSQL (DB1) │ │ PostgreSQL (DB2) │
|
||||
│ - broker_jobs │ │ - broker_jobs │
|
||||
│ - broker_queueinstance│ │ - broker_queueinstance│
|
||||
│ - broker_schedule │ │ - broker_schedule │
|
||||
└──────────────────────┘ └──────────────────────┘
|
||||
```
|
||||
|
||||
**Key Points**:
|
||||
- One broker process can manage multiple databases
|
||||
- Each database has exactly ONE active broker instance
|
||||
- Each database instance has its own queues and workers
|
||||
- Validation prevents multiple broker processes from connecting to the same database
|
||||
- Different databases can have different queue counts
|
||||
|
||||
## Installation
|
||||
|
||||
### From Source
|
||||
|
||||
```bash
|
||||
git clone git.warky.dev/wdevs/pgsql-broker
|
||||
cd pgsql-broker
|
||||
make build
|
||||
```
|
||||
|
||||
The binary will be available in `bin/pgsql-broker`.
|
||||
|
||||
### As a Library
|
||||
|
||||
```bash
|
||||
go get git.warky.dev/wdevs/pgsql-broker
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Setup Database
|
||||
|
||||
Install the required tables and stored procedures:
|
||||
|
||||
```bash
|
||||
# Using the CLI (recommended)
|
||||
./bin/pgsql-broker install --config broker.yaml
|
||||
|
||||
# Or with make
|
||||
make sql-install
|
||||
|
||||
# Verify installation
|
||||
./bin/pgsql-broker install --verify-only --config broker.yaml
|
||||
|
||||
# Or manually with psql:
|
||||
psql -f pkg/broker/install/sql/tables/00_install.sql
|
||||
psql -f pkg/broker/install/sql/procedures/00_install.sql
|
||||
```
|
||||
|
||||
### 2. Configure
|
||||
|
||||
Create a configuration file `broker.yaml`:
|
||||
|
||||
```yaml
|
||||
databases:
|
||||
- name: db1
|
||||
host: localhost
|
||||
port: 5432
|
||||
database: broker_db1
|
||||
user: postgres
|
||||
password: your_password
|
||||
sslmode: disable
|
||||
queue_count: 4
|
||||
|
||||
# Optional: add more databases
|
||||
- name: db2
|
||||
host: localhost
|
||||
port: 5432
|
||||
database: broker_db2
|
||||
user: postgres
|
||||
password: your_password
|
||||
sslmode: disable
|
||||
queue_count: 2
|
||||
|
||||
broker:
|
||||
name: pgsql-broker
|
||||
enable_debug: false
|
||||
|
||||
logging:
|
||||
level: info
|
||||
format: json
|
||||
```
|
||||
|
||||
**Note**: Each database requires a unique `name` identifier and can have its own `queue_count` configuration.
|
||||
|
||||
### 3. Run the Broker
|
||||
|
||||
```bash
|
||||
# Using the binary
|
||||
./bin/pgsql-broker start --config broker.yaml
|
||||
|
||||
# Or with make
|
||||
make run
|
||||
|
||||
# Or with custom log level
|
||||
./bin/pgsql-broker start --log-level debug
|
||||
```
|
||||
|
||||
### 4. Add a Job
|
||||
|
||||
```sql
|
||||
SELECT broker_add_job(
|
||||
'My Job', -- job_name
|
||||
'SELECT do_something()', -- execute_str
|
||||
1, -- job_queue (default: 1)
|
||||
0, -- job_priority (default: 0)
|
||||
'sql', -- job_language (default: 'sql')
|
||||
NULL, -- run_as
|
||||
NULL, -- user_login
|
||||
NULL -- schedule_id
|
||||
);
|
||||
```
|
||||
|
||||
## Usage as a Library
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"git.warky.dev/wdevs/pgsql-broker/pkg/broker"
|
||||
"git.warky.dev/wdevs/pgsql-broker/pkg/broker/adapter"
|
||||
"git.warky.dev/wdevs/pgsql-broker/pkg/broker/config"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// Load config
|
||||
cfg, _ := config.LoadConfig("broker.yaml")
|
||||
|
||||
// Create logger
|
||||
logger := adapter.NewSlogLogger(slog.LevelInfo)
|
||||
|
||||
// Create database adapter
|
||||
dbAdapter := adapter.NewPostgresAdapter(cfg.Database.ToPostgresConfig(), logger)
|
||||
|
||||
// Create and start broker
|
||||
instance, _ := broker.New(cfg, dbAdapter, logger, "1.0.0")
|
||||
instance.Start()
|
||||
|
||||
// ... wait for shutdown signal ...
|
||||
|
||||
instance.Stop()
|
||||
}
|
||||
```
|
||||
|
||||
See the [examples](./examples/) directory for complete examples.
|
||||
|
||||
## Database Schema
|
||||
|
||||
### Tables
|
||||
|
||||
- **broker_queueinstance**: Tracks active broker queue instances (one per database)
|
||||
- **broker_jobs**: Job queue with status tracking
|
||||
- **broker_schedule**: Scheduled jobs (cron-like functionality)
|
||||
|
||||
### Stored Procedures
|
||||
|
||||
- **broker_get**: Fetch the next job from a queue
|
||||
- **broker_run**: Execute a job
|
||||
- **broker_set**: Set runtime options (user, application_name, etc.)
|
||||
- **broker_add_job**: Add a new job to the queue
|
||||
- **broker_register_instance**: Register a broker instance
|
||||
- **broker_ping_instance**: Update instance heartbeat
|
||||
- **broker_shutdown_instance**: Mark instance as shutdown
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
See [broker.example.yaml](./broker.example.yaml) for a complete configuration example.
|
||||
|
||||
### Database Settings
|
||||
|
||||
The `databases` array can contain multiple database configurations. Each entry supports:
|
||||
|
||||
| Setting | Description | Default |
|
||||
|---------|-------------|---------|
|
||||
| `name` | Unique identifier for this database | **Required** |
|
||||
| `host` | PostgreSQL host | `localhost` |
|
||||
| `port` | PostgreSQL port | `5432` |
|
||||
| `database` | Database name | **Required** |
|
||||
| `user` | Database user | **Required** |
|
||||
| `password` | Database password | - |
|
||||
| `sslmode` | SSL mode | `disable` |
|
||||
| `max_open_conns` | Max open connections | `25` |
|
||||
| `max_idle_conns` | Max idle connections | `5` |
|
||||
| `conn_max_lifetime` | Connection max lifetime | `5m` |
|
||||
| `conn_max_idle_time` | Connection max idle time | `10m` |
|
||||
| `queue_count` | Number of queues for this database | `4` |
|
||||
|
||||
### Broker Settings
|
||||
|
||||
Global settings applied to all database instances:
|
||||
|
||||
| Setting | Description | Default |
|
||||
|---------|-------------|---------|
|
||||
| `name` | Broker instance name | `pgsql-broker` |
|
||||
| `fetch_query_que_size` | Jobs per fetch cycle | `100` |
|
||||
| `queue_timer_sec` | Seconds between polls | `10` |
|
||||
| `queue_buffer_size` | Job buffer size | `50` |
|
||||
| `worker_idle_timeout_sec` | Worker idle timeout | `10` |
|
||||
| `notify_retry_seconds` | NOTIFY retry interval | `30s` |
|
||||
| `enable_debug` | Enable debug logging | `false` |
|
||||
|
||||
## Development
|
||||
|
||||
### Building
|
||||
|
||||
```bash
|
||||
make build # Build the binary
|
||||
make clean # Clean build artifacts
|
||||
make deps # Download dependencies
|
||||
make fmt # Format code
|
||||
make vet # Run go vet
|
||||
make test # Run tests
|
||||
```
|
||||
|
||||
### Project Structure
|
||||
|
||||
```
|
||||
pgsql-broker/
|
||||
├── cmd/broker/ # CLI application
|
||||
├── pkg/broker/ # Core broker package
|
||||
│ ├── adapter/ # Database & logger interfaces
|
||||
│ ├── config/ # Configuration management
|
||||
│ ├── models/ # Data models
|
||||
│ ├── queue/ # Queue management
|
||||
│ ├── worker/ # Worker implementation
|
||||
│ └── install/ # Database installer with embedded SQL
|
||||
│ └── sql/ # SQL schema (embedded in binary)
|
||||
│ ├── tables/ # Table definitions
|
||||
│ └── procedures/ # Stored procedures
|
||||
├── examples/ # Usage examples
|
||||
└── Makefile # Build automation
|
||||
```
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions are welcome! Please ensure:
|
||||
- Code is formatted with `go fmt`
|
||||
- Tests pass with `go test ./...`
|
||||
- Documentation is updated
|
||||
|
||||
## License
|
||||
|
||||
See [LICENSE](./LICENSE) file for details.
|
||||
Reference in New Issue
Block a user