Added a scripts execution ability
Some checks failed
CI / Test (1.24) (push) Successful in -25m17s
CI / Test (1.25) (push) Successful in -25m15s
CI / Build (push) Successful in -25m45s
CI / Lint (push) Successful in -25m31s
Integration Tests / Integration Tests (push) Failing after -25m58s

This commit is contained in:
2025-12-31 00:44:14 +02:00
parent 1d193c84d7
commit adfe126758
11 changed files with 2402 additions and 0 deletions

View File

@@ -0,0 +1,160 @@
# SQL Directory Reader
The SQL Directory Reader (`sqldir`) reads SQL scripts from a directory structure and populates the `Scripts` field of a `Schema`. It supports recursive directory scanning and extracts priority, sequence, and name information from filenames.
## File Naming Convention
Scripts must follow this naming pattern (supports both underscores and hyphens as separators):
```
{priority}_{sequence}_{name}.{sql|pgsql}
{priority}-{sequence}-{name}.{sql|pgsql}
```
### Components
- **priority**: Integer (0-9999) - Defines execution order (lower executes first)
- **sequence**: Integer (0-9999) - Defines order within the same priority level
- **separator**: Underscore `_` or hyphen `-` (can be mixed)
- **name**: Descriptive name (alphanumeric, underscores, hyphens allowed)
- **extension**: `.sql` or `.pgsql`
### Examples
```
migrations/
├── 1_001_create_schema.sql # Priority 1, Sequence 1 (underscore format)
├── 1-002-create-users-table.sql # Priority 1, Sequence 2 (hyphen format)
├── 1_003_create_posts_table.pgsql # Priority 1, Sequence 3 (underscore format)
├── 2-001-add-indexes.sql # Priority 2, Sequence 1 (hyphen format)
├── 2_002_add_constraints.sql # Priority 2, Sequence 2 (underscore format)
├── 10-10-create-newid.pgsql # Priority 10, Sequence 10 (hyphen format)
└── subdirectory/
└── 3_001_seed_data.sql # Priority 3, Sequence 1 (subdirs supported)
```
**Execution Order**: 1→2→3→4→5→6→7 (sorted by Priority ascending, then Sequence ascending)
**Both formats can be mixed** in the same directory - the reader handles both seamlessly.
### Invalid Filenames (Ignored)
- `migration.sql` - Missing priority/sequence
- `1_create_users.sql` - Missing sequence
- `create_users.sql` - Missing priority/sequence
- `1_001_test.txt` - Wrong extension
- `readme.md` - Not a SQL file
## Usage
### Basic Usage
```go
import (
"git.warky.dev/wdevs/relspecgo/pkg/readers"
"git.warky.dev/wdevs/relspecgo/pkg/readers/sqldir"
)
reader := sqldir.NewReader(&readers.ReaderOptions{
FilePath: "/path/to/migrations",
Metadata: map[string]any{
"schema_name": "public", // Optional, defaults to "public"
"database_name": "myapp", // Optional, defaults to "database"
},
})
// Read all scripts
database, err := reader.ReadDatabase()
if err != nil {
log.Fatal(err)
}
// Access scripts
for _, schema := range database.Schemas {
for _, script := range schema.Scripts {
fmt.Printf("Script: %s (P:%d S:%d)\n",
script.Name, script.Priority, script.Sequence)
fmt.Printf("SQL: %s\n", script.SQL)
}
}
```
### Read Schema Only
```go
schema, err := reader.ReadSchema()
if err != nil {
log.Fatal(err)
}
fmt.Printf("Found %d scripts\n", len(schema.Scripts))
```
## Features
- **Recursive Directory Scanning**: Automatically scans all subdirectories
- **Multiple Extensions**: Supports both `.sql` and `.pgsql` files
- **Flexible Naming**: Extract metadata from filename patterns
- **Error Handling**: Validates directory existence and file accessibility
- **Schema Integration**: Scripts are added to the standard RelSpec `Schema` model
## Script Model
Each script is stored as a `models.Script`:
```go
type Script struct {
Name string // Extracted from filename (e.g., "create_users")
Description string // Auto-generated description with file path
SQL string // Complete SQL content from file
Priority int // Execution priority from filename
Sequence uint // Execution sequence from filename
// ... other fields available but not populated by this reader
}
```
## Integration with SQL Executor
The SQL Directory Reader is designed to work seamlessly with the SQL Executor Writer:
```go
// Read scripts
reader := sqldir.NewReader(&readers.ReaderOptions{
FilePath: "./migrations",
})
db, _ := reader.ReadDatabase()
// Execute scripts
writer := sqlexec.NewWriter(&writers.WriterOptions{
Metadata: map[string]any{
"connection_string": "postgres://localhost/mydb",
},
})
writer.WriteDatabase(db) // Executes in Priority→Sequence order
```
See `pkg/writers/sqlexec/README.md` for more details on script execution.
## Error Handling
The reader will return errors for:
- Non-existent directory paths
- Inaccessible directories or files
- Invalid file permissions
- File read failures
Files that don't match the naming pattern are silently ignored (not treated as errors).
## Testing
Run tests:
```bash
go test ./pkg/readers/sqldir/
```
Tests include:
- Valid file parsing
- Recursive directory scanning
- Invalid filename handling
- Empty directory handling
- Error conditions

View File

@@ -0,0 +1,127 @@
package sqldir_test
import (
"fmt"
"log"
"git.warky.dev/wdevs/relspecgo/pkg/readers"
"git.warky.dev/wdevs/relspecgo/pkg/readers/sqldir"
"git.warky.dev/wdevs/relspecgo/pkg/writers"
"git.warky.dev/wdevs/relspecgo/pkg/writers/sqlexec"
)
// Example demonstrates how to read SQL scripts from a directory and execute them
func Example() {
// Step 1: Read SQL scripts from a directory
// Directory structure example:
// migrations/
// 1_001_create_schema.sql
// 1_002_create_users_table.sql
// 1_003_create_posts_table.pgsql
// 2_001_add_indexes.sql
// 2_002_seed_data.sql
reader := sqldir.NewReader(&readers.ReaderOptions{
FilePath: "/path/to/migrations",
Metadata: map[string]any{
"schema_name": "public",
"database_name": "myapp",
},
})
// Read the database schema with scripts
database, err := reader.ReadDatabase()
if err != nil {
log.Fatalf("Failed to read scripts: %v", err)
}
fmt.Printf("Read %d schemas\n", len(database.Schemas))
fmt.Printf("Found %d scripts in schema '%s'\n",
len(database.Schemas[0].Scripts),
database.Schemas[0].Name)
// Step 2: Execute the scripts against a PostgreSQL database
writer := sqlexec.NewWriter(&writers.WriterOptions{
Metadata: map[string]any{
"connection_string": "postgres://user:password@localhost:5432/myapp?sslmode=disable",
},
})
// Execute all scripts in Priority then Sequence order
if err := writer.WriteDatabase(database); err != nil {
log.Fatalf("Failed to execute scripts: %v", err)
}
fmt.Println("All scripts executed successfully!")
}
// Example_withSingleSchema shows how to read and execute scripts for a single schema
func Example_withSingleSchema() {
// Read scripts
reader := sqldir.NewReader(&readers.ReaderOptions{
FilePath: "/path/to/migrations",
})
schema, err := reader.ReadSchema()
if err != nil {
log.Fatalf("Failed to read schema: %v", err)
}
// Execute scripts
writer := sqlexec.NewWriter(&writers.WriterOptions{
Metadata: map[string]any{
"connection_string": "postgres://localhost/testdb",
},
})
if err := writer.WriteSchema(schema); err != nil {
log.Fatalf("Failed to execute scripts: %v", err)
}
fmt.Println("Schema scripts executed successfully!")
}
// Example_fileNamingConvention shows the expected file naming pattern
func Example_fileNamingConvention() {
// File naming pattern: {priority}_{sequence}_{name}.sql or .pgsql
// OR: {priority}-{sequence}-{name}.sql or .pgsql
//
// Both underscore (_) and hyphen (-) separators are supported and can be mixed.
//
// Components:
// - priority: Integer (0-9999) - Scripts with lower priority execute first
// - sequence: Integer (0-9999) - Within same priority, lower sequence executes first
// - separator: Underscore (_) or hyphen (-)
// - name: Descriptive name (alphanumeric, underscores, hyphens)
// - extension: .sql or .pgsql
//
// Examples (underscore format):
// ✓ 1_001_create_users.sql (Priority=1, Sequence=1)
// ✓ 1_002_create_posts.sql (Priority=1, Sequence=2)
// ✓ 2_001_add_indexes.pgsql (Priority=2, Sequence=1)
// ✓ 10_100_migration.sql (Priority=10, Sequence=100)
//
// Examples (hyphen format):
// ✓ 1-001-create-users.sql (Priority=1, Sequence=1)
// ✓ 1-002-create-posts.sql (Priority=1, Sequence=2)
// ✓ 2-001-add-indexes.pgsql (Priority=2, Sequence=1)
// ✓ 10-10-create-newid.pgsql (Priority=10, Sequence=10)
//
// Mixed format (both in same directory):
// ✓ 1_001_create_users.sql (underscore format)
// ✓ 1-002-create-posts.sql (hyphen format)
// ✓ 2_001_add_indexes.sql (underscore format)
//
// Execution order for mixed examples:
// 1. 1_001_create_users.sql (Priority 1, Sequence 1)
// 2. 1-002-create-posts.sql (Priority 1, Sequence 2)
// 3. 2_001_add_indexes.sql (Priority 2, Sequence 1)
//
// Invalid filenames (will be ignored):
// ✗ migration.sql (missing priority/sequence)
// ✗ 1_create_users.sql (missing sequence)
// ✗ create_users.sql (missing priority/sequence)
// ✗ 1_001_create_users.txt (wrong extension)
fmt.Println("See comments for file naming conventions")
}

View File

@@ -0,0 +1,171 @@
package sqldir
import (
"fmt"
"os"
"path/filepath"
"regexp"
"strconv"
"git.warky.dev/wdevs/relspecgo/pkg/models"
"git.warky.dev/wdevs/relspecgo/pkg/readers"
)
// Reader implements the readers.Reader interface for SQL script directories
type Reader struct {
options *readers.ReaderOptions
}
// NewReader creates a new SQL directory reader
func NewReader(options *readers.ReaderOptions) *Reader {
return &Reader{
options: options,
}
}
// ReadDatabase reads all SQL scripts from a directory into a Database
func (r *Reader) ReadDatabase() (*models.Database, error) {
if r.options.FilePath == "" {
return nil, fmt.Errorf("directory path is required")
}
// Check if directory exists
info, err := os.Stat(r.options.FilePath)
if err != nil {
return nil, fmt.Errorf("failed to access directory: %w", err)
}
if !info.IsDir() {
return nil, fmt.Errorf("path is not a directory: %s", r.options.FilePath)
}
// Read scripts from directory
scripts, err := r.readScripts()
if err != nil {
return nil, fmt.Errorf("failed to read scripts: %w", err)
}
// Get schema name from metadata or use default
schemaName := "public"
if name, ok := r.options.Metadata["schema_name"].(string); ok && name != "" {
schemaName = name
}
// Create schema with scripts
schema := &models.Schema{
Name: schemaName,
Scripts: scripts,
}
// Get database name from metadata or use default
dbName := "database"
if name, ok := r.options.Metadata["database_name"].(string); ok && name != "" {
dbName = name
}
// Create database with schema
database := &models.Database{
Name: dbName,
Schemas: []*models.Schema{schema},
}
// Set back-reference
schema.RefDatabase = database
return database, nil
}
// ReadSchema reads all SQL scripts from a directory into a Schema
func (r *Reader) ReadSchema() (*models.Schema, error) {
db, err := r.ReadDatabase()
if err != nil {
return nil, err
}
if len(db.Schemas) == 0 {
return nil, fmt.Errorf("no schema found")
}
return db.Schemas[0], nil
}
// ReadTable is not applicable for SQL script directories
func (r *Reader) ReadTable() (*models.Table, error) {
return nil, fmt.Errorf("ReadTable is not supported for SQL script directories")
}
// readScripts recursively scans the directory for SQL files and parses them into Script models
func (r *Reader) readScripts() ([]*models.Script, error) {
var scripts []*models.Script
// Regular expression to parse filename: {priority}{sep}{sequence}{sep}{name}.sql or .pgsql
// Separator can be underscore (_) or hyphen (-)
// Example: 1_001_create_users.sql -> priority=1, sequence=001, name=create_users
// Example: 2_005_add_indexes.pgsql -> priority=2, sequence=005, name=add_indexes
// Example: 10-10-create-newid.pgsql -> priority=10, sequence=10, name=create-newid
pattern := regexp.MustCompile(`^(\d+)[_-](\d+)[_-](.+)\.(sql|pgsql)$`)
err := filepath.WalkDir(r.options.FilePath, func(path string, d os.DirEntry, err error) error {
if err != nil {
return err
}
// Skip directories
if d.IsDir() {
return nil
}
// Get filename
filename := d.Name()
// Match against pattern
matches := pattern.FindStringSubmatch(filename)
if matches == nil {
// Skip files that don't match the pattern
return nil
}
// Parse priority
priority, err := strconv.Atoi(matches[1])
if err != nil {
return fmt.Errorf("invalid priority in filename %s: %w", filename, err)
}
// Parse sequence
sequence, err := strconv.ParseUint(matches[2], 10, 64)
if err != nil {
return fmt.Errorf("invalid sequence in filename %s: %w", filename, err)
}
// Extract name
name := matches[3]
// Read SQL content
content, err := os.ReadFile(path)
if err != nil {
return fmt.Errorf("failed to read file %s: %w", path, err)
}
// Get relative path from base directory
relPath, err := filepath.Rel(r.options.FilePath, path)
if err != nil {
relPath = path
}
// Create Script model
script := &models.Script{
Name: name,
Description: fmt.Sprintf("SQL script from %s", relPath),
SQL: string(content),
Priority: priority,
Sequence: uint(sequence),
}
scripts = append(scripts, script)
return nil
})
if err != nil {
return nil, err
}
return scripts, nil
}

View File

@@ -0,0 +1,375 @@
package sqldir
import (
"os"
"path/filepath"
"testing"
"git.warky.dev/wdevs/relspecgo/pkg/readers"
)
func TestReader_ReadDatabase(t *testing.T) {
// Create temporary test directory
tempDir, err := os.MkdirTemp("", "sqldir-test-*")
if err != nil {
t.Fatalf("Failed to create temp directory: %v", err)
}
defer os.RemoveAll(tempDir)
// Create test SQL files with both underscore and hyphen separators
testFiles := map[string]string{
"1_001_create_users.sql": "CREATE TABLE users (id SERIAL PRIMARY KEY, name TEXT);",
"1_002_create_posts.sql": "CREATE TABLE posts (id SERIAL PRIMARY KEY, user_id INT);",
"2_001_add_indexes.sql": "CREATE INDEX idx_posts_user_id ON posts(user_id);",
"1_003_seed_data.pgsql": "INSERT INTO users (name) VALUES ('Alice'), ('Bob');",
"10-10-create-newid.pgsql": "CREATE TABLE newid (id SERIAL PRIMARY KEY);",
"2-005-add-column.sql": "ALTER TABLE users ADD COLUMN email TEXT;",
}
for filename, content := range testFiles {
filePath := filepath.Join(tempDir, filename)
if err := os.WriteFile(filePath, []byte(content), 0644); err != nil {
t.Fatalf("Failed to create test file %s: %v", filename, err)
}
}
// Create subdirectory with additional script
subDir := filepath.Join(tempDir, "migrations")
if err := os.MkdirAll(subDir, 0755); err != nil {
t.Fatalf("Failed to create subdirectory: %v", err)
}
subFile := filepath.Join(subDir, "3_001_add_column.sql")
if err := os.WriteFile(subFile, []byte("ALTER TABLE users ADD COLUMN email TEXT;"), 0644); err != nil {
t.Fatalf("Failed to create subdirectory file: %v", err)
}
// Create reader
reader := NewReader(&readers.ReaderOptions{
FilePath: tempDir,
Metadata: map[string]any{
"schema_name": "test_schema",
"database_name": "test_db",
},
})
// Read database
db, err := reader.ReadDatabase()
if err != nil {
t.Fatalf("ReadDatabase failed: %v", err)
}
// Verify database
if db.Name != "test_db" {
t.Errorf("Expected database name 'test_db', got '%s'", db.Name)
}
if len(db.Schemas) != 1 {
t.Fatalf("Expected 1 schema, got %d", len(db.Schemas))
}
schema := db.Schemas[0]
if schema.Name != "test_schema" {
t.Errorf("Expected schema name 'test_schema', got '%s'", schema.Name)
}
// Verify scripts (should be 7 total: 4 underscore + 2 hyphen + 1 subdirectory)
if len(schema.Scripts) != 7 {
t.Fatalf("Expected 7 scripts, got %d", len(schema.Scripts))
}
// Verify script details
expectedScripts := []struct {
name string
priority int
sequence uint
}{
{"create_users", 1, 1},
{"create_posts", 1, 2},
{"seed_data", 1, 3},
{"add_indexes", 2, 1},
{"add-column", 2, 5},
{"add_column", 3, 1},
{"create-newid", 10, 10},
}
scriptMap := make(map[string]*struct {
priority int
sequence uint
sql string
})
for _, script := range schema.Scripts {
scriptMap[script.Name] = &struct {
priority int
sequence uint
sql string
}{
priority: script.Priority,
sequence: script.Sequence,
sql: script.SQL,
}
}
for _, expected := range expectedScripts {
script, exists := scriptMap[expected.name]
if !exists {
t.Errorf("Expected script '%s' not found", expected.name)
continue
}
if script.priority != expected.priority {
t.Errorf("Script '%s': expected priority %d, got %d",
expected.name, expected.priority, script.priority)
}
if script.sequence != expected.sequence {
t.Errorf("Script '%s': expected sequence %d, got %d",
expected.name, expected.sequence, script.sequence)
}
if script.sql == "" {
t.Errorf("Script '%s': SQL content is empty", expected.name)
}
}
}
func TestReader_ReadSchema(t *testing.T) {
// Create temporary test directory
tempDir, err := os.MkdirTemp("", "sqldir-test-*")
if err != nil {
t.Fatalf("Failed to create temp directory: %v", err)
}
defer os.RemoveAll(tempDir)
// Create test SQL file
testFile := filepath.Join(tempDir, "1_001_test.sql")
if err := os.WriteFile(testFile, []byte("SELECT 1;"), 0644); err != nil {
t.Fatalf("Failed to create test file: %v", err)
}
// Create reader
reader := NewReader(&readers.ReaderOptions{
FilePath: tempDir,
})
// Read schema
schema, err := reader.ReadSchema()
if err != nil {
t.Fatalf("ReadSchema failed: %v", err)
}
// Verify schema
if schema.Name != "public" {
t.Errorf("Expected default schema name 'public', got '%s'", schema.Name)
}
if len(schema.Scripts) != 1 {
t.Fatalf("Expected 1 script, got %d", len(schema.Scripts))
}
}
func TestReader_InvalidDirectory(t *testing.T) {
reader := NewReader(&readers.ReaderOptions{
FilePath: "/nonexistent/directory",
})
_, err := reader.ReadDatabase()
if err == nil {
t.Error("Expected error for nonexistent directory, got nil")
}
}
func TestReader_EmptyDirectory(t *testing.T) {
// Create temporary empty directory
tempDir, err := os.MkdirTemp("", "sqldir-test-*")
if err != nil {
t.Fatalf("Failed to create temp directory: %v", err)
}
defer os.RemoveAll(tempDir)
reader := NewReader(&readers.ReaderOptions{
FilePath: tempDir,
})
db, err := reader.ReadDatabase()
if err != nil {
t.Fatalf("ReadDatabase failed: %v", err)
}
if len(db.Schemas[0].Scripts) != 0 {
t.Errorf("Expected 0 scripts in empty directory, got %d", len(db.Schemas[0].Scripts))
}
}
func TestReader_InvalidFilename(t *testing.T) {
// Create temporary test directory
tempDir, err := os.MkdirTemp("", "sqldir-test-*")
if err != nil {
t.Fatalf("Failed to create temp directory: %v", err)
}
defer os.RemoveAll(tempDir)
// Create files with various invalid patterns
invalidFiles := []string{
"invalid.sql", // No priority/sequence
"1_test.sql", // Missing sequence
"test_1_2.sql", // Wrong order
"a_001_test.sql", // Non-numeric priority
"1_abc_test.sql", // Non-numeric sequence
"1_001_test.txt", // Wrong extension
"1_001_test.sql.backup", // Wrong extension
}
for _, filename := range invalidFiles {
filePath := filepath.Join(tempDir, filename)
if err := os.WriteFile(filePath, []byte("SELECT 1;"), 0644); err != nil {
t.Fatalf("Failed to create test file %s: %v", filename, err)
}
}
// Create one valid file
validFile := filepath.Join(tempDir, "1_001_valid.sql")
if err := os.WriteFile(validFile, []byte("SELECT 1;"), 0644); err != nil {
t.Fatalf("Failed to create valid file: %v", err)
}
reader := NewReader(&readers.ReaderOptions{
FilePath: tempDir,
})
db, err := reader.ReadDatabase()
if err != nil {
t.Fatalf("ReadDatabase failed: %v", err)
}
// Should only have the valid file
if len(db.Schemas[0].Scripts) != 1 {
t.Errorf("Expected 1 script (invalid files should be skipped), got %d", len(db.Schemas[0].Scripts))
}
if db.Schemas[0].Scripts[0].Name != "valid" {
t.Errorf("Expected script name 'valid', got '%s'", db.Schemas[0].Scripts[0].Name)
}
}
func TestReader_ReadTable(t *testing.T) {
reader := NewReader(&readers.ReaderOptions{})
_, err := reader.ReadTable()
if err == nil {
t.Error("Expected error for ReadTable (not supported), got nil")
}
}
func TestReader_HyphenFormat(t *testing.T) {
// Create temporary test directory
tempDir, err := os.MkdirTemp("", "sqldir-test-hyphen-*")
if err != nil {
t.Fatalf("Failed to create temp directory: %v", err)
}
defer os.RemoveAll(tempDir)
// Create test files with hyphen separators
testFiles := map[string]string{
"1-001-create-table.sql": "CREATE TABLE test (id INT);",
"1-002-insert-data.pgsql": "INSERT INTO test VALUES (1);",
"10-10-create-newid.pgsql": "CREATE TABLE newid (id SERIAL);",
"2-005-add-index.sql": "CREATE INDEX idx_test ON test(id);",
}
for filename, content := range testFiles {
filePath := filepath.Join(tempDir, filename)
if err := os.WriteFile(filePath, []byte(content), 0644); err != nil {
t.Fatalf("Failed to create test file %s: %v", filename, err)
}
}
// Create reader
reader := NewReader(&readers.ReaderOptions{
FilePath: tempDir,
})
// Read database
db, err := reader.ReadDatabase()
if err != nil {
t.Fatalf("ReadDatabase failed: %v", err)
}
schema := db.Schemas[0]
if len(schema.Scripts) != 4 {
t.Fatalf("Expected 4 scripts, got %d", len(schema.Scripts))
}
// Verify specific hyphen-formatted scripts
expectedScripts := map[string]struct {
priority int
sequence uint
}{
"create-table": {1, 1},
"insert-data": {1, 2},
"add-index": {2, 5},
"create-newid": {10, 10},
}
for _, script := range schema.Scripts {
expected, exists := expectedScripts[script.Name]
if !exists {
t.Errorf("Unexpected script: %s", script.Name)
continue
}
if script.Priority != expected.priority {
t.Errorf("Script '%s': expected priority %d, got %d",
script.Name, expected.priority, script.Priority)
}
if script.Sequence != expected.sequence {
t.Errorf("Script '%s': expected sequence %d, got %d",
script.Name, expected.sequence, script.Sequence)
}
}
}
func TestReader_MixedFormat(t *testing.T) {
// Test that both underscore and hyphen formats can be mixed
tempDir, err := os.MkdirTemp("", "sqldir-test-mixed-*")
if err != nil {
t.Fatalf("Failed to create temp directory: %v", err)
}
defer os.RemoveAll(tempDir)
testFiles := map[string]string{
"1_001_underscore.sql": "SELECT 1;",
"1-002-hyphen.sql": "SELECT 2;",
"2_003_underscore.sql": "SELECT 3;",
"2-004-hyphen.sql": "SELECT 4;",
}
for filename, content := range testFiles {
filePath := filepath.Join(tempDir, filename)
if err := os.WriteFile(filePath, []byte(content), 0644); err != nil {
t.Fatalf("Failed to create test file %s: %v", filename, err)
}
}
reader := NewReader(&readers.ReaderOptions{
FilePath: tempDir,
})
db, err := reader.ReadDatabase()
if err != nil {
t.Fatalf("ReadDatabase failed: %v", err)
}
schema := db.Schemas[0]
if len(schema.Scripts) != 4 {
t.Fatalf("Expected 4 scripts (mixed format), got %d", len(schema.Scripts))
}
// Verify both formats are parsed correctly
names := make(map[string]bool)
for _, script := range schema.Scripts {
names[script.Name] = true
}
expectedNames := []string{"underscore", "hyphen", "underscore", "hyphen"}
for _, name := range expectedNames {
if !names[name] {
t.Errorf("Expected script name '%s' not found", name)
}
}
}

View File

@@ -0,0 +1,226 @@
# SQL Executor Writer
The SQL Executor Writer (`sqlexec`) executes SQL scripts from `models.Script` objects against a PostgreSQL database. Scripts are executed in order based on Priority (ascending) and Sequence (ascending).
## Features
- **Ordered Execution**: Scripts execute in Priority→Sequence order
- **PostgreSQL Support**: Uses `pgx/v5` driver for robust PostgreSQL connectivity
- **Stop on Error**: Execution halts immediately on first error (default behavior)
- **Progress Reporting**: Prints execution status to stdout
- **Multiple Schemas**: Can execute scripts from multiple schemas in a database
## Usage
### Basic Usage
```go
import (
"git.warky.dev/wdevs/relspecgo/pkg/writers"
"git.warky.dev/wdevs/relspecgo/pkg/writers/sqlexec"
)
writer := sqlexec.NewWriter(&writers.WriterOptions{
Metadata: map[string]any{
"connection_string": "postgres://user:password@localhost:5432/dbname?sslmode=disable",
},
})
// Execute all scripts from database
err := writer.WriteDatabase(database)
if err != nil {
log.Fatalf("Execution failed: %v", err)
}
```
### Execute Single Schema
```go
err := writer.WriteSchema(schema)
if err != nil {
log.Fatalf("Schema execution failed: %v", err)
}
```
### Complete Example with SQL Directory Reader
```go
import (
"log"
"git.warky.dev/wdevs/relspecgo/pkg/readers"
"git.warky.dev/wdevs/relspecgo/pkg/readers/sqldir"
"git.warky.dev/wdevs/relspecgo/pkg/writers"
"git.warky.dev/wdevs/relspecgo/pkg/writers/sqlexec"
)
func main() {
// Read SQL scripts from directory
reader := sqldir.NewReader(&readers.ReaderOptions{
FilePath: "./migrations",
})
db, err := reader.ReadDatabase()
if err != nil {
log.Fatal(err)
}
// Execute scripts against PostgreSQL
writer := sqlexec.NewWriter(&writers.WriterOptions{
Metadata: map[string]any{
"connection_string": "postgres://localhost/myapp",
},
})
if err := writer.WriteDatabase(db); err != nil {
log.Fatal(err)
}
}
```
## Configuration
### Required Metadata
- **connection_string**: PostgreSQL connection string (required)
### Connection String Format
```
postgres://[user[:password]@][host][:port][/dbname][?param1=value1&...]
```
Examples:
```
postgres://localhost/mydb
postgres://user:pass@localhost:5432/mydb?sslmode=disable
postgres://user@localhost/mydb?sslmode=require
postgresql://user:pass@prod-db.example.com:5432/production
```
## Execution Order
Scripts are sorted and executed based on:
1. **Priority** (ascending): Lower priority values execute first
2. **Sequence** (ascending): Within same priority, lower sequence values execute first
### Example Execution Order
Given these scripts:
```
Script A: Priority=2, Sequence=1
Script B: Priority=1, Sequence=3
Script C: Priority=1, Sequence=1
Script D: Priority=1, Sequence=2
Script E: Priority=3, Sequence=1
```
Execution order: **C → D → B → A → E**
## Output
The writer prints progress to stdout:
```
Executing script: create_users (Priority=1, Sequence=1)
✓ Successfully executed: create_users
Executing script: create_posts (Priority=1, Sequence=2)
✓ Successfully executed: create_posts
Executing script: add_indexes (Priority=2, Sequence=1)
✓ Successfully executed: add_indexes
```
## Error Handling
### Connection Errors
If the database connection fails, execution stops immediately:
```
Error: failed to connect to database: connection refused
```
### Script Execution Errors
If a script fails, execution stops and returns the error with context:
```
Error: failed to execute script add_indexes (Priority=2, Sequence=1):
syntax error at or near "IDNEX"
```
**Behavior**: Stop on first error (scripts executed before the error remain committed)
### Empty Script Handling
Scripts with empty SQL content are skipped silently.
## Database Support
Currently supports:
- ✅ PostgreSQL (via pgx/v5)
Future support planned for:
- MySQL/MariaDB
- SQLite
- Generic SQL via database/sql
## Transaction Behavior
**Current**: Each script executes in its own implicit transaction (PostgreSQL default behavior)
**Future Enhancement**: Option to wrap all scripts in a single transaction for atomic execution with rollback on error.
## Performance Considerations
- Scripts execute sequentially (not in parallel)
- Each script creates a database round-trip
- For large migrations, consider:
- Combining related statements into fewer scripts
- Using PostgreSQL's COPY command for bulk data
- Running during low-traffic periods
## Testing
Run tests:
```bash
go test ./pkg/writers/sqlexec/
```
Current tests include:
- Validation and error handling
- Script sorting logic
- Configuration validation
### Integration Tests
For integration testing with a real database:
```bash
# Start PostgreSQL (example with Docker)
docker run -d --name postgres-test \
-e POSTGRES_PASSWORD=test \
-e POSTGRES_DB=testdb \
-p 5432:5432 \
postgres:16
# Run your integration tests
go test -tags=integration ./pkg/writers/sqlexec/
# Cleanup
docker stop postgres-test
docker rm postgres-test
```
## Limitations
- `WriteTable()` is not supported (returns error)
- Requires PostgreSQL connection (no offline mode)
- No built-in transaction wrapping (yet)
- No rollback script support (yet, though `models.Script.Rollback` field exists)
## Related
- **SQL Directory Reader**: `pkg/readers/sqldir/` - Read scripts from filesystem
- **Script Model**: `pkg/models/models.go` - Script structure definition
- **pgx Documentation**: https://github.com/jackc/pgx - PostgreSQL driver docs

View File

@@ -0,0 +1,125 @@
package sqlexec
import (
"context"
"fmt"
"sort"
"github.com/jackc/pgx/v5"
"git.warky.dev/wdevs/relspecgo/pkg/models"
"git.warky.dev/wdevs/relspecgo/pkg/writers"
)
// Writer implements the writers.Writer interface for executing SQL scripts
type Writer struct {
options *writers.WriterOptions
}
// NewWriter creates a new SQL executor writer
func NewWriter(options *writers.WriterOptions) *Writer {
return &Writer{
options: options,
}
}
// WriteDatabase executes all scripts from all schemas in the database
func (w *Writer) WriteDatabase(db *models.Database) error {
if db == nil {
return fmt.Errorf("database is nil")
}
// Get connection string from metadata
connString, ok := w.options.Metadata["connection_string"].(string)
if !ok || connString == "" {
return fmt.Errorf("connection_string is required in writer metadata")
}
// Connect to database
ctx := context.Background()
conn, err := pgx.Connect(ctx, connString)
if err != nil {
return fmt.Errorf("failed to connect to database: %w", err)
}
defer conn.Close(ctx)
// Execute scripts from all schemas
for _, schema := range db.Schemas {
if err := w.executeScripts(ctx, conn, schema.Scripts); err != nil {
return fmt.Errorf("failed to execute scripts from schema %s: %w", schema.Name, err)
}
}
return nil
}
// WriteSchema executes all scripts from a single schema
func (w *Writer) WriteSchema(schema *models.Schema) error {
if schema == nil {
return fmt.Errorf("schema is nil")
}
// Get connection string from metadata
connString, ok := w.options.Metadata["connection_string"].(string)
if !ok || connString == "" {
return fmt.Errorf("connection_string is required in writer metadata")
}
// Connect to database
ctx := context.Background()
conn, err := pgx.Connect(ctx, connString)
if err != nil {
return fmt.Errorf("failed to connect to database: %w", err)
}
defer conn.Close(ctx)
// Execute scripts
if err := w.executeScripts(ctx, conn, schema.Scripts); err != nil {
return fmt.Errorf("failed to execute scripts: %w", err)
}
return nil
}
// WriteTable is not applicable for SQL script execution
func (w *Writer) WriteTable(table *models.Table) error {
return fmt.Errorf("WriteTable is not supported for SQL script execution")
}
// executeScripts executes scripts in Priority then Sequence order
func (w *Writer) executeScripts(ctx context.Context, conn *pgx.Conn, scripts []*models.Script) error {
if len(scripts) == 0 {
return nil
}
// Sort scripts by Priority (ascending) then Sequence (ascending)
sortedScripts := make([]*models.Script, len(scripts))
copy(sortedScripts, scripts)
sort.Slice(sortedScripts, func(i, j int) bool {
if sortedScripts[i].Priority != sortedScripts[j].Priority {
return sortedScripts[i].Priority < sortedScripts[j].Priority
}
return sortedScripts[i].Sequence < sortedScripts[j].Sequence
})
// Execute each script in order
for _, script := range sortedScripts {
if script.SQL == "" {
continue
}
fmt.Printf("Executing script: %s (Priority=%d, Sequence=%d)\n",
script.Name, script.Priority, script.Sequence)
// Execute the SQL script
_, err := conn.Exec(ctx, script.SQL)
if err != nil {
return fmt.Errorf("failed to execute script %s (Priority=%d, Sequence=%d): %w",
script.Name, script.Priority, script.Sequence, err)
}
fmt.Printf("✓ Successfully executed: %s\n", script.Name)
}
return nil
}

View File

@@ -0,0 +1,201 @@
package sqlexec
import (
"testing"
"git.warky.dev/wdevs/relspecgo/pkg/models"
"git.warky.dev/wdevs/relspecgo/pkg/writers"
)
func TestNewWriter(t *testing.T) {
opts := &writers.WriterOptions{
Metadata: map[string]any{
"connection_string": "postgres://localhost/test",
},
}
writer := NewWriter(opts)
if writer == nil {
t.Fatal("Expected non-nil writer")
}
if writer.options != opts {
t.Error("Writer options not set correctly")
}
}
func TestWriter_WriteDatabase_NilDatabase(t *testing.T) {
writer := NewWriter(&writers.WriterOptions{
Metadata: map[string]any{
"connection_string": "postgres://localhost/test",
},
})
err := writer.WriteDatabase(nil)
if err == nil {
t.Error("Expected error for nil database, got nil")
}
}
func TestWriter_WriteDatabase_MissingConnectionString(t *testing.T) {
writer := NewWriter(&writers.WriterOptions{
Metadata: map[string]any{},
})
db := &models.Database{
Name: "test",
Schemas: []*models.Schema{
{
Name: "public",
Scripts: []*models.Script{
{Name: "test", SQL: "SELECT 1;"},
},
},
},
}
err := writer.WriteDatabase(db)
if err == nil {
t.Error("Expected error for missing connection_string, got nil")
}
}
func TestWriter_WriteSchema_NilSchema(t *testing.T) {
writer := NewWriter(&writers.WriterOptions{
Metadata: map[string]any{
"connection_string": "postgres://localhost/test",
},
})
err := writer.WriteSchema(nil)
if err == nil {
t.Error("Expected error for nil schema, got nil")
}
}
func TestWriter_WriteSchema_MissingConnectionString(t *testing.T) {
writer := NewWriter(&writers.WriterOptions{
Metadata: map[string]any{},
})
schema := &models.Schema{
Name: "public",
Scripts: []*models.Script{
{Name: "test", SQL: "SELECT 1;"},
},
}
err := writer.WriteSchema(schema)
if err == nil {
t.Error("Expected error for missing connection_string, got nil")
}
}
func TestWriter_WriteTable(t *testing.T) {
writer := NewWriter(&writers.WriterOptions{})
err := writer.WriteTable(&models.Table{})
if err == nil {
t.Error("Expected error for WriteTable (not supported), got nil")
}
}
// TestScriptSorting verifies that scripts are sorted correctly by Priority then Sequence
func TestScriptSorting(t *testing.T) {
scripts := []*models.Script{
{Name: "script1", Priority: 2, Sequence: 1, SQL: "SELECT 1;"},
{Name: "script2", Priority: 1, Sequence: 3, SQL: "SELECT 2;"},
{Name: "script3", Priority: 1, Sequence: 1, SQL: "SELECT 3;"},
{Name: "script4", Priority: 1, Sequence: 2, SQL: "SELECT 4;"},
{Name: "script5", Priority: 3, Sequence: 1, SQL: "SELECT 5;"},
{Name: "script6", Priority: 2, Sequence: 2, SQL: "SELECT 6;"},
}
// Create a copy and sort it using the same logic as executeScripts
sortedScripts := make([]*models.Script, len(scripts))
copy(sortedScripts, scripts)
// Use the same sorting logic from executeScripts
for i := 0; i < len(sortedScripts)-1; i++ {
for j := i + 1; j < len(sortedScripts); j++ {
if sortedScripts[i].Priority > sortedScripts[j].Priority ||
(sortedScripts[i].Priority == sortedScripts[j].Priority &&
sortedScripts[i].Sequence > sortedScripts[j].Sequence) {
sortedScripts[i], sortedScripts[j] = sortedScripts[j], sortedScripts[i]
}
}
}
// Expected order after sorting
expectedOrder := []string{
"script3", // Priority 1, Sequence 1
"script4", // Priority 1, Sequence 2
"script2", // Priority 1, Sequence 3
"script1", // Priority 2, Sequence 1
"script6", // Priority 2, Sequence 2
"script5", // Priority 3, Sequence 1
}
for i, expected := range expectedOrder {
if sortedScripts[i].Name != expected {
t.Errorf("Position %d: expected %s, got %s", i, expected, sortedScripts[i].Name)
}
}
// Verify priorities are ascending
for i := 0; i < len(sortedScripts)-1; i++ {
if sortedScripts[i].Priority > sortedScripts[i+1].Priority {
t.Errorf("Priority not ascending at position %d: %d > %d",
i, sortedScripts[i].Priority, sortedScripts[i+1].Priority)
}
// Within same priority, sequences should be ascending
if sortedScripts[i].Priority == sortedScripts[i+1].Priority &&
sortedScripts[i].Sequence > sortedScripts[i+1].Sequence {
t.Errorf("Sequence not ascending at position %d with same priority %d: %d > %d",
i, sortedScripts[i].Priority, sortedScripts[i].Sequence, sortedScripts[i+1].Sequence)
}
}
}
func TestWriter_WriteSchema_EmptyScripts(t *testing.T) {
// This test verifies that writing an empty script list doesn't cause errors
// even without a database connection (should return early)
writer := NewWriter(&writers.WriterOptions{
Metadata: map[string]any{
"connection_string": "postgres://invalid/test",
},
})
schema := &models.Schema{
Name: "public",
Scripts: []*models.Script{},
}
// Note: This will try to connect even with empty scripts
// In a real scenario, the executeScripts function returns early for empty scripts
// but the connection is made before that. This test documents the behavior.
err := writer.WriteSchema(schema)
// We expect a connection error since we're using an invalid connection string
if err == nil {
t.Error("Expected connection error, got nil")
}
}
// NOTE: Integration tests for actual database execution should be added separately
// Those tests would require:
// 1. A running PostgreSQL instance
// 2. Test database setup/teardown
// 3. Verification of actual script execution
// 4. Testing error handling during execution
// 5. Testing transaction behavior if added
//
// Example integration test structure:
// func TestWriter_Integration_ExecuteScripts(t *testing.T) {
// if testing.Short() {
// t.Skip("Skipping integration test")
// }
// // Setup test database
// // Create test scripts
// // Execute scripts
// // Verify results
// // Cleanup
// }