Added a scripts execution ability

2025-12-31 00:44:14 +02:00
parent 1d193c84d7
commit adfe126758
11 changed files with 2402 additions and 0 deletions
--- a/docs/SCRIPTS_COMMAND.md
+++ b/docs/SCRIPTS_COMMAND.md
@@ -0,0 +1,360 @@
+# RelSpec Scripts Command
+
+The `relspec scripts` command provides tools for managing and executing SQL migration scripts from a directory structure.
+
+## Overview
+
+The scripts command supports two main operations:
+- **list**: List SQL scripts from a directory in execution order
+- **execute**: Execute SQL scripts against a PostgreSQL database
+
+Scripts are read from a directory (recursively) and executed in a deterministic order based on **Priority** (ascending) and **Sequence** (ascending).
+
+## File Naming Convention
+
+SQL scripts must follow this naming pattern (both separators are supported):
+
+```
+{priority}_{sequence}_{name}.{sql|pgsql}  (underscore format)
+{priority}-{sequence}-{name}.{sql|pgsql}  (hyphen format)
+```
+
+### Components
+
+- **priority**: Integer (0-9999) - Execution priority level (lower executes first)
+- **sequence**: Integer (0-9999) - Order within priority level (lower executes first)
+- **separator**: Underscore `_` or hyphen `-` (both formats can be mixed)
+- **name**: Descriptive name (alphanumeric, underscores, hyphens)
+- **extension**: `.sql` or `.pgsql`
+
+### Valid Examples
+
+**Underscore format:**
+```
+1_001_create_users.sql          # Priority 1, Sequence 1
+1_002_create_posts.sql          # Priority 1, Sequence 2
+1_003_create_comments.pgsql     # Priority 1, Sequence 3
+2_001_add_indexes.sql           # Priority 2, Sequence 1
+2_002_add_constraints.sql       # Priority 2, Sequence 2
+3_001_seed_users.sql            # Priority 3, Sequence 1
+```
+
+**Hyphen format:**
+```
+1-001-create-users.sql          # Priority 1, Sequence 1
+1-002-create-posts.sql          # Priority 1, Sequence 2
+1-003-create-comments.pgsql     # Priority 1, Sequence 3
+10-10-create-newid.pgsql        # Priority 10, Sequence 10
+```
+
+**Mixed format (both in same directory):**
+```
+1_001_create_users.sql          # Priority 1, Sequence 1 (underscore)
+1-002-create-posts.sql          # Priority 1, Sequence 2 (hyphen)
+2_001_add_indexes.sql           # Priority 2, Sequence 1 (underscore)
+```
+
+**Execution Order**: 1→2→3→4→5→6 (sorted by Priority, then Sequence)
+
+### Invalid Examples (Will be ignored)
+
+```
+migration.sql                   # Missing priority/sequence
+create_users.sql                # Missing priority/sequence
+1_create_users.sql              # Missing sequence
+1_001_test.txt                  # Wrong extension
+README.md                       # Not a SQL file
+```
+
+## Directory Structure
+
+Scripts can be organized in subdirectories. The scanner recursively finds all matching SQL files:
+
+```
+migrations/
+├── 1_001_create_schema.sql
+├── 1_002_create_users.sql
+├── tables/
+│   ├── 1_003_create_posts.sql
+│   └── 1_004_create_comments.pgsql
+├── indexes/
+│   └── 2_001_add_indexes.sql
+└── data/
+    └── 3_001_seed_data.sql
+```
+
+All files will be found and executed in Priority→Sequence order regardless of directory structure.
+
+## Commands
+
+### relspec scripts list
+
+List all SQL scripts in a directory and show their execution order.
+
+**Usage:**
+```bash
+relspec scripts list --dir <directory> [flags]
+```
+
+**Flags:**
+- `--dir <path>` (required): Directory containing SQL scripts
+- `--schema <name>`: Schema name (default: "public")
+- `--database <name>`: Database name (default: "database")
+
+**Example:**
+```bash
+relspec scripts list --dir ./migrations
+```
+
+**Output:**
+```
+=== SQL Scripts List ===
+Directory: ./migrations
+
+Found 5 script(s) in execution order:
+
+No.  Priority   Sequence Name                           Lines
+---- --------   -------- ------------------------------ -----
+1    1          1        create_users                   7
+2    1          2        create_posts                   8
+3    2          1        add_indexes                    4
+4    2          2        add_constraints                6
+5    3          1        seed_data                      4
+```
+
+### relspec scripts execute
+
+Execute SQL scripts from a directory against a PostgreSQL database.
+
+**Usage:**
+```bash
+relspec scripts execute --dir <directory> --conn <connection-string> [flags]
+```
+
+**Flags:**
+- `--dir <path>` (required): Directory containing SQL scripts
+- `--conn <string>` (required): PostgreSQL connection string
+- `--schema <name>`: Schema name (default: "public")
+- `--database <name>`: Database name (default: "database")
+
+**Connection String Formats:**
+
+```bash
+# Standard PostgreSQL URLs
+postgres://username:password@localhost:5432/database_name
+postgres://username:password@localhost/database_name
+postgresql://user:pass@host:5432/dbname?sslmode=disable
+postgresql://user:pass@host/dbname?sslmode=require
+
+# Key-value format
+host=localhost port=5432 user=username password=pass dbname=mydb sslmode=disable
+```
+
+**Examples:**
+
+```bash
+# Execute migration scripts
+relspec scripts execute \
+  --dir ./migrations \
+  --conn "postgres://user:pass@localhost:5432/mydb"
+
+# Execute with custom schema
+relspec scripts execute \
+  --dir ./migrations \
+  --conn "postgres://localhost/mydb" \
+  --schema public
+
+# Execute with SSL disabled
+relspec scripts execute \
+  --dir ./sql \
+  --conn "postgres://user:pass@localhost/db?sslmode=disable"
+
+# Execute using key-value connection string
+relspec scripts execute \
+  --dir ./migrations \
+  --conn "host=localhost port=5432 user=admin password=secret dbname=prod"
+```
+
+**Output:**
+```
+=== SQL Scripts Execution ===
+Started at: 2025-12-30 22:30:15
+Directory: ./migrations
+Database:  postgres://user:***@localhost:5432/mydb
+
+[1/2] Reading SQL scripts...
+      ✓ Found 4 script(s)
+
+[2/2] Executing scripts in order (Priority → Sequence)...
+
+Executing script: create_users (Priority=1, Sequence=1)
+✓ Successfully executed: create_users
+Executing script: create_posts (Priority=1, Sequence=2)
+✓ Successfully executed: create_posts
+Executing script: add_indexes (Priority=2, Sequence=1)
+✓ Successfully executed: add_indexes
+Executing script: seed_data (Priority=2, Sequence=2)
+✓ Successfully executed: seed_data
+
+=== Execution Complete ===
+Completed at: 2025-12-30 22:30:16
+Successfully executed 4 script(s)
+```
+
+## Execution Behavior
+
+### Execution Order
+
+Scripts are **always** executed in this order:
+1. Sort by **Priority** (ascending)
+2. Within same priority, sort by **Sequence** (ascending)
+
+Example:
+```
+Priority 1, Sequence 1   → Executes 1st
+Priority 1, Sequence 2   → Executes 2nd
+Priority 1, Sequence 10  → Executes 3rd
+Priority 2, Sequence 1   → Executes 4th
+Priority 2, Sequence 5   → Executes 5th
+Priority 10, Sequence 1  → Executes 6th
+```
+
+### Error Handling
+
+- **Stop on First Error**: Execution stops immediately when any script fails
+- **No Automatic Rollback**: Scripts executed before the failure remain committed
+- **Error Details**: Full error message with script name, priority, and sequence
+
+Example error output:
+```
+Executing script: add_indexes (Priority=2, Sequence=1)
+Error: execution failed: failed to execute script add_indexes (Priority=2, Sequence=1):
+       ERROR: syntax error at or near "IDNEX" (SQLSTATE 42601)
+```
+
+### Transaction Behavior
+
+- Each script executes in its own implicit transaction (PostgreSQL default)
+- No automatic transaction wrapping across multiple scripts
+- For atomic migrations, manually wrap SQL in `BEGIN/COMMIT` blocks
+
+### Empty Scripts
+
+Scripts with empty SQL content are silently skipped.
+
+## Use Cases
+
+### Development Migrations
+
+Organize database changes by priority levels:
+
+```
+migrations/
+├── 1_xxx_schema.sql         # Priority 1: Core schema
+├── 1_xxx_tables.sql
+├── 2_xxx_indexes.sql        # Priority 2: Performance
+├── 2_xxx_constraints.sql
+└── 3_xxx_seed.sql           # Priority 3: Data
+```
+
+### Multi-Environment Deployments
+
+Use priority levels for environment-specific scripts:
+
+```
+deploy/
+├── 1_xxx_core_schema.sql    # Priority 1: All environments
+├── 2_xxx_dev_data.sql       # Priority 2: Dev only
+├── 2_xxx_staging_data.sql   # Priority 2: Staging only
+└── 3_xxx_prod_data.sql      # Priority 3: Production only
+```
+
+### Incremental Rollouts
+
+Use sequence for ordered feature rollouts:
+
+```
+features/
+├── 1_001_feature_a_schema.sql
+├── 1_002_feature_a_data.sql
+├── 1_003_feature_b_schema.sql
+├── 1_004_feature_b_data.sql
+```
+
+## Integration with RelSpec
+
+The scripts command uses:
+- **Reader**: `pkg/readers/sqldir/` - Reads SQL files into `models.Schema.Scripts`
+- **Writer**: `pkg/writers/sqlexec/` - Executes scripts from `models.Schema.Scripts`
+
+You can use these packages programmatically:
+
+```go
+import (
+    "git.warky.dev/wdevs/relspecgo/pkg/readers"
+    "git.warky.dev/wdevs/relspecgo/pkg/readers/sqldir"
+    "git.warky.dev/wdevs/relspecgo/pkg/writers"
+    "git.warky.dev/wdevs/relspecgo/pkg/writers/sqlexec"
+)
+
+// Read scripts
+reader := sqldir.NewReader(&readers.ReaderOptions{
+    FilePath: "./migrations",
+})
+db, _ := reader.ReadDatabase()
+
+// Execute scripts
+writer := sqlexec.NewWriter(&writers.WriterOptions{
+    Metadata: map[string]any{
+        "connection_string": "postgres://localhost/mydb",
+    },
+})
+writer.WriteDatabase(db)
+```
+
+## Best Practices
+
+### Naming
+
+- Use zero-padded sequences: `001`, `002`, `010` (not `1`, `2`, `10`)
+- Use descriptive names: `create_users_table`, not `table1`
+- Group related changes: same priority for related DDL
+
+### Organization
+
+- Keep scripts small and focused (one logical change per file)
+- Use priority levels to organize phases (schema → indexes → data)
+- Document complex migrations with SQL comments
+
+### Safety
+
+- Always test migrations in development first
+- Use `scripts list` to verify execution order before running
+- Back up production databases before executing
+- Consider using transactions for critical changes
+- Review generated SQL before execution
+
+### Version Control
+
+- Commit scripts to version control
+- Never modify executed scripts (create new ones instead)
+- Use meaningful commit messages
+- Tag releases with migration checkpoints
+
+## Limitations
+
+- PostgreSQL only (currently)
+- No built-in rollback support
+- No migration state tracking (no "already executed" detection)
+- No dry-run mode
+- Stops on first error (no partial execution tracking)
+
+## Future Enhancements
+
+Potential future features:
+- Migration state tracking (executed scripts table)
+- Rollback script support (using `models.Script.Rollback` field)
+- Dry-run mode (validate without executing)
+- Transaction wrapping (all-or-nothing execution)
+- Multi-database support (MySQL, SQLite, etc.)
+- Parallel execution for independent scripts
--- a/docs/SCRIPTS_EXAMPLES.md
+++ b/docs/SCRIPTS_EXAMPLES.md
@@ -0,0 +1,393 @@
+# RelSpec Scripts Command - Quick Examples
+
+## Basic Workflow
+
+### 1. Create migration directory structure
+
+```bash
+mkdir -p migrations
+```
+
+### 2. Create migration scripts
+
+Both underscore and hyphen formats are supported. Examples below use underscore format,
+but you can also use: `1-001-create-users-table.sql`
+
+```bash
+# Priority 1: Core schema
+cat > migrations/1_001_create_users_table.sql << 'EOF'
+CREATE TABLE users (
+    id SERIAL PRIMARY KEY,
+    username VARCHAR(100) NOT NULL UNIQUE,
+    email VARCHAR(255) NOT NULL UNIQUE,
+    password_hash VARCHAR(255) NOT NULL,
+    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+);
+
+CREATE INDEX idx_users_username ON users(username);
+CREATE INDEX idx_users_email ON users(email);
+EOF
+
+cat > migrations/1_002_create_posts_table.sql << 'EOF'
+CREATE TABLE posts (
+    id SERIAL PRIMARY KEY,
+    user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+    title VARCHAR(200) NOT NULL,
+    content TEXT,
+    published BOOLEAN DEFAULT FALSE,
+    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+);
+EOF
+
+# Priority 2: Additional indexes
+cat > migrations/2_001_add_post_indexes.sql << 'EOF'
+CREATE INDEX idx_posts_user_id ON posts(user_id);
+CREATE INDEX idx_posts_published ON posts(published);
+CREATE INDEX idx_posts_created_at ON posts(created_at);
+EOF
+
+# Priority 3: Seed data
+cat > migrations/3_001_seed_admin_user.sql << 'EOF'
+INSERT INTO users (username, email, password_hash)
+VALUES ('admin', 'admin@example.com', 'hashed_password_here')
+ON CONFLICT (username) DO NOTHING;
+EOF
+```
+
+### 3. List scripts to verify order
+
+```bash
+relspec scripts list --dir migrations
+```
+
+Output:
+```
+=== SQL Scripts List ===
+Directory: migrations
+
+Found 4 script(s) in execution order:
+
+No.  Priority   Sequence Name                           Lines
+---- --------   -------- ------------------------------ -----
+1    1          1        create_users_table             13
+2    1          2        create_posts_table             11
+3    2          1        add_post_indexes               4
+4    3          1        seed_admin_user                4
+```
+
+### 4. Execute against database
+
+```bash
+relspec scripts execute \
+  --dir migrations \
+  --conn "postgres://myuser:mypass@localhost:5432/myapp"
+```
+
+## Real-World Examples
+
+### Example 1: E-commerce Database Setup
+
+```bash
+# Directory structure
+migrations/
+├── 1_001_create_users.sql
+├── 1_002_create_products.sql
+├── 1_003_create_orders.sql
+├── 1_004_create_order_items.sql
+├── 2_001_add_indexes.sql
+├── 2_002_add_constraints.sql
+├── 3_001_seed_categories.sql
+└── 3_002_seed_sample_products.sql
+
+# Execute
+relspec scripts execute \
+  --dir migrations \
+  --conn "postgres://ecommerce_user:pass@db.example.com:5432/ecommerce_prod?sslmode=require"
+```
+
+### Example 2: Multi-Schema Database
+
+```bash
+# Organize by schema using subdirectories
+migrations/
+├── public/
+│   ├── 1_001_create_users.sql
+│   └── 1_002_create_sessions.sql
+├── analytics/
+│   ├── 1_001_create_events.sql
+│   └── 2_001_create_views.sql
+└── reporting/
+    └── 1_001_create_reports.sql
+
+# Execute (all schemas processed together)
+relspec scripts execute \
+  --dir migrations \
+  --conn "postgres://localhost/multi_schema_db" \
+  --schema public
+```
+
+### Example 3: Development Environment Setup
+
+```bash
+# Create local development database
+createdb myapp_dev
+
+# Run migrations
+relspec scripts execute \
+  --dir ./db/migrations \
+  --conn "postgres://localhost/myapp_dev?sslmode=disable"
+
+# Verify
+psql myapp_dev -c "\dt"
+```
+
+### Example 4: CI/CD Pipeline
+
+```yaml
+# .github/workflows/deploy.yml
+- name: Run database migrations
+  run: |
+    relspec scripts list --dir migrations
+    relspec scripts execute \
+      --dir migrations \
+      --conn "${{ secrets.DATABASE_URL }}"
+```
+
+### Example 5: Docker Compose Integration
+
+```yaml
+# docker-compose.yml
+services:
+  postgres:
+    image: postgres:16
+    environment:
+      POSTGRES_DB: myapp
+      POSTGRES_USER: myuser
+      POSTGRES_PASSWORD: mypass
+    ports:
+      - "5432:5432"
+
+  migrate:
+    image: relspec:latest
+    depends_on:
+      - postgres
+    volumes:
+      - ./migrations:/migrations
+    command: >
+      scripts execute
+      --dir /migrations
+      --conn "postgres://myuser:mypass@postgres:5432/myapp"
+```
+
+```bash
+# Run migrations with docker-compose
+docker-compose up -d postgres
+sleep 5  # Wait for postgres to be ready
+docker-compose run --rm migrate
+```
+
+### Example 6: Incremental Feature Rollout
+
+```bash
+# Feature branch structure
+migrations/
+├── 1_100_user_profiles_schema.sql      # Feature: User profiles
+├── 1_101_user_profiles_constraints.sql
+├── 1_102_user_profiles_indexes.sql
+├── 2_100_notifications_schema.sql      # Feature: Notifications
+├── 2_101_notifications_constraints.sql
+└── 2_102_notifications_indexes.sql
+
+# Deploy just user profiles (Priority 1)
+# Then later deploy notifications (Priority 2)
+```
+
+### Example 7: Rollback Strategy (Manual)
+
+```bash
+# Forward migration
+cat > migrations/1_001_add_column.sql << 'EOF'
+ALTER TABLE users ADD COLUMN phone VARCHAR(20);
+EOF
+
+# Create manual rollback script (not auto-executed)
+cat > rollbacks/1_001_remove_column.sql << 'EOF'
+ALTER TABLE users DROP COLUMN phone;
+EOF
+
+# If needed, manually execute rollback
+psql myapp -f rollbacks/1_001_remove_column.sql
+```
+
+### Example 8: Complex Schema Changes
+
+```bash
+# migrations/1_001_alter_users_table.sql
+BEGIN;
+
+-- Add new column
+ALTER TABLE users ADD COLUMN full_name VARCHAR(200);
+
+-- Populate from existing data
+UPDATE users SET full_name = username WHERE full_name IS NULL;
+
+-- Make it required
+ALTER TABLE users ALTER COLUMN full_name SET NOT NULL;
+
+-- Add index
+CREATE INDEX idx_users_full_name ON users(full_name);
+
+COMMIT;
+```
+
+Execute:
+```bash
+relspec scripts execute \
+  --dir migrations \
+  --conn "postgres://localhost/myapp"
+```
+
+## File Naming Format Examples
+
+### Underscore Format (Traditional)
+```
+migrations/
+├── 1_001_create_users.sql
+├── 1_002_create_posts.sql
+├── 2_001_add_indexes.sql
+└── 3_001_seed_data.sql
+```
+
+### Hyphen Format (Alternative)
+```
+migrations/
+├── 1-001-create-users.sql
+├── 1-002-create-posts.sql
+├── 10-10-create-newid.pgsql
+└── 2-001-add-indexes.sql
+```
+
+### Mixed Format (Both in Same Directory)
+```
+migrations/
+├── 1_001_create_users.sql          # Underscore format
+├── 1-002-create-posts.sql          # Hyphen format
+├── 2_001_add_indexes.sql           # Underscore format
+└── 10-10-special-migration.pgsql   # Hyphen format
+```
+
+**Note:** All three approaches work identically - use whichever naming style you prefer!
+
+## Common Patterns
+
+### Pattern 1: Schema → Indexes → Constraints → Data
+
+```
+1_xxx_*.sql   # Tables and basic structure
+2_xxx_*.sql   # Indexes for performance
+3_xxx_*.sql   # Foreign keys and constraints
+4_xxx_*.sql   # Seed/reference data
+```
+
+### Pattern 2: Feature-Based Organization
+
+```
+1_001_feature_auth_users.sql
+1_002_feature_auth_sessions.sql
+1_003_feature_auth_permissions.sql
+2_001_feature_blog_posts.sql
+2_002_feature_blog_comments.sql
+3_001_feature_payments_transactions.sql
+```
+
+### Pattern 3: Date-Based Versioning
+
+```
+1_20250130_create_users.sql
+2_20250131_add_user_indexes.sql
+3_20250201_create_posts.sql
+```
+
+### Pattern 4: Environment-Specific Scripts
+
+```bash
+# Base migrations (all environments)
+migrations/base/
+├── 1_001_create_users.sql
+├── 1_002_create_products.sql
+
+# Development-specific
+migrations/dev/
+└── 9_001_seed_test_data.sql
+
+# Production-specific
+migrations/prod/
+└── 9_001_seed_production_config.sql
+
+# Execute different paths based on environment
+ENV=dev
+relspec scripts execute \
+  --dir migrations/base \
+  --conn "postgres://localhost/myapp_${ENV}"
+
+relspec scripts execute \
+  --dir migrations/${ENV} \
+  --conn "postgres://localhost/myapp_${ENV}"
+```
+
+## Troubleshooting
+
+### Check script order before execution
+```bash
+relspec scripts list --dir migrations
+```
+
+### Test against local database first
+```bash
+# Create test database
+createdb myapp_test
+
+# Test migrations
+relspec scripts execute \
+  --dir migrations \
+  --conn "postgres://localhost/myapp_test"
+
+# Inspect results
+psql myapp_test
+
+# Cleanup
+dropdb myapp_test
+```
+
+### Validate SQL syntax
+```bash
+# Use PostgreSQL to check syntax without executing
+for f in migrations/*.sql; do
+  echo "Checking $f..."
+  psql myapp -c "BEGIN; \i $f; ROLLBACK;" --single-transaction
+done
+```
+
+### Debug connection issues
+```bash
+# Test connection string
+psql "postgres://user:pass@localhost:5432/myapp"
+
+# If that works, use the same string for relspec
+relspec scripts execute \
+  --dir migrations \
+  --conn "postgres://user:pass@localhost:5432/myapp"
+```
+
+## Tips
+
+1. **Always review execution order** with `list` before running `execute`
+2. **Test in development** before running against production
+3. **Use zero-padded sequences** (001, 002, not 1, 2) for consistent sorting
+4. **Keep scripts idempotent** when possible (use IF NOT EXISTS, ON CONFLICT, etc.)
+5. **Back up production** before running migrations
+6. **Use transactions** for complex multi-statement migrations
+7. **Document breaking changes** with SQL comments in the migration files
+8. **Version control everything** - commit migrations with code changes