Added a scripts execution ability
This commit is contained in:
360
docs/SCRIPTS_COMMAND.md
Normal file
360
docs/SCRIPTS_COMMAND.md
Normal file
@@ -0,0 +1,360 @@
|
||||
# RelSpec Scripts Command
|
||||
|
||||
The `relspec scripts` command provides tools for managing and executing SQL migration scripts from a directory structure.
|
||||
|
||||
## Overview
|
||||
|
||||
The scripts command supports two main operations:
|
||||
- **list**: List SQL scripts from a directory in execution order
|
||||
- **execute**: Execute SQL scripts against a PostgreSQL database
|
||||
|
||||
Scripts are read from a directory (recursively) and executed in a deterministic order based on **Priority** (ascending) and **Sequence** (ascending).
|
||||
|
||||
## File Naming Convention
|
||||
|
||||
SQL scripts must follow this naming pattern (both separators are supported):
|
||||
|
||||
```
|
||||
{priority}_{sequence}_{name}.{sql|pgsql} (underscore format)
|
||||
{priority}-{sequence}-{name}.{sql|pgsql} (hyphen format)
|
||||
```
|
||||
|
||||
### Components
|
||||
|
||||
- **priority**: Integer (0-9999) - Execution priority level (lower executes first)
|
||||
- **sequence**: Integer (0-9999) - Order within priority level (lower executes first)
|
||||
- **separator**: Underscore `_` or hyphen `-` (both formats can be mixed)
|
||||
- **name**: Descriptive name (alphanumeric, underscores, hyphens)
|
||||
- **extension**: `.sql` or `.pgsql`
|
||||
|
||||
### Valid Examples
|
||||
|
||||
**Underscore format:**
|
||||
```
|
||||
1_001_create_users.sql # Priority 1, Sequence 1
|
||||
1_002_create_posts.sql # Priority 1, Sequence 2
|
||||
1_003_create_comments.pgsql # Priority 1, Sequence 3
|
||||
2_001_add_indexes.sql # Priority 2, Sequence 1
|
||||
2_002_add_constraints.sql # Priority 2, Sequence 2
|
||||
3_001_seed_users.sql # Priority 3, Sequence 1
|
||||
```
|
||||
|
||||
**Hyphen format:**
|
||||
```
|
||||
1-001-create-users.sql # Priority 1, Sequence 1
|
||||
1-002-create-posts.sql # Priority 1, Sequence 2
|
||||
1-003-create-comments.pgsql # Priority 1, Sequence 3
|
||||
10-10-create-newid.pgsql # Priority 10, Sequence 10
|
||||
```
|
||||
|
||||
**Mixed format (both in same directory):**
|
||||
```
|
||||
1_001_create_users.sql # Priority 1, Sequence 1 (underscore)
|
||||
1-002-create-posts.sql # Priority 1, Sequence 2 (hyphen)
|
||||
2_001_add_indexes.sql # Priority 2, Sequence 1 (underscore)
|
||||
```
|
||||
|
||||
**Execution Order**: 1→2→3→4→5→6 (sorted by Priority, then Sequence)
|
||||
|
||||
### Invalid Examples (Will be ignored)
|
||||
|
||||
```
|
||||
migration.sql # Missing priority/sequence
|
||||
create_users.sql # Missing priority/sequence
|
||||
1_create_users.sql # Missing sequence
|
||||
1_001_test.txt # Wrong extension
|
||||
README.md # Not a SQL file
|
||||
```
|
||||
|
||||
## Directory Structure
|
||||
|
||||
Scripts can be organized in subdirectories. The scanner recursively finds all matching SQL files:
|
||||
|
||||
```
|
||||
migrations/
|
||||
├── 1_001_create_schema.sql
|
||||
├── 1_002_create_users.sql
|
||||
├── tables/
|
||||
│ ├── 1_003_create_posts.sql
|
||||
│ └── 1_004_create_comments.pgsql
|
||||
├── indexes/
|
||||
│ └── 2_001_add_indexes.sql
|
||||
└── data/
|
||||
└── 3_001_seed_data.sql
|
||||
```
|
||||
|
||||
All files will be found and executed in Priority→Sequence order regardless of directory structure.
|
||||
|
||||
## Commands
|
||||
|
||||
### relspec scripts list
|
||||
|
||||
List all SQL scripts in a directory and show their execution order.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
relspec scripts list --dir <directory> [flags]
|
||||
```
|
||||
|
||||
**Flags:**
|
||||
- `--dir <path>` (required): Directory containing SQL scripts
|
||||
- `--schema <name>`: Schema name (default: "public")
|
||||
- `--database <name>`: Database name (default: "database")
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
relspec scripts list --dir ./migrations
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```
|
||||
=== SQL Scripts List ===
|
||||
Directory: ./migrations
|
||||
|
||||
Found 5 script(s) in execution order:
|
||||
|
||||
No. Priority Sequence Name Lines
|
||||
---- -------- -------- ------------------------------ -----
|
||||
1 1 1 create_users 7
|
||||
2 1 2 create_posts 8
|
||||
3 2 1 add_indexes 4
|
||||
4 2 2 add_constraints 6
|
||||
5 3 1 seed_data 4
|
||||
```
|
||||
|
||||
### relspec scripts execute
|
||||
|
||||
Execute SQL scripts from a directory against a PostgreSQL database.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
relspec scripts execute --dir <directory> --conn <connection-string> [flags]
|
||||
```
|
||||
|
||||
**Flags:**
|
||||
- `--dir <path>` (required): Directory containing SQL scripts
|
||||
- `--conn <string>` (required): PostgreSQL connection string
|
||||
- `--schema <name>`: Schema name (default: "public")
|
||||
- `--database <name>`: Database name (default: "database")
|
||||
|
||||
**Connection String Formats:**
|
||||
|
||||
```bash
|
||||
# Standard PostgreSQL URLs
|
||||
postgres://username:password@localhost:5432/database_name
|
||||
postgres://username:password@localhost/database_name
|
||||
postgresql://user:pass@host:5432/dbname?sslmode=disable
|
||||
postgresql://user:pass@host/dbname?sslmode=require
|
||||
|
||||
# Key-value format
|
||||
host=localhost port=5432 user=username password=pass dbname=mydb sslmode=disable
|
||||
```
|
||||
|
||||
**Examples:**
|
||||
|
||||
```bash
|
||||
# Execute migration scripts
|
||||
relspec scripts execute \
|
||||
--dir ./migrations \
|
||||
--conn "postgres://user:pass@localhost:5432/mydb"
|
||||
|
||||
# Execute with custom schema
|
||||
relspec scripts execute \
|
||||
--dir ./migrations \
|
||||
--conn "postgres://localhost/mydb" \
|
||||
--schema public
|
||||
|
||||
# Execute with SSL disabled
|
||||
relspec scripts execute \
|
||||
--dir ./sql \
|
||||
--conn "postgres://user:pass@localhost/db?sslmode=disable"
|
||||
|
||||
# Execute using key-value connection string
|
||||
relspec scripts execute \
|
||||
--dir ./migrations \
|
||||
--conn "host=localhost port=5432 user=admin password=secret dbname=prod"
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```
|
||||
=== SQL Scripts Execution ===
|
||||
Started at: 2025-12-30 22:30:15
|
||||
Directory: ./migrations
|
||||
Database: postgres://user:***@localhost:5432/mydb
|
||||
|
||||
[1/2] Reading SQL scripts...
|
||||
✓ Found 4 script(s)
|
||||
|
||||
[2/2] Executing scripts in order (Priority → Sequence)...
|
||||
|
||||
Executing script: create_users (Priority=1, Sequence=1)
|
||||
✓ Successfully executed: create_users
|
||||
Executing script: create_posts (Priority=1, Sequence=2)
|
||||
✓ Successfully executed: create_posts
|
||||
Executing script: add_indexes (Priority=2, Sequence=1)
|
||||
✓ Successfully executed: add_indexes
|
||||
Executing script: seed_data (Priority=2, Sequence=2)
|
||||
✓ Successfully executed: seed_data
|
||||
|
||||
=== Execution Complete ===
|
||||
Completed at: 2025-12-30 22:30:16
|
||||
Successfully executed 4 script(s)
|
||||
```
|
||||
|
||||
## Execution Behavior
|
||||
|
||||
### Execution Order
|
||||
|
||||
Scripts are **always** executed in this order:
|
||||
1. Sort by **Priority** (ascending)
|
||||
2. Within same priority, sort by **Sequence** (ascending)
|
||||
|
||||
Example:
|
||||
```
|
||||
Priority 1, Sequence 1 → Executes 1st
|
||||
Priority 1, Sequence 2 → Executes 2nd
|
||||
Priority 1, Sequence 10 → Executes 3rd
|
||||
Priority 2, Sequence 1 → Executes 4th
|
||||
Priority 2, Sequence 5 → Executes 5th
|
||||
Priority 10, Sequence 1 → Executes 6th
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
|
||||
- **Stop on First Error**: Execution stops immediately when any script fails
|
||||
- **No Automatic Rollback**: Scripts executed before the failure remain committed
|
||||
- **Error Details**: Full error message with script name, priority, and sequence
|
||||
|
||||
Example error output:
|
||||
```
|
||||
Executing script: add_indexes (Priority=2, Sequence=1)
|
||||
Error: execution failed: failed to execute script add_indexes (Priority=2, Sequence=1):
|
||||
ERROR: syntax error at or near "IDNEX" (SQLSTATE 42601)
|
||||
```
|
||||
|
||||
### Transaction Behavior
|
||||
|
||||
- Each script executes in its own implicit transaction (PostgreSQL default)
|
||||
- No automatic transaction wrapping across multiple scripts
|
||||
- For atomic migrations, manually wrap SQL in `BEGIN/COMMIT` blocks
|
||||
|
||||
### Empty Scripts
|
||||
|
||||
Scripts with empty SQL content are silently skipped.
|
||||
|
||||
## Use Cases
|
||||
|
||||
### Development Migrations
|
||||
|
||||
Organize database changes by priority levels:
|
||||
|
||||
```
|
||||
migrations/
|
||||
├── 1_xxx_schema.sql # Priority 1: Core schema
|
||||
├── 1_xxx_tables.sql
|
||||
├── 2_xxx_indexes.sql # Priority 2: Performance
|
||||
├── 2_xxx_constraints.sql
|
||||
└── 3_xxx_seed.sql # Priority 3: Data
|
||||
```
|
||||
|
||||
### Multi-Environment Deployments
|
||||
|
||||
Use priority levels for environment-specific scripts:
|
||||
|
||||
```
|
||||
deploy/
|
||||
├── 1_xxx_core_schema.sql # Priority 1: All environments
|
||||
├── 2_xxx_dev_data.sql # Priority 2: Dev only
|
||||
├── 2_xxx_staging_data.sql # Priority 2: Staging only
|
||||
└── 3_xxx_prod_data.sql # Priority 3: Production only
|
||||
```
|
||||
|
||||
### Incremental Rollouts
|
||||
|
||||
Use sequence for ordered feature rollouts:
|
||||
|
||||
```
|
||||
features/
|
||||
├── 1_001_feature_a_schema.sql
|
||||
├── 1_002_feature_a_data.sql
|
||||
├── 1_003_feature_b_schema.sql
|
||||
├── 1_004_feature_b_data.sql
|
||||
```
|
||||
|
||||
## Integration with RelSpec
|
||||
|
||||
The scripts command uses:
|
||||
- **Reader**: `pkg/readers/sqldir/` - Reads SQL files into `models.Schema.Scripts`
|
||||
- **Writer**: `pkg/writers/sqlexec/` - Executes scripts from `models.Schema.Scripts`
|
||||
|
||||
You can use these packages programmatically:
|
||||
|
||||
```go
|
||||
import (
|
||||
"git.warky.dev/wdevs/relspecgo/pkg/readers"
|
||||
"git.warky.dev/wdevs/relspecgo/pkg/readers/sqldir"
|
||||
"git.warky.dev/wdevs/relspecgo/pkg/writers"
|
||||
"git.warky.dev/wdevs/relspecgo/pkg/writers/sqlexec"
|
||||
)
|
||||
|
||||
// Read scripts
|
||||
reader := sqldir.NewReader(&readers.ReaderOptions{
|
||||
FilePath: "./migrations",
|
||||
})
|
||||
db, _ := reader.ReadDatabase()
|
||||
|
||||
// Execute scripts
|
||||
writer := sqlexec.NewWriter(&writers.WriterOptions{
|
||||
Metadata: map[string]any{
|
||||
"connection_string": "postgres://localhost/mydb",
|
||||
},
|
||||
})
|
||||
writer.WriteDatabase(db)
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Naming
|
||||
|
||||
- Use zero-padded sequences: `001`, `002`, `010` (not `1`, `2`, `10`)
|
||||
- Use descriptive names: `create_users_table`, not `table1`
|
||||
- Group related changes: same priority for related DDL
|
||||
|
||||
### Organization
|
||||
|
||||
- Keep scripts small and focused (one logical change per file)
|
||||
- Use priority levels to organize phases (schema → indexes → data)
|
||||
- Document complex migrations with SQL comments
|
||||
|
||||
### Safety
|
||||
|
||||
- Always test migrations in development first
|
||||
- Use `scripts list` to verify execution order before running
|
||||
- Back up production databases before executing
|
||||
- Consider using transactions for critical changes
|
||||
- Review generated SQL before execution
|
||||
|
||||
### Version Control
|
||||
|
||||
- Commit scripts to version control
|
||||
- Never modify executed scripts (create new ones instead)
|
||||
- Use meaningful commit messages
|
||||
- Tag releases with migration checkpoints
|
||||
|
||||
## Limitations
|
||||
|
||||
- PostgreSQL only (currently)
|
||||
- No built-in rollback support
|
||||
- No migration state tracking (no "already executed" detection)
|
||||
- No dry-run mode
|
||||
- Stops on first error (no partial execution tracking)
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
Potential future features:
|
||||
- Migration state tracking (executed scripts table)
|
||||
- Rollback script support (using `models.Script.Rollback` field)
|
||||
- Dry-run mode (validate without executing)
|
||||
- Transaction wrapping (all-or-nothing execution)
|
||||
- Multi-database support (MySQL, SQLite, etc.)
|
||||
- Parallel execution for independent scripts
|
||||
Reference in New Issue
Block a user