relspecgo/docs/JOB_FILES.md

# RelSpec Job Files

Job files let you declare named, repeatable RelSpec workflows in YAML and run
them with `relspec job run <name>` instead of retyping long command lines.

```bash
relspec job list                     # deterministic list of discovered jobs
relspec job run build-schema --plan  # validate + print plan, execute nothing
relspec job run build-schema         # run the job (and its dependencies)
```

## Design contract (first release)

This is the smallest coherent contract that is safe and useful end to end.
Anything not listed under "Supported" is intentionally deferred.

### Not a shell

`command` is a **closed allow-list**. There is no field anywhere that accepts a
shell string, an executable path, or arbitrary arguments. Adding a new command
means adding a vetted adapter in the RelSpec source.

| command        | what it does                                                        |
|----------------|--------------------------------------------------------------------|
| `convert`      | read one or more input schemas, additively merge them, write one output |
| `merge`        | like `convert` but requires ≥2 inputs and exposes `skip_*` merge options |
| `scripts-list` | deterministically list SQL scripts across one or more directories   |

Deferred (documented, not implemented here): `scripts` execution against a live
database, `split`, `inspect`, `diff`, `templ`, job-to-job output wiring,
log rotation/retention. Live SQL execution already exists as
`relspec scripts execute`; wiring it into the job runner is a follow-up because
it needs live database credentials and cannot be covered by offline tests.

### Discovery and precedence

`relspec job` (no `--file`) scans `--dir` (default `.`) for:

1. `relspec.yml` / `relspec.yaml` (the default file), then
2. `relspec.<name>.yml` / `relspec.<name>.yaml` (extra files),

each group sorted lexically. Order is stable across runs. Use `--file <path>`
(repeatable) to load explicit files and skip discovery.

All discovered/selected files are merged into one job namespace. A job name
defined by **more than one file is a hard error** naming both files. YAML maps
already forbid duplicate keys within a single file.

### Paths

* Every path (`inputs[].path`, `output.path`, `script_dirs[]`, `logfile`) is
  **relative to the directory containing the job file that declared the job**,
  not the process working directory.
* Absolute paths, `~`-relative paths and any path that resolves outside the job
  file directory (`../`, `a/../../b`, …) are **rejected during validation** —
  before anything runs.

### Credentials

* Database inputs (`format: pgsql` / `mssql`) and database execution outputs
  (`format: pgsql` with `conn_env`) reference an **environment variable name**
  via `conn_env:`. The connection string itself is never stored in the
  manifest.
* A `conn_env` value that looks like a connection string (contains `:`, `/`,
  `@`, `=`, spaces) is rejected.
* Missing/empty environment variables are reported during pre-flight, before
  execution.
* Job logs and `--plan` output show `env:<NAME>`, never the value. Resolved
  secret values and anything matching a connection-string password are
  redacted (`***`) from the logfile and diagnostics.

### Validation happens before execution

`relspec job list` and `relspec job run` both fully validate the selected set
first. Nothing is read, written, connected to, or executed if validation fails.
Checks include:

* schema `version` (must be `1`), unknown YAML fields rejected
* duplicate job names across files
* unknown / missing `command`
* per-command input/output shape (`convert`/`merge` need inputs + output;
  `scripts-list` needs `script_dirs` and forbids inputs/output)
* unknown input/output `format`
* path traversal / absolute / home-relative paths
* `depends_on` targets exist
* dependency cycles (reported as `a -> b -> c -> a`)

Then, immediately before running, per-job pre-flight resolves paths and checks:

* every input file exists and is a file
* every `script_dir` exists and is a directory
* every `conn_env` variable is set
* `output.path` does not already exist unless `output.overwrite: true`

If any pre-flight check fails for **any** job in the plan, **no** job runs.

### Execution and exit codes

* `relspec job run <name>` runs the job's `depends_on` closure first, in
  topological order (deterministic), then the job. `--no-deps` runs only the
  named job.
* `--dry-run` (alias `--plan`) prints the resolved plan and exits 0 without
  touching inputs, outputs or databases.
* A failing job returns the underlying non-zero status (the process exits 1)
  and the error names the job. The logfile records `FAILED: <error>`; a
  successful job records `OK`. No separate success-marker file is written, so a
  failure can never leave a stale "success".

## Schema reference

```yaml
version: 1                       # required, must be 1
jobs:
  <job-name>:
    command: convert | merge | scripts-list   # required
    description: "free text"                   # optional, shown by `job list`
    depends_on: [other-job, ...]               # optional
    inputs:                                    # convert (≥1) / merge (≥2)
      - path: relative/file.dbml               # file inputs
        format: dbml
      - format: pgsql                          # live-connection inputs
        conn_env: SOURCE_DB_URL                # env var NAME
    script_dirs:                               # scripts-list (≥1)
      - migrations/core
      - migrations/tenant
    output:                                    # convert / merge (required)
      format: pgsql
      path: build/schema.sql                   # file output, OR:
      conn_env: TARGET_DB_URL                  # execute against DB (pgsql only)
      overwrite: false                         # default false
    options:
      flatten_schema: false
      schema: public
      package: models                          # for gorm/bun output
      continue_on_error: false                 # pgsql output
      skip_relations: false                    # merge only
      skip_enums: false
      skip_views: false
      skip_domains: false
      skip_sequences: false
    logfile: .relspec/log/<job-name>.log       # optional; appended to
```

### Supported input formats

`dbml`, `dctx`, `drawdb`, `graphql`, `json`, `yaml`, `gorm`, `bun`, `drizzle`,
`prisma`, `typeorm`, `sqlite` (file, via `path`); `pgsql`, `mssql`
(live, via `conn_env`).

### Supported output formats

`dbml`, `dctx`, `drawdb`, `graphql`, `json`, `yaml`, `gorm`, `bun`, `drizzle`,
`prisma`, `typeorm`, `pgsql`, `mssql`, `sqlite` (file, via `path`); `pgsql` also
supports `conn_env` to execute the generated DDL against a live database.

## Examples

### Merge many schema files, emit PostgreSQL DDL

```yaml
version: 1
jobs:
  build-schema:
    command: convert
    inputs:
      - { path: schema/core.dbml, format: dbml }
      - { path: schema/billing.dbml, format: dbml }
      - { path: schema/tenant.dbml, format: dbml }
    output:
      format: pgsql
      path: build/schema.sql
      overwrite: true
    logfile: .relspec/log/build-schema.log
```

### Multiple script directories

```yaml
version: 1
jobs:
  migration-order:
    command: scripts-list
    script_dirs:
      - migrations/core
      - migrations/tenant
      - migrations/reporting
    logfile: .relspec/log/migration-order.log
```

### Job depending on another job

```yaml
version: 1
jobs:
  build-schema:
    command: convert
    inputs:
      - { path: schema/core.dbml, format: dbml }
      - { path: schema/tenant.dbml, format: dbml }
    output: { format: json, path: build/schema.json, overwrite: true }
  build-docs:
    command: convert
    depends_on: [build-schema]
    inputs:
      - { path: schema/core.dbml, format: dbml }
    output: { format: yaml, path: build/schema.yaml, overwrite: true }
```

### Reading from a remote database

```yaml
version: 1
jobs:
  snapshot-prod:
    command: convert
    inputs:
      - format: pgsql
        conn_env: PROD_DB_URL      # export PROD_DB_URL=postgres://...
    output:
      format: dbml
      path: snapshots/prod.dbml
      overwrite: true
```