Relspec job files #20

New Issue

2026-08-27T19:02:33Z

warkanum commented

2026-08-27 19:02:33 +00:00

We need a way for relspec to detect files, paths and commands to run from a job file instead of command line args.
e.g. relspec job a, relspec job b
Then we have a file or files called relspec.yml or relpsec.a.yml with the job name defined, command to action, inputs, output config and a logfile etc.
Basically something to easy auto building of schemas.
The reason for this. Some jobs might contain many schema files to merge, then convert it.
Some might contain files to run on a remote database etc.

This issue needs to be refined, but thats the rought idea.

We need a way for relspec to detect files, paths and commands to run from a job file instead of command line args. e.g. relspec job a, relspec job b Then we have a file or files called relspec.yml or relpsec.a.yml with the job name defined, command to action, inputs, output config and a logfile etc. Basically something to easy auto building of schemas. The reason for this. Some jobs might contain many schema files to merge, then convert it. Some might contain files to run on a remote database etc. This issue needs to be refined, but thats the rought idea.

sgcommand commented

2026-08-28 04:19:23 +00:00

Research update (2026-08-28)

Current implementation findings

RelSpec already has scripts list and scripts execute commands. The current workflow discovers SQL files from a directory, sorts them by priority/sequence/name, and executes them against a PostgreSQL connection.
The current CLI is argument-driven: cmd/relspec/scripts.go exposes directory, connection, schema/database, and ignore-errors flags. There is no job-file loader, job-name resolver, declarative input/output model, or per-job logfile configuration.
YAML is already a supported schema format and the project already depends on YAML-capable configuration patterns, so a job manifest can be introduced without inventing a second serialization format.
A job file must not become an arbitrary shell-execution feature by accident. Remote database connection strings, file paths, and commands need explicit validation and safe logging; credentials must never be persisted in job output or logs.

Proposed job-file contract

Support a default relspec.yml plus explicitly selected files such as relspec.<name>.yml, with a command such as:

relspec job list
relspec job run build-schema

A first schema could contain:

version: 1
jobs:
  build-schema:
    command: convert
    inputs:
      - path: schema.dbml
        format: dbml
    output:
      format: pgsql
      path: build/schema.sql
    options:
      flatten_schema: false
    logfile: .relspec/log/build-schema.log

The design should define whether command is a closed enum (convert, merge, split, inspect, scripts) or an allowlisted subcommand graph. Arbitrary shell strings should be rejected by default. Remote database jobs should reference environment variable names or an existing credential mechanism rather than embedding secrets.

Required behavior to settle

File discovery precedence and merge rules when multiple job files exist.
Job-name uniqueness and deterministic listing order.
Relative-path base: job-file directory versus process working directory.
Input/output validation, parent-directory creation, overwrite policy, and atomic output behavior.
Dependency handling between jobs, cycle detection, and whether a job may consume another job's output.
Per-job log path, redaction, rotation/retention, and whether logs are stdout/stderr mirrors.
Dry-run/plan output and exit-code propagation from the underlying command.
Schema versioning and clear line/field-level validation errors.

Suggested acceptance tests

Discover default and explicitly named job files deterministically.
job list shows stable names and source files.
Run a file-based conversion job in a temporary directory and verify output.
Reject duplicate names, unknown commands, missing inputs, path traversal where disallowed, invalid formats, and dependency cycles before executing anything.
Verify relative paths resolve from the job file location.
Verify secrets are absent from logs and generated diagnostics.
Verify a failed job returns the underlying non-zero status and does not leave a misleading success marker.
Verify a job file can express the existing multi-file merge workflow without requiring shell quoting.

This should be implemented in phases: schema/parser and validation, job list, one safe command adapter, then dependency execution/logging and broader command coverage.

Research update (2026-08-28) ## Current implementation findings - RelSpec already has `scripts list` and `scripts execute` commands. The current workflow discovers SQL files from a directory, sorts them by priority/sequence/name, and executes them against a PostgreSQL connection. - The current CLI is argument-driven: `cmd/relspec/scripts.go` exposes directory, connection, schema/database, and ignore-errors flags. There is no job-file loader, job-name resolver, declarative input/output model, or per-job logfile configuration. - YAML is already a supported schema format and the project already depends on YAML-capable configuration patterns, so a job manifest can be introduced without inventing a second serialization format. - A job file must not become an arbitrary shell-execution feature by accident. Remote database connection strings, file paths, and commands need explicit validation and safe logging; credentials must never be persisted in job output or logs. ## Proposed job-file contract Support a default `relspec.yml` plus explicitly selected files such as `relspec.<name>.yml`, with a command such as: ```text relspec job list relspec job run build-schema ``` A first schema could contain: ```yaml version: 1 jobs: build-schema: command: convert inputs: - path: schema.dbml format: dbml output: format: pgsql path: build/schema.sql options: flatten_schema: false logfile: .relspec/log/build-schema.log ``` The design should define whether `command` is a closed enum (`convert`, `merge`, `split`, `inspect`, `scripts`) or an allowlisted subcommand graph. Arbitrary shell strings should be rejected by default. Remote database jobs should reference environment variable names or an existing credential mechanism rather than embedding secrets. ## Required behavior to settle 1. File discovery precedence and merge rules when multiple job files exist. 2. Job-name uniqueness and deterministic listing order. 3. Relative-path base: job-file directory versus process working directory. 4. Input/output validation, parent-directory creation, overwrite policy, and atomic output behavior. 5. Dependency handling between jobs, cycle detection, and whether a job may consume another job's output. 6. Per-job log path, redaction, rotation/retention, and whether logs are stdout/stderr mirrors. 7. Dry-run/plan output and exit-code propagation from the underlying command. 8. Schema versioning and clear line/field-level validation errors. ## Suggested acceptance tests - Discover default and explicitly named job files deterministically. - `job list` shows stable names and source files. - Run a file-based conversion job in a temporary directory and verify output. - Reject duplicate names, unknown commands, missing inputs, path traversal where disallowed, invalid formats, and dependency cycles before executing anything. - Verify relative paths resolve from the job file location. - Verify secrets are absent from logs and generated diagnostics. - Verify a failed job returns the underlying non-zero status and does not leave a misleading success marker. - Verify a job file can express the existing multi-file merge workflow without requiring shell quoting. This should be implemented in phases: schema/parser and validation, `job list`, one safe command adapter, then dependency execution/logging and broader command coverage.

warkanum commented

2026-08-28 04:32:46 +00:00

We need support for multiple script directories as well

sgcommand referenced this issue from a commit

2026-09-01 22:44:42 +00:00

feat(job): declarative YAML job files for named relspec workflows

sgcommand referenced this issue

2026-09-01 22:45:15 +00:00

feat(job): declarative YAML job files for named relspec workflows (#20) #24

sgcommand commented

2026-09-01 22:45:57 +00:00

Implemented in PR #24 → #24

branch: issue-20-job-files
head SHA: 4d299fda986bb7df0affa6ec876ce3c44e3d84ce (verified == origin)
base: master · state: open · mergeable · 14 files

v1 contract (docs/JOB_FILES.md)

relspec job list / relspec job run <name> driven by relspec.yml +
relspec.<name>.yml; --file overrides discovery; --dir sets scan root.
command is a closed allow-list — convert, merge, scripts-list.
No field accepts a shell string or executable path.
Deterministic discovery (default file first, then named files lexically);
all files merged into one namespace; duplicate job names across files are a
hard error.
Every path resolves relative to the job file's directory; absolute,
~-relative and directory-escaping paths rejected at validation.
DB credentials referenced by env-var name via conn_env:; connection
strings never stored; redacted (***) from logs and --plan.
Full validation (version, unknown fields, command/format, per-command
input/output shape, path traversal, depends_on targets, dependency cycles)
runs before anything is read/written/executed; per-job pre-flight then checks
input existence, script dirs, env vars and overwrite policy for the whole
plan — any failure ⇒ no job runs.
depends_on closure runs in deterministic topological order; --no-deps
runs only the named job.
--dry-run (alias --plan) prints the resolved plan, exits 0, touches
nothing.
Failing job propagates the underlying non-zero exit status, logs FAILED
(never OK), writes no success marker.
Multiple script directories supported (comment above).

Acceptance coverage

All acceptance tests from the research comment are covered: deterministic
discovery/listing, temp-dir conversion output, rejection of duplicate names /
unknown commands / arbitrary shell strings / missing inputs / path traversal /
invalid formats / dependency cycles before execution, relative-path resolution
from the job file, secret redaction, failed-job exit propagation with no
misleading success marker, and the multi-file merge workflow without shell
quoting. 17 unit tests (pkg/jobs) + 11 CLI tests (cmd/relspec).

Verification

gofmt -l clean · go vet ./... clean · go build ./... ok ·
go test ./... all pass (40 pkgs) · go test -race ./pkg/jobs/... ./cmd/relspec/...
pass · git diff --check clean · manual smoke run against examples/jobs/.

Caveats / deferred

Live scripts execute against a database from a job is deferred (needs live
credentials, not offline-testable); relspec scripts execute still covers it.
split / inspect / diff / templ job commands deferred.
Job-to-job output file wiring deferred; depends_on only orders execution.
Log rotation/retention deferred (append-only stderr mirror).
Output written in place (parent dirs created), not atomic-rename.
Path safety is string/filepath.Rel-based; symlink-escape hardening deferred.
golangci-lint not run in this environment: installed binary is v1 while the
repo config is v2 (pre-existing mismatch, unrelated to this change).

Implemented in PR #24 → https://git.warky.dev/wdevs/relspecgo/pulls/24 - branch: `issue-20-job-files` - head SHA: `4d299fda986bb7df0affa6ec876ce3c44e3d84ce` (verified == origin) - base: `master` · state: open · mergeable · 14 files ## v1 contract (docs/JOB_FILES.md) - `relspec job list` / `relspec job run <name>` driven by `relspec.yml` + `relspec.<name>.yml`; `--file` overrides discovery; `--dir` sets scan root. - `command` is a **closed allow-list** — `convert`, `merge`, `scripts-list`. No field accepts a shell string or executable path. - Deterministic discovery (default file first, then named files lexically); all files merged into one namespace; duplicate job names across files are a hard error. - Every path resolves relative to the **job file's directory**; absolute, `~`-relative and directory-escaping paths rejected at validation. - DB credentials referenced by **env-var name** via `conn_env:`; connection strings never stored; redacted (`***`) from logs and `--plan`. - Full validation (version, unknown fields, command/format, per-command input/output shape, path traversal, `depends_on` targets, dependency cycles) runs before anything is read/written/executed; per-job pre-flight then checks input existence, script dirs, env vars and overwrite policy for the whole plan — any failure ⇒ no job runs. - `depends_on` closure runs in deterministic topological order; `--no-deps` runs only the named job. - `--dry-run` (alias `--plan`) prints the resolved plan, exits 0, touches nothing. - Failing job propagates the underlying non-zero exit status, logs `FAILED` (never `OK`), writes no success marker. - Multiple script directories supported (comment above). ## Acceptance coverage All acceptance tests from the research comment are covered: deterministic discovery/listing, temp-dir conversion output, rejection of duplicate names / unknown commands / arbitrary shell strings / missing inputs / path traversal / invalid formats / dependency cycles before execution, relative-path resolution from the job file, secret redaction, failed-job exit propagation with no misleading success marker, and the multi-file merge workflow without shell quoting. 17 unit tests (pkg/jobs) + 11 CLI tests (cmd/relspec). ## Verification `gofmt -l` clean · `go vet ./...` clean · `go build ./...` ok · `go test ./...` all pass (40 pkgs) · `go test -race ./pkg/jobs/... ./cmd/relspec/...` pass · `git diff --check` clean · manual smoke run against `examples/jobs/`. ## Caveats / deferred - Live `scripts execute` against a database from a job is deferred (needs live credentials, not offline-testable); `relspec scripts execute` still covers it. - `split` / `inspect` / `diff` / `templ` job commands deferred. - Job-to-job output file wiring deferred; `depends_on` only orders execution. - Log rotation/retention deferred (append-only stderr mirror). - Output written in place (parent dirs created), not atomic-rename. - Path safety is string/`filepath.Rel`-based; symlink-escape hardening deferred. - `golangci-lint` not run in this environment: installed binary is v1 while the repo config is v2 (pre-existing mismatch, unrelated to this change).

warkanum referenced this issue from a commit

2026-09-02 04:16:40 +00:00

Merge pull request 'feat(job): declarative YAML job files for named relspec workflows (#20)' (#24) from issue-20-job-files into master

warkanum commented

2026-09-07 04:12:58 +00:00

Add the templ command to the job files.

sgcommand commented

2026-09-08 04:03:56 +00:00

Dispatcher assignment — 2026-09-08 06:03 SAST

• Forge/repository: Gitea wdevs/relspecgo
• Issue: #20 — Relspec job files; follow-up scope explicitly requested by owner: add the templ command to job files.
• Classification: simple bounded extension of the existing job-file command allow-list and adapter, with focused tests/docs; no new architecture decision beyond the existing templ CLI contract.
• Selected backend: custom:litellm-warky / ornith:9b. Bounded probe returned ORNITH_DISPATCH_PROBE_OK.
• Worker session/process: not launched yet; assignment comment precedes worker launch.
• Worktree: /home/hermes/work/issue-agents/wdevs/relspecgo/issue-20-templ-job-command
• Branch: issue-20-templ-job-command (created locally from synchronized origin/master); PR does not yet exist.
• Kanban card: t_beeb3f4f, created blocked and read back to suppress duplicate board dispatch.
• Plan: read AGENTS.md/CLAUDE.md and existing job-file plus templ CLI contracts, implement the smallest safe templ adapter and validation/docs/examples as appropriate, add focused regression coverage, run gofmt/vet/build/tests/diff-check, commit, push, create a non-empty PR, and post exact SHA/verification here.
• Required skills: cron-model-selection-policy, cron-worker-contract.
• Safety: no force-push, history rewrite, destructive reset, issue close, PR merge, deployment, secrets, or dangerous permission bypass.

Dispatcher assignment — 2026-09-08 06:03 SAST • Forge/repository: Gitea wdevs/relspecgo • Issue: #20 — Relspec job files; follow-up scope explicitly requested by owner: add the templ command to job files. • Classification: simple bounded extension of the existing job-file command allow-list and adapter, with focused tests/docs; no new architecture decision beyond the existing templ CLI contract. • Selected backend: custom:litellm-warky / ornith:9b. Bounded probe returned ORNITH_DISPATCH_PROBE_OK. • Worker session/process: not launched yet; assignment comment precedes worker launch. • Worktree: /home/hermes/work/issue-agents/wdevs/relspecgo/issue-20-templ-job-command • Branch: issue-20-templ-job-command (created locally from synchronized origin/master); PR does not yet exist. • Kanban card: t_beeb3f4f, created blocked and read back to suppress duplicate board dispatch. • Plan: read AGENTS.md/CLAUDE.md and existing job-file plus templ CLI contracts, implement the smallest safe templ adapter and validation/docs/examples as appropriate, add focused regression coverage, run gofmt/vet/build/tests/diff-check, commit, push, create a non-empty PR, and post exact SHA/verification here. • Required skills: cron-model-selection-policy, cron-worker-contract. • Safety: no force-push, history rewrite, destructive reset, issue close, PR merge, deployment, secrets, or dangerous permission bypass.

sgcommand referenced this issue

2026-09-08 04:20:52 +00:00

feat(job): support templ command #25

sgcommand commented

2026-09-08 04:21:28 +00:00

Worker delivery update — 2026-09-08 06:20 SAST

• PR: #25
• Branch: issue-20-templ-job-command
• SHA: 3d57c947cd

Implemented the explicitly requested templ command in the declarative job-file system. The closed allow-list, templ-specific input/mode/path/output validation, template-file preflight, execution through the existing template writer, focused database-mode regression coverage, and JOB_FILES.md documentation are included.

Verification:
• gofmt -l cmd/relspec pkg/jobs — clean
• go vet ./... — passed
• go build ./... — passed
• go test ./... — passed
• go test -race ./pkg/jobs/... ./cmd/relspec/... — passed
• git diff --check — passed

Caveat: golangci-lint was not run because the environment has a v1 binary while the repository configuration requires v2. No issue close or PR merge performed; PR remains open for review.

Worker delivery update — 2026-09-08 06:20 SAST • PR: https://git.warky.dev/wdevs/relspecgo/pulls/25 • Branch: issue-20-templ-job-command • SHA: 3d57c947cd10e05c9f7dc9ea8a987dc1e2fd95d5 Implemented the explicitly requested templ command in the declarative job-file system. The closed allow-list, templ-specific input/mode/path/output validation, template-file preflight, execution through the existing template writer, focused database-mode regression coverage, and JOB_FILES.md documentation are included. Verification: • gofmt -l cmd/relspec pkg/jobs — clean • go vet ./... — passed • go build ./... — passed • go test ./... — passed • go test -race ./pkg/jobs/... ./cmd/relspec/... — passed • git diff --check — passed Caveat: golangci-lint was not run because the environment has a v1 binary while the repository configuration requires v2. No issue close or PR merge performed; PR remains open for review.

warkanum referenced this issue from a commit

2026-09-08 12:32:18 +00:00

feat(job): complete deferred job-file features

warkanum referenced a pull request that will close this issue

2026-09-08 12:32:29 +00:00

feat(job): complete deferred job-file features (#20) #26

warkanum commented

2026-09-08 12:32:53 +00:00

Completed in PR #26 (branch issue-20-complete-job-files).

Delivered on top of the initial job system (convert/merge/scripts-list/templ):

forward-permissive version (accept >= 1, warn on newer)
from_job job-to-job output→input wiring
logfile size-rotation (5MB / keep 3 defaults, per-job + defaults: overrides)
new commands: split, inspect, diff, scripts-exec
atomic single-file output writes
symlink-escape hardening in path resolution

Docs and shipped example updated. make build/test/lint all green.

Completed in PR #26 (branch `issue-20-complete-job-files`). Delivered on top of the initial job system (`convert`/`merge`/`scripts-list`/`templ`): - forward-permissive `version` (accept `>= 1`, warn on newer) - `from_job` job-to-job output→input wiring - logfile size-rotation (5MB / keep 3 defaults, per-job + `defaults:` overrides) - new commands: `split`, `inspect`, `diff`, `scripts-exec` - atomic single-file output writes - symlink-escape hardening in path resolution Docs and shipped example updated. `make build`/`test`/`lint` all green.

warkanum closed this issue

2026-09-08 12:32:54 +00:00

warkanum referenced this issue from a commit

2026-09-08 14:18:59 +00:00

Merge pull request 'feat(job): complete deferred job-file features (#20)' (#26) from issue-20-complete-job-files into master

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: wdevs/relspecgo#20