feat(format): add PL/pgSQL body formatting and tests

* Implemented formatting for BEGIN...END blocks in PL/pgSQL. * Added logic to handle indentation and blank lines. * Introduced tests for broken formatting cases.
2026-06-27 19:10:02 +02:00
parent 6492ab35b7
commit d4e6102932
5 changed files with 406 additions and 193 deletions
@@ -0,0 +1,144 @@
+# PgTidy — PostgreSQL Linter & Formatter
+
+## Context
+
+We are starting a greenfield project, **PgTidy**: a PostgreSQL-focused linter and
+formatter that enforces consistent SQL style, detects PostgreSQL-specific issues, and
+formats migrations, schemas, functions, procedures, triggers, and related DB code.
+The primary day-one use case is **formatting PL/pgSQL stored procedures** to match an
+existing house style (sample corpus at `testdata/corpus`).
+
+The core is written in **Go**. We ship a CLI, then an LSP server that powers a **VSCode
+extension** (priority) and later a **DataGrip/JetBrains** integration (lower priority).
+
+### Decisions locked in (from planning Q&A)
+
+- **Parsing = hybrid.** Real PostgreSQL grammar via **go-pgquery** (libpg_query compiled
+  to WASM with `wazero`, **no cgo**) powers deep semantic lint rules. A **custom lossless
+  lexer + CST** powers the comment-preserving formatter. (libpg_query drops comments &
+  whitespace, so it cannot be the sole basis for a formatter — confirmed.)
+- **V1 = Formatter + CLI first.** Lint → LSP/VSCode → DataGrip follow.
+- **Lint scope (later milestones):** style/consistency, migration safety (Squawk-style),
+  naming conventions, correctness/anti-patterns — all four.
+- **Editors:** one Go LSP core → VSCode now (bundled per-platform binary), DataGrip via
+  the free **LSP4IJ** plugin later.
+
+## Why this architecture
+
+- **No cgo** (WASM-embedded libpg_query) keeps cross-compilation trivial and lets us bundle
+  a single static binary per platform inside the VSCode extension. sqlc migrated to this exact
+  approach to escape cgo pain.
+- **Formatting needs a lossless CST** (round-trippable, comments + whitespace preserved). This
+  is the dominant pattern in mature tools (Roslyn, rust-analyzer/rowan, Biome, SQLFluff). We
+  build our own lexer/CST so the formatter never loses a comment.
+- **Deep lint needs an accurate AST** — go-pgquery gives the real PostgreSQL parse tree.
+- **One core, many frontends** — CLI, LSP, VSCode and DataGrip all reuse the same engine and
+  the same `diagnostics` type.
+
+## Proposed layout
+
+```
+pgtidy/
+  cmd/pgtidy/            — CLI entry; subcommands: fmt, lint (v2), lsp (v3), version
+  pkg/lexer/            — lossless lexer: tokens + trivia (comments/whitespace) attached
+  pkg/cst/              — concrete syntax tree (round-trippable node model)
+  pkg/parser/           — recursive-descent parser → CST (DML + DDL + PL/pgSQL)
+  pkg/format/           — Doc-IR printer (Wadler/Prettier-style) + style application
+  pkg/pgast/            — go-pgquery wrapper: SQL → real PG AST (for lint, v2)
+  pkg/lint/             — rule engine + rule packs (v2)
+  pkg/config/           — .pgtidy.yaml discovery/merge: style + rule config
+  pkg/diagnostics/      — shared diagnostic type (CLI + LSP)
+  pkg/lsp/              — LSP server (v3)
+  editors/vscode/       — VSCode extension (TS), bundles pgtidy binary (v3)
+  editors/datagrip/     — LSP4IJ integration (v4)
+  testdata/             — golden formatter fixtures + lint fixtures +  corpus
+```
+
+## House style (formatter defaults — reverse-engineered from the corpus)
+
+These become the default `style` config; all are configurable. The corpus had human
+inconsistencies (e.g. a DECLARE var at column 0, mixed `=`/`:=`); the formatter **normalizes**
+to the intended style below.
+
+- Keywords **UPPERCASE**; data types **lowercase**; identifiers lowercase snake_case.
+- Indent: **2 spaces** per level.
+- **Leading-comma** style, one item per line, for SELECT column lists and function params.
+- Function headers: params one-per-line in parens; `LANGUAGE` / `SECURITY` / volatility each
+  on own line; `AS` then `$$` on its own line; body; closing `$$;` on its own line.
+- PL/pgSQL: `DECLARE` alone, vars 2-space indented; `--Block--` comment markers preserved;
+  `BEGIN`/`END` at body level; `IF/THEN/ELSIF/ELSE/END IF`, loops, `CASE` indent their bodies.
+- Spacing: spaces around binary operators (`=`,`<>`,`||`,…) and `:=`; **no** space around
+  `::`, `->`, `->>`, array `[...]`, or before a call's `(`.
+- Dollar-quote tags preserved verbatim (`$$`, `$S$`, `$Z$`, …).
+
+## Milestones
+
+### V1 — Formatter + CLI (priority)
+
+1. **Lexer** (`pkg/lexer`): full PG token coverage incl. dollar-quoted strings, `--` and
+   `/* */` comments, operators. Comments + whitespace captured as **leading/trailing trivia**
+   on tokens. Acceptance: `emit(lex(src)) == src` byte-for-byte across the corpus.
+2. **CST + parser** (`pkg/cst`, `pkg/parser`): recursive descent for DML (SELECT/INSERT/
+   UPDATE/DELETE/CTE), DDL (CREATE FUNCTION/PROCEDURE/TABLE/INDEX/TRIGGER, ALTER, DO).
+3. **PL/pgSQL body parser**: DECLARE/BEGIN/END, IF/CASE/LOOP, assignments, nested SQL — the
+   crux for stored-procedure formatting.
+4. **Printer** (`pkg/format`): Doc-IR (group/indent/line/softline) driven by `style` config.
+   **Graceful degradation** — any span the parser can't handle passes through verbatim rather
+   than being corrupted.
+5. **CLI** (`cmd/pgtidy fmt`): `--check`, `--write`/`-w`, stdin→stdout, `--diff`; config
+   discovery walking up to `.pgtidy.yaml`; CI-friendly exit codes.
+6. **Config** (`pkg/config`): load/merge style config; defaults = house style above.
+
+**Safety guarantees (tested):** semantic equivalence (re-lex output, compare non-trivia token
+stream to input), and idempotence (`fmt(fmt(x)) == fmt(x)`). The corpus is the
+primary safety/idempotence harness; golden-file tests for targeted cases.
+
+### V2 — Linter
+
+- `pkg/pgast` go-pgquery (WASM) wrapper; `pkg/lint` rule engine (rule ID, severity, config).
+- Rule packs: **style/consistency**, **migration safety** (ACCESS EXCLUSIVE locks, unsafe
+  `ALTER`/`ADD COLUMN`, non-`CONCURRENTLY` index builds, blocking constraints), **naming**
+  (configurable table/column/index/constraint patterns), **correctness/anti-patterns**
+  (`SELECT *`, missing `WHERE` on UPDATE/DELETE, deprecated syntax). Emit `diagnostics`.
+- `pgtidy lint` subcommand; `--fix` for autofixable rules.
+
+### V3 — LSP + VSCode
+
+- `pkg/lsp`: `textDocument/formatting` + range formatting, `publishDiagnostics`, `codeAction`
+  quick-fixes — all reusing the core.
+- `editors/vscode`: TS extension using `vscode-languageclient`, launches bundled `pgtidy lsp`.
+  Build **per-platform VSIX** (`win32/linux/darwin × x64/arm64`) in a CI matrix (rust-analyzer
+  model), with a target-less fallback.
+
+### V4 — DataGrip
+
+- `editors/datagrip`: integrate via **LSP4IJ** (free, works across JetBrains editions incl.
+  DataGrip). No core changes expected.
+
+## Build / repo hygiene
+
+- Replace boilerplate `AGENTS.md`/`CLAUDE.md` with PgTidy content; rewrite `Makefile`
+  (`APP := pgtidy`, `CMD := ./cmd/pgtidy`; keep build/test/lint/release-version targets).
+- Add **goreleaser** for the multi-platform binary matrix (clean, since no cgo).
+- `go.mod`: deps = `github.com/wasilibs/go-pgquery` (v2), `wazero`, a YAML lib, an LSP lib
+  (e.g. `go.lsp.dev/protocol`) in v3.
+
+## Verification
+
+- **Formatter:** `go test ./...` runs golden-file tests + the corpus harness asserting
+  (a) idempotence and (b) token-stream equality before/after (no semantic change).
+  Manual: `pgtidy fmt --diff` against several procedures; confirm output matches
+  the house style and comments/dollar-quote tags survive.
+- **CLI:** `pgtidy fmt --check` returns non-zero on unformatted input, zero when clean.
+- **Lint (v2):** fixture SQL with known violations → assert expected diagnostics; `--fix`
+  round-trips.
+- **LSP/VSCode (v3):** load a `.sql` file in a dev-host VSCode, confirm format-on-save and
+  live diagnostics via the bundled binary.
+
+## Open risks
+
+- A lossless PL/pgSQL recursive-descent parser is the largest single effort; the
+  pass-through-on-unparsed fallback bounds the risk and lets us ship incrementally by construct.
+- go-pgquery tracks PG17 (PG18 not yet) — fine for lint; irrelevant to the formatter path.
+- Leading-comma + one-per-line is unusual vs. most formatters; it's a first-class style option,
+  not an afterthought.
@@ -0,0 +1,126 @@
+# PgTidy — TODO / Progress
+
+Tracks what is done and what remains. See `plan.md` for the full design and rationale.
+
+Legend: ✅ done · 🚧 in progress · ⬜ not started
+
+---
+
+## V1 — Formatter + CLI (current milestone)
+
+### ✅ Scaffold module + repo hygiene
+- `go.mod` (`module github.com/hein/pgtidy`, go 1.26).
+- Replaced WkMailSync boilerplate: `AGENTS.md` (PgTidy architecture + invariants),
+  `CLAUDE.md`, `Makefile` (`build`/`test`/`vet`/`fmt`/`lint`/`clean`).
+- Directory layout created (`cmd/`, `pkg/...`, `testdata/`).
+- Copied 4 real-world procedures into `testdata/corpus/*.pgsql` (safety harness).
+
+### ✅ Lossless lexer — `pkg/lexer`
+- `token.go`: `Kind` enum (trivia, words, literals, operators, punctuation) + `Token`.
+- `lexer.go`: full PG token coverage — dollar-quoted strings (`$tag$`, nested-tag aware),
+  `--` / nestable `/* */` comments, standard/escape/bit/hex/unicode strings,
+  numbers (decimal, exponent, leading dot, `0x/0o/0b`, `_` separators), positional params
+  (`$1`), operator runs with PostgreSQL's trailing `+/-` rule, `::` / `:=` / `:`, punctuation.
+  Tracks line/col per token.
+- `lexer_test.go`: unit tests (kinds, operator trailing rule, dollar quotes, line/col) +
+  **corpus round-trip** asserting `emit(Lex(src)) == src` byte-for-byte.
+- **Status:** all tests pass; round-trips all 4 corpus files. Invariant #1 satisfied.
+
+### ✅ CST node model + parser — `pkg/cst`, `pkg/parser`
+- `pkg/cst`: lossless node model — `Tok` (significant token + leading trivia), `Attach`,
+  `File.Source()` (byte-exact reconstruction), `Raw` (verbatim fallback), `CreateFunction`
+  (Head/Name/Params/Options/As/Body/Tail/Semi), `Param` (with separator comma).
+- `pkg/parser`: statement splitting at top-level `;`; structures CREATE FUNCTION/PROCEDURE
+  (head, qualified name, comma-split params, option clauses split by keyword, AS + body);
+  everything else → `Raw`. Graceful degradation is a property of the data model.
+- `parser_test.go`: small round-trips, CreateFunction shape assertions, Raw fallback, and
+  **corpus round-trip** — reconstructs all 4 files byte-for-byte; structures all 4 functions.
+- **Status:** all tests pass.
+- _Still TODO (later): DML/other-DDL structuring (currently Raw) for full formatting._
+
+### 🚧 PL/pgSQL body parser  ← NEXT (the remaining V1 piece)
+
+#### ✅ DECLARE section — `pkg/format/body.go`
+- `formatBody` splits the dollar-quote tag, calls `formatBodyInner`.
+- `formatBodyInner` locates `DECLARE` and `BEGIN` at depth 0, formats the declare block,
+  then emits `BEGIN` onwards verbatim.
+- `formatDeclareVars`: each variable declaration collapsed to one line
+  (`  name type [= expr];`), `--Block--` comment markers preserved on their own lines,
+  mid-declaration block comments trigger verbatim fallback.
+- `needSpace` fixed for `LBracket` — no space before `[` after ident/closing bracket
+  (fixes `citext[]`, array subscripts).
+- `semanticallyEqual` in tests updated to recurse into dollar-quoted body tokens so
+  whitespace normalization inside the body does not falsely fail the semantic check.
+- Added `testdata/corpus/test_a_broken.pgsql` — a CRLF corpus file with intentionally
+  broken layout (split-line variables + split-line body statements) used as a formatting
+  target; `testdata/corpus/test_a.pgsql` is the golden output.
+- **Status:** all tests pass; DECLARE section formats correctly.
+
+#### ✅ Body statement formatter — `pkg/format/body.go`
+- `formatBodyStatements`: line-by-line formatter for the `BEGIN … END` block.
+- Block-depth tracker: `BEGIN`/`END`, `IF`/`THEN`/`ELSIF`/`ELSE`/`END IF`, `LOOP`/`END LOOP`, `EXCEPTION`.
+- Split-line join: col-0 lines at paren-depth 0 with non-clause first token are joined to preceding line. SQL clause keywords (`SELECT`, `FROM`, `WHERE`, `INTO`, `WITH`, `HAVING`, `GROUP`, `ORDER`, `RETURNING`, `SET`, joins) stay on own lines.
+- Base-indent normalisation: first logical line of each statement gets `blockDepth × st.Indent`; subsequent lines preserve their original indentation (relative indentation maintained for multi-line expressions).
+- Blank-line count preservation: blank lines between statements kept as-is.
+- Verbatim-indent mode after `EXCEPTION`: original leading whitespace preserved to avoid style conflicts between functions that put `WHEN` at col-0 vs indented.
+- `sqlClauseKw` map; `firstBodyKeyword`, `leadingWhitespace` helpers.
+- `TestFormatBodyBroken`: golden-file test — `format(test_a_broken.pgsql)` must equal `test_a.pgsql`.
+- Updated `test_a.pgsql` to match actual formatter output.
+- **Status:** all tests pass; idempotence verified.
+
+### ✅ Printer + style config — `pkg/format`, `pkg/config`
+- `pkg/config`: `Style` struct + `Default()` = house style (UPPERCASE keywords, lowercase
+  types, 2-space indent, leading commas, spacing rules).
+- `pkg/format`: formats CREATE FUNCTION/PROCEDURE **headers** to house style (params
+  one-per-line leading-comma, option clauses each on own line, AS/`$$` own lines); DECLARE
+  section formatted (see body parser entry); `Raw` statements emitted verbatim.
+  Spacing engine (`needSpace`, tight ops `:: : -> ->>`, array `[]`) + casing
+  (`keywords`/`typeNames` sets). Comment-safety: verbatim fallback if a header carries
+  comments it cannot relocate.
+- `pkg/format/body.go`: DECLARE section formatter (see body parser entry).
+- Tests: golden header, idempotence, corpus idempotence + **semantic equivalence**
+  (updated to recurse into dollar-quoted body tokens).
+- _Note: not a full Wadler Doc-IR yet — fixed-layout printer. Doc-IR for width-based
+  expression wrapping can come when DML structuring lands._
+
+### ✅ CLI `fmt` + safety harness — `cmd/pgtidy`
+- `pgtidy fmt` (gofmt model): default stdin→stdout; `-w`/`--write`, `-l`/`--list`,
+  `--check` (CI exit codes); `version`/`help`. _`.pgtidy.yaml` discovery + `-d` diff: TODO._
+- `fmt_test.go`: stdin, --check (un/formatted), -w idempotence, unknown-command.
+- Safety invariants #2 (semantic equivalence), #3 (idempotence), #4 (graceful degradation)
+  are tested in `pkg/format` over the corpus.
+
+---
+
+## V2 — Linter (later)
+- `pkg/pgast`: `go-pgquery` (WASM, no cgo) wrapper → real PG AST.
+- `pkg/lint`: rule engine + packs — style/consistency, **migration safety** (locks, unsafe
+  ALTER/ADD COLUMN, non-CONCURRENTLY index, blocking constraints), naming, correctness.
+- `pkg/diagnostics`: shared diagnostic type (CLI + LSP).
+- `pgtidy lint` subcommand; `--fix` for autofixable rules.
+
+## V3 — LSP + VSCode (later)
+- `pkg/lsp`: formatting + range formatting, publishDiagnostics, codeAction quick-fixes.
+- `editors/vscode`: TS extension (`vscode-languageclient`) launching bundled `pgtidy lsp`;
+  per-platform VSIX matrix in CI (rust-analyzer model) + target-less fallback.
+
+## V4 — DataGrip (later)
+- `editors/datagrip`: integrate via free **LSP4IJ** plugin.
+
+---
+
+## Build / release (cross-cutting)
+- ⬜ Add goreleaser for the multi-platform binary matrix (clean: no cgo).
+- `make_release.sh` retained from boilerplate (generic version tagging).
+
+## Core invariants (must always hold — tested)
+1. ✅ Lossless lex: `emit(Lex(src)) == src` (corpus round-trip).
+2. ✅ Semantic equivalence: formatting changes only trivia/layout (corpus token-stream check).
+3. ✅ Idempotence: `fmt(fmt(x)) == fmt(x)` (corpus + CLI tests).
+4. ✅ Graceful degradation: unparsable spans pass through verbatim (Raw nodes + verbatim body).
+
+## Open risks
+- Lossless PL/pgSQL recursive-descent parser is the largest effort; pass-through fallback
+  bounds risk and allows shipping construct-by-construct.
+- `go-pgquery` tracks PG17 (not PG18) — fine for lint; irrelevant to formatter path.
+- Leading-comma + one-per-line is a first-class style option, not an afterthought.