Hein
5cdec88299
Replace the bare SemanticallyEqual call in every frontend (CLI fmt, LSP
formatting + rangeFormatting) with format.VerifySafe, which runs four
checks before any formatted output is emitted:
- semantic equivalence - the code token stream is unchanged
- comment preservation - no -- or /* */ comment is dropped, merged,
split, reordered, or reworded (line endings / indentation normalised
away; recurses into dollar-quoted bodies)
- structural balance - the () [] and BEGIN/CASE/IF/LOOP...END nesting
profile matches, ignoring anything inside a comment or a string
- idempotence - a second format pass would not change it
On failure the CLI now prints the specific reason and keeps the original.
The comment check surfaced two real formatter bugs, both fixed in
formatBodyStatements:
- multi-line /* */ comments inside a PL/pgSQL body had their interior
lines re-split and reindented as if they were statements; they are
now tracked and carried verbatim with the opening line
- a column-0 -- line was glued onto the preceding line by the
split-line-join, which merged consecutive comment lines into one
Regenerate testdata/corpus/test_mm_proc.pgsql (was carrying the mangled
output). TestCorpusIdempotentAndSafe now runs the full VerifySafe bundle;
add safety_test.go with targeted cases.
149 lines
8.4 KiB
Markdown
149 lines
8.4 KiB
Markdown
## Project Overview
|
||
|
||
**PgTidy** is a PostgreSQL-focused linter and formatter written in Go. It enforces a
|
||
consistent SQL style, detects PostgreSQL-specific issues, and formats migrations, schemas,
|
||
functions, procedures, triggers, and related database code. The primary day-one use case is
|
||
formatting PL/pgSQL stored procedures to a defined house style.
|
||
|
||
It ships as a CLI, an LSP server, a VSCode extension, and (later) a DataGrip integration.
|
||
|
||
# Agent Rules
|
||
|
||
Keep your answers short. Question everything, never assume or guess. Ask the user if you are
|
||
unsure about anything.
|
||
|
||
# Architecture
|
||
|
||
PgTidy uses a **hybrid** strategy:
|
||
|
||
- **Formatting** is driven by a custom **lossless lexer + CST** (concrete syntax tree) that
|
||
preserves every comment and whitespace span, so formatting can round-trip safely.
|
||
- **Linting** (later milestones) is driven by the real PostgreSQL grammar via
|
||
`go-pgquery` (libpg_query compiled to WASM with wazero — no cgo), which yields an accurate
|
||
AST for deep semantic rules.
|
||
|
||
A single Go core backs every frontend (CLI, LSP, editors) and shares one `diagnostics` type.
|
||
|
||
```
|
||
cmd/pgtidy/ — CLI entry; subcommands: fmt, lint (v2), lsp (v3), version
|
||
pkg/lexer/ — lossless lexer: tokens + trivia (comments/whitespace) attached
|
||
pkg/cst/ — concrete syntax tree (round-trippable node model)
|
||
pkg/parser/ — recursive-descent parser → CST (DML + DDL + PL/pgSQL)
|
||
pkg/format/ — Doc-IR printer (Wadler/Prettier-style) + style application
|
||
pkg/config/ — .pgtidy.yaml discovery/merge: style + rule config
|
||
pkg/diagnostics/ — shared diagnostic type (CLI + LSP)
|
||
pkg/pgast/ — go-pgquery wrapper: SQL → real PG AST (lint, v2)
|
||
pkg/lint/ — rule engine + rule packs (v2)
|
||
pkg/lsp/ — LSP server (v3)
|
||
editors/vscode/ — VSCode extension (v3)
|
||
editors/datagrip/ — native JetBrains plugin, shells out to the CLI (v4)
|
||
testdata/corpus/ — real-world .pgsql procedures used as the safety/idempotence harness
|
||
```
|
||
|
||
## Why this architecture
|
||
|
||
- **No cgo** (WASM-embedded libpg_query) keeps cross-compilation trivial and lets us bundle a
|
||
single static binary per platform inside the VSCode extension.
|
||
- **Formatting needs a lossless CST** — libpg_query drops comments and whitespace, so it cannot
|
||
be the sole basis for a formatter. We build our own lexer/CST so the formatter never loses a
|
||
comment.
|
||
- **Deep lint needs an accurate AST** — go-pgquery gives the real PostgreSQL parse tree.
|
||
- **One core, many frontends** — CLI, LSP, VSCode, and DataGrip all reuse the same engine and
|
||
the same `diagnostics` type.
|
||
|
||
## Core invariants (the formatter must never violate these)
|
||
|
||
1. **Lossless lex**: `emit(lex(src)) == src` byte-for-byte. The lexer keeps all trivia.
|
||
2. **Semantic equivalence**: formatting only changes trivia/layout. Verify by re-lexing the
|
||
output and comparing the non-trivia token stream against the input.
|
||
3. **Idempotence**: `fmt(fmt(x)) == fmt(x)`.
|
||
4. **Graceful degradation**: any span the parser cannot handle is passed through verbatim
|
||
rather than corrupted.
|
||
5. **Runtime safety gate**: every frontend (CLI `fmt`, LSP `textDocument/formatting` and
|
||
`rangeFormatting`) calls `format.VerifySafe(src, out, style)` before writing or returning
|
||
formatted output. If it returns a non-nil error the formatter has a bug — the caller must
|
||
refuse to write/emit the result and keep the original source, never guess or best-effort
|
||
it. This is enforced at runtime, not just in `pkg/format/*_test.go`, so a formatter bug can
|
||
never silently drop or alter code. `VerifySafe` runs four checks:
|
||
- **Semantic equivalence** (`SemanticallyEqual`): re-lex both sides, compare the non-trivia
|
||
token streams — case-insensitive for identifiers/keywords, exact otherwise, recursing
|
||
into dollar-quoted bodies. Comments and whitespace are trivia and are ignored here.
|
||
- **Comment preservation** (`CommentsPreserved`): every `--` and `/* */` comment in `src`
|
||
reappears in `out`, in order, with the same content (line endings and indentation are
|
||
normalised away; dropping, merging, splitting, reordering, or rewording a comment is
|
||
not). Descends into dollar-quoted bodies.
|
||
- **Structural balance** (`StructurallyBalanced`): the `( ) [ ]` and BEGIN/CASE/IF/LOOP…END
|
||
nesting profile of `out` matches `src`, counting only real code tokens (comment and
|
||
string/dollar-quote contents are skipped).
|
||
- **Idempotence**: re-formatting `out` yields `out` unchanged.
|
||
|
||
**Line endings**: the formatter re-emits all layout with `st.Newline` (default `\n`), so a
|
||
CRLF input file is normalised to LF on write. This includes `\r\n` that sits *inside* a
|
||
multi-line string literal — a line-ending change there is layout normalisation, not a change
|
||
of code content, so `SemanticallyEqual` compares string tokens modulo `\r\n` ↔ `\n`
|
||
(`normNL` in `pkg/format/safety.go`). Set `newline: "\r\n"` in `.pgtidy.yaml` to keep CRLF.
|
||
|
||
## Commands
|
||
|
||
```bash
|
||
go build ./cmd/pgtidy # Build the binary
|
||
go test ./... # Run all tests (incl. corpus safety harness)
|
||
go vet ./... # Static analysis
|
||
go fmt ./... # Format Go code
|
||
```
|
||
|
||
## House style (formatter defaults; all configurable via .pgtidy.yaml)
|
||
|
||
- Keywords UPPERCASE; data types lowercase; identifiers lowercase snake_case.
|
||
- Indent: 2 spaces per level.
|
||
- Leading-comma style, one item per line, for SELECT column lists and function params.
|
||
- Function headers: params one-per-line in parens; LANGUAGE / SECURITY / volatility each on
|
||
their own line; `AS` then `$$` on its own line; body; closing `$$;` on its own line.
|
||
- PL/pgSQL: `DECLARE` alone, vars 2-space indented; `--Block--` comment markers preserved;
|
||
`BEGIN`/`END` at body level; IF/ELSIF/ELSE/END IF, loops, CASE indent their bodies.
|
||
- Spacing: spaces around binary operators (`=`, `<>`, `||`, …) and `:=`; no space around
|
||
`::`, `->`, `->>`, array `[...]`, or before a call's `(`.
|
||
- Dollar-quote tags preserved verbatim (`$$`, `$S$`, `$Z$`, …).
|
||
|
||
## Milestones
|
||
|
||
### V1 — Formatter + CLI (current priority)
|
||
1. **Lexer** (`pkg/lexer`): full PG token coverage incl. dollar-quoted strings, `--` and
|
||
`/* */` comments, operators. Comments + whitespace as leading/trailing trivia on tokens.
|
||
Acceptance: `emit(lex(src)) == src` byte-for-byte across the corpus.
|
||
2. **CST + parser** (`pkg/cst`, `pkg/parser`): recursive descent for DML (SELECT/INSERT/
|
||
UPDATE/DELETE/CTE), DDL (CREATE FUNCTION/PROCEDURE/TABLE/INDEX/TRIGGER, ALTER, DO).
|
||
3. **PL/pgSQL body parser**: DECLARE/BEGIN/END, IF/CASE/LOOP, assignments, nested SQL.
|
||
4. **Printer** (`pkg/format`): Doc-IR (group/indent/line/softline) driven by `style` config.
|
||
Graceful degradation — unparsed spans pass through verbatim.
|
||
5. **CLI** (`cmd/pgtidy fmt`): `--check`, `--write`/`-w`, stdin→stdout, `--diff`; config
|
||
discovery walking up to `.pgtidy.yaml`; CI-friendly exit codes.
|
||
6. **Config** (`pkg/config`): load/merge style config; defaults = house style above.
|
||
|
||
### V2 — Linter
|
||
- `pkg/pgast` go-pgquery (WASM) wrapper; `pkg/lint` rule engine (rule ID, severity, config).
|
||
- Rule packs: style/consistency, migration safety (ACCESS EXCLUSIVE locks, unsafe ALTER/ADD
|
||
COLUMN, non-CONCURRENTLY index builds, blocking constraints), naming conventions,
|
||
correctness/anti-patterns (SELECT *, missing WHERE on UPDATE/DELETE). Emit `diagnostics`.
|
||
- `pgtidy lint` subcommand; `--fix` for autofixable rules.
|
||
|
||
### V3 — LSP + VSCode
|
||
- `pkg/lsp`: `textDocument/formatting` + range formatting, `publishDiagnostics`, `codeAction`
|
||
quick-fixes — all reusing the core.
|
||
- `editors/vscode`: TS extension using `vscode-languageclient`, launches bundled `pgtidy lsp`.
|
||
Per-platform VSIX (`win32/linux/darwin × x64/arm64`) built in CI matrix.
|
||
|
||
### V4 — DataGrip
|
||
- `editors/datagrip`: native JetBrains plugin (no LSP4IJ dependency) that shells out to the
|
||
`pgtidy` binary directly. Formatting and version info run `pgtidy fmt`/`version`; lint
|
||
diagnostics and quick-fixes run via a native `ExternalAnnotator` calling `pgtidy lint --json`.
|
||
|
||
## Verification
|
||
- **Formatter:** `go test ./...` runs golden-file tests + corpus harness asserting idempotence
|
||
and token-stream equality before/after (no semantic change).
|
||
- **CLI:** `pgtidy fmt --check` returns non-zero on unformatted input, zero when clean.
|
||
- **Lint (v2):** fixture SQL with known violations → assert expected diagnostics; `--fix`
|
||
round-trips.
|
||
- **LSP/VSCode (v3):** load a `.sql` file in a dev-host VSCode, confirm format-on-save and
|
||
live diagnostics via the bundled binary.
|