diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..b836be9 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,124 @@ +## Project Overview + +**PgTidy** is a PostgreSQL-focused linter and formatter written in Go. It enforces a +consistent SQL style, detects PostgreSQL-specific issues, and formats migrations, schemas, +functions, procedures, triggers, and related database code. The primary day-one use case is +formatting PL/pgSQL stored procedures to a defined house style. + +It ships as a CLI, an LSP server, a VSCode extension, and (later) a DataGrip integration. + +# Agent Rules + +Keep your answers short. Question everything, never assume or guess. Ask the user if you are +unsure about anything. + +# Architecture + +PgTidy uses a **hybrid** strategy: + +- **Formatting** is driven by a custom **lossless lexer + CST** (concrete syntax tree) that + preserves every comment and whitespace span, so formatting can round-trip safely. +- **Linting** (later milestones) is driven by the real PostgreSQL grammar via + `go-pgquery` (libpg_query compiled to WASM with wazero — no cgo), which yields an accurate + AST for deep semantic rules. + +A single Go core backs every frontend (CLI, LSP, editors) and shares one `diagnostics` type. + +``` +cmd/pgtidy/ — CLI entry; subcommands: fmt, lint (v2), lsp (v3), version +pkg/lexer/ — lossless lexer: tokens + trivia (comments/whitespace) attached +pkg/cst/ — concrete syntax tree (round-trippable node model) +pkg/parser/ — recursive-descent parser → CST (DML + DDL + PL/pgSQL) +pkg/format/ — Doc-IR printer (Wadler/Prettier-style) + style application +pkg/config/ — .pgtidy.yaml discovery/merge: style + rule config +pkg/diagnostics/ — shared diagnostic type (CLI + LSP) +pkg/pgast/ — go-pgquery wrapper: SQL → real PG AST (lint, v2) +pkg/lint/ — rule engine + rule packs (v2) +pkg/lsp/ — LSP server (v3) +editors/vscode/ — VSCode extension (v3) +editors/datagrip/ — LSP4IJ integration (v4) +testdata/corpus/ — real-world .pgsql procedures used as the safety/idempotence harness +``` + +## Why this architecture + +- **No cgo** (WASM-embedded libpg_query) keeps cross-compilation trivial and lets us bundle a + single static binary per platform inside the VSCode extension. +- **Formatting needs a lossless CST** — libpg_query drops comments and whitespace, so it cannot + be the sole basis for a formatter. We build our own lexer/CST so the formatter never loses a + comment. +- **Deep lint needs an accurate AST** — go-pgquery gives the real PostgreSQL parse tree. +- **One core, many frontends** — CLI, LSP, VSCode, and DataGrip all reuse the same engine and + the same `diagnostics` type. + +## Core invariants (the formatter must never violate these) + +1. **Lossless lex**: `emit(lex(src)) == src` byte-for-byte. The lexer keeps all trivia. +2. **Semantic equivalence**: formatting only changes trivia/layout. Verify by re-lexing the + output and comparing the non-trivia token stream against the input. +3. **Idempotence**: `fmt(fmt(x)) == fmt(x)`. +4. **Graceful degradation**: any span the parser cannot handle is passed through verbatim + rather than corrupted. + +## Commands + +```bash +go build ./cmd/pgtidy # Build the binary +go test ./... # Run all tests (incl. corpus safety harness) +go vet ./... # Static analysis +go fmt ./... # Format Go code +``` + +## House style (formatter defaults; all configurable via .pgtidy.yaml) + +- Keywords UPPERCASE; data types lowercase; identifiers lowercase snake_case. +- Indent: 2 spaces per level. +- Leading-comma style, one item per line, for SELECT column lists and function params. +- Function headers: params one-per-line in parens; LANGUAGE / SECURITY / volatility each on + their own line; `AS` then `$$` on its own line; body; closing `$$;` on its own line. +- PL/pgSQL: `DECLARE` alone, vars 2-space indented; `--Block--` comment markers preserved; + `BEGIN`/`END` at body level; IF/ELSIF/ELSE/END IF, loops, CASE indent their bodies. +- Spacing: spaces around binary operators (`=`, `<>`, `||`, …) and `:=`; no space around + `::`, `->`, `->>`, array `[...]`, or before a call's `(`. +- Dollar-quote tags preserved verbatim (`$$`, `$S$`, `$Z$`, …). + +## Milestones + +### V1 — Formatter + CLI (current priority) +1. **Lexer** (`pkg/lexer`): full PG token coverage incl. dollar-quoted strings, `--` and + `/* */` comments, operators. Comments + whitespace as leading/trailing trivia on tokens. + Acceptance: `emit(lex(src)) == src` byte-for-byte across the corpus. +2. **CST + parser** (`pkg/cst`, `pkg/parser`): recursive descent for DML (SELECT/INSERT/ + UPDATE/DELETE/CTE), DDL (CREATE FUNCTION/PROCEDURE/TABLE/INDEX/TRIGGER, ALTER, DO). +3. **PL/pgSQL body parser**: DECLARE/BEGIN/END, IF/CASE/LOOP, assignments, nested SQL. +4. **Printer** (`pkg/format`): Doc-IR (group/indent/line/softline) driven by `style` config. + Graceful degradation — unparsed spans pass through verbatim. +5. **CLI** (`cmd/pgtidy fmt`): `--check`, `--write`/`-w`, stdin→stdout, `--diff`; config + discovery walking up to `.pgtidy.yaml`; CI-friendly exit codes. +6. **Config** (`pkg/config`): load/merge style config; defaults = house style above. + +### V2 — Linter +- `pkg/pgast` go-pgquery (WASM) wrapper; `pkg/lint` rule engine (rule ID, severity, config). +- Rule packs: style/consistency, migration safety (ACCESS EXCLUSIVE locks, unsafe ALTER/ADD + COLUMN, non-CONCURRENTLY index builds, blocking constraints), naming conventions, + correctness/anti-patterns (SELECT *, missing WHERE on UPDATE/DELETE). Emit `diagnostics`. +- `pgtidy lint` subcommand; `--fix` for autofixable rules. + +### V3 — LSP + VSCode +- `pkg/lsp`: `textDocument/formatting` + range formatting, `publishDiagnostics`, `codeAction` + quick-fixes — all reusing the core. +- `editors/vscode`: TS extension using `vscode-languageclient`, launches bundled `pgtidy lsp`. + Per-platform VSIX (`win32/linux/darwin × x64/arm64`) built in CI matrix. + +### V4 — DataGrip +- `editors/datagrip`: integrate via LSP4IJ (free, works across JetBrains editions incl. + DataGrip). No core changes expected. + +## Verification +- **Formatter:** `go test ./...` runs golden-file tests + corpus harness asserting idempotence + and token-stream equality before/after (no semantic change). +- **CLI:** `pgtidy fmt --check` returns non-zero on unformatted input, zero when clean. +- **Lint (v2):** fixture SQL with known violations → assert expected diagnostics; `--fix` + round-trips. +- **LSP/VSCode (v3):** load a `.sql` file in a dev-host VSCode, confirm format-on-save and + live diagnostics via the bundled binary. diff --git a/AI_USE.md b/AI_USE.md new file mode 100644 index 0000000..397857c --- /dev/null +++ b/AI_USE.md @@ -0,0 +1,35 @@ +# AI Usage Declaration + +This Go project utilizes AI tools for the following purposes: + +- Generating and improving documentation +- Writing and enhancing tests +- Refactoring and optimizing existing code + +AI is **not** used for core design or architecture decisions. +All design decisions are deferred to human discussion. +AI is employed only for enhancements to human-written code. + +We are aware of significant AI hallucinations; all AI-generated content is to be reviewed and verified by humans. + + + .-""""""-. + .' '. + / O O \ + : ` : + | | + : .------. : + \ ' ' / + '. .' + '-......-' + MEGAMIND AI + [============] + + ___________ + /___________\ + /_____________\ + | ASSIMILATE | + | RESISTANCE | + | IS FUTILE | + \_____________/ + \___________/ diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..548047a --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,4 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. +You must now read the ./AGENTS.md diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..98eb0ae --- /dev/null +++ b/Makefile @@ -0,0 +1,29 @@ +APP := pgtidy +CMD := ./cmd/pgtidy +DIST := dist + +.PHONY: build test lint vet fmt clean + +## build: compile binary for the current platform +build: + go build -trimpath -ldflags "-s -w -X main.version=$$(git describe --tags --abbrev=0 2>/dev/null || echo dev)" -o $(DIST)/$(APP) $(CMD) + +## test: run tests (incl. corpus safety harness) +test: + go test ./... + +## vet: static analysis +vet: + go vet ./... + +## fmt: format Go code +fmt: + go fmt ./... + +## lint: vet + format check +lint: vet + test -z "$$(gofmt -l .)" || (echo "gofmt needed:"; gofmt -l .; exit 1) + +## clean: remove build artifacts +clean: + rm -rf $(DIST) diff --git a/cmd/pgtidy/fmt.go b/cmd/pgtidy/fmt.go new file mode 100644 index 0000000..0a02073 --- /dev/null +++ b/cmd/pgtidy/fmt.go @@ -0,0 +1,100 @@ +package main + +import ( + "fmt" + "io" + "os" + + "github.com/hein/pgtidy/pkg/config" + "github.com/hein/pgtidy/pkg/format" + "github.com/hein/pgtidy/pkg/parser" +) + +// cmdFmt implements `pgtidy fmt`. It follows the gofmt model: with no flags it +// prints the formatted result to stdout; -w rewrites in place; -l lists files +// that differ; --check exits non-zero if any input is unformatted. +func cmdFmt(args []string, stdin io.Reader, stdout, stderr io.Writer) int { + var ( + write bool + list bool + check bool + files []string + ) + for _, a := range args { + switch a { + case "-w", "--write": + write = true + case "-l", "--list": + list = true + case "--check": + check = true + case "-h", "--help": + usage(stdout) + return 0 + default: + if len(a) > 1 && a[0] == '-' { + fmt.Fprintf(stderr, "pgtidy fmt: unknown flag %q\n", a) + return 2 + } + files = append(files, a) + } + } + + // TODO: discover and parse .pgtidy.yaml; for now use the house-style default. + st := config.Default() + + // stdin → stdout when no files are given. + if len(files) == 0 { + src, err := io.ReadAll(stdin) + if err != nil { + fmt.Fprintf(stderr, "pgtidy: reading stdin: %v\n", err) + return 2 + } + out := format.File(parser.Parse(string(src)), st) + if check { + if out != string(src) { + return 1 + } + return 0 + } + io.WriteString(stdout, out) + return 0 + } + + exit := 0 + anyDiff := false + for _, path := range files { + src, err := os.ReadFile(path) + if err != nil { + fmt.Fprintf(stderr, "pgtidy: %v\n", err) + exit = 2 + continue + } + out := format.File(parser.Parse(string(src)), st) + changed := out != string(src) + if changed { + anyDiff = true + } + switch { + case write: + if changed { + if err := os.WriteFile(path, []byte(out), 0o644); err != nil { + fmt.Fprintf(stderr, "pgtidy: writing %s: %v\n", path, err) + exit = 2 + } + } + case list: + if changed { + fmt.Fprintln(stdout, path) + } + case check: + // handled after loop via anyDiff + default: + io.WriteString(stdout, out) + } + } + if check && anyDiff && exit == 0 { + return 1 + } + return exit +} diff --git a/cmd/pgtidy/fmt_test.go b/cmd/pgtidy/fmt_test.go new file mode 100644 index 0000000..85dbb5c --- /dev/null +++ b/cmd/pgtidy/fmt_test.go @@ -0,0 +1,69 @@ +package main + +import ( + "bytes" + "os" + "path/filepath" + "strings" + "testing" +) + +func TestFmtStdin(t *testing.T) { + in := strings.NewReader("create function f() returns int language sql as $$ select 1 $$;\n") + var out, errb bytes.Buffer + rc := run([]string{"fmt"}, in, &out, &errb) + if rc != 0 { + t.Fatalf("rc=%d stderr=%s", rc, errb.String()) + } + if !strings.HasPrefix(out.String(), "CREATE FUNCTION f(") { + t.Errorf("unexpected output:\n%s", out.String()) + } +} + +func TestFmtCheckStdin(t *testing.T) { + unformatted := "create function f() returns int language sql as $$ select 1 $$;" + + // Unformatted input → rc 1. + var out, errb bytes.Buffer + if rc := run([]string{"fmt", "--check"}, strings.NewReader(unformatted), &out, &errb); rc != 1 { + t.Errorf("unformatted input: rc=%d, want 1", rc) + } + + // Its own formatted output → rc 0 (idempotent + check agree). + out.Reset() + errb.Reset() + run([]string{"fmt"}, strings.NewReader(unformatted), &out, &errb) + formatted := out.String() + var out2, errb2 bytes.Buffer + if rc := run([]string{"fmt", "--check"}, strings.NewReader(formatted), &out2, &errb2); rc != 0 { + t.Errorf("formatted input: rc=%d, want 0\noutput was:\n%s", rc, formatted) + } +} + +func TestFmtWriteInPlace(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "f.pgsql") + orig := "create function f() returns int language sql as $$ select 1 $$;\n" + if err := os.WriteFile(path, []byte(orig), 0o644); err != nil { + t.Fatal(err) + } + var out, errb bytes.Buffer + if rc := run([]string{"fmt", "-w", path}, nil, &out, &errb); rc != 0 { + t.Fatalf("rc=%d stderr=%s", rc, errb.String()) + } + got, _ := os.ReadFile(path) + if string(got) == orig { + t.Error("file was not rewritten") + } + // Second write is a no-op (idempotent). + if rc := run([]string{"fmt", "--check", path}, nil, &out, &errb); rc != 0 { + t.Errorf("after -w, --check rc=%d, want 0", rc) + } +} + +func TestUnknownCommand(t *testing.T) { + var out, errb bytes.Buffer + if rc := run([]string{"frobnicate"}, nil, &out, &errb); rc != 2 { + t.Errorf("rc=%d, want 2", rc) + } +} diff --git a/cmd/pgtidy/main.go b/cmd/pgtidy/main.go new file mode 100644 index 0000000..a4f997c --- /dev/null +++ b/cmd/pgtidy/main.go @@ -0,0 +1,52 @@ +// Command pgtidy is the PgTidy CLI: a PostgreSQL formatter (and, later, linter) +// and LSP server. Today it provides the `fmt` subcommand. +package main + +import ( + "fmt" + "io" + "os" +) + +// version is overridden at build time via -ldflags "-X main.version=...". +var version = "dev" + +func main() { + os.Exit(run(os.Args[1:], os.Stdin, os.Stdout, os.Stderr)) +} + +func run(args []string, stdin io.Reader, stdout, stderr io.Writer) int { + if len(args) == 0 { + usage(stderr) + return 2 + } + switch args[0] { + case "fmt", "format": + return cmdFmt(args[1:], stdin, stdout, stderr) + case "version", "--version", "-v": + fmt.Fprintf(stdout, "pgtidy %s\n", version) + return 0 + case "help", "-h", "--help": + usage(stdout) + return 0 + default: + fmt.Fprintf(stderr, "pgtidy: unknown command %q\n", args[0]) + usage(stderr) + return 2 + } +} + +func usage(w io.Writer) { + fmt.Fprint(w, `pgtidy — PostgreSQL formatter and linter + +Usage: + pgtidy fmt [flags] [files...] Format SQL/PL-pgSQL (stdin if no files) + pgtidy version Print version + pgtidy help Show this help + +fmt flags: + -w, --write Rewrite files in place + -l, --list List files whose formatting differs (no writes) + --check Exit non-zero if any input is not already formatted (CI) +`) +} diff --git a/go.mod b/go.mod new file mode 100644 index 0000000..72681b2 --- /dev/null +++ b/go.mod @@ -0,0 +1,3 @@ +module github.com/hein/pgtidy + +go 1.26 diff --git a/make_release.sh b/make_release.sh new file mode 100755 index 0000000..8831ad9 --- /dev/null +++ b/make_release.sh @@ -0,0 +1,55 @@ +#!/bin/bash + +# Detect the latest version tag +latest_tag=$(git tag --sort=-v:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | head -1) + +if [[ -z "$latest_tag" ]]; then + suggested_version="v1.0.0" +else + # Increment the patch version + IFS='.' read -r major minor patch <<< "${latest_tag#v}" + suggested_version="v${major}.${minor}.$((patch + 1))" +fi + +echo "Latest tag: ${latest_tag:-none}" +echo "Suggested next version: $suggested_version" + +read -p "Do you want to make a release version? (y/n): " make_release + +if [[ $make_release =~ ^[Yy]$ ]]; then + read -p "Enter the version number [$suggested_version]: " version + version="${version:-$suggested_version}" + + # Validate the version number + if ! [[ $version =~ ^(v)?([0-9]+\.){0,2}[0-9]+(\.[0-9]+)?(\.beta)?$ ]]; then + echo "Invalid version number format. Please use a valid format like 'v1.0.0'" + exit 1 + fi + + # Prepend 'v' to the version if it doesn't start with it + if ! [[ $version =~ ^v ]]; then + version="v$version" + fi + + # Build tag message with recent commits since last tag + if [[ -n "$latest_tag" ]]; then + commit_log=$(git log "${latest_tag}..HEAD" --oneline --no-decorate) + else + commit_log=$(git log --oneline --no-decorate -20) + fi + + tag_message="Released $version + +Changes since ${latest_tag:-beginning}: +$commit_log" + + # Create an annotated tag + git tag -a "$version" -m "$tag_message" + + # Push the tag to the remote repository + git push origin "$version" + + echo "Tag $version created and pushed to the remote repository." +else + echo "No release version created." +fi diff --git a/pkg/config/config.go b/pkg/config/config.go new file mode 100644 index 0000000..1f8c0d9 --- /dev/null +++ b/pkg/config/config.go @@ -0,0 +1,55 @@ +// Package config defines PgTidy's formatter (and, later, linter) configuration. +// +// Defaults encode the project house style reverse-engineered from the corpus. +// A future change will load/merge these from a discovered .pgtidy.yaml file. +package config + +// Case controls keyword/identifier casing. +type Case string + +const ( + CaseUpper Case = "upper" + CaseLower Case = "lower" + // CasePreserve leaves the token text unchanged. + CasePreserve Case = "preserve" +) + +// CommaStyle controls where separators sit in multi-line lists. +type CommaStyle string + +const ( + // CommaLeading puts the comma at the start of the continuation line + // (",col"), the house style. + CommaLeading CommaStyle = "leading" + // CommaTrailing puts the comma at the end of the preceding line ("col,"). + CommaTrailing CommaStyle = "trailing" +) + +// Style is the formatter configuration. +type Style struct { + // Indent is one indentation level (default two spaces). + Indent string + // Newline is the line terminator emitted by the formatter. + Newline string + // KeywordCase controls SQL keyword casing (types excluded — see TypeCase). + KeywordCase Case + // IdentCase controls unquoted identifier casing (quoted identifiers are + // never touched). + IdentCase Case + // TypeCase controls built-in type-name casing. + TypeCase Case + // Commas controls list separator placement. + Commas CommaStyle +} + +// Default returns the house-style configuration. +func Default() Style { + return Style{ + Indent: " ", + Newline: "\n", + KeywordCase: CaseUpper, + IdentCase: CaseLower, + TypeCase: CaseLower, + Commas: CommaLeading, + } +} diff --git a/pkg/cst/cst.go b/pkg/cst/cst.go new file mode 100644 index 0000000..5355061 --- /dev/null +++ b/pkg/cst/cst.go @@ -0,0 +1,173 @@ +// Package cst defines PgTidy's concrete syntax tree. +// +// The CST is lossless: every lexer token (significant tokens and the trivia — +// whitespace/comments — attached to them) is reachable, so File.Source() +// reproduces the original input byte-for-byte. The parser builds structured +// nodes only where it is confident; everything else is captured verbatim in a +// Raw node. This makes "graceful degradation" a property of the data model +// rather than something the printer must remember to do. +package cst + +import ( + "strings" + + "github.com/hein/pgtidy/pkg/lexer" +) + +// Trivia is a run of whitespace/comment tokens preceding a significant token. +type Trivia []lexer.Token + +// Tok is a significant (non-trivia) token together with the trivia that +// immediately precedes it in the source. +type Tok struct { + Lead Trivia + Tok lexer.Token +} + +// Text returns the significant token's text. +func (t Tok) Text() string { return t.Tok.Text } + +// Is reports whether the token is an unquoted word equal (case-insensitively) +// to kw. +func (t Tok) Is(kw string) bool { + return t.Tok.Kind == lexer.Ident && strings.EqualFold(t.Tok.Text, kw) +} + +// Comments returns just the comment tokens from the leading trivia. +func (t Tok) Comments() []lexer.Token { + var out []lexer.Token + for _, tr := range t.Lead { + if tr.Kind == lexer.LineComment || tr.Kind == lexer.BlockComment { + out = append(out, tr) + } + } + return out +} + +// Attach converts a lossless lexer stream into significant tokens, each +// carrying its leading trivia, plus any trailing trivia before EOF. +func Attach(toks []lexer.Token) (sig []Tok, trailing Trivia) { + var lead Trivia + for _, t := range toks { + if t.Kind == lexer.EOF { + trailing = lead + break + } + if t.IsTrivia() { + lead = append(lead, t) + continue + } + sig = append(sig, Tok{Lead: lead, Tok: t}) + lead = nil + } + return sig, trailing +} + +// Node is any element of a File. +type Node interface { + appendTokens(*[]Tok) +} + +// Tokens returns all significant tokens of a node in source order. +func Tokens(n Node) []Tok { + var t []Tok + n.appendTokens(&t) + return t +} + +// File is the whole parsed input. +type File struct { + Items []Node + Trailing Trivia // trivia after the last significant token +} + +// Source reconstructs the original input from the tree. With a correct parse +// this is byte-for-byte identical to the lexer input. +func (f *File) Source() string { + var toks []Tok + for _, it := range f.Items { + it.appendTokens(&toks) + } + var b strings.Builder + for _, t := range toks { + for _, tr := range t.Lead { + b.WriteString(tr.Text) + } + b.WriteString(t.Tok.Text) + } + for _, tr := range f.Trailing { + b.WriteString(tr.Text) + } + return b.String() +} + +// Raw is an unstructured statement: a verbatim run of significant tokens. This +// is the graceful-degradation fallback for anything the parser does not (yet) +// structure. +type Raw struct { + Toks []Tok +} + +func (r *Raw) appendTokens(out *[]Tok) { *out = append(*out, r.Toks...) } + +// Param is one entry in a function/procedure parameter list, with the comma +// that separates it from the next entry (nil on the last parameter). +type Param struct { + Toks []Tok + Sep *Tok // trailing comma, nil if last +} + +// CreateFunction is a parsed CREATE [OR REPLACE] FUNCTION|PROCEDURE statement. +// +// Field order matches source order; appendTokens emits them contiguously so the +// node round-trips exactly. +type CreateFunction struct { + Head []Tok // CREATE [OR REPLACE] FUNCTION|PROCEDURE + Name []Tok // name (possibly schema-qualified) + LParen Tok // + Params []Param // parameter list (may be empty) + RParen Tok // + Options [][]Tok // option clauses before the body (RETURNS/LANGUAGE/VOLATILE/...) + As *Tok // the AS keyword, if present + Body *Tok // the body token (dollar-quoted string or string), if present + Tail [][]Tok // option clauses after the body (rare) + Semi *Tok // terminating semicolon, if present +} + +// IsProcedure reports whether this is a PROCEDURE (vs FUNCTION). +func (c *CreateFunction) IsProcedure() bool { + for _, t := range c.Head { + if t.Is("procedure") { + return true + } + } + return false +} + +func (c *CreateFunction) appendTokens(out *[]Tok) { + *out = append(*out, c.Head...) + *out = append(*out, c.Name...) + *out = append(*out, c.LParen) + for _, p := range c.Params { + *out = append(*out, p.Toks...) + if p.Sep != nil { + *out = append(*out, *p.Sep) + } + } + *out = append(*out, c.RParen) + for _, cl := range c.Options { + *out = append(*out, cl...) + } + if c.As != nil { + *out = append(*out, *c.As) + } + if c.Body != nil { + *out = append(*out, *c.Body) + } + for _, cl := range c.Tail { + *out = append(*out, cl...) + } + if c.Semi != nil { + *out = append(*out, *c.Semi) + } +} diff --git a/pkg/format/format.go b/pkg/format/format.go new file mode 100644 index 0000000..ba8a4ab --- /dev/null +++ b/pkg/format/format.go @@ -0,0 +1,310 @@ +// Package format renders a cst.File back to source text in the configured +// house style. +// +// Scope (V1): CREATE FUNCTION/PROCEDURE headers are laid out to house style +// (one parameter per line with leading commas, each option clause on its own +// line, AS/$$ on their own lines). The PL/pgSQL body and any statement the +// parser left as cst.Raw are emitted verbatim — this upholds the safety +// invariants while body formatting is built out (task #4). +// +// Guarantees: formatting changes only trivia/layout (never literal/identifier +// semantics), and is idempotent. +package format + +import ( + "strings" + + "github.com/hein/pgtidy/pkg/config" + "github.com/hein/pgtidy/pkg/cst" + "github.com/hein/pgtidy/pkg/lexer" +) + +// File formats a parsed file with the given style. +func File(f *cst.File, st config.Style) string { + p := &printer{st: st} + for i, item := range f.Items { + toks := cst.Tokens(item) + if len(toks) == 0 { + continue + } + lead := toks[0].Lead + if i > 0 { + p.nl() + if hasBlankLine(lead) { + p.nl() + } + } + p.leadingComments(lead) + p.writeItem(item) + } + p.trailingComments(f.Trailing) + return ensureTrailingNewline(p.b.String(), st.Newline) +} + +type printer struct { + st config.Style + b strings.Builder +} + +func (p *printer) nl() { p.b.WriteString(p.st.Newline) } + +func (p *printer) writeItem(n cst.Node) { + switch v := n.(type) { + case *cst.CreateFunction: + p.writeCreateFunction(v) + default: + p.b.WriteString(verbatimSpan(cst.Tokens(n))) + } +} + +func (p *printer) writeCreateFunction(cf *cst.CreateFunction) { + // Safety: if the header carries comments we cannot confidently relocate, + // emit the whole statement verbatim rather than risk dropping them. + if headerHasComments(cf) { + p.b.WriteString(verbatimSpan(cst.Tokens(cf))) + return + } + + head := p.inline(cf.Head) + name := p.inline(cf.Name) + p.b.WriteString(head) + if name != "" { + p.b.WriteString(" ") + p.b.WriteString(name) + } + p.b.WriteString("(") + + first := p.st.Indent + " " // align item text one column past the comma + cont := p.st.Indent + for i, param := range cf.Params { + p.nl() + text := p.inline(param.Toks) + if i == 0 || p.st.Commas != config.CommaLeading { + p.b.WriteString(first) + p.b.WriteString(text) + if p.st.Commas == config.CommaTrailing && i < len(cf.Params)-1 { + p.b.WriteString(",") + } + } else { + p.b.WriteString(cont) + p.b.WriteString(",") + p.b.WriteString(text) + } + } + p.nl() + p.b.WriteString(")") + + for _, clause := range cf.Options { + p.nl() + p.b.WriteString(p.inline(clause)) + } + if cf.As != nil { + p.nl() + p.b.WriteString(p.inline([]cst.Tok{{Tok: cf.As.Tok}})) + } + if cf.Body != nil { + p.nl() + p.b.WriteString(cf.Body.Tok.Text) // body emitted verbatim (formatted later) + } + for _, clause := range cf.Tail { + p.nl() + p.b.WriteString(p.inline(clause)) + } + if cf.Semi != nil { + p.b.WriteString(";") + } +} + +// inline renders a run of tokens on one line, applying spacing and casing. +// If the run contains comment trivia it is emitted verbatim to avoid losing +// or misplacing the comments. +func (p *printer) inline(toks []cst.Tok) string { + if len(toks) == 0 { + return "" + } + // A comment on the first token's lead is the span's leading/separation + // comment, handled by the caller and never emitted here, so it does not + // force verbatim. Internal comments (on later tokens) do. + if anyComment(toks[1:]) { + return verbatimSpan(toks) + } + var b strings.Builder + for i, t := range toks { + if i > 0 && needSpace(toks[i-1].Tok, t.Tok) { + b.WriteByte(' ') + } + b.WriteString(caseText(t.Tok, p.st)) + } + return b.String() +} + +func (p *printer) leadingComments(lead cst.Trivia) { + for _, tr := range lead { + if tr.Kind == lexer.LineComment || tr.Kind == lexer.BlockComment { + p.b.WriteString(strings.TrimRight(tr.Text, " \t")) + p.nl() + } + } +} + +func (p *printer) trailingComments(lead cst.Trivia) { + cs := commentsOf(lead) + for _, c := range cs { + p.nl() + p.b.WriteString(strings.TrimRight(c.Text, " \t")) + } +} + +// --- spacing & casing --- + +// tightOps are operators printed without surrounding spaces. +var tightOps = map[string]bool{"::": true, ":": true, "->": true, "->>": true} + +func needSpace(a, b lexer.Token) bool { + // No space after. + switch a.Kind { + case lexer.LParen, lexer.LBracket, lexer.Dot: + return false + case lexer.Operator: + if tightOps[a.Text] { + return false + } + } + // No space before. + switch b.Kind { + case lexer.RParen, lexer.RBracket, lexer.Comma, lexer.Semicolon, lexer.Dot: + return false + case lexer.LParen: + switch a.Kind { + case lexer.Ident, lexer.QuotedIdent, lexer.RParen, lexer.RBracket, lexer.Param: + return false // function call / type modifier + } + case lexer.Operator: + if tightOps[b.Text] { + return false + } + } + return true +} + +func caseText(t lexer.Token, st config.Style) string { + if t.Kind != lexer.Ident { + return t.Text // only unquoted words are re-cased + } + low := lowerASCII(t.Text) + switch { + case isTypeName(low): + return applyCase(t.Text, st.TypeCase) + case isKeyword(low): + return applyCase(t.Text, st.KeywordCase) + default: + return applyCase(t.Text, st.IdentCase) + } +} + +func applyCase(s string, c config.Case) string { + switch c { + case config.CaseUpper: + return strings.ToUpper(s) + case config.CaseLower: + return strings.ToLower(s) + default: + return s + } +} + +// --- helpers --- + +// headerHasComments reports whether the function header carries comment trivia +// the formatter cannot confidently relocate. The first token's leading trivia +// is excluded: that is the statement's leading comment, which File() emits +// separately. The body token's own text is excluded too (it is emitted +// verbatim), but a comment in front of the body is caught. +func headerHasComments(cf *cst.CreateFunction) bool { + all := cst.Tokens(cf) + for i, t := range all { + if i == 0 { + continue + } + if t.Tok.Kind == lexer.Semicolon { + continue + } + if hasComment(t) { + return true + } + } + return false +} + +func anyComment(toks []cst.Tok) bool { + for _, t := range toks { + if hasComment(t) { + return true + } + } + return false +} + +func hasComment(t cst.Tok) bool { + for _, tr := range t.Lead { + if tr.Kind == lexer.LineComment || tr.Kind == lexer.BlockComment { + return true + } + } + return false +} + +func commentsOf(lead cst.Trivia) []lexer.Token { + var out []lexer.Token + for _, tr := range lead { + if tr.Kind == lexer.LineComment || tr.Kind == lexer.BlockComment { + out = append(out, tr) + } + } + return out +} + +// verbatimSpan emits tokens exactly as in source, excluding the leading trivia +// of the first token (separation is controlled by the caller). +func verbatimSpan(toks []cst.Tok) string { + var b strings.Builder + for i, t := range toks { + if i > 0 { + for _, tr := range t.Lead { + b.WriteString(tr.Text) + } + } + b.WriteString(t.Tok.Text) + } + return b.String() +} + +// hasBlankLine reports whether leading whitespace trivia contains a blank line +// (two or more newlines), indicating the author wanted statements separated. +func hasBlankLine(lead cst.Trivia) bool { + n := 0 + for _, tr := range lead { + if tr.Kind == lexer.Whitespace { + n += strings.Count(tr.Text, "\n") + } + } + return n >= 2 +} + +func ensureTrailingNewline(s, nl string) string { + s = strings.TrimRight(s, "\n") + if s == "" { + return s + } + return s + nl +} + +func lowerASCII(s string) string { + b := []byte(s) + for i, c := range b { + if c >= 'A' && c <= 'Z' { + b[i] = c + 32 + } + } + return string(b) +} diff --git a/pkg/format/format_test.go b/pkg/format/format_test.go new file mode 100644 index 0000000..2e7fefb --- /dev/null +++ b/pkg/format/format_test.go @@ -0,0 +1,118 @@ +package format + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/hein/pgtidy/pkg/config" + "github.com/hein/pgtidy/pkg/lexer" + "github.com/hein/pgtidy/pkg/parser" +) + +func format(src string) string { + return File(parser.Parse(src), config.Default()) +} + +func TestFormatHeaderGolden(t *testing.T) { + src := "--select * from dropall('resolvespec_login');\n" + + "create or replace function resolvespec_login(\n" + + "INOUT p_data jsonb, OUT p_success boolean, OUT p_error text)\n" + + "language plpgsql volatile security definer\n" + + "as $$\nbegin end;\n$$;\n" + + want := "--select * from dropall('resolvespec_login');\n" + + "CREATE OR REPLACE FUNCTION resolvespec_login(\n" + + " INOUT p_data jsonb\n" + + " ,OUT p_success boolean\n" + + " ,OUT p_error text\n" + + ")\n" + + "LANGUAGE plpgsql\n" + + "VOLATILE\n" + + "SECURITY DEFINER\n" + + "AS\n" + + "$$\nbegin end;\n$$;\n" + + got := format(src) + if got != want { + t.Errorf("header format mismatch\n--- got ---\n%s\n--- want ---\n%s", got, want) + } +} + +func TestIdempotentSmall(t *testing.T) { + src := "create function f(a int,b text) returns void language sql as $$ select 1 $$;" + once := format(src) + twice := format(once) + if once != twice { + t.Errorf("not idempotent\n--- once ---\n%s\n--- twice ---\n%s", once, twice) + } +} + +func TestCorpusIdempotentAndSafe(t *testing.T) { + dir := filepath.Join("..", "..", "testdata", "corpus") + entries, err := os.ReadDir(dir) + if err != nil { + t.Skipf("no corpus: %v", err) + } + var seen int + for _, e := range entries { + if e.IsDir() || !strings.HasSuffix(e.Name(), ".pgsql") { + continue + } + seen++ + data, err := os.ReadFile(filepath.Join(dir, e.Name())) + if err != nil { + t.Fatal(err) + } + src := string(data) + once := format(src) + twice := format(once) + if once != twice { + t.Errorf("%s: not idempotent", e.Name()) + } + if !semanticallyEqual(src, once) { + t.Errorf("%s: formatting changed semantics", e.Name()) + } + } + if seen == 0 { + t.Skip("no corpus files") + } + t.Logf("formatted %d corpus files (idempotent + semantically equal)", seen) +} + +// semanticallyEqual compares the non-trivia token streams of two sources, +// treating unquoted identifiers/keywords case-insensitively and everything +// else (strings, numbers, dollar bodies, operators, punctuation) exactly. This +// validates that formatting changed only layout/casing, never meaning. +func semanticallyEqual(a, b string) bool { + ta := significant(a) + tb := significant(b) + if len(ta) != len(tb) { + return false + } + for i := range ta { + if ta[i].Kind != tb[i].Kind { + return false + } + if ta[i].Kind == lexer.Ident { + if !strings.EqualFold(ta[i].Text, tb[i].Text) { + return false + } + } else if ta[i].Text != tb[i].Text { + return false + } + } + return true +} + +func significant(src string) []lexer.Token { + var out []lexer.Token + for _, t := range lexer.Lex(src) { + if t.Kind == lexer.EOF || t.IsTrivia() { + continue + } + out = append(out, t) + } + return out +} diff --git a/pkg/format/keywords.go b/pkg/format/keywords.go new file mode 100644 index 0000000..8a6c06f --- /dev/null +++ b/pkg/format/keywords.go @@ -0,0 +1,55 @@ +package format + +// keywords are SQL / PL/pgSQL words the formatter may re-case via KeywordCase. +// Type-name words are deliberately excluded (see typeNames) so the house style +// can keep types lowercase while keywords are uppercased. +var keywords = words(` +add after all alter analyze and any array as asc atomic begin between by +called cascade case cast check close coalesce collate column commit +concurrently constraint create cross cube current_date current_time +current_timestamp current_user cursor declare default deferrable definer delete desc +distinct do drop each else elsif end except exception execute exists external +fetch filter first for foreach foreign from full function get grant group +grouping having if ilike immutable in index inner inout insert intersect into +into invoker is join key language last leakproof left like limit localtime +localtimestamp loop materialized natural new next no not nothing notify null +nulls of off offset old on only open or order out outer over overriding +parallel partition perform precision primary procedure raise +references refresh rename replace reset restrict return returning returns +revoke right rollback row rows safe schema secdef security select sequence set +setof some stable stacked strict table temp temporary then to transaction +trigger truncate union unique unsafe update using vacuum values variadic view +volatile when where while window with within +`) + +// typeNames are built-in/common type words kept lowercase by the house style. +var typeNames = words(` +bigint bigserial bit bool boolean box bytea char character cidr circle citext +date daterange decimal double float float4 float8 hstore inet int int2 int4 +int8 integer interval json jsonb line lseg macaddr macaddr8 money numeric oid +path pg_lsn point polygon real serial serial2 serial4 serial8 smallint +smallserial text time timestamp timestamptz timetz tsquery tsrange tstzrange +tsvector uuid varbit varchar xml +`) + +func words(s string) map[string]bool { + m := make(map[string]bool) + w := "" + for _, r := range s { + if r == ' ' || r == '\n' || r == '\t' || r == '\r' { + if w != "" { + m[w] = true + w = "" + } + continue + } + w += string(r) + } + if w != "" { + m[w] = true + } + return m +} + +func isKeyword(lower string) bool { return keywords[lower] } +func isTypeName(lower string) bool { return typeNames[lower] } diff --git a/pkg/lexer/lexer.go b/pkg/lexer/lexer.go new file mode 100644 index 0000000..38739d7 --- /dev/null +++ b/pkg/lexer/lexer.go @@ -0,0 +1,370 @@ +package lexer + +import "strings" + +// opChars are the characters PostgreSQL allows in operator names. +const opChars = "+-*/<>=~!@#%^&|`?" + +// opSpecial is the subset whose presence lets an operator end in + or -. +const opSpecial = "~!@#%^&|`?" + +// Lex tokenizes src into a lossless token stream. The concatenation of every +// returned token's Text equals src. The final token is always EOF (empty Text). +func Lex(src string) []Token { + l := &lexer{src: src, line: 1, col: 1} + return l.run() +} + +type lexer struct { + src string + pos int + line int + col int + out []Token + + // position of the token currently being scanned, captured each iteration + tokOff, tokLine, tokCol int +} + +func (l *lexer) run() []Token { + n := len(l.src) + for l.pos < n { + l.tokOff, l.tokLine, l.tokCol = l.pos, l.line, l.col + start := l.pos + c := l.src[l.pos] + switch { + case isSpace(c): + l.scanWhile(isSpace) + l.emit(Whitespace, start) + case c == '-' && l.peek(1) == '-': + l.scanLineComment() + l.emit(LineComment, start) + case c == '/' && l.peek(1) == '*': + l.scanBlockComment() + l.emit(BlockComment, start) + case c == '\'': + l.scanString('\'', false) + l.emit(String, start) + case c == '"': + l.scanString('"', false) + l.emit(QuotedIdent, start) + case c == '$': + l.scanDollar(start) + case c == '(': + l.advance(1) + l.emit(LParen, start) + case c == ')': + l.advance(1) + l.emit(RParen, start) + case c == '[': + l.advance(1) + l.emit(LBracket, start) + case c == ']': + l.advance(1) + l.emit(RBracket, start) + case c == ',': + l.advance(1) + l.emit(Comma, start) + case c == ';': + l.advance(1) + l.emit(Semicolon, start) + case c == ':': + // :: (cast), := (assign), or bare : (slice). + if l.peek(1) == ':' || l.peek(1) == '=' { + l.advance(2) + } else { + l.advance(1) + } + l.emit(Operator, start) + case c == '.': + if isDigit(l.peek(1)) { + l.scanNumber() + l.emit(Number, start) + } else { + l.advance(1) + l.emit(Dot, start) + } + case isDigit(c): + l.scanNumber() + l.emit(Number, start) + case isIdentStart(c): + l.scanWord(start) + case strings.IndexByte(opChars, c) >= 0: + l.scanOperator() + l.emit(Operator, start) + default: + l.advance(1) + l.emit(Unknown, start) + } + } + l.out = append(l.out, Token{Kind: EOF, Off: l.pos, Line: l.line, Col: l.col}) + return l.out +} + +// scanWord handles identifiers and the typed-string prefixes E' B' X' U&' / U&". +func (l *lexer) scanWord(start int) { + c := l.src[l.pos] + switch c { + case 'E', 'e': + if l.peek(1) == '\'' { + l.advance(1) + l.scanString('\'', true) + l.emit(EscapeString, start) + return + } + case 'B', 'b': + if l.peek(1) == '\'' { + l.advance(1) + l.scanString('\'', false) + l.emit(BitString, start) + return + } + case 'X', 'x': + if l.peek(1) == '\'' { + l.advance(1) + l.scanString('\'', false) + l.emit(HexString, start) + return + } + case 'U', 'u': + if l.peek(1) == '&' && (l.peek(2) == '\'' || l.peek(2) == '"') { + q := l.peek(2) + l.advance(2) + l.scanString(q, false) + l.emit(UnicodeString, start) + return + } + } + l.scanWhile(isIdentCont) + l.emit(Ident, start) +} + +// scanDollar handles dollar-quoted strings ($tag$...$tag$), positional +// parameters ($1), and a lone $. +func (l *lexer) scanDollar(start int) { + if tag, ok := l.dollarTag(); ok { + // Opening delimiter is "$tag$"; find the matching close. + open := "$" + tag + "$" + l.advance(len(open)) + if idx := strings.Index(l.src[l.pos:], open); idx >= 0 { + l.advance(idx + len(open)) + } else { + l.advance(len(l.src) - l.pos) // unterminated: consume to EOF + } + l.emit(DollarString, start) + return + } + if isDigit(l.peek(1)) { + l.advance(1) + l.scanWhile(isDigit) + l.emit(Param, start) + return + } + l.advance(1) + l.emit(Unknown, start) +} + +// dollarTag checks whether the current $ begins a dollar-quote opening +// delimiter and, if so, returns the (possibly empty) tag. +func (l *lexer) dollarTag() (string, bool) { + // src[pos] == '$' + j := l.pos + 1 + n := len(l.src) + if j < n && l.src[j] == '$' { + return "", true // "$$" + } + k := j + if k < n && isIdentStart(l.src[k]) { + k++ + // Tag chars follow identifier rules but exclude '$' (which closes the tag). + for k < n && isIdentCont(l.src[k]) && l.src[k] != '$' { + k++ + } + if k < n && l.src[k] == '$' { + return l.src[j:k], true + } + } + return "", false +} + +func (l *lexer) scanLineComment() { + // Consume until newline (exclusive); the newline is whitespace. + for l.pos < len(l.src) && l.src[l.pos] != '\n' { + l.advance(1) + } +} + +func (l *lexer) scanBlockComment() { + l.advance(2) // /* + depth := 1 + for l.pos < len(l.src) && depth > 0 { + if l.src[l.pos] == '/' && l.peek(1) == '*' { + l.advance(2) + depth++ + } else if l.src[l.pos] == '*' && l.peek(1) == '/' { + l.advance(2) + depth-- + } else { + l.advance(1) + } + } +} + +// scanString consumes a quoted run beginning at the current quote character. +// A doubled quote (”) is an embedded quote; with backslash=true a backslash +// escapes the next byte (escape-string syntax). +func (l *lexer) scanString(quote byte, backslash bool) { + l.advance(1) // opening quote + for l.pos < len(l.src) { + c := l.src[l.pos] + if backslash && c == '\\' && l.pos+1 < len(l.src) { + l.advance(2) + continue + } + if c == quote { + if l.peek(1) == quote { + l.advance(2) // doubled quote + continue + } + l.advance(1) // closing quote + return + } + l.advance(1) + } +} + +func (l *lexer) scanNumber() { + // Non-decimal integer literals: 0x.., 0o.., 0b.. + if l.src[l.pos] == '0' { + switch l.peek(1) { + case 'x', 'X', 'o', 'O', 'b', 'B': + l.advance(2) + l.scanWhile(isHexOrSep) + return + } + } + l.scanWhile(isDigitOrSep) + if l.pos < len(l.src) && l.src[l.pos] == '.' { + l.advance(1) + l.scanWhile(isDigitOrSep) + } + if c := l.cur(); c == 'e' || c == 'E' { + if p := l.peek(1); isDigit(p) || ((p == '+' || p == '-') && isDigit(l.peek(2))) { + l.advance(1) + if l.cur() == '+' || l.cur() == '-' { + l.advance(1) + } + l.scanWhile(isDigit) + } + } +} + +// scanOperator consumes a run of operator characters, applying PostgreSQL's +// rule that a multi-character operator may not end in + or - unless it also +// contains one of ~ ! @ # % ^ & | ` ?. +func (l *lexer) scanOperator() { + start := l.pos + hasSpecial := false + for l.pos < len(l.src) { + c := l.src[l.pos] + if strings.IndexByte(opChars, c) < 0 { + break + } + if c == '-' && l.peek(1) == '-' { + break // comment start + } + if c == '/' && l.peek(1) == '*' { + break // comment start + } + if strings.IndexByte(opSpecial, c) >= 0 { + hasSpecial = true + } + l.advance(1) + } + if !hasSpecial { + for l.pos-1 > start { + last := l.src[l.pos-1] + if last != '+' && last != '-' { + break + } + l.backup() + } + } +} + +// --- low-level helpers --- + +func (l *lexer) cur() byte { + if l.pos < len(l.src) { + return l.src[l.pos] + } + return 0 +} + +func (l *lexer) peek(k int) byte { + if l.pos+k < len(l.src) { + return l.src[l.pos+k] + } + return 0 +} + +func (l *lexer) scanWhile(pred func(byte) bool) { + for l.pos < len(l.src) && pred(l.src[l.pos]) { + l.advance(1) + } +} + +// advance moves forward k bytes, maintaining line/col. +func (l *lexer) advance(k int) { + for i := 0; i < k && l.pos < len(l.src); i++ { + if l.src[l.pos] == '\n' { + l.line++ + l.col = 1 + } else { + l.col++ + } + l.pos++ + } +} + +// backup moves back one byte. Only used within scanOperator, which never +// spans a newline, so column bookkeeping is safe. +func (l *lexer) backup() { + l.pos-- + l.col-- +} + +// emit appends a token covering src[start:pos], using the start position +// captured at the top of the scan loop. +func (l *lexer) emit(kind Kind, start int) { + l.out = append(l.out, Token{ + Kind: kind, + Text: l.src[start:l.pos], + Off: l.tokOff, + Line: l.tokLine, + Col: l.tokCol, + }) +} + +func isSpace(c byte) bool { + switch c { + case ' ', '\t', '\n', '\r', '\v', '\f': + return true + } + return false +} + +func isDigit(c byte) bool { return c >= '0' && c <= '9' } +func isDigitOrSep(c byte) bool { return isDigit(c) || c == '_' } + +func isHexOrSep(c byte) bool { + return isDigit(c) || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F') || c == '_' +} + +func isIdentStart(c byte) bool { + return c == '_' || (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || c >= 0x80 +} + +func isIdentCont(c byte) bool { + return isIdentStart(c) || isDigit(c) || c == '$' +} diff --git a/pkg/lexer/lexer_test.go b/pkg/lexer/lexer_test.go new file mode 100644 index 0000000..4cf797d --- /dev/null +++ b/pkg/lexer/lexer_test.go @@ -0,0 +1,146 @@ +package lexer + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// emit reconstructs the source from a token stream. +func emit(toks []Token) string { + var b strings.Builder + for _, t := range toks { + b.WriteString(t.Text) + } + return b.String() +} + +func TestRoundTripSmall(t *testing.T) { + cases := []string{ + "", + "SELECT 1;", + "select * from t where a = b;", + "-- a comment\nSELECT 1", + "/* block /* nested */ still */ SELECT 1", + "SELECT 'it''s', E'a\\nb', $$dollar$$, $tag$x$tag$, $1;", + "a->>'b'::text", + "x := y + 1;", + "a=-b", + "SELECT 1.5, .5, 1e10, 0xFF, 1_000;", + "arr[1:2]", + "\"Quoted Ident\".col", + "U&'d\\0061t'", + } + for _, src := range cases { + got := emit(Lex(src)) + if got != src { + t.Errorf("round-trip mismatch\n in: %q\nout: %q", src, got) + } + } +} + +func TestKinds(t *testing.T) { + toks := nonEOF(Lex("a->>'b'::text")) + want := []Kind{Ident, Operator, String, Operator, Ident} + if len(toks) != len(want) { + t.Fatalf("got %d tokens, want %d: %v", len(toks), len(want), toks) + } + for i, k := range want { + if toks[i].Kind != k { + t.Errorf("token %d: got %s, want %s (text %q)", i, toks[i].Kind, k, toks[i].Text) + } + } +} + +func TestOperatorTrailingRule(t *testing.T) { + // "=-" must split into "=" and "-" (no special char, can't end in -). + toks := nonEOF(Lex("a=-b")) + if len(toks) != 4 || toks[1].Text != "=" || toks[2].Text != "-" { + t.Fatalf("a=-b mis-lexed: %v", toks) + } + // "@-" keeps trailing - because @ is special. + toks = nonEOF(Lex("a@-b")) + if toks[1].Text != "@-" { + t.Fatalf("@- should stay one operator, got %q", toks[1].Text) + } +} + +func TestDollarQuote(t *testing.T) { + toks := nonEOF(Lex("$func$ body $$ inner $func$")) + if len(toks) != 1 || toks[0].Kind != DollarString { + t.Fatalf("dollar quote not single token: %v", toks) + } +} + +func TestLineColumns(t *testing.T) { + toks := nonEOF(Lex("ab\n cd")) + // ab(1,1) ws cd(2,3) + if toks[0].Line != 1 || toks[0].Col != 1 { + t.Errorf("ab at %d:%d, want 1:1", toks[0].Line, toks[0].Col) + } + cd := toks[len(toks)-1] + if cd.Text != "cd" || cd.Line != 2 || cd.Col != 3 { + t.Errorf("cd at %d:%d, want 2:3", cd.Line, cd.Col) + } +} + +// TestCorpusRoundTrip is the core lossless invariant: lexing then re-emitting +// every real-world .pgsql fixture must reproduce it byte-for-byte. +func TestCorpusRoundTrip(t *testing.T) { + dir := filepath.Join("..", "..", "testdata", "corpus") + entries, err := os.ReadDir(dir) + if err != nil { + t.Skipf("no corpus dir: %v", err) + } + var seen int + for _, e := range entries { + if e.IsDir() || !strings.HasSuffix(e.Name(), ".pgsql") { + continue + } + seen++ + path := filepath.Join(dir, e.Name()) + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read %s: %v", path, err) + } + src := string(data) + if got := emit(Lex(src)); got != src { + t.Errorf("%s: round-trip mismatch (len in=%d out=%d)", e.Name(), len(src), len(got)) + reportFirstDiff(t, e.Name(), src, got) + } + } + if seen == 0 { + t.Skip("corpus dir has no .pgsql files") + } + t.Logf("round-tripped %d corpus files", seen) +} + +func reportFirstDiff(t *testing.T, name, a, b string) { + t.Helper() + n := len(a) + if len(b) < n { + n = len(b) + } + for i := 0; i < n; i++ { + if a[i] != b[i] { + lo := i - 20 + if lo < 0 { + lo = 0 + } + t.Logf("%s: first diff at byte %d\n in: %q\n out: %q", name, i, a[lo:min(i+20, len(a))], b[lo:min(i+20, len(b))]) + return + } + } +} + +func nonEOF(toks []Token) []Token { + var out []Token + for _, t := range toks { + if t.Kind == EOF { + continue + } + out = append(out, t) + } + return out +} diff --git a/pkg/lexer/token.go b/pkg/lexer/token.go new file mode 100644 index 0000000..3adbf81 --- /dev/null +++ b/pkg/lexer/token.go @@ -0,0 +1,102 @@ +// Package lexer implements a lossless lexer for PostgreSQL SQL and PL/pgSQL. +// +// "Lossless" means every byte of the input is represented by exactly one token, +// including whitespace and comments. Concatenating the Text of all tokens in +// order reproduces the original source byte-for-byte: +// +// emit(Lex(src)) == src +// +// This property is the foundation of the formatter: comments and whitespace are +// first-class tokens (trivia) so the parser/printer can preserve them. +package lexer + +// Kind classifies a token. +type Kind int + +const ( + EOF Kind = iota + + // Trivia — insignificant to the grammar but preserved losslessly. + Whitespace + LineComment // -- ... (up to, not including, the newline) + BlockComment // /* ... */ (nestable) + + // Words. + Ident // unquoted identifier or keyword (keyword-ness resolved later) + QuotedIdent // "..." + + // Literals. + String // '...' + EscapeString // E'...' + BitString // B'...' + HexString // X'...' + UnicodeString // U&'...' or U&"..." + DollarString // $tag$...$tag$ + Number // 123, 1.5, .5, 1e10, 0xff, 1_000 + Param // $1, $2 + + // Operators (incl. ::, :=, :, ->, ->>, and op-char runs). + Operator + + // Structural punctuation. + LParen + RParen + LBracket + RBracket + Comma + Semicolon + Dot + + Unknown // a byte that fits no other category +) + +var kindNames = map[Kind]string{ + EOF: "EOF", + Whitespace: "Whitespace", + LineComment: "LineComment", + BlockComment: "BlockComment", + Ident: "Ident", + QuotedIdent: "QuotedIdent", + String: "String", + EscapeString: "EscapeString", + BitString: "BitString", + HexString: "HexString", + UnicodeString: "UnicodeString", + DollarString: "DollarString", + Number: "Number", + Param: "Param", + Operator: "Operator", + LParen: "LParen", + RParen: "RParen", + LBracket: "LBracket", + RBracket: "RBracket", + Comma: "Comma", + Semicolon: "Semicolon", + Dot: "Dot", + Unknown: "Unknown", +} + +func (k Kind) String() string { + if s, ok := kindNames[k]; ok { + return s + } + return "Kind(?)" +} + +// Token is a single lexical unit covering Text == src[Off:Off+len(Text)]. +type Token struct { + Kind Kind + Text string + Off int // byte offset of the first byte + Line int // 1-based line of the first byte + Col int // 1-based byte column of the first byte +} + +// IsTrivia reports whether the token is whitespace or a comment. +func (t Token) IsTrivia() bool { + switch t.Kind { + case Whitespace, LineComment, BlockComment: + return true + } + return false +} diff --git a/pkg/parser/parser.go b/pkg/parser/parser.go new file mode 100644 index 0000000..33504fc --- /dev/null +++ b/pkg/parser/parser.go @@ -0,0 +1,263 @@ +// Package parser builds a cst.File from PostgreSQL source. +// +// The parser is intentionally partial: it splits input into statements and +// structures the constructs the formatter currently understands (today: +// CREATE FUNCTION/PROCEDURE). Anything else is preserved verbatim as a cst.Raw +// node. This guarantees the parser never loses or corrupts input — see the +// round-trip test. +package parser + +import ( + "github.com/hein/pgtidy/pkg/cst" + "github.com/hein/pgtidy/pkg/lexer" +) + +// Parse lexes and parses src into a lossless cst.File. +func Parse(src string) *cst.File { + sig, trailing := cst.Attach(lexer.Lex(src)) + f := &cst.File{Trailing: trailing} + for _, stmt := range splitStatements(sig) { + f.Items = append(f.Items, parseStatement(stmt)) + } + return f +} + +// splitStatements breaks the significant-token stream into statements at +// top-level (paren-depth 0) semicolons. The terminating semicolon is included +// in the statement it ends. +func splitStatements(sig []cst.Tok) [][]cst.Tok { + var stmts [][]cst.Tok + var cur []cst.Tok + depth := 0 + for _, t := range sig { + cur = append(cur, t) + switch t.Tok.Kind { + case lexer.LParen: + depth++ + case lexer.RParen: + if depth > 0 { + depth-- + } + case lexer.Semicolon: + if depth == 0 { + stmts = append(stmts, cur) + cur = nil + } + } + } + if len(cur) > 0 { + stmts = append(stmts, cur) + } + return stmts +} + +func parseStatement(stmt []cst.Tok) cst.Node { + if isCreateFunction(stmt) { + if cf, ok := parseCreateFunction(stmt); ok { + return cf + } + } + return &cst.Raw{Toks: stmt} +} + +// isCreateFunction reports whether stmt begins with CREATE ... FUNCTION|PROCEDURE. +func isCreateFunction(stmt []cst.Tok) bool { + if len(stmt) == 0 || !stmt[0].Is("create") { + return false + } + for i := 1; i < len(stmt) && i < 4; i++ { + if stmt[i].Is("function") || stmt[i].Is("procedure") { + return true + } + } + return false +} + +// parseCreateFunction structures a CREATE FUNCTION/PROCEDURE statement. It +// returns ok=false (so the caller falls back to Raw) if the shape is not what +// it expects. +func parseCreateFunction(stmt []cst.Tok) (*cst.CreateFunction, bool) { + cf := &cst.CreateFunction{} + i := 0 + + // Head: CREATE [OR REPLACE] (FUNCTION|PROCEDURE) + if i >= len(stmt) || !stmt[i].Is("create") { + return nil, false + } + i++ + if i+1 < len(stmt) && stmt[i].Is("or") && stmt[i+1].Is("replace") { + i += 2 + } + if i >= len(stmt) || !(stmt[i].Is("function") || stmt[i].Is("procedure")) { + return nil, false + } + i++ + cf.Head = stmt[:i] + + // Name: everything up to the opening '(' of the parameter list. + nameStart := i + for i < len(stmt) && stmt[i].Tok.Kind != lexer.LParen { + i++ + } + if i >= len(stmt) { + return nil, false // no parameter list + } + cf.Name = stmt[nameStart:i] + cf.LParen = stmt[i] + i++ + + // Parameters: up to the matching ')'. + params, rparen, ok := parseParamList(stmt, i) + if !ok { + return nil, false + } + cf.Params = params + cf.RParen = stmt[rparen] + i = rparen + 1 + + // Trailing semicolon (handle now so it isn't swept into options/tail). + end := len(stmt) + if end > 0 && stmt[end-1].Tok.Kind == lexer.Semicolon { + cf.Semi = &stmt[end-1] + end-- + } + + // Options up to AS; then the body; then any tail options. + asIdx := indexOfKeyword(stmt, i, end, "as") + if asIdx < 0 { + // No AS clause (e.g. RETURN / BEGIN ATOMIC bodies). Keep options + // structured but leave the body unstructured. + cf.Options = splitClauses(stmt[i:end]) + return cf, true + } + cf.Options = splitClauses(stmt[i:asIdx]) + cf.As = &stmt[asIdx] + bi := asIdx + 1 + if bi < end && isBodyToken(stmt[bi]) { + cf.Body = &stmt[bi] + bi++ + } + if bi < end { + cf.Tail = splitClauses(stmt[bi:end]) + } + return cf, true +} + +// parseParamList parses comma-separated parameters starting at index `start` +// (the token after '('). It returns the parameters, the index of the matching +// ')', and ok. +func parseParamList(stmt []cst.Tok, start int) ([]cst.Param, int, bool) { + var params []cst.Param + depth := 0 + itemStart := start + i := start + flush := func(endExclusive int, sep *cst.Tok) { + toks := stmt[itemStart:endExclusive] + if len(toks) == 0 && sep == nil { + return // empty parameter list () + } + params = append(params, cst.Param{Toks: toks, Sep: sep}) + } + for ; i < len(stmt); i++ { + switch stmt[i].Tok.Kind { + case lexer.LParen, lexer.LBracket: + depth++ + case lexer.RParen: + if depth == 0 { + // Closing the parameter list. + flush(i, nil) + return params, i, true + } + depth-- + case lexer.RBracket: + if depth > 0 { + depth-- + } + case lexer.Comma: + if depth == 0 { + sep := &stmt[i] + flush(i, sep) + itemStart = i + 1 + } + } + } + return nil, 0, false // unbalanced +} + +// splitClauses groups a run of option tokens into clauses, each beginning at a +// recognized clause keyword (at paren depth 0). Tokens before the first +// recognized keyword, if any, are attached to a leading clause so nothing is +// dropped. +func splitClauses(toks []cst.Tok) [][]cst.Tok { + if len(toks) == 0 { + return nil + } + var clauses [][]cst.Tok + depth := 0 + start := 0 + for i, t := range toks { + switch t.Tok.Kind { + case lexer.LParen, lexer.LBracket: + depth++ + case lexer.RParen, lexer.RBracket: + if depth > 0 { + depth-- + } + } + if i > start && depth == 0 && isClauseStarter(t) { + clauses = append(clauses, toks[start:i]) + start = i + } + } + clauses = append(clauses, toks[start:]) + return clauses +} + +var clauseStarters = map[string]bool{ + "returns": true, "language": true, "transform": true, "window": true, + "immutable": true, "stable": true, "volatile": true, "leakproof": true, + "not": true, "called": true, "strict": true, "external": true, + "security": true, "parallel": true, "cost": true, "rows": true, + "support": true, "set": true, "as": true, +} + +func isClauseStarter(t cst.Tok) bool { + if t.Tok.Kind != lexer.Ident { + return false + } + return clauseStarters[lowerASCII(t.Tok.Text)] +} + +func isBodyToken(t cst.Tok) bool { + return t.Tok.Kind == lexer.DollarString || t.Tok.Kind == lexer.String +} + +// indexOfKeyword returns the index in [from,to) of the first top-level (paren +// depth 0) token equal to kw, or -1. +func indexOfKeyword(stmt []cst.Tok, from, to int, kw string) int { + depth := 0 + for i := from; i < to; i++ { + switch stmt[i].Tok.Kind { + case lexer.LParen, lexer.LBracket: + depth++ + case lexer.RParen, lexer.RBracket: + if depth > 0 { + depth-- + } + } + if depth == 0 && stmt[i].Is(kw) { + return i + } + } + return -1 +} + +func lowerASCII(s string) string { + b := []byte(s) + for i, c := range b { + if c >= 'A' && c <= 'Z' { + b[i] = c + 32 + } + } + return string(b) +} diff --git a/pkg/parser/parser_test.go b/pkg/parser/parser_test.go new file mode 100644 index 0000000..b8bf7ba --- /dev/null +++ b/pkg/parser/parser_test.go @@ -0,0 +1,124 @@ +package parser + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/hein/pgtidy/pkg/cst" +) + +func TestParseRoundTripSmall(t *testing.T) { + cases := []string{ + "", + "SELECT 1;", + "-- lead\nSELECT 1; SELECT 2;", + "CREATE FUNCTION f() RETURNS int LANGUAGE sql AS $$ SELECT 1 $$;", + "create or replace function s.g(a int, b text default 'x') returns void language plpgsql as $$ begin end; $$;\n", + "INSERT INTO t (a,b) VALUES (1,2); -- trailing\n", + } + for _, src := range cases { + if got := Parse(src).Source(); got != src { + t.Errorf("round-trip mismatch\n in: %q\nout: %q", src, got) + } + } +} + +func TestParseCreateFunctionShape(t *testing.T) { + src := "CREATE OR REPLACE FUNCTION resolvespec_login(\n" + + " INOUT p_data jsonb\n" + + " ,OUT p_success boolean\n" + + " ,OUT p_error text\n" + + ")\nLANGUAGE plpgsql VOLATILE\nSECURITY DEFINER\nAS\n$$\nbegin end;\n$$;\n" + + f := Parse(src) + if len(f.Items) != 1 { + t.Fatalf("want 1 item, got %d", len(f.Items)) + } + cf, ok := f.Items[0].(*cst.CreateFunction) + if !ok { + t.Fatalf("want *CreateFunction, got %T", f.Items[0]) + } + if cf.IsProcedure() { + t.Error("should be FUNCTION not PROCEDURE") + } + if len(cf.Params) != 3 { + t.Fatalf("want 3 params, got %d", len(cf.Params)) + } + // Leading-comma style: in source the comma precedes the next param, but the + // parser associates each comma as the separator after the preceding param. + if cf.Params[0].Sep == nil || cf.Params[1].Sep == nil || cf.Params[2].Sep != nil { + t.Errorf("param separators wrong: %v %v %v", + cf.Params[0].Sep != nil, cf.Params[1].Sep != nil, cf.Params[2].Sep != nil) + } + if cf.Body == nil || !strings.Contains(cf.Body.Text(), "begin") { + t.Errorf("body not captured: %+v", cf.Body) + } + if cf.As == nil { + t.Error("AS not captured") + } + // Options should include a LANGUAGE clause, a VOLATILE clause and a SECURITY clause. + var langs, vols, secs int + for _, cl := range cf.Options { + if len(cl) == 0 { + continue + } + switch { + case cl[0].Is("language"): + langs++ + case cl[0].Is("volatile"): + vols++ + case cl[0].Is("security"): + secs++ + } + } + if langs != 1 || vols != 1 || secs != 1 { + t.Errorf("clause split wrong: language=%d volatile=%d security=%d", langs, vols, secs) + } +} + +func TestParseFallbackRaw(t *testing.T) { + f := Parse("SELECT a, b FROM t WHERE c = 1;") + if len(f.Items) != 1 { + t.Fatalf("want 1 item, got %d", len(f.Items)) + } + if _, ok := f.Items[0].(*cst.Raw); !ok { + t.Fatalf("want *Raw, got %T", f.Items[0]) + } +} + +// TestCorpusRoundTrip proves the parser is lossless on real-world input: the +// reconstructed source must equal the original byte-for-byte. +func TestCorpusRoundTrip(t *testing.T) { + dir := filepath.Join("..", "..", "testdata", "corpus") + entries, err := os.ReadDir(dir) + if err != nil { + t.Skipf("no corpus dir: %v", err) + } + var seen, fns int + for _, e := range entries { + if e.IsDir() || !strings.HasSuffix(e.Name(), ".pgsql") { + continue + } + seen++ + data, err := os.ReadFile(filepath.Join(dir, e.Name())) + if err != nil { + t.Fatal(err) + } + src := string(data) + f := Parse(src) + if got := f.Source(); got != src { + t.Errorf("%s: round-trip mismatch (len in=%d out=%d)", e.Name(), len(src), len(got)) + } + for _, it := range f.Items { + if _, ok := it.(*cst.CreateFunction); ok { + fns++ + } + } + } + if seen == 0 { + t.Skip("no corpus files") + } + t.Logf("round-tripped %d corpus files; structured %d CREATE FUNCTION/PROCEDURE", seen, fns) +} diff --git a/plan.md b/plan.md new file mode 100644 index 0000000..b0407f2 --- /dev/null +++ b/plan.md @@ -0,0 +1,144 @@ +# PgTidy — PostgreSQL Linter & Formatter + +## Context + +We are starting a greenfield project, **PgTidy**: a PostgreSQL-focused linter and +formatter that enforces consistent SQL style, detects PostgreSQL-specific issues, and +formats migrations, schemas, functions, procedures, triggers, and related DB code. +The primary day-one use case is **formatting PL/pgSQL stored procedures** to match an +existing house style (sample corpus at `testdata/corpus`). + +The core is written in **Go**. We ship a CLI, then an LSP server that powers a **VSCode +extension** (priority) and later a **DataGrip/JetBrains** integration (lower priority). + +### Decisions locked in (from planning Q&A) + +- **Parsing = hybrid.** Real PostgreSQL grammar via **go-pgquery** (libpg_query compiled + to WASM with `wazero`, **no cgo**) powers deep semantic lint rules. A **custom lossless + lexer + CST** powers the comment-preserving formatter. (libpg_query drops comments & + whitespace, so it cannot be the sole basis for a formatter — confirmed.) +- **V1 = Formatter + CLI first.** Lint → LSP/VSCode → DataGrip follow. +- **Lint scope (later milestones):** style/consistency, migration safety (Squawk-style), + naming conventions, correctness/anti-patterns — all four. +- **Editors:** one Go LSP core → VSCode now (bundled per-platform binary), DataGrip via + the free **LSP4IJ** plugin later. + +## Why this architecture + +- **No cgo** (WASM-embedded libpg_query) keeps cross-compilation trivial and lets us bundle + a single static binary per platform inside the VSCode extension. sqlc migrated to this exact + approach to escape cgo pain. +- **Formatting needs a lossless CST** (round-trippable, comments + whitespace preserved). This + is the dominant pattern in mature tools (Roslyn, rust-analyzer/rowan, Biome, SQLFluff). We + build our own lexer/CST so the formatter never loses a comment. +- **Deep lint needs an accurate AST** — go-pgquery gives the real PostgreSQL parse tree. +- **One core, many frontends** — CLI, LSP, VSCode and DataGrip all reuse the same engine and + the same `diagnostics` type. + +## Proposed layout + +``` +pgtidy/ + cmd/pgtidy/ — CLI entry; subcommands: fmt, lint (v2), lsp (v3), version + pkg/lexer/ — lossless lexer: tokens + trivia (comments/whitespace) attached + pkg/cst/ — concrete syntax tree (round-trippable node model) + pkg/parser/ — recursive-descent parser → CST (DML + DDL + PL/pgSQL) + pkg/format/ — Doc-IR printer (Wadler/Prettier-style) + style application + pkg/pgast/ — go-pgquery wrapper: SQL → real PG AST (for lint, v2) + pkg/lint/ — rule engine + rule packs (v2) + pkg/config/ — .pgtidy.yaml discovery/merge: style + rule config + pkg/diagnostics/ — shared diagnostic type (CLI + LSP) + pkg/lsp/ — LSP server (v3) + editors/vscode/ — VSCode extension (TS), bundles pgtidy binary (v3) + editors/datagrip/ — LSP4IJ integration (v4) + testdata/ — golden formatter fixtures + lint fixtures + corpus +``` + +## House style (formatter defaults — reverse-engineered from the corpus) + +These become the default `style` config; all are configurable. The corpus had human +inconsistencies (e.g. a DECLARE var at column 0, mixed `=`/`:=`); the formatter **normalizes** +to the intended style below. + +- Keywords **UPPERCASE**; data types **lowercase**; identifiers lowercase snake_case. +- Indent: **2 spaces** per level. +- **Leading-comma** style, one item per line, for SELECT column lists and function params. +- Function headers: params one-per-line in parens; `LANGUAGE` / `SECURITY` / volatility each + on own line; `AS` then `$$` on its own line; body; closing `$$;` on its own line. +- PL/pgSQL: `DECLARE` alone, vars 2-space indented; `--Block--` comment markers preserved; + `BEGIN`/`END` at body level; `IF/THEN/ELSIF/ELSE/END IF`, loops, `CASE` indent their bodies. +- Spacing: spaces around binary operators (`=`,`<>`,`||`,…) and `:=`; **no** space around + `::`, `->`, `->>`, array `[...]`, or before a call's `(`. +- Dollar-quote tags preserved verbatim (`$$`, `$S$`, `$Z$`, …). + +## Milestones + +### V1 — Formatter + CLI (priority) + +1. **Lexer** (`pkg/lexer`): full PG token coverage incl. dollar-quoted strings, `--` and + `/* */` comments, operators. Comments + whitespace captured as **leading/trailing trivia** + on tokens. Acceptance: `emit(lex(src)) == src` byte-for-byte across the corpus. +2. **CST + parser** (`pkg/cst`, `pkg/parser`): recursive descent for DML (SELECT/INSERT/ + UPDATE/DELETE/CTE), DDL (CREATE FUNCTION/PROCEDURE/TABLE/INDEX/TRIGGER, ALTER, DO). +3. **PL/pgSQL body parser**: DECLARE/BEGIN/END, IF/CASE/LOOP, assignments, nested SQL — the + crux for stored-procedure formatting. +4. **Printer** (`pkg/format`): Doc-IR (group/indent/line/softline) driven by `style` config. + **Graceful degradation** — any span the parser can't handle passes through verbatim rather + than being corrupted. +5. **CLI** (`cmd/pgtidy fmt`): `--check`, `--write`/`-w`, stdin→stdout, `--diff`; config + discovery walking up to `.pgtidy.yaml`; CI-friendly exit codes. +6. **Config** (`pkg/config`): load/merge style config; defaults = house style above. + +**Safety guarantees (tested):** semantic equivalence (re-lex output, compare non-trivia token +stream to input), and idempotence (`fmt(fmt(x)) == fmt(x)`). The corpus is the +primary safety/idempotence harness; golden-file tests for targeted cases. + +### V2 — Linter + +- `pkg/pgast` go-pgquery (WASM) wrapper; `pkg/lint` rule engine (rule ID, severity, config). +- Rule packs: **style/consistency**, **migration safety** (ACCESS EXCLUSIVE locks, unsafe + `ALTER`/`ADD COLUMN`, non-`CONCURRENTLY` index builds, blocking constraints), **naming** + (configurable table/column/index/constraint patterns), **correctness/anti-patterns** + (`SELECT *`, missing `WHERE` on UPDATE/DELETE, deprecated syntax). Emit `diagnostics`. +- `pgtidy lint` subcommand; `--fix` for autofixable rules. + +### V3 — LSP + VSCode + +- `pkg/lsp`: `textDocument/formatting` + range formatting, `publishDiagnostics`, `codeAction` + quick-fixes — all reusing the core. +- `editors/vscode`: TS extension using `vscode-languageclient`, launches bundled `pgtidy lsp`. + Build **per-platform VSIX** (`win32/linux/darwin × x64/arm64`) in a CI matrix (rust-analyzer + model), with a target-less fallback. + +### V4 — DataGrip + +- `editors/datagrip`: integrate via **LSP4IJ** (free, works across JetBrains editions incl. + DataGrip). No core changes expected. + +## Build / repo hygiene + +- Replace boilerplate `AGENTS.md`/`CLAUDE.md` with PgTidy content; rewrite `Makefile` + (`APP := pgtidy`, `CMD := ./cmd/pgtidy`; keep build/test/lint/release-version targets). +- Add **goreleaser** for the multi-platform binary matrix (clean, since no cgo). +- `go.mod`: deps = `github.com/wasilibs/go-pgquery` (v2), `wazero`, a YAML lib, an LSP lib + (e.g. `go.lsp.dev/protocol`) in v3. + +## Verification + +- **Formatter:** `go test ./...` runs golden-file tests + the corpus harness asserting + (a) idempotence and (b) token-stream equality before/after (no semantic change). + Manual: `pgtidy fmt --diff` against several procedures; confirm output matches + the house style and comments/dollar-quote tags survive. +- **CLI:** `pgtidy fmt --check` returns non-zero on unformatted input, zero when clean. +- **Lint (v2):** fixture SQL with known violations → assert expected diagnostics; `--fix` + round-trips. +- **LSP/VSCode (v3):** load a `.sql` file in a dev-host VSCode, confirm format-on-save and + live diagnostics via the bundled binary. + +## Open risks + +- A lossless PL/pgSQL recursive-descent parser is the largest single effort; the + pass-through-on-unparsed fallback bounds the risk and lets us ship incrementally by construct. +- go-pgquery tracks PG17 (PG18 not yet) — fine for lint; irrelevant to the formatter path. +- Leading-comma + one-per-line is unusual vs. most formatters; it's a first-class style option, + not an afterthought. diff --git a/testdata/corpus/test_a.pgsql b/testdata/corpus/test_a.pgsql new file mode 100755 index 0000000..5934f17 --- /dev/null +++ b/testdata/corpus/test_a.pgsql @@ -0,0 +1,169 @@ +--select * from dropall('resolvespec_login'); +CREATE OR REPLACE FUNCTION resolvespec_login( + INOUT p_data jsonb + ,OUT p_success boolean + ,OUT p_error text +) +LANGUAGE plpgsql VOLATILE +SECURITY DEFINER +AS +$$ +DECLARE + --Error Handling-- + m_funcname text = 'resolvespec_login'; + m_errmsg text; + m_errcontext text; + m_errdetail text; + m_errhint text; + m_errstate text; + m_retval integer; + --Error Handling-- + m_rid_user integer; + m_rid_hub integer; + m_pass_hashed citext[]; + m_session jsonb; + m_allow_hash_auth boolean; +BEGIN + m_allow_hash_auth = _try_integer( p_data->'claims'->>'rid_user',0) > 0; + create extension if not exists pgcrypto; + + perform log_event(m_funcname,format('API Login username: %s hh=%s claims: %s',p_data->>'username',m_allow_hash_auth,p_data->'claims'), bt_enum('eventlog','local notice')); + + select h.rid_hub + from public.user h + where h.usercode = p_data->>'username' + into m_rid_hub; + + if m_rid_hub is null + and exists (select 1 from information_schema.tables t where t.table_schema = 'public' and t.table_name = 'users') + then + select u.rid_hub, u.rid_user + from public.users u + where u.rid_user = _try_integer( p_data->'claims'->>'rid_user',0) + into m_rid_hub,m_rid_user; + + end if; + + if m_rid_hub is null + then + raise exception 'Invalid username / password'; + end if; + + m_pass_hashed = array[encode(digest(format('%s:%s',p_data->>'username',p_data->>'password'), 'sha512'), 'hex') + ,encode(digest(format('%s:%s',p_data->>'username',p_data->>'password'), 'md5'), 'hex') + ,encode(digest(format('%s',p_data->>'password'), 'md5'), 'hex') + ]::citext[]; + + if m_allow_hash_auth + then + m_pass_hashed := m_pass_hashed || array[ + p_data->>'password' + ]::citext[]; + end if; + + --select $A${"meta": null, "claims": {"rid_user": 30000024}, "password": "c4ca4238a0b923820dcc509a6f75849b", "username": "SUPPORT"}$A$::jsonb->>'password' + --c4ca4238a0b923820dcc509a6f75849b + --select * from v_eventlog + + if exists ( + select 1 + from information_schema.tables t + where t.table_schema = 'public' + and t.table_name = 'users' + ) + then + if not exists ( + select 1 + from public.user h + left outer join public.users usr on usr.rid_hub = h.rid_hub + where h.rid_hub = m_rid_hub + and ( + h.password = any (m_pass_hashed) + and nv(h.password) <> '' + or usr.password = any(m_pass_hashed) + and nv(usr.password) <> '' + ) + ) + then + raise exception 'Password incorrect'; + end if; + + elsif not exists ( + select 1 + from public.user h + where h.rid_hub = m_rid_hub + and h.password = any(m_pass_hashed) + and nv(h.password) <> '' + ) + then + raise exception 'Password incorrect'; + end if; + + if _try_bool(p_data->'jsonvalue'->>'issecurity',false) + and not exists ( + select h.rid_hub + from public.user h + inner join public.user_all_parents(m_rid_hub) p on p.parent_rid_hub = h.rid_hub + where h.hubname ilike '%Access Control%' + ) + then + raise exception 'Cannot login with security mode. User must be in Access Control group'; + end if; + + with newsession as ( + insert into core._loginsession (createtm, modifytm, rid_user, usertable, sessionid, token, useragent, location, + ipaddress, expiretm, jsonvalue) + select now(), now(), m_rid_hub, 'hub',newid(), newid(),p_data->>'user-agent',p_data->>'fromurl' + ,p_data->>'host', (now() + '31 days'::interval), p_data->'jsonvalue' + returning * + ) + select to_jsonb(newsession) + from newsession + into m_session; + + if _try_integer(m_session->>'rid_user',0) > 0 + then + update public.user u + set jsonvalue = _jsonb_object_cat(u.jsonvalue, jsonb_build_object('lastlogin',to_char(now(),'YYYY-MM-DD HH24:mi:SS'))) + where u.rid_hub = m_rid_hub; + end if; + + + + + select jsonb_build_object('token',m_session->>'token' + ,'session',m_session->>'session' + , 'user', _jsonb_object_cat(jsonb_build_object( + 'user_id', h.rid_hub + ,'username', h.usercode + ,'email',null + ,'user_level', 0 + ,'roles', jsonb_build_array() + ,'session_id', m_session->>'sessionid' + ,'token', m_session->>'token' + ,'session_rid', _try_integer(m_session->>'id') + ), ( + select jsonb_build_object('program_user_table', r.tablename,'program_user_id', _try_integer(r.key,0)) + from public.user_tableinfo(m_rid_hub) r + ) + ) + ,'expires_in', 86400 + ) + from public.user h + where h.rid_hub = m_rid_hub + into p_data; + + p_success = true; +EXCEPTION +WHEN others THEN + GET STACKED DIAGNOSTICS + m_errmsg = MESSAGE_TEXT + ,m_errcontext = PG_EXCEPTION_CONTEXT + ,m_errdetail = PG_EXCEPTION_DETAIL + ,m_errhint = PG_EXCEPTION_HINT + ,m_errstate = RETURNED_SQLSTATE; + + p_error := get_err_msg(m_funcname, m_errmsg, m_errcontext, m_errdetail, m_errhint, m_errstate); + p_success := false; +END; +$$; \ No newline at end of file diff --git a/testdata/corpus/test_b.pgsql b/testdata/corpus/test_b.pgsql new file mode 100755 index 0000000..85a308d --- /dev/null +++ b/testdata/corpus/test_b.pgsql @@ -0,0 +1,790 @@ +--select * from dropall('migrations_read','meta'); +CREATE OR REPLACE FUNCTION meta.migration_read( + p_id_migration_model integer default null + ,OUT p_retval integer + ,OUT p_errmsg text + ,OUT p_info jsonb +) +LANGUAGE plpgsql VOLATILE +SECURITY DEFINER +AS +$$ +DECLARE + m_funcname text = 'migrations_read'; + m_errmsg text; + m_errcontext text; + m_errdetail text; + m_errhint text; + m_errstate text; + m_retval integer; + m_id_migration_model integer; + m_payload text; + m_tm timestamp; + + m_xml xml; + m_json json; + j_err jsonb; +BEGIN + m_tm = clock_timestamp(); + + select f.id_migration_model + ,convert_from(case when f_iscompressed(f.modelfile) = 'gzip' then pl_gzip_bytes(0,f.modelfile) else f.modelfile end, 'utf8') + from meta.migration_model f + where f.id_migration_model = p_id_migration_model + or p_id_migration_model is null + order by f.version desc , f.id_migration_model desc + into m_id_migration_model, m_payload; + + if m_payload ilike '%<%>%' + then + m_xml = m_payload::xml; + raise notice 'XML File set'; + elseif m_payload ilike '%{%}%' + then + raise notice 'JSON File set'; + m_json = m_payload::json; + end if; + + if m_xml is not null + then + + perform pg_notify('upgrade.events', json_build_object('type','upgrade','status',1,'objecttype',m_funcname)::text); + + p_info = jsonb_build_object('format','xml'); + + delete from meta.migration_table + where ismodel; + + delete from meta.migration_index + where ismodel; + + delete from meta.migration_relation_col d + where d.rid_migration_relation in ( + select r.id_migration_relation + from meta.migration_relation r + where r.ismodel + ); + + delete from meta.migration_relation + where ismodel; + + raise notice 'inserting meta.migration_table @ %', (clock_timestamp() - m_tm)::interval; + m_tm = clock_timestamp(); + + drop table if exists tmp_meta_migration_table; + create temp table tmp_meta_migration_table as + select + lower((xpath('/table/tableident/text()', node.x))[1]::text)::citext as tableident + , lower((xpath('/table/prefix/text()', node.x))[1]::text)::citext as tableprefix + , lower((xpath('/table/tablename/text()', node.x))[1]::text)::citext as tablename + , lower(coalesce((xpath('/table/schema/text()', node.x))[1]::text,(xpath('/table/schemaname/text()', node.x))[1]::text)) as schemaname + , lower((xpath('/table/version/text()', node.x))[1]::text)::citext as version + , coalesce((xpath('/table/@seq', node.x))[1]::text,'0')::integer + 1 as schemapriority + , node.x as xml + from unnest(xpath('/root/tables/table', m_xml)) node(x) + ; + + insert into meta.migration_table(rid_migration_model ,guid,prefix, tablename, schemaname, version,ismodel, isdb,schemapriority) + select distinct on (r.tableident,r.tablename,r.tableprefix,r.schemaname) + m_id_migration_model + ,r.tableident + ,r.tableprefix + ,r.tablename + ,r.schemaname + ,r.version + ,true + ,false + ,r.schemapriority + from tmp_meta_migration_table r + ; + + + raise notice 'inserting meta.migration_column @ %', (clock_timestamp() - m_tm)::interval; + m_tm = clock_timestamp(); + + insert into meta.migration_column(rid_migration_table, columnname, guid, columntype,indextype,seqseed, columnlen, precision, defaultval) + select distinct on (r.id_migration_table,r.columnname,r.columnident) + r.id_migration_table,r.columnname,r.columnident,r.columntype,r.indextype,r.seq_seed,r.columnlen,r.precision + ,r.defaultval + from ( + select + mt.id_migration_table + , lower((xpath('/column/columnname/text()', col.x))[1]::text) as columnname + , (xpath('/column/columnident/text()', col.x))[1]::text as columnident + , (xpath('/column/columntype/text()', col.x))[1]::text as columntype + , (xpath('/column/indextype/text()', col.x))[1]::text as indextype + , coalesce((xpath('/column/seed/text()', col.x))[1]::text,'0')::bigint as seq_seed + , coalesce((xpath('/column/columnlen/text()', col.x))[1]::text, '0')::integer as columnlen + , coalesce((xpath('/column/precision/text()', col.x))[1]::text, '')::text as precision + , coalesce((xpath('/column/defaultval/text()', col.x))[1]::text, '')::text as defaultval + from tmp_meta_migration_table tbl + cross join unnest(xpath('/table/columns/column', tbl.xml)) col(x) + inner join meta.migration_table mt on mt.ismodel + and mt.rid_migration_model = m_id_migration_model + -- and lower(mt.guid) = lower((xpath('/table/tableident/text()', tbl.x))[1]::text) + and lower(mt.tablename) = lower(tbl.tablename) + and lower(mt.schemaname) =lower(tbl.schemaname) + ) r + ; + + + raise notice 'inserting meta.migration_index @ %', (clock_timestamp() - m_tm)::interval; + m_tm = clock_timestamp(); + + insert into meta.migration_index(rid_migration_model,rid_migration_table, indexname, indextype, ispk,isduplicate, ispartial,partialstr, isunique, sequence, guid,ismodel) + select distinct on (r.id_migration_table,r.indexname) + m_id_migration_model + ,r.id_migration_table + ,r.indexname + + ,null + ,r.indexprimary in ('1','true') + ,r.indexduplicate in ('1','true') + ,length(r.indexpartial) > 3 + ,r.indexpartial + ,(not coalesce(r.indexduplicate,'') in ('1','true') and (r.indexunique in ('1','true') or r.indexprimary in ('1','true'))) + ,row_number() over (order by r.indexname) + ,r.indexname + ,true + from ( + select + mt.id_migration_table + , mt.schemaname + , mt.tablename + , lower((xpath('/index/indexname/text()', idx.x))[1]::text) as indexname + , coalesce(lower((xpath('/index/indexprimary/text()', idx.x))[1]::text),'') as indexprimary + , coalesce(lower((xpath('/index/indexduplicate/text()', idx.x))[1]::text),'') as indexduplicate + , coalesce(lower((xpath('/index/indexpartial/text()', idx.x))[1]::text),'') as indexpartial + , coalesce(lower((xpath('/index/indexunique/text()', idx.x))[1]::text),'') as indexunique + from tmp_meta_migration_table tbl + cross join unnest(xpath('/table/indexes/index', tbl.xml)) idx(x) + inner join meta.migration_table mt on mt.ismodel + and mt.rid_migration_model = m_id_migration_model + and lower(mt.tablename) = lower(tbl.tablename) + and lower(mt.schemaname) =lower(tbl.schemaname) + ) r + ; + + + update meta.migration_index u + set isunique = true + where u.indexname ilike 'uk_%' + and not u.ispk + and not u.ispartial + and not u.isduplicate + ; + +-- update meta.migration_index u +-- set ispartial = true +-- where u.indexname ilike 'k_%' +-- and not u.isunique +-- and not u.ispk +-- ; + + insert into meta.migration_index_col(rid_migration_index,rid_migration_column_parent,sequence) + select distinct on (r.id_migration_index,r.id_migration_column) + r.id_migration_index + ,r.id_migration_column + ,r.seq + from ( + select + midx.id_migration_index + ,mc.id_migration_column + ,coalesce((xpath('/indexcolumn/@seq', idxcol.x))[1]::text,'0')::integer as seq + from tmp_meta_migration_table tbl + cross join unnest(xpath('/table/indexes/index', tbl.xml)) idx(x) + inner join meta.migration_table mt on mt.ismodel + and mt.rid_migration_model = m_id_migration_model + and lower(mt.tablename) = lower(tbl.tablename) + and lower(mt.schemaname) =lower(tbl.schemaname) + inner join meta.migration_index midx on midx.rid_migration_table = mt.id_migration_table + and midx.indexname = lower((xpath('/index/indexname/text()', idx.x))[1]::text) + + cross join unnest(xpath('/index/indexcolumns/indexcolumn', idx.x)) idxcol(x) + inner join meta.migration_column mc on mt.id_migration_table = mc.rid_migration_table + and lower(mc.columnname) = lower((xpath('/indexcolumn/text()', idxcol.x))[1]::text) + ) r + ; + + raise notice 'inserting meta.migration_relation temp table @ %', (clock_timestamp() - m_tm)::interval; + m_tm = clock_timestamp(); + + drop table if exists tmp_meta_gigration_relation; + create temp table tmp_meta_gigration_relation as + select + tbl.tablename + ,tbl.schemaname + , lower((xpath('/relation/childtable/text()', rel.x))[1]::text) as childtable + , lower((xpath('/relation/relationname/text()', rel.x))[1]::text) as relationname + , lower((xpath('/relation/relationguid/text()', rel.x))[1]::text) as relationguid + , lower((xpath('/relation/deleteconstraint/text()', rel.x))[1]::text) as deleteconstraint + , lower((xpath('/relation/updateconstraint/text()', rel.x))[1]::text) as updateconstraint + ,rel.x as relation + from tmp_meta_migration_table tbl + cross join unnest(xpath('/table/relations/relation', tbl.xml)) rel(x) + ; + + raise notice 'inserting meta.migration_relation insert @ %', (clock_timestamp() - m_tm)::interval; + m_tm = clock_timestamp(); + + insert + into meta.migration_relation( rid_migration_model + , rid_migration_table_parent + , rid_migration_table_child + , relationname + , guid + , updateconstraint + , deleteconstraint + , sequence + , ismodel) + select m_id_migration_model + ,mt.id_migration_table + ,ct.id_migration_table as rid_child_table + ,src.relationname + ,src.relationguid + ,src.updateconstraint + ,src.deleteconstraint + ,row_number() over (order by src.relationname) + ,true + from tmp_meta_gigration_relation as src + inner join meta.migration_table mt on mt.ismodel + and mt.rid_migration_model = m_id_migration_model + and lower(mt.tablename) = lower(src.tablename) + and lower(mt.schemaname) = lower(src.schemaname) + inner join meta.migration_table ct on ct.ismodel + and ct.rid_migration_model = m_id_migration_model + and lower(ct.schemaname) = lower(mt.schemaname) + and lower(ct.tablename) = lower(src.childtable) + ; + + raise notice 'inserting meta.migration_relation_col @ %', (clock_timestamp() - m_tm)::interval; + m_tm = clock_timestamp(); + + insert into meta.migration_relation_col(rid_migration_relation, rid_migration_column_parent, rid_migration_column_child, sequence) + select mrel.id_migration_relation + , parcol.id_migration_column + , cldcol.id_migration_column + ,coalesce((xpath('/keyfields/@seq', key.x))[1]::text,'0')::integer as seq + from tmp_meta_gigration_relation src + inner join meta.migration_table mt on mt.ismodel + and mt.rid_migration_model = m_id_migration_model + and lower(mt.tablename) = lower(src.tablename) + and lower(mt.schemaname) = lower(src.schemaname) + inner join meta.migration_relation mrel on mrel.rid_migration_table_parent = mt.id_migration_table + and lower(mrel.relationname) = lower(src.relationname) + cross join unnest(xpath('/relation/keyfields', src.relation)) key(x) + inner join meta.migration_column parcol on mrel.rid_migration_table_parent = parcol.rid_migration_table + and lower(parcol.columnname) = lower((xpath('/keyfields/parentcolumn/text()', key.x))[1]::text) + inner join meta.migration_column cldcol on mrel.rid_migration_table_child = cldcol.rid_migration_table + and lower(cldcol.columnname) = lower((xpath('/keyfields/childcolumn/text()', key.x))[1]::text) + ; + + raise notice 'inserting meta.migration_object @ %', (clock_timestamp() - m_tm)::interval; + m_tm = clock_timestamp(); + + delete from meta.migration_object + where ismodel; + + insert into meta.migration_object(rid_migration_model,objecttype,objectname, schema, version, checksum, sequence, priority, guid, body, ismodel, isdb) + select m_id_migration_model + ,format('script:%s',r.scripttype) + ,format('%s [%s]',r.scriptname,r.id_scriptcode) + ,r.dbschema,r.version + , case when coalesce(r.scriptchecksum) <> '' then r.scriptchecksum else encode(sha256(convert_to(r.scriptcode,'utf8')),'hex') end + , r.sequence, r.priority, r.id_scriptcode::text,r.scriptcode,true,false + from ( + select (xpath('/script/priority/text()', node.x))[1]::text::integer as priority + , (xpath('/script/sequence/text()', node.x))[1]::text::integer as sequence + , lower((xpath('/script/scriptname/text()', node.x))[1]::text)::citext as scriptname + , lower((xpath('/script/scripttype/text()', node.x))[1]::text)::citext as scripttype + , lower((xpath('/script/dbschema/text()', node.x))[1]::text)::citext as dbschema + , lower((xpath('/script/programname/text()', node.x))[1]::text)::citext as programname + , lower((xpath('/script/version/text()', node.x))[1]::text)::citext as version + , lower((xpath('/script/scriptchecksum/text()', node.x))[1]::text)::citext as scriptchecksum + , xml_extract_value('/script/code', node.x)::text as scriptcode + ,'dct' as source + ,(xpath('/script/id_scriptcode/text()', node.x))[1]::text::integer as id_scriptcode + from unnest(xpath('/root/scripts/script', m_xml )) node(x) + ) r + ; + + elsif m_json is not null + then + p_info = jsonb_build_object('format','json'); + raise exception 'Not yet supported'; + else + p_info = jsonb_build_object('format','unknown'); + raise exception 'Unsupported input file, Content: %', substr(m_payload, 1,20); + end if; + + + + + insert into meta.migration_column(rid_migration_table, columnname, columntype, guid) + select distinct on (t.id_migration_table) + t.id_migration_table, 'updatecnt', 'integer',format('updatecnt_%s', t.id_migration_table) + from meta.migration_table t + left outer join meta.migration_column c on c.rid_migration_table = t.id_migration_table + and c.columnname = 'updatecnt' + where t.ismodel + and t.tablename not in ( + select uut.tablename + from meta.f_upgrade_table() uut + ) + and c.id_migration_column is null + ; + + raise notice 'updates section @ %', (clock_timestamp() - m_tm)::interval; + m_tm = clock_timestamp(); + + + --Set primary key field from indexes + update meta.migration_column u + set ispk = true + from meta.migration_column c + inner join meta.migration_table t on t.id_migration_table = c.rid_migration_table + and t.ismodel + inner join meta.migration_index idx on idx.rid_migration_table = t.id_migration_table + inner join meta.migration_index_col idxcol on idxcol.rid_migration_index = idx.id_migration_index + and idxcol.rid_migration_column_parent = c.id_migration_column + where idx.ispk + and u.id_migration_column = c.id_migration_column + ; + + + + --Set length for strings + update meta.migration_column + set columnlen = reverse(substr(reverse(columntype),2,strpos(reverse(columntype),'(')-2))::integer + - (case when columntype ilike '%cstring%' then 1 else 0 end) + where rid_migration_table in (select t.id_migration_table from meta.migration_table t where t.ismodel) + and columntype ilike '%string%' + ; + + ---Set the length and precision values + update meta.migration_column u + set columnlen = case when coalesce(u.columnlen,0) = 0 and (r.precision > 0 or r.scale > 0) + then coalesce(r.precision,0) + coalesce(r.scale,0) + else u.columnlen + end + ,precision = format('%s,%s',r.precision,r.scale) + from ( + select id_migration_column + ,substring(columntype FROM '\((\d+),(\d+)\)')::integer AS precision + , substring(columntype FROM '\(\d+,(\d+)\)')::integer AS scale + from meta.migration_column + where + rid_migration_table in ( + select t.id_migration_table from meta.migration_table t where t.ismodel + ) + and columntype ilike '%(%)' + ) r + where u.id_migration_column = r.id_migration_column + ; + + --Set default values for prefixes + update meta.migration_column u + set defaultval = ( + select quote_literal(upper(t.prefix)) + from meta.migration_table t + where t.id_migration_table = u.rid_migration_table + and t.ismodel + and nv(t.prefix) <> '' + ) + where u.rid_migration_table in (select t.id_migration_table from meta.migration_table t where t.ismodel) + and u.columnname = 'prefix' + and coalesce(u.defaultval,'') = '' + ; + + --Set GUID field types. e.g. text, uuid + with option as ( + select name, value + from meta.migration_option + where name = 'guidtype' + ) + update meta.migration_column u + set columntype = option.value + ,defaultval = case when nv(u.defaultval) = '' then 'newid()' else u.defaultval end + from option + where rid_migration_table in ( + select t.id_migration_table + from meta.migration_table t + where t.ismodel + and not exists ( + select 1 + from meta.migration_option o + cross join regexp_split_to_table(o.value, ',') rex(v) + where o.name = 'textguid' + and rex.v::citext = t.tablename + and t.schemaname = 'public' + ) + ) + and u.columnname in ('guid','uuid') + and u.columntype is distinct from option.value + ; + + + --Limit length constraints + with option as ( + select name, value::numeric as numvalue + from meta.migration_option + where name = 'max_constraint' + and value ~ '^-?[0-9]+(\.[0-9]+)?$' + ) + update meta.migration_column u + set columnlen = 0 + from option + where rid_migration_table in (select t.id_migration_table from meta.migration_table t where t.ismodel) + and u.columnlen >= option.numvalue + ; + + --Force names if json type options + with option as ( + select name, value + from meta.migration_option + where name = 'settype_names_jsonb' + and value is not null + ) + update meta.migration_column u + set columntype = 'jsonb' + from option + where rid_migration_table in (select t.id_migration_table from meta.migration_table t where t.ismodel) + and lower(u.columnname) in ( + SELECT lower(t.v) from regexp_split_to_table(option.value, ',') t(v) + ) + ; + + --Convert the program types to postgres types + update meta.migration_column u + set columntype = meta.f_datatype_map(u.columntype) + where rid_migration_table in (select t.id_migration_table from meta.migration_table t where t.ismodel) + and meta.f_datatype_map(u.columntype) not ilike '%unknown%' + ; + + update meta.migration_column u + set columntype = case when u.columntype in ('date','text','citext') and (u.columnname ilike '%datetime%' or u.columnname ilike '%timestamp%') + then 'timestamp' + else u.columntype + end + where rid_migration_table in (select t.id_migration_table from meta.migration_table t where t.ismodel) + and meta.f_datatype_map(u.columntype) not ilike '%unknown%' + ; + + + --Larges objects has no lengths + update meta.migration_column u + set columnlen = 0 + where rid_migration_table in (select t.id_migration_table from meta.migration_table t where t.ismodel) + and lower(u.columntype) in ('blob,0','jsonb','json','blob','bytea') + ; + + j_err = null; + select jsonb_agg( + jsonb_build_object('tablename',t.tablename,'schemaname',t.schemaname,'word', kw.word) + ) + from meta.migration_table t + inner join pg_get_keywords() kw on lower(kw.word) = lower(t.tablename) + and lower(kw.catdesc::text) = 'reserved' + where t.ismodel + into j_err + ; + + if jsonb_typeof(j_err) = 'array' + then + p_info = p_info || jsonb_build_object('table_reserved_words',j_err) ; + end if; + + j_err = null; + select jsonb_agg( + jsonb_build_object('columnname',u.columnname,'tablename',t.tablename,'schemaname',t.schemaname,'word', kw.word) + ) + from meta.migration_column u + inner join meta.migration_table t on t.id_migration_table = u.rid_migration_table + inner join pg_get_keywords() kw on lower(kw.word) = lower(u.columnname) + and lower(kw.catdesc::text) = 'reserved' + where u.rid_migration_table in (select t.id_migration_table from meta.migration_table t where t.ismodel) + into j_err + ; + + if jsonb_typeof(j_err) = 'array' + then + p_info = p_info || jsonb_build_object('column_reserved_words',j_err) ; + end if; + + ----Set the default value to the identity if the pk type is identity + update meta.migration_column u + set defaultval = r.def + from ( + select + c.id_migration_column + ,format($S$nextval('%s.identity_%s_%s'::regclass)$S$,t.schemaname,t.tablename,c.columnname) as def + from meta.migration_table t + inner join meta.migration_column c on c.rid_migration_table = t.id_migration_table + where + t.ismodel + and c.ispk + and nv(c.defaultval) = '' + and c.indextype = 'identity' + ) r + where u.id_migration_column = r.id_migration_column; + + update meta.migration_table u + set ismodel = false + where format('%s_%s',u.schemaname, u.tablename) in ( + select format('%s_%s',split_part(o.name,':',2) + ,t.name + ) + from meta.migration_option o + cross join regexp_split_to_table(o.value,',') t(name) + where o.name ilike 'exclude:%' + and t.name <> '' + ) + and u.ismodel + ; + + raise notice 'duplicates section @ %', (clock_timestamp() - m_tm)::interval; + m_tm = clock_timestamp(); + + + --Fallback fix missing primary keys + + update meta.migration_column u + set ispk = true + from meta.migration_table t + where t.id_migration_table = u.rid_migration_table + and t.ismodel + and not exists ( + select 1 + from meta.migration_column c2 + where c2.rid_migration_table = t.id_migration_table + and c2.ispk + ) + and u.indextype = 'identity' + and u.columntype ilike '%int%' + and u.columnname ilike '%id_%' + ; + + + --Move duplicate tables that exists in the settings to their default location. Move the keys as well. + with move as ( + select + t1.schemaname + , t1.tablename + , t1.id_migration_table as id_migration_table_dest + , t2.schemaname + , t2.tablename + , t2.id_migration_table as id_migration_table_src + ,format('%s.%s',t1.schemaname, t1.tablename) as named + from meta.migration_table t1 + inner join meta.migration_table t2 on t2.ismodel + and t2.tablename = t1.tablename + and t2.schemaname is distinct from t1.schemaname + where + t1.ismodel + --and not t1.isdb + and ( + format('%s.%s',t1.schemaname, t1.tablename) in ( + select format('%s.%s',o.value,lower(split_part(o.name, ':', 2))) + from meta.migration_option o + where o.name ilike 'default_schema:%' + ) + ) + ), cols as ( + update meta.migration_column u + set rid_migration_table = m.id_migration_table_dest + from move m + inner join meta.migration_column c1 on c1.rid_migration_table = m.id_migration_table_src + and c1.columnname not in ( + select c2.columnname + from meta.migration_column c2 + where c2.rid_migration_table = m.id_migration_table_dest + ) + where u.id_migration_column = c1.id_migration_column + returning * + ), relations1 as ( + update meta.migration_relation u + set rid_migration_table_parent = m.id_migration_table_dest + ,relationname = format('%s_merged',u.relationname) + from move m + where u.rid_migration_table_parent = m.id_migration_table_src + returning * + ), relations2 as ( + update meta.migration_relation u + set rid_migration_table_child = m.id_migration_table_dest + ,relationname = format('%s_merged',u.relationname) + from move m + where u.rid_migration_table_child = m.id_migration_table_src + returning * + ), relationscols as ( + update meta.migration_relation_col u + set rid_migration_column_child = mc2.id_migration_column + from move m + inner join meta.migration_column mc on mc.rid_migration_table = m.id_migration_table_src + inner join meta.migration_column mc2 on mc2.rid_migration_table = m.id_migration_table_dest + and mc2.columnname = mc.columnname + inner join meta.migration_relation_col rc on rc.rid_migration_column_child = mc.id_migration_column + where u.id_migration_relationcol = rc.id_migration_relationcol + ), relationscols2 as ( + update meta.migration_relation_col u + set rid_migration_column_parent = mc2.id_migration_column + from move m + inner join meta.migration_column mc on mc.rid_migration_table = m.id_migration_table_src + inner join meta.migration_column mc2 on mc2.rid_migration_table = m.id_migration_table_dest + and mc2.columnname = mc.columnname + inner join meta.migration_relation_col rc on rc.rid_migration_column_parent = mc.id_migration_column + where u.id_migration_relationcol = rc.id_migration_relationcol + ), idx as ( + update meta.migration_index u + set rid_migration_table = m.id_migration_table_dest + from move m + where u.rid_migration_table = m.id_migration_table_src + returning * + ), idxcols as ( + update meta.migration_index_col u + set rid_migration_column_parent = col.id_migration_column + from move m + inner join meta.migration_column col on col.rid_migration_table = m.id_migration_table_src + inner join meta.migration_column col2 on col2.rid_migration_table = m.id_migration_table_dest + and col2.columnname = col.columnname + where u.rid_migration_column_parent = col.id_migration_column + ) + update meta.migration_table u + set ismodel = false + from move mv + where u.id_migration_table = mv.id_migration_table_src; + + + update meta.migration_index u + set indexname = format('%s_%s_%s',u.indexname,tbl.schemaname,tbl.tablename) + from meta.migration_table tbl + where u.rid_migration_table = tbl.id_migration_table + and u.indexname not ilike '%' || tbl.schemaname ||'%' || tbl.tablename || '%s' + ; + + + + + --disable and report duplicates. Take the high priority schema + + with dup as ( + select + t.tablename as tablename_src + , t2.tablename as tablename_dest + , t.schemaname as schemaname_src + , t2.schemaname as schemaname_dest + , t.id_migration_table as id_migration_table_dest + , t2.id_migration_table as id_migration_table_src + from meta.migration_table t + inner join meta.migration_table t2 on t2.ismodel + and t.tablename = t2.tablename + and t.schemaname is distinct from t2.schemaname + and t.schemapriority < t2.schemapriority + where t.ismodel + and exists ( + select * + from meta.migration_option o + where o.name ilike 'default_schema:%' + and (split_part(o.name, ':', 2) = t.tablename + or split_part(o.name, ':', 2) = t2.tablename + ) + ) + ), upd as ( + update meta.migration_table u + set ismodel = false + from dup + where u.id_migration_table = dup.id_migration_table_src + returning * + ) + select jsonb_agg(to_jsonb(dup)) + from dup + into m_json; + + + + insert into meta.table_prefix(schemaname,tablename, prefix) + select distinct t.schemaname,t.tablename,t.prefix + from meta.migration_table t + where t.ismodel + and coalesce(t.prefix,'') <> '' + and not exists ( + select 1 from meta.table_prefix p where p.schemaname = t.schemaname and p.tablename = t.tablename + ); + + update meta.table_prefix u + set prefix = t.prefix + from meta.migration_table t + where t.ismodel + and u.tablename = t.tablename + and u.schemaname = t.schemaname + and u.prefix is distinct from t.prefix + ; + + update meta.migration_relation u + set ismodel = false + from ( + select max(mr.rid_migration_table_child) as rid_migration_table_child + , max(mr.rid_migration_table_parent) as rid_migration_table_parent + , count(1) as cnt + , array_agg(mr.id_migration_relation) as a_id_migration_relation + , min(mr.id_migration_relation) as id_migration_relation_keep + , array_agg(mr.relationname) + ,string_agg(c1.columnname,'_') + from meta.migration_relation mr + inner join meta.migration_relation_col rc on rc.rid_migration_relation = mr.id_migration_relation + inner join meta.migration_table t1 on t1.id_migration_table = mr.rid_migration_table_parent + inner join meta.migration_table t2 on t2.id_migration_table = mr.rid_migration_table_child + inner join meta.migration_column c1 on c1.id_migration_column = rc.rid_migration_column_parent + inner join meta.migration_column c2 on c2.id_migration_column = rc.rid_migration_column_child + where mr.ismodel + + group by t1.schemaname, t1.tablename,t2.schemaname,t2.tablename,c1.columnname,c2.columnname + having count(1) > 1 + ) r + where u.id_migration_relation = any(a_id_migration_relation) + and u.id_migration_relation <> r.id_migration_relation_keep + ; + + /* + update meta.migration_relation u + set ismodel = false + ,isdb = false + from ( + select mr.relationname + , mr.rid_migration_table_child + , mr.rid_migration_table_parent + , count(1) as cnt + , array_agg(mr.id_migration_relation) as a_id_migration_relation + , min(mr.id_migration_relation) as id_migration_relation + from meta.migration_relation mr + where mr.ismodel + group by mr.relationname, mr.rid_migration_table_child, mr.rid_migration_table_parent + ) r + where u.id_migration_relation = any(r.a_id_migration_relation) + and u.id_migration_relation is distinct from r.id_migration_relation + ; +*/ + + + raise notice 'done @ %', (clock_timestamp() - m_tm)::interval; + m_tm = clock_timestamp(); + + + + p_info = coalesce(jsonb_concat(p_info, m_json::jsonb),p_info); + perform pg_notify('upgrade.events', json_build_object('type','upgrade','status',2,'objecttype',m_funcname)::text); + +EXCEPTION + WHEN others THEN + GET STACKED DIAGNOSTICS + m_errmsg = MESSAGE_TEXT + ,m_errcontext = PG_EXCEPTION_CONTEXT + ,m_errdetail = PG_EXCEPTION_DETAIL + ,m_errhint = PG_EXCEPTION_HINT + ,m_errstate = RETURNED_SQLSTATE; + + p_retval = 1; + p_errmsg = format('%s Context %s State: %s', m_errmsg, m_errcontext, m_errstate); + raise warning '% % hint:% state:% context:%',m_errmsg,m_errdetail,m_errhint,m_errstate,m_errcontext; + perform pg_notify('upgrade.events', json_build_object('type','upgrade','status',3,'error',m_errmsg,'objecttype',m_funcname)::text); +END; +$$; \ No newline at end of file diff --git a/todo.md b/todo.md new file mode 100644 index 0000000..3835ab7 --- /dev/null +++ b/todo.md @@ -0,0 +1,100 @@ +# PgTidy — TODO / Progress + +Tracks what is done and what remains. See `plan.md` for the full design and rationale. + +Legend: ✅ done · 🚧 in progress · ⬜ not started + +--- + +## V1 — Formatter + CLI (current milestone) + +### ✅ Scaffold module + repo hygiene +- `go.mod` (`module github.com/hein/pgtidy`, go 1.26). +- Replaced WkMailSync boilerplate: `AGENTS.md` (PgTidy architecture + invariants), + `CLAUDE.md`, `Makefile` (`build`/`test`/`vet`/`fmt`/`lint`/`clean`). +- Directory layout created (`cmd/`, `pkg/...`, `testdata/`). +- Copied 4 real-world procedures into `testdata/corpus/*.pgsql` (safety harness). + +### ✅ Lossless lexer — `pkg/lexer` +- `token.go`: `Kind` enum (trivia, words, literals, operators, punctuation) + `Token`. +- `lexer.go`: full PG token coverage — dollar-quoted strings (`$tag$`, nested-tag aware), + `--` / nestable `/* */` comments, standard/escape/bit/hex/unicode strings, + numbers (decimal, exponent, leading dot, `0x/0o/0b`, `_` separators), positional params + (`$1`), operator runs with PostgreSQL's trailing `+/-` rule, `::` / `:=` / `:`, punctuation. + Tracks line/col per token. +- `lexer_test.go`: unit tests (kinds, operator trailing rule, dollar quotes, line/col) + + **corpus round-trip** asserting `emit(Lex(src)) == src` byte-for-byte. +- **Status:** all tests pass; round-trips all 4 corpus files. Invariant #1 satisfied. + +### ✅ CST node model + parser — `pkg/cst`, `pkg/parser` +- `pkg/cst`: lossless node model — `Tok` (significant token + leading trivia), `Attach`, + `File.Source()` (byte-exact reconstruction), `Raw` (verbatim fallback), `CreateFunction` + (Head/Name/Params/Options/As/Body/Tail/Semi), `Param` (with separator comma). +- `pkg/parser`: statement splitting at top-level `;`; structures CREATE FUNCTION/PROCEDURE + (head, qualified name, comma-split params, option clauses split by keyword, AS + body); + everything else → `Raw`. Graceful degradation is a property of the data model. +- `parser_test.go`: small round-trips, CreateFunction shape assertions, Raw fallback, and + **corpus round-trip** — reconstructs all 4 files byte-for-byte; structures all 4 functions. +- **Status:** all tests pass. +- _Still TODO (later): DML/other-DDL structuring (currently Raw) for full formatting._ + +### ⬜ PL/pgSQL body parser ← NEXT (the remaining V1 piece) +- Parse `$$ ... $$` bodies: DECLARE/BEGIN/END, IF/ELSIF/ELSE/END IF, LOOP/FOR/WHILE, + CASE, assignments (`:=` / `=`), RAISE, nested SQL statements, EXCEPTION blocks. +- The crux for stored-procedure formatting (primary use case). Currently the body is + emitted verbatim; this milestone formats inside it. Reuse `inline`/spacing/casing from + `pkg/format` and keep verbatim fallback for unparsable constructs. + +### ✅ Printer + style config — `pkg/format`, `pkg/config` +- `pkg/config`: `Style` struct + `Default()` = house style (UPPERCASE keywords, lowercase + types, 2-space indent, leading commas, spacing rules). +- `pkg/format`: formats CREATE FUNCTION/PROCEDURE **headers** to house style (params + one-per-line leading-comma, option clauses each on own line, AS/`$$` own lines); body and + `Raw` statements emitted verbatim. Spacing engine (`needSpace`, tight ops `:: : -> ->>`) + + casing (`keywords`/`typeNames` sets). Comment-safety: verbatim fallback if a header + carries comments it cannot relocate. +- Tests: golden header, idempotence, corpus idempotence + **semantic equivalence**. +- _Note: not a full Wadler Doc-IR yet — fixed-layout printer. Doc-IR for width-based + expression wrapping can come when DML structuring lands._ + +### ✅ CLI `fmt` + safety harness — `cmd/pgtidy` +- `pgtidy fmt` (gofmt model): default stdin→stdout; `-w`/`--write`, `-l`/`--list`, + `--check` (CI exit codes); `version`/`help`. _`.pgtidy.yaml` discovery + `-d` diff: TODO._ +- `fmt_test.go`: stdin, --check (un/formatted), -w idempotence, unknown-command. +- Safety invariants #2 (semantic equivalence), #3 (idempotence), #4 (graceful degradation) + are tested in `pkg/format` over the corpus. + +--- + +## V2 — Linter (later) +- `pkg/pgast`: `go-pgquery` (WASM, no cgo) wrapper → real PG AST. +- `pkg/lint`: rule engine + packs — style/consistency, **migration safety** (locks, unsafe + ALTER/ADD COLUMN, non-CONCURRENTLY index, blocking constraints), naming, correctness. +- `pkg/diagnostics`: shared diagnostic type (CLI + LSP). +- `pgtidy lint` subcommand; `--fix` for autofixable rules. + +## V3 — LSP + VSCode (later) +- `pkg/lsp`: formatting + range formatting, publishDiagnostics, codeAction quick-fixes. +- `editors/vscode`: TS extension (`vscode-languageclient`) launching bundled `pgtidy lsp`; + per-platform VSIX matrix in CI (rust-analyzer model) + target-less fallback. + +## V4 — DataGrip (later) +- `editors/datagrip`: integrate via free **LSP4IJ** plugin. + +--- + +## Build / release (cross-cutting) +- ⬜ Add goreleaser for the multi-platform binary matrix (clean: no cgo). +- `make_release.sh` retained from boilerplate (generic version tagging). + +## Core invariants (must always hold — tested) +1. ✅ Lossless lex: `emit(Lex(src)) == src` (corpus round-trip). +2. ✅ Semantic equivalence: formatting changes only trivia/layout (corpus token-stream check). +3. ✅ Idempotence: `fmt(fmt(x)) == fmt(x)` (corpus + CLI tests). +4. ✅ Graceful degradation: unparsable spans pass through verbatim (Raw nodes + verbatim body). + +## Open risks +- Lossless PL/pgSQL recursive-descent parser is the largest effort; pass-through fallback + bounds risk and allows shipping construct-by-construct. +- `go-pgquery` tracks PG17 (not PG18) — fine for lint; irrelevant to formatter path. +- Leading-comma + one-per-line is a first-class style option, not an afterthought.