Compare commits

...
Author SHA1 Message Date
warkanum 32cf25ff73 chore(release): update package version to 1.0.91
Release / test (push) Successful in 54s
Release / release (push) Successful in 2m31s
Release / AUR package (relspec) (push) Successful in 50s
Release / pkg-rpm (push) Successful in 1m50s
Release / AUR package (relspec-bin) (push) Successful in 52s
Release / pkg-deb (push) Successful in 1m32s
2026-10-12 22:15:53 +02:00
warkanum 032093e8c0 feat(packaging): add relspec-bin AUR package
- Add linux/arch/relspec-bin/PKGBUILD using prebuilt release tarballs
- Move source PKGBUILD to linux/arch/relspec/PKGBUILD
- Release: build static binaries, pack linux tarballs (binary + LICENSE)
- AUR job: matrix over relspec and relspec-bin
- Makefile: bump pkgver in both PKGBUILDs
2026-10-12 22:15:07 +02:00
warkanum 69a27784d1 chore(release): update package version to 1.0.90
Release / test (push) Successful in 54s
Release / release (push) Successful in 3m12s
Release / pkg-aur (push) Successful in 52s
Release / pkg-deb (push) Successful in 1m22s
Release / pkg-rpm (push) Successful in 1m55s
2026-10-12 22:06:59 +02:00
warkanum 1d93255575 fix(assets): improve error reporting for asset failures
fix(embed): update embedDirectives to return result directly

fix(executor): clean up import order and formatting

fix(syntax_test): change file permission syntax to octal

fix(mysql): update file permission syntax to octal

fix(pgsql): update file permission syntax to octal

fix(sqlex): simplify Walk function parameter handling

fix(transform): adjust identifier comparison logic

fix(template): clean up function formatting and error handling
2026-10-12 22:04:40 +02:00
warkanum 05466a92b1 fix: preserve MySQL schema fidelity 2026-10-12 21:56:48 +02:00
warkanum dd97413b8e fix: contain generated files and assets and harden SQL rendering 2026-10-12 20:59:46 +02:00
warkanum 2fa48335d3 fix: address audit execution failures and unsafe type conversions 2026-10-12 19:53:19 +02:00
Hein bfe9852826 chore(Makefile): update govulncheck command to use pinned toolchain 2026-10-12 09:49:25 +02:00
Hein e3bc6cf491 chore(go): require Go 1.26.9 for stdlib vulnerability fixes
Release / test (push) Successful in 2m10s
Release / release (push) Successful in 2m48s
Release / pkg-aur (push) Successful in 33m34s
Release / pkg-deb (push) Successful in 1m35s
Release / pkg-rpm (push) Successful in 2m0s
go 1.26.0 made CI's setup-go install a toolchain with 27 known stdlib
vulnerabilities (fixed in 1.26.1-1.26.9).
2026-10-12 09:48:05 +02:00
Hein fa6a5d0339 chore(release): update package version to 1.0.89
Release / test (push) Failing after 2m37s
Release / release (push) Skipped
Release / pkg-aur (push) Skipped
Release / pkg-deb (push) Skipped
Release / pkg-rpm (push) Skipped
2026-10-12 09:33:34 +02:00
Hein 337af294f3 chore(deps): update all dependencies
Fixes GO-2026-6629 (golang.org/x/text precis panic). Bumps bun and
bun/dialect/pgdialect together to v1.3.0; go directive now 1.26.0.
2026-10-12 09:31:55 +02:00
Hein a4c68e5661 ci(release): attach source archives to release and use them for AUR package
Gitea's /archive/ endpoint is disabled, which broke the AUR source download.
Build source tar.gz/zip with git archive, upload as release assets, and point
the PKGBUILD and checksum step at the release asset.
2026-10-12 09:28:52 +02:00
warkanum 9f0269f591 chore(release): update package version to 1.0.88
Release / test (push) Failing after 1m36s
Release / release (push) Skipped
Release / pkg-aur (push) Skipped
Release / pkg-deb (push) Skipped
Release / pkg-rpm (push) Skipped
2026-10-08 00:08:26 +02:00
warkanum b4f18f088b feat(pgsql): unique constraints on apply, matching unique index names, FK deferral
- migration writer now emits unique constraints (direct apply and live diff
  previously dropped them); idempotent via the existing DO block
- unique constraint adopts a same-named model unique index with
  ADD CONSTRAINT ... UNIQUE USING INDEX instead of building a second index
- dbml reader names unnamed unique indexes ukey_<table>_<cols> to match the
  unique constraint naming
- foreign key deferral: Constraint.Deferrable/InitiallyDeferred are *bool
  (nil = unspecified, never diffed); writers emit DEFERRABLE, DEFERRABLE
  INITIALLY DEFERRED and NOT DEFERRABLE; live reader reads condeferrable and
  condeferred; dbml reader/writer support deferrable and initially deferred
- tests and README/naming docs updated
2026-10-08 00:07:49 +02:00
Hein 20f8dd82b6 fix(models): ensure unique GUIDs in InitTable function
Release / test (push) Successful in 2m47s
Release / release (push) Successful in 2m31s
Release / pkg-aur (push) Successful in 1m0s
Release / pkg-deb (push) Successful in 2m7s
Release / pkg-rpm (push) Successful in 2m8s
2026-10-05 17:29:23 +02:00
Hein d26d720e93 chore(release): update package version to 1.0.87
Release / release (push) Canceled after 0s
Release / pkg-aur (push) Canceled after 0s
Release / pkg-deb (push) Canceled after 0s
Release / pkg-rpm (push) Canceled after 0s
Release / test (push) Canceled after 2m42s
2026-10-05 17:27:11 +02:00
warkanum 2796ebe381 chore(release): update package version to 1.0.86
Release / test (push) Failing after 2m29s
Release / release (push) Skipped
Release / pkg-aur (push) Skipped
Release / pkg-deb (push) Skipped
Release / pkg-rpm (push) Skipped
2026-10-03 23:23:20 +02:00
912 changed files with 1392737 additions and 4173807 deletions
+44 -10
View File
@@ -71,11 +71,18 @@ jobs:
EXT=""
[ "$GOOS" = "windows" ] && EXT=".exe"
NAME="relspec-${GOOS}-${GOARCH}${EXT}"
GOOS="$GOOS" GOARCH="$GOARCH" go build \
GOOS="$GOOS" GOARCH="$GOARCH" CGO_ENABLED=0 go build \
-trimpath \
-ldflags "$LDFLAGS" \
-o "$NAME" ./cmd/relspec
echo "Built $NAME"
if [ "$GOOS" = "linux" ]; then
TARNAME="relspec_${VERSION#v}_${GOOS}_${GOARCH}.tar.gz"
cp "$NAME" relspec
tar czf "$TARNAME" relspec LICENSE
rm relspec
echo "Packed $TARNAME"
fi
done
- name: Build Windows installer
@@ -85,6 +92,13 @@ jobs:
VERSION="${{ github.event.inputs.tag || github.ref_name }}"
makensis -DVERSION="${VERSION#v}" -DEXE="$PWD/relspec-windows-amd64.exe" -DOUT="$PWD/relspec-setup-windows-amd64.exe" windows/installer.nsi
- name: Build source archives
run: |
TAG="${{ github.event.inputs.tag || github.ref_name }}"
PREFIX="relspec-${TAG#v}"
git archive --format=tar.gz --prefix="${PREFIX}/" -o "relspec-${TAG}-source.tar.gz" "$TAG"
git archive --format=zip --prefix="${PREFIX}/" -o "relspec-${TAG}-source.zip" "$TAG"
- name: Create release and upload assets
run: |
TAG="${{ github.event.inputs.tag || github.ref_name }}"
@@ -114,7 +128,7 @@ jobs:
exit 1
fi
for f in relspec-*; do
for f in relspec-* relspec_*.tar.gz; do
echo "Uploading $f..."
curl -s -X POST "${UPLOAD_URL}?name=${f}" \
-H "Authorization: token ${GITHUB_TOKEN}" \
@@ -125,8 +139,13 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
pkg-aur:
name: AUR package (${{ matrix.pkg }})
needs: release
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
pkg: [relspec, relspec-bin]
steps:
- uses: actions/checkout@v4
@@ -138,6 +157,7 @@ jobs:
VERSION="${{ github.event.inputs.tag || github.ref_name }}"
PKGVER="${VERSION#v}"
PKG="${{ matrix.pkg }}"
AUR_KEY_PATH="$HOME/.ssh/aur"
AUR_KNOWN_HOSTS="$HOME/.ssh/known_hosts"
@@ -176,10 +196,15 @@ jobs:
# Clone AUR repo
GIT_SSH_COMMAND="ssh -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=$AUR_KNOWN_HOSTS -i $AUR_KEY_PATH" \
git clone ssh://aur@aur.archlinux.org/relspec.git aur-repo
git clone "ssh://aur@aur.archlinux.org/${PKG}.git" aur-repo
CURRENT_PKGVER=$(awk -F= '/^pkgver=/ {print $2; exit}' aur-repo/PKGBUILD | tr -d "[:space:]")
CURRENT_PKGREL=$(awk -F= '/^pkgrel=/ {print $2; exit}' aur-repo/PKGBUILD | tr -d "[:space:]")
if [ -f aur-repo/PKGBUILD ]; then
CURRENT_PKGVER=$(awk -F= '/^pkgver=/ {print $2; exit}' aur-repo/PKGBUILD | tr -d "[:space:]")
CURRENT_PKGREL=$(awk -F= '/^pkgrel=/ {print $2; exit}' aur-repo/PKGBUILD | tr -d "[:space:]")
else
CURRENT_PKGVER=""
CURRENT_PKGREL=0
fi
if [ "$CURRENT_PKGVER" = "$PKGVER" ]; then
case "$CURRENT_PKGREL" in
@@ -195,16 +220,25 @@ jobs:
PKGREL=1
fi
echo "Publishing AUR package version ${PKGVER}-${PKGREL}"
echo "Publishing AUR package ${PKG} ${PKGVER}-${PKGREL}"
# Compute SHA256 of the source archive from the same URL the PKGBUILD will download.
SHA=$(curl -fsSL "https://git.warky.dev/wdevs/relspecgo/archive/v${PKGVER}.zip" | sha256sum | cut -d' ' -f1)
# Compute SHA256 of the artifacts from the same URLs the PKGBUILD will download.
DL="https://git.warky.dev/wdevs/relspecgo/releases/download/v${PKGVER}"
if [ "$PKG" = "relspec-bin" ]; then
SHA_AMD64=$(curl -fsSL "${DL}/relspec_${PKGVER}_linux_amd64.tar.gz" | sha256sum | cut -d' ' -f1)
SHA_ARM64=$(curl -fsSL "${DL}/relspec_${PKGVER}_linux_arm64.tar.gz" | sha256sum | cut -d' ' -f1)
SUMS=(-e "s/^sha256sums_x86_64=.*/sha256sums_x86_64=('${SHA_AMD64}')/" \
-e "s/^sha256sums_aarch64=.*/sha256sums_aarch64=('${SHA_ARM64}')/")
else
SHA=$(curl -fsSL "${DL}/relspec-v${PKGVER}-source.zip" | sha256sum | cut -d' ' -f1)
SUMS=(-e "s/^sha256sums=.*/sha256sums=('${SHA}')/")
fi
# Update PKGBUILD — keep remote source URL, bump version/checksum, and increment pkgrel for same-version rebuilds.
sed -e "s/^pkgver=.*/pkgver=${PKGVER}/" \
-e "s/^pkgrel=.*/pkgrel=${PKGREL}/" \
-e "s/^sha256sums=.*/sha256sums=('${SHA}')/" \
linux/arch/PKGBUILD > aur-repo/PKGBUILD
"${SUMS[@]}" \
"linux/arch/${PKG}/PKGBUILD" > aur-repo/PKGBUILD
# Generate .SRCINFO inside an Arch container (docker cp avoids DinD volume mount issues)
CID=$(docker run -d archlinux:latest sleep infinity)
+20 -10
View File
@@ -16,8 +16,9 @@ GOCLEAN=$(GOCMD) clean
# Tool versions (compiled on demand via `go run` so they match the local toolchain)
GOLANGCI_LINT = go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest
STATICCHECK = go run honnef.co/go/tools/cmd/staticcheck@latest
GOVULNCHECK = go run golang.org/x/vuln/cmd/govulncheck@latest
STATICCHECK = build/tools/staticcheck
# Scan with the toolchain pinned in go.mod (what CI uses), so stdlib vulns match CI.
GOVULNCHECK = GOTOOLCHAIN=go$(shell awk '/^go /{print $$2; exit}' go.mod) go run golang.org/x/vuln/cmd/govulncheck@latest
# Version information
# VERSION file is the single source of truth (bumped by `make release-version`).
@@ -63,7 +64,16 @@ fmt-check: ## Check formatting (gofumpt + goimports via golangci-lint)
@diff=$$($(GOLANGCI_LINT) fmt --diff --config=.golangci.json); \
if [ -n "$$diff" ]; then echo "$$diff"; echo "Run: make fmt"; exit 1; fi
staticcheck: ## Run staticcheck
# staticcheck is built against the latest golang.org/x/tools so it can read export
# data from the newest Go toolchain (`go run ...@latest` uses an older, pinned x/tools).
$(STATICCHECK):
@echo "Building staticcheck..."
@mkdir -p build/tools/src
@cd build/tools/src && { [ -f go.mod ] || go mod init staticcheck-build >/dev/null 2>&1; } && \
go get honnef.co/go/tools/cmd/staticcheck@latest golang.org/x/tools@latest && \
go build -o ../staticcheck honnef.co/go/tools/cmd/staticcheck
staticcheck: $(STATICCHECK) ## Run staticcheck
@echo "Running staticcheck..."
$(STATICCHECK) ./...
@@ -209,7 +219,7 @@ docker-test-integration: docker-up ## Start DB and run integration tests
$(GOTEST) -v ./pkg/readers/pgsql/ -count=1 || (make docker-down && exit 1)
@make docker-down
release: lint fmt-check test build ## Run lint, format check, tests, build, then create and push a new release tag
release: lint fmt-check govulncheck test build ## Run lint, format check, govulncheck, tests, build, then create and push a new release tag
@echo "Creating new release..."
@latest_tag=$$(git describe --tags --abbrev=0 2>/dev/null || echo ""); \
if [ -z "$$latest_tag" ]; then \
@@ -228,9 +238,9 @@ release: lint fmt-check test build ## Run lint, format check, tests, build, then
PKGVER=$${version#v}; \
echo "Updating package metadata to $$PKGVER..."; \
echo "$$PKGVER" > VERSION; \
sed -i "s/^pkgver=.*/pkgver=$$PKGVER/" linux/arch/PKGBUILD; \
sed -i "s/^pkgver=.*/pkgver=$$PKGVER/" linux/arch/relspec/PKGBUILD linux/arch/relspec-bin/PKGBUILD; \
sed -i "s/^Version:.*/Version: $$PKGVER/" linux/centos/relspec.spec; \
git add VERSION linux/arch/PKGBUILD linux/centos/relspec.spec; \
git add VERSION linux/arch/relspec/PKGBUILD linux/arch/relspec-bin/PKGBUILD linux/centos/relspec.spec; \
if ! git diff --cached --quiet; then \
git commit -m "chore(release): update package version to $$PKGVER"; \
fi; \
@@ -243,7 +253,7 @@ release: lint fmt-check test build ## Run lint, format check, tests, build, then
git push origin "$$version"; \
echo "Tag $$version created and pushed to remote repository."
release-version: lint fmt-check ## Run lint and format check, then auto-increment patch version, update VERSION/package files, commit, tag, and push
release-version: lint fmt-check staticcheck govulncheck ## Run lint, format check, staticcheck and govulncheck, then auto-increment patch version, update VERSION/package files, commit, tag, and push
@CURRENT=$$([ -f VERSION ] && cat VERSION | tr -d '[:space:]' || (git describe --tags --abbrev=0 2>/dev/null | sed 's/^v//') || echo "0.0.0"); \
MAJOR=$$(echo $$CURRENT | cut -d. -f1); \
MINOR=$$(echo $$CURRENT | cut -d. -f2); \
@@ -252,15 +262,15 @@ release-version: lint fmt-check ## Run lint and format check, then auto-incremen
NEXT="v$$PKGVER"; \
echo "Current: v$$CURRENT → Next: $$NEXT"; \
echo "$$PKGVER" > VERSION; \
sed -i "s/^pkgver=.*/pkgver=$$PKGVER/" linux/arch/PKGBUILD; \
sed -i "s/^pkgver=.*/pkgver=$$PKGVER/" linux/arch/relspec/PKGBUILD linux/arch/relspec-bin/PKGBUILD; \
sed -i "s/^Version:.*/Version: $$PKGVER/" linux/centos/relspec.spec; \
git add VERSION linux/arch/PKGBUILD linux/centos/relspec.spec; \
git add VERSION linux/arch/relspec/PKGBUILD linux/arch/relspec-bin/PKGBUILD linux/centos/relspec.spec; \
git commit -m "chore(release): update package version to $$PKGVER"; \
git tag -a "$$NEXT" -m "Release $$NEXT"; \
git push origin HEAD "$$NEXT"; \
echo "Pushed $$NEXT — release workflow triggered"
rerelease: lint fmt-check ## Move the latest tag to HEAD and force push it
rerelease: lint fmt-check govulncheck ## Move the latest tag to HEAD and force push it
@TAG=$$(git describe --tags --abbrev=0 2>/dev/null); \
if [ -z "$$TAG" ]; then echo "No existing tags found"; exit 1; fi; \
echo "Moving $$TAG to $$(git rev-parse --short HEAD)"; \
+76
View File
@@ -41,6 +41,9 @@ relspec convert --from pgsql --from-conn "postgres://user:pass@localhost/mydb" \
# DBML → PostgreSQL DDL
relspec convert --from dbml --from-path schema.dbml --to pgsql --to-path schema.sql
# Lowercase model names and structured references before output
relspec convert --from dbml --from-path schema.dbml --to pgsql --to-path schema.sql --lowercase
# PostgreSQL → SQLite (auto flattens schemas)
relspec convert --from pgsql --from-conn "postgres://..." --to sqlite --to-path schema.sql
@@ -57,6 +60,44 @@ connections or `--dry-run`). Source files are polled every `--watch-interval`
is ignored so generating into the source tree does not loop. Conversion errors
are printed and watching continues.
#### Identifier casing and PostgreSQL quoting
PostgreSQL output always double-quotes model identifiers, including reserved
words, and preserves their case by default. For example, schema `Sales` and table
`Order` produce `"Sales"."Order"`; with `--lowercase`, they produce
`"sales"."order"`. Quoted names are case-sensitive in PostgreSQL.
`--lowercase` is available on `convert`, `batch`, `merge`, `split`, `templ`, and
`job run`. It lowercases model names and structured references on a copy and
rejects collisions such as `Users` and `users` in the same schema. SQL expressions,
defaults, comments, enum values, and data retain their case. Adjust identifier
references inside your own SQL expressions when opting into lowercasing.
Generated PostgreSQL SQL also escapes identifier quotes and literal values and
chooses dollar-quote delimiters that do not collide with function or procedural
block contents. Raw SQL expressions remain executable input. See the
[PostgreSQL writer documentation](pkg/writers/pgsql/README.md#identifier-and-value-handling)
for library options and details.
#### MySQL fidelity and strict conversion
MySQL output now preserves foreign keys and their actions, checks, indexes,
unsigned types, dimensions, generated columns, and native defaults. Known
unsupported features produce warnings. Add `--strict-conversion` to reject
lossy MySQL output before writing a file or executing DDL:
```bash
relspec convert --from mysql --from-conn "$MYSQL_DSN" \
--to mysql --to-path schema.sql --strict-conversion
```
Schema jobs can set `options.strict_conversion: true`. This check currently
covers MySQL output; it does not guarantee lossless conversion to every format.
The live suite covers MySQL 8.4 / InnoDB, including JSON/YAML intermediates.
See the [MySQL capability matrix](docs/MYSQL_FIDELITY.md) for unsupported objects,
metadata, SQL-mode requirements, and execution limits. Output creates tables in
existing databases; it is not a data copy or a rerunnable migration.
### `batch` — Convert many inputs in one run
Converts each input independently (one output per input, unlike `--from-list`
@@ -165,6 +206,7 @@ jobs:
overwrite: true
options:
flatten_schema: false
lowercase_identifiers: true # opt in; defaults to false
logfile: .relspec/log/build-schema.log
```
@@ -177,6 +219,40 @@ any job runs. Path-like fields also support `${NAME}` environment-variable
references, which are expanded and safety-checked during pre-flight. See
[docs/JOB_FILES.md](docs/JOB_FILES.md).
Use `options.lowercase_identifiers: true` per job, or
`relspec job run build-schema --lowercase` to enable lowercasing for schema outputs
throughout the selected job plan. For MySQL schema output,
`options.strict_conversion: true` or `--strict-conversion` rejects known lossy
conversions before writing.
### `scripts` — SQL scripts and embedded assets
```bash
relspec scripts list --dir ./migrations
relspec scripts execute --dir ./migrations --conn "$DATABASE_URL"
```
Execution stops on the first error by default. `--continue-on-error` (alias
`--coe`) attempts later scripts and still exits nonzero if any script fails.
Scripts can embed nearby files using a standalone directive comment:
```sql
-- @embed: path=assets/message.txt var=:message mode=text
INSERT INTO messages (body) VALUES (:message);
```
Embed replacement preserves dollar text such as `$1`, `${2}`, and `$HOME` exactly.
Placeholders are replaced only in executable SQL; quoted strings, quoted
identifiers, comments, casts, and dollar-quoted bodies retain their contents.
Inserted content is never scanned for further placeholders. Asset-call named
parameters follow the same SQL lexical boundaries. See
[docs/SCRIPTS_COMMAND.md](docs/SCRIPTS_COMMAND.md) for file containment, limits,
binary embedding, and execution details.
MySQL DDL execution keeps each generated statement whole, preserving multiline
defaults containing semicolons or comment-like text.
### `edit` — Interactive TUI editor
```bash
+1 -1
View File
@@ -1 +1 @@
1.0.85
1.0.91
+2 -1
View File
@@ -203,7 +203,8 @@ func runAssetsExecute(cmd *cobra.Command, args []string) error {
if len(failures) > 0 {
fmt.Fprintf(os.Stderr, "Failed: %d\n", len(failures))
fmt.Fprintf(os.Stderr, "\n⚠ Failed Entries Summary (%d failed):\n", len(failures))
for i, f := range failures {
for i := range failures {
f := &failures[i]
fmt.Fprintf(os.Stderr, " %d. %s (Priority=%d, Sequence=%d)\n Error: %v\n",
i+1, filepath.Base(f.item.Entry.File), f.item.Priority, f.item.Sequence, f.err)
}
+1
View File
@@ -68,6 +68,7 @@ func init() {
batchCmd.Flags().StringVar(&batchNullableTypes, "types", "", "Nullable type package for code-gen writers (bun/gorm)")
batchCmd.Flags().StringVar(&batchNullableArrays, "array-nullable", "", "Nullable array representation for the Bun writer")
batchCmd.Flags().BoolVar(&batchContinueOnError, "continue-on-error", false, "Prepend \\set ON_ERROR_STOP off to generated SQL (pgsql output only)")
batchCmd.Flags().BoolVar(&batchContinueOnError, "coe", false, "Alias for --continue-on-error")
batchCmd.Flags().BoolVar(&batchKeepGoing, "keep-going", false, "Process remaining inputs after a failure; still exits non-zero")
batchCmd.Flags().BoolVar(&batchDryRun, "dry-run", false, "Read and validate every input and print the plan without writing any output")
+31 -4
View File
@@ -28,6 +28,7 @@ import (
"git.warky.dev/wdevs/relspecgo/pkg/readers/sqlite"
"git.warky.dev/wdevs/relspecgo/pkg/readers/typeorm"
"git.warky.dev/wdevs/relspecgo/pkg/readers/yaml"
"git.warky.dev/wdevs/relspecgo/pkg/transform"
"git.warky.dev/wdevs/relspecgo/pkg/writers"
wbun "git.warky.dev/wdevs/relspecgo/pkg/writers/bun"
wdbml "git.warky.dev/wdevs/relspecgo/pkg/writers/dbml"
@@ -180,12 +181,12 @@ Examples:
}
func init() {
convertCmd.Flags().StringVar(&convertSourceType, "from", "", "Source format (dbml, dctx, drawdb, graphql, json, yaml, gorm, bun, drizzle, prisma, typeorm, pgsql, sqlite)")
convertCmd.Flags().StringVar(&convertSourceType, "from", "", "Source format (dbml, dctx, drawdb, graphql, json, yaml, gorm, bun, drizzle, prisma, typeorm, pgsql, mssql, mysql, sqlite)")
convertCmd.Flags().StringVar(&convertSourcePath, "from-path", "", "Source file path (for file-based formats)")
convertCmd.Flags().StringVar(&convertSourceConn, "from-conn", "", "Source connection string (for pgsql) or file path (for sqlite)")
convertCmd.Flags().StringVar(&convertSourceConn, "from-conn", "", "Source connection string (pgsql, mssql, mysql) or file path (sqlite)")
convertCmd.Flags().StringSliceVar(&convertFromList, "from-list", nil, "Comma-separated list of source file paths to read and merge (mutually exclusive with --from-path)")
convertCmd.Flags().StringVar(&convertTargetType, "to", "", "Target format (dbml, dctx, drawdb, graphql, json, yaml, gorm, bun, drizzle, prisma, typeorm, pgsql)")
convertCmd.Flags().StringVar(&convertTargetType, "to", "", "Target format (dbml, dctx, drawdb, graphql, json, yaml, gorm, bun, drizzle, prisma, typeorm, pgsql, mssql, mysql, sqlite)")
convertCmd.Flags().StringVar(&convertTargetPath, "to-path", "", "Target output path (file or directory)")
convertCmd.Flags().StringVar(&convertPackageName, "package", "", "Package name (for code generation formats like gorm/bun)")
convertCmd.Flags().StringVar(&convertSchemaFilter, "schema", "", "Filter to a specific schema by name (required for formats like dctx that only support single schemas)")
@@ -193,6 +194,7 @@ func init() {
convertCmd.Flags().StringVar(&convertNullableTypes, "types", "", "Nullable type package for code-gen writers (bun/gorm): 'baselib' (default, Go pointer types), 'stdlib' (database/sql), or 'sqltypes'")
convertCmd.Flags().StringVar(&convertNullableArrays, "array-nullable", "", "Nullable PostgreSQL array representation for the Bun writer in stdlib/baselib --types mode: 'slice' (default, plain slice) or 'pointer_slice' (*[]T, distinguishes NULL from '{}')")
convertCmd.Flags().BoolVar(&convertContinueOnError, "continue-on-error", false, "Prepend \\set ON_ERROR_STOP off to generated SQL so psql continues past errors (pgsql output only)")
convertCmd.Flags().BoolVar(&convertContinueOnError, "coe", false, "Alias for --continue-on-error")
convertCmd.Flags().StringVar(&convertExtraFields, "extra-fields", "", "Path to JSON file containing extra Bun model fields to inject (bun output only); fields support target_table, name, type, bun_tag, json_tag, comment")
convertCmd.Flags().BoolVar(&convertDryRun, "dry-run", false, "Read and validate the input and print the plan without writing any output")
@@ -447,10 +449,22 @@ func readDatabaseForConvert(dbType, filePath, connString string) (*models.Databa
return db, nil
}
func writeDatabase(db *models.Database, dbType, outputPath, packageName, schemaFilter string, flattenSchema bool, nullableTypes, nullableArrays string, continueOnError bool, extraFields string) error {
func writeDatabase(db *models.Database, dbType, outputPath, packageName, schemaFilter string, flattenSchema bool, nullableTypes, nullableArrays string, continueOnError bool, extraFields string, strict ...bool) error {
if lowercaseIdentifiers {
var err error
schemaFilter = strings.ToLower(schemaFilter)
db, err = transform.LowercaseIdentifiers(db)
if err != nil {
return err
}
}
var writer writers.Writer
writerOpts := newWriterOptions(outputPath, packageName, flattenSchema, nullableTypes, nullableArrays, continueOnError)
if len(strict) > 0 {
writerOpts.StrictConversion = writerOpts.StrictConversion || strict[0]
}
if extraFields != "" {
extraFieldsJSON, err := loadExtraFields(dbType, extraFields)
if err != nil {
@@ -600,6 +614,19 @@ func validateWriteTarget(db *models.Database, dbType, packageName, schemaFilter,
"pgsql", "postgres", "postgresql", "sql",
"mssql", "sqlserver", "mssql2016", "mssql2017", "mssql2019", "mssql2022",
"sqlite", "sqlite3", "prisma", "typeorm", "graphql", "gql":
case "mysql", "mariadb":
selected := db
if schemaFilter != "" {
selected = &models.Database{}
for _, s := range db.Schemas {
if s.Name == schemaFilter {
selected.Schemas = append(selected.Schemas, s)
}
}
}
if _, err := wmysql.NewWriter(newWriterOptions("", packageName, false, "", "", false)).Statements(selected); err != nil {
return err
}
case "gorm", "bun":
if packageName == "" {
return fmt.Errorf("package name is required for %s format (use --package flag)", dbType)
+22 -2
View File
@@ -19,6 +19,7 @@ import (
"git.warky.dev/wdevs/relspecgo/pkg/models"
"git.warky.dev/wdevs/relspecgo/pkg/readers"
"git.warky.dev/wdevs/relspecgo/pkg/readers/sqldir"
"git.warky.dev/wdevs/relspecgo/pkg/transform"
"git.warky.dev/wdevs/relspecgo/pkg/writers"
wpgsql "git.warky.dev/wdevs/relspecgo/pkg/writers/pgsql"
"git.warky.dev/wdevs/relspecgo/pkg/writers/sqlexec"
@@ -522,6 +523,12 @@ func runTemplJob(rj *resolvedJob, lg *jobLogger) error {
return fmt.Errorf("schema not found: %s", schema)
}
}
if lowercaseIdentifiers || rj.job.Options.LowercaseIdentifiers {
db, err = transform.LowercaseIdentifiers(db)
if err != nil {
return err
}
}
mode := rj.job.Mode
if mode == "" {
mode = "database"
@@ -777,6 +784,8 @@ func runScriptsExecJob(rj *resolvedJob, lg *jobLogger) error {
lg.logf("executing %d script(s) against database env:%s", len(combined.Scripts), rj.outputConnEnv)
writer := sqlexec.NewWriter(&writers.WriterOptions{
ContinueOnError: rj.job.Options.ContinueOnError,
AtomicExecution: rj.job.Options.AtomicExecution,
Metadata: map[string]any{
"connection_string": rj.outputConn,
"ignore_errors": rj.job.Options.ContinueOnError,
@@ -791,7 +800,7 @@ func runScriptsExecJob(rj *resolvedJob, lg *jobLogger) error {
success, _ := opts.Metadata["execution_success"].(int)
failed, _ := opts.Metadata["execution_failed"].(int)
lg.logf("executed %d script(s): %d succeeded, %d failed", total, success, failed)
if failed > 0 && !rj.job.Options.ContinueOnError {
if failed > 0 {
return fmt.Errorf("%d script(s) failed", failed)
}
return nil
@@ -835,7 +844,17 @@ func inputLabel(ri resolvedInput) string {
// writeJobOutput writes db to the job's output target (file or database).
func writeJobOutput(rj *resolvedJob, db *models.Database, lg *jobLogger) error {
if lowercaseIdentifiers || rj.job.Options.LowercaseIdentifiers {
var err error
db, err = transform.LowercaseIdentifiers(db)
if err != nil {
return err
}
}
o := rj.job.Options
if lowercaseIdentifiers || o.LowercaseIdentifiers {
o.Schema = strings.ToLower(o.Schema)
}
format := strings.ToLower(rj.job.Output.Format)
if rj.outputConn != "" {
@@ -844,6 +863,7 @@ func writeJobOutput(rj *resolvedJob, db *models.Database, lg *jobLogger) error {
}
lg.logf("writing output to database env:%s", rj.outputConnEnv)
writerOpts := newWriterOptions("", o.Package, o.FlattenSchema, o.Types, o.ArrayNullable, o.ContinueOnError)
writerOpts.AtomicExecution = o.AtomicExecution
writerOpts.Metadata = map[string]interface{}{"connection_string": rj.outputConn, "full_ddl": o.FullDDL}
return wpgsql.NewWriter(writerOpts).WriteDatabase(db)
}
@@ -854,7 +874,7 @@ func writeJobOutput(rj *resolvedJob, db *models.Database, lg *jobLogger) error {
lg.logf("writing output: %s (%s)", rj.outputPath, format)
write := func(target string) error {
return writeDatabase(db, format, target, o.Package, o.Schema, o.FlattenSchema, o.Types, o.ArrayNullable, o.ContinueOnError, "")
return writeDatabase(db, format, target, o.Package, o.Schema, o.FlattenSchema, o.Types, o.ArrayNullable, o.ContinueOnError, "", o.StrictConversion)
}
// Single-file formats are written to a temp file and renamed into place so
// a failure never leaves a partial or truncated output. Directory-emitting
+105
View File
@@ -2,11 +2,15 @@ package main
import (
"bytes"
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/jackc/pgx/v5"
"github.com/spf13/cobra"
"git.warky.dev/wdevs/relspecgo/pkg/jobs"
@@ -741,3 +745,104 @@ jobs:
t.Fatalf("expected templ format rejection, got %v", err)
}
}
func TestLive_JobPostgresFailureStopsDependents(t *testing.T) {
connString := os.Getenv("RELSPEC_TEST_PG_CONN")
if connString == "" {
t.Skip("RELSPEC_TEST_PG_CONN not set")
}
t.Setenv("RELSPEC_TEST_JOB_OUTPUT_CONN", connString)
for _, continueOnError := range []string{"false", "true"} {
t.Run("continue="+continueOnError, func(t *testing.T) {
manifest := `version: 1
jobs:
apply:
command: convert
inputs:
- path: schema/core.dbml
format: dbml
output:
format: pgsql
conn_env: RELSPEC_TEST_JOB_OUTPUT_CONN
options:
full_ddl: true
continue_on_error: ` + continueOnError + `
logfile: .relspec/apply.log
dependent:
command: convert
depends_on: [apply]
inputs:
- path: schema/core.dbml
format: dbml
output:
format: json
path: build/dependent.json
logfile: .relspec/dependent.log
`
dir := jobFixture(t, manifest)
schema := fmt.Sprintf("job_audit_%d", time.Now().UnixNano())
conn, err := pgx.Connect(context.Background(), connString)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
_, _ = conn.Exec(context.Background(), "DROP SCHEMA IF EXISTS "+schema+" CASCADE")
_ = conn.Close(context.Background())
})
// An invalid column type prevents table creation in this isolated schema.
writeFile(t, filepath.Join(dir, "schema", "core.dbml"), "Table "+schema+".users {\n id no_such_type_audit_xyz\n}\n")
set := mustLoadSet(t, filepath.Join(dir, "relspec.yml"))
err = executeJobPlan(set, "dependent", false, false, &bytes.Buffer{})
if err == nil || !strings.Contains(err.Error(), `job "apply" failed`) {
t.Fatalf("expected database failure propagation, got %v", err)
}
for _, path := range []string{"build/dependent.json", ".relspec/dependent.log"} {
if _, err := os.Stat(filepath.Join(dir, path)); !os.IsNotExist(err) {
t.Fatalf("dependent must not start: path=%s err=%v", path, err)
}
}
logData, err := os.ReadFile(filepath.Join(dir, ".relspec/apply.log"))
if err != nil || !strings.Contains(string(logData), "FAILED") {
t.Fatalf("failed application must be logged: %s err=%v", logData, err)
}
})
}
}
func TestLive_JobScriptContinuationStillStopsDependents(t *testing.T) {
connection := os.Getenv("RELSPEC_TEST_PG_CONN")
if connection == "" {
t.Skip("RELSPEC_TEST_PG_CONN not set")
}
t.Setenv("RELSPEC_TEST_EXEC_CONN", connection)
dir := jobFixture(t, `version: 1
jobs:
migrate:
command: scripts-exec
script_dirs: [migrations]
output:
conn_env: RELSPEC_TEST_EXEC_CONN
options:
continue_on_error: true
logfile: .relspec/migrate.log
dependent:
command: convert
depends_on: [migrate]
inputs:
- path: schema/core.dbml
format: dbml
output:
format: json
path: build/dependent.json
`)
writeFile(t, filepath.Join(dir, "migrations", "1_001_bad.sql"), "SELECT 1 / 0;\n")
writeFile(t, filepath.Join(dir, "migrations", "1_002_good.sql"), "SELECT 1;\n")
set := mustLoadSet(t, filepath.Join(dir, "relspec.yml"))
err := executeJobPlan(set, "dependent", false, false, &bytes.Buffer{})
if err == nil || !strings.Contains(err.Error(), `job "migrate" failed`) {
t.Fatalf("failed scripts must fail job: %v", err)
}
if _, err := os.Stat(filepath.Join(dir, "build", "dependent.json")); !os.IsNotExist(err) {
t.Fatalf("dependent ran: %v", err)
}
}
+82
View File
@@ -0,0 +1,82 @@
package main
import (
"bytes"
"os"
"path/filepath"
"strings"
"testing"
"git.warky.dev/wdevs/relspecgo/pkg/models"
)
func TestLowercaseFlagAndFileOutput(t *testing.T) {
prior := lowercaseIdentifiers
t.Cleanup(func() { lowercaseIdentifiers = prior })
for _, cmd := range []struct {
name string
flag bool
}{{"convert", convertCmd.Flags().Lookup("lowercase") != nil}, {"batch", batchCmd.Flags().Lookup("lowercase") != nil}, {"merge", mergeCmd.Flags().Lookup("lowercase") != nil}, {"templ", templCmd.Flags().Lookup("lowercase") != nil}, {"job", jobCmd.PersistentFlags().Lookup("lowercase") != nil}} {
if !cmd.flag {
t.Fatalf("missing %s --lowercase", cmd.name)
}
}
if err := convertCmd.Flags().Set("lowercase", "true"); err != nil {
t.Fatal(err)
}
path := filepath.Join(t.TempDir(), "schema.sql")
db := models.InitDatabase("DB")
s := models.InitSchema("App")
table := models.InitTable("Users", "App")
col := models.InitColumn("Name", "Users", "App")
col.Type = "text"
col.Default = "Keep CASE"
table.Columns[col.Name] = col
s.Tables = append(s.Tables, table)
db.Schemas = append(db.Schemas, s)
if err := writeDatabase(db, "pgsql", path, "", "", false, "", "", false, ""); err != nil {
t.Fatal(err)
}
data, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(data), `"app"."users"`) || !strings.Contains(string(data), `"name" text DEFAULT 'Keep CASE'`) {
t.Fatalf("%s", data)
}
if table.Name != "Users" {
t.Fatal("input mutated")
}
}
func TestJobLowercaseOption(t *testing.T) {
prior := lowercaseIdentifiers
lowercaseIdentifiers = false
t.Cleanup(func() { lowercaseIdentifiers = prior })
dir := t.TempDir()
writeTestJSON(t, filepath.Join(dir, "input.json"), []string{"MixedCase"})
writeFile(t, filepath.Join(dir, "relspec.yml"), `version: 1
jobs:
lower:
command: convert
inputs:
- format: json
path: input.json
output:
format: json
path: output.json
options:
lowercase_identifiers: true
`)
set := mustLoadSet(t, filepath.Join(dir, "relspec.yml"))
if err := executeJobPlan(set, "lower", false, false, &bytes.Buffer{}); err != nil {
t.Fatal(err)
}
data, err := os.ReadFile(filepath.Join(dir, "output.json"))
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(data), "mixedcase") || strings.Contains(string(data), "MixedCase") {
t.Fatalf("%s", data)
}
}
+9
View File
@@ -24,6 +24,7 @@ import (
"git.warky.dev/wdevs/relspecgo/pkg/readers/sqlite"
"git.warky.dev/wdevs/relspecgo/pkg/readers/typeorm"
"git.warky.dev/wdevs/relspecgo/pkg/readers/yaml"
"git.warky.dev/wdevs/relspecgo/pkg/transform"
"git.warky.dev/wdevs/relspecgo/pkg/writers"
wbun "git.warky.dev/wdevs/relspecgo/pkg/writers/bun"
wdbml "git.warky.dev/wdevs/relspecgo/pkg/writers/dbml"
@@ -407,6 +408,14 @@ func readDatabaseForMerge(dbType, filePath, connString, label string) (*models.D
}
func writeDatabaseForMerge(dbType, filePath, connString string, db *models.Database, label string, flattenSchema bool) error {
if lowercaseIdentifiers {
var err error
db, err = transform.LowercaseIdentifiers(db)
if err != nil {
return err
}
}
var writer writers.Writer
switch strings.ToLower(dbType) {
+11 -9
View File
@@ -22,14 +22,16 @@ func newReaderOptions(filePath, connString string) *readers.ReaderOptions {
func newWriterOptions(outputPath, packageName string, flattenSchema bool, nullableTypes, nullableArrays string, continueOnError bool) *writers.WriterOptions {
return &writers.WriterOptions{
OutputPath: outputPath,
PackageName: packageName,
FlattenSchema: flattenSchema,
NullableTypes: nullableTypes,
NullableArrays: nullableArrays,
TypeMappings: typeMappings,
Prisma7: prisma7,
ContinueOnError: continueOnError,
StrictDirectives: strictDirectives,
OutputPath: outputPath,
PackageName: packageName,
FlattenSchema: flattenSchema,
LowercaseIdentifiers: lowercaseIdentifiers,
NullableTypes: nullableTypes,
NullableArrays: nullableArrays,
TypeMappings: typeMappings,
Prisma7: prisma7,
ContinueOnError: continueOnError,
StrictDirectives: strictDirectives,
StrictConversion: strictConversion,
}
}
+15 -8
View File
@@ -12,14 +12,16 @@ import (
// version/buildDate mirror pkg/buildinfo so existing call sites keep working.
// The actual values are set there via ldflags (see Makefile).
var (
version = buildinfo.Version
buildDate = buildinfo.BuildDate
prisma7 bool
noVersion bool
silent bool
strictDirectives bool
typeMapFlags []string
typeMappings map[string]string
version = buildinfo.Version
buildDate = buildinfo.BuildDate
prisma7 bool
noVersion bool
silent bool
strictDirectives bool
strictConversion bool
lowercaseIdentifiers bool
typeMapFlags []string
typeMappings map[string]string
)
var rootCmd = &cobra.Command{
@@ -39,12 +41,16 @@ JSON, YAML, SQL, etc.).`,
}
func init() {
for _, cmd := range []*cobra.Command{convertCmd, batchCmd, mergeCmd, splitCmd, templCmd} {
cmd.Flags().BoolVar(&lowercaseIdentifiers, "lowercase", false, "Lowercase all model identifiers and structured references before output (SQL expressions and data keep their case)")
}
rootCmd.AddCommand(convertCmd)
rootCmd.AddCommand(batchCmd)
rootCmd.AddCommand(diffCmd)
rootCmd.AddCommand(inspectCmd)
rootCmd.AddCommand(scriptsCmd)
rootCmd.AddCommand(jobCmd)
jobCmd.PersistentFlags().BoolVar(&lowercaseIdentifiers, "lowercase", false, "Lowercase all model identifiers and structured references before schema output")
rootCmd.AddCommand(assetsCmd)
rootCmd.AddCommand(templCmd)
rootCmd.AddCommand(editCmd)
@@ -57,6 +63,7 @@ func init() {
rootCmd.PersistentFlags().StringArrayVar(&typeMapFlags, "type-map", nil, "Override a SQL-to-Go type mapping for bun/gorm output as sqltype=gotype (repeatable), e.g. --type-map uuid=uuid.UUID --type-map numeric=decimal.Decimal")
rootCmd.PersistentFlags().BoolVar(&noVersion, "no-version", false, "Suppress the RelSpec version header")
rootCmd.PersistentFlags().BoolVar(&silent, "silent", false, "Suppress progress and status messages (errors are still shown)")
rootCmd.PersistentFlags().BoolVar(&strictConversion, "strict-conversion", false, "Reject known lossy MySQL schema conversions before writing or executing output")
rootCmd.PersistentFlags().BoolVar(&strictDirectives, "strict-directives", false, "Fail on unknown or untranslatable DBML dialect directives (@postgres:, @sqlite:, …)")
}
+3 -1
View File
@@ -63,7 +63,7 @@ var scriptsExecuteCmd = &cobra.Command{
Long: `Execute SQL scripts from a directory against a PostgreSQL database.
Scripts are executed in order: Priority (ascending), Sequence (ascending), Name (alphabetical).
By default, execution stops immediately on the first error. Use --ignore-errors to continue execution.
By default, execution stops immediately on the first error. Use --ignore-errors (or --continue-on-error / --coe) to attempt later scripts; any failure still exits nonzero.
The directory is scanned recursively for all subdirectories and files matching the patterns:
{priority}_{sequence}_{name}.sql or .pgsql (underscore format)
@@ -112,6 +112,8 @@ func init() {
scriptsExecuteCmd.Flags().StringVar(&scriptsSchemaName, "schema", "public", "Schema name (optional, default: public)")
scriptsExecuteCmd.Flags().StringVar(&scriptsDBName, "database", "database", "Database name (optional, default: database)")
scriptsExecuteCmd.Flags().BoolVar(&scriptsIgnoreErrors, "ignore-errors", false, "Continue executing scripts even if errors occur")
scriptsExecuteCmd.Flags().BoolVar(&scriptsIgnoreErrors, "continue-on-error", false, "Attempt later scripts after failures; still exit nonzero")
scriptsExecuteCmd.Flags().BoolVar(&scriptsIgnoreErrors, "coe", false, "Alias for --continue-on-error")
err = scriptsExecuteCmd.MarkFlagRequired("dir")
if err != nil {
+41
View File
@@ -0,0 +1,41 @@
package main
import (
"io"
"os"
"path/filepath"
"strings"
"testing"
"git.warky.dev/wdevs/relspecgo/pkg/jobs"
"git.warky.dev/wdevs/relspecgo/pkg/models"
)
func TestStrictConversionOptionAndDryRun(t *testing.T) {
prior := strictConversion
strictConversion = true
t.Cleanup(func() { strictConversion = prior })
if flag := rootCmd.PersistentFlags().Lookup("strict-conversion"); flag == nil {
t.Fatal("missing --strict-conversion")
}
db := models.InitDatabase("shop")
s := models.InitSchema("shop")
table := models.InitTable("users", "shop")
col := models.InitColumn("value", "users", "shop")
col.Type = "unknown_type"
table.Columns[col.Name] = col
s.Tables = append(s.Tables, table)
db.Schemas = append(db.Schemas, s)
if err := validateWriteTarget(db, "mysql", "", "", ""); err == nil || !strings.Contains(err.Error(), "lossy MySQL") {
t.Fatalf("dry run bypassed strict validation: %v", err)
}
strictConversion = false
out := filepath.Join(t.TempDir(), "out.sql")
rj := &resolvedJob{job: &jobs.Job{Output: &jobs.Output{Format: "mysql"}, Options: jobs.Options{StrictConversion: true}}, outputPath: out}
if err := writeJobOutput(rj, db, &jobLogger{file: io.Discard}); err == nil || !strings.Contains(err.Error(), "lossy MySQL") {
t.Fatalf("job option bypassed strict validation: %v", err)
}
if _, err := os.Stat(out); !os.IsNotExist(err) {
t.Fatalf("strict job created output: %v", err)
}
}
+8
View File
@@ -7,6 +7,7 @@ import (
"github.com/spf13/cobra"
"git.warky.dev/wdevs/relspecgo/pkg/models"
"git.warky.dev/wdevs/relspecgo/pkg/transform"
"git.warky.dev/wdevs/relspecgo/pkg/writers"
wtemplate "git.warky.dev/wdevs/relspecgo/pkg/writers/template"
)
@@ -142,6 +143,13 @@ func runTempl(cmd *cobra.Command, args []string) error {
}
}
if lowercaseIdentifiers {
db, err = transform.LowercaseIdentifiers(db)
if err != nil {
return err
}
}
// Create template writer
fmt.Fprintf(os.Stderr, "Loading template: %s\n", templTemplatePath)
fmt.Fprintf(os.Stderr, "Execution mode: %s\n", templMode)
+2
View File
@@ -191,6 +191,8 @@ jobs:
# (file output = full DDL); set options.full_ddl to force full DDL
overwrite: false # default false
options:
strict_conversion: false # reject known lossy MySQL output before writing
lowercase_identifiers: false # lowercase names/references; SQL expressions and data keep their case
flatten_schema: false
schema: public
package: models # for gorm/bun output
+131
View File
@@ -0,0 +1,131 @@
# MySQL schema fidelity
The MySQL reader/writer preserves the table-schema features below. It generates
DDL for **new tables in existing databases**, not migrations or data copies.
MySQL 8.4 / InnoDB is exercised by the live suite. The reader uses MySQL 8.0.23+
metadata (including functional indexes, checks, spatial metadata, and invisible
columns); older MySQL versions and the `mariadb` alias are not certified by this
suite and may return explicit metadata-query errors.
## Conversion diagnostics
MySQL output warns on stderr when it encounters known unsupported features.
Add `--strict-conversion` to reject these conversions before opening an output
file or connecting to an output database:
```bash
relspec convert --from mysql --from-conn "$MYSQL_DSN" \
--to mysql --to-path schema.sql --strict-conversion
relspec convert --from json --from-path schema.json \
--to mysql --to-path schema.sql --strict-conversion --dry-run
```
The flag also applies to MySQL output from `batch`, `split`, and schema jobs.
Jobs can set `options.strict_conversion: true`. Library callers use
`writers.WriterOptions.StrictConversion`; `Warning func(string)` collects
non-strict diagnostics instead of printing them. `mysql.Writer.Statements`
performs the same preflight without opening a destination and returns structured
SQL statements suitable for execution without splitting on semicolons.
Strict conversion currently covers **MySQL output**. It is not a general
losslessness check for other writers or a validator of arbitrary SQL fragments.
Warnings identify the object and omitted or changed feature. Inspect them before
using non-strict output; unsupported index types are omitted, while a partial
index's predicate and INCLUDE columns cannot be represented in MySQL.
## MySQL capability matrix
| Feature | Reader → model | Model → MySQL DDL |
| --- | --- | --- |
| Ordered columns, nullability, primary/composite keys | Preserved | Preserved |
| Unique/composite keys | Preserved, including backing indexes | Preserved once; no duplicate unique index |
| Foreign/composite keys and DELETE/UPDATE actions | Preserved | Added after all tables, allowing forward references/cycles |
| CHECK and NOT ENFORCED CHECK | Preserved | Preserved |
| BTREE, FULLTEXT, SPATIAL indexes | Preserved | Emitted; engine restrictions still apply |
| Expression/prefix/descending/invisible index parts | Preserved in `Index.KeyParts` and metadata | Preserved |
| HASH indexes | Preserved | Engine-dependent; incompatible engine produces a diagnostic |
| Integer unsignedness; mediumint; enum/set; binary/blob/text variants | Canonical type plus native declaration | Native declaration preserved |
| Length, numeric precision/scale, fractional seconds | Typed dimensions plus native declaration | Preserved; canonical dimensions also supported |
| Defaults, including SQL expressions, bit/binary literals and multiline text | Values plus native SQL spelling from SHOW CREATE | Preserved |
| Auto-increment and current next value | Preserved | Preserved |
| Stored/virtual generated columns, ON UPDATE, invisible columns | Preserved | Preserved |
| Engine, table/column collation, charset and comments | Preserved | Preserved |
| Views, triggers, routines, events, partitioned tables | Names recorded in `mysql_unsupported` | Diagnostic; bodies/partition definitions are not emitted |
| Nonempty table CREATE_OPTIONS; native parser/storage/engine attributes; spatial SRID | Recorded as unsupported attributes | Diagnostic; not emitted |
| Partial/INCLUDE indexes, unsupported index methods | Retained when supplied in the model | Diagnostic |
| Deferrable constraints, identity semantics, timestamptz, jsonb storage | Retained when supplied in the model | Diagnostic for unsupported/different semantics |
| Standalone sequences/enums/domains, ownership/permissions, scripts and dialect directives | Retained when supplied in the model | Diagnostic; not emitted |
| Data, grants, users, server/plugin configuration | Outside the reader's schema contract | Outside the writer's contract |
Diagnostics depend on the objects visible to the connection account. They cover
known modeled/discovered features, not every engine/plugin extension. The server
still validates SQL expressions, engine rules, and references at execution time.
For example, source and target engines must support the constraints requested.
## Preservation across intermediate formats
| Path | Scope |
| --- | --- |
| MySQL → model → MySQL | Live-tested for the features above on MySQL 8.4 / InnoDB |
| MySQL → JSON → model → MySQL | Live-tested; preserves typed index parts and MySQL metadata |
| MySQL → YAML → model → MySQL | YAML serializes the same fields/metadata; covered by the live round-trip test |
| Other source/target formats | Their own mappings apply; MySQL-specific metadata and semantics are not certified to survive |
Native declarations and defaults are stored in column metadata as `mysql_type`
and `mysql_default_sql`. Reader-generated signatures (`mysql_type_signature`,
`mysql_default_value`) prevent these spellings from silently overriding edited
types/dimensions/defaults. Editing canonical dimensions invalidates the native
type signature, produces a diagnostic, and uses the canonical mapping instead.
For a binary default, the logical default is its SQL hex/literal spelling so it
survives serialization without invalid UTF-8.
Other metadata includes `mysql_charset`, `mysql_generation_storage`,
`mysql_default_expression`, `mysql_on_update`, `mysql_invisible`, table
`mysql_engine` / `mysql_collation` / `mysql_auto_increment`, index
`mysql_visible`, and check `mysql_enforced`. Unknown `mysql_*` metadata produces
a diagnostic. For a manually authored expression default, set
`mysql_default_expression: true`; otherwise strings are literal values.
`Index.KeyParts` is authoritative when present; `Index.Columns` remains the
compatibility representation for simple indexes.
Raw native type declarations, default SQL, generation/check/index expressions,
and ON UPDATE expressions are executable SQL input. Lowercasing changes names
and structured index parts but leaves expressions untouched; update references
inside raw SQL yourself when using `--lowercase`.
## Applying the output
Create the destination database/schema first. Qualified names are preserved,
including foreign-key target schemas. Adjust them explicitly when restoring to
a different database. Output assumes standard MySQL backslash escaping;
`NO_BACKSLASH_ESCAPES` or `ANSI_QUOTES` on the source produces a conversion
diagnostic, and live execution refuses these target modes before DDL. File consumers must use the same
supported mode. Other SQL-mode incompatibilities surface as server errors.
`CREATE TABLE IF NOT EXISTS` does not reconcile existing definitions. Foreign
keys use `ALTER TABLE ADD CONSTRAINT` and therefore repeated application can
fail on existing constraints. This is creation DDL, not a rerunnable migration.
Live execution stops on the first SQL error by default. Explicit
`ContinueOnError` attempts later statements and still returns an error if any
fail. MySQL DDL implicitly commits; `AtomicExecution` is rejected. PostgreSQL's
shared execution-report support is not yet implemented for this writer.
## Verification
```bash
./tests/dbtest/dbtest.sh test mysql
# Or use your own disposable database:
RELSPEC_TEST_MYSQL_CONN="$MYSQL_DSN" go test -mod=vendor ./pkg/readers/mysql ./pkg/writers/mysql
```
The live tests compare MySQL's `SHOW CREATE TABLE` definitions, normalizing only
temporary target-name suffixes and nonsemantic ordinary-index display order.
They re-read the target, exercise constraint enforcement/default values, and
verify unsupported objects produce warnings or strict failures without touching
an existing output file. Test fixtures are removed after each run.
Metadata references: [COLUMNS](https://dev.mysql.com/doc/refman/8.4/en/information-schema-columns-table.html),
[STATISTICS](https://dev.mysql.com/doc/refman/8.4/en/information-schema-statistics-table.html),
[TABLE_CONSTRAINTS](https://dev.mysql.com/doc/refman/8.4/en/information-schema-table-constraints-table.html),
and [REFERENTIAL_CONSTRAINTS](https://dev.mysql.com/doc/refman/8.4/en/information-schema-referential-constraints-table.html).
+30 -2
View File
@@ -96,11 +96,39 @@ INSERT INTO assets (message, payload)
VALUES (:message, decode(:payload, 'base64')::bytea);
```
- `path`: File path resolved relative to the SQL file containing the directive
- `path`: Relative file path beneath the SQL file’s directory; traversal, absolute paths, and symlinks are rejected
- `var`: Named placeholder to replace, such as `:message`
- `mode`: `text` embeds an escaped SQL string literal; `base64` embeds a base64 string literal
The directive comment is removed from the SQL, and every matching placeholder is replaced before the script is listed or executed.
A directive must be a standalone SQL line comment. Directive-like text inside
quoted strings, quoted identifiers, block comments, or dollar-quoted bodies is
ordinary SQL content. The directive is removed, and every matching placeholder in
executable SQL is replaced once before listing or execution. Replacements are
literal, preserving `$1`, `${2}`, `$HOME`, and other dollar text; inserted content
is never scanned for another placeholder. Duplicate embed variables or variables
used only inside strings/comments are errors. PostgreSQL casts (`::type`) remain
unchanged. Asset-call named parameters follow these same lexical boundaries;
repeated parameters share one positional argument.
Embed and manifest asset reads share root-scoped containment and a 64 MiB per-file
limit. Only regular files are accepted. The SQL directory reader uses its selected
script tree as an explicit asset root; embed paths remain relative to each script.
Symlinks in asset paths are rejected, and containment is enforced again at open time.
Manifest entries are checked at discovery and execution, so replacing a file with an
escaping symlink after scanning cannot expose an external file. No per-directive
external-file override is supported; copy required assets into the allowed tree.
Library callers can choose a trusted root and size limit with
`assetloader.ProcessEmbedDirectivesWithOptions(..., assetloader.EmbedOptions{
AssetRoot: root, MaxFileBytes: limit})`; zero means the 64 MiB default, and negative
or unbounded limits are rejected. Manifest items retain `AssetRoot` and
`MaxFileBytes` for execution-time validation.
For scripts loaded from a model rather than a SQL directory, the executor requires
trusted writer configuration `Metadata["asset_root"]` before expanding directives.
Model `source_path` metadata supplies resolution context only; it does not grant
local-file access. Missing or out-of-root source paths fail before reading assets.
## Commands
+10
View File
@@ -570,3 +570,13 @@ COMMIT;
- [RelSpec Documentation](../README.md)
- [Model Structure Reference](../pkg/models/)
- [Example Templates](../examples/templates/)
## Output path safety
In schema, domain, script, and table modes, `--output` always names a directory,
including a nonexistent directory without a trailing slash. An empty value uses
the current directory. Database mode uses a single output file as before.
Generated paths must stay relative to that directory. Traversal, absolute or
Windows-specific unsafe paths, symlinks, filename collisions (including case
variants), and file/directory collisions are rejected before publishing artifacts.
Nested patterns such as `{{.ParentSchema.Name}}/{{.Name}}.txt` remain supported.
+359
View File
@@ -0,0 +1,359 @@
# Project audit 1
Date: 2026-10-12
Revision reviewed: `bfe98528265b45be7bfdcdf025710981bae06a7b`
Perspectives: software engineering, security, maintenance, and databases.
## Repair progress
Implementation batches (2026-10-12):
- [x] **A01, PostgreSQL failure propagation:** stop on the first SQL failure by default; explicit continuation still returns a structured `ExecutionFailure`. Reports include skipped entries, and requested report write/close failures are errors. Live tests cover a middle-statement failure in both modes and verify dependent jobs do not start.
- [x] **A02, implicit replacement removed:** both full-DDL and diff type changes propagate conversion errors without renaming/replacing the column. Live populated-data tests verify original values, column count/type, unique-index identity, and constraint enforcement survive incompatible conversions.
- [x] **Related defect exposed by A01:** full-DDL column additions now use native `ADD COLUMN IF NOT EXISTS`, fixing duplicate-column errors for flattened output that previously appeared successful.
- [x] **A01, shared execution results:** PostgreSQL, SQLite, SQL Server, and raw script execution now use `writers.ExecutionReport` / `ExecutionFailure`. Stop/continue modes always fail on SQL errors; counters/results survive early failure and aggregate across script schemas. Reports distinguish applied, failed, skipped, rolled-back, and uncertain operations. Generated DDL supports optional atomic execution; raw scripts report script-managed outcomes without promising statement-level commit/rollback. Requested report failures are errors in every executor. Live PostgreSQL, SQL Server, SQLite, and dependent-job regressions pass.
- [x] **A02, conversion expressions and preflight:** column metadata `pgsql_using` configures an explicit SQL conversion expression; `pgsql_preflight` adds a row predicate. Full-DDL and diff plans evaluate every existing value before schema mutations, and live preflight failure stops even with continuation enabled. Tests cover later invalid rows, preservation of a constraint scheduled for removal, custom conversions, predicate rejection, invalid configuration, and block-delimiter collisions. Standalone schema generation also includes preflights.
- [x] **A03, contained template output:** multi-file output has an explicit directory root, portable relative filename validation, and preflight collision/symlink checks. Rendering/validation complete before writing; exclusive per-file staging and root-scoped rename prevent generated filenames from escaping the root. Multi-file atomic publication remains A16 work.
- [x] **A04, contained asset reads:** embed directives and manifest execution use shared root-scoped reads, symlink checks, regular-file validation, and a bounded 64 MiB default. Manifest paths are rechecked at execution. Imported source-path metadata cannot authorize file reads without a trusted configured asset root. Library callers can explicitly configure a root/size limit; directives cannot opt out of containment.
- [x] **A05, PostgreSQL quoting:** schema/full-DDL/diff paths and embedded templates always quote model identifiers, quote literal/catalog/regclass values, sanitize names in generated line comments, and choose collision-safe dollar delimiters for procedural and audit function bodies. NUL is rejected before output; migration comparisons preserve identifier case. Raw SQL expressions remain an explicit executable-input boundary.
- [x] **Requested lowercase option:** `--lowercase` on convert/batch/merge/split/templ/job run, job `options.lowercase_identifiers`, and PostgreSQL `WriterOptions.LowercaseIdentifiers` normalize names and structured references on a copy, rejecting collisions. SQL expressions, defaults, comments, enum values, and data keep their case; callers must adjust references inside raw SQL themselves.
- [x] **A08, SQL-aware substitution/execution:** embeds use one-pass literal substitution, preserving dollar characters and preventing replacement rescanning. Named parameters and embed directives respect SQL strings, quoted identifiers, nested comments, casts, and dollar bodies; duplicate/unused embed variables fail. MySQL executes structured generated statements without splitting multiline literals or stripping their contents.
- [x] **A06, nullable MySQL catalogs:** constraint references, functional-index columns/expressions, index prefix/direction fields, column dimensions/collations/generation metadata, and optional table auto-increment counters use nullable destinations. Queries qualify joined columns. Live primary/unique/composite/functional-key reads pass, including unsigned auto-increment values beyond signed 64-bit range.
- [x] **A07, scoped MySQL fidelity and diagnostics:** native types/default spellings and typed dimensions/index parts survive MySQL → model → JSON/YAML → DDL. Output emits foreign keys/actions after all tables, checks/enforcement, unique/ordinary/functional/prefix/descending/invisible/fulltext indexes, generated columns, comments/collations/charsets, ON UPDATE, invisible columns, and auto-increment counters. Known unsupported objects/attributes produce warnings or `--strict-conversion` / job `options.strict_conversion` / library `StrictConversion` failures before output or connection. See [the scoped capability matrix](MYSQL_FIDELITY.md); this covers new-table DDL on MySQL 8.4 / InnoDB, not all engines, MariaDB versions, other format mappings, or migrations.
A06–A07 verification (2026-10-12): live MySQL 8.4.11 tests compare source/target `SHOW CREATE TABLE` definitions (normalizing temporary target suffixes and ordinary-index display order), re-read the target, and verify FK/check enforcement and binary/bit/multiline/backslash/default values. The suite covers composite/functional unique keys, reverse table dependency order, generated/check/index expressions containing SQL literals, precision, unsigned ranges, and multiple FKs to the same target. It exposed and fixed truncated binary defaults in `information_schema` by extracting native defaults from SQL-aware `SHOW CREATE TABLE` definitions. Native SHOW CREATE also avoids stale auto-increment statistics, and escaped catalog expressions are decoded before rendering. Unsupported views/triggers/partitioned tables produce warnings and strict failures; strict failure preserves existing files and happens before opening an output connection. CLI dry-run and per-job strict options are tested. The MySQL container test runner now selects reader/writer packages by default. `go test -mod=vendor -p 1 ./...` passed with `RELSPEC_TEST_MYSQL_CONN` configured; `go vet -mod=vendor ./...`, CLI strict/dry-run checks, shell syntax, and `git diff --check` passed. The MySQL container runner also passed using its schema-scoped account; its disposable server enables trigger-fixture creation after final-server readiness. PostgreSQL and SQL Server live variables were unset for this batch. Checks used `go1.27.1-X:nodwarf5 linux/amd64`; declared-minimum-toolchain coverage remains A13 work. Disposable MySQL instances were removed after verification. A08 now also has live MySQL coverage for embedded statement delimiters/default contents.
A05–A08 verification: unit regressions and live PostgreSQL 18 checks cover unusual names, comments/defaults, rerunnable file/statement output, diff migrations, sequences, primary-key replacement, audit functions/triggers, and both standard-string modes. MySQL statement preservation is unit-tested; a live MySQL instance was not used for A08. `go test -mod=vendor -p 1 ./...` passed with `RELSPEC_TEST_PG_CONN` set to the disposable instance; `go vet -mod=vendor ./...`, executable CLI help checks, and `git diff --check` passed. SQL Server tests were not configured for this batch. The disposable instance was removed after verification.
Additional observation: a parallel live full-suite run hit a PostgreSQL reader catalog race (`could not open relation with OID`) while another package's test dropped schema objects. The writer safely fell back to full DDL and its preflight still stopped all mutations, but the existing test assumed no successful read-only preflights before failure. Running live packages serially passed. Reader resilience to concurrent catalog changes remains outside A05/A08; A10 specifically concerns cancellation/timeouts.
A03–A04 verification: `go test -mod=vendor ./...`, targeted regression tests, `go vet -mod=vendor ./...`, and `git diff --check` passed. Database-dependent tests were not configured for this filesystem-only batch. Windows test binaries were cross-compiled for the template and asset packages; Windows runtime filesystem tests remain unperformed.
Verification: `go test -mod=vendor ./...` passed with `RELSPEC_TEST_PG_CONN` and `RELSPEC_TEST_MSSQL_CONN` pointing to disposable PostgreSQL 18 and SQL Server 2022 Developer instances; SQLite regressions use temporary database files. A minimal public table was seeded for the existing PostgreSQL `ReadTable` test, which requires a nonempty default schema. `go vet -mod=vendor ./...` and `git diff --check` passed. Executable CLI checks verified `scripts execute --ignore-errors`, `--continue-on-error`, and `--coe` all attempt a later script after failure and exit 1. Raw script tests intentionally make no statement-level rollback claim. A09 remains open for concurrency, history, and migration recovery; sequence/external side effects are outside transaction rollback guarantees. Original audit evidence is retained for historical context.
## Assessment
RelSpec has a useful architecture, broad format support, and substantial tests. Its most serious weaknesses are in the boundary between generating a schema and safely applying that schema to a real database. A successful process exit does not reliably mean a successful migration. Some conversions silently discard database semantics, and several file/SQL generation paths do not preserve the safety guarantees implemented elsewhere.
**Recommendation: fix the high-priority findings before relying on unattended production database writes or processing externally supplied specifications.** Continue using generated output with review and disposable database validation while these issues are addressed. This is a repository audit, not a certification of production readiness.
### Existing strengths
- Readers, writers, the intermediate model, diffing, inspection, and transformations are separated into packages.
- There are many unit and regression tests, including live PostgreSQL writer tests and SQLite tests.
- Job files have semantic validation, dependency-cycle detection, environment-based connection references, path containment checks, and symlink checks.
- Single-file job conversions already use temporary output followed by rename.
- The asset manifest executor binds file bytes as database parameters.
- The repository includes a private vulnerability reporting policy, vendored dependencies, and release-time vet/static analysis/vulnerability-check steps.
These controls should be extended and made consistent, rather than replaced wholesale.
## Scope and verification
Reviewed the CLI, job handling, database readers/writers, PostgreSQL templates and migration generation, asset embedding, generic templates, connection builder, updater, build/release workflows, documentation, and the VS Code extension. Review depth was greatest for database mutation and input/output trust boundaries; every parser and format combination was not exhaustively audited.
Checks performed:
| Check | Result |
| --- | --- |
| `GOCACHE=/tmp/relspec-audit-gocache go test -mod=vendor ./...` | Initial sandbox run failed because HTTP tests could not open loopback sockets. Approved rerun outside the sandbox passed. |
| `GOCACHE=/tmp/relspec-audit-gocache go vet -mod=vendor ./...` | Passed. |
| `make fmt-check GOLANGCI_LINT=false` | Incorrectly passed, confirming that a failed formatter can be hidden. |
| Temporary Go reproduction using exported package APIs | Confirmed template path escape, embed path escape, embed dollar corruption, invalid PostgreSQL identifier/literal rendering, omitted MySQL FK/index output, and placeholder matching inside SQL strings. |
The local compiler reported `go1.27.1-X:nodwarf5 linux/amd64`; `go.mod` requires `1.26.9`. This was not a verification under the declared minimum toolchain. No live PostgreSQL, MySQL, or SQL Server connection was configured for the audit. Tests requiring those connections skipped; a passing package does not demonstrate live database correctness. Race testing, fuzz campaigns, benchmarks, `staticcheck`, and a fresh `govulncheck` scan were not performed. No claims about specific dependency CVEs are made.
Reproduction files and logs were placed in `/tmp`; the only intended repository change is this document. No real credentials or user data were used in reproductions. Database injection and migration effects were assessed from code/generated SQL, not executed against a database.
### Priority and evidence notation
- **P1 / High:** address before unattended database use or handling less-trusted inputs; credible data-integrity, confidentiality, or integrity impact.
- **P2 / Medium:** schedule in the next maintenance cycle; operational reliability, reproducibility, or defense-in-depth issue.
- **P3 / Low:** documentation or developer-experience issue.
- **Reproduced:** a local executable check demonstrated the stated behavior.
- **Code-confirmed:** the implementation establishes the behavior, but the complete external-system scenario was not executed.
## Findings
### A01 — PostgreSQL writes report success after statement failures
**Repair status: addressed.** The shared executor now enforces stop/continue behavior and nonzero failure across PostgreSQL, SQLite, SQL Server, and raw scripts, including requested report errors. Reports include individual outcomes, skipped operations, confirmed rollbacks, and uncertain finalization. Atomic generated-DDL runs are optional; raw scripts explicitly report script-managed execution. A09 remains a separate migration-management/concurrency contract.
**P1 / High · Engineering, database · Code-confirmed**
Evidence: `pkg/writers/pgsql/writer.go:2200–2322`, especially `executeStatements` and `finishReport`; `pkg/writers/pgsql/live_execute_test.go:144` explicitly tests continuing after failure.
Every failed statement is recorded and execution continues. `finishReport` prints an error count but returns `nil`. Failure to write the requested report is also only a warning. This behavior is unconditional in this execution path. A deployment or dependent job can proceed after an incomplete schema change because the writer returned success.
**Fix:** return a structured error when any statement fails; stop on the first error by default. A continue-on-error option should control continuation, separately from the final exit status. Aggregate results consistently across PostgreSQL, SQLite, SQL Server, and script execution. Treat failure to produce an explicitly requested execution report as an error.
**Acceptance:** force a middle statement to fail; the CLI exits nonzero, dependent jobs do not run, and the report distinguishes applied, failed, skipped, and rolled-back operations. Test both stop and continue modes.
### A02 — Failed type conversion silently substitutes an empty column
**Repair status: addressed.** The shared type-change template propagates conversion errors. Configurable `pgsql_using` expressions and `pgsql_preflight` predicates are supported, with automatic whole-column cast evaluation before schema mutations. Full-DDL and diff live tests verify preservation of populated columns and unique dependencies, custom conversions, and rejected predicates. Preflight is advisory under concurrent writes; atomic generated-DDL execution is available for transactionally supported changes, and separately reviewed rename/replacement remains required.
**P1 / High · Database · Code-confirmed**
Evidence: `pkg/writers/pgsql/templates/alter_column_type_with_check.tmpl:18–27`; `pkg/writers/pgsql/migration_writer.go:491`; `pkg/writers/pgsql/writer.go:514`.
The type-change block catches `WHEN OTHERS`, renames the existing column, and adds a fresh column with the original name. For example, converting populated text values containing non-numeric data to an integer can fail the cast and activate this fallback. The application then sees an empty replacement column. Existing indexes, constraints, and dependencies may remain attached to the renamed column. Original values are not necessarily deleted, but the application's expected column no longer contains them.
The broad exception handler also treats unrelated failures as permission to attempt this schema rewrite.
**Fix:** fail on conversion errors. Require a separately reviewed migration for any rename-and-replace operation, including backfill, validation, dependency handling, and recovery. Support an explicit conversion expression and preflight checks where appropriate.
**Acceptance:** migrating populated incompatible data leaves the original column, values, and dependencies intact and returns an actionable failure. No rename/replacement occurs implicitly.
### A03 — Generated template filenames escape the output directory
**Repair status: addressed.** Multi-file roots, generated paths, collisions, symlinks, and root-scoped writes are covered by regression tests. A nonexistent output path now consistently denotes the directory itself rather than its parent.
**P1 / High · Security · Reproduced**
Evidence: `pkg/writers/template/writer.go:249–308`.
The default filename pattern uses `{{.Name}}.txt`. `generateFilename` joins the rendered name with the destination without checking containment; `writeOutput` creates directories and writes the resulting path. A schema named `../escaped` wrote `escaped.txt` beside the selected output directory in the reproduction.
An imported model can therefore cause writes outside the intended destination, including overwriting files writable by the current user. Job-level checks validate the declared output path, not every subsequently generated filename. Exploitation requires the user to process the supplied model in a multi-file template mode.
**Fix:** validate every rendered filename against a clearly defined output root; reject traversal, unsafe absolute paths, and symlink escapes. Detect filename collisions before writing. Prefer root-scoped filesystem operations so validation and opening cannot be separated by a symlink replacement.
**Acceptance:** malicious names, filename patterns, symlink parents, duplicate names, and platform-specific paths cannot write outside the root or silently overwrite another generated artifact.
### A04 — SQL embed directives can read outside the script directory
**Repair status: addressed.** Shared bounded root-scoped reads protect directives and manifest assets, including revalidation after discovery. Model source-path metadata alone no longer grants local file access.
**P1 / High · Security · Reproduced**
Evidence: `pkg/assetloader/embed.go:73–74`; compare the containment checks in `pkg/assetloader/loader.go` and `pkg/jobs/jobs.go`.
Embed paths are cleaned and joined to the script directory, then passed directly to `os.ReadFile`. A directive using `path=../synthetic-secret.txt` successfully embedded a sibling file. There is no equivalent root or symlink policy for this path. Script metadata also supplies the source path used for resolution.
A supplied script can cause local readable files to be copied into generated SQL or sent to the destination database. Permission to execute SQL against a database should not implicitly grant unrestricted local-file access. This is a local input trust-boundary issue, not an unauthenticated network endpoint.
**Fix:** define an explicit asset root, contain all embed reads within it, check symlink resolution, limit file size, and document any opt-in allowance for external files. Reuse the policy across manifest assets and embed directives.
**Acceptance:** traversal and symlink escape fail before reading; valid in-root text and binary assets still work; diagnostics do not print file contents.
### A05 — PostgreSQL SQL quoting is incomplete across syntactic contexts
**P1 / High · Security, database · Reproduced rendering defects**
Evidence: `pkg/writers/pgsql/template_functions.go:134`; `pkg/writers/pgsql/templates/add_column_with_check.tmpl:5–7`; `create_foreign_key_with_check.tmpl`; `alter_column_type_with_check.tmpl`.
Two separate defects were confirmed:
1. `quoteIdent` does not actually check reserved keywords, despite its comment. The template executor generated `CREATE TABLE IF NOT EXISTS public.order (select text);` for legal names requiring quoting.
2. Catalog lookups insert names directly into single-quoted SQL strings. A table named `O'Reilly` generated `table_name = 'O'Reilly'`. Quoting a name for an identifier elsewhere does not escape it as a string literal here.
Fixed `DO $$ ... $$` delimiters create another context that must be protected from embedded delimiter text. Besides breaking valid database names, insufficient context-aware quoting is an injection risk when less-trusted metadata is rendered and executed using privileged database credentials. A complete injection exploit was not executed during this audit. Raw SQL expressions are a separate, intentional executable-input boundary and should be documented as such.
**Fix:** centralize identifier, literal, qualified-name, and block-delimiter handling. Always quote identifiers where practical; parameterize values where supported. Generate collision-safe dollar delimiters or avoid interpolating arbitrary content into fixed blocks. Apply this across all templates, not only the examples above.
**Acceptance:** generated SQL executes correctly for reserved words, apostrophes, double quotes, Unicode, newlines, and dollar delimiters, and cannot introduce additional SQL statements through a name.
PostgreSQL documents the distinct identifier, string, and dollar-quote rules in [Lexical Structure](https://www.postgresql.org/docs/current/sql-syntax-lexical.html).
### A06 — MySQL constraint reading scans nullable metadata into strings
**Repair status: addressed.** Nullable metadata scans and qualified queries are covered by live MySQL 8.4 round trips with primary, unique, composite FK, and functional-index definitions.
**P1 / High · Database · Code-confirmed; live MySQL not exercised**
Evidence: `pkg/readers/mysql/reader.go:181–220`, specifically the `rs`, `rt`, and `rc` string variables passed to `rows.Scan`.
The constraint query reads `REFERENCED_TABLE_SCHEMA`, `REFERENCED_TABLE_NAME`, and `REFERENCED_COLUMN_NAME` for primary and unique constraints as well as foreign keys. Non-foreign-key rows have no referenced object and can contain SQL NULL. Scanning these into Go strings fails, aborting inspection of ordinary tables with a primary key.
**Fix:** use nullable destinations such as `sql.NullString`, or explicit `COALESCE` where an empty string has the intended meaning. Review other nullable metadata, including expression-index column names, in the same reader.
**Acceptance:** live MySQL tests read tables with primary, unique, composite foreign-key, and expression-index definitions without scan failures and preserve their metadata. The current reader tests only cover type mapping and a missing connection string.
Reference: [MySQL KEY_COLUMN_USAGE documentation](https://dev.mysql.com/doc/refman/8.0/en/information-schema-key-column-usage-table.html) and [MySQL's documented non-FK NULL behavior](https://bugs.mysql.com/bug.php?id=9587).
### A07 — MySQL conversion silently loses constraints, indexes, and type detail
**Repair status: addressed within the [documented MySQL table-schema contract](MYSQL_FIDELITY.md).** Supported features survive live round trips through JSON/YAML; known unsupported modeled/discovered features produce diagnostics or strict conversion failures. MySQL output remains creation DDL for existing databases, with engine/version/SQL-mode limits stated in the capability matrix.
**P1 / High · Database · Reproduced omissions; additional losses code-confirmed**
Evidence: `pkg/writers/mysql/writer.go:96–163`; `pkg/readers/mysql/reader.go:153–178`; `pkg/mariadb/datatypes.go`.
The writer emits columns, primary keys, and unique constraints, but does not emit foreign keys, checks, or ordinary indexes from the model. A reproduction containing both a foreign key and an index returned success while emitting neither. The reader maps `COLUMN_TYPE` to a canonical base type without populating the separate length/precision/scale fields; its tests explicitly map `bigint unsigned` to `int64`.
Consequences include lost referential integrity, missing performance-critical indexes, changed ranges, and changed string/decimal definitions. Advertising a format as supported does not communicate these losses.
**Fix:** preserve and emit supported semantics; return explicit diagnostics for every unsupported object or attribute. Add a strict conversion mode that refuses lossy output and a per-format capability matrix. Preserve unsignedness, dimensions, generated expressions, and FK actions through the intermediate representation.
**Acceptance:** inspect → model → DDL → disposable database → inspect tests compare semantic equivalence for representative MySQL schemas. Unsupported features produce visible, testable failures or warnings, never silent success.
### A08 — Embed substitution corrupts dollar characters; placeholder parsing ignores SQL syntax
**P2 / Medium · Engineering, database · Reproduced**
Evidence: `pkg/assetloader/embed.go:159–160`; `pkg/assetloader/executor.go:32–87`.
`replaceEmbedPlaceholder` passes file content into the replacement argument of `regexp.ReplaceAllString`. Dollar characters are interpreted as capture references. Embedding `price $1 ${2} $HOME $$` produced `price ; $` in the local reproduction, rather than preserving the input.
Separately, `BuildQuery` matches placeholders inside quoted strings and comments. `SELECT ':literal', :bytes` fails with an unknown-placeholder error even though `:literal` is SQL string content. Similar text-based processing makes the MySQL executor's `strings.Split(..., ";\n")` fragile for multiline defaults containing that sequence.
**Fix:** use literal replacement callbacks or explicit span assembly for embed content. Parse SQL lexical contexts for named placeholders. Execute a structured list of generated statements instead of splitting SQL text on delimiters.
**Acceptance:** dollar text, repeated placeholders, quoted strings, comments, casts, dollar-quoted bodies, and embedded statement delimiters preserve exact intended semantics.
### A09 — Migration execution lacks a coherent atomicity and concurrency contract
**P1 / High · Database, engineering · Code-confirmed**
Evidence: `pkg/writers/pgsql/writer.go:2200–2300`; `pkg/writers/sqlexec/writer.go:107–222`; `pkg/writers/sqlite/writer.go:118–164`; `pkg/writers/mysql/writer.go:165–188`.
Executors issue independent statements/scripts without a transaction spanning the operation, migration-wide lock, or applied-script ledger in these paths. A failure can leave earlier work committed. Re-running script execution repeats supplied scripts; concurrent invocations can interleave. A script may contain its own transaction, but that does not establish a consistent executor-level guarantee.
The necessary behavior differs by database and statement type: wrapping all MySQL DDL in a transaction would not provide the same rollback guarantee as transactional PostgreSQL DDL.
**Fix:** define explicit execution modes with transaction boundaries, concurrency control, retry/recovery semantics, and accurate partial-application reports. If `scripts execute` remains a raw runner, document that clearly. If it is intended as a migration manager, add migration identity, checksums, applied history, and duplicate/concurrent-run protection.
**Acceptance:** failure injection and simultaneous-run tests establish predictable behavior; supported transactional runs roll back; nontransactional runs report exactly what committed and how to resume safely.
### A10 — Database operations cannot consistently inherit cancellation or deadlines
**P2 / Medium · Engineering, operations · Code-confirmed**
Evidence: `pkg/readers/reader.go`; `pkg/writers/writer.go`; `pkg/writers/pgsql/writer.go:2205`; `pkg/writers/sqlexec/writer.go`; `pkg/readers/mysql/reader.go:25`; `pkg/writers/mysql/writer.go:184`.
Reader/writer interfaces do not accept a caller context, and several implementations start with `context.Background()`. This disconnects CLI/job cancellation from connection establishment, catalog reads, and SQL execution. A blocked statement or lock wait has no consistent application-level deadline.
**Fix:** carry caller contexts through reader/writer operations; configure connection, statement, and lock-wait limits separately. Use a signal-aware CLI root context and preserve cancellation errors in job results.
**Acceptance:** a blocked database query terminates promptly on cancellation or timeout, cleanup is bounded, and the job exits nonzero without starting dependents.
### A11 — Windows updater executes downloads without artifact authentication
**P2 / Medium · Security, supply chain · Code-confirmed**
Evidence: `cmd/relspec/update.go:115–152`; `.gitea/workflows/release.yml`.
After confirmation or `--yes`, the updater downloads the asset URL from release metadata and starts the resulting executable. There is no signature verification, trusted-publisher check, content-size limit, or asset-origin/redirect restriction in this path. The default release API uses HTTPS, which is a useful transport control, but it is not independent artifact authentication. Compromised release metadata or publication infrastructure could substitute an installer.
**Fix:** sign installers or release manifests and verify against an independently trusted identity before launch. Require HTTPS throughout redirects, apply a deliberate origin policy, cap downloads, and clean up failed downloads. A checksum fetched from the same compromised endpoint alone is insufficient authentication.
**Acceptance:** tampered, unsigned, wrong-publisher, oversized, and disallowed-origin assets are rejected before execution.
### A12 — Connection and development-service defaults permit avoidable exposure
**P2 / Medium · Security · Code-confirmed**
Evidence: `pkg/ui/connstring.go:42–49`; `docker-compose.yml`; `Makefile`'s `docker-up` target.
The UI starts PostgreSQL and SQL Server connection fields with encryption disabled; changing the host does not change that setting. Compose publishes database ports without a loopback host restriction and supplies fixed development passwords. These are development credentials, not evidence of leaked production secrets, but the services can become reachable on host network interfaces.
**Fix:** default remote connections to authenticated encrypted transport and make local exceptions explicit. Bind development database ports to `127.0.0.1` by default; document how to deliberately expose a test service. Prefer a limited inspection role over administrative defaults.
**Acceptance:** changing a connection to a remote host does not silently retain an insecure default; standard local test startup only exposes loopback ports.
### A13 — CI does not consistently test the declared toolchain or live writer paths
**P1 / High · Maintenance, database assurance · Code-confirmed configuration defects**
Evidence: `.github/workflows/ci.yml:15,58,79`; `.github/workflows/integration-tests.yml:59–76`; `.gitea/workflows/release.yml:48–49`; `Makefile:92–108`; `tests/dbtest/dbs/mysql.sh`.
- GitHub CI specifies Go 1.24/1.25 while `go.mod` requires 1.26.9. Automatic toolchain switching can make that matrix misleading; with switching disabled those jobs cannot build the module as declared.
- The GitHub integration job runs directly on the host but connects to `postgres`, the service-container name, rather than the published loopback port. GitHub documents `localhost` for this job layout. This networking conclusion applies to GitHub-hosted execution; the Gitea runner configuration was not inspected.
- `make test-integration` runs the PostgreSQL reader and `tests/integration`, but omits live tests under `pkg/writers/pgsql` and `pkg/writers/sqlexec`. Release `go test ./...` has no configured database service/connection, so those tests can skip there too.
- MySQL has container scaffolding but no Go tests consuming `RELSPEC_TEST_MYSQL_CONN`; this leaves A06 undetected.
- `make test-unit` excludes the entire PostgreSQL reader package, including `queries_pure_test.go`.
**Fix:** use `go-version-file: go.mod`, explicitly test supported toolchains, correct GitHub service networking, and run all live readers and writers against disposable databases in required CI jobs. Make unexpected integration-test skips fail the integration job. Verify which workflows the actual Gitea deployment discovers; CI badges alone do not establish execution.
**Acceptance:** CI artifacts show real reader/writer execution, not just green packages with skipped live tests, and demonstrate MySQL constraint inspection and failing-migration behavior.
Reference: [GitHub PostgreSQL service-container networking](https://docs.github.com/en/actions/tutorials/use-containerized-services/create-postgresql-service-containers).
### A14 — Formatting gate ignores formatter failure
**P2 / Medium · Maintenance · Reproduced**
Evidence: `Makefile:62–66`.
The recipe captures formatter stdout and checks whether it is empty, without preserving the command's failure status. `make fmt-check GOLANGCI_LINT=false` prints `Checking formatting...` and exits successfully. A missing/broken formatter or dependency/toolchain failure can therefore pass this gate without checking formatting.
**Fix:** explicitly fail when the formatter command fails before inspecting its diff output. Pin the formatter version so the gate is reproducible.
**Acceptance:** a formatter that exits nonzero with empty stdout fails the target; clean formatting passes; a real diff fails.
### A15 — Release artifacts can disagree with the requested release tag
**P2 / Medium · Maintenance, supply chain · Code-confirmed**
Evidence: `.gitea/workflows/release.yml:55–94,241–269,297–314`; upload steps around lines 113–130, 278–289, and 360–371; `Makefile:84,143,256,273`.
For manual dispatch with a tag input, checkout does not explicitly select that tag. Binaries and packages are built from the dispatched checkout, while the source archives use the supplied tag and version strings claim that version. These can represent different commits. Uploads use `curl -s` without failing on HTTP error status, so publication failures can also look successful.
Additional reproducibility gaps: ordinary `make build` runs `go mod tidy` through `deps`; analysis tools use `@latest`; `release-version` lacks test/build prerequisites; `rerelease` force-moves an existing release tag.
**Fix:** resolve and validate the release tag once, check out its commit in every artifact job, and record that commit in a manifest. Fail on HTTP errors and verify the uploaded inventory. Keep normal builds read-only with respect to dependency manifests, pin build tools, and publish immutable versions with consistent quality gates.
**Acceptance:** dispatching a historical tag from a newer branch produces binaries and source from the same historical commit; failed uploads fail the workflow; normal builds leave the working tree unchanged.
### A16 — Output replacement guarantees differ across commands and writers
**P2 / Medium · Engineering, maintenance · Code-confirmed**
Evidence: `cmd/relspec/job.go:859–886`; `pkg/writers/template/writer.go:288–308`; `pkg/writers/mysql/writer.go:54–68`; `pkg/writers/json/writer.go:56–58`.
Job single-file conversion has an atomic-write wrapper, but direct writers and template output write or truncate final paths in place. Some writer cleanup functions ignore close errors. A disk error, failed generation, or interrupted multi-file run can replace a previously valid artifact with incomplete output or leave a mixed generation.
**Fix:** centralize output staging and finalization for direct CLI use and jobs. Handle close/flush errors, use exclusive unpredictable temporary files, preflight filename collisions, and define a staging/manifest strategy for multi-file generations. Preserve existing files when generation fails.
**Acceptance:** injected generation/write/finalization failures leave the previous valid output intact and return an error; no command claims success with a partial replacement.
## Perspective-specific maintenance needs
These are follow-up investments and scope gaps, not additional proven vulnerabilities.
### Software engineering
- **A common capability registry:** format construction is repeated across `convert.go`, `diff.go`, `merge.go`, `inspect.go`, and `edit.go`. Job format validation is another list. Centralize reader/writer creation and capabilities so adding a format does not require synchronized switches throughout the CLI.
- **Typed execution options and results:** connection settings, execution counters, and flags are carried through string-keyed metadata maps. Introduce typed configuration/result structures for live execution while retaining metadata for extensions.
- **Model validation before generation:** JSON/YAML readers primarily unmarshal and return models. Establish a shared normalization/validation boundary for nil entries, duplicate objects, unresolved references, incompatible attributes, and absent required fields. The inspector is useful but should not be the only optional path to structural safety.
- **Smaller database writer components:** `pkg/writers/pgsql/writer.go` is 2,643 lines and combines generation, execution, reporting, type/index policy, and statement parsing. Separate these responsibilities to make security and migration changes easier to review.
- **Adversarial parser tests:** no `Fuzz...` tests were found in `pkg` or `cmd`. Add fuzz targets for DBML/directives, ORM readers, embed handling, SQL lexical processing, and invalid intermediate models. Add large-schema benchmarks before introducing concurrency or caching.
### Security
- Document the trust model: SQL scripts, expressions/defaults, asset calls, schema metadata, templates, and release assets have different execution and local-file capabilities.
- Test redaction across every CLI/library logging path, not just the job logger and masked UI preview. Cover URL-encoded passwords, DSN parse errors, and database error messages without printing real credentials in tests.
- Run pinned vulnerability checks in required change-validation workflows and record the tool/database versions. Produce release checksums and an SBOM; signing and verification are covered by A11.
- Use scoped database roles: inspection should not require DDL rights, and schema application should not normally use a server administrator account.
### Maintenance
- Update contradictory documentation. README advertises Go 1.24.0; CONTRIBUTING says 1.25+; `go.mod` requires 1.26.9. TODO says MySQL, watch mode, type mappings, Windows installer, and installer CI are missing even though implementations exist. `tests/dbtest/README.md` still says there is no MySQL driver/reader.
- Document actual feature completeness, not just format names. Distinguish implemented, partial, experimental, and unsupported behavior with links to tests.
- The VS Code extension's `renderTemplate` is a placeholder and syntax validation mainly counts delimiters per line (`vscode-extension/relspec-template-editor/src/extension.ts`). Label it experimental or connect it to the actual Go parser/renderer. Reuse and dispose a single diagnostics collection rather than creating one on each validation.
- Align GitHub and Gitea workflow ownership, tool versions, required checks, release policies, and developer setup. Add Windows execution tests for filesystem behavior and installer/updater code, rather than relying solely on cross-compilation.
### Database specialization
- Publish an object/attribute fidelity matrix covering every supported source/destination, including views, sequences, enums, generated/identity columns, FK actions/deferral, expression/partial indexes, precision/scale, unsigned types, and permissions.
- PostgreSQL generation needs explicit fidelity review beyond tables: the model includes views, enums, permissions, and standalone sequences, but the main PostgreSQL writer does not provide corresponding complete emission paths. Its sequence generation focuses on table primary keys; scanning views to infer extensions does not emit the views. Unsupported objects need warnings or rejection.
- Establish migration safety policy: destructive-change approval in the product, reviewed plans, populated-data preflight, lock budgets, backfills, recovery, and post-apply verification. A dry run that only validates options is not proof that a migration can execute safely.
- Test semantic convergence: after applying a model, re-inspect and diff; a second application should generate no unintended change. Include populated tables, cross-schema dependencies, generated columns, reserved names, and failure recovery.
- Keep database-specific transaction and DDL behavior explicit. Do not promise uniform rollback or schema support across PostgreSQL, SQLite, SQL Server, and MySQL.
## Prioritized repair plan
| Order | Work | Completion evidence |
| --- | --- | --- |
| 1 | Fix misleading success and unsafe type fallback: A01–A02 | Failure-injection tests prove nonzero exit and preservation of existing data/schema. |
| 2 | Close file and SQL rendering boundaries: A03–A05, A08 | Adversarial filename, embed, identifier, literal, and delimiter cases pass. |
| 3 | Repair MySQL reading and fidelity: A06–A07 | Live round trips preserve constraints, indexes, and type dimensions; unsupported features fail explicitly. |
| 4 | Restore trustworthy CI coverage: A13–A14 | All relevant live writers run, toolchain matches policy, and tool failures fail gates. |
| 5 | Define execution safety: A09–A10 | Documented transaction/concurrency modes, cancellation, recovery, and precise results. |
| 6 | Harden distribution and output: A11–A12, A15–A16 | Verified installer artifacts, consistent release commit, safer defaults, and staged file replacement. |
| 7 | Reduce maintenance debt | Capability registry, typed options, accurate docs, supported-feature matrix, fuzzing, and measured performance baseline. |
Avoid expanding the format list or adding a web/API layer until the existing execution and fidelity contracts are dependable. The current code already offers enough functionality that correctness, safety, and maintainability will yield more value than additional breadth.
+23 -25
View File
@@ -1,27 +1,26 @@
module git.warky.dev/wdevs/relspecgo
go 1.25.13
go 1.26.9
require (
github.com/gdamore/tcell/v2 v2.13.9
github.com/go-sql-driver/mysql v1.9.3
github.com/gdamore/tcell/v2 v2.13.10
github.com/go-sql-driver/mysql v1.10.1
github.com/google/uuid v1.6.0
github.com/jackc/pgx/v5 v5.9.2
github.com/microsoft/go-mssqldb v1.10.0
github.com/jackc/pgx/v5 v5.11.0
github.com/microsoft/go-mssqldb v1.11.2
github.com/rivo/tview v0.42.0
github.com/spf13/cobra v1.10.2
github.com/stretchr/testify v1.11.1
github.com/uptrace/bun v1.2.18
github.com/uptrace/bun/dialect/pgdialect v1.2.18
golang.org/x/text v0.39.0
github.com/stretchr/testify v1.12.1
github.com/uptrace/bun v1.3.0
github.com/uptrace/bun/dialect/pgdialect v1.3.0
golang.org/x/text v0.43.0
gopkg.in/yaml.v3 v3.0.1
modernc.org/sqlite v1.50.1
modernc.org/sqlite v1.60.1
)
require (
filippo.io/edwards25519 v1.1.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
filippo.io/edwards25519 v1.2.0 // indirect
github.com/dustin/go-humanize v1.1.0 // indirect
github.com/gdamore/encoding v1.0.1 // indirect
github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9 // indirect
github.com/golang-sql/sqlexp v0.1.0 // indirect
@@ -31,25 +30,24 @@ require (
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/jinzhu/inflection v1.0.0 // indirect
github.com/kr/pretty v0.3.1 // indirect
github.com/lucasb-eyer/go-colorful v1.4.0 // indirect
github.com/mattn/go-isatty v0.0.22 // indirect
github.com/ncruces/go-strftime v1.0.0 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/lucasb-eyer/go-colorful v1.4.1 // indirect
github.com/mattn/go-isatty v0.0.24 // indirect
github.com/ncruces/go-strftime v1.1.0 // indirect
github.com/puzpuzpuz/xsync/v3 v3.5.1 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/rogpeppe/go-internal v1.14.1 // indirect
github.com/shopspring/decimal v1.4.0 // indirect
github.com/shopspring/decimal v1.5.0 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc // indirect
github.com/vmihailenco/msgpack/v5 v5.4.1 // indirect
github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
golang.org/x/crypto v0.53.0 // indirect
golang.org/x/net v0.56.0 // indirect
golang.org/x/sync v0.21.0 // indirect
golang.org/x/sys v0.46.0 // indirect
golang.org/x/term v0.44.0 // indirect
modernc.org/libc v1.72.3 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
golang.org/x/crypto v0.58.0 // indirect
golang.org/x/sync v0.24.0 // indirect
golang.org/x/sys v0.49.0 // indirect
golang.org/x/term v0.47.0 // indirect
modernc.org/libc v1.78.0 // indirect
modernc.org/mathutil v1.7.1 // indirect
modernc.org/memory v1.11.0 // indirect
modernc.org/memory v1.12.1 // indirect
)
+66 -67
View File
@@ -1,38 +1,36 @@
filippo.io/edwards25519 v1.1.0 h1:FNf4tywRC1HmFuKW5xopWpigGjJKiJSV0Cqo0cJWDaA=
filippo.io/edwards25519 v1.1.0/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.1 h1:jHb/wfvRikGdxMXYV3QG/SzUOPYN9KEUUuC0Yd0/vC0=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.1/go.mod h1:pzBXCYn05zvYIrwLgtK8Ap8QcjRg+0i76tMQdWN6wOk=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.13.1 h1:Hk5QBxZQC1jb2Fwj6mpzme37xbCDdNTxU7O9eb5+LB4=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.13.1/go.mod h1:IYus9qsFobWIc2YVwe/WPjcnyCkPKtnHAqUYeebc8z0=
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.1 h1:zvXfGJCWvywnCA814d8ZiVyt+fm9nnTE8xSb99zRyfo=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.1/go.mod h1:iptorS+VYKFL2N6PnebpS91dubG35eAOEERnT4PJbQU=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.1 h1:u93s+zU2JD62im61Bm5CZIc1ZrOJaIAWEg0WOrMVkEo=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.1/go.mod h1:oXtinPO4OLj9d1DOTrqrL1oRwGhcqadvAmrl6wTeGlk=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0 h1:fhqpLE3UEXi9lPaBRpQ6XuRW0nU7hgg4zlmZZa+a9q4=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0/go.mod h1:7dCRMLwisfRH3dBupKeNCioWYUZ4SS09Z14H+7i8ZoY=
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azkeys v1.4.0 h1:E4MgwLBGeVB5f2MdcIVD3ELVAWpr+WD6MUe1i+tM/PA=
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azkeys v1.4.0/go.mod h1:Y2b/1clN4zsAoUd/pgNAQHjLDnTis/6ROkUfyob6psM=
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azkeys v1.5.0 h1:MaKvxE6D0KkjOg6Wd9M00iqP5PR0kUxCfiezes4JweM=
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azkeys v1.5.0/go.mod h1:i2h9fsTFKZorh8RdV2IcSUf/Qj98GlTkrTvUbX/s8as=
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/internal v1.2.0 h1:nCYfgcSyHZXJI8J0IWE5MsCGlb2xp9fJiXyxWgmOFg4=
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/internal v1.2.0/go.mod h1:ucUjca2JtSZboY8IoUqyQyuuXvwbMBVwFOm0vdQPNhA=
github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0 h1:XRzhVemXdgvJqCH0sFfrBUTnUJSBrBf7++ypk+twtRs=
github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0/go.mod h1:HKpQxkWaGLJ+D/5H8QRpyQXA1eKjxkFlOMwck5+33Jk=
github.com/AzureAD/microsoft-authentication-library-for-go v1.8.0 h1:Nljr4q1GRA/5vCrMONS+g4u4LRHNgOXVSh3O43J2CnI=
github.com/AzureAD/microsoft-authentication-library-for-go v1.8.0/go.mod h1:Y33QHnf0FfdVewFFISOGe20mkZbxX4H839o955/PoeI=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/dustin/go-humanize v1.1.0 h1:dbKTrvD0klcbBV/h4AWJdMuZogJACoMlvWIWZ5b2xWg=
github.com/dustin/go-humanize v1.1.0/go.mod h1:hc1CvRkJMsgxqjmjMQF3QNRAZBwY8AXBAzKYoSX9sFI=
github.com/gdamore/encoding v1.0.1 h1:YzKZckdBL6jVt2Gc+5p82qhrGiqMdG/eNs6Wy0u3Uhw=
github.com/gdamore/encoding v1.0.1/go.mod h1:0Z0cMFinngz9kS1QfMjCP8TY7em3bZYeeklsSDPivEo=
github.com/gdamore/tcell/v2 v2.13.9 h1:uI5l3DYPcFvHINKlGft+en23evOKL+dwtD21QR8ejVA=
github.com/gdamore/tcell/v2 v2.13.9/go.mod h1:+Wfe208WDdB7INEtCsNrAN6O2m+wsTPk1RAovjaILlo=
github.com/go-sql-driver/mysql v1.9.3 h1:U/N249h2WzJ3Ukj8SowVFjdtZKfu9vlLZxjPXV1aweo=
github.com/go-sql-driver/mysql v1.9.3/go.mod h1:qn46aNg1333BRMNU69Lq93t8du/dwxI64Gl8i5p1WMU=
github.com/gdamore/tcell/v2 v2.13.10 h1:Afs3JKt83HnhuUKdZ3MnxUgOqQRWftj5JyDqv1LLynA=
github.com/gdamore/tcell/v2 v2.13.10/go.mod h1:+Wfe208WDdB7INEtCsNrAN6O2m+wsTPk1RAovjaILlo=
github.com/go-sql-driver/mysql v1.10.1 h1:arlSnNLq6a5yxGxV7qg9lF4j0C+KwD6NbQyKr9QL6ME=
github.com/go-sql-driver/mysql v1.10.1/go.mod h1:M+cqaI7+xxXGG9swrdeUIoPG3Y3KCkF0pZej+SK+nWk=
github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY=
github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9 h1:au07oEsX2xN0ktxqI+Sida1w446QrXBRJ0nee3SNZlA=
github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9/go.mod h1:8vg3r2VgvsThLBIFL93Qb5yWzgyZWhEmBwUJWevAkK0=
github.com/golang-sql/sqlexp v0.1.0 h1:ZCD6MBpcuOVfGVqsEmY5/4FtYiKz6tSyUv9LPEDei6A=
github.com/golang-sql/sqlexp v0.1.0/go.mod h1:J4ad9Vo8ZCWQ2GMrC4UCQy1JpCbwU9m3EOqtpKwwwHI=
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs=
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA=
github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3 h1:LMLX+LgTNWpfvCBdFebv6EsYotImrt/Ppc5cXIriCSo=
github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3/go.mod h1:jl5iWTm0/hd5PjEYEOuwAJ57L/CibdZfrqZ5XA5GrCk=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
@@ -43,8 +41,8 @@ github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsI
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgx/v5 v5.9.2 h1:3ZhOzMWnR4yJ+RW1XImIPsD1aNSz4T4fyP7zlQb56hw=
github.com/jackc/pgx/v5 v5.9.2/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/pgx/v5 v5.11.0 h1:IzBBtyK9AHqf98cctWFifYSci2hgQR/cd56wB4p+ogg=
github.com/jackc/pgx/v5 v5.11.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/jinzhu/inflection v1.0.0 h1:K317FqzuhWc8YvSVlFMCCUb36O/S9MCKRDI7QkRKD/E=
@@ -55,18 +53,17 @@ github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/lucasb-eyer/go-colorful v1.4.0 h1:UtrWVfLdarDgc44HcS7pYloGHJUjHV/4FwW4TvVgFr4=
github.com/lucasb-eyer/go-colorful v1.4.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
github.com/mattn/go-isatty v0.0.22 h1:j8l17JJ9i6VGPUFUYoTUKPSgKe/83EYU2zBC7YNKMw4=
github.com/mattn/go-isatty v0.0.22/go.mod h1:ZXfXG4SQHsB/w3ZeOYbR0PrPwLy+n6xiMrJlRFqopa4=
github.com/microsoft/go-mssqldb v1.10.0 h1:pHEt+Qz6YFPWqREq10mqSE524QQo+/QremwTCQht7TY=
github.com/microsoft/go-mssqldb v1.10.0/go.mod h1:mnG7lGa9iYJbzJqGCXyuQCegStKMr3kogDLD6+bmggg=
github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
github.com/lucasb-eyer/go-colorful v1.4.1 h1:1EO+WB73+EH8EVbzlrG3KLAfEypQWVHIBqlTf+2hNss=
github.com/lucasb-eyer/go-colorful v1.4.1/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI=
github.com/mattn/go-isatty v0.0.24/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A=
github.com/microsoft/go-mssqldb v1.11.2 h1:FCgeBIK8um2+X4tbun6Q71N1KsfyCDPKY41e1yGVjSE=
github.com/microsoft/go-mssqldb v1.11.2/go.mod h1:CYgwG5AMXFojbjTg+GNP5G/y6uz1BhTyZaPqQWzkGnQ=
github.com/ncruces/go-strftime v1.1.0 h1:vtPmPWMkb539RGV9WpWT9EwJGvrOCKvMNI4obWL47Hk=
github.com/ncruces/go-strftime v1.1.0/go.mod h1:yVaq7iwCN09vHRrKBrrflXT0M9tTMG2KmZ9Fj7xeG/g=
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c h1:+mdjkGKdHQG3305AYmdv1U2eRNDiU2ErMBj1gwrq8eQ=
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c/go.mod h1:7rwL4CYBLnjLxUqIJNnCWiEdr3bn6IUYi15bNlnbCCU=
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/puzpuzpuz/xsync/v3 v3.5.1 h1:GJYJZwO6IdxN/IKbneznS6yPkVC+c3zyY/j19c++5Fg=
github.com/puzpuzpuz/xsync/v3 v3.5.1/go.mod h1:VjzYrABPabuM4KyBh1Ftq6u8nhwY5tBPKP9jpmh0nnA=
@@ -80,8 +77,8 @@ github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/f
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k=
github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
github.com/shopspring/decimal v1.5.0 h1:HxSWk31RmJv0EyY4oLiGQB9u7WPludUwPBdvQZnYFPc=
github.com/shopspring/decimal v1.5.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
@@ -90,65 +87,67 @@ github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3A
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc h1:9lRDQMhESg+zvGYmW5DyG0UqvY96Bu5QYsTLvCHdrgo=
github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc/go.mod h1:bciPuU6GHm1iF1pBvUfxfsH0Wmnc2VbpgvbI9ZWuIRs=
github.com/uptrace/bun v1.2.18 h1:3HnRcMfS6OBPMG1eSOzlbFJ/X/AyMEJb7rMxE6VQvDU=
github.com/uptrace/bun v1.2.18/go.mod h1:wNltaKJk4JtOt4SG5I5zmA7v0/Mzjh1+/S906Rayd3Y=
github.com/uptrace/bun/dialect/pgdialect v1.2.18 h1:IZ6nM2+OYrL8lkEAy7UkSEZvoa3vluTAUlZfPtlRB2k=
github.com/uptrace/bun/dialect/pgdialect v1.2.18/go.mod h1:Tqdf4QP1okrGYpXfodXvCOK6Ob1OOTwSaoAzCgBB3IU=
github.com/uptrace/bun v1.3.0 h1:hUcbfRpbQk4/upFSI+uZf6jSJw5A0lkIKt0XdpvbUZo=
github.com/uptrace/bun v1.3.0/go.mod h1:wNltaKJk4JtOt4SG5I5zmA7v0/Mzjh1+/S906Rayd3Y=
github.com/uptrace/bun/dialect/pgdialect v1.3.0 h1:0ZkGqGugxaPCxko4DV2f4d7i+FnM2Qj3rNpChfhiv54=
github.com/uptrace/bun/dialect/pgdialect v1.3.0/go.mod h1:kP5HGEpAX8iJCqj3WHm04XnoeBPacdCN1P2HiwMtHpg=
github.com/vmihailenco/msgpack/v5 v5.4.1 h1:cQriyiUvjTwOHg8QZaPihLWeRAAVoCpE00IUPn0Bjt8=
github.com/vmihailenco/msgpack/v5 v5.4.1/go.mod h1:GaZTsDaehaPpQVyxrf5mtQlH+pc21PIudVV/E3rRQok=
github.com/vmihailenco/tagparser/v2 v2.0.0 h1:y09buUbR+b5aycVFQs/g70pqKVZNBmxwAhO7/IwNM9g=
github.com/vmihailenco/tagparser/v2 v2.0.0/go.mod h1:Wri+At7QHww0WTrCBeu4J6bNtoV6mEfg5OIWRZA9qds=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto=
golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54=
golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c=
golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/net v0.60.0 h1:79p50tfZlm0J9YfoDsSi639qSXNGVwEzOPLCxM2FsYU=
golang.org/x/net v0.60.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.44.0 h1:0rLvDRCtNj0gZkyIXhCyOb2OAzEhLVqc4B+hrsBhrmc=
golang.org/x/term v0.44.0/go.mod h1:7ze4MdzUzLXpSAoFP1H0bOI9aXDqveSvatT5vKcFh2Y=
golang.org/x/term v0.47.0 h1:cpyOwQkr2Nzxe2p6KnrTG7Ei2wtzjsuSjhpXXepZ/BM=
golang.org/x/term v0.47.0/go.mod h1:efAfAuG6RKABvg1ArGebRkvX6LXul14SM67o2e4mUCY=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.39.0 h1:UbZz4pLOvn600D6Oh6GGEI6VAmndrEBLv8/6BEXzyus=
golang.org/x/text v0.39.0/go.mod h1:3UwRclnC2g0TU9x8PZiyfOajCd1zaUNHF9cvqcQZ+ZM=
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
golang.org/x/tools v0.51.0 h1:k4Xc/1Om9jwkBJBo4NVLMSARBoWtK10mx+W5BnXCeAI=
golang.org/x/tools v0.51.0/go.mod h1:9eEncMayCV6zRMGhR5eZEC2iBx98qWcF1HZ9Z7wJOoA=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
@@ -156,30 +155,30 @@ gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EV
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
modernc.org/cc/v4 v4.28.2 h1:3tQ0lf2ADtoby2EtSP+J7IE2SHwEJdP8ioR59wx7XpY=
modernc.org/cc/v4 v4.28.2/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=
modernc.org/ccgo/v4 v4.34.0 h1:yRLPFZieg532OT4rp4JFNIVcquwalMX26G95WQDqwCQ=
modernc.org/ccgo/v4 v4.34.0/go.mod h1:AS5WYMyBakQ+fhsHhtP8mWB82KTGPkNNJDGfGQCe0/A=
modernc.org/cc/v4 v4.30.1 h1:ZXInYH2CD24GPWSwOFPwB6JrnFt4/6d7ceW1Ukd30Bk=
modernc.org/cc/v4 v4.30.1/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=
modernc.org/ccgo/v4 v4.36.2 h1:j6d5HuXtBxF00vEcI5RE0O6EIrG6RyRpgMiGV2IK9ys=
modernc.org/ccgo/v4 v4.36.2/go.mod h1:Wh8QlPZq/US2DrsCc31FxJVBOEEcN2BBfla6OQDHydA=
modernc.org/fileutil v1.4.0 h1:j6ZzNTftVS054gi281TyLjHPp6CPHr2KCxEXjEbD6SM=
modernc.org/fileutil v1.4.0/go.mod h1:EqdKFDxiByqxLk8ozOxObDSfcVOv/54xDs/DUHdvCUU=
modernc.org/gc/v2 v2.6.5 h1:nyqdV8q46KvTpZlsw66kWqwXRHdjIlJOhG6kxiV/9xI=
modernc.org/gc/v2 v2.6.5/go.mod h1:YgIahr1ypgfe7chRuJi2gD7DBQiKSLMPgBQe9oIiito=
modernc.org/gc/v3 v3.1.2 h1:ZtDCnhonXSZexk/AYsegNRV1lJGgaNZJuKjJSWKyEqo=
modernc.org/gc/v3 v3.1.2/go.mod h1:HFK/6AGESC7Ex+EZJhJ2Gni6cTaYpSMmU/cT9RmlfYY=
modernc.org/gc/v3 v3.2.0 h1:/Y3LGDMQaoYf55PebP1qsZ5FDGgAZyG1RYKEHeQzh1o=
modernc.org/gc/v3 v3.2.0/go.mod h1:HFK/6AGESC7Ex+EZJhJ2Gni6cTaYpSMmU/cT9RmlfYY=
modernc.org/goabi0 v0.2.0 h1:HvEowk7LxcPd0eq6mVOAEMai46V+i7Jrj13t4AzuNks=
modernc.org/goabi0 v0.2.0/go.mod h1:CEFRnnJhKvWT1c1JTI3Avm+tgOWbkOu5oPA8eH8LnMI=
modernc.org/libc v1.72.3 h1:ZnDF4tXn4NBXFutMMQC4vtbTFSXhhKzR73fv0beZEAU=
modernc.org/libc v1.72.3/go.mod h1:dn0dZNnnn1clLyvRxLxYExxiKRZIRENOfqQ8XEeg4Qs=
modernc.org/libc v1.78.0 h1:Lpm+e9K6ruVYEGwWfuBO1xXtYEaXDouFOG1SuWjFDP4=
modernc.org/libc v1.78.0/go.mod h1:wvxR55nOF2/66NkJoNV7/coXBTCvT3HR22rcqW6lOuA=
modernc.org/mathutil v1.7.1 h1:GCZVGXdaN8gTqB1Mf/usp1Y/hSqgI2vAGGP4jZMCxOU=
modernc.org/mathutil v1.7.1/go.mod h1:4p5IwJITfppl0G4sUEDtCr4DthTaT47/N3aT6MhfgJg=
modernc.org/memory v1.11.0 h1:o4QC8aMQzmcwCK3t3Ux/ZHmwFPzE6hf2Y5LbkRs+hbI=
modernc.org/memory v1.11.0/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw=
modernc.org/memory v1.12.1 h1:nFMiWrpStgZczNl6XI9GnIk/rWhYIyHGUaR04pGbp9g=
modernc.org/memory v1.12.1/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw=
modernc.org/opt v0.2.0 h1:tGyef5ApycA7FSEOMraay9SaTk5zmbx7Tu+cJs4QKZg=
modernc.org/opt v0.2.0/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns=
modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w=
modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE=
modernc.org/sqlite v1.50.1 h1:l+cQvn0sd0zJJtfygGHuQJ5AjlrwXmWPw4KP3ZMwr9w=
modernc.org/sqlite v1.50.1/go.mod h1:tcNzv5p84E0skkmJn038y+hWJbLQXQqEnQfeh5r2JLM=
modernc.org/sqlite v1.60.1 h1:/blz53O951KWFOso4QQvEs/Fq6cDBKLtMVrYNSeJVKw=
modernc.org/sqlite v1.60.1/go.mod h1:1dIoEagfDE72QytD5scH1lxARtaUgKgHC/NuApA27r0=
modernc.org/strutil v1.2.1 h1:UneZBkQA+DX2Rp35KcM69cSsNES9ly8mQWD71HKlOA0=
modernc.org/strutil v1.2.1/go.mod h1:EHkiggD70koQxjVdSBM3JKM7k6L0FbGE5eymy9i3B9A=
modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
+24
View File
@@ -0,0 +1,24 @@
# Maintainer: Hein (Warky Devs) <hein@warky.dev>
pkgname=relspec-bin
pkgver=1.0.91
pkgrel=1
pkgdesc="RelSpec is a comprehensive database relations management tool that reads, transforms, and writes database table specifications across multiple formats and ORMs."
arch=('x86_64' 'aarch64')
url="https://git.warky.dev/wdevs/relspecgo"
license=('MIT')
provides=('relspec')
conflicts=('relspec')
options=('!debug')
source_x86_64=("$pkgname-$pkgver-x86_64.tar.gz::$url/releases/download/v$pkgver/relspec_${pkgver}_linux_amd64.tar.gz")
source_aarch64=("$pkgname-$pkgver-aarch64.tar.gz::$url/releases/download/v$pkgver/relspec_${pkgver}_linux_arm64.tar.gz")
sha256sums_x86_64=('SKIP')
sha256sums_aarch64=('SKIP')
package() {
# Binary
install -Dm755 "relspec" "$pkgdir/usr/bin/relspec"
install -Dm644 "LICENSE" "$pkgdir/usr/share/licenses/$pkgname/LICENSE"
# Default config dir
install -dm755 "$pkgdir/etc/relspec"
}
@@ -1,17 +1,17 @@
# Maintainer: Hein (Warky Devs) <hein@warky.dev>
pkgname=relspec
pkgver=1.0.85
pkgver=1.0.91
pkgrel=1
pkgdesc="RelSpec is a comprehensive database relations management tool that reads, transforms, and writes database table specifications across multiple formats and ORMs."
arch=('x86_64' 'aarch64')
url="https://git.warky.dev/wdevs/relspecgo"
license=('MIT')
makedepends=('go')
source=("$pkgname-$pkgver.zip::$url/archive/v$pkgver.zip")
source=("$pkgname-$pkgver.zip::$url/releases/download/v$pkgver/relspec-v$pkgver-source.zip")
sha256sums=('SKIP')
build() {
cd "relspecgo"
cd "$pkgname-$pkgver"
export CGO_ENABLED=0
go build \
-trimpath \
@@ -20,12 +20,12 @@ build() {
}
check() {
cd "relspecgo"
cd "$pkgname-$pkgver"
go test ./...
}
package() {
cd "relspecgo"
cd "$pkgname-$pkgver"
# Binary
install -Dm755 "$pkgname" "$pkgdir/usr/bin/$pkgname"
+1 -1
View File
@@ -1,5 +1,5 @@
Name: relspec
Version: 1.0.85
Version: 1.0.91
Release: 1%{?dist}
Summary: RelSpec is a comprehensive database relations management tool that reads, transforms, and writes database table specifications across multiple formats and ORMs.
+93 -18
View File
@@ -9,10 +9,22 @@ import (
"strconv"
"strings"
"unicode/utf8"
"git.warky.dev/wdevs/relspecgo/pkg/safefile"
"git.warky.dev/wdevs/relspecgo/pkg/sqllex"
)
const ScriptSourcePathMetadataKey = "source_path"
const DefaultMaxAssetBytes int64 = 64 << 20 // 64 MiB per manifest/embedded asset
// EmbedOptions are trusted caller configuration, not script/model metadata.
// An empty AssetRoot defaults to the SQL file's directory; external reads are rejected.
type EmbedOptions struct {
AssetRoot string
MaxFileBytes int64
}
var (
embedDirectivePattern = regexp.MustCompile(`(?m)^\s*--\s*@embed:\s*(.+?)\s*$`)
embedAttrPattern = regexp.MustCompile(`([a-zA-Z_][a-zA-Z0-9_]*)=("[^"]*"|'[^']*'|\S+)`)
@@ -27,7 +39,28 @@ var (
// string literal. Base64 mode embeds a quoted base64 literal suitable for
// decode(:var, 'base64').
func ProcessEmbedDirectives(sqlPath, sql string) (string, error) {
directives := embedDirectivePattern.FindAllStringSubmatch(sql, -1)
return ProcessEmbedDirectivesWithOptions(sqlPath, sql, EmbedOptions{})
}
func embedDirectives(sql string) (result string, directives []string) {
// Only standalone line comments are directives, and only in SQL code.
result = sqllex.Walk(sql, nil, func(comment string) string {
if match := embedDirectivePattern.FindStringSubmatch(comment); match != nil {
directives = append(directives, match[1])
return ""
}
return comment
})
return result, directives
}
func HasEmbedDirectives(sql string) bool {
_, directives := embedDirectives(sql)
return len(directives) > 0
}
func ProcessEmbedDirectivesWithOptions(sqlPath, sql string, options EmbedOptions) (string, error) {
result, directives := embedDirectives(sql)
if len(directives) == 0 {
return sql, nil
}
@@ -36,22 +69,63 @@ func ProcessEmbedDirectives(sqlPath, sql string) (string, error) {
return "", fmt.Errorf("sql path is required for embed directives")
}
result := embedDirectivePattern.ReplaceAllString(sql, "")
assetRoot := options.AssetRoot
if assetRoot == "" {
assetRoot = filepath.Dir(sqlPath)
}
absRoot, err := filepath.Abs(assetRoot)
if err != nil {
return "", err
}
absSQL, err := filepath.Abs(sqlPath)
if err != nil {
return "", err
}
scriptRel, err := filepath.Rel(absRoot, absSQL)
if err != nil {
return "", err
}
if _, err := safefile.Relative(filepath.ToSlash(scriptRel)); err != nil {
return "", fmt.Errorf("SQL source path is outside asset root: %w", err)
}
root, err := openAssetRoot(absRoot)
if err != nil {
return "", err
}
defer root.Close()
limit := options.MaxFileBytes
if limit == 0 {
limit = DefaultMaxAssetBytes
}
literals := map[string]string{}
used := map[string]bool{}
for i, directive := range directives {
literal, placeholder, err := embedDirectiveLiteral(sqlPath, directive[1], i+1)
literal, placeholder, err := embedDirectiveLiteral(root, filepath.Dir(scriptRel), limit, sqlPath, directive, i+1)
if err != nil {
return "", err
}
if !embedPlaceholderPattern(placeholder).MatchString(result) {
return "", fmt.Errorf("%s embed directive %d: placeholder %s not found", sqlPath, i+1, placeholder)
if _, exists := literals[placeholder]; exists {
return "", fmt.Errorf("duplicate embed placeholder %s", placeholder)
}
literals[placeholder] = literal
}
result = sqllex.Walk(result, func(name string) string {
if literal, ok := literals[name]; ok {
used[name] = true
return literal
}
return name
}, nil)
for name := range literals {
if !used[name] {
return "", fmt.Errorf("%s: embed placeholder %s not found in executable SQL", sqlPath, name)
}
result = replaceEmbedPlaceholder(result, placeholder, literal)
}
return result, nil
}
func embedDirectiveLiteral(sqlPath, raw string, directiveNumber int) (literal, placeholder string, err error) {
func embedDirectiveLiteral(root *os.Root, scriptDir string, maxBytes int64, sqlPath, raw string, directiveNumber int) (literal, placeholder string, err error) {
attrs, err := parseEmbedAttrs(raw)
if err != nil {
return "", "", fmt.Errorf("%s embed directive %d: %w", sqlPath, directiveNumber, err)
@@ -70,8 +144,12 @@ func embedDirectiveLiteral(sqlPath, raw string, directiveNumber int) (literal, p
return "", "", fmt.Errorf("%s embed directive %d: mode must be text or base64", sqlPath, directiveNumber)
}
resolved := filepath.Join(filepath.Dir(sqlPath), filepath.Clean(pathValue))
data, err := os.ReadFile(resolved)
relative, err := safefile.Relative(pathValue)
if err != nil {
return "", "", fmt.Errorf("%s embed directive %d: %w", sqlPath, directiveNumber, err)
}
resolved := filepath.Join(scriptDir, relative)
data, err := safefile.Read(root, resolved, maxBytes)
if err != nil {
return "", "", fmt.Errorf("%s embed directive %d: reading %s: %w", sqlPath, directiveNumber, resolved, err)
}
@@ -153,13 +231,10 @@ func sqlStringLiteral(value string) (string, error) {
if strings.ContainsRune(value, '\x00') {
return "", fmt.Errorf("embedded text contains NUL byte")
}
return "'" + strings.ReplaceAll(value, "'", "''") + "'", nil
}
func replaceEmbedPlaceholder(sql, placeholder, literal string) string {
return embedPlaceholderPattern(placeholder).ReplaceAllString(sql, "${1}"+literal+"${2}")
}
func embedPlaceholderPattern(placeholder string) *regexp.Regexp {
return regexp.MustCompile(`(^|[^a-zA-Z0-9_:])` + regexp.QuoteMeta(placeholder) + `([^a-zA-Z0-9_]|$)`)
prefix := ""
if strings.Contains(value, "\\") {
prefix = "E"
value = strings.ReplaceAll(value, "\\", "\\\\")
}
return prefix + "'" + strings.ReplaceAll(value, "'", "''") + "'", nil
}
+111
View File
@@ -141,3 +141,114 @@ SELECT :body, :body_extra;
t.Fatalf("placeholder boundary was not respected:\n%s", got)
}
}
func TestEmbedRejectsTraversalAbsolutePathsAndSymlinks(t *testing.T) {
root := t.TempDir()
scriptDir := filepath.Join(root, "scripts")
if err := os.Mkdir(scriptDir, 0o755); err != nil {
t.Fatal(err)
}
secret := filepath.Join(root, "secret.txt")
if err := os.WriteFile(secret, []byte("never print this secret"), 0o600); err != nil {
t.Fatal(err)
}
for _, path := range []string{"../secret.txt", secret, `..\secret.txt`, `C:\secret.txt`, `\\server\secret.txt`} {
sql := "-- @embed: path='" + path + "' var=:body mode=text\nSELECT :body;"
got, err := ProcessEmbedDirectives(filepath.Join(scriptDir, "seed.sql"), sql)
if err == nil || got != "" {
t.Fatalf("unsafe embed path accepted %q: %q %v", path, got, err)
}
if strings.Contains(err.Error(), "never print this secret") {
t.Fatal("diagnostic leaked asset bytes")
}
}
if err := os.Symlink(root, filepath.Join(scriptDir, "link")); err != nil {
t.Skip(err)
}
if _, err := ProcessEmbedDirectives(filepath.Join(scriptDir, "seed.sql"), "-- @embed: path=link/secret.txt var=:body mode=text\nSELECT :body;"); err == nil {
t.Fatal("symlink parent must be rejected")
}
if err := os.Symlink(secret, filepath.Join(scriptDir, "leaf.txt")); err != nil {
t.Fatal(err)
}
if _, err := ProcessEmbedDirectives(filepath.Join(scriptDir, "seed.sql"), "-- @embed: path=leaf.txt var=:body mode=text\nSELECT :body;"); err == nil {
t.Fatal("symlink asset must be rejected")
}
}
func TestEmbedExplicitRootAndSizeLimit(t *testing.T) {
root := t.TempDir()
dir := filepath.Join(root, "scripts")
if err := os.Mkdir(dir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "body.txt"), []byte("12345"), 0o600); err != nil {
t.Fatal(err)
}
sql := "-- @embed: path=body.txt var=:body mode=text\nSELECT :body;"
path := filepath.Join(dir, "seed.sql")
if _, err := ProcessEmbedDirectivesWithOptions(path, sql, EmbedOptions{AssetRoot: root, MaxFileBytes: 4}); err == nil || !strings.Contains(err.Error(), "byte limit") {
t.Fatalf("expected size limit error: %v", err)
}
if got, err := ProcessEmbedDirectivesWithOptions(path, sql, EmbedOptions{AssetRoot: root, MaxFileBytes: 5}); err != nil || !strings.Contains(got, "SELECT '12345';") {
t.Fatalf("valid bounded embed: %q %v", got, err)
}
if _, err := ProcessEmbedDirectivesWithOptions(path, sql, EmbedOptions{AssetRoot: t.TempDir()}); err == nil {
t.Fatal("source path outside explicit root must fail")
}
if _, err := ProcessEmbedDirectivesWithOptions(path, sql, EmbedOptions{MaxFileBytes: -1}); err == nil {
t.Fatal("negative limits must fail")
}
}
func TestManifestReadRechecksContainmentAndSize(t *testing.T) {
root, external := t.TempDir(), t.TempDir()
file := filepath.Join(root, "body.txt")
if err := os.WriteFile(file, []byte("valid"), 0o600); err != nil {
t.Fatal(err)
}
item := Item{AssetRoot: root, MaxFileBytes: 4, Entry: ManifestEntry{File: file}}
if _, err := readItemAsset(item); err == nil {
t.Fatal("manifest size bound must apply at execution")
}
item.MaxFileBytes = 5
if got, err := readItemAsset(item); err != nil || string(got) != "valid" {
t.Fatalf("valid manifest asset: %q %v", got, err)
}
secret := filepath.Join(external, "secret.txt")
if err := os.WriteFile(secret, []byte("secret"), 0o600); err != nil {
t.Fatal(err)
}
if err := os.Remove(file); err != nil {
t.Fatal(err)
}
if err := os.Symlink(secret, file); err != nil {
t.Skip(err)
}
if _, err := readItemAsset(item); err == nil {
t.Fatal("replacement symlink after scanning must fail")
}
item.Entry.File = secret
if _, err := readItemAsset(item); err == nil {
t.Fatal("forged asset path must fail")
}
}
func TestEmbedDefaultLimitRejectsOversizedFile(t *testing.T) {
dir := t.TempDir()
file, err := os.Create(filepath.Join(dir, "large.bin"))
if err != nil {
t.Fatal(err)
}
if err := file.Truncate(DefaultMaxAssetBytes + 1); err != nil {
file.Close()
t.Fatal(err)
}
if err := file.Close(); err != nil {
t.Fatal(err)
}
_, err = ProcessEmbedDirectives(filepath.Join(dir, "seed.sql"), "-- @embed: path=large.bin var=:payload mode=base64\nSELECT :payload;")
if err == nil || !strings.Contains(err.Error(), "byte limit") {
t.Fatalf("expected default size bound: %v", err)
}
}
+5 -18
View File
@@ -3,20 +3,13 @@ package assetloader
import (
"context"
"fmt"
"os"
"path/filepath"
"regexp"
"strings"
"git.warky.dev/wdevs/relspecgo/pkg/sqllex"
"github.com/jackc/pgx/v5"
)
// namedPlaceholder matches :identifier patterns (but not ::cast syntax).
var namedPlaceholder = regexp.MustCompile(`:([a-zA-Z_][a-zA-Z0-9_]*)`)
// pgCastMarker temporarily replaces :: to protect PostgreSQL cast syntax.
const pgCastMarker = "\x00PGCAST\x00"
// BuildQuery converts a SQL call that uses :name named placeholders into a
// pgx-compatible positional-parameter query ($1, $2, …) and returns the
// corresponding argument slice.
@@ -30,13 +23,10 @@ const pgCastMarker = "\x00PGCAST\x00"
// placeholder (not built-in and not in staticParams) returns an error.
// PostgreSQL cast syntax (::type) is left untouched.
func BuildQuery(call string, fileBytes []byte, filename string, staticParams map[string]string) (query string, args []any, err error) {
// Protect :: casts before running the placeholder regex.
protected := strings.ReplaceAll(call, "::", pgCastMarker)
paramIndex := map[string]int{} // name → 1-based position
var firstErr error
query = namedPlaceholder.ReplaceAllStringFunc(protected, func(match string) string {
query = sqllex.Walk(call, func(match string) string {
if firstErr != nil {
return match
}
@@ -70,15 +60,12 @@ func BuildQuery(call string, fileBytes []byte, filename string, staticParams map
paramIndex[name] = idx
args = append(args, val)
return fmt.Sprintf("$%d", idx)
})
}, nil)
if firstErr != nil {
return "", nil, firstErr
}
// Restore :: casts.
query = strings.ReplaceAll(query, pgCastMarker, "::")
return query, args, nil
}
@@ -86,7 +73,7 @@ func BuildQuery(call string, fileBytes []byte, filename string, staticParams map
// absolute path set by ScanDir) and executes the configured SQL call via conn.
// The file's raw bytes are bound as a []byte parameter — no encoding or escaping.
func ExecuteItem(ctx context.Context, conn *pgx.Conn, item Item) error {
data, err := os.ReadFile(item.Entry.File)
data, err := readItemAsset(item)
if err != nil {
return fmt.Errorf("reading asset file %s: %w", item.Entry.File, err)
}
+36
View File
@@ -0,0 +1,36 @@
package assetloader
import (
"fmt"
"os"
"path/filepath"
"git.warky.dev/wdevs/relspecgo/pkg/safefile"
)
func openAssetRoot(path string) (*os.Root, error) {
if err := safefile.CheckRoot(path); err != nil {
return nil, err
}
return os.OpenRoot(path)
}
func readItemAsset(item Item) ([]byte, error) {
if item.AssetRoot == "" {
return nil, fmt.Errorf("asset root is required for manifest execution")
}
root, err := openAssetRoot(item.AssetRoot)
if err != nil {
return nil, err
}
defer root.Close()
relative, err := filepath.Rel(item.AssetRoot, item.Entry.File)
if err != nil {
return nil, err
}
limit := item.MaxFileBytes
if limit == 0 {
limit = DefaultMaxAssetBytes
}
return safefile.Read(root, relative, limit)
}
+38 -30
View File
@@ -18,9 +18,10 @@ import (
"regexp"
"sort"
"strconv"
"strings"
"gopkg.in/yaml.v3"
"git.warky.dev/wdevs/relspecgo/pkg/safefile"
)
// ManifestEntry describes a single file to load from an assets.yaml manifest.
@@ -37,6 +38,9 @@ type ManifestEntry struct {
// Item combines a manifest entry with its ordering metadata and the resolved
// directory where the manifest and asset file reside.
type Item struct {
// AssetRoot is the trusted scan root, retained for validation at execution time.
AssetRoot string
MaxFileBytes int64
// Priority and Sequence come from the parent directory's naming pattern.
Priority int
Sequence uint
@@ -57,7 +61,12 @@ var dirPattern = regexp.MustCompile(`^(\d+)[_-](\d+)[_-](.+)$`)
// absent or contains invalid YAML.
func LoadManifest(dir string) ([]ManifestEntry, error) {
manifestPath := filepath.Join(dir, "assets.yaml")
data, err := os.ReadFile(manifestPath)
root, err := openAssetRoot(dir)
if err != nil {
return nil, err
}
defer root.Close()
data, err := safefile.Read(root, "assets.yaml", DefaultMaxAssetBytes)
if err != nil {
return nil, fmt.Errorf("reading %s: %w", manifestPath, err)
}
@@ -92,7 +101,7 @@ func ScanDir(baseDir string) ([]Item, error) {
if d.IsDir() {
return nil
}
if d.Name() != "assets.yaml" {
if d.Type()&os.ModeSymlink != 0 || d.Name() != "assets.yaml" {
return nil
}
@@ -119,10 +128,12 @@ func ScanDir(baseDir string) ([]Item, error) {
}
items = append(items, Item{
Priority: priority,
Sequence: sequence,
DirName: dirName,
Dir: manifestDir,
AssetRoot: absBase,
MaxFileBytes: DefaultMaxAssetBytes,
Priority: priority,
Sequence: sequence,
DirName: dirName,
Dir: manifestDir,
Entry: ManifestEntry{
File: resolved, // absolute path, safe to read
Call: entry.Call,
@@ -170,31 +181,28 @@ func parseDirName(dir string) (priority int, sequence uint, name string) {
// Returns the absolute path, a skip flag (true when the entry should be silently
// dropped), and any hard error.
func resolveAssetPath(absBase, manifestDir, file string) (absPath string, skip bool, err error) {
// Clean and join before any symlink resolution so we can detect traversal.
joined := filepath.Join(manifestDir, filepath.Clean(file))
// Ensure the cleaned path is still inside absBase.
rel, err := filepath.Rel(absBase, joined)
if err != nil || strings.HasPrefix(rel, "..") {
// Path escapes the base directory; skip silently.
if _, err := safefile.Relative(file); err != nil {
return "", true, nil
}
// Walk each component to detect symlinks.
parts := strings.Split(rel, string(filepath.Separator))
current := absBase
for _, part := range parts {
current = filepath.Join(current, part)
info, statErr := os.Lstat(current)
if statErr != nil {
// File doesn't exist; skip.
return "", true, nil
}
if info.Mode()&os.ModeSymlink != 0 {
// Symlink in path; skip silently.
return "", true, nil
}
joined := filepath.Join(manifestDir, file)
rel, err := filepath.Rel(absBase, joined)
if err != nil {
return "", true, nil
}
root, err := openAssetRoot(absBase)
if err != nil {
return "", false, err
}
defer root.Close()
if err := safefile.Check(root, rel); err != nil {
return "", true, nil
}
info, err := root.Stat(rel)
if err != nil || !info.Mode().IsRegular() {
return "", true, nil
}
if info.Size() > DefaultMaxAssetBytes {
return "", false, fmt.Errorf("asset %q exceeds %d byte limit", rel, DefaultMaxAssetBytes)
}
return joined, false, nil
}
+60
View File
@@ -0,0 +1,60 @@
package assetloader
import (
"os"
"path/filepath"
"strings"
"testing"
)
func TestBuildQuerySQLContexts(t *testing.T) {
call := `SELECT ':unknown', ":unknown", $$:unknown$$, E'it\':unknown', :bytes::bytea, :filename, :bytes /* :unknown /* :unknown */ */ -- :unknown
`
got, args, err := BuildQuery(call, []byte("$1"), "File.txt", nil)
if err != nil {
t.Fatal(err)
}
if len(args) != 2 || !strings.Contains(got, "$1::bytea, $2, $1") || !strings.Contains(got, "$$:unknown$$") {
t.Fatalf("%s %#v", got, args)
}
}
func TestEmbedLiteralOnePassAndSQLContexts(t *testing.T) {
dir := t.TempDir()
data := "price $1 ${2} $HOME $$ :b 'quote' \\path"
for file, text := range map[string]string{"a.txt": data, "b.txt": "Second"} {
if err := os.WriteFile(filepath.Join(dir, file), []byte(text), 0o600); err != nil {
t.Fatal(err)
}
}
sql := `-- @embed: path=a.txt var=:a mode=text
-- @embed: path=b.txt var=:b mode=text
SELECT :a,:a,:b, ':a', ":a", $$:a$$, 1::integer; -- :a
/* :a */`
got, err := ProcessEmbedDirectives(filepath.Join(dir, "script.sql"), sql)
if err != nil {
t.Fatal(err)
}
literal, err := sqlStringLiteral(data)
if err != nil {
t.Fatal(err)
}
if strings.Count(got, literal) != 2 || !strings.Contains(got, "'Second'") || !strings.Contains(got, "':a'") || !strings.Contains(got, "$$:a$$") {
t.Fatalf("%s", got)
}
for _, sql := range []string{"SELECT $body$\n-- @embed: path=missing var=:x mode=text\n$body$", "SELECT '\n-- @embed: path=missing var=:x mode=text\n'", "/*\n-- @embed: path=missing var=:x mode=text\n*/", "SELECT 1; -- @embed: path=missing var=:x mode=text"} {
if HasEmbedDirectives(sql) {
t.Fatalf("false directive: %s", sql)
}
got, err := ProcessEmbedDirectives("", sql)
if err != nil || got != sql {
t.Fatalf("%q: %v", got, err)
}
}
for _, sql := range []string{"-- @embed: path=a.txt var=:a mode=text\nSELECT ':a'", "-- @embed: path=a.txt var=:a mode=text\n-- @embed: path=b.txt var=:a mode=text\nSELECT :a"} {
if _, err := ProcessEmbedDirectives(filepath.Join(dir, "script.sql"), sql); err == nil {
t.Fatal("expected invalid directive use")
}
}
}
+3
View File
@@ -504,6 +504,9 @@ func compareConstraintDetails(source, target *models.Constraint) map[string]any
if normalizeConstraintAction(source.OnUpdate) != normalizeConstraintAction(target.OnUpdate) {
changes["on_update"] = map[string]string{"source": source.OnUpdate, "target": target.OnUpdate}
}
if !models.DeferralMatches(source, target) {
changes["deferral"] = map[string]string{"source": source.DeferralClause(), "target": target.DeferralClause()}
}
return changes
}
+14 -3
View File
@@ -277,14 +277,17 @@ func parseHumanSize(s string) (int64, error) {
// Options carries the subset of command flags a job file may set.
type Options struct {
FlattenSchema bool `yaml:"flatten_schema"`
Schema string `yaml:"schema"`
Package string `yaml:"package"`
StrictConversion bool `yaml:"strict_conversion"`
LowercaseIdentifiers bool `yaml:"lowercase_identifiers"`
FlattenSchema bool `yaml:"flatten_schema"`
Schema string `yaml:"schema"`
Package string `yaml:"package"`
// Types and ArrayNullable mirror the existing convert/split writer flags.
// They are passed through only to writers that already support them.
Types string `yaml:"types"`
ArrayNullable string `yaml:"array_nullable"`
ContinueOnError bool `yaml:"continue_on_error"`
AtomicExecution bool `yaml:"atomic_execution"`
// FullDDL makes direct pgsql output execute the full idempotent DDL instead of
// diffing against the live database first.
FullDDL bool `yaml:"full_ddl"`
@@ -479,6 +482,14 @@ func (s *Set) Validate() error {
func (j *Job) validate() []string {
var e []string
if j.Options.AtomicExecution {
if j.Options.ContinueOnError {
e = append(e, "atomic_execution cannot be combined with continue_on_error")
}
if (j.Command != CommandConvert && j.Command != CommandMerge) || j.Output == nil || j.Output.ConnEnv == "" || j.Output.Format != "pgsql" {
e = append(e, "atomic_execution requires convert/merge with pgsql database output")
}
}
switch j.Command {
case CommandConvert, CommandMerge, CommandScriptsList, CommandScriptsExec,
+19 -19
View File
@@ -61,6 +61,7 @@ var MariaDBToCanonicalTypes = map[string]string{
// CanonicalToMariaDBTypes maps canonical types to MariaDB/MySQL types.
var CanonicalToMariaDBTypes = map[string]string{
"bool": "TINYINT(1)",
"boolean": "TINYINT(1)",
"int8": "TINYINT",
"int16": "SMALLINT",
"int": "INT",
@@ -114,20 +115,26 @@ func NormalizeMariaDBType(baseType string) string {
// ConvertMariaDBToCanonical converts a MariaDB/MySQL type name to the canonical type.
// Strips dimension parameters and normalizes aliases. Defaults to "string".
func ConvertMariaDBToCanonical(mariadbType string) string {
base := strings.ToLower(strings.TrimSpace(mariadbType))
if idx := strings.Index(base, "("); idx >= 0 {
base = strings.TrimSpace(base[:idx])
lower := strings.ToLower(strings.TrimSpace(mariadbType))
base := lower
if idx := strings.IndexAny(base, "( "); idx >= 0 {
base = base[:idx]
}
if canonical, ok := MariaDBToCanonicalTypes[base]; ok {
return canonical
}
// Prefix match for composite types (e.g., "unsigned bigint")
for key, canonical := range MariaDBToCanonicalTypes {
if strings.HasPrefix(base, key) {
return canonical
canonical, ok := MariaDBToCanonicalTypes[base]
if ok {
if strings.Contains(lower, "unsigned") || strings.Contains(lower, "zerofill") {
switch canonical {
case "int8":
return "uint8"
case "int16":
return "uint16"
case "int":
return "uint"
case "int64":
return "uint64"
}
}
return canonical
}
return "string"
@@ -145,12 +152,5 @@ func ConvertCanonicalToMariaDB(canonicalType string) string {
return mariadbType
}
// Prefix fallback
for canonical, mariadb := range CanonicalToMariaDBTypes {
if strings.HasPrefix(lower, canonical) {
return mariadb
}
}
return "VARCHAR(255)"
}
+13
View File
@@ -0,0 +1,13 @@
package mariadb
import "testing"
func TestIntegerSignedness(t *testing.T) {
for _, tc := range []struct{ input, want string }{
{"tinyint(3) unsigned", "uint8"}, {"smallint unsigned", "uint16"}, {"mediumint unsigned", "uint"}, {"int unsigned", "uint"}, {"bigint unsigned", "uint64"}, {"bigint zerofill", "uint64"}, {"bigint", "int64"}, {"varchar(32)", "string"},
} {
if got := ConvertMariaDBToCanonical(tc.input); got != tc.want {
t.Errorf("%s: got %s want %s", tc.input, got, tc.want)
}
}
}
+10 -2
View File
@@ -605,8 +605,8 @@ func cloneConstraint(constraint *models.Constraint) *models.Constraint {
OnDelete: constraint.OnDelete,
Expression: constraint.Expression,
Name: constraint.Name,
Deferrable: constraint.Deferrable,
InitiallyDeferred: constraint.InitiallyDeferred,
Deferrable: copyBoolPtr(constraint.Deferrable),
InitiallyDeferred: copyBoolPtr(constraint.InitiallyDeferred),
Sequence: constraint.Sequence,
}
copy(newConstraint.Columns, constraint.Columns)
@@ -823,3 +823,11 @@ func min(a, b int) int {
}
return b
}
func copyBoolPtr(b *bool) *bool {
if b == nil {
return nil
}
v := *b
return &v
}
+48
View File
@@ -0,0 +1,48 @@
package models
// DeferralClause returns the SQL deferral clause for a foreign key constraint
// ("DEFERRABLE", "DEFERRABLE INITIALLY DEFERRED" or "NOT DEFERRABLE"), or ""
// when the model does not specify one. InitiallyDeferred without Deferrable
// implies DEFERRABLE; InitiallyDeferred=false alone specifies nothing.
func (d *Constraint) DeferralClause() string {
deferrable, initially, set := d.effectiveDeferral()
switch {
case !set:
return ""
case !deferrable:
return "NOT DEFERRABLE"
case initially != nil && *initially:
return "DEFERRABLE INITIALLY DEFERRED"
default:
return "DEFERRABLE"
}
}
// effectiveDeferral resolves the optional flags. set is false when the model says nothing
// about deferral; initially is nil when the initial mode is unspecified.
func (d *Constraint) effectiveDeferral() (deferrable bool, initially *bool, set bool) {
switch {
case d.Deferrable != nil:
return *d.Deferrable, d.InitiallyDeferred, true
case d.InitiallyDeferred != nil && *d.InitiallyDeferred:
return true, d.InitiallyDeferred, true
}
return false, nil, false
}
// DeferralMatches reports whether two constraints agree on deferral. An unspecified side
// matches anything, and the initial mode is only compared when both sides specify it.
func DeferralMatches(a, b *Constraint) bool {
da, ia, aSet := a.effectiveDeferral()
db, ib, bSet := b.effectiveDeferral()
if !aSet || !bSet {
return true
}
if da != db {
return false
}
if !da || ia == nil || ib == nil {
return true
}
return *ia == *ib
}
+59
View File
@@ -0,0 +1,59 @@
package models
import "testing"
func boolPtr(b bool) *bool { return &b }
func TestConstraint_DeferralClause(t *testing.T) {
tests := []struct {
name string
c Constraint
want string
}{
{"unspecified", Constraint{}, ""},
{"deferrable", Constraint{Deferrable: boolPtr(true)}, "DEFERRABLE"},
{"deferrable initially deferred", Constraint{Deferrable: boolPtr(true), InitiallyDeferred: boolPtr(true)}, "DEFERRABLE INITIALLY DEFERRED"},
{"deferrable initially immediate", Constraint{Deferrable: boolPtr(true), InitiallyDeferred: boolPtr(false)}, "DEFERRABLE"},
{"initially deferred implies deferrable", Constraint{InitiallyDeferred: boolPtr(true)}, "DEFERRABLE INITIALLY DEFERRED"},
{"initially immediate alone is unspecified", Constraint{InitiallyDeferred: boolPtr(false)}, ""},
{"explicitly not deferrable", Constraint{Deferrable: boolPtr(false)}, "NOT DEFERRABLE"},
{"not deferrable ignores initially deferred", Constraint{Deferrable: boolPtr(false), InitiallyDeferred: boolPtr(true)}, "NOT DEFERRABLE"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := tt.c.DeferralClause(); got != tt.want {
t.Errorf("DeferralClause() = %q, want %q", got, tt.want)
}
})
}
}
func TestDeferralMatches(t *testing.T) {
live := func(def, init bool) *Constraint {
return &Constraint{Deferrable: boolPtr(def), InitiallyDeferred: boolPtr(init)}
}
tests := []struct {
name string
model, other *Constraint
want bool
}{
{"model unspecified vs deferrable live", &Constraint{}, live(true, false), true},
{"model unspecified vs not deferrable live", &Constraint{}, live(false, false), true},
{"deferrable vs deferrable live", &Constraint{Deferrable: boolPtr(true)}, live(true, true), true},
{"deferrable vs not deferrable live", &Constraint{Deferrable: boolPtr(true)}, live(false, false), false},
{"not deferrable vs deferrable live", &Constraint{Deferrable: boolPtr(false)}, live(true, false), false},
{"initially deferred vs initially immediate live", &Constraint{InitiallyDeferred: boolPtr(true)}, live(true, false), false},
{"initially deferred vs initially deferred live", &Constraint{InitiallyDeferred: boolPtr(true)}, live(true, true), true},
{"explicit initially immediate vs initially deferred live", &Constraint{Deferrable: boolPtr(true), InitiallyDeferred: boolPtr(false)}, live(true, true), false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := DeferralMatches(tt.model, tt.other); got != tt.want {
t.Errorf("DeferralMatches() = %v, want %v", got, tt.want)
}
if got := DeferralMatches(tt.other, tt.model); got != tt.want {
t.Errorf("DeferralMatches() reversed = %v, want %v", got, tt.want)
}
})
}
}
+13 -2
View File
@@ -256,9 +256,19 @@ func (d *Column) SQLName() string {
return strings.ToLower(d.Name)
}
// IndexPart is an ordered MySQL index key part. Name and Expression are mutually
// exclusive; expressions are executable SQL and are not identifier-normalized.
type IndexPart struct {
Name string `json:"name,omitempty" yaml:"name,omitempty" xml:"name,omitempty"`
Expression string `json:"expression,omitempty" yaml:"expression,omitempty" xml:"expression,omitempty"`
Length int `json:"length,omitempty" yaml:"length,omitempty" xml:"length,omitempty"`
Descending bool `json:"descending,omitempty" yaml:"descending,omitempty" xml:"descending,omitempty"`
}
// Index represents a database index for optimizing query performance.
// Indexes can be unique, partial, or include additional columns.
type Index struct {
KeyParts []IndexPart `json:"key_parts,omitempty" yaml:"key_parts,omitempty" xml:"key_parts,omitempty"`
Name string `json:"name" yaml:"name" xml:"name"`
Description string `json:"description,omitempty" yaml:"description,omitempty" xml:"description,omitempty"`
Table string `json:"table,omitempty" yaml:"table,omitempty" xml:"table,omitempty"`
@@ -318,6 +328,7 @@ func (d *Relationship) SQLName() string {
// Constraint represents a database constraint that enforces data integrity rules.
// Constraints can be primary keys, foreign keys, unique constraints, check constraints, or not-null constraints.
type Constraint struct {
Metadata map[string]any `json:"metadata,omitempty" yaml:"metadata,omitempty" xml:"-"`
Name string `json:"name" yaml:"name" xml:"name"`
Type ConstraintType `json:"type" yaml:"type" xml:"type"`
Columns []string `json:"columns" yaml:"columns" xml:"columns"`
@@ -329,8 +340,8 @@ type Constraint struct {
ReferencedColumns []string `json:"referenced_columns" yaml:"referenced_columns" xml:"referenced_columns"`
OnDelete string `json:"on_delete" yaml:"on_delete" xml:"on_delete"` // CASCADE, SET NULL, RESTRICT, etc.
OnUpdate string `json:"on_update" yaml:"on_update" xml:"on_update"`
Deferrable bool `json:"deferrable,omitempty" yaml:"deferrable,omitempty" xml:"deferrable,omitempty"`
InitiallyDeferred bool `json:"initially_deferred,omitempty" yaml:"initially_deferred,omitempty" xml:"initially_deferred,omitempty"`
Deferrable *bool `json:"deferrable,omitempty" yaml:"deferrable,omitempty" xml:"deferrable,omitempty"` // nil: unspecified, left as is
InitiallyDeferred *bool `json:"initially_deferred,omitempty" yaml:"initially_deferred,omitempty" xml:"initially_deferred,omitempty"` // nil: unspecified, left as is
Sequence uint `json:"sequence,omitempty" yaml:"sequence,omitempty" xml:"sequence,omitempty"`
GUID string `json:"guid" yaml:"guid" xml:"guid"`
}
+2 -1
View File
@@ -226,7 +226,8 @@ func TestInitConstructors(t *testing.T) {
}
// GUIDs are unique per call.
if InitTable("t", "s").GUID == InitTable("t", "s").GUID {
t1, t2 := InitTable("t", "s"), InitTable("t", "s")
if t1.GUID == t2.GUID {
t.Error("GUIDs must be unique")
}
}
+25 -1
View File
@@ -86,16 +86,40 @@ Ref: posts.user_id > users.id [delete: cascade]
- Table definitions with columns
- Primary keys (`pk`)
- Not null constraints (`not null`)
- Unique constraints (`unique`)
- Unique constraints (`unique`) → constraint `ukey_<table>_<col>`
- Default values (`default`)
- Inline references (`ref`)
- Standalone `Ref` blocks
- FK deferral: `deferrable`, `initially deferred` (see below)
- Commented cross-file refs (`// Ref:` — see below)
- Indexes and composite indexes
- Table notes and column notes
- Enums
- Dialect directives (`@postgres:` / `@sqlite:` — see below)
## Naming
| Source | Generated name |
| ------ | -------------- |
| Column `[unique]` | constraint `ukey_<table>_<col>` |
| Unnamed unique index | index `ukey_<table>_<cols>` (same name as the constraint on the same columns) |
| Unnamed index | `idx_<table>_<cols>` |
Explicit `name:` is never changed. Leading `_` in the table name is stripped.
## FK deferral
```
col int [ref: > t.id, deferrable, initially deferred]
Ref: a.x > b.id [delete: cascade, deferrable]
```
| Setting | SQL |
| ------- | --- |
| `deferrable` | `DEFERRABLE` |
| `initially deferred` | `DEFERRABLE INITIALLY DEFERRED` |
| neither | unspecified (existing FK left as is) |
## Commented cross-file refs
`// Ref:` / `// ref:` lines (ignored by dbdiagram) become FKs + relationships once both ends are loaded.
+26 -3
View File
@@ -848,6 +848,13 @@ func (r *Reader) parseColumn(line, tableName, schemaName string) (*models.Column
}
}
// "deferrable" / "initially deferred" column settings apply to the inline foreign key
if constraint != nil && constraint.Type == models.ForeignKeyConstraint && attrs != "" {
for _, attr := range splitColumnAttrs(attrs) {
applyDeferralSetting(constraint, attr)
}
}
// Parse inline comment
if inlineComment != "" {
column.Comment = inlineComment
@@ -1081,11 +1088,13 @@ func (r *Reader) parseIndex(line, tableName, schemaName string) *models.Index {
// Generate name if not provided
if index.Name == "" {
prefix := "idx"
// Unique indexes share the ukey_ naming of unique constraints, so an index and a
// constraint on the same columns get the same name and can be merged on write.
prefix, name := "idx", tableName
if index.Unique {
prefix = "uidx"
prefix, name = "ukey", strings.TrimLeft(tableName, "_")
}
index.Name = fmt.Sprintf("%s_%s_%s", prefix, tableName, strings.Join(columns, "_"))
index.Name = fmt.Sprintf("%s_%s_%s", prefix, name, strings.Join(columns, "_"))
}
return index
@@ -1182,6 +1191,8 @@ func (r *Reader) parseRef(refStr string) *models.Constraint {
constraint.OnDelete = strings.TrimSpace(strings.TrimPrefix(action, "ondelete:"))
} else if strings.HasPrefix(action, "onupdate:") {
constraint.OnUpdate = strings.TrimSpace(strings.TrimPrefix(action, "onupdate:"))
} else {
applyDeferralSetting(constraint, action)
}
}
}
@@ -1190,6 +1201,18 @@ func (r *Reader) parseRef(refStr string) *models.Constraint {
return constraint
}
// applyDeferralSetting applies a "deferrable" or "initially deferred" Ref setting to a
// foreign key constraint. Other settings are ignored.
func applyDeferralSetting(constraint *models.Constraint, setting string) {
yes := true
switch strings.Join(strings.Fields(strings.ToLower(setting)), " ") {
case "deferrable":
constraint.Deferrable = &yes
case "initially deferred", "initially_deferred":
constraint.InitiallyDeferred = &yes
}
}
// parseTableRef parses a table reference like "schema.table.(column1, column2)" or "schema"."table"."column"
func (r *Reader) parseTableRef(ref string) (schema, table string, columns []string) {
// Extract columns if present in parentheses format
+70 -3
View File
@@ -3,6 +3,7 @@ package dbml
import (
"os"
"path/filepath"
"strings"
"testing"
"git.warky.dev/wdevs/relspecgo/pkg/models"
@@ -842,9 +843,9 @@ func TestConstraintNaming(t *testing.T) {
t.Logf("Available constraints: %v", getKeys(postsTable.Constraints))
}
// Test unique index naming: uidx_table_columns
if _, exists := postsTable.Indexes["uidx_posts_slug"]; !exists {
t.Error("Expected unique index 'uidx_posts_slug' not found")
// Test unique index naming: ukey_table_columns, matching the unique constraint name
if _, exists := postsTable.Indexes["ukey_posts_slug"]; !exists {
t.Error("Expected unique index 'ukey_posts_slug' not found")
t.Logf("Available indexes: %v", getKeys(postsTable.Indexes))
}
@@ -1121,3 +1122,69 @@ func TestReader_DuplicateIndexNameInTableKept(t *testing.T) {
}
}
}
func TestReader_ForeignKeyDeferral(t *testing.T) {
dbml := `Table users {
id integer [pk]
}
Table posts {
id integer [pk]
user_id integer [ref: > users.id, deferrable, initially deferred]
editor_id integer
plain_id integer [ref: > users.id]
}
Table comments {
id integer [pk]
post_id integer
author_id integer
}
Ref: comments.post_id > posts.id [delete: cascade, deferrable]
Ref: comments.author_id > users.id [deferrable, initially deferred]
`
path := filepath.Join(t.TempDir(), "deferral.dbml")
if err := os.WriteFile(path, []byte(dbml), 0o600); err != nil {
t.Fatal(err)
}
db, err := NewReader(&readers.ReaderOptions{FilePath: path}).ReadDatabase()
if err != nil {
t.Fatalf("ReadDatabase() error = %v", err)
}
find := func(table, column string) *models.Constraint {
for _, s := range db.Schemas {
for _, tb := range s.Tables {
if tb.Name != table {
continue
}
for _, c := range tb.Constraints {
if c.Type == models.ForeignKeyConstraint && len(c.Columns) == 1 && c.Columns[0] == column {
return c
}
}
}
}
t.Fatalf("FK %s.%s not found", table, column)
return nil
}
tests := []struct {
table, column string
want string
}{
{"posts", "user_id", "DEFERRABLE INITIALLY DEFERRED"},
{"posts", "plain_id", ""},
{"comments", "post_id", "DEFERRABLE"},
{"comments", "author_id", "DEFERRABLE INITIALLY DEFERRED"},
}
for _, tt := range tests {
if got := find(tt.table, tt.column).DeferralClause(); got != tt.want {
t.Errorf("%s.%s deferral = %q, want %q", tt.table, tt.column, got, tt.want)
}
}
if got := find("comments", "post_id").OnDelete; !strings.EqualFold(got, "cascade") {
t.Errorf("delete action lost, got %q", got)
}
}
+194
View File
@@ -0,0 +1,194 @@
package mysql
import (
"regexp"
"strings"
)
// columnDefaults extracts native DEFAULT expressions from SHOW CREATE TABLE.
// MySQL can truncate binary defaults in information_schema (for example X'00FF'
// appears as just "0x"). SHOW CREATE is authoritative for their SQL spelling.
// Separators and whitespace inside quoted strings, identifiers and nested
// expressions are never treated as definition/default boundaries.
func columnDefaults(create string) map[string]string {
out := map[string]string{}
depth, start := 0, 0
var quoted byte
definitions := []string{}
for i := 0; i < len(create); i++ {
ch := create[i]
if quoted != 0 {
if ch == '\\' && quoted != '`' && i+1 < len(create) {
i++
continue
}
if ch == quoted {
if i+1 < len(create) && create[i+1] == quoted {
i++
} else {
quoted = 0
}
}
continue
}
if ch == '\'' || ch == '"' || ch == '`' {
quoted = ch
continue
}
switch ch {
case '(':
depth++
if depth == 1 {
start = i + 1
}
case ',':
if depth == 1 {
definitions = append(definitions, create[start:i])
start = i + 1
}
case ')':
depth--
if depth == 0 {
definitions = append(definitions, create[start:i])
}
}
if ch == ')' && depth == 0 {
break
}
}
for _, def := range definitions {
def = strings.TrimSpace(def)
if len(def) == 0 || def[0] != '`' {
continue
}
var name strings.Builder
end := 1
for end < len(def) {
if def[end] == '`' {
if end+1 < len(def) && def[end+1] == '`' {
name.WriteByte('`')
end += 2
continue
}
end++
break
}
name.WriteByte(def[end])
end++
}
// Consume full tokens so DEFAULT inside a generated expression or literal
// cannot masquerade as a column default.
for end < len(def) {
for end < len(def) && isSpace(def[end]) {
end++
}
begin := end
end = tokenEnd(def, end)
if end == begin {
break
}
if strings.EqualFold(def[begin:end], "DEFAULT") {
for end < len(def) && isSpace(def[end]) {
end++
}
finish := tokenEnd(def, end)
out[name.String()] = def[end:finish]
break
}
}
}
return out
}
func isSpace(b byte) bool { return b == ' ' || b == '\n' || b == '\r' || b == '\t' }
func tokenEnd(s string, start int) int {
depth := 0
var quoted byte
for i := start; i < len(s); i++ {
ch := s[i]
if quoted != 0 {
if ch == '\\' && quoted != '`' && i+1 < len(s) {
i++
continue
}
if ch == quoted {
if i+1 < len(s) && s[i+1] == quoted {
i++
} else {
quoted = 0
}
}
continue
}
if ch == '\'' || ch == '"' || ch == '`' {
quoted = ch
continue
}
if ch == '(' {
depth++
}
if ch == ')' {
depth--
}
if depth == 0 && isSpace(ch) {
return i
}
}
return len(s)
}
// SHOW CREATE has native options not exposed by STATISTICS/COLUMNS. Ignore
// quoted identifiers and values when detecting options outside our contract.
func unquotedSQL(create string) string {
var code strings.Builder
var quoted byte
for i := 0; i < len(create); i++ {
ch := create[i]
if quoted != 0 {
if ch == '\\' && quoted != '`' && i+1 < len(create) {
i++
continue
}
if ch == quoted {
if i+1 < len(create) && create[i+1] == quoted {
i++
} else {
quoted = 0
code.WriteByte(' ')
}
}
continue
}
if ch == '\'' || ch == '"' || ch == '`' {
quoted = ch
code.WriteByte(' ')
continue
}
code.WriteByte(ch)
}
return code.String()
}
var autoIncrementOption = regexp.MustCompile(`(?i)\bAUTO_INCREMENT\s*=\s*([0-9]+)`)
func autoIncrementValue(create string) string {
if match := autoIncrementOption.FindStringSubmatch(unquotedSQL(create)); len(match) > 1 {
return match[1]
}
return "1"
}
func unsupportedNativeOptions(create string) []string {
words := strings.FieldsFunc(strings.ToUpper(unquotedSQL(create)), func(r rune) bool { return (r < 'A' || r > 'Z') && r != '_' })
var out []string
for i, w := range words {
switch w {
case "ENGINE_ATTRIBUTE", "SECONDARY_ENGINE_ATTRIBUTE", "KEY_BLOCK_SIZE", "COLUMN_FORMAT", "STORAGE":
out = append(out, w)
case "PARSER":
if i > 0 && words[i-1] == "WITH" {
out = append(out, "WITH PARSER")
}
}
}
return out
}
+54
View File
@@ -0,0 +1,54 @@
package mysql
import "testing"
func TestUnescapeExpression(t *testing.T) {
for _, tc := range []struct{ input, want string }{
{`_utf8mb4\'one\\ntwo\'`, `_utf8mb4'one\ntwo'`},
{`concat(\'don\\\'t\',\' change\')`, `concat('don\'t',' change')`},
{`(uuid())`, `(uuid())`},
} {
if got := unescapeExpression(tc.input); got != tc.want {
t.Errorf("%q: got %q want %q", tc.input, got, tc.want)
}
}
}
func TestNativeColumnDefaults(t *testing.T) {
create := "CREATE TABLE `odd(table` (\n" +
" `a,b` varbinary(8) DEFAULT 0x0027FF,\n" +
" `tick``name` varchar(30) DEFAULT 'don''t, (split)',\n" +
" `expr` varchar(30) DEFAULT (concat('DEFAULT, ', (uuid()))),\n" +
" `generated` varchar(30) GENERATED ALWAYS AS (concat('DEFAULT',`expr`)) VIRTUAL,\n" +
" `new\nline` int DEFAULT 3,\n" +
" PRIMARY KEY (`a,b`)\n) ENGINE=InnoDB"
got := columnDefaults(create)
want := map[string]string{"a,b": "0x0027FF", "tick`name": "'don''t, (split)'", "expr": "(concat('DEFAULT, ', (uuid())))", "new\nline": "3"}
if len(got) != len(want) {
t.Fatalf("defaults: %v", got)
}
for name, value := range want {
if got[name] != value {
t.Errorf("%s: got %q want %q", name, got[name], value)
}
}
}
func TestNativeOptionsIgnoreQuotedValues(t *testing.T) {
create := "CREATE TABLE `WITH PARSER` (`STORAGE` int COMMENT 'KEY_BLOCK_SIZE', KEY `ENGINE_ATTRIBUTE` (`STORAGE`) WITH PARSER `ngram`) ENGINE=InnoDB"
options := unsupportedNativeOptions(create)
if len(options) != 1 || options[0] != "WITH PARSER" {
t.Fatalf("native options: %v", options)
}
}
func TestNativeAutoIncrementIgnoresQuotedValues(t *testing.T) {
for _, tc := range []struct{ create, want string }{
{"CREATE TABLE `x` (`id` bigint AUTO_INCREMENT) ENGINE=InnoDB AUTO_INCREMENT=18446744073709551615", "18446744073709551615"},
{"CREATE TABLE `x` (`id` bigint AUTO_INCREMENT) COMMENT='AUTO_INCREMENT=12'", "1"},
} {
if got := autoIncrementValue(tc.create); got != tc.want {
t.Errorf("got %s want %s", got, tc.want)
}
}
}
+252 -43
View File
@@ -4,9 +4,10 @@ import (
"context"
"database/sql"
"fmt"
"regexp"
"strings"
_ "github.com/go-sql-driver/mysql"
mysqlDriver "github.com/go-sql-driver/mysql"
"git.warky.dev/wdevs/relspecgo/pkg/mariadb"
"git.warky.dev/wdevs/relspecgo/pkg/models"
@@ -31,12 +32,13 @@ func (r *Reader) ReadDatabase() (*models.Database, error) {
return nil, fmt.Errorf("failed to connect: %w", err)
}
defer r.close()
var name, version string
if err := r.db.QueryRowContext(r.ctx, "SELECT DATABASE()").Scan(&name); err != nil {
var name, version, sqlMode string
if err := r.db.QueryRowContext(r.ctx, "SELECT DATABASE(), @@SESSION.sql_mode").Scan(&name, &sqlMode); err != nil {
return nil, fmt.Errorf("failed to get database name: %w", err)
}
_ = r.db.QueryRowContext(r.ctx, "SELECT VERSION()").Scan(&version)
db := models.InitDatabase(name)
db.Metadata["mysql_sql_mode"] = sqlMode
db.DatabaseType, db.SourceFormat, db.DatabaseVersion = models.MySQLDatabaseType, "mysql", version
schemas, err := r.querySchemas(name)
if err != nil {
@@ -48,8 +50,10 @@ func (r *Reader) ReadDatabase() (*models.Database, error) {
return nil, err
}
schema.Tables = tables
schema.Metadata["mysql_sql_mode"] = sqlMode
for _, table := range tables {
table.Columns, err = r.queryColumns(schema.Name, table.Name)
table.Metadata["mysql_sql_mode"] = sqlMode
table.Columns, err = r.queryColumns(schema.Name, table)
if err != nil {
return nil, err
}
@@ -62,6 +66,15 @@ func (r *Reader) ReadDatabase() (*models.Database, error) {
return nil, err
}
table.RefSchema = schema
for _, c := range table.Constraints {
if c.Type == models.PrimaryKeyConstraint {
for _, n := range c.Columns {
if col := table.Columns[n]; col != nil {
col.IsPrimaryKey = true
}
}
}
}
for _, c := range table.Constraints {
if c.Type == models.ForeignKeyConstraint {
r.deriveRelationship(table, c)
@@ -71,6 +84,9 @@ func (r *Reader) ReadDatabase() (*models.Database, error) {
schema.RefDatabase = db
db.Schemas = append(db.Schemas, schema)
}
if err := r.readUnsupported(db, name); err != nil {
return nil, err
}
return db, nil
}
@@ -97,7 +113,17 @@ func (r *Reader) ReadTable() (*models.Table, error) {
}
func (r *Reader) connect() error {
db, err := sql.Open("mysql", r.options.ConnectionString)
cfg, err := mysqlDriver.ParseDSN(r.options.ConnectionString)
if err != nil {
return err
}
if cfg.Params == nil {
cfg.Params = map[string]string{}
}
// SHOW CREATE parsing requires quoted column identifiers, including when the
// caller's metadata connection would otherwise disable show-create quoting.
cfg.Params["sql_quote_show_create"] = "1"
db, err := sql.Open("mysql", cfg.FormatDSN())
if err != nil {
return err
}
@@ -134,61 +160,147 @@ func (r *Reader) querySchemas(current string) ([]*models.Schema, error) {
}
func (r *Reader) queryTables(schema string) ([]*models.Table, error) {
rows, err := r.db.QueryContext(r.ctx, "SELECT TABLE_NAME FROM information_schema.TABLES WHERE TABLE_SCHEMA = ? AND TABLE_TYPE = 'BASE TABLE' ORDER BY TABLE_NAME", schema)
rows, err := r.db.QueryContext(r.ctx, "SELECT TABLE_NAME, ENGINE, TABLE_COLLATION, TABLE_COMMENT, CREATE_OPTIONS, AUTO_INCREMENT FROM information_schema.TABLES WHERE TABLE_SCHEMA = ? AND TABLE_TYPE = 'BASE TABLE' ORDER BY TABLE_NAME", schema)
if err != nil {
return nil, err
}
defer rows.Close()
var out []*models.Table
for rows.Next() {
var n string
if err := rows.Scan(&n); err != nil {
var n, engine, collation, comment, options string
var auto sql.NullString
if err := rows.Scan(&n, &engine, &collation, &comment, &options, &auto); err != nil {
return nil, err
}
out = append(out, models.InitTable(n, schema))
t := models.InitTable(n, schema)
t.Comment = comment
t.Metadata["mysql_engine"], t.Metadata["mysql_collation"], t.Metadata["mysql_create_options"] = engine, collation, options
if auto.Valid {
t.Metadata["mysql_auto_increment"] = auto.String
}
out = append(out, t)
}
return out, rows.Err()
}
func (r *Reader) queryColumns(schema, table string) (map[string]*models.Column, error) {
rows, err := r.db.QueryContext(r.ctx, `SELECT COLUMN_NAME, COLUMN_TYPE, IS_NULLABLE, COLUMN_DEFAULT, ORDINAL_POSITION, EXTRA, COLUMN_COMMENT FROM information_schema.COLUMNS WHERE TABLE_SCHEMA = ? AND TABLE_NAME = ? ORDER BY ORDINAL_POSITION`, schema, table)
var onUpdateExtra = regexp.MustCompile(`on update current_timestamp(?:\(\d*\))?`)
func (r *Reader) queryColumns(schema string, table *models.Table) (map[string]*models.Column, error) {
rows, err := r.db.QueryContext(r.ctx, `SELECT COLUMN_NAME, COLUMN_TYPE, IS_NULLABLE, COLUMN_DEFAULT, ORDINAL_POSITION, EXTRA, COLUMN_COMMENT,
CHARACTER_MAXIMUM_LENGTH, NUMERIC_PRECISION, NUMERIC_SCALE, DATETIME_PRECISION, COLLATION_NAME, CHARACTER_SET_NAME, GENERATION_EXPRESSION, SRS_ID
FROM information_schema.COLUMNS WHERE TABLE_SCHEMA=? AND TABLE_NAME=? ORDER BY ORDINAL_POSITION`, schema, table.Name)
if err != nil {
return nil, err
return nil, fmt.Errorf("columns %s.%s: %w", schema, table.Name, err)
}
defer rows.Close()
out := map[string]*models.Column{}
for rows.Next() {
var name, typ, nullable, extra, comment string
var def sql.NullString
var def, collation, charset, generation sql.NullString
var length, precision, scale, datetime, srid sql.NullInt64
var pos int
if err := rows.Scan(&name, &typ, &nullable, &def, &pos, &extra, &comment); err != nil {
if err := rows.Scan(&name, &typ, &nullable, &def, &pos, &extra, &comment, &length, &precision, &scale, &datetime, &collation, &charset, &generation, &srid); err != nil {
return nil, err
}
c := models.InitColumn(name, table, schema)
c.Type = r.mapDataType(typ)
c.NotNull = strings.EqualFold(nullable, "NO")
c.Sequence = uint(pos)
c.Comment = comment
c := models.InitColumn(name, table.Name, schema)
c.Type, c.NotNull, c.Sequence, c.Comment = r.mapDataType(typ), nullable == "NO", uint(pos), comment
// The native declaration preserves mediumint, enum/set, binary/blob sizes,
// unsigned/zerofill and timestamp versus datetime. Canonical dimensions remain editable.
c.Length, c.Precision, c.Scale = int(length.Int64), int(precision.Int64), int(scale.Int64)
if datetime.Valid {
c.Precision = int(datetime.Int64)
}
c.Collation = collation.String
c.Metadata["mysql_type"] = typ
c.Metadata["mysql_type_signature"] = typeSignature(c)
c.Metadata["mysql_charset"] = charset.String
c.Metadata["mysql_datetime_precision"] = datetime.Int64
c.Metadata["mysql_default_expression"] = strings.Contains(extra, "DEFAULT_GENERATED") || strings.HasPrefix(strings.ToLower(def.String), "current_timestamp") && (c.Type == "timestamp")
if def.Valid {
c.Default = def.String
if strings.Contains(extra, "DEFAULT_GENERATED") {
c.Default = unescapeExpression(def.String)
}
if strings.HasPrefix(typ, "bit(") && def.Valid {
c.Metadata["mysql_default_expression"] = true
}
}
c.AutoIncrement = strings.Contains(extra, "auto_increment")
c.Generated, c.GenerationExpression = generation.String != "", unescapeExpression(generation.String)
if strings.Contains(extra, "STORED GENERATED") {
c.Metadata["mysql_generation_storage"] = "STORED"
} else if c.Generated {
c.Metadata["mysql_generation_storage"] = "VIRTUAL"
}
c.Metadata["mysql_invisible"] = strings.Contains(extra, "INVISIBLE")
remaining := onUpdateExtra.ReplaceAllString(strings.ToLower(extra), "")
for _, known := range []string{"auto_increment", "default_generated", "stored generated", "virtual generated", "invisible"} {
remaining = strings.ReplaceAll(remaining, known, "")
}
if remaining = strings.TrimSpace(remaining); remaining != "" {
c.Metadata["mysql_unsupported_extra"] = remaining
}
if i := strings.Index(strings.ToLower(extra), "on update "); i >= 0 {
c.Metadata["mysql_on_update"] = strings.Fields(extra[i+10:])[0]
}
if srid.Valid {
c.Metadata["mysql_srs_id"] = srid.Int64
}
c.AutoIncrement = strings.Contains(strings.ToLower(extra), "auto_increment")
out[name] = c
}
return out, rows.Err()
if err := rows.Err(); err != nil {
return nil, err
}
var tableName, create string
qualified := "`" + strings.ReplaceAll(schema, "`", "``") + "`.`" + strings.ReplaceAll(table.Name, "`", "``") + "`"
if err := r.db.QueryRowContext(r.ctx, "SHOW CREATE TABLE "+qualified).Scan(&tableName, &create); err != nil {
return nil, fmt.Errorf("native defaults %s.%s: %w", schema, table.Name, err)
}
// information_schema AUTO_INCREMENT may be stale because of statistics caching.
for _, c := range out {
if c.AutoIncrement {
table.Metadata["mysql_auto_increment"] = autoIncrementValue(create)
break
}
}
if options := unsupportedNativeOptions(create); len(options) > 0 {
table.Metadata["mysql_unsupported_options"] = strings.Join(options, ", ")
}
for name, sql := range columnDefaults(create) {
c := out[name]
if c == nil || c.Default == nil {
continue
}
if c.Type == "bytea" {
c.Default = sql
}
c.Metadata["mysql_default_sql"] = sql
c.Metadata["mysql_default_value"] = fmt.Sprint(c.Default)
}
return out, nil
}
func typeSignature(c *models.Column) string {
return fmt.Sprintf("%s/%d/%d/%d", c.Type, c.Length, c.Precision, c.Scale)
}
func (r *Reader) queryConstraints(schema, table string) (map[string]*models.Constraint, error) {
rows, err := r.db.QueryContext(r.ctx, `SELECT CONSTRAINT_NAME, CONSTRAINT_TYPE, COLUMN_NAME, REFERENCED_TABLE_SCHEMA, REFERENCED_TABLE_NAME, REFERENCED_COLUMN_NAME, ORDINAL_POSITION FROM information_schema.KEY_COLUMN_USAGE k JOIN information_schema.TABLE_CONSTRAINTS t USING (CONSTRAINT_SCHEMA, TABLE_NAME, CONSTRAINT_NAME) WHERE k.TABLE_SCHEMA=? AND k.TABLE_NAME=? ORDER BY CONSTRAINT_NAME, ORDINAL_POSITION`, schema, table)
rows, err := r.db.QueryContext(r.ctx, `SELECT k.CONSTRAINT_NAME, t.CONSTRAINT_TYPE, k.COLUMN_NAME,
k.REFERENCED_TABLE_SCHEMA, k.REFERENCED_TABLE_NAME, k.REFERENCED_COLUMN_NAME, k.ORDINAL_POSITION,
rc.DELETE_RULE, rc.UPDATE_RULE FROM information_schema.KEY_COLUMN_USAGE k
JOIN information_schema.TABLE_CONSTRAINTS t USING (CONSTRAINT_SCHEMA,TABLE_NAME,CONSTRAINT_NAME)
LEFT JOIN information_schema.REFERENTIAL_CONSTRAINTS rc ON rc.CONSTRAINT_SCHEMA=k.CONSTRAINT_SCHEMA AND rc.TABLE_NAME=k.TABLE_NAME AND rc.CONSTRAINT_NAME=k.CONSTRAINT_NAME
WHERE k.TABLE_SCHEMA=? AND k.TABLE_NAME=? ORDER BY k.CONSTRAINT_NAME,k.ORDINAL_POSITION`, schema, table)
if err != nil {
return nil, err
return nil, fmt.Errorf("constraints %s.%s: %w", schema, table, err)
}
defer rows.Close()
out := map[string]*models.Constraint{}
for rows.Next() {
var name, typ, col, rs, rt, rc string
var name, typ string
var col, rs, rt, rc, del, upd sql.NullString
var pos int
if err := rows.Scan(&name, &typ, &col, &rs, &rt, &rc, &pos); err != nil {
if err := rows.Scan(&name, &typ, &col, &rs, &rt, &rc, &pos, &del, &upd); err != nil {
rows.Close()
return nil, err
}
c := out[name]
@@ -201,31 +313,54 @@ func (r *Reader) queryConstraints(schema, table string) (map[string]*models.Cons
ct = models.ForeignKeyConstraint
}
c = models.InitConstraint(name, ct)
c.Schema = schema
c.Table = table
c.ReferencedSchema = rs
c.ReferencedTable = rt
c.Schema, c.Table = schema, table
c.ReferencedSchema, c.ReferencedTable = rs.String, rt.String
c.OnDelete, c.OnUpdate = del.String, upd.String
out[name] = c
}
c.Columns = append(c.Columns, col)
if rc != "" {
c.ReferencedColumns = append(c.ReferencedColumns, rc)
if col.Valid {
c.Columns = append(c.Columns, col.String)
}
if rc.Valid {
c.ReferencedColumns = append(c.ReferencedColumns, rc.String)
}
}
err = rows.Err()
rows.Close()
if err != nil {
return nil, err
}
rows, err = r.db.QueryContext(r.ctx, `SELECT t.CONSTRAINT_NAME,c.CHECK_CLAUSE,t.ENFORCED FROM information_schema.TABLE_CONSTRAINTS t JOIN information_schema.CHECK_CONSTRAINTS c USING (CONSTRAINT_SCHEMA,CONSTRAINT_NAME) WHERE t.TABLE_SCHEMA=? AND t.TABLE_NAME=? AND t.CONSTRAINT_TYPE='CHECK' ORDER BY t.CONSTRAINT_NAME`, schema, table)
if err != nil {
return nil, fmt.Errorf("checks %s.%s (requires MySQL 8.0.16+): %w", schema, table, err)
}
defer rows.Close()
for rows.Next() {
var name, expr, enforced string
if err := rows.Scan(&name, &expr, &enforced); err != nil {
return nil, err
}
c := models.InitConstraint(name, models.CheckConstraint)
c.Schema, c.Table, c.Expression = schema, table, unescapeExpression(expr)
c.Metadata = map[string]any{"mysql_enforced": enforced == "YES"}
out[name] = c
}
return out, rows.Err()
}
func (r *Reader) queryIndexes(schema, table string) (map[string]*models.Index, error) {
rows, err := r.db.QueryContext(r.ctx, `SELECT INDEX_NAME, NON_UNIQUE, COLUMN_NAME, SEQ_IN_INDEX, INDEX_TYPE FROM information_schema.STATISTICS WHERE TABLE_SCHEMA=? AND TABLE_NAME=? AND INDEX_NAME <> 'PRIMARY' ORDER BY INDEX_NAME, SEQ_IN_INDEX`, schema, table)
rows, err := r.db.QueryContext(r.ctx, `SELECT INDEX_NAME, NON_UNIQUE, COLUMN_NAME, SEQ_IN_INDEX, INDEX_TYPE, EXPRESSION, SUB_PART, COLLATION, IS_VISIBLE, INDEX_COMMENT FROM information_schema.STATISTICS WHERE TABLE_SCHEMA=? AND TABLE_NAME=? ORDER BY INDEX_NAME,SEQ_IN_INDEX`, schema, table)
if err != nil {
return nil, err
return nil, fmt.Errorf("indexes %s.%s (requires MySQL 8.0.13+): %w", schema, table, err)
}
defer rows.Close()
out := map[string]*models.Index{}
for rows.Next() {
var name, col, typ string
var name, typ, visible, comment string
var col, expr, direction sql.NullString
var prefix sql.NullInt64
var non, seq int
if err := rows.Scan(&name, &non, &col, &seq, &typ); err != nil {
if err := rows.Scan(&name, &non, &col, &seq, &typ, &expr, &prefix, &direction, &visible, &comment); err != nil {
return nil, err
}
i := out[name]
@@ -233,22 +368,96 @@ func (r *Reader) queryIndexes(schema, table string) (map[string]*models.Index, e
i = models.InitIndex(name, table, schema)
i.Unique = non == 0
i.Type = strings.ToLower(typ)
i.Comment = comment
i.Metadata["mysql_visible"] = visible == "YES"
out[name] = i
}
i.Columns = append(i.Columns, col)
i.Columns = append(i.Columns, col.String)
i.KeyParts = append(i.KeyParts, models.IndexPart{Name: col.String, Expression: unescapeExpression(expr.String), Length: int(prefix.Int64), Descending: direction.String == "D"})
}
return out, rows.Err()
}
// Record objects outside the supported table-schema contract rather than silently
// discarding them. Writers can warn or reject the conversion before any output.
func (r *Reader) readUnsupported(db *models.Database, schema string) error {
queries := []struct{ kind, sql string }{
{"view", `SELECT TABLE_NAME FROM information_schema.VIEWS WHERE TABLE_SCHEMA=?`},
{"trigger", `SELECT TRIGGER_NAME FROM information_schema.TRIGGERS WHERE TRIGGER_SCHEMA=?`},
{"routine", `SELECT ROUTINE_NAME FROM information_schema.ROUTINES WHERE ROUTINE_SCHEMA=?`},
{"event", `SELECT EVENT_NAME FROM information_schema.EVENTS WHERE EVENT_SCHEMA=?`},
{"partitioned table", `SELECT DISTINCT TABLE_NAME FROM information_schema.PARTITIONS WHERE TABLE_SCHEMA=? AND PARTITION_NAME IS NOT NULL`},
}
var unsupported []string
for _, q := range queries {
rows, err := r.db.QueryContext(r.ctx, q.sql, schema)
if err != nil {
return fmt.Errorf("discover %s: %w", q.kind, err)
}
for rows.Next() {
var name string
if err := rows.Scan(&name); err != nil {
rows.Close()
return err
}
unsupported = append(unsupported, q.kind+" "+schema+"."+name)
}
err = rows.Err()
rows.Close()
if err != nil {
return err
}
}
if len(unsupported) > 0 {
db.Metadata["mysql_unsupported"] = unsupported
for _, s := range db.Schemas {
s.Metadata["mysql_unsupported"] = unsupported
}
if r.options.Progress != nil {
for _, item := range unsupported {
r.options.Progress("warning: unsupported MySQL " + item)
}
}
}
return nil
}
func (r *Reader) deriveRelationship(t *models.Table, c *models.Constraint) {
n := fmt.Sprintf("%s_to_%s", t.Name, c.ReferencedTable)
n := fmt.Sprintf("%s_%s", t.Name, c.Name)
rel := models.InitRelationship(n, models.OneToMany)
rel.FromTable = t.Name
rel.FromSchema = t.Schema
rel.FromTable, rel.FromSchema = t.Name, t.Schema
rel.FromColumns = append([]string(nil), c.Columns...)
rel.ToTable = c.ReferencedTable
rel.ToSchema = c.ReferencedSchema
rel.ToTable, rel.ToSchema = c.ReferencedTable, c.ReferencedSchema
rel.ToColumns = append([]string(nil), c.ReferencedColumns...)
rel.ForeignKey = c.Name
t.Relationships[n] = rel
}
// MySQL information_schema escapes SQL expressions by an additional layer
// (including quote delimiters). This covers defaults, generated columns, checks,
// and functional indexes. Literal defaults are already values and bypass this.
func unescapeExpression(s string) string {
var out strings.Builder
for i := 0; i < len(s); i++ {
if s[i] == '\\' && i+1 < len(s) {
i++
switch s[i] {
case 'n':
out.WriteByte('\n')
case 'r':
out.WriteByte('\r')
case 't':
out.WriteByte('\t')
case '0':
out.WriteByte(0)
case 'Z':
out.WriteByte(26)
default:
out.WriteByte(s[i])
}
} else {
out.WriteByte(s[i])
}
}
return out.String()
}
+16 -1
View File
@@ -3,12 +3,13 @@ package mysql
import (
"testing"
"git.warky.dev/wdevs/relspecgo/pkg/models"
"git.warky.dev/wdevs/relspecgo/pkg/readers"
)
func TestReaderMapDataType(t *testing.T) {
r := NewReader(&readers.ReaderOptions{})
for _, tc := range []struct{ input, want string }{{"varchar(64)", "string"}, {"bigint unsigned", "int64"}, {"datetime", "timestamp"}, {"json", "json"}} {
for _, tc := range []struct{ input, want string }{{"varchar(64)", "string"}, {"bigint unsigned", "uint64"}, {"datetime", "timestamp"}, {"json", "json"}} {
if got := r.mapDataType(tc.input); got != tc.want {
t.Errorf("mapDataType(%q) = %q, want %q", tc.input, got, tc.want)
}
@@ -20,3 +21,17 @@ func TestReaderRequiresConnectionString(t *testing.T) {
t.Fatal("expected missing connection string error")
}
}
func TestRelationshipNamesCannotCollapseDistinctForeignKeys(t *testing.T) {
table := models.InitTable("t", "app")
reader := NewReader(&readers.ReaderOptions{})
for _, c := range []*models.Constraint{
{Name: "a_to_b", ReferencedTable: "c"},
{Name: "a", ReferencedTable: "b_to_c"},
} {
reader.deriveRelationship(table, c)
}
if len(table.Relationships) != 2 {
t.Fatalf("distinct foreign keys collapsed: %+v", table.Relationships)
}
}
+12 -2
View File
@@ -439,7 +439,9 @@ func (r *Reader) queryForeignKeys(schemaName string) (map[string][]*models.Const
fcl.relname AS referenced_table_name,
fatt.attname AS referenced_column,
` + actionCase("con.confupdtype") + ` AS update_rule,
` + actionCase("con.confdeltype") + ` AS delete_rule
` + actionCase("con.confdeltype") + ` AS delete_rule,
con.condeferrable AS deferrable,
con.condeferred AS initially_deferred
FROM pg_catalog.pg_constraint con
JOIN pg_catalog.pg_class cl ON cl.oid = con.conrelid
JOIN pg_catalog.pg_namespace ns ON ns.oid = cl.relnamespace
@@ -470,6 +472,8 @@ func (r *Reader) queryForeignKeys(schemaName string) (map[string][]*models.Const
referencedColumns []string
updateRule string
deleteRule string
deferrable bool
initiallyDeferred bool
}
fkMap := make(map[string]*fkData)
@@ -479,8 +483,9 @@ func (r *Reader) queryForeignKeys(schemaName string) (map[string][]*models.Const
var foreignSchema, foreignTable, foreignColumn string
var referencedSchema, referencedTable, referencedColumn string
var updateRule, deleteRule string
var deferrable, initiallyDeferred bool
if err := rows.Scan(&schema, &tableName, &constraintName, &foreignSchema, &foreignTable, &foreignColumn, &referencedSchema, &referencedTable, &referencedColumn, &updateRule, &deleteRule); err != nil {
if err := rows.Scan(&schema, &tableName, &constraintName, &foreignSchema, &foreignTable, &foreignColumn, &referencedSchema, &referencedTable, &referencedColumn, &updateRule, &deleteRule, &deferrable, &initiallyDeferred); err != nil {
return nil, err
}
@@ -497,6 +502,8 @@ func (r *Reader) queryForeignKeys(schemaName string) (map[string][]*models.Const
referencedColumns: []string{},
updateRule: updateRule,
deleteRule: deleteRule,
deferrable: deferrable,
initiallyDeferred: initiallyDeferred,
}
}
@@ -517,6 +524,9 @@ func (r *Reader) queryForeignKeys(schemaName string) (map[string][]*models.Const
constraint.ReferencedColumns = fk.referencedColumns
constraint.OnUpdate = fk.updateRule
constraint.OnDelete = fk.deleteRule
deferrable, initiallyDeferred := fk.deferrable, fk.initiallyDeferred
constraint.Deferrable = &deferrable
constraint.InitiallyDeferred = &initiallyDeferred
tableKey := fk.schema + "." + fk.tableName
foreignKeys[tableKey] = append(foreignKeys[tableKey], constraint)
+1 -1
View File
@@ -152,7 +152,7 @@ func (r *Reader) readScripts() ([]*models.Script, error) {
if err != nil {
return fmt.Errorf("failed to read file %s: %w", path, err)
}
sql, err := assetloader.ProcessEmbedDirectives(path, string(content))
sql, err := assetloader.ProcessEmbedDirectivesWithOptions(path, string(content), assetloader.EmbedOptions{AssetRoot: r.options.FilePath})
if err != nil {
return err
}
+136
View File
@@ -0,0 +1,136 @@
// Package safefile provides portable path validation and root-scoped file access.
package safefile
import (
"fmt"
"io"
"os"
"path/filepath"
"strings"
"unicode"
)
// Relative validates artifact paths consistently on Unix and Windows. It rejects
// traversal rather than normalizing it, so aliases cannot hide filename collisions.
func Relative(name string) (string, error) {
if name == "" || !filepath.IsLocal(name) || strings.ContainsAny(name, `\:<>"|?*`) {
return "", fmt.Errorf("unsafe relative path %q", name)
}
for _, r := range name {
if unicode.IsControl(r) {
return "", fmt.Errorf("control character in path %q", name)
}
}
for _, part := range strings.Split(filepath.ToSlash(name), "/") {
if part == "" || part == "." || part == ".." || strings.HasSuffix(part, ".") || strings.HasSuffix(part, " ") {
return "", fmt.Errorf("unsafe path component in %q", name)
}
base := strings.ToUpper(strings.SplitN(part, ".", 2)[0])
device := base == "CON" || base == "PRN" || base == "AUX" || base == "NUL" || base == "CONIN$" || base == "CONOUT$"
for _, prefix := range []string{"COM", "LPT"} {
if strings.HasPrefix(base, prefix) {
suffix := []rune(strings.TrimPrefix(base, prefix))
device = device || (len(suffix) == 1 && strings.ContainsRune("123456789¹²³", suffix[0]))
}
}
if device {
return "", fmt.Errorf("reserved device path %q", name)
}
}
return filepath.FromSlash(name), nil
}
// Check rejects existing symlinks and special files; os.Root separately enforces
// containment at open/rename time, including if a symlink changes after this check.
func Check(root *os.Root, name string) error {
if _, err := Relative(filepath.ToSlash(name)); err != nil {
return err
}
parts := strings.Split(name, string(filepath.Separator))
current := ""
for i, part := range parts {
current = filepath.Join(current, part)
info, err := root.Lstat(current)
if os.IsNotExist(err) {
return nil
}
if err != nil {
return err
}
if info.Mode()&os.ModeSymlink != 0 {
return fmt.Errorf("symlink is not allowed in path %q", name)
}
if i < len(parts)-1 && !info.IsDir() {
return fmt.Errorf("parent is not a directory in path %q", name)
}
if i == len(parts)-1 && !info.Mode().IsRegular() {
return fmt.Errorf("path is not a regular file: %q", name)
}
}
return nil
}
// Read opens through a pinned root and bounds both the declared and actual size.
func Read(root *os.Root, name string, maxBytes int64) ([]byte, error) {
if maxBytes <= 0 || maxBytes == (1<<63)-1 {
return nil, fmt.Errorf("file size limit must be positive and bounded")
}
if err := Check(root, name); err != nil {
return nil, err
}
file, err := root.Open(name)
if err != nil {
return nil, err
}
defer file.Close()
info, err := file.Stat()
if err != nil {
return nil, err
}
if !info.Mode().IsRegular() {
return nil, fmt.Errorf("asset is not a regular file: %q", name)
}
if info.Size() > maxBytes {
return nil, fmt.Errorf("asset %q exceeds %d byte limit", name, maxBytes)
}
data, err := io.ReadAll(io.LimitReader(file, maxBytes+1))
if err != nil {
return nil, err
}
if int64(len(data)) > maxBytes {
return nil, fmt.Errorf("asset %q exceeds %d byte limit", name, maxBytes)
}
return data, nil
}
// CheckRoot rejects symlinks in the configured root or its existing ancestors.
// The configured root is a trusted caller-selected boundary; OpenRoot pins it for
// subsequent operations. Missing components are allowed for output root creation.
func CheckRoot(path string) error {
absolute, err := filepath.Abs(path)
if err != nil {
return err
}
volume := filepath.VolumeName(absolute)
current := volume + string(filepath.Separator)
for _, part := range strings.Split(strings.TrimPrefix(absolute, current), string(filepath.Separator)) {
if part == "" {
continue
}
current = filepath.Join(current, part)
info, err := os.Lstat(current)
if os.IsNotExist(err) {
return nil
}
if err != nil {
return err
}
if info.Mode()&os.ModeSymlink != 0 {
return fmt.Errorf("symlink is not allowed in root path %q", path)
}
if !info.IsDir() {
return fmt.Errorf("root parent is not a directory: %q", current)
}
}
return nil
}
+101
View File
@@ -0,0 +1,101 @@
// Package sqllex recognizes PostgreSQL lexical boundaries without parsing SQL.
package sqllex
import "strings"
func start(c byte) bool { return c >= 128 || c == '_' || c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' }
func ident(c byte) bool { return start(c) || c >= '0' && c <= '9' || c >= 128 }
// Walk visits named parameters and standalone line comments in executable SQL. Quoted
// strings/identifiers, dollar bodies, casts and nested block comments are opaque.
// Returning replacement text is literal: it is never scanned again.
func Walk(sql string, parameter, lineComment func(string) string) string {
var out strings.Builder
for i := 0; i < len(sql); {
begin := i
switch {
case strings.HasPrefix(sql[i:], "--"):
i += 2
for i < len(sql) && sql[i] != '\n' && sql[i] != '\r' {
i++
}
text := sql[begin:i]
if lineComment != nil && strings.TrimSpace(sql[strings.LastIndexAny(sql[:begin], "\r\n")+1:begin]) == "" {
text = lineComment(text)
}
out.WriteString(text)
continue
case strings.HasPrefix(sql[i:], "/*"):
i += 2
depth := 1
for i < len(sql) && depth > 0 {
if strings.HasPrefix(sql[i:], "/*") {
depth++
i += 2
} else if strings.HasPrefix(sql[i:], "*/") {
depth--
i += 2
} else {
i++
}
}
case sql[i] == '\'' || sql[i] == '"':
q := sql[i]
escapes := q == '\'' && i > 0 && (sql[i-1] == 'E' || sql[i-1] == 'e') && (i == 1 || !ident(sql[i-2]))
i++
for i < len(sql) {
if escapes && sql[i] == '\\' {
i++
if i < len(sql) {
i++
}
continue
}
if sql[i] == q {
i++
if i < len(sql) && sql[i] == q {
i++
continue
}
break
}
i++
}
case sql[i] == '$' && (i == 0 || !ident(sql[i-1]) && sql[i-1] != '$'):
j := i + 1
if j < len(sql) && start(sql[j]) {
for j < len(sql) && ident(sql[j]) {
j++
}
}
if j < len(sql) && sql[j] == '$' {
tag := sql[i : j+1]
end := strings.Index(sql[j+1:], tag)
if end < 0 {
i = len(sql)
} else {
i = j + 1 + end + len(tag)
}
} else {
i++
}
case strings.HasPrefix(sql[i:], "::"):
i += 2
case sql[i] == ':' && i+1 < len(sql) && start(sql[i+1]) && (i == 0 || !ident(sql[i-1]) && sql[i-1] != ':'):
i += 2
for i < len(sql) && ident(sql[i]) {
i++
}
text := sql[begin:i]
if parameter != nil {
text = parameter(text)
}
out.WriteString(text)
continue
default:
i++
}
out.WriteString(sql[begin:i])
}
return out.String()
}
+28
View File
@@ -0,0 +1,28 @@
package sqllex
import (
"strings"
"testing"
)
func TestWalkSQLContexts(t *testing.T) {
sql := `SELECT :x,:x, :x::text, ':hidden', 'it'':hidden', E'it\':hidden', ":hidden", $$:hidden$$, $名$:hidden$名$;
-- :hidden
/* :hidden /* :hidden */ :hidden */ SELECT :y, value::integer, x := 1;`
var names []string
got := Walk(sql, func(name string) string { names = append(names, name); return "$1 ${2} :inserted" }, nil)
if strings.Join(names, ",") != ":x,:x,:x,:y" {
t.Fatalf("parameters: %v", names)
}
if strings.Count(got, "$1 ${2} :inserted") != 4 || !strings.Contains(got, "value::integer") {
t.Fatalf("result: %s", got)
}
}
func TestUnterminatedSQLContextsAreOpaque(t *testing.T) {
for _, sql := range []string{"SELECT ':x", `SELECT ":x`, "DO $tag$:x", "/* :x", "-- :x"} {
if got := Walk(sql, func(string) string { t.Fatal("unexpected parameter"); return "" }, nil); got != sql {
t.Fatalf("changed %q", sql)
}
}
}
+157
View File
@@ -0,0 +1,157 @@
package transform
import (
"fmt"
"reflect"
"strings"
"git.warky.dev/wdevs/relspecgo/pkg/models"
)
// LowercaseIdentifiers copies the model and lowercases names and structured
// references. SQL expressions, defaults, enum values, comments and metadata are
// preserved. Callers must adjust identifiers inside their own SQL expressions.
// Collisions are rejected before any output is written.
func LowercaseIdentifiers(db *models.Database) (*models.Database, error) {
if db == nil {
return nil, fmt.Errorf("database is required")
}
seen := map[any]reflect.Value{}
nameFields := map[string]bool{}
for _, name := range strings.Fields("Name Schema Table Tablespace Collation TableName SchemaName FromTable FromSchema ToTable ToSchema ThroughTable ThroughSchema ForeignKey ReferencedTable ReferencedSchema OwnedByTable OwnedByColumn") {
nameFields[name] = true
}
listFields := map[string]bool{"Columns": true, "Include": true, "ReferencedColumns": true, "FromColumns": true, "ToColumns": true, "RunAfter": true}
var copyValue func(reflect.Value) (reflect.Value, error)
copyValue = func(v reflect.Value) (reflect.Value, error) {
if !v.IsValid() {
return v, nil
}
switch v.Kind() {
case reflect.Pointer:
if v.IsNil() || v.Type().Elem().PkgPath() != reflect.TypeOf(models.Database{}).PkgPath() {
return v, nil
}
if prior, ok := seen[v.Interface()]; ok {
return prior, nil
}
out := reflect.New(v.Type().Elem())
seen[v.Interface()] = out
copied, err := copyValue(v.Elem())
if err != nil {
return reflect.Value{}, err
}
out.Elem().Set(copied)
return out, nil
case reflect.Struct:
out := reflect.New(v.Type()).Elem()
out.Set(v)
for i := 0; i < v.NumField(); i++ {
field := v.Type().Field(i).Name
value := v.Field(i)
if nameFields[field] && value.Kind() == reflect.String {
out.Field(i).SetString(strings.ToLower(value.String()))
continue
}
if (v.Type() != reflect.TypeOf(models.Index{}) || field != "Columns") && listFields[field] && value.Kind() == reflect.Slice && value.Type().Elem().Kind() == reflect.String {
list := reflect.MakeSlice(value.Type(), value.Len(), value.Len())
for j := 0; j < value.Len(); j++ {
list.Index(j).SetString(strings.ToLower(value.Index(j).String()))
}
out.Field(i).Set(list)
continue
}
if field == "Metadata" || field == "Default" || field == "Properties" || field == "Permissions" {
continue
}
copied, err := copyValue(value)
if err != nil {
return reflect.Value{}, err
}
out.Field(i).Set(copied)
}
if v.Type() == reflect.TypeOf(models.Table{}) {
original := v.Interface().(models.Table)
copied := out.Addr().Interface().(*models.Table)
for originalKey, index := range original.Indexes {
if index == nil {
continue
}
target := copied.Indexes[strings.ToLower(originalKey)]
target.Columns = append([]string(nil), index.Columns...)
for i, reference := range index.Columns {
for _, column := range original.Columns {
if column != nil && (reference == column.Name || reference == `"`+strings.ReplaceAll(column.Name, `"`, `""`)+`"`) {
target.Columns[i] = strings.ToLower(column.Name)
break
}
}
}
}
}
return out, nil
case reflect.Slice:
if v.IsNil() {
return v, nil
}
out := reflect.MakeSlice(v.Type(), v.Len(), v.Len())
names := map[string]bool{}
for i := 0; i < v.Len(); i++ {
copied, err := copyValue(v.Index(i))
if err != nil {
return reflect.Value{}, err
}
out.Index(i).Set(copied)
if copied.Kind() == reflect.Pointer && !copied.IsNil() && copied.Elem().Kind() == reflect.Struct {
name := copied.Elem().FieldByName("Name")
if name.IsValid() {
n := name.String()
if names[n] {
return reflect.Value{}, fmt.Errorf("lowercase identifier collision: %q", n)
}
names[n] = true
}
}
}
return out, nil
case reflect.Map:
if v.IsNil() || v.Type().Elem().Kind() != reflect.Pointer {
return v, nil
}
out := reflect.MakeMap(v.Type())
names := map[string]bool{}
iter := v.MapRange()
for iter.Next() {
key := iter.Key()
if key.Kind() == reflect.String {
key = reflect.ValueOf(strings.ToLower(key.String())).Convert(key.Type())
}
if out.MapIndex(key).IsValid() {
return reflect.Value{}, fmt.Errorf("lowercase identifier collision: %v", key.Interface())
}
copied, err := copyValue(iter.Value())
if err != nil {
return reflect.Value{}, err
}
if copied.Kind() == reflect.Pointer && !copied.IsNil() && copied.Elem().Kind() == reflect.Struct {
name := copied.Elem().FieldByName("Name")
if name.IsValid() {
n := name.String()
if names[n] {
return reflect.Value{}, fmt.Errorf("lowercase identifier collision: %q", n)
}
names[n] = true
}
}
out.SetMapIndex(key, copied)
}
return out, nil
}
return v, nil
}
value, err := copyValue(reflect.ValueOf(db))
if err != nil {
return nil, err
}
return value.Interface().(*models.Database), nil
}
+53
View File
@@ -0,0 +1,53 @@
package transform
import (
"testing"
"git.warky.dev/wdevs/relspecgo/pkg/models"
)
func TestLowercaseIdentifiersReferencesAndData(t *testing.T) {
db := models.InitDatabase("DB")
s := models.InitSchema("App")
tbl := models.InitTable("Users", "App")
col := models.InitColumn("ID", "Users", "App")
tbl.Tablespace = "FastStore"
col.Collation = "MyCollation"
col.Default = "Keep CASE"
col.GenerationExpression = `"ID" + 1`
tbl.Columns[col.Name] = col
tbl.Constraints["FK"] = &models.Constraint{Name: "FK", Columns: []string{"ID"}, ReferencedSchema: "Other", ReferencedTable: "Target", ReferencedColumns: []string{"Key"}}
tbl.Indexes["IDX"] = &models.Index{Name: "IDX", Columns: []string{"ID", "lower('VALUE')"}, Include: []string{"ID"}, Where: `"ID" > 0`}
tbl.RefSchema = s
s.Tables = append(s.Tables, tbl)
db.Schemas = append(db.Schemas, s)
got, err := LowercaseIdentifiers(db)
if err != nil {
t.Fatal(err)
}
table := got.Schemas[0].Tables[0]
if table.Tablespace != "faststore" || table.Columns["id"].Collation != "mycollation" || got.Name != "db" || table.Name != "users" || table.Schema != "app" || table.Columns["id"].Name != "id" || table.Constraints["fk"].ReferencedColumns[0] != "key" || table.Indexes["idx"].Include[0] != "id" || table.RefSchema != got.Schemas[0] {
t.Fatalf("bad normalized model: %+v", table)
}
if table.Columns["id"].Default != "Keep CASE" || table.Columns["id"].GenerationExpression != `"ID" + 1` || table.Indexes["idx"].Where != `"ID" > 0` || table.Indexes["idx"].Columns[1] != "lower('VALUE')" {
t.Fatal("data/expression changed")
}
if tbl.Name != "Users" || tbl.Columns["ID"] != col {
t.Fatal("input mutated")
}
}
func TestLowercaseIdentifierCollisions(t *testing.T) {
db := models.InitDatabase("db")
s := models.InitSchema("app")
db.Schemas = append(db.Schemas, s)
s.Tables = []*models.Table{models.InitTable("Users", "app"), models.InitTable("users", "app")}
if _, err := LowercaseIdentifiers(db); err == nil {
t.Fatal("expected table collision")
}
s.Tables = s.Tables[:1]
s.Tables[0].Columns = map[string]*models.Column{"ID": {Name: "ID"}, "id": {Name: "id"}}
if _, err := LowercaseIdentifiers(db); err == nil {
t.Fatal("expected column collision")
}
}
+7
View File
@@ -273,6 +273,13 @@ func (w *Writer) constraintToDBML(c *models.Constraint, t *models.Table) string
actions = append(actions, fmt.Sprintf("update: %s", c.OnUpdate))
}
if c.Deferrable != nil && *c.Deferrable {
actions = append(actions, "deferrable")
}
if c.InitiallyDeferred != nil && *c.InitiallyDeferred {
actions = append(actions, "initially deferred")
}
refLine := fmt.Sprintf("Ref: %s %s %s", fromRef, relationship, toRef)
if len(actions) > 0 {
refLine += fmt.Sprintf(" [%s]", strings.Join(actions, ", "))
+28
View File
@@ -171,3 +171,31 @@ func TestWriter_WriteDatabase_OneToOneRelationship(t *testing.T) {
assert.Contains(t, output, "Ref: public.profiles.user_id - public.users.id")
}
func TestWriter_WriteDatabase_ForeignKeyDeferral(t *testing.T) {
yes := true
db := models.InitDatabase("test_db")
schema := models.InitSchema("public")
users := models.InitTable("users", "public")
users.Columns["id"] = models.InitColumn("id", "users", "public")
users.Columns["id"].Type = "bigint"
posts := models.InitTable("posts", "public")
posts.Columns["user_id"] = models.InitColumn("user_id", "posts", "public")
posts.Columns["user_id"].Type = "bigint"
fk := models.InitConstraint("fk_posts_user", models.ForeignKeyConstraint)
fk.Table, fk.Schema = "posts", "public"
fk.Columns = []string{"user_id"}
fk.ReferencedTable, fk.ReferencedSchema = "users", "public"
fk.ReferencedColumns = []string{"id"}
fk.Deferrable = &yes
fk.InitiallyDeferred = &yes
posts.Constraints[fk.Name] = fk
schema.Tables = append(schema.Tables, users, posts)
db.Schemas = append(db.Schemas, schema)
outputPath := filepath.Join(t.TempDir(), "test.dbml")
assert.NoError(t, NewWriter(&writers.WriterOptions{OutputPath: outputPath}).WriteDatabase(db))
content, err := os.ReadFile(outputPath)
assert.NoError(t, err)
assert.Contains(t, string(content), "Ref: public.posts.user_id > public.users.id [deferrable, initially deferred]")
}
+239
View File
@@ -0,0 +1,239 @@
package writers
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"strings"
"time"
)
// ExecutionReport describes an entire invocation, including operations never attempted.
// ExecutedStatements counts operations still applied, excluding confirmed rollbacks.
type ExecutionReport struct {
Dialect string `json:"dialect,omitempty"`
TotalStatements int `json:"total_statements"`
ExecutedStatements int `json:"executed_statements"`
FailedStatements int `json:"failed_statements"`
SkippedStatements int `json:"skipped_statements"`
RolledBackStatements int `json:"rolled_back_statements"`
Outcome string `json:"outcome"`
Operations []ExecutionOperation `json:"operations"`
Schemas []SchemaReport `json:"schemas"`
Errors []ExecutionError `json:"errors,omitempty"`
StartTime string `json:"start_time"`
EndTime string `json:"end_time"`
}
type SchemaReport struct {
Name string `json:"name"`
Tables []TableReport `json:"tables"`
}
type TableReport struct {
Name string `json:"name"`
Created bool `json:"created"`
Error string `json:"error,omitempty"`
}
type ExecutionError struct {
StatementNumber int `json:"statement_number"`
Statement string `json:"statement"`
Error string `json:"error"`
}
// ExecutionOperation is one generated statement or supplied script, in execution order.
// A failed raw script may have committed some of its own statements; see Outcome.
type ExecutionOperation struct {
StatementNumber int `json:"statement_number"`
Name string `json:"name,omitempty"`
Statement string `json:"statement"`
Status string `json:"status"` // applied, executed (raw script), failed, skipped, rolled_back, unknown
Error string `json:"error,omitempty"`
Skip bool `json:"-"`
StopOnFailure bool `json:"-"`
}
// ExecutionFailure is returned even when continuation was explicitly requested.
type ExecutionFailure struct{ Report *ExecutionReport }
func (e *ExecutionFailure) Error() string {
message := fmt.Sprintf("%s execution failed: %d failed, %d executed, %d skipped, %d rolled back",
e.Report.Dialect, e.Report.FailedStatements, e.Report.ExecutedStatements,
e.Report.SkippedStatements, e.Report.RolledBackStatements)
if len(e.Report.Errors) > 0 {
first := e.Report.Errors[0]
message += fmt.Sprintf("; statement %d: %s", first.StatementNumber, first.Error)
}
return message
}
// ExecutionBackend operates on one connection. Begin switches Exec to a transaction.
// Atomic execution stops at the first failure and rolls back the whole invocation.
type ExecutionBackend struct {
Exec func(context.Context, int, string) error
Begin func(context.Context) error
Commit func(context.Context) error
Rollback func(context.Context) error
// ScriptManaged means arbitrary scripts can commit/rollback internally.
ScriptManaged bool
}
func (o *WriterOptions) ShouldContinue() bool {
legacy, _ := o.Metadata["ignore_errors"].(bool)
return o.ContinueOnError || legacy
}
// RunExecution applies a plan, records every outcome, and publishes metadata and reports.
func RunExecution(ctx context.Context, opts *WriterOptions, report *ExecutionReport, operations []ExecutionOperation, backend ExecutionBackend) error {
report.TotalStatements = len(operations)
report.Operations = append([]ExecutionOperation(nil), operations...)
report.ExecutedStatements, report.FailedStatements, report.SkippedStatements, report.RolledBackStatements = 0, 0, 0, 0
report.Errors = nil
report.StartTime = time.Now().Format(time.RFC3339Nano)
report.Outcome = "not_started"
for i := range report.Operations {
report.Operations[i].StatementNumber = i + 1
report.Operations[i].Status = "skipped"
}
var runErr error
atomic := opts.AtomicExecution
if atomic && opts.ShouldContinue() {
runErr = errors.New("atomic execution cannot be combined with continue-on-error")
} else if atomic && (backend.Begin == nil || backend.Commit == nil || backend.Rollback == nil) {
runErr = errors.New("atomic execution is not supported by this executor")
} else if atomic {
runErr = backend.Begin(ctx)
}
if runErr == nil {
report.Outcome = "nontransactional"
if backend.ScriptManaged {
report.Outcome = "script_managed"
}
stopped := false
for i := range report.Operations {
op := &report.Operations[i]
if stopped || op.Skip || strings.TrimSpace(op.Statement) == "" {
continue
}
if err := backend.Exec(ctx, i, op.Statement); err != nil {
runErr = errors.Join(runErr, err)
op.Status, op.Error = "failed", err.Error()
report.FailedStatements++
report.Errors = append(report.Errors, ExecutionError{i + 1, op.Statement, err.Error()})
stopped = op.StopOnFailure || !opts.ShouldContinue()
} else {
op.Status = "applied"
if backend.ScriptManaged {
op.Status = "executed"
}
report.ExecutedStatements++
}
}
if atomic {
if report.FailedStatements > 0 {
// Cleanup uses a bounded context independent of caller cancellation.
cleanup, cancel := context.WithTimeout(context.Background(), 5*time.Second)
rollbackErr := backend.Rollback(cleanup)
cancel()
if rollbackErr == nil {
report.Outcome = "rolled_back"
for i := range report.Operations {
if report.Operations[i].Status == "applied" {
report.Operations[i].Status = "rolled_back"
report.RolledBackStatements++
}
}
report.ExecutedStatements = 0
} else {
report.Outcome = "unknown"
runErr = errors.Join(runErr, fmt.Errorf("rollback failed: %w", rollbackErr))
}
} else {
runErr = backend.Commit(ctx)
if runErr == nil {
report.Outcome = "committed"
} else {
report.Outcome = "unknown"
}
}
if report.Outcome == "rolled_back" || report.Outcome == "unknown" {
for i := range report.Schemas {
for j := range report.Schemas[i].Tables {
report.Schemas[i].Tables[j].Created = false
}
}
}
if report.Outcome == "unknown" {
for i := range report.Operations {
if report.Operations[i].Status == "applied" {
report.Operations[i].Status = "unknown"
}
}
report.ExecutedStatements = 0
}
}
}
for _, op := range report.Operations {
if op.Status == "skipped" {
report.SkippedStatements++
}
}
report.EndTime = time.Now().Format(time.RFC3339Nano)
if report.FailedStatements > 0 {
runErr = errors.Join(&ExecutionFailure{report}, runErr)
}
return FinishExecution(opts, report, runErr)
}
// FinishExecution publishes legacy counters and a typed report even on failure.
func FinishExecution(opts *WriterOptions, report *ExecutionReport, runErr error) error {
if opts.Metadata == nil {
opts.Metadata = map[string]interface{}{}
}
opts.Metadata["execution_report"] = report
opts.Metadata["execution_total"] = report.TotalStatements
opts.Metadata["execution_success"] = report.ExecutedStatements
opts.Metadata["execution_failed"] = report.FailedStatements
opts.Metadata["execution_skipped"] = report.SkippedStatements
opts.Metadata["execution_rolled_back"] = report.RolledBackStatements
if path, _ := opts.Metadata["report_path"].(string); path != "" {
if err := WriteExecutionReport(path, report); err != nil {
runErr = errors.Join(runErr, fmt.Errorf("failed to write requested execution report: %w", err))
}
}
return runErr
}
func WriteExecutionReport(path string, report *ExecutionReport) error {
file, err := os.Create(path)
if err != nil {
return fmt.Errorf("failed to create report file: %w", err)
}
encoder := json.NewEncoder(file)
encoder.SetIndent("", " ")
return errors.Join(encoder.Encode(report), file.Close())
}
// StatementOperations creates a plan from the writers' structured statement lists.
func StatementOperations(statements []string) []ExecutionOperation {
ops := make([]ExecutionOperation, len(statements))
for i, stmt := range statements {
ops[i] = ExecutionOperation{Statement: stmt, StopOnFailure: strings.HasPrefix(strings.TrimSpace(stmt), "DO $relspec_preflight"), Skip: strings.TrimSpace(stmt) == "" || strings.HasPrefix(strings.TrimSpace(stmt), "--")}
}
return ops
}
// FailExecution records an invocation that failed before issuing any SQL.
func FailExecution(opts *WriterOptions, dialect string, operations []ExecutionOperation, cause error) error {
report := &ExecutionReport{Dialect: dialect, TotalStatements: len(operations), SkippedStatements: len(operations), Outcome: "not_started", Operations: append([]ExecutionOperation(nil), operations...)}
report.StartTime = time.Now().Format(time.RFC3339Nano)
report.EndTime = report.StartTime
for i := range report.Operations {
report.Operations[i].StatementNumber = i + 1
report.Operations[i].Status = "skipped"
}
return FinishExecution(opts, report, cause)
}
+96
View File
@@ -0,0 +1,96 @@
package writers
import (
"context"
"errors"
"path/filepath"
"strings"
"testing"
)
func TestExecutionOutcomes(t *testing.T) {
for _, atomic := range []bool{false, true} {
for _, continuation := range []bool{false, true} {
t.Run(strings.Join([]string{map[bool]string{false: "independent", true: "atomic"}[atomic], map[bool]string{false: "stop", true: "continue"}[continuation]}, "/"), func(t *testing.T) {
attempted, rollbacks := 0, 0
cause := errors.New("injected failure")
backend := ExecutionBackend{
Exec: func(_ context.Context, i int, _ string) error {
attempted++
if i == 1 {
return cause
}
return nil
},
Begin: func(context.Context) error { return nil }, Commit: func(context.Context) error { t.Fatal("must not commit"); return nil },
Rollback: func(context.Context) error { rollbacks++; return nil },
}
opts := &WriterOptions{ContinueOnError: continuation, AtomicExecution: atomic}
report := &ExecutionReport{Dialect: "test"}
err := RunExecution(context.Background(), opts, report, StatementOperations([]string{"one", "bad", "three"}), backend)
if atomic && continuation {
if err == nil || attempted != 0 || report.SkippedStatements != 3 {
t.Fatalf("invalid options must not execute: %+v %v", report, err)
}
return
}
var failure *ExecutionFailure
if !errors.As(err, &failure) || !errors.Is(err, cause) {
t.Fatalf("must preserve typed report and cause: %v", err)
}
if report.FailedStatements != 1 || report.TotalStatements != 3 || report.Errors[0].StatementNumber != 2 {
t.Fatalf("failure report: %+v", report)
}
if atomic {
if rollbacks != 1 || report.ExecutedStatements != 0 || report.RolledBackStatements != 1 || report.Operations[0].Status != "rolled_back" || report.Outcome != "rolled_back" {
t.Fatalf("rollback report: %+v", report)
}
} else if continuation {
if attempted != 3 || report.ExecutedStatements != 2 || report.SkippedStatements != 0 {
t.Fatalf("continuation report: %+v", report)
}
} else if attempted != 2 || report.ExecutedStatements != 1 || report.SkippedStatements != 1 {
t.Fatalf("stop report: %+v", report)
}
if opts.Metadata["execution_report"] != report {
t.Fatal("typed results missing")
}
})
}
}
}
func TestExecutionUncertainFinalization(t *testing.T) {
for _, failStatement := range []bool{false, true} {
t.Run(map[bool]string{false: "commit", true: "rollback"}[failStatement], func(t *testing.T) {
finalErr := errors.New("connection lost during finalization")
backend := ExecutionBackend{
Begin: func(context.Context) error { return nil },
Exec: func(_ context.Context, i int, _ string) error {
if failStatement && i == 1 {
return errors.New("statement failed")
}
return nil
},
Commit: func(context.Context) error { return finalErr }, Rollback: func(context.Context) error { return finalErr },
}
report := &ExecutionReport{}
err := RunExecution(context.Background(), &WriterOptions{AtomicExecution: true}, report, StatementOperations([]string{"one", "two"}), backend)
if !errors.Is(err, finalErr) || report.Outcome != "unknown" || report.Operations[0].Status != "unknown" || report.RolledBackStatements != 0 || report.ExecutedStatements != 0 {
t.Fatalf("uncertain outcome misreported: %+v %v", report, err)
}
})
}
}
func TestExecutionReportFailureAndFatalPreflight(t *testing.T) {
opts := &WriterOptions{ContinueOnError: true, Metadata: map[string]interface{}{"report_path": filepath.Join(t.TempDir(), "missing", "report.json")}}
report := &ExecutionReport{}
calls := 0
ops := []ExecutionOperation{{Statement: "preflight", StopOnFailure: true}, {Statement: "must not run"}}
err := RunExecution(context.Background(), opts, report, ops, ExecutionBackend{Exec: func(context.Context, int, string) error { calls++; return errors.New("invalid data") }})
var failure *ExecutionFailure
if !errors.As(err, &failure) || !strings.Contains(err.Error(), "requested execution report") || calls != 1 || report.SkippedStatements != 1 {
t.Fatalf("must retain both failures and stop: %+v %v", report, err)
}
}
+16 -16
View File
@@ -502,30 +502,30 @@ func (w *Writer) executeDatabaseSQL(db *models.Database, connString string) erro
// Connect to database
dbConn, err := sql.Open("mssql", connString)
if err != nil {
return fmt.Errorf("failed to connect to database: %w", err)
return writers.FailExecution(w.options, "SQL Server", writers.StatementOperations(statements), fmt.Errorf("failed to connect to database: %w", err))
}
defer dbConn.Close()
ctx := context.Background()
if err = dbConn.PingContext(ctx); err != nil {
return fmt.Errorf("failed to ping database: %w", err)
return writers.FailExecution(w.options, "SQL Server", writers.StatementOperations(statements), fmt.Errorf("failed to ping database: %w", err))
}
// Execute statements
executedCount := 0
for i, stmt := range statements {
fmt.Fprintf(os.Stderr, "Executing statement %d/%d...\n", i+1, len(statements))
if _, execErr := dbConn.ExecContext(ctx, stmt); execErr != nil {
fmt.Fprintf(os.Stderr, "⚠ Warning: Statement failed: %v\n", execErr)
continue
}
executedCount++
var tx *sql.Tx
backend := writers.ExecutionBackend{
Exec: func(ctx context.Context, _ int, stmt string) error {
if tx != nil {
_, err := tx.ExecContext(ctx, stmt)
return err
}
_, err := dbConn.ExecContext(ctx, stmt)
return err
},
Begin: func(ctx context.Context) error { var err error; tx, err = dbConn.BeginTx(ctx, nil); return err },
Commit: func(context.Context) error { return tx.Commit() },
Rollback: func(context.Context) error { return tx.Rollback() },
}
fmt.Fprintf(os.Stderr, "✓ Successfully executed %d statements\n", executedCount)
return nil
return writers.RunExecution(ctx, w.options, &writers.ExecutionReport{Dialect: "SQL Server"}, writers.StatementOperations(statements), backend)
}
// generateStatements renders the same script WriteDatabase would produce and
+69
View File
@@ -0,0 +1,69 @@
package mssql
import (
"database/sql"
"errors"
"fmt"
"os"
"testing"
"time"
"git.warky.dev/wdevs/relspecgo/pkg/models"
"git.warky.dev/wdevs/relspecgo/pkg/writers"
)
func TestLive_ExecutionFailureModes(t *testing.T) {
connection := os.Getenv("RELSPEC_TEST_MSSQL_CONN")
if connection == "" {
t.Skip("RELSPEC_TEST_MSSQL_CONN not set")
}
conn, err := sql.Open("mssql", connection)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
for _, mode := range []string{"stop", "continue", "atomic"} {
t.Run(mode, func(t *testing.T) {
prefix := fmt.Sprintf("audit_%d_", time.Now().UnixNano())
db := models.InitDatabase("audit")
schema := models.InitSchema("dbo")
for _, name := range []string{"first", "bad", "last"} {
table := models.InitTable(prefix+name, "dbo")
col := models.InitColumn("id", table.Name, "dbo")
col.Type = "integer"
if name == "bad" {
duplicate := models.InitColumn("id", table.Name, "dbo")
duplicate.Type = "integer"
table.Columns["duplicate"] = duplicate
}
table.Columns[col.Name] = col
schema.Tables = append(schema.Tables, table)
}
db.Schemas = append(db.Schemas, schema)
t.Cleanup(func() {
for _, name := range []string{"first", "bad", "last"} {
_, _ = conn.Exec("DROP TABLE IF EXISTS dbo." + prefix + name)
}
})
opts := &writers.WriterOptions{ContinueOnError: mode == "continue", AtomicExecution: mode == "atomic", Metadata: map[string]interface{}{"connection_string": connection}}
err := NewWriter(opts).WriteDatabase(db)
var failure *writers.ExecutionFailure
if !errors.As(err, &failure) || failure.Report.FailedStatements == 0 {
t.Fatalf("expected structured error: %v", err)
}
for _, name := range []string{"first", "last"} {
var count int
if err := conn.QueryRow("SELECT count(*) FROM sys.tables WHERE name=?", prefix+name).Scan(&count); err != nil {
t.Fatal(err)
}
want := mode != "atomic" && (name == "first" || mode == "continue")
if (count == 1) != want {
t.Fatalf("%s exists=%v want=%v", name, count == 1, want)
}
}
if mode == "atomic" && (failure.Report.Outcome != "rolled_back" || failure.Report.RolledBackStatements == 0 || failure.Report.ExecutedStatements != 0) {
t.Fatalf("incorrect rollback report: %+v", failure.Report)
}
})
}
}
+145
View File
@@ -0,0 +1,145 @@
package mysql
import (
"os"
"path/filepath"
"strings"
"testing"
"git.warky.dev/wdevs/relspecgo/pkg/models"
"git.warky.dev/wdevs/relspecgo/pkg/transform"
"git.warky.dev/wdevs/relspecgo/pkg/writers"
)
func TestStrictConversionPreflightsAllObjects(t *testing.T) {
db := shopDB()
unsupported := models.InitTable("unsupported", "shop")
col := models.InitColumn("opaque", "unsupported", "shop")
col.Type = "unknown_type"
unsupported.Columns[col.Name] = col
index := models.InitIndex("partial", "unsupported", "shop")
index.Columns = []string{col.Name}
index.Where = "opaque IS NOT NULL"
index.Include = []string{"other"}
unsupported.Indexes[index.Name] = index
db.Schemas[0].Tables = append(db.Schemas[0].Tables, unsupported)
out := filepath.Join(t.TempDir(), "existing.sql")
if err := os.WriteFile(out, []byte("keep existing contents"), 0o600); err != nil {
t.Fatal(err)
}
opts := &writers.WriterOptions{StrictConversion: true, OutputPath: out}
err := NewWriter(opts).WriteDatabase(db)
for _, want := range []string{"lossy MySQL conversion rejected", "unknown type", "partial predicate", "INCLUDE"} {
if err == nil || !strings.Contains(err.Error(), want) {
t.Fatalf("expected %s: %v", want, err)
}
}
bytes, err := os.ReadFile(out)
if err != nil || string(bytes) != "keep existing contents" {
t.Fatalf("destination changed: %q %v", bytes, err)
}
opts.Metadata = map[string]any{"connection_string": "invalid DSN"}
if err := NewWriter(opts).WriteDatabase(db); err == nil || !strings.Contains(err.Error(), "lossy MySQL") {
t.Fatalf("connected before strict validation: %v", err)
}
opts.StrictConversion = false
opts.Metadata = nil
var warnings []string
opts.Warning = func(s string) { warnings = append(warnings, s) }
if err := NewWriter(opts).WriteDatabase(db); err != nil {
t.Fatal(err)
}
if len(warnings) != 3 {
t.Fatalf("warnings: %v", warnings)
}
}
func TestUnsupportedModelFeaturesDiagnosed(t *testing.T) {
db := shopDB()
s := db.Schemas[0]
table := s.Tables[0]
s.Metadata["mysql_unsupported"] = []any{"trigger shop.audit", "partitioned table shop.archive"}
c := table.Columns["id"]
c.Type = "timestamptz"
c.Identity = true
c.Metadata["mysql_srs_id"] = 4326
check := models.InitConstraint("deferred", models.CheckConstraint)
check.Expression = "id > 0"
yes := true
check.Deferrable = &yes
table.Constraints[check.Name] = check
i := models.InitIndex("gin", "users", "shop")
i.Type = "gin"
i.Columns = []string{"id"}
table.Indexes[i.Name] = i
_, err := NewWriter(&writers.WriterOptions{StrictConversion: true}).Statements(db)
for _, want := range []string{"trigger shop.audit", "partitioned table", "time zone", "identity", "mysql_srs_id", "deferrable", "unsupported index type gin"} {
if err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("missing diagnostic %q: %v", want, err)
}
}
}
func TestCanonicalDimensionsAndNativeEdit(t *testing.T) {
db := shopDB()
c := db.Schemas[0].Tables[0].Columns["age"]
c.Type = "decimal"
c.Precision = 12
c.Scale = 4
if got := columnType(c); got != "DECIMAL(12,4)" {
t.Fatal(got)
}
c.Type = "uint64"
c.Precision = 0
c.Scale = 0
if got := columnType(c); got != "BIGINT UNSIGNED" {
t.Fatal(got)
}
c.Metadata["mysql_type"] = "bigint unsigned"
c.Metadata["mysql_type_signature"] = signature(c)
c.Type = "int64"
_, err := NewWriter(&writers.WriterOptions{StrictConversion: true}).Statements(db)
if err == nil || !strings.Contains(err.Error(), "override native declaration") {
t.Fatalf("native metadata silently overrode edit: %v", err)
}
if got := columnType(c); got != "BIGINT" {
t.Fatal(got)
}
}
func TestLowercaseIndexPartsPreserveSQL(t *testing.T) {
db := shopDB()
table := db.Schemas[0].Tables[0]
c := models.InitColumn("Mixed", "users", "shop")
c.Type = "string"
table.Columns[c.Name] = c
i := models.InitIndex("MixedIndex", "users", "shop")
i.KeyParts = []models.IndexPart{{Name: "Mixed", Length: 8, Descending: true}, {Expression: "lower(`Mixed`)"}}
table.Indexes[i.Name] = i
model, err := transform.LowercaseIdentifiers(db)
if err != nil {
t.Fatal(err)
}
parts := model.Schemas[0].Tables[0].Indexes["mixedindex"].KeyParts
if parts[0].Name != "mixed" || parts[1].Expression != "lower(`Mixed`)" || i.KeyParts[0].Name != "Mixed" {
t.Fatalf("lowercase parts: %+v", parts)
}
}
func TestLowercasePrimaryIndexIsNotDuplicated(t *testing.T) {
db := shopDB()
table := db.Schemas[0].Tables[0]
i := models.InitIndex("PRIMARY", table.Name, table.Schema)
i.Unique = true
i.Type = "btree"
i.Columns = []string{"id"}
i.KeyParts = []models.IndexPart{{Name: "id", Descending: true}}
table.Indexes[i.Name] = i
statements, err := NewWriter(&writers.WriterOptions{StrictConversion: true, LowercaseIdentifiers: true}).Statements(db)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(statements[0], "PRIMARY KEY (`id` DESC)") || strings.Contains(statements[0], "KEY `primary`") {
t.Fatalf("primary key duplicated or attributes lost: %s", statements[0])
}
}
+184
View File
@@ -0,0 +1,184 @@
package mysql
import (
"fmt"
"sort"
"strings"
"git.warky.dev/wdevs/relspecgo/pkg/mariadb"
"git.warky.dev/wdevs/relspecgo/pkg/models"
)
// Diagnostics cover modeled schema semantics and features discovered by the
// MySQL reader. They do not certify arbitrary SQL expressions or catalog grants.
func conversionDiagnostics(db *models.Database) []string {
var out []string
warn := func(path, feature string) { out = append(out, path+": "+feature) }
unsupported := func(m map[string]any) {
switch values := m["mysql_unsupported"].(type) {
case []string:
for _, v := range values {
warn(v, "unsupported object")
}
case []any:
for _, v := range values {
warn(fmt.Sprint(v), "unsupported object")
}
}
}
metadata := func(path string, m map[string]any, allowed string) {
if len(models.GetDirectives(m)) > 0 {
warn(path, "dialect directives are not emitted")
}
for _, key := range keys(m) {
if strings.HasPrefix(key, "mysql_") && !strings.Contains(" "+allowed+" ", " "+key+" ") {
warn(path, "unsupported metadata "+key+": "+fmt.Sprint(m[key]))
}
}
}
metadata("database", db.Metadata, "mysql_unsupported mysql_sql_mode")
modeWarning := func(path string, m map[string]any) {
if mode := unsupportedMode(metaString(m, "mysql_sql_mode")); mode != "" {
warn(path, "unsupported source SQL mode "+mode+"; generated SQL requires standard quotes and backslash escaping")
}
}
modeWarning("database", db.Metadata)
unsupported(db.Metadata)
if len(db.Domains) > 0 {
warn("database", "domains are not emitted")
}
for _, s := range db.Schemas {
path := s.Name
metadata(path, s.Metadata, "mysql_unsupported mysql_sql_mode")
modeWarning(path, s.Metadata)
unsupported(s.Metadata)
if len(s.Views) > 0 {
warn(path, "views are not emitted")
}
if len(s.Sequences) > 0 {
warn(path, "standalone sequences are not emitted")
}
if len(s.Enums) > 0 {
warn(path, "standalone enum types are not emitted; use native column enum types")
}
if len(s.Scripts) > 0 {
warn(path, "scripts are not emitted")
}
if s.Owner != "" || len(s.Permissions) > 0 {
warn(path, "ownership and permissions are not emitted")
}
for _, t := range s.Tables {
tp := path + "." + t.Name
modeWarning(tp, t.Metadata)
if t.Tablespace != "" {
warn(tp, "tablespace is not emitted")
}
metadata(tp, t.Metadata, "mysql_engine mysql_collation mysql_create_options mysql_auto_increment mysql_sql_mode")
if options := metaString(t.Metadata, "mysql_create_options"); options != "" {
warn(tp, "table CREATE_OPTIONS are not emitted: "+options)
}
for _, name := range keys(t.Columns) {
c := t.Columns[name]
cp := tp + "." + c.Name
metadata(cp, c.Metadata, "mysql_type mysql_type_signature mysql_charset mysql_datetime_precision mysql_default_expression mysql_default_sql mysql_default_value mysql_generation_storage mysql_invisible mysql_on_update")
if _, known := mariadb.CanonicalToMariaDBTypes[strings.ToLower(c.Type)]; !known && metaString(c.Metadata, "mysql_type") == "" {
warn(cp, "unknown type "+c.Type+" maps to "+columnType(c))
}
if native := metaString(c.Metadata, "mysql_type"); native != "" && metaString(c.Metadata, "mysql_type_signature") != signature(c) {
warn(cp, "edited canonical type/dimensions override native declaration "+native)
}
if c.Type == "timestamptz" {
warn(cp, "time zone semantics are not supported")
}
if c.Type == "jsonb" {
warn(cp, "jsonb storage semantics become JSON")
}
if c.Identity {
warn(cp, "identity semantics are not emitted; use auto_increment")
}
if c.Generated && c.GenerationExpression == "" {
warn(cp, "generated column has no expression")
}
if !c.Generated && c.GenerationExpression != "" {
warn(cp, "generation expression requires generated=true")
}
if c.Generated && (c.Default != nil || c.AutoIncrement) {
warn(cp, "generated column cannot preserve a default or auto_increment")
}
if storage := metaString(c.Metadata, "mysql_generation_storage"); storage != "" && storage != "STORED" && storage != "VIRTUAL" {
warn(cp, "unsupported generation storage "+storage)
}
}
for _, name := range keys(t.Constraints) {
c := t.Constraints[name]
cp := tp + " constraint " + c.Name
metadata(cp, c.Metadata, "mysql_enforced")
if c.Deferrable != nil && *c.Deferrable || c.InitiallyDeferred != nil && *c.InitiallyDeferred {
warn(cp, "deferrable constraints are not supported")
}
switch c.Type {
case models.PrimaryKeyConstraint, models.UniqueConstraint:
case models.CheckConstraint:
if c.Expression == "" {
warn(cp, "check constraint has no expression")
}
case models.ForeignKeyConstraint:
engine := metaString(t.Metadata, "mysql_engine")
if engine != "" && !strings.EqualFold(engine, "InnoDB") && !strings.EqualFold(engine, "NDB") {
warn(cp, "foreign keys are not supported by engine "+engine)
}
for _, action := range []string{c.OnDelete, c.OnUpdate} {
switch strings.ToUpper(action) {
case "", "CASCADE", "SET NULL", "RESTRICT", "NO ACTION":
default:
warn(cp, "unsupported referential action is not emitted: "+action)
}
}
default:
warn(cp, "unsupported constraint type "+string(c.Type))
}
}
for _, name := range keys(t.Indexes) {
i := t.Indexes[name]
ip := tp + " index " + i.Name
metadata(ip, i.Metadata, "mysql_visible")
if i.Where != "" {
warn(ip, "partial predicate is not emitted")
}
if len(i.Include) > 0 {
warn(ip, "INCLUDE columns are not emitted")
}
if i.Concurrent {
warn(ip, "concurrent creation is not supported")
}
if strings.EqualFold(i.Type, "hash") && !strings.EqualFold(metaString(t.Metadata, "mysql_engine"), "MEMORY") {
warn(ip, "HASH index requires a compatible engine; InnoDB converts it to BTREE")
}
switch strings.ToUpper(i.Type) {
case "", "BTREE", "HASH", "FULLTEXT", "SPATIAL":
default:
warn(ip, "unsupported index type "+i.Type+"; index is not emitted")
}
for _, p := range i.KeyParts {
if (p.Name == "") == (p.Expression == "") {
warn(ip, "key part requires exactly one name or expression")
}
}
}
for _, name := range keys(t.Relationships) {
r := t.Relationships[name]
if t.Constraints[r.ForeignKey] == nil {
warn(tp+" relationship "+r.Name, "no backing foreign key constraint; relationship is not emitted")
}
}
}
}
sort.Strings(out)
unique := out[:0]
for _, d := range out {
if len(unique) == 0 || unique[len(unique)-1] != d {
unique = append(unique, d)
}
}
return unique
}
+142
View File
@@ -0,0 +1,142 @@
package mysql_test
import (
"database/sql"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
driver "github.com/go-sql-driver/mysql"
"git.warky.dev/wdevs/relspecgo/pkg/models"
"git.warky.dev/wdevs/relspecgo/pkg/readers"
rmysql "git.warky.dev/wdevs/relspecgo/pkg/readers/mysql"
"git.warky.dev/wdevs/relspecgo/pkg/writers"
wmysql "git.warky.dev/wdevs/relspecgo/pkg/writers/mysql"
)
func TestLiveUnsupportedObjectsDiagnosed(t *testing.T) {
conn := os.Getenv("RELSPEC_TEST_MYSQL_CONN")
if conn == "" {
t.Skip("RELSPEC_TEST_MYSQL_CONN is not set")
}
db, err := sql.Open("mysql", conn)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { db.Close() })
prefix := fmt.Sprintf("unsupported_%d", time.Now().UnixNano())
table, view, trigger := prefix+"_table", prefix+"_view", prefix+"_trigger"
t.Cleanup(func() {
for _, stmt := range []string{"DROP VIEW IF EXISTS " + view, "DROP TRIGGER IF EXISTS " + trigger, "DROP TABLE IF EXISTS " + table} {
if _, err := db.Exec(stmt); err != nil {
t.Error(err)
}
}
})
for _, stmt := range []string{
"CREATE TABLE " + table + " (id INT PRIMARY KEY) PARTITION BY HASH(id) PARTITIONS 2",
"CREATE VIEW " + view + " AS SELECT id FROM " + table,
"CREATE TRIGGER " + trigger + " BEFORE INSERT ON " + table + " FOR EACH ROW SET NEW.id = NEW.id + 1",
} {
if _, err := db.Exec(stmt); err != nil {
t.Fatalf("%s: %v", stmt, err)
}
}
model, err := rmysql.NewReader(&readers.ReaderOptions{ConnectionString: conn}).ReadDatabase()
if err != nil {
t.Fatal(err)
}
path := filepath.Join(t.TempDir(), "schema.sql")
if err := os.WriteFile(path, []byte("original"), 0o600); err != nil {
t.Fatal(err)
}
err = wmysql.NewWriter(&writers.WriterOptions{StrictConversion: true, OutputPath: path}).WriteDatabase(model)
for _, want := range []string{"partitioned table", view, trigger} {
if err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("missing diagnostic %s: %v", want, err)
}
}
bytes, readErr := os.ReadFile(path)
if readErr != nil || string(bytes) != "original" {
t.Fatalf("strict failure touched file: %s %v", bytes, readErr)
}
// Schema-filtered/library writes retain discovery diagnostics as well.
err = wmysql.NewWriter(&writers.WriterOptions{StrictConversion: true}).WriteSchema(model.Schemas[0])
if err == nil || !strings.Contains(err.Error(), trigger) {
t.Fatalf("filtered schema lost diagnostics: %v", err)
}
var warnings []string
_, err = wmysql.NewWriter(&writers.WriterOptions{Warning: func(s string) { warnings = append(warnings, s) }}).Statements(model)
if err != nil {
t.Fatal(err)
}
for _, want := range []string{view, trigger, table} {
found := false
for _, warning := range warnings {
if strings.Contains(warning, want) {
found = true
}
}
if !found {
t.Errorf("no warning for %s: %v", want, warnings)
}
}
}
func TestLiveUnsupportedSQLModeFailsBeforeDDL(t *testing.T) {
conn := os.Getenv("RELSPEC_TEST_MYSQL_CONN")
if conn == "" {
t.Skip("RELSPEC_TEST_MYSQL_CONN is not set")
}
for _, mode := range []string{"NO_BACKSLASH_ESCAPES", "ANSI_QUOTES"} {
t.Run(mode, func(t *testing.T) {
cfg, err := driver.ParseDSN(conn)
if err != nil {
t.Fatal(err)
}
if cfg.Params == nil {
cfg.Params = map[string]string{}
}
cfg.Params["sql_mode"] = "'" + mode + "'"
modeConn := cfg.FormatDSN()
model, err := rmysql.NewReader(&readers.ReaderOptions{ConnectionString: modeConn}).ReadDatabase()
if err != nil {
t.Fatal(err)
}
if _, err := wmysql.NewWriter(&writers.WriterOptions{StrictConversion: true}).Statements(model); err == nil || !strings.Contains(err.Error(), mode) {
t.Fatalf("source mode not diagnosed: %v", err)
}
db, err := sql.Open("mysql", conn)
if err != nil {
t.Fatal(err)
}
defer db.Close()
name := fmt.Sprintf("mode_%d", time.Now().UnixNano())
defer db.Exec("DROP TABLE IF EXISTS " + name)
target := models.InitDatabase(cfg.DBName)
s := models.InitSchema(cfg.DBName)
target.Schemas = append(target.Schemas, s)
table := models.InitTable(name, s.Name)
s.Tables = append(s.Tables, table)
c := models.InitColumn("id", name, s.Name)
c.Type = "int"
table.Columns[c.Name] = c
err = wmysql.NewWriter(&writers.WriterOptions{StrictConversion: true, Metadata: map[string]any{"connection_string": modeConn}}).WriteDatabase(target)
if err == nil || !strings.Contains(err.Error(), mode) {
t.Fatalf("target mode not rejected: %v", err)
}
var count int
if err := db.QueryRow("SELECT COUNT(*) FROM information_schema.TABLES WHERE TABLE_SCHEMA=? AND TABLE_NAME=?", cfg.DBName, name).Scan(&count); err != nil {
t.Fatal(err)
}
if count != 0 {
t.Fatal("mode rejection occurred after DDL")
}
})
}
}
+244
View File
@@ -0,0 +1,244 @@
package mysql_test
import (
"database/sql"
"encoding/json"
"fmt"
"os"
"reflect"
"sort"
"strings"
"testing"
"time"
"gopkg.in/yaml.v3"
driver "github.com/go-sql-driver/mysql"
"git.warky.dev/wdevs/relspecgo/pkg/models"
"git.warky.dev/wdevs/relspecgo/pkg/readers"
rmysql "git.warky.dev/wdevs/relspecgo/pkg/readers/mysql"
"git.warky.dev/wdevs/relspecgo/pkg/writers"
wmysql "git.warky.dev/wdevs/relspecgo/pkg/writers/mysql"
)
func TestLiveSchemaRoundTrip(t *testing.T) {
conn := os.Getenv("RELSPEC_TEST_MYSQL_CONN")
if conn == "" {
t.Skip("RELSPEC_TEST_MYSQL_CONN is not set")
}
db, err := sql.Open("mysql", conn)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { db.Close() })
prefix := fmt.Sprintf("rt_%d", time.Now().UnixNano())
parent, child := prefix+"_zparent", prefix+"_achild"
q := func(n string) string { return "`" + strings.ReplaceAll(n, "`", "``") + "`" }
exec := func(stmt string) {
t.Helper()
if _, err := db.Exec(stmt); err != nil {
t.Fatalf("%v\n%s", err, stmt)
}
}
t.Cleanup(func() {
for _, name := range []string{child + "_copy", child, parent + "_copy", parent} {
if _, err := db.Exec("DROP TABLE IF EXISTS " + q(name)); err != nil {
t.Errorf("cleanup %s: %v", name, err)
}
}
})
exec("CREATE TABLE " + q(parent) + ` (
id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
code CHAR(3) NOT NULL,
email VARCHAR(100) NOT NULL,
label VARCHAR(64) DEFAULT 'don''t (change)' COMMENT 'column''s comment',
amount DECIMAL(12,4) NOT NULL DEFAULT 3.2500,
medium MEDIUMINT UNSIGNED,
flags BIT(8) DEFAULT b'00001010',
status ENUM('one','two','don''t, split') DEFAULT 'don''t, split',
choices SET('a','b') DEFAULT 'a,b',
stamp TIMESTAMP(6) NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6),
duration TIME(4),
bytes VARBINARY(16),
binary_def VARBINARY(8) DEFAULT X'00275CFF',
slash VARCHAR(64) DEFAULT 'C:\\temp\\file',
uuid_value CHAR(36) DEFAULT (uuid()),
hidden_col INT INVISIBLE,
dt DATETIME(6) DEFAULT '2020-01-02 03:04:05.123456',
payload MEDIUMBLOB,
doc TEXT DEFAULT ('first;
-- data, not a comment
last'),
generated_amount DECIMAL(13,4) GENERATED ALWAYS AS (amount * 2) STORED,
virtual_label VARCHAR(80) GENERATED ALWAYS AS (concat('prefix:',lower(label))) VIRTUAL,
PRIMARY KEY (id,code),
UNIQUE KEY email_unique (email),
UNIQUE KEY pair_unique (label,email),
UNIQUE KEY unique_expression ((concat(lower(code),'-x')),email),
KEY prefix_desc (label(10) DESC),
KEY expression_key ((lower(email))),
KEY invisible_key (medium) INVISIBLE COMMENT 'index''s comment',
FULLTEXT KEY text_search (doc),
CONSTRAINT ` + q(prefix+"_amount_check") + ` CHECK (amount >= 0),
CONSTRAINT ` + q(prefix+"_optional_check") + ` CHECK (medium > 0 AND status <> 'forbidden;-- value') NOT ENFORCED
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci COMMENT='table''s comment'`)
exec("CREATE TABLE " + q(child) + ` (
id INT PRIMARY KEY,
parent_id BIGINT UNSIGNED,
parent_code CHAR(3),
parent_email VARCHAR(100),
CONSTRAINT ` + q(prefix+"_composite_fk") + ` FOREIGN KEY (parent_id,parent_code) REFERENCES ` + q(parent) + ` (id,code) ON DELETE SET NULL ON UPDATE CASCADE,
CONSTRAINT ` + q(prefix+"_email_fk") + ` FOREIGN KEY (parent_email) REFERENCES ` + q(parent) + ` (email) ON DELETE CASCADE ON UPDATE RESTRICT
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci`)
readerConfig, err := driver.ParseDSN(conn)
if err != nil {
t.Fatal(err)
}
if readerConfig.Params == nil {
readerConfig.Params = map[string]string{}
}
readerConfig.Params["sql_quote_show_create"] = "0" // reader must ensure quoted native metadata
read := func() *models.Database {
t.Helper()
model, err := rmysql.NewReader(&readers.ReaderOptions{ConnectionString: readerConfig.FormatDSN()}).ReadDatabase()
if err != nil {
t.Fatal(err)
}
return model
}
model := read()
var selected []*models.Table
for _, table := range model.Schemas[0].Tables {
if table.Name == parent || table.Name == child {
selected = append(selected, table)
}
}
if len(selected) != 2 {
t.Fatalf("read %d fixture tables", len(selected))
}
model.Schemas[0].Tables = selected
model.Metadata = nil
model.Schemas[0].Metadata = nil // unrelated pre-existing test objects are outside this fixture
for _, table := range selected {
if table.Name == parent {
if table.Columns["id"].Type != "uint64" {
t.Fatal("unsigned type lost")
}
if table.Columns["amount"].Precision != 12 || table.Columns["amount"].Scale != 4 {
t.Fatal("decimal dimensions lost")
}
if table.Indexes["expression_key"].KeyParts[0].Expression == "" {
t.Fatal("functional index expression lost")
}
} else if len(table.Relationships) != 2 {
t.Fatalf("foreign key relationships collapsed: %d", len(table.Relationships))
}
}
// Exercise the serialized model boundary as well as the native reader objects.
encoded, err := json.Marshal(model)
if err != nil {
t.Fatal(err)
}
var target models.Database
if err := json.Unmarshal(encoded, &target); err != nil {
t.Fatal(err)
}
yamlModel, err := yaml.Marshal(&target)
if err != nil {
t.Fatal(err)
}
target = models.Database{}
if err := yaml.Unmarshal(yamlModel, &target); err != nil {
t.Fatal(err)
}
for _, table := range target.Schemas[0].Tables {
table.Name += "_copy"
for _, col := range table.Columns {
col.Table = table.Name
}
for _, index := range table.Indexes {
index.Table = table.Name
}
constraints := map[string]*models.Constraint{}
for _, c := range table.Constraints {
c.Table = table.Name
if c.Type == models.ForeignKeyConstraint {
c.ReferencedTable += "_copy"
}
if c.Type == models.ForeignKeyConstraint || c.Type == models.CheckConstraint {
c.Name += "_copy"
}
constraints[c.Name] = c
}
table.Constraints = constraints
for _, r := range table.Relationships {
r.FromTable += "_copy"
r.ToTable += "_copy"
r.ForeignKey += "_copy"
}
}
opts := &writers.WriterOptions{StrictConversion: true, Metadata: map[string]any{"connection_string": conn}}
if err := wmysql.NewWriter(opts).WriteDatabase(&target); err != nil {
t.Fatal(err)
}
show := func(table string) string {
t.Helper()
var name, ddl string
if err := db.QueryRow("SHOW CREATE TABLE "+q(table)).Scan(&name, &ddl); err != nil {
t.Fatal(err)
}
ddl = strings.ReplaceAll(ddl, "_copy", "")
// Index creation order is not semantic. Keep column and constraint order intact.
lines := strings.Split(ddl, "\n")
var indexes []string
for _, line := range lines {
if strings.HasPrefix(strings.TrimSpace(line), "KEY ") {
indexes = append(indexes, line)
}
}
sort.Strings(indexes)
n := 0
for i, line := range lines {
if strings.HasPrefix(strings.TrimSpace(line), "KEY ") {
lines[i] = indexes[n]
n++
}
}
return strings.Join(lines, "\n")
}
for _, name := range []string{parent, child} {
a, b := show(name), show(name+"_copy")
if a != b {
t.Errorf("schema mismatch\nsource:\n%s\ntarget:\n%s", a, b)
}
}
again := read()
for _, table := range again.Schemas[0].Tables {
if table.Name == parent+"_copy" && !reflect.DeepEqual(table.Columns["amount"].Metadata, selected[1].Columns["amount"].Metadata) {
t.Error("type metadata changed on second read")
}
}
// Verify constraints and defaults as behavior, after comparing untouched schemas.
exec("INSERT INTO " + q(parent+"_copy") + " (id,code,email) VALUES (18446744073709551614,'abc','one@example.com')")
exec("INSERT INTO " + q(child+"_copy") + " (id,parent_id,parent_code,parent_email) VALUES (1,18446744073709551614,'abc','one@example.com')")
if _, err := db.Exec("INSERT INTO " + q(child+"_copy") + " (id,parent_id,parent_code) VALUES (2,123,'abc')"); err == nil {
t.Error("foreign key is not enforced")
}
if _, err := db.Exec("UPDATE " + q(parent+"_copy") + " SET amount=-1"); err == nil {
t.Error("check is not enforced")
}
var label, doc, binaryHex, slash string
var flags int
if err := db.QueryRow("SELECT label,doc,flags+0,HEX(binary_def),slash FROM "+q(parent+"_copy")).Scan(&label, &doc, &flags, &binaryHex, &slash); err != nil {
t.Fatal(err)
}
for _, table := range read().Schemas[0].Tables {
if table.Name == parent+"_copy" && table.Metadata["mysql_auto_increment"] != "18446744073709551615" {
t.Errorf("large auto-increment counter lost: %v", table.Metadata["mysql_auto_increment"])
}
}
if label != "don't (change)" || doc != "first;\n-- data, not a comment\nlast" || flags != 10 || binaryHex != "00275CFF" || slash != `C:\temp\file` {
t.Errorf("defaults corrupted: %q %q %d %s %q", label, doc, flags, binaryHex, slash)
}
}
+355 -110
View File
@@ -7,12 +7,14 @@ import (
"io"
"os"
"sort"
"strconv"
"strings"
_ "github.com/go-sql-driver/mysql"
"git.warky.dev/wdevs/relspecgo/pkg/mariadb"
"git.warky.dev/wdevs/relspecgo/pkg/models"
"git.warky.dev/wdevs/relspecgo/pkg/transform"
"git.warky.dev/wdevs/relspecgo/pkg/writers"
)
@@ -22,83 +24,202 @@ type Writer struct {
}
func NewWriter(options *writers.WriterOptions) *Writer { return &Writer{options: options} }
func (w *Writer) WriteDatabase(db *models.Database) error {
// Statements preflights the whole model, emits conversion diagnostics, and returns
// executable units. No output files or database connections are opened here.
func (w *Writer) Statements(db *models.Database) ([]string, error) {
if w.options == nil {
return fmt.Errorf("writer options are required")
return nil, fmt.Errorf("writer options are required")
}
if conn, ok := w.options.Metadata["connection_string"].(string); ok && conn != "" {
return w.execute(db, conn)
if db == nil {
return nil, fmt.Errorf("database is required")
}
release, err := w.openOutput()
if err != nil {
return err
}
defer release()
return w.writeDatabaseDDL(db)
}
func (w *Writer) writeDatabaseDDL(db *models.Database) error {
if _, err := fmt.Fprintf(w.writer, "-- MySQL Database Schema\n-- Database: %s\n-- Generated by RelSpec\n\n", db.Name); err != nil {
return err
}
for _, s := range db.Schemas {
if err := w.WriteSchema(s); err != nil {
return err
}
}
return nil
}
// openOutput points w.writer at the configured destination (output file or
// stdout) when none is set, and returns a func that releases it again.
func (w *Writer) openOutput() (func(), error) {
if w.writer != nil {
return func() {}, nil
}
if w.options != nil && w.options.OutputPath != "" {
f, err := os.Create(w.options.OutputPath)
if w.options.LowercaseIdentifiers {
var err error
db, err = transform.LowercaseIdentifiers(db)
if err != nil {
return nil, err
}
w.writer = f
return func() {
f.Close()
w.writer = nil
}, nil
}
w.writer = os.Stdout
return func() { w.writer = nil }, nil
diagnostics := conversionDiagnostics(db)
if len(diagnostics) > 0 && w.options.StrictConversion {
return nil, fmt.Errorf("lossy MySQL conversion rejected:\n- %s", strings.Join(diagnostics, "\n- "))
}
for _, d := range diagnostics {
if w.options.Warning != nil {
w.options.Warning(d)
} else {
fmt.Fprintln(os.Stderr, "warning: MySQL conversion: "+d)
}
}
return databaseStatements(db), nil
}
func (w *Writer) WriteSchema(s *models.Schema) error {
release, err := w.openOutput()
func (w *Writer) WriteDatabase(db *models.Database) error {
statements, err := w.Statements(db)
if err != nil {
return err
}
defer release()
for _, t := range s.Tables {
if err := w.writeTable(s, t); err != nil {
if conn, ok := w.options.Metadata["connection_string"].(string); ok && conn != "" {
return w.execute(statements, conn)
}
if w.writer != nil {
return writeStatements(w.writer, statements, db.Name)
}
if w.options.OutputPath == "" {
return writeStatements(os.Stdout, statements, db.Name)
}
f, err := os.Create(w.options.OutputPath)
if err != nil {
return err
}
err = writeStatements(f, statements, db.Name)
closeErr := f.Close()
if err != nil {
return err
}
return closeErr
}
func writeStatements(out io.Writer, statements []string, name string) error {
name = strings.NewReplacer("\n", " ", "\r", " ").Replace(name)
if _, err := fmt.Fprintf(out, "-- Database: %s\n", name); err != nil {
return err
}
if _, err := io.WriteString(out, "-- MySQL table schema; generated by RelSpec\n\n"); err != nil {
return err
}
for _, stmt := range statements {
if _, err := io.WriteString(out, stmt); err != nil {
return err
}
}
return nil
}
func (w *Writer) WriteTable(t *models.Table) error {
release, err := w.openOutput()
if err != nil {
return err
}
defer release()
return w.writeTable(nil, t)
func (w *Writer) WriteSchema(s *models.Schema) error {
return w.WriteDatabase(&models.Database{Schemas: []*models.Schema{s}})
}
func (w *Writer) writeTable(s *models.Schema, t *models.Table) error {
name := t.Name
func (w *Writer) WriteTable(t *models.Table) error {
return w.WriteSchema(&models.Schema{Name: "", Tables: []*models.Table{t}})
}
func qualified(schema, name string) string {
if schema == "" {
return quote(name)
}
return quote(schema) + "." + quote(name)
}
func keys[T any](m map[string]T) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
func metaString(m map[string]any, key string) string { v, _ := m[key].(string); return v }
func metaBool(m map[string]any, key string, def bool) bool {
v, ok := m[key].(bool)
if ok {
return v
}
return def
}
func signature(c *models.Column) string {
return fmt.Sprintf("%s/%d/%d/%d", c.Type, c.Length, c.Precision, c.Scale)
}
func columnType(c *models.Column) string {
if native := metaString(c.Metadata, "mysql_type"); native != "" && metaString(c.Metadata, "mysql_type_signature") == signature(c) {
return native
}
typ := mariadb.ConvertCanonicalToMariaDB(c.Type)
switch strings.ToLower(c.Type) {
case "string":
if c.Length > 0 {
typ = fmt.Sprintf("VARCHAR(%d)", c.Length)
}
case "decimal":
if c.Precision > 0 {
typ = fmt.Sprintf("DECIMAL(%d,%d)", c.Precision, c.Scale)
}
case "timestamp", "time":
if c.Precision > 0 {
typ += fmt.Sprintf("(%d)", c.Precision)
}
}
return typ
}
// MySQL string literals use doubled quotes and escaped backslashes. The output
// assumes the server's standard SQL mode (NO_BACKSLASH_ESCAPES is not enabled).
func literal(s string) string {
return "'" + strings.ReplaceAll(strings.ReplaceAll(s, "\\", "\\\\"), "'", "''") + "'"
}
func defaultSQL(c *models.Column) string {
value := fmt.Sprint(c.Default)
if native := metaString(c.Metadata, "mysql_default_sql"); native != "" && metaString(c.Metadata, "mysql_default_value") == value {
return native
}
if strings.HasPrefix(metaString(c.Metadata, "mysql_type"), "bit(") && strings.HasPrefix(value, "b'") && strings.HasSuffix(value, "'") && strings.Trim(value[2:len(value)-1], "01") == "" {
return value
}
if metaBool(c.Metadata, "mysql_default_expression", false) {
if strings.HasPrefix(strings.ToUpper(value), "CURRENT_TIMESTAMP") {
return value
}
return "(" + value + ")"
}
switch strings.ToLower(c.Type) {
case "text", "json", "jsonb", "bytea":
return "(" + literal(value) + ")"
case "string", "uuid", "date", "time", "timestamp", "timestamptz":
return literal(value)
case "bool", "boolean":
if value == "true" {
return "1"
}
if value == "false" {
return "0"
}
}
// Numeric defaults are values, never executable fragments.
if _, err := strconv.ParseFloat(value, 64); err == nil {
return value
}
return literal(value)
}
func indexParts(i *models.Index) string {
if len(i.KeyParts) == 0 {
return quoted(i.Columns)
}
parts := make([]string, 0, len(i.KeyParts))
for _, p := range i.KeyParts {
s := quote(p.Name)
if p.Expression != "" {
s = "(" + p.Expression + ")"
}
if p.Length > 0 {
s += fmt.Sprintf("(%d)", p.Length)
}
if p.Descending {
s += " DESC"
}
parts = append(parts, s)
}
return strings.Join(parts, ", ")
}
func tableStatement(s *models.Schema, t *models.Table) string {
schema := ""
if s != nil {
name = fmt.Sprintf("%s.%s", quote(s.Name), quote(t.Name))
} else {
name = quote(name)
schema = s.Name
}
cols := make([]*models.Column, 0, len(t.Columns))
for _, c := range t.Columns {
@@ -113,92 +234,198 @@ func (w *Writer) writeTable(s *models.Schema, t *models.Table) error {
defs := []string{}
pk := []string{}
for _, c := range cols {
d := fmt.Sprintf(" %s %s", quote(c.Name), mariadb.ConvertCanonicalToMariaDB(c.Type))
if c.Length > 0 && strings.EqualFold(c.Type, "string") {
d = fmt.Sprintf(" %s VARCHAR(%d)", quote(c.Name), c.Length)
d := " " + quote(c.Name) + " " + columnType(c)
if charset := metaString(c.Metadata, "mysql_charset"); charset != "" && c.Collation != metaString(t.Metadata, "mysql_collation") {
d += " CHARACTER SET " + quote(charset)
}
if c.Collation != "" && c.Collation != metaString(t.Metadata, "mysql_collation") {
d += " COLLATE " + quote(c.Collation)
}
if c.Generated {
d += " GENERATED ALWAYS AS (" + c.GenerationExpression + ")"
if strings.EqualFold(metaString(c.Metadata, "mysql_generation_storage"), "STORED") {
d += " STORED"
} else {
d += " VIRTUAL"
}
}
if c.NotNull {
d += " NOT NULL"
} else if !c.Generated {
d += " NULL"
}
if c.AutoIncrement {
d += " AUTO_INCREMENT"
}
if c.Default != nil {
d += fmt.Sprintf(" DEFAULT %s", writers.QuoteDefaultValue(fmt.Sprint(c.Default), c.Type))
if c.Default != nil && !c.Generated {
d += " DEFAULT " + defaultSQL(c)
}
if upd := metaString(c.Metadata, "mysql_on_update"); upd != "" {
d += " ON UPDATE " + upd
}
if metaBool(c.Metadata, "mysql_invisible", false) {
d += " INVISIBLE"
}
if c.Comment != "" {
d += " COMMENT " + literal(c.Comment)
}
defs = append(defs, d)
if c.IsPrimaryKey {
pk = append(pk, quote(c.Name))
pk = append(pk, c.Name)
}
}
constraintNames := make([]string, 0, len(t.Constraints))
for name := range t.Constraints {
constraintNames = append(constraintNames, name)
}
sort.Strings(constraintNames)
for _, name := range constraintNames {
for _, name := range keys(t.Constraints) {
c := t.Constraints[name]
if c.Type == models.PrimaryKeyConstraint {
pk = nil
for _, n := range c.Columns {
pk = append(pk, quote(n))
}
pk = c.Columns
break
}
}
if len(pk) > 0 {
defs = append(defs, " PRIMARY KEY ("+strings.Join(pk, ", ")+")")
}
for _, name := range constraintNames {
c := t.Constraints[name]
if c.Type == models.UniqueConstraint {
defs = append(defs, fmt.Sprintf(" CONSTRAINT %s UNIQUE (%s)", quote(c.Name), quoted(c.Columns)))
var primaryIndex *models.Index
for name, i := range t.Indexes {
if strings.EqualFold(name, "PRIMARY") {
primaryIndex = i
break
}
}
sql := fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s (\n%s\n) ENGINE=InnoDB;\n\n", name, strings.Join(defs, ",\n"))
if _, err := io.WriteString(w.writer, sql); err != nil {
return err
if i := primaryIndex; i != nil {
defs = append(defs, " PRIMARY KEY ("+indexParts(i)+")")
} else if len(pk) > 0 {
defs = append(defs, " PRIMARY KEY ("+quoted(pk)+")")
}
return nil
for _, name := range keys(t.Constraints) {
c := t.Constraints[name]
switch c.Type {
case models.UniqueConstraint:
if t.Indexes[c.Name] == nil {
defs = append(defs, " CONSTRAINT "+quote(c.Name)+" UNIQUE ("+quoted(c.Columns)+")")
}
case models.CheckConstraint:
d := " CONSTRAINT " + quote(c.Name) + " CHECK (" + c.Expression + ")"
if !metaBool(c.Metadata, "mysql_enforced", true) {
d += " NOT ENFORCED"
}
defs = append(defs, d)
}
}
for _, name := range keys(t.Indexes) {
i := t.Indexes[name]
if strings.EqualFold(name, "PRIMARY") {
continue
}
method := strings.ToUpper(i.Type)
if method != "" && method != "BTREE" && method != "HASH" && method != "FULLTEXT" && method != "SPATIAL" {
continue
}
kind := "KEY"
if i.Unique {
kind = "UNIQUE KEY"
}
if method == "FULLTEXT" || method == "SPATIAL" {
kind = method + " KEY"
}
d := " " + kind + " " + quote(i.Name) + " (" + indexParts(i) + ")"
if method == "HASH" {
d += " USING " + method
}
if !metaBool(i.Metadata, "mysql_visible", true) {
d += " INVISIBLE"
}
if i.Comment != "" {
d += " COMMENT " + literal(i.Comment)
}
defs = append(defs, d)
}
engine := metaString(t.Metadata, "mysql_engine")
if engine == "" {
engine = "InnoDB"
}
tail := " ENGINE=" + quote(engine)
if engine == "InnoDB" {
tail = " ENGINE=InnoDB"
}
if coll := metaString(t.Metadata, "mysql_collation"); coll != "" {
tail += " COLLATE=" + quote(coll)
}
if auto := metaString(t.Metadata, "mysql_auto_increment"); auto != "" {
if _, err := strconv.ParseUint(auto, 10, 64); err == nil {
tail += " AUTO_INCREMENT=" + auto
}
}
if t.Comment != "" {
tail += " COMMENT=" + literal(t.Comment)
}
return "CREATE TABLE IF NOT EXISTS " + qualified(schema, t.Name) + " (\n" + strings.Join(defs, ",\n") + "\n)" + tail + ";\n\n"
}
func (w *Writer) execute(dbm *models.Database, conn string) error {
var b strings.Builder
old := w.writer
w.writer = &b
if err := w.writeDatabaseDDL(dbm); err != nil {
w.writer = old
return err
func databaseStatements(db *models.Database) []string {
var statements []string
for _, s := range db.Schemas {
for _, t := range s.Tables {
statements = append(statements, tableStatement(s, t))
}
}
// References are added after all tables, supporting cycles and forward references.
for _, s := range db.Schemas {
for _, t := range s.Tables {
for _, name := range keys(t.Constraints) {
c := t.Constraints[name]
if c.Type != models.ForeignKeyConstraint {
continue
}
rs := c.ReferencedSchema
if rs == "" {
rs = s.Name
}
stmt := "ALTER TABLE " + qualified(s.Name, t.Name) + " ADD CONSTRAINT " + quote(c.Name) + " FOREIGN KEY (" + quoted(c.Columns) + ") REFERENCES " + qualified(rs, c.ReferencedTable) + " (" + quoted(c.ReferencedColumns) + ")"
if validAction(c.OnDelete) && c.OnDelete != "" {
stmt += " ON DELETE " + strings.ToUpper(c.OnDelete)
}
if validAction(c.OnUpdate) && c.OnUpdate != "" {
stmt += " ON UPDATE " + strings.ToUpper(c.OnUpdate)
}
statements = append(statements, stmt+";\n\n")
}
}
}
return statements
}
func (w *Writer) execute(statements []string, conn string) error {
if w.options.AtomicExecution {
return fmt.Errorf("MySQL DDL implicitly commits; atomic execution is unsupported")
}
w.writer = old
db, err := sql.Open("mysql", conn)
if err != nil {
return fmt.Errorf("failed to connect: %w", err)
}
defer db.Close()
for _, stmt := range strings.Split(b.String(), ";\n") {
stmt = stripComments(strings.TrimSpace(stmt))
if stmt == "" {
continue
}
if _, err := db.ExecContext(context.Background(), stmt); err != nil {
return fmt.Errorf("failed to execute SQL: %w", err)
session, err := db.Conn(context.Background())
if err != nil {
return fmt.Errorf("failed to execute SQL: %w", err)
}
defer session.Close()
var mode string
if err := session.QueryRowContext(context.Background(), "SELECT @@SESSION.sql_mode").Scan(&mode); err != nil {
return fmt.Errorf("read MySQL SQL mode: %w", err)
}
if mode = unsupportedMode(mode); mode != "" {
return fmt.Errorf("MySQL execution requires standard quotes and backslash escaping; %s is enabled", mode)
}
var failures []string
for n, stmt := range statements {
if _, err := session.ExecContext(context.Background(), stmt); err != nil {
failures = append(failures, fmt.Sprintf("statement %d: %v", n+1, err))
if !w.options.ContinueOnError {
break
}
}
}
if len(failures) > 0 {
return fmt.Errorf("failed to execute SQL: %s", strings.Join(failures, "; "))
}
return nil
}
func stripComments(sqlText string) string {
lines := strings.Split(sqlText, "\n")
kept := lines[:0]
for _, line := range lines {
if !strings.HasPrefix(strings.TrimSpace(line), "--") {
kept = append(kept, line)
}
}
return strings.TrimSpace(strings.Join(kept, "\n"))
}
func quote(s string) string { return "`" + strings.ReplaceAll(s, "`", "``") + "`" }
func quoted(xs []string) string {
out := make([]string, len(xs))
@@ -207,3 +434,21 @@ func quoted(xs []string) string {
}
return strings.Join(out, ", ")
}
func validAction(action string) bool {
switch strings.ToUpper(action) {
case "", "CASCADE", "SET NULL", "RESTRICT", "NO ACTION":
return true
}
return false
}
func unsupportedMode(mode string) string {
var modes []string
for _, unsupported := range []string{"NO_BACKSLASH_ESCAPES", "ANSI_QUOTES"} {
if strings.Contains(strings.ToUpper(mode), unsupported) {
modes = append(modes, unsupported)
}
}
return strings.Join(modes, ", ")
}
+23 -7
View File
@@ -9,13 +9,6 @@ import (
"git.warky.dev/wdevs/relspecgo/pkg/writers"
)
func TestStripComments(t *testing.T) {
got := stripComments("-- header\nCREATE TABLE `users` (\n `id` INT\n);")
if strings.HasPrefix(got, "--") || !strings.HasPrefix(got, "CREATE TABLE") {
t.Fatalf("stripComments() = %q", got)
}
}
func TestWriterGeneratesMySQLDDL(t *testing.T) {
db := models.InitDatabase("app")
s := models.InitSchema("app")
@@ -42,3 +35,26 @@ func TestWriterGeneratesMySQLDDL(t *testing.T) {
}
}
}
func TestDatabaseStatementsKeepEmbeddedDelimiters(t *testing.T) {
db := models.InitDatabase("app")
s := models.InitSchema("app")
db.Schemas = append(db.Schemas, s)
for _, name := range []string{"one", "two"} {
table := models.InitTable(name, s.Name)
col := models.InitColumn("text", name, s.Name)
col.Type = "text"
col.Default = "first;\n-- data, not a comment\nlast"
table.Columns[col.Name] = col
s.Tables = append(s.Tables, table)
}
statements := databaseStatements(db)
if len(statements) != 2 {
t.Fatalf("got %d statements", len(statements))
}
for _, stmt := range statements {
if !strings.Contains(stmt, "first;\n-- data, not a comment\nlast") {
t.Fatalf("default corrupted: %s", stmt)
}
}
}
+13 -13
View File
@@ -8,8 +8,8 @@ Standardized naming rules for all database objects in RelSpec PostgreSQL output.
| ----------------- | ----------- | ---------------------------------- | ------------------------ |
| Primary Key | `pk_` | `pk_<schema>_<table>` | `pk_public_users` |
| Foreign Key | `fk_` | `fk_<table>_<referenced_table>` | `fk_posts_users` |
| Unique Constraint | `uk_` | `uk_<table>_<column>` | `uk_users_email` |
| Unique Index | `uidx_` | `uidx_<table>_<column>` | `uidx_users_email` |
| Unique Constraint | `ukey_` | `ukey_<table>_<column>` | `ukey_users_email` |
| Unique Index | `ukey_` | `ukey_<table>_<column>` | `ukey_users_email` |
| Regular Index | `idx_` | `idx_<table>_<column>` | `idx_posts_user_id` |
| Check Constraint | `chk_` | `chk_<table>_<constraint_purpose>` | `chk_users_age_positive` |
| Sequence | `identity_` | `identity_<table>_<column>` | `identity_users_id` |
@@ -43,25 +43,25 @@ Standardized naming rules for all database objects in RelSpec PostgreSQL output.
### Unique Constraints
**Pattern:** `uk_<table>_<column>`
**Pattern:** `ukey_<table>_<column>`
- Use `uk_` prefix strictly for database constraints (CONSTRAINT type)
- Generated by the DBML reader for column `[unique]`
- Include column name for clarity
- Examples:
- `uk_users_email`
- `uk_users_username`
- `uk_products_sku`
- `ukey_users_email`
- `ukey_users_username`
- `ukey_products_sku`
### Unique Indexes
**Pattern:** `uidx_<table>_<column>`
**Pattern:** `ukey_<table>_<column>`
- Use `uidx_` prefix strictly for index type objects
- Distinguished from constraints for clarity and implementation flexibility
- Unnamed unique indexes use the same name as a unique constraint on the same columns
- Writers then build the constraint on the index (`ADD CONSTRAINT ... UNIQUE USING INDEX`), so no duplicate index is created
- Examples:
- `uidx_users_email`
- `uidx_sessions_token`
- `uidx_api_keys_key`
- `ukey_users_email`
- `ukey_sessions_token`
- `ukey_api_keys_key`
### Regular Indexes
+109
View File
@@ -17,6 +17,28 @@ The PostgreSQL Writer converts RelSpec's internal database model representation
- Includes audit triggers (optional)
- Handles PostgreSQL-specific data types
## Identifier and value handling
Generated identifiers are always double-quoted, including reserved words, and
retain the model's case by default. Supply raw model names, without SQL quoting.
Identifiers containing double quotes are escaped by doubling them; NUL is rejected
before database/file output. Catalog checks, comments, audit values, and regclass
names use SQL literal quoting; generated procedural/function bodies choose dollar
delimiters absent from their rendered content. Names in generated line comments
cannot create new SQL lines.
Use CLI `--lowercase`, job option `lowercase_identifiers: true`, or PostgreSQL
`WriterOptions.LowercaseIdentifiers: true` to lowercase model names and structured
references. The transformation copies the model and rejects name collisions. Other
library writers can use `transform.LowercaseIdentifiers` before writing.
SQL expressions are executable input: types, defaults, generated expressions,
checks, index expressions/predicates, dialect directive clauses, audit user
expressions, and `pgsql_using` / `pgsql_preflight` are not sanitized or case-folded.
Adjust identifier references inside these expressions yourself when lowercasing.
Enum values, comments, and other data retain their case. Migration matching respects
identifier case; `"Users"` and `"users"` are distinct objects.
## Usage
### Basic Example
@@ -154,6 +176,29 @@ When `migration_mode` is enabled, generates migration scripts with:
- Transactional DDL
- Rollback support
### Unique constraints and indexes
- Unique constraint → `ADD CONSTRAINT <name> UNIQUE (cols)`, idempotent (checked by name)
- Migration mode and direct apply both emit them (not only `convert`)
- Same-named model unique index → `UNIQUE USING INDEX` (no duplicate index). Requires: same name, same columns, btree, non-partial, no `INCLUDE`
- Otherwise a plain `ADD CONSTRAINT` (second index is built)
- Naming: unique constraint and unique index both `ukey_<table>_<cols>`; see `NAMING_CONVENTIONS.md`
- Renaming `uidx_…` → `ukey_…` drops the old index and re-adds the constraint on the next diff
### FK deferral
| Model (`Deferrable` / `InitiallyDeferred`) | SQL |
| ------------------------------------------ | --- |
| unset | create: `DEFERRABLE` (migration/apply), none (full DDL constraints); existing FK never diffed |
| `true` / unset or `false` | `DEFERRABLE` |
| `true` / `true`, or unset / `true` | `DEFERRABLE INITIALLY DEFERRED` |
| `false` | `NOT DEFERRABLE` |
- Both fields are `*bool`; `nil` = unspecified
- Specified value differing from the live DB → FK dropped and re-added
- Live reader reads `condeferrable` / `condeferred`
- Initial mode compared only when both sides state it
### Audit Triggers
When `include_audit` is enabled, adds:
@@ -323,3 +368,67 @@ metadata:
- Default values are properly quoted
- Constraint names follow PostgreSQL conventions
- Compatible with PostgreSQL 12+
## Live execution failures
Live PostgreSQL output stops after the first failed statement by default and returns
`*writers.ExecutionFailure` (also available as `*pgsql.ExecutionFailure`; inspect it
with `errors.As`). `WriterOptions.ContinueOnError` or job
`options.continue_on_error: true` attempts later statements but still returns an error
if any fail, preventing dependent jobs from running. A failed type preflight always
stops live execution, including when continuation is enabled.
PostgreSQL, SQLite, SQL Server, and the PostgreSQL script runner share execution
reports through `WriterOptions.Metadata["execution_report"]`. Optional
`Metadata["report_path"]` writes the report as JSON; write and close failures are
errors. Reports include each operation's status and original number, plus applied,
failed, skipped, and rolled-back counts. Blank and comment entries count as skipped.
Unknown commit/rollback outcomes are explicitly marked `unknown` and are never
reported as confirmed rollback. Connection failures record all planned operations as
skipped. Raw scripts have different reporting limits; see the sqlexec README.
Generated DDL uses independent statements by default: earlier changes can remain
committed after failure. Set `WriterOptions.AtomicExecution = true` to use one
transaction with PostgreSQL, SQLite, or SQL Server generated DDL. For database output
jobs, use:
```yaml
options:
atomic_execution: true
```
Atomic execution and continuation are mutually exclusive. Rollback covers database
transactional effects; sequence advancement and external effects in SQL expressions
are outside that guarantee. Concurrency locks, migration history, and resumable
migration management remain separate work (audit A09).
## Column type conversions
Type changes propagate errors and never implicitly rename or replace a column.
Before schema changes, full-DDL and diff plans evaluate every existing value through
the conversion expression and target cast. Diff preflights run before dropping
constraints/indexes or creating sequences; database-wide preflights run before
changing any schema. Matching types, missing tables/columns, generated columns, and
identity columns need no conversion check.
Column metadata in JSON/YAML models supports two optional SQL expressions:
```yaml
metadata:
pgsql_using: "split_part(name, ' ', 1)::integer"
pgsql_preflight: "name ~ '^[0-9]+ units$'"
```
`pgsql_using` replaces the default `column::target_type` expression in `USING`.
`pgsql_preflight` is a row predicate; false or NULL rejects the conversion. These
fields must contain nonempty strings. Both are trusted executable SQL, like defaults
and check expressions; they are not values or a sandbox. Preflights evaluate the
conversion expression, so use expressions without side effects. New types/extensions
needed by an existing-column conversion must already be available for preflight.
Preflight checks can race concurrent writes and do not prove that locks, defaults,
or every dependency will permit the ALTER. The ALTER still validates the conversion;
use atomic execution when preceding operations also need rollback. Generated SQL
clients must stop on preflight failure; live RelSpec execution enforces this even
with continuation enabled. Rename/replacement/backfill operations still require a
separately reviewed migration.
+5 -4
View File
@@ -22,8 +22,9 @@ func TestCurrentColumnHasDescription(t *testing.T) {
want bool
}{
{"nil table", nil, &models.Column{Name: "email", Description: "x"}, false},
{"match ignoring case and whitespace", table, &models.Column{Name: "email", Description: "the email"}, true},
{"different description", table, &models.Column{Name: "email", Description: "other"}, false},
{"match ignoring whitespace", table, &models.Column{Name: "Email", Description: "the email"}, true},
{"different description", table, &models.Column{Name: "Email", Description: "other"}, false},
{"different identifier case", table, &models.Column{Name: "email", Description: "the email"}, false},
{"column missing", table, &models.Column{Name: "age", Description: "x"}, false},
}
for _, tt := range tests {
@@ -41,12 +42,12 @@ func TestExecuteCommentColumn(t *testing.T) {
t.Fatal(err)
}
got, err := te.ExecuteCommentColumn(CommentColumnData{
SchemaName: "public", TableName: "users", ColumnName: "email", Comment: "it''s",
SchemaName: "public", TableName: "users", ColumnName: "email", Comment: "it's",
})
if err != nil {
t.Fatal(err)
}
if !strings.Contains(got, "COMMENT ON COLUMN") || !strings.Contains(got, "public.users") ||
if !strings.Contains(got, "COMMENT ON COLUMN") || !strings.Contains(got, "\"public\".\"users\"") ||
!strings.Contains(got, "email") || !strings.Contains(got, "IS 'it''s';") {
t.Errorf("unexpected output: %s", got)
}
+11 -11
View File
@@ -57,8 +57,8 @@ func TestDiffStatements(t *testing.T) {
current: func() *models.Database {
return diffTestDB("public", map[string]map[string]string{"users": {"id": "integer"}})
},
wantContain: []string{"CREATE TABLE IF NOT EXISTS public.posts"},
wantAbsent: []string{"public.users"},
wantContain: []string{"CREATE TABLE IF NOT EXISTS \"public\".\"posts\""},
wantAbsent: []string{"\"public\".\"users\""},
},
{
name: "missing column is added and existing columns are skipped",
@@ -68,7 +68,7 @@ func TestDiffStatements(t *testing.T) {
current: func() *models.Database {
return diffTestDB("public", map[string]map[string]string{"users": {"id": "integer"}})
},
wantContain: []string{"ADD COLUMN IF NOT EXISTS email text"},
wantContain: []string{"ADD COLUMN IF NOT EXISTS \"email\" text"},
wantAbsent: []string{"COLUMN id"},
},
{
@@ -79,7 +79,7 @@ func TestDiffStatements(t *testing.T) {
current: func() *models.Database {
return diffTestDB("public", map[string]map[string]string{"users": {"id": "integer"}})
},
wantContain: []string{"ALTER COLUMN id TYPE bigint"},
wantContain: []string{"ALTER COLUMN \"id\" TYPE bigint"},
},
{
name: "missing non-public schema is created before its tables",
@@ -88,8 +88,8 @@ func TestDiffStatements(t *testing.T) {
},
current: func() *models.Database { return models.InitDatabase("testdb") },
wantContain: []string{
"CREATE SCHEMA IF NOT EXISTS app",
"CREATE TABLE IF NOT EXISTS app.users",
"CREATE SCHEMA IF NOT EXISTS \"app\"",
"CREATE TABLE IF NOT EXISTS \"app\".\"users\"",
},
},
}
@@ -149,7 +149,7 @@ func TestDiffStatements_NewTableCreatesPrimaryKeySequence(t *testing.T) {
id := models.InitColumn("id", "users", "public")
id.Type = "integer"
id.IsPrimaryKey = true
id.Default = "nextval('public.users_id_seq'::regclass)"
id.Default = "nextval('\"public\".\"users_id_seq\"'::regclass)"
table.Columns["id"] = id
schema.Tables = append(schema.Tables, table)
model.Schemas = append(model.Schemas, schema)
@@ -160,7 +160,7 @@ func TestDiffStatements_NewTableCreatesPrimaryKeySequence(t *testing.T) {
t.Fatalf("diffStatements failed: %v", err)
}
out := diffJoin(stmts)
if !strings.Contains(out, "CREATE SEQUENCE IF NOT EXISTS public.users_id_seq") {
if !strings.Contains(out, "CREATE SEQUENCE IF NOT EXISTS \"public\".\"users_id_seq\"") {
t.Fatalf("expected sequence creation, got:\n%s", out)
}
@@ -193,7 +193,7 @@ func TestDiffStatements_NewIndexAndChangedIndexRecreated(t *testing.T) {
if err != nil {
t.Fatalf("diffStatements failed: %v", err)
}
if !strings.Contains(diffJoin(stmts), "CREATE INDEX IF NOT EXISTS idx_users_lookup") {
if !strings.Contains(diffJoin(stmts), "CREATE INDEX IF NOT EXISTS \"idx_users_lookup\"") {
t.Fatalf("expected index creation, got:\n%s", diffJoin(stmts))
}
@@ -329,7 +329,7 @@ func TestDiffStatements_ChangedPrimaryKeyIsRecreated(t *testing.T) {
if err != nil {
t.Fatalf("diffStatements failed: %v", err)
}
if !strings.Contains(diffJoin(stmts), "DROP CONSTRAINT IF EXISTS pk_public_users") {
if !strings.Contains(diffJoin(stmts), "DROP CONSTRAINT IF EXISTS \"pk_public_users\"") {
t.Fatalf("expected old primary key to be dropped, got:\n%s", diffJoin(stmts))
}
}
@@ -345,7 +345,7 @@ func TestDiffStatements_ChangedCommentEmitted(t *testing.T) {
if err != nil {
t.Fatalf("diffStatements failed: %v", err)
}
if !strings.Contains(diffJoin(stmts), "COMMENT ON TABLE public.users IS 'new'") {
if !strings.Contains(diffJoin(stmts), "COMMENT ON TABLE \"public\".\"users\" IS 'new'") {
t.Fatalf("expected changed comment, got:\n%s", diffJoin(stmts))
}
}
+2 -2
View File
@@ -55,7 +55,7 @@ func TestPgDirectives_WriteDatabasePath(t *testing.T) {
for _, want := range []string{
") PARTITION BY RANGE (created_at) TABLESPACE fast_data",
"GENERATED ALWAYS AS IDENTITY",
"WITH (fillfactor=90) TABLESPACE idx_space",
"WITH (fillfactor=90) TABLESPACE \"idx_space\"",
} {
if !strings.Contains(out, want) {
t.Errorf("missing %q in:\n%s", want, out)
@@ -78,7 +78,7 @@ func TestPgDirectives_WriteSchemaPath(t *testing.T) {
if !strings.Contains(out, ") PARTITION BY RANGE (created_at) TABLESPACE fast_data;") {
t.Errorf("table suffix missing from WriteSchema path:\n%s", out)
}
if !strings.Contains(out, "WITH (fillfactor=90) TABLESPACE idx_space") {
if !strings.Contains(out, "WITH (fillfactor=90) TABLESPACE \"idx_space\"") {
t.Errorf("index clauses missing from WriteSchema path:\n%s", out)
}
}
+2 -2
View File
@@ -88,8 +88,8 @@ func TestGeneratedColumns_FullDDL(t *testing.T) {
}
out := diffJoin(stmts)
for _, s := range []string{
"full_name text GENERATED ALWAYS AS (first || ' ' || last) STORED",
"seq bigint GENERATED ALWAYS AS IDENTITY NOT NULL",
"\"full_name\" text GENERATED ALWAYS AS (first || ' ' || last) STORED",
"\"seq\" bigint GENERATED ALWAYS AS IDENTITY NOT NULL",
} {
if !strings.Contains(out, s) {
t.Errorf("missing %q:\n%s", s, out)
+116
View File
@@ -0,0 +1,116 @@
package pgsql
import (
"fmt"
"reflect"
"strings"
"git.warky.dev/wdevs/relspecgo/pkg/models"
"git.warky.dev/wdevs/relspecgo/pkg/transform"
"git.warky.dev/wdevs/relspecgo/pkg/writers"
)
// prepareModel rejects NUL before creating files or executing SQL. SQL names
// cannot contain NUL; pgx's identifier sanitizer would otherwise remove it.
func prepareModel(db *models.Database, lower bool) (*models.Database, error) {
if db == nil {
return nil, fmt.Errorf("database is required")
}
seen := map[any]bool{}
var validate func(reflect.Value) error
validate = func(v reflect.Value) error {
switch v.Kind() {
case reflect.Pointer:
if v.IsNil() || v.Type().Elem().PkgPath() != reflect.TypeOf(models.Database{}).PkgPath() {
return nil
}
if seen[v.Interface()] {
return nil
}
seen[v.Interface()] = true
return validate(v.Elem())
case reflect.Struct:
for i := 0; i < v.NumField(); i++ {
if v.Type().Field(i).Name == "Metadata" {
continue
}
if err := validate(v.Field(i)); err != nil {
return err
}
}
case reflect.String:
if strings.ContainsRune(v.String(), 0) {
return fmt.Errorf("PostgreSQL model contains a NUL byte")
}
case reflect.Slice:
for i := 0; i < v.Len(); i++ {
if err := validate(v.Index(i)); err != nil {
return err
}
}
case reflect.Map:
if v.Type().Elem().Kind() == reflect.Pointer {
iter := v.MapRange()
for iter.Next() {
if err := validate(iter.Value()); err != nil {
return err
}
}
}
}
return nil
}
if err := validate(reflect.ValueOf(db)); err != nil {
return nil, err
}
if lower {
return transform.LowercaseIdentifiers(db)
}
return db, nil
}
func (w *Writer) prepareSchema(schema *models.Schema) (*models.Schema, error) {
db, err := prepareModel(&models.Database{Schemas: []*models.Schema{schema}}, w.options.LowercaseIdentifiers)
if err != nil {
return nil, err
}
return db.Schemas[0], nil
}
// regclassLastName decodes quoted identifiers in a qualified regclass spelling.
func regclassLastName(text string) string {
var name strings.Builder
quoted := false
for i := 0; i < len(text); i++ {
c := text[i]
if c == '"' {
if quoted && i+1 < len(text) && text[i+1] == '"' {
name.WriteByte('"')
i++
} else {
quoted = !quoted
}
continue
}
if c == '.' && !quoted {
name.Reset()
continue
}
name.WriteByte(c)
}
return name.String()
}
// quoteDefaultValue keeps explicit SQL expressions intact and uses the same
// backslash-safe literal quoting as other PostgreSQL value contexts.
func quoteDefaultValue(value, sqlType string) string {
trimmed := strings.TrimSpace(value)
if len(trimmed) >= 3 && (trimmed[0] == 'E' || trimmed[0] == 'e') && trimmed[1] == '\'' && trimmed[len(trimmed)-1] == '\'' {
return trimmed
}
formatted := writers.QuoteDefaultValue(value, sqlType)
if formatted == "'"+strings.ReplaceAll(trimmed, "'", "''")+"'" {
return quote(trimmed)
}
return formatted
}
+178 -4
View File
@@ -3,9 +3,11 @@ package pgsql
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
@@ -141,15 +143,16 @@ func TestLive_WriteDatabaseEmptyThenIdenticalThenDrifted(t *testing.T) {
}
}
func TestLive_WriteDatabaseFailedStatementContinues(t *testing.T) {
func TestLive_WriteDatabaseFailedStatementReturnsError(t *testing.T) {
connString := liveWriterConn(t)
schema, _ := liveWriterSchema(t, connString)
reportPath := filepath.Join(t.TempDir(), "report.json")
db := liveModel(schema, map[string]string{"bad": "no_such_type_xyz"})
rep, err := runLiveWrite(t, connString, db, map[string]interface{}{"full_ddl": true, "report_path": reportPath})
if err != nil {
t.Fatalf("failed statements must not abort the run: %v", err)
var failure *ExecutionFailure
if !errors.As(err, &failure) {
t.Fatalf("expected structured failure, got: %v", err)
}
if rep.FailedStatements == 0 || len(rep.Errors) != rep.FailedStatements {
t.Fatalf("expected recorded failures: %+v", rep)
@@ -230,7 +233,7 @@ func TestLive_ExecuteStatementsSkipsCommentsAndBlank(t *testing.T) {
t.Fatal(err)
}
r := w.executionReport
if r.ExecutedStatements != 2 || r.FailedStatements != 0 || r.TotalStatements != 5 {
if r.ExecutedStatements != 2 || r.FailedStatements != 0 || r.TotalStatements != 5 || r.SkippedStatements != 3 {
t.Errorf("counts: %+v", r)
}
if len(r.Schemas) != 1 || r.Schemas[0].Name != schema || len(r.Schemas[0].Tables) != 1 || !r.Schemas[0].Tables[0].Created {
@@ -241,3 +244,174 @@ func TestLive_ExecuteStatementsSkipsCommentsAndBlank(t *testing.T) {
t.Errorf("table not created: %v", err)
}
}
func TestLive_ExecuteStatementsFailureModes(t *testing.T) {
connString := liveWriterConn(t)
for _, continueOnError := range []bool{false, true} {
t.Run(fmt.Sprintf("continue=%v", continueOnError), func(t *testing.T) {
schema, conn := liveWriterSchema(t, connString)
reportPath := filepath.Join(t.TempDir(), "report.json")
w := NewWriter(&writers.WriterOptions{
ContinueOnError: continueOnError,
Metadata: map[string]interface{}{"report_path": reportPath},
})
w.executionReport = &ExecutionReport{}
err := w.executeStatements([]string{
"CREATE SCHEMA " + schema,
"SELECT 1 / 0",
"CREATE TABLE " + schema + ".after_failure (id int)",
}, connString)
var failure *ExecutionFailure
if !errors.As(err, &failure) || failure.Report != w.executionReport {
t.Fatalf("expected structured execution failure, got %v", err)
}
var report ExecutionReport
data, err := os.ReadFile(reportPath)
if err != nil {
t.Fatal(err)
}
if err := json.Unmarshal(data, &report); err != nil {
t.Fatal(err)
}
executed, skipped := 1, 1
if continueOnError {
executed, skipped = 2, 0
}
if report.TotalStatements != 3 || report.ExecutedStatements != executed || report.FailedStatements != 1 || report.SkippedStatements != skipped || report.EndTime == "" {
t.Fatalf("incorrect partial application report: %+v", report)
}
if len(report.Errors) != 1 || report.Errors[0].StatementNumber != 2 {
t.Fatalf("incorrect failure details: %+v", report.Errors)
}
if exists := columnExists(t, conn, schema, "after_failure", "id"); exists != continueOnError {
t.Fatalf("later statement executed=%v, continue=%v", exists, continueOnError)
}
})
}
}
func TestLive_IncompatibleTypeChangePreservesColumnAndData(t *testing.T) {
connString := liveWriterConn(t)
schema, conn := liveWriterSchema(t, connString)
ctx := context.Background()
_, err := conn.Exec(ctx, "CREATE SCHEMA "+schema+"; CREATE TABLE "+schema+".accounts (id integer PRIMARY KEY, name text UNIQUE); INSERT INTO "+schema+".accounts VALUES (1, 'not-an-integer')")
if err != nil {
t.Fatal(err)
}
var originalIndex uint32
if err := conn.QueryRow(ctx, "SELECT to_regclass($1)::oid", schema+".accounts_name_key").Scan(&originalIndex); err != nil {
t.Fatal(err)
}
// Exercise both full-DDL and diff generation against populated data.
for _, fullDDL := range []bool{false, true} {
t.Run(fmt.Sprintf("full_ddl=%v", fullDDL), func(t *testing.T) {
model := liveModel(schema, map[string]string{"name": "integer"})
unique := models.InitConstraint("accounts_name_key", models.UniqueConstraint)
unique.Columns = []string{"name"}
model.Schemas[0].Tables[0].Constraints[unique.Name] = unique
report, err := runLiveWrite(t, connString, model, map[string]interface{}{"full_ddl": fullDDL})
var failure *ExecutionFailure
if !errors.As(err, &failure) || report.FailedStatements != 1 {
t.Fatalf("expected failed conversion, got report=%+v err=%v", report, err)
}
var value, typ string
if err := conn.QueryRow(ctx, "SELECT name, pg_typeof(name)::text FROM "+schema+".accounts WHERE id=1").Scan(&value, &typ); err != nil {
t.Fatal(err)
}
if value != "not-an-integer" || typ != "text" {
t.Fatalf("original column changed: value=%q type=%q", value, typ)
}
var columns int
if err := conn.QueryRow(ctx, "SELECT count(*) FROM information_schema.columns WHERE table_schema=$1 AND table_name='accounts'", schema).Scan(&columns); err != nil || columns != 2 {
t.Fatalf("unexpected replacement column: columns=%d err=%v", columns, err)
}
var index uint32
if err := conn.QueryRow(ctx, "SELECT to_regclass($1)::oid", schema+".accounts_name_key").Scan(&index); err != nil || index != originalIndex {
t.Fatalf("unique index changed: index=%d original=%d err=%v", index, originalIndex, err)
}
if _, err := conn.Exec(ctx, "INSERT INTO "+schema+".accounts VALUES (2, 'not-an-integer')"); err == nil {
t.Fatal("unique constraint no longer enforced")
}
})
}
}
func TestLive_AtomicExecutionRollback(t *testing.T) {
connString := liveWriterConn(t)
schema, conn := liveWriterSchema(t, connString)
w := NewWriter(&writers.WriterOptions{AtomicExecution: true})
w.executionReport = &ExecutionReport{}
err := w.executeStatements([]string{
"-- Schema: " + schema,
"CREATE SCHEMA " + schema,
"CREATE TABLE " + schema + ".created (id int)",
"SELECT 1 / 0",
"SELECT 1",
}, connString)
var failure *ExecutionFailure
if !errors.As(err, &failure) {
t.Fatalf("expected execution failure: %v", err)
}
report := failure.Report
if report.Outcome != "rolled_back" || report.RolledBackStatements != 2 || report.ExecutedStatements != 0 || report.FailedStatements != 1 || report.SkippedStatements != 2 {
t.Fatalf("rollback report: %+v", report)
}
if report.Schemas[0].Tables[0].Created {
t.Fatal("rolled-back table must not be reported created")
}
if columnExists(t, conn, schema, "created", "id") {
t.Fatal("table survived transaction rollback")
}
}
func TestLive_TypePreflightPrecedesDropsEvenWithContinuation(t *testing.T) {
connString := liveWriterConn(t)
schema, conn := liveWriterSchema(t, connString)
_, err := conn.Exec(context.Background(), "CREATE SCHEMA "+schema+"; CREATE TABLE "+schema+".accounts (id integer PRIMARY KEY, name text UNIQUE); INSERT INTO "+schema+".accounts VALUES (1, '123'), (2, 'bad')")
if err != nil {
t.Fatal(err)
}
// The desired model would drop the unique constraint. Incompatible data must
// be caught first, including bad data after the first row.
model := liveModel(schema, map[string]string{"name": "integer"})
w := NewWriter(&writers.WriterOptions{ContinueOnError: true, Metadata: map[string]interface{}{"connection_string": connString}})
err = w.WriteDatabase(model)
var failure *ExecutionFailure
if !errors.As(err, &failure) || failure.Report.FailedStatements != 1 || failure.Report.ExecutedStatements != 0 {
t.Fatalf("preflight must stop mutations: %+v %v", w.executionReport, err)
}
var exists bool
if err := conn.QueryRow(context.Background(), "SELECT to_regclass($1) IS NOT NULL", schema+".accounts_name_key").Scan(&exists); err != nil || !exists {
t.Fatalf("constraint was dropped before preflight: %v", err)
}
}
func TestLive_CustomTypeConversionAndPredicate(t *testing.T) {
connString := liveWriterConn(t)
for _, fullDDL := range []bool{false, true} {
t.Run(fmt.Sprintf("full_ddl=%v", fullDDL), func(t *testing.T) {
schema, conn := liveWriterSchema(t, connString)
_, err := conn.Exec(context.Background(), "CREATE SCHEMA "+schema+"; CREATE TABLE "+schema+".accounts (id integer PRIMARY KEY, name text); INSERT INTO "+schema+".accounts VALUES (1,'42 units'),(2,'7 units')")
if err != nil {
t.Fatal(err)
}
model := liveModel(schema, map[string]string{"name": "integer"})
col := model.Schemas[0].Tables[0].Columns["name"]
col.Metadata["pgsql_using"] = "split_part(name, ' ', 1)::integer"
col.Metadata["pgsql_preflight"] = "name ~ '^[0-9]+ units$' AND name NOT LIKE '7 %'"
meta := map[string]interface{}{"full_ddl": fullDDL}
if _, err := runLiveWrite(t, connString, model, meta); err == nil || !strings.Contains(err.Error(), "preflight rejected rows") {
t.Fatalf("predicate must reject: %v", err)
}
col.Metadata["pgsql_preflight"] = "name ~ '^[0-9]+ units$'"
if _, err := runLiveWrite(t, connString, model, meta); err != nil {
t.Fatal(err)
}
var value int
if err := conn.QueryRow(context.Background(), "SELECT sum(name) FROM "+schema+".accounts").Scan(&value); err != nil || value != 49 {
t.Fatalf("custom conversion result %d: %v", value, err)
}
})
}
}
+153 -39
View File
@@ -61,6 +61,16 @@ func (w *MigrationWriter) GenerateScripts(model, current *models.Database) ([]Mi
current = models.InitDatabase(model.Name)
}
var err error
model, err = prepareModel(model, w.options.LowercaseIdentifiers)
if err != nil {
return nil, err
}
current, err = prepareModel(current, false)
if err != nil {
return nil, err
}
// Check if audit is configured in metadata
var auditConfig *AuditConfig
if w.options.Metadata != nil {
@@ -93,7 +103,7 @@ func (w *MigrationWriter) GenerateScripts(model, current *models.Database) ([]Mi
if cs == nil {
continue
}
if strings.EqualFold(cs.Name, modelSchema.Name) {
if cs.Name == modelSchema.Name {
currentSchema = cs
break
}
@@ -166,7 +176,7 @@ func (w *MigrationWriter) WriteMigration(model, current *models.Database) error
// Write header
fmt.Fprintf(w.writer, "-- PostgreSQL Migration Script\n")
fmt.Fprintf(w.writer, "-- Generated by RelSpec\n")
fmt.Fprintf(w.writer, "-- Source: %s -> %s\n", current.Name, model.Name)
fmt.Fprintf(w.writer, "-- Source: %s -> %s\n", commentText(current.Name), commentText(model.Name))
if w.options.ContinueOnError {
fmt.Fprintf(w.writer, "\\set ON_ERROR_STOP off\n")
}
@@ -175,7 +185,7 @@ func (w *MigrationWriter) WriteMigration(model, current *models.Database) error
// Write scripts
for _, script := range scripts {
fmt.Fprintf(w.writer, "-- Priority: %d | Type: %s | Object: %s\n",
script.Priority, script.ObjectType, script.ObjectName)
script.Priority, commentText(script.ObjectType), commentText(script.ObjectName))
fmt.Fprintf(w.writer, "%s\n\n", script.Body)
}
@@ -222,6 +232,13 @@ func (w *MigrationWriter) generateSchemaScripts(model, current *models.Schema) (
}
scripts = append(scripts, indexScripts...)
// Phase 4.5: Create unique constraints (Priority 185)
uniqueScripts, err := w.generateUniqueConstraintScripts(model, current)
if err != nil {
return nil, fmt.Errorf("failed to generate unique constraint scripts: %w", err)
}
scripts = append(scripts, uniqueScripts...)
// Phase 5: Create foreign keys (Priority 195)
fkScripts, err := w.generateForeignKeyScripts(model, current, droppedFKs)
if err != nil {
@@ -249,12 +266,12 @@ func (w *MigrationWriter) generateDropScripts(model, current *models.Schema) ([]
// Build map of model tables for quick lookup
modelTables := make(map[string]*models.Table)
for _, table := range model.Tables {
modelTables[strings.ToLower(table.Name)] = table
modelTables[table.Name] = table
}
// Find constraints to drop
for _, currentTable := range current.Tables {
modelTable, existsInModel := modelTables[strings.ToLower(currentTable.Name)]
modelTable, existsInModel := modelTables[currentTable.Name]
if !existsInModel {
continue
@@ -289,7 +306,7 @@ func (w *MigrationWriter) generateDropScripts(model, current *models.Schema) ([]
if refSchema == "" {
refSchema = current.Name
}
if strings.EqualFold(refTable, currentTable.Name) && strings.EqualFold(refSchema, current.Name) {
if (refTable == currentTable.Name) && (refSchema == current.Name) {
fkKey := fmt.Sprintf("%s.%s.%s", current.Name, otherTable.Name, fkName)
if !droppedFKs[fkKey] {
droppedFKs[fkKey] = true
@@ -388,13 +405,13 @@ func (w *MigrationWriter) generateTableScripts(model, current *models.Schema) ([
currentTables := make(map[string]*models.Table)
if current != nil {
for _, table := range current.Tables {
currentTables[strings.ToLower(table.Name)] = table
currentTables[table.Name] = table
}
}
// Process each model table
for _, modelTable := range model.Tables {
currentTable, exists := currentTables[strings.ToLower(modelTable.Name)]
currentTable, exists := currentTables[modelTable.Name]
if !exists {
// Table doesn't exist, create it
@@ -432,12 +449,12 @@ func (w *MigrationWriter) generateAlterTableScripts(schema *models.Schema, model
// Build map of current columns
currentColumns := make(map[string]*models.Column)
for name, col := range currentTable.Columns {
currentColumns[strings.ToLower(name)] = col
currentColumns[name] = col
}
// Check each model column
for _, modelCol := range sortColumns(modelTable.Columns) {
currentCol, exists := currentColumns[strings.ToLower(modelCol.Name)]
currentCol, exists := currentColumns[modelCol.Name]
if !exists {
// Column doesn't exist, add it
@@ -480,15 +497,16 @@ func (w *MigrationWriter) generateAlterTableScripts(schema *models.Schema, model
} else if !columnsEqual(modelCol, currentCol) {
// Column exists but properties changed
if !columnTypesEqual(modelCol, currentCol) {
newType := effectiveAlterColumnSQLType(modelCol)
sql, err := w.executor.ExecuteAlterColumnTypeWithCheck(AlterColumnTypeWithCheckData{
SchemaName: schema.Name,
TableName: modelTable.Name,
ColumnName: modelCol.Name,
NewType: newType,
EquivalentTypes: equivalentTypeListSQL(newType),
UsingExpr: buildAlterColumnUsingExpression(modelCol.Name, newType),
data, err := conversionData(schema.Name, modelTable.Name, modelCol)
if err != nil {
return nil, err
}
scripts = append(scripts, MigrationScript{
ObjectName: fmt.Sprintf("%s.%s.%s", schema.Name, modelTable.Name, modelCol.Name),
ObjectType: "preflight column type", Schema: schema.Name, Priority: -10000,
Sequence: len(scripts), Body: preflightColumnType(data, w.options.FlattenSchema),
})
sql, err := w.executor.ExecuteAlterColumnTypeWithCheck(data)
if err != nil {
return nil, err
}
@@ -566,13 +584,13 @@ func (w *MigrationWriter) generateIndexScripts(model, current *models.Schema) ([
currentTables := make(map[string]*models.Table)
if current != nil {
for _, table := range current.Tables {
currentTables[strings.ToLower(table.Name)] = table
currentTables[table.Name] = table
}
}
// Process each model table
for _, modelTable := range model.Tables {
currentTable := currentTables[strings.ToLower(modelTable.Name)]
currentTable := currentTables[modelTable.Name]
// Process primary keys first - check explicit constraints
foundExplicitPK := false
@@ -595,7 +613,7 @@ func (w *MigrationWriter) generateIndexScripts(model, current *models.Schema) ([
SchemaName: model.Name,
TableName: modelTable.Name,
ConstraintName: constraintName,
Columns: strings.Join(constraint.Columns, ", "),
Columns: quotedIdentifiers(constraint.Columns),
})
if err != nil {
return nil, err
@@ -619,12 +637,12 @@ func (w *MigrationWriter) generateIndexScripts(model, current *models.Schema) ([
pkColumns := []string{}
for _, col := range modelTable.Columns {
if col.IsPrimaryKey {
pkColumns = append(pkColumns, col.SQLName())
pkColumns = append(pkColumns, col.Name)
}
}
if len(pkColumns) > 0 {
sort.Strings(pkColumns)
constraintName := fmt.Sprintf("pk_%s_%s", model.SQLName(), modelTable.SQLName())
constraintName := fmt.Sprintf("pk_%s_%s", model.Name, modelTable.Name)
shouldCreate := true
if currentTable != nil {
@@ -642,7 +660,7 @@ func (w *MigrationWriter) generateIndexScripts(model, current *models.Schema) ([
SchemaName: model.Name,
TableName: modelTable.Name,
ConstraintName: constraintName,
Columns: strings.Join(pkColumns, ", "),
Columns: quotedIdentifiers(pkColumns),
})
if err != nil {
return nil, err
@@ -742,7 +760,7 @@ func buildIndexColumnExpressionsFiltered(table *models.Table, index *models.Inde
continue
}
colExpr := col.SQLName()
colExpr := quoteIdent(col.Name)
if opClass := indexOperatorClassForColumn(col, indexType, index.Comment); opClass != "" {
colExpr = fmt.Sprintf("%s %s", colExpr, opClass)
}
@@ -751,6 +769,92 @@ func buildIndexColumnExpressionsFiltered(table *models.Table, index *models.Inde
return columnExprs
}
// generateUniqueConstraintScripts generates ADD CONSTRAINT UNIQUE scripts for model unique
// constraints that are missing from, or differ in, the live schema.
func (w *MigrationWriter) generateUniqueConstraintScripts(model, current *models.Schema) ([]MigrationScript, error) {
scripts := make([]MigrationScript, 0)
currentTables := make(map[string]*models.Table)
if current != nil {
for _, table := range current.Tables {
currentTables[table.Name] = table
}
}
for _, modelTable := range model.Tables {
currentTable := currentTables[modelTable.Name]
for _, constraint := range sortConstraints(modelTable.Constraints) {
if constraint.Type != models.UniqueConstraint {
continue
}
if currentTable != nil {
if currentConstraint, exists := lookupConstraint(currentTable.Constraints, constraint.Name); exists && constraintsEqual(constraint, currentConstraint) {
continue
}
}
columnExprs := make([]string, 0, len(constraint.Columns))
for _, colName := range constraint.Columns {
if col, ok := modelTable.Columns[colName]; ok {
columnExprs = append(columnExprs, quoteIdent(col.Name))
}
}
if len(columnExprs) == 0 {
continue
}
sql, err := w.executor.ExecuteCreateUniqueConstraint(CreateUniqueConstraintData{
SchemaName: model.Name,
TableName: modelTable.Name,
ConstraintName: constraint.Name,
Columns: strings.Join(columnExprs, ", "),
UsingIndex: adoptableUniqueIndex(modelTable, constraint),
})
if err != nil {
return nil, err
}
scripts = append(scripts, MigrationScript{
ObjectName: fmt.Sprintf("%s.%s.%s", model.Name, modelTable.Name, constraint.Name),
ObjectType: "create unique constraint",
Schema: model.Name,
Priority: 185,
Sequence: len(scripts),
Body: sql,
})
}
}
return scripts, nil
}
// adoptableUniqueIndex returns the name of the model's unique index that the given unique
// constraint can be built on with ADD CONSTRAINT ... UNIQUE USING INDEX, so the constraint does
// not create a second index (or collide with the index's name). The index must carry the
// constraint's name, be a plain non-partial btree index and cover exactly the constraint's
// columns. Index creation runs before constraint creation, so this holds whether or not the
// live database already has the index. Returns "" when none qualifies.
func adoptableUniqueIndex(table *models.Table, constraint *models.Constraint) string {
index, ok := table.Indexes[constraint.Name]
if !ok || !index.Unique || index.Where != "" || len(index.Include) > 0 {
return ""
}
if index.Type != "" && !strings.EqualFold(index.Type, "btree") {
return ""
}
if len(index.Columns) != len(constraint.Columns) {
return ""
}
for i, col := range index.Columns {
if col != constraint.Columns[i] {
return ""
}
}
return index.Name
}
// generateForeignKeyScripts generates ADD CONSTRAINT FOREIGN KEY scripts using templates.
// forceRecreate is a set of FK constraint keys (schema.table.name) that must be recreated
// even if unchanged, because their referenced PK was dropped and recreated.
@@ -761,13 +865,13 @@ func (w *MigrationWriter) generateForeignKeyScripts(model, current *models.Schem
currentTables := make(map[string]*models.Table)
if current != nil {
for _, table := range current.Tables {
currentTables[strings.ToLower(table.Name)] = table
currentTables[table.Name] = table
}
}
// Process each model table
for _, modelTable := range model.Tables {
currentTable := currentTables[strings.ToLower(modelTable.Name)]
currentTable := currentTables[modelTable.Name]
// Process each constraint
for _, constraint := range sortConstraints(modelTable.Constraints) {
@@ -800,16 +904,23 @@ func (w *MigrationWriter) generateForeignKeyScripts(model, current *models.Schem
onUpdate = strings.ToUpper(constraint.OnUpdate)
}
// An unspecified deferral keeps the historical default of DEFERRABLE.
deferral := constraint.DeferralClause()
if deferral == "" {
deferral = "DEFERRABLE"
}
sql, err := w.executor.ExecuteCreateForeignKey(CreateForeignKeyData{
SchemaName: model.Name,
TableName: modelTable.Name,
ConstraintName: constraintName,
SourceColumns: strings.Join(constraint.Columns, ", "),
SourceColumns: quotedIdentifiers(constraint.Columns),
TargetSchema: constraint.ReferencedSchema,
TargetTable: constraint.ReferencedTable,
TargetColumns: strings.Join(constraint.ReferencedColumns, ", "),
TargetColumns: quotedIdentifiers(constraint.ReferencedColumns),
OnDelete: onDelete,
OnUpdate: onUpdate,
Deferral: deferral,
})
if err != nil {
return nil, err
@@ -838,20 +949,20 @@ func (w *MigrationWriter) generateCommentScripts(model, current *models.Schema)
currentTables := make(map[string]*models.Table)
if current != nil {
for _, table := range current.Tables {
currentTables[strings.ToLower(table.Name)] = table
currentTables[table.Name] = table
}
}
// Process each model table
for _, modelTable := range model.Tables {
currentTable := currentTables[strings.ToLower(modelTable.Name)]
currentTable := currentTables[modelTable.Name]
// Table comment (skipped when the live table already carries the same comment)
if modelTable.Description != "" && (currentTable == nil || strings.TrimSpace(currentTable.Description) != strings.TrimSpace(modelTable.Description)) {
sql, err := w.executor.ExecuteCommentTable(CommentTableData{
SchemaName: model.Name,
TableName: modelTable.Name,
Comment: escapeQuote(modelTable.Description),
Comment: modelTable.Description,
})
if err != nil {
return nil, err
@@ -875,7 +986,7 @@ func (w *MigrationWriter) generateCommentScripts(model, current *models.Schema)
SchemaName: model.Name,
TableName: modelTable.Name,
ColumnName: col.Name,
Comment: escapeQuote(col.Description),
Comment: col.Description,
})
if err != nil {
return nil, err
@@ -1128,27 +1239,30 @@ func constraintsEqual(c1, c2 *models.Constraint) bool {
return false
}
for i, col := range c1.Columns {
if !strings.EqualFold(col, c2.Columns[i]) {
if col != c2.Columns[i] {
return false
}
}
// For foreign keys, also compare referenced table and columns
if c1.Type == models.ForeignKeyConstraint {
if !strings.EqualFold(c1.ReferencedTable, c2.ReferencedTable) {
if c1.ReferencedTable != c2.ReferencedTable {
return false
}
if len(c1.ReferencedColumns) != len(c2.ReferencedColumns) {
return false
}
for i, col := range c1.ReferencedColumns {
if !strings.EqualFold(col, c2.ReferencedColumns[i]) {
if col != c2.ReferencedColumns[i] {
return false
}
}
if !fkActionsEqual(c1.OnDelete, c2.OnDelete) || !fkActionsEqual(c1.OnUpdate, c2.OnUpdate) {
return false
}
if !models.DeferralMatches(c1, c2) {
return false
}
}
return true
@@ -1169,7 +1283,7 @@ func indexesEqual(idx1, idx2 *models.Index) bool {
return false
}
for i, col := range idx1.Columns {
if !strings.EqualFold(col, idx2.Columns[i]) {
if col != idx2.Columns[i] {
return false
}
}
@@ -1223,7 +1337,7 @@ func currentColumnHasDescription(currentTable *models.Table, col *models.Column)
return false
}
for _, cc := range currentTable.Columns {
if strings.EqualFold(cc.Name, col.Name) {
if cc.Name == col.Name {
return strings.TrimSpace(cc.Description) == strings.TrimSpace(col.Description)
}
}
@@ -1251,7 +1365,7 @@ func primaryKeyColumnsMatch(modelTable *models.Table, current *models.Constraint
return false
}
for i, col := range modelCols {
if !strings.EqualFold(col, current.Columns[i]) {
if col != current.Columns[i] {
return false
}
}
+345 -36
View File
@@ -89,7 +89,7 @@ func TestWriteMigration_ArrayDefault(t *testing.T) {
}
output := buf.String()
if !strings.Contains(output, "tags text[] DEFAULT '{}' NOT NULL") {
if !strings.Contains(output, "\"tags\" text[] DEFAULT '{}' NOT NULL") {
t.Fatalf("expected normalized array default in migration, got:\n%s", output)
}
if strings.Contains(output, "'''{}'''") {
@@ -128,15 +128,15 @@ func TestWriteMigration_AltersColumnTypeWhenActualTypeDiffers(t *testing.T) {
}
output := buf.String()
if !strings.Contains(output, "ALTER TABLE public.learnings") || !strings.Contains(output, "ALTER COLUMN details TYPE text") {
if !strings.Contains(output, "ALTER TABLE \"public\".\"learnings\"") || !strings.Contains(output, "ALTER COLUMN \"details\" TYPE text") {
t.Fatalf("expected migration to alter mismatched column type, got:\n%s", output)
}
if !strings.Contains(output, `ALTER COLUMN details TYPE text USING details::text;`) {
if !strings.Contains(output, `ALTER COLUMN "details" TYPE text USING "details"::text;`) {
t.Fatalf("expected migration type alter to include USING cast, got:\n%s", output)
}
}
func TestWriteMigration_AltersColumnTypeFallsBackToRenameAndAddOnConversionFailure(t *testing.T) {
func TestWriteMigration_AltersColumnTypePreservesConversionFailure(t *testing.T) {
current := models.InitDatabase("testdb")
currentSchema := models.InitSchema("public")
currentTable := models.InitTable("learnings", "public")
@@ -167,17 +167,13 @@ func TestWriteMigration_AltersColumnTypeFallsBackToRenameAndAddOnConversionFailu
}
output := buf.String()
if !strings.Contains(output, "EXCEPTION WHEN OTHERS THEN") {
t.Fatalf("expected migration to guard the type conversion with an exception handler, got:\n%s", output)
if !strings.Contains(output, "ALTER COLUMN \"details\" TYPE integer USING \"details\"::integer;") {
t.Fatalf("expected explicit type conversion, got:\n%s", output)
}
if !strings.Contains(output, "RENAME COLUMN details TO %I") {
t.Fatalf("expected migration to rename the old column (derived from the live type) on conversion failure, got:\n%s", output)
}
if !strings.Contains(output, "renamed_column := 'details_' || trim(both '_' from regexp_replace(lower(current_type)") {
t.Fatalf("expected migration to derive the renamed column name from the live type, got:\n%s", output)
}
if !strings.Contains(output, "ADD COLUMN details integer") {
t.Fatalf("expected migration to add a fresh column with the new type on conversion failure, got:\n%s", output)
for _, unsafe := range []string{"EXCEPTION WHEN OTHERS", "renamed_column", "RENAME COLUMN"} {
if strings.Contains(output, unsafe) {
t.Fatalf("type conversion must propagate errors without replacing the column (%s):\n%s", unsafe, output)
}
}
}
@@ -214,7 +210,7 @@ func TestWriteMigration_AltersColumnNullabilityWhenNotNullDiffers(t *testing.T)
}
output := buf.String()
if !strings.Contains(output, "ALTER COLUMN rid_service_instance_type DROP NOT NULL") {
if !strings.Contains(output, "ALTER COLUMN \"rid_service_instance_type\" DROP NOT NULL") {
t.Fatalf("expected migration to drop NOT NULL on existing column, got:\n%s", output)
}
}
@@ -335,13 +331,13 @@ func TestWriteMigration_UsesStorageTypeForSerialAlterStatements(t *testing.T) {
}
output := buf.String()
if !strings.Contains(output, "ALTER COLUMN id TYPE bigint") {
if !strings.Contains(output, "ALTER COLUMN \"id\" TYPE bigint") {
t.Fatalf("expected serial alter to use bigint storage type, got:\n%s", output)
}
if strings.Contains(output, "ALTER COLUMN id TYPE bigserial;") {
if strings.Contains(output, "ALTER COLUMN \"id\" TYPE bigserial;") {
t.Fatalf("did not expect invalid bigserial alter statement, got:\n%s", output)
}
if !strings.Contains(output, `ALTER COLUMN id TYPE bigint USING id::bigint;`) {
if !strings.Contains(output, `ALTER COLUMN "id" TYPE bigint USING "id"::bigint;`) {
t.Fatalf("expected serial alter to include USING cast, got:\n%s", output)
}
}
@@ -377,7 +373,7 @@ func TestWriteMigration_ArrayAlterIncludesUsingCast(t *testing.T) {
}
output := buf.String()
if !strings.Contains(output, `ALTER COLUMN tags TYPE text[] USING tags::text[];`) {
if !strings.Contains(output, `ALTER COLUMN "tags" TYPE text[] USING "tags"::text[];`) {
t.Fatalf("expected array alter to include USING cast, got:\n%s", output)
}
}
@@ -414,7 +410,7 @@ func TestWriteMigration_DoesNotAlterEquivalentNormalizedColumnType(t *testing.T)
}
output := buf.String()
if strings.Contains(output, "ALTER COLUMN email TYPE") {
if strings.Contains(output, "ALTER COLUMN \"email\" TYPE") {
t.Fatalf("did not expect alter type for equivalent normalized types, got:\n%s", output)
}
}
@@ -497,7 +493,7 @@ func TestWriteMigration_GinIndexOnTextUsesTrigramOperatorClass(t *testing.T) {
if !strings.Contains(output, "CREATE EXTENSION IF NOT EXISTS pg_trgm;") {
t.Fatalf("expected trigram extension for text GIN migration index, got:\n%s", output)
}
if !strings.Contains(output, "USING gin (title gin_trgm_ops)") {
if !strings.Contains(output, "USING gin (\"title\" gin_trgm_ops)") {
t.Fatalf("expected GIN text index to include gin_trgm_ops, got:\n%s", output)
}
}
@@ -518,7 +514,7 @@ func TestWriteMigration_GinIndexOnQuotedTextColumnUsesTrigramOperatorClass(t *te
index := &models.Index{
Name: "idx_agent_personas_name_gin",
Type: "gin",
Columns: []string{`"name"`},
Columns: []string{"name"},
}
table.Indexes[index.Name] = index
@@ -537,7 +533,7 @@ func TestWriteMigration_GinIndexOnQuotedTextColumnUsesTrigramOperatorClass(t *te
}
output := buf.String()
if !strings.Contains(output, "USING gin (name gin_trgm_ops)") {
if !strings.Contains(output, "USING gin (\"name\" gin_trgm_ops)") {
t.Fatalf("expected quoted text column GIN index to include gin_trgm_ops, got:\n%s", output)
}
}
@@ -577,7 +573,7 @@ func TestWriteMigration_GinIndexOnTextArrayDoesNotUseTrigramOperatorClass(t *tes
}
output := buf.String()
if !strings.Contains(output, "USING gin (tags array_ops)") {
if !strings.Contains(output, "USING gin (\"tags\" array_ops)") {
t.Fatalf("expected GIN array index with array_ops, got:\n%s", output)
}
if strings.Contains(output, "gin_trgm_ops") {
@@ -620,7 +616,7 @@ func TestWriteMigration_GinIndexOnJSONBUsesJSONBOperatorClass(t *testing.T) {
}
output := buf.String()
if !strings.Contains(output, "USING gin (details jsonb_ops)") {
if !strings.Contains(output, "USING gin (\"details\" jsonb_ops)") {
t.Fatalf("expected GIN jsonb index to include jsonb_ops, got:\n%s", output)
}
if strings.Contains(output, "gin_trgm_ops") {
@@ -664,7 +660,7 @@ func TestWriteMigration_GinIndexOnJSONBIgnoresIncompatibleTrigramOperatorClass(t
}
output := buf.String()
if !strings.Contains(output, "USING gin (details jsonb_ops)") {
if !strings.Contains(output, "USING gin (\"details\" jsonb_ops)") {
t.Fatalf("expected incompatible trigram hint on jsonb to fall back to jsonb_ops, got:\n%s", output)
}
}
@@ -725,20 +721,20 @@ func TestWriteMigration_WithAudit(t *testing.T) {
t.Logf("Generated migration with audit:\n%s", output)
// Verify audit tables
if !strings.Contains(output, "CREATE TABLE IF NOT EXISTS public.atevent") {
if !strings.Contains(output, "CREATE TABLE IF NOT EXISTS \"public\".atevent") {
t.Error("Migration missing atevent table")
}
if !strings.Contains(output, "CREATE TABLE IF NOT EXISTS public.atdetail") {
if !strings.Contains(output, "CREATE TABLE IF NOT EXISTS \"public\".atdetail") {
t.Error("Migration missing atdetail table")
}
// Verify audit function
if !strings.Contains(output, "CREATE OR REPLACE FUNCTION public.tf_audit_users()") {
if !strings.Contains(output, "CREATE OR REPLACE FUNCTION \"public\".\"tf_audit_users\"()") {
t.Error("Migration missing audit function")
}
// Verify audit trigger
if !strings.Contains(output, "CREATE TRIGGER t_audit_users") {
if !strings.Contains(output, "CREATE TRIGGER \"t_audit_users\"") {
t.Error("Migration missing audit trigger")
}
@@ -770,13 +766,13 @@ func TestTemplateExecutor_CreateTable(t *testing.T) {
t.Logf("Generated SQL:\n%s", sql)
if !strings.Contains(sql, "CREATE TABLE IF NOT EXISTS public.test_table") {
if !strings.Contains(sql, "CREATE TABLE IF NOT EXISTS \"public\".\"test_table\"") {
t.Error("SQL missing CREATE TABLE statement")
}
if !strings.Contains(sql, "id integer NOT NULL") {
if !strings.Contains(sql, "\"id\" integer NOT NULL") {
t.Error("SQL missing id column definition")
}
if !strings.Contains(sql, "name text DEFAULT 'unknown'") {
if !strings.Contains(sql, "\"name\" text DEFAULT 'unknown'") {
t.Error("SQL missing name column definition")
}
}
@@ -814,7 +810,7 @@ func TestTemplateExecutor_AuditFunction(t *testing.T) {
t.Logf("Generated SQL:\n%s", sql)
if !strings.Contains(sql, "CREATE OR REPLACE FUNCTION public.tf_audit_users()") {
if !strings.Contains(sql, "CREATE OR REPLACE FUNCTION \"public\".\"tf_audit_users\"()") {
t.Error("SQL missing function definition")
}
if !strings.Contains(sql, "IF TG_OP = 'INSERT'") {
@@ -987,8 +983,8 @@ func TestWriteMigration_VectorAndPostGISIndexes(t *testing.T) {
"CREATE EXTENSION IF NOT EXISTS vector;",
"vector(1536)",
"geometry(Point,4326)",
"USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100)",
"USING gist (location)",
"USING ivfflat (\"embedding\" vector_cosine_ops) WITH (lists = 100)",
"USING gist (\"location\")",
} {
if !strings.Contains(output, want) {
t.Fatalf("expected migration to contain %q, got:\n%s", want, output)
@@ -1027,3 +1023,316 @@ func TestIndexesEqual_OperatorClassAndStorageParameters(t *testing.T) {
})
}
}
func uniqueConstraintTestModel() *models.Database {
model := models.InitDatabase("testdb")
schema := models.InitSchema("public")
table := models.InitTable("login", "public")
idCol := models.InitColumn("id", "login", "public")
idCol.Type = "integer"
idCol.NotNull = true
table.Columns["id"] = idCol
guidCol := models.InitColumn("guid", "login", "public")
guidCol.Type = "uuid"
table.Columns["guid"] = guidCol
uc := models.InitConstraint("ukey_login_guid", models.UniqueConstraint)
uc.Schema = "public"
uc.Table = "login"
uc.Columns = []string{"guid"}
table.Constraints[uc.Name] = uc
schema.Tables = append(schema.Tables, table)
model.Schemas = append(model.Schemas, schema)
return model
}
func TestWriteMigration_UniqueConstraint(t *testing.T) {
tests := []struct {
name string
current func() *models.Database
wantConstrnt bool
}{
{
name: "new table",
current: func() *models.Database {
c := models.InitDatabase("testdb")
c.Schemas = append(c.Schemas, models.InitSchema("public"))
return c
},
wantConstrnt: true,
},
{
name: "existing table without constraint",
current: func() *models.Database {
c := uniqueConstraintTestModel()
delete(c.Schemas[0].Tables[0].Constraints, "ukey_login_guid")
return c
},
wantConstrnt: true,
},
{
name: "existing table with constraint",
current: uniqueConstraintTestModel,
wantConstrnt: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var buf bytes.Buffer
writer, err := NewMigrationWriter(&writers.WriterOptions{})
if err != nil {
t.Fatalf("Failed to create writer: %v", err)
}
writer.writer = &buf
if err := writer.WriteMigration(uniqueConstraintTestModel(), tt.current()); err != nil {
t.Fatalf("WriteMigration failed: %v", err)
}
output := buf.String()
has := strings.Contains(output, "ADD CONSTRAINT") && strings.Contains(output, "ukey_login_guid") && strings.Contains(output, "UNIQUE")
if has != tt.wantConstrnt {
t.Fatalf("unique constraint present = %v, want %v; output:\n%s", has, tt.wantConstrnt, output)
}
if strings.Contains(output, "DROP CONSTRAINT") {
t.Fatalf("unexpected DROP CONSTRAINT; output:\n%s", output)
}
})
}
}
func TestWriteMigration_UniqueConstraintAdoptsSameNamedModelIndex(t *testing.T) {
newIndex := func(name string, unique bool, where string) *models.Index {
idx := models.InitIndex(name, "login", "public")
idx.Columns = []string{"guid"}
idx.Unique = unique
idx.Type = "btree"
idx.Where = where
return idx
}
tests := []struct {
name string
modelIdx *models.Index
liveIdx *models.Index // nil: fresh table
wantAdopt bool
wantDrop bool
}{
{
name: "fresh table: same-named unique index is promoted",
modelIdx: newIndex("ukey_login_guid", true, ""),
wantAdopt: true,
},
{
name: "existing live unique index is promoted",
modelIdx: newIndex("ukey_login_guid", true, ""),
liveIdx: newIndex("ukey_login_guid", true, ""),
wantAdopt: true,
},
{
name: "live unique index not in model is dropped and constraint built fresh",
liveIdx: newIndex("ukey_login_guid", true, ""),
wantDrop: true,
},
{
name: "partial model index is not promoted",
modelIdx: newIndex("ukey_login_guid", true, "guid IS NOT NULL"),
},
{
name: "non-unique same-named index is not promoted",
modelIdx: newIndex("ukey_login_guid", false, ""),
},
{
name: "differently named index is not promoted",
modelIdx: newIndex("idx_login_guid", true, ""),
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
model := uniqueConstraintTestModel()
if tt.modelIdx != nil {
model.Schemas[0].Tables[0].Indexes[tt.modelIdx.Name] = tt.modelIdx
}
current := models.InitDatabase("testdb")
current.Schemas = append(current.Schemas, models.InitSchema("public"))
if tt.liveIdx != nil {
current = uniqueConstraintTestModel()
curTable := current.Schemas[0].Tables[0]
delete(curTable.Constraints, "ukey_login_guid")
curTable.Indexes[tt.liveIdx.Name] = tt.liveIdx
}
var buf bytes.Buffer
writer, err := NewMigrationWriter(&writers.WriterOptions{})
if err != nil {
t.Fatalf("Failed to create writer: %v", err)
}
writer.writer = &buf
if err := writer.WriteMigration(model, current); err != nil {
t.Fatalf("WriteMigration failed: %v", err)
}
output := buf.String()
if got := strings.Contains(output, "UNIQUE USING INDEX"); got != tt.wantAdopt {
t.Errorf("USING INDEX present = %v, want %v; output:\n%s", got, tt.wantAdopt, output)
}
if got := strings.Contains(output, "UNIQUE (\"guid\")"); got == tt.wantAdopt {
t.Errorf("column-list ADD CONSTRAINT present = %v, want %v; output:\n%s", got, !tt.wantAdopt, output)
}
if got := strings.Contains(output, "DROP INDEX"); got != tt.wantDrop {
t.Errorf("DROP INDEX present = %v, want %v; output:\n%s", got, tt.wantDrop, output)
}
})
}
}
func TestWriteDatabase_UniqueConstraintAdoptsSameNamedIndex(t *testing.T) {
db := uniqueConstraintTestModel()
idx := models.InitIndex("ukey_login_guid", "login", "public")
idx.Columns = []string{"guid"}
idx.Unique = true
db.Schemas[0].Tables[0].Indexes[idx.Name] = idx
var buf bytes.Buffer
w := NewWriter(&writers.WriterOptions{})
w.writer = &buf
if err := w.WriteDatabase(db); err != nil {
t.Fatalf("WriteDatabase failed: %v", err)
}
output := buf.String()
if !strings.Contains(output, "UNIQUE USING INDEX") {
t.Fatalf("expected UNIQUE USING INDEX in output:\n%s", output)
}
if strings.Index(output, "CREATE UNIQUE INDEX") > strings.Index(output, "UNIQUE USING INDEX") {
t.Fatalf("index must be created before the constraint adopts it:\n%s", output)
}
}
func fkDeferralTestModel(def, init *bool) *models.Database {
db := models.InitDatabase("testdb")
schema := models.InitSchema("public")
users := models.InitTable("users", "public")
uid := models.InitColumn("id", "users", "public")
uid.Type = "integer"
uid.NotNull = true
users.Columns["id"] = uid
posts := models.InitTable("posts", "public")
pid := models.InitColumn("id", "posts", "public")
pid.Type = "integer"
pid.NotNull = true
posts.Columns["id"] = pid
puid := models.InitColumn("user_id", "posts", "public")
puid.Type = "integer"
posts.Columns["user_id"] = puid
fk := models.InitConstraint("fk_posts_user_id", models.ForeignKeyConstraint)
fk.Schema = "public"
fk.Table = "posts"
fk.Columns = []string{"user_id"}
fk.ReferencedSchema = "public"
fk.ReferencedTable = "users"
fk.ReferencedColumns = []string{"id"}
fk.Deferrable = def
fk.InitiallyDeferred = init
posts.Constraints[fk.Name] = fk
schema.Tables = append(schema.Tables, users, posts)
db.Schemas = append(db.Schemas, schema)
return db
}
func migrationOutput(t *testing.T, model, current *models.Database) string {
t.Helper()
var buf bytes.Buffer
writer, err := NewMigrationWriter(&writers.WriterOptions{})
if err != nil {
t.Fatalf("Failed to create writer: %v", err)
}
writer.writer = &buf
if err := writer.WriteMigration(model, current); err != nil {
t.Fatalf("WriteMigration failed: %v", err)
}
return buf.String()
}
func TestWriteMigration_ForeignKeyDeferral(t *testing.T) {
yes, no := true, false
empty := func() *models.Database {
c := models.InitDatabase("testdb")
c.Schemas = append(c.Schemas, models.InitSchema("public"))
return c
}
tests := []struct {
name string
def *bool
init *bool
wantFrag string
}{
{"unspecified keeps DEFERRABLE default", nil, nil, "ON UPDATE NO ACTION\n DEFERRABLE;"},
{"deferrable", &yes, nil, "DEFERRABLE;"},
{"initially deferred", &yes, &yes, "DEFERRABLE INITIALLY DEFERRED;"},
{"not deferrable", &no, nil, "NOT DEFERRABLE;"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
out := migrationOutput(t, fkDeferralTestModel(tt.def, tt.init), empty())
if !strings.Contains(out, tt.wantFrag) {
t.Fatalf("expected %q in output:\n%s", tt.wantFrag, out)
}
})
}
}
func TestWriteMigration_ForeignKeyDeferralDiff(t *testing.T) {
yes, no := true, false
live := func(def, init bool) *models.Database {
db := fkDeferralTestModel(&def, &init)
return db
}
tests := []struct {
name string
def, init *bool
liveDef bool
liveInit bool
wantRecreate bool
}{
{"unspecified vs deferrable live: untouched", nil, nil, true, false, false},
{"unspecified vs not deferrable live: untouched", nil, nil, false, false, false},
{"deferrable matches live", &yes, nil, true, true, false},
{"deferrable vs not deferrable live: recreated", &yes, nil, false, false, true},
{"not deferrable vs deferrable live: recreated", &no, nil, true, false, true},
{"initially deferred vs initially immediate live: recreated", &yes, &yes, true, false, true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
out := migrationOutput(t, fkDeferralTestModel(tt.def, tt.init), live(tt.liveDef, tt.liveInit))
got := strings.Contains(out, "ADD CONSTRAINT") && strings.Contains(out, "FOREIGN KEY")
if got != tt.wantRecreate {
t.Fatalf("FK recreated = %v, want %v; output:\n%s", got, tt.wantRecreate, out)
}
})
}
}
func TestWriteDatabase_ForeignKeyDeferral(t *testing.T) {
yes := true
var buf bytes.Buffer
w := NewWriter(&writers.WriterOptions{})
w.writer = &buf
if err := w.WriteDatabase(fkDeferralTestModel(&yes, &yes)); err != nil {
t.Fatalf("WriteDatabase failed: %v", err)
}
if !strings.Contains(buf.String(), "DEFERRABLE INITIALLY DEFERRED;") {
t.Fatalf("expected DEFERRABLE INITIALLY DEFERRED in output:\n%s", buf.String())
}
}
+251
View File
@@ -0,0 +1,251 @@
package pgsql
import (
"bytes"
"context"
"os"
"path/filepath"
"strings"
"testing"
"git.warky.dev/wdevs/relspecgo/pkg/models"
"git.warky.dev/wdevs/relspecgo/pkg/writers"
)
func TestQuotedNamesAndLowercaseOption(t *testing.T) {
db := models.InitDatabase("DB\nSELECT 'header';")
s := models.InitSchema("MySchema")
tbl := models.InitTable("Select", "MySchema")
col := models.InitColumn("Key", "Select", "MySchema")
col.Type = "text"
col.Default = "Keep CASE"
tbl.Columns[col.Name] = col
s.Tables = append(s.Tables, tbl)
db.Schemas = append(db.Schemas, s)
for _, lower := range []bool{false, true} {
w := NewWriter(&writers.WriterOptions{LowercaseIdentifiers: lower})
stmts, err := w.GenerateDatabaseStatements(db)
if err != nil {
t.Fatal(err)
}
out := strings.Join(stmts, "\n")
want := `"MySchema"."Select"`
if lower {
want = `"myschema"."select"`
}
if !strings.Contains(out, want) || !strings.Contains(out, "'Keep CASE'") || strings.Contains(out, "\nSELECT 'header';") {
t.Fatalf("%s", out)
}
}
if db.Name != "DB\nSELECT 'header';" || tbl.Name != "Select" {
t.Fatal("input changed")
}
path := filepath.Join(t.TempDir(), "out.sql")
if err := os.WriteFile(path, []byte("keep"), 0o600); err != nil {
t.Fatal(err)
}
col.Name = "bad\x00name"
if err := NewWriter(&writers.WriterOptions{OutputPath: path}).WriteDatabase(db); err == nil {
t.Fatal("expected NUL rejection")
}
data, _ := os.ReadFile(path)
if string(data) != "keep" {
t.Fatal("output changed before validation")
}
}
func TestAllBlockTemplatesAvoidDelimiterCollisions(t *testing.T) {
te, err := NewTemplateExecutor(false)
if err != nil {
t.Fatal(err)
}
name := "名'\"\\\n$$ $body$ $relspec_0$ $relspec_1$"
cases := []struct {
name string
data any
}{
{"alter_column_default_with_check.tmpl", AlterColumnDefaultWithCheckData{SchemaName: name, TableName: name, ColumnName: name, SetDefault: true, DefaultValue: quote(name)}},
{"alter_column_nullability_with_check.tmpl", AlterColumnNullabilityWithCheckData{SchemaName: name, TableName: name, ColumnName: name}},
{"create_check_constraint.tmpl", CreateCheckConstraintData{SchemaName: name, TableName: name, ConstraintName: name, Expression: "true"}},
{"create_foreign_key_with_check.tmpl", CreateForeignKeyWithCheckData{SchemaName: name, TableName: name, ConstraintName: name}},
{"create_primary_key.tmpl", CreatePrimaryKeyData{SchemaName: name, TableName: name, ConstraintName: name}},
{"create_primary_key_with_autogen_check.tmpl", CreatePrimaryKeyWithAutoGenCheckData{SchemaName: name, TableName: name, ConstraintName: name}},
{"create_unique_constraint.tmpl", CreateUniqueConstraintData{SchemaName: name, TableName: name, ConstraintName: name}},
{"set_sequence_value.tmpl", SetSequenceValueData{SchemaName: name, TableName: name, SequenceName: name, ColumnName: name}},
{"audit_function.tmpl", AuditFunctionData{SchemaName: name, FunctionName: name, TableName: name, PrimaryKey: name, AuditSchema: name}},
{"audit_trigger.tmpl", AuditTriggerData{SchemaName: name, TableName: name, TriggerName: name, FunctionName: name}},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
out, err := te.execute(tc.name, tc.data)
if err != nil {
t.Fatal(err)
}
if strings.Count(out, "$relspec_2$") != 2 || !strings.Contains(out, quoteIdent(name)) || !strings.Contains(out, quote(name)) {
t.Fatalf("%s", out)
}
})
}
}
func TestLive_AdversarialNamesAndAudit(t *testing.T) {
connString := liveWriterConn(t)
base, conn := liveWriterSchema(t, connString)
ctx := context.Background()
schemaName := base + "'\"\\\n名$relspec_0$"
tableName := "select"
key := "Key'\"\\\n$relspec_0$"
valueName := "名'\"\\\n$body$"
comment := "Comment ' \\ $$ $relspec_0$\nUnicode 名"
t.Cleanup(func() { _, _ = conn.Exec(ctx, "DROP SCHEMA IF EXISTS "+quoteIdent(schemaName)+" CASCADE") })
db := models.InitDatabase("test\nSELECT invalid_header;")
s := models.InitSchema(schemaName)
tbl := models.InitTable(tableName, schemaName)
id := models.InitColumn(key, tableName, schemaName)
id.Type = "integer"
id.IsPrimaryKey = true
id.NotNull = true
tbl.Columns[key] = id
value := models.InitColumn(valueName, tableName, schemaName)
value.Type = "text"
value.Default = quote(comment)
value.Description = comment
value.Comment = comment
tbl.Columns[valueName] = value
tbl.Description = comment
tbl.Comment = comment
tbl.Constraints["unique"] = &models.Constraint{Name: "unique'\"\\\n$relspec_0$", Type: models.UniqueConstraint, Columns: []string{valueName}}
tbl.Constraints["check"] = &models.Constraint{Name: "check'\"\\\n$relspec_0$", Type: models.CheckConstraint, Expression: quoteIdent(key) + ">0"}
tbl.Indexes["index"] = &models.Index{Name: "index'\"\\\n$relspec_0$", Type: "btree", Columns: []string{valueName}}
child := models.InitTable("from", schemaName)
child.Columns[key] = &models.Column{Name: key, Type: "integer"}
child.Constraints["fk"] = &models.Constraint{Name: "foreign'\"\\\n$relspec_0$", Type: models.ForeignKeyConstraint, Columns: []string{key}, ReferencedSchema: schemaName, ReferencedTable: tableName, ReferencedColumns: []string{key}}
s.Tables = []*models.Table{tbl, child}
db.Schemas = []*models.Schema{s}
w := NewWriter(&writers.WriterOptions{})
var output bytes.Buffer
w.writer = &output
if err := w.WriteDatabase(db); err != nil {
t.Fatal(err)
}
if _, err := conn.Exec(ctx, output.String()); err != nil {
t.Fatalf("file SQL: %v\n%s", err, output.String())
}
if _, err := conn.Exec(ctx, "SET standard_conforming_strings=off"); err != nil {
t.Fatal(err)
}
statements, err := w.GenerateDatabaseStatements(db)
if err != nil {
t.Fatal(err)
}
for _, stmt := range statements {
if _, err := conn.Exec(ctx, stmt); err != nil {
t.Fatalf("statement: %v\n%s", err, stmt)
}
}
target := w.qualTable(schemaName, tableName)
if _, err := conn.Exec(ctx, "INSERT INTO "+target+" ("+quoteIdent(key)+") VALUES (1)"); err != nil {
t.Fatal(err)
}
var got string
if err := conn.QueryRow(ctx, "SELECT "+quoteIdent(valueName)+" FROM "+target).Scan(&got); err != nil || got != comment {
t.Fatalf("default: %q %v", got, err)
}
if err := conn.QueryRow(ctx, `SELECT col_description(c.oid,a.attnum) FROM pg_class c JOIN pg_namespace n ON n.oid=c.relnamespace JOIN pg_attribute a ON a.attrelid=c.oid WHERE n.nspname=$1 AND c.relname=$2 AND a.attname=$3`, schemaName, tableName, valueName).Scan(&got); err != nil || got != comment {
t.Fatalf("comment: %q %v", got, err)
}
// Migration templates must preserve the same identifier and literal boundaries.
current := models.InitDatabase("current")
currentSchema := models.InitSchema(schemaName)
currentTable := models.InitTable(tableName, schemaName)
currentValue := *value
currentValue.Type = "text"
currentTable.Columns[valueName] = &currentValue
currentID := *id
currentTable.Columns[key] = &currentID
currentTable.Constraints = tbl.Constraints
currentTable.Indexes = tbl.Indexes
currentSchema.Tables = []*models.Table{currentTable, child}
current.Schemas = []*models.Schema{currentSchema}
value.Type = "varchar"
value.Length = 200
migration, err := NewMigrationWriter(&writers.WriterOptions{})
if err != nil {
t.Fatal(err)
}
var migrationOutput bytes.Buffer
migration.writer = &migrationOutput
if err := migration.WriteMigration(db, current); err != nil {
t.Fatal(err)
}
if _, err := conn.Exec(ctx, migrationOutput.String()); err != nil {
t.Fatalf("migration: %v\n%s", err, migrationOutput.String())
}
if err := conn.QueryRow(ctx, "SELECT "+quoteIdent(valueName)+" FROM "+target+" /* after migration */").Scan(&got); err != nil || got != comment {
t.Fatalf("migration value: %q %v", got, err)
}
te := w.executor
sequenceName := "Seq'\"\\\n$relspec_0$"
sequenceSQL, err := te.ExecuteCreateSequence(CreateSequenceData{SchemaName: schemaName, SequenceName: sequenceName, Increment: 1, MinValue: 1, MaxValue: 1000, StartValue: 1, CacheSize: 1})
if err != nil {
t.Fatal(err)
}
if _, err := conn.Exec(ctx, sequenceSQL); err != nil {
t.Fatal(err)
}
setvalSQL, err := te.ExecuteSetSequenceValue(SetSequenceValueData{SchemaName: schemaName, TableName: tableName, SequenceName: sequenceName, ColumnName: key})
if err != nil {
t.Fatal(err)
}
if _, err := conn.Exec(ctx, setvalSQL); err != nil {
t.Fatalf("sequence: %v\n%s", err, setvalSQL)
}
var next int
if err := conn.QueryRow(ctx, "SELECT nextval($1::regclass)", w.qualTable(schemaName, sequenceName)).Scan(&next); err != nil || next != 2 {
t.Fatalf("sequence value: %d %v", next, err)
}
autoTable := w.qualTable(schemaName, "auto")
if _, err := conn.Exec(ctx, "CREATE TABLE "+autoTable+" (old integer PRIMARY KEY, new integer)"); err != nil {
t.Fatal(err)
}
pkSQL, err := te.ExecuteCreatePrimaryKeyWithAutoGenCheck(CreatePrimaryKeyWithAutoGenCheckData{SchemaName: schemaName, TableName: "auto", ConstraintName: "new'key", AutoGenNames: formatStringList([]string{"auto_pkey"}), Columns: quotedIdentifiers([]string{"new"}), ColumnNames: formatStringList([]string{"new"})})
if err != nil {
t.Fatal(err)
}
if _, err := conn.Exec(ctx, pkSQL); err != nil {
t.Fatalf("primary key: %v\n%s", err, pkSQL)
}
auditTables, err := te.ExecuteAuditTables(AuditTablesData{AuditSchema: schemaName})
if err != nil {
t.Fatal(err)
}
if _, err := conn.Exec(ctx, auditTables); err != nil {
t.Fatal(err)
}
config := &TableAuditConfig{AuditInsert: true, AuditUpdate: true, AuditDelete: true, TablePrefix: comment}
functionData := BuildAuditFunctionData(schemaName, tbl, id, config, schemaName, "current_user")
function, err := te.ExecuteAuditFunction(functionData)
if err != nil {
t.Fatal(err)
}
if _, err := conn.Exec(ctx, function); err != nil {
t.Fatalf("audit function: %v\n%s", err, function)
}
trigger, err := te.ExecuteAuditTrigger(AuditTriggerData{SchemaName: schemaName, TableName: tableName, FunctionName: functionData.FunctionName, TriggerName: "trigger'\"\\\n$relspec_0$", Events: "INSERT OR UPDATE OR DELETE"})
if err != nil {
t.Fatal(err)
}
if _, err := conn.Exec(ctx, trigger); err != nil {
t.Fatalf("audit trigger: %v\n%s", err, trigger)
}
for _, sql := range []string{"INSERT INTO " + target + " VALUES (2, 'second')", "UPDATE " + target + " SET " + quoteIdent(valueName) + "='third' WHERE " + quoteIdent(key) + "=2", "DELETE FROM " + target + " WHERE " + quoteIdent(key) + "=2"} {
if _, err := conn.Exec(ctx, sql); err != nil {
t.Fatal(err)
}
}
var count int
if err := conn.QueryRow(ctx, "SELECT count(*) FROM "+w.qualTable(schemaName, "atevent")+" WHERE tableprefix=$1", comment).Scan(&count); err != nil || count != 3 {
t.Fatalf("audit events: %d %v", count, err)
}
}
+1 -1
View File
@@ -110,7 +110,7 @@ func TestDiffStatements_ExistingTableNewSequenceIsSetPastData(t *testing.T) {
t.Fatalf("diffStatements failed: %v", err)
}
out := diffJoin(stmts)
seqIdx := strings.Index(out, "CREATE SEQUENCE IF NOT EXISTS identity.login_event_id_login_event_seq")
seqIdx := strings.Index(out, "CREATE SEQUENCE IF NOT EXISTS \"identity\".\"login_event_id_login_event_seq\"")
setvalIdx := strings.Index(out, "setval(")
if seqIdx < 0 || setvalIdx < 0 || seqIdx > setvalIdx {
t.Fatalf("expected sequence creation before setval, got:\n%s", out)
+13 -6
View File
@@ -2,6 +2,7 @@ package pgsql
import (
"encoding/json"
"errors"
"os"
"path/filepath"
"strings"
@@ -130,7 +131,7 @@ func TestFirstBareIdentAndHelpers(t *testing.T) {
"abc,def": "abc",
"abc;": "abc",
"\n abc\tdef": "abc",
`"a b" c`: `"a`,
`"a b" c`: `"a b"`,
" tbl (x int)": "tbl",
}
for in, want := range bare {
@@ -237,8 +238,9 @@ func TestWriteAndFinishReport(t *testing.T) {
options: &writers.WriterOptions{Metadata: map[string]interface{}{"report_path": path}},
executionReport: report,
}
if err := w.finishReport(); err != nil {
t.Fatalf("finishReport: %v", err)
var failure *ExecutionFailure
if err := w.finishReport(); !errors.As(err, &failure) || failure.Report != report {
t.Fatalf("expected structured execution failure, got: %v", err)
}
data, err := os.ReadFile(path)
@@ -270,10 +272,15 @@ func TestWriteReportBadPath(t *testing.T) {
if err := w.writeReport(filepath.Join(t.TempDir(), "missing", "r.json")); err == nil {
t.Error("expected error for unwritable path")
}
// finishReport must swallow the report error.
// Explicitly requested reports are part of successful execution.
w.options.Metadata = map[string]interface{}{"report_path": filepath.Join(t.TempDir(), "missing", "r.json")}
if err := w.finishReport(); err != nil {
t.Errorf("finishReport must not fail on report write error: %v", err)
if err := w.finishReport(); err == nil {
t.Error("finishReport must fail on report write error")
}
w.executionReport.FailedStatements = 1
var failure *ExecutionFailure
if err := w.finishReport(); !errors.As(err, &failure) || !strings.Contains(err.Error(), "requested execution report") {
t.Fatalf("must preserve both execution and report errors: %v", err)
}
}
+12 -34
View File
@@ -24,6 +24,7 @@ func TemplateFunctions() map[string]interface{} {
"escape": escape,
"safe_identifier": safeIdentifier,
"quote_ident": quoteIdent,
"comment_text": commentText,
// Type conversion
"goTypeToSQL": goTypeToSQL,
@@ -99,6 +100,9 @@ func indent(spaces int, text string) string {
// quote quotes a string value for SQL (escapes single quotes)
func quote(s string) string {
if strings.Contains(s, "\\") {
return sqlLiteral(s)
}
return "'" + strings.ReplaceAll(s, "'", "''") + "'"
}
@@ -125,41 +129,15 @@ func safeIdentifier(s string) string {
return strings.ToLower(safe)
}
// quoteIdent quotes a PostgreSQL identifier if necessary
// Identifiers need quoting if they:
// - Start with a digit
// - Contain special characters
// - Are reserved keywords
// - Contain uppercase letters (to preserve case)
func quoteIdent(s string) string {
if s == "" {
return `""`
// quoteIdent always quotes an identifier, preserving case and reserved words.
func quoteIdent(s string) string { return `"` + strings.ReplaceAll(s, `"`, `""`) + `"` }
func quotedIdentifiers(names []string) string {
out := make([]string, len(names))
for i, name := range names {
out[i] = quoteIdent(name)
}
// Check if quoting is needed
needsQuoting := unicode.IsDigit(rune(s[0]))
// Starts with digit
// Contains uppercase letters or special characters
for _, r := range s {
if unicode.IsUpper(r) {
needsQuoting = true
break
}
if !unicode.IsLetter(r) && !unicode.IsDigit(r) && r != '_' {
needsQuoting = true
break
}
}
if needsQuoting {
// Escape double quotes by doubling them
escaped := strings.ReplaceAll(s, `"`, `""`)
return `"` + escaped + `"`
}
return s
return strings.Join(out, ", ")
}
// Type conversion functions
+3 -3
View File
@@ -106,7 +106,7 @@ func TestQuoteIdent(t *testing.T) {
input string
expected string
}{
{"valid_name", "valid_name"},
{"valid_name", `"valid_name"`},
{"ValidName", `"ValidName"`},
{"123column", `"123column"`},
{"215162_fk_constraint", `"215162_fk_constraint"`},
@@ -114,8 +114,8 @@ func TestQuoteIdent(t *testing.T) {
{"user@domain", `"user@domain"`},
{`"quoted"`, `"""quoted"""`},
{"", `""`},
{"lowercase", "lowercase"},
{"with_underscore", "with_underscore"},
{"lowercase", `"lowercase"`},
{"with_underscore", `"with_underscore"`},
}
for _, tt := range tests {
+78 -162
View File
@@ -8,7 +8,6 @@ import (
"text/template"
"git.warky.dev/wdevs/relspecgo/pkg/models"
"git.warky.dev/wdevs/relspecgo/pkg/writers"
)
//go:embed templates/*.tmpl
@@ -29,7 +28,7 @@ func NewTemplateExecutor(flattenSchema bool) (*TemplateExecutor, error) {
}
// qual_table returns a quoted, schema-qualified identifier.
// With flatten=false: "schema"."table" (or unquoted equivalents).
// With flatten=false: "schema"."table".
// With flatten=true: "schema_table".
funcMap["qual_table"] = func(schema, name string) string {
if schema == "" {
@@ -41,20 +40,20 @@ func NewTemplateExecutor(flattenSchema bool) (*TemplateExecutor, error) {
return quoteIdent(schema) + "." + quoteIdent(name)
}
// qual_table_raw is the same as qual_table but without identifier quoting.
funcMap["qual_table_raw"] = func(schema, name string) string {
if schema == "" {
return name
}
if flattenSchema {
return schema + "_" + name
}
return schema + "." + name
}
tmpl, err := template.New("").Funcs(funcMap).ParseFS(templateFS, "templates/*.tmpl")
funcMap["dollar_delimiter"] = func() string { return "$$" }
tmpl := template.New("").Funcs(funcMap)
entries, err := templateFS.ReadDir("templates")
if err != nil {
return nil, fmt.Errorf("failed to parse templates: %w", err)
return nil, err
}
for _, entry := range entries {
source, err := templateFS.ReadFile("templates/" + entry.Name())
if err != nil {
return nil, err
}
if _, err := tmpl.New(entry.Name()).Parse(string(source)); err != nil {
return nil, fmt.Errorf("parse %s: %w", entry.Name(), err)
}
}
return &TemplateExecutor{
@@ -62,6 +61,31 @@ func NewTemplateExecutor(flattenSchema bool) (*TemplateExecutor, error) {
}, nil
}
// execute chooses a delimiter absent from rendered data, including SQL expressions.
// Only explicit template calls emit delimiters; user text is never rewritten.
func (te *TemplateExecutor) execute(name string, data any) (string, error) {
for n := 0; ; n++ {
delimiter := fmt.Sprintf("$relspec_%d$", n)
count := 0
tmpl, err := te.templates.Clone()
if err != nil {
return "", err
}
tmpl.Funcs(template.FuncMap{"dollar_delimiter": func() string { count++; return delimiter }})
var buf bytes.Buffer
if err := tmpl.ExecuteTemplate(&buf, name, data); err != nil {
return "", fmt.Errorf("execute %s: %w", name, err)
}
rendered := buf.String()
if strings.ContainsRune(rendered, 0) {
return "", fmt.Errorf("%s contains a NUL byte", name)
}
if strings.Count(rendered, delimiter) == count {
return rendered, nil
}
}
}
// Template data structures
// CreateTableData contains data for create table template
@@ -99,15 +123,15 @@ type AddColumnData struct {
// AlterColumnTypeWithCheckData contains data for the guarded alter column
// type template, which only alters existing columns whose live type
// differs from the desired one, and falls back to renaming the old column
// and adding a fresh one when the in-place conversion is not possible.
// differs from the desired one. Conversion errors propagate unchanged.
type AlterColumnTypeWithCheckData struct {
SchemaName string
TableName string
ColumnName string
NewType string
EquivalentTypes string
UsingExpr string
SchemaName string
TableName string
ColumnName string
NewType string
EquivalentTypes string
UsingExpr string
PreflightPredicate string
}
// AlterColumnDefaultWithCheckData contains data for the guarded alter
@@ -164,6 +188,7 @@ type CreateForeignKeyData struct {
TargetColumns string
OnDelete string
OnUpdate string
Deferral string // DEFERRABLE / DEFERRABLE INITIALLY DEFERRED / NOT DEFERRABLE, or empty
}
// DropConstraintData contains data for drop constraint template
@@ -238,6 +263,7 @@ type CreateUniqueConstraintData struct {
TableName string
ConstraintName string
Columns string
UsingIndex string // when set, promotes this existing unique index instead of building a new one
}
// CreateCheckConstraintData contains data for create check constraint template
@@ -259,7 +285,7 @@ type CreateForeignKeyWithCheckData struct {
TargetColumns string
OnDelete string
OnUpdate string
Deferrable bool
Deferral string // DEFERRABLE / DEFERRABLE INITIALLY DEFERRED / NOT DEFERRABLE, or empty
}
// SetSequenceValueData contains data for set sequence value template
@@ -303,227 +329,117 @@ type CreatePrimaryKeyWithAutoGenCheckData struct {
// ExecuteCreateTable executes the create table template
func (te *TemplateExecutor) ExecuteCreateTable(data CreateTableData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "create_table.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute create_table template: %w", err)
}
return buf.String(), nil
return te.execute("create_table.tmpl", data)
}
// ExecuteAddColumn executes the add column template
func (te *TemplateExecutor) ExecuteAddColumn(data AddColumnData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "add_column.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute add_column template: %w", err)
}
return buf.String(), nil
return te.execute("add_column.tmpl", data)
}
// ExecuteAlterColumnTypeWithCheck executes the guarded alter column type
// template shared by the full-schema writer and the diff-based migration
// writer.
func (te *TemplateExecutor) ExecuteAlterColumnTypeWithCheck(data AlterColumnTypeWithCheckData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "alter_column_type_with_check.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute alter_column_type_with_check template: %w", err)
}
return buf.String(), nil
return te.execute("alter_column_type_with_check.tmpl", data)
}
// ExecuteAlterColumnDefaultWithCheck executes the guarded alter column
// default template shared by the full-schema writer and the diff-based
// migration writer.
func (te *TemplateExecutor) ExecuteAlterColumnDefaultWithCheck(data AlterColumnDefaultWithCheckData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "alter_column_default_with_check.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute alter_column_default_with_check template: %w", err)
}
return buf.String(), nil
return te.execute("alter_column_default_with_check.tmpl", data)
}
// ExecuteAlterColumnNullabilityWithCheck executes the guarded alter column
// nullability template.
func (te *TemplateExecutor) ExecuteAlterColumnNullabilityWithCheck(data AlterColumnNullabilityWithCheckData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "alter_column_nullability_with_check.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute alter_column_nullability_with_check template: %w", err)
}
return buf.String(), nil
return te.execute("alter_column_nullability_with_check.tmpl", data)
}
// ExecuteCreatePrimaryKey executes the create primary key template
func (te *TemplateExecutor) ExecuteCreatePrimaryKey(data CreatePrimaryKeyData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "create_primary_key.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute create_primary_key template: %w", err)
}
return buf.String(), nil
return te.execute("create_primary_key.tmpl", data)
}
// ExecuteCreateIndex executes the create index template
func (te *TemplateExecutor) ExecuteCreateIndex(data CreateIndexData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "create_index.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute create_index template: %w", err)
}
return buf.String(), nil
return te.execute("create_index.tmpl", data)
}
// ExecuteCreateForeignKey executes the create foreign key template
func (te *TemplateExecutor) ExecuteCreateForeignKey(data CreateForeignKeyData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "create_foreign_key.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute create_foreign_key template: %w", err)
}
return buf.String(), nil
return te.execute("create_foreign_key.tmpl", data)
}
// ExecuteDropConstraint executes the drop constraint template
func (te *TemplateExecutor) ExecuteDropConstraint(data DropConstraintData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "drop_constraint.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute drop_constraint template: %w", err)
}
return buf.String(), nil
return te.execute("drop_constraint.tmpl", data)
}
// ExecuteDropIndex executes the drop index template
func (te *TemplateExecutor) ExecuteDropIndex(data DropIndexData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "drop_index.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute drop_index template: %w", err)
}
return buf.String(), nil
return te.execute("drop_index.tmpl", data)
}
// ExecuteCommentTable executes the table comment template
func (te *TemplateExecutor) ExecuteCommentTable(data CommentTableData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "comment_table.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute comment_table template: %w", err)
}
return buf.String(), nil
return te.execute("comment_table.tmpl", data)
}
// ExecuteCommentColumn executes the column comment template
func (te *TemplateExecutor) ExecuteCommentColumn(data CommentColumnData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "comment_column.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute comment_column template: %w", err)
}
return buf.String(), nil
return te.execute("comment_column.tmpl", data)
}
// ExecuteAuditTables executes the audit tables template
func (te *TemplateExecutor) ExecuteAuditTables(data AuditTablesData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "audit_tables.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute audit_tables template: %w", err)
}
return buf.String(), nil
return te.execute("audit_tables.tmpl", data)
}
// ExecuteAuditFunction executes the audit function template
func (te *TemplateExecutor) ExecuteAuditFunction(data AuditFunctionData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "audit_function.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute audit_function template: %w", err)
}
return buf.String(), nil
return te.execute("audit_function.tmpl", data)
}
// ExecuteAuditTrigger executes the audit trigger template
func (te *TemplateExecutor) ExecuteAuditTrigger(data AuditTriggerData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "audit_trigger.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute audit_trigger template: %w", err)
}
return buf.String(), nil
return te.execute("audit_trigger.tmpl", data)
}
// ExecuteCreateUniqueConstraint executes the create unique constraint template
func (te *TemplateExecutor) ExecuteCreateUniqueConstraint(data CreateUniqueConstraintData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "create_unique_constraint.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute create_unique_constraint template: %w", err)
}
return buf.String(), nil
return te.execute("create_unique_constraint.tmpl", data)
}
// ExecuteCreateCheckConstraint executes the create check constraint template
func (te *TemplateExecutor) ExecuteCreateCheckConstraint(data CreateCheckConstraintData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "create_check_constraint.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute create_check_constraint template: %w", err)
}
return buf.String(), nil
return te.execute("create_check_constraint.tmpl", data)
}
// ExecuteCreateForeignKeyWithCheck executes the create foreign key with check template
func (te *TemplateExecutor) ExecuteCreateForeignKeyWithCheck(data CreateForeignKeyWithCheckData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "create_foreign_key_with_check.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute create_foreign_key_with_check template: %w", err)
}
return buf.String(), nil
return te.execute("create_foreign_key_with_check.tmpl", data)
}
// ExecuteSetSequenceValue executes the set sequence value template
func (te *TemplateExecutor) ExecuteSetSequenceValue(data SetSequenceValueData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "set_sequence_value.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute set_sequence_value template: %w", err)
}
return buf.String(), nil
return te.execute("set_sequence_value.tmpl", data)
}
// ExecuteCreateSequence executes the create sequence template
func (te *TemplateExecutor) ExecuteCreateSequence(data CreateSequenceData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "create_sequence.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute create_sequence template: %w", err)
}
return buf.String(), nil
return te.execute("create_sequence.tmpl", data)
}
// ExecuteAddColumnWithCheck executes the add column with check template
func (te *TemplateExecutor) ExecuteAddColumnWithCheck(data AddColumnWithCheckData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "add_column_with_check.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute add_column_with_check template: %w", err)
}
return buf.String(), nil
return te.execute("add_column_with_check.tmpl", data)
}
// ExecuteCreatePrimaryKeyWithAutoGenCheck executes the create primary key with auto-generated key check template
func (te *TemplateExecutor) ExecuteCreatePrimaryKeyWithAutoGenCheck(data CreatePrimaryKeyWithAutoGenCheckData) (string, error) {
var buf bytes.Buffer
err := te.templates.ExecuteTemplate(&buf, "create_primary_key_with_autogen_check.tmpl", data)
if err != nil {
return "", fmt.Errorf("failed to execute create_primary_key_with_autogen_check template: %w", err)
}
return buf.String(), nil
return te.execute("create_primary_key_with_autogen_check.tmpl", data)
}
// Helper functions to build template data from models
@@ -556,7 +472,7 @@ func BuildCreateTableData(schemaName string, table *models.Table) CreateTableDat
colData.Generated, colData.Identity = columnGenerationClauses(col)
if col.Default != nil && colData.Generated == "" && colData.Identity == "" {
if value, ok := col.Default.(string); ok {
colData.Default = writers.QuoteDefaultValue(stripBackticks(value), col.Type)
colData.Default = quoteDefaultValue(stripBackticks(value), col.Type)
} else {
colData.Default = fmt.Sprintf("%v", col.Default)
}
@@ -607,7 +523,7 @@ func BuildAuditFunctionData(
updateComparisons := make([]string, 0)
for _, col := range auditedColumns {
updateComparisons = append(updateComparisons,
fmt.Sprintf("old.%s IS DISTINCT FROM new.%s", col.Name, col.Name))
fmt.Sprintf("old.%s IS DISTINCT FROM new.%s", quoteIdent(col.Name), quoteIdent(col.Name)))
}
updateCondition := strings.Join(updateComparisons, " OR ")
@@ -622,8 +538,8 @@ func BuildAuditFunctionData(
}
}
oldValue := fmt.Sprintf("old.%s::text", col.Name)
newValue := fmt.Sprintf("new.%s::text", col.Name)
oldValue := fmt.Sprintf("old.%s::text", quoteIdent(col.Name))
newValue := fmt.Sprintf("new.%s::text", quoteIdent(col.Name))
if isEncrypted {
oldValue = "'****************'"
@@ -648,7 +564,7 @@ func BuildAuditFunctionData(
}
}
oldValue := fmt.Sprintf("old.%s::text", col.Name)
oldValue := fmt.Sprintf("old.%s::text", quoteIdent(col.Name))
if isEncrypted {
oldValue = "'****************'"
}
@@ -661,7 +577,7 @@ func BuildAuditFunctionData(
tablePrefix := "NULL"
if config.TablePrefix != "" {
tablePrefix = fmt.Sprintf("'%s'", config.TablePrefix)
tablePrefix = quote(config.TablePrefix)
}
return AuditFunctionData{
@@ -1,12 +1 @@
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_schema = '{{.SchemaName}}'
AND table_name = '{{.TableName}}'
AND column_name = '{{.ColumnName}}'
) THEN
ALTER TABLE {{qual_table .SchemaName .TableName}} ADD COLUMN {{.ColumnDefinition}};
END IF;
END;
$$;
ALTER TABLE {{qual_table .SchemaName .TableName}} ADD COLUMN IF NOT EXISTS {{.ColumnDefinition}};
@@ -1,4 +1,4 @@
DO $$
DO {{dollar_delimiter}}
DECLARE
current_default text;
BEGIN
@@ -8,9 +8,9 @@ BEGIN
JOIN pg_class t ON t.oid = a.attrelid
JOIN pg_namespace n ON n.oid = t.relnamespace
LEFT JOIN pg_attrdef d ON d.adrelid = a.attrelid AND d.adnum = a.attnum
WHERE n.nspname = '{{.SchemaName}}'
AND t.relname = '{{.TableName}}'
AND a.attname = '{{.ColumnName}}'
WHERE n.nspname = {{quote .SchemaName}}
AND t.relname = {{quote .TableName}}
AND a.attname = {{quote .ColumnName}}
AND a.attnum > 0
AND NOT a.attisdropped;
@@ -26,4 +26,4 @@ BEGIN
END IF;
{{- end }}
END;
$$;
{{dollar_delimiter}};
@@ -1,4 +1,4 @@
DO $$
DO {{dollar_delimiter}}
DECLARE
current_not_null boolean;
BEGIN
@@ -7,9 +7,9 @@ BEGIN
FROM pg_attribute a
JOIN pg_class t ON t.oid = a.attrelid
JOIN pg_namespace n ON n.oid = t.relnamespace
WHERE n.nspname = '{{.SchemaName}}'
AND t.relname = '{{.TableName}}'
AND a.attname = '{{.ColumnName}}'
WHERE n.nspname = {{quote .SchemaName}}
AND t.relname = {{quote .TableName}}
AND a.attname = {{quote .ColumnName}}
AND a.attnum > 0
AND NOT a.attisdropped;
@@ -23,4 +23,4 @@ BEGIN
{{- end }}
END IF;
END;
$$;
{{dollar_delimiter}};
@@ -1,30 +1,22 @@
DO $$
DO {{dollar_delimiter}}
DECLARE
current_type text;
renamed_column text;
BEGIN
SELECT pg_catalog.format_type(a.atttypid, a.atttypmod)
INTO current_type
FROM pg_attribute a
JOIN pg_class t ON t.oid = a.attrelid
JOIN pg_namespace n ON n.oid = t.relnamespace
WHERE n.nspname = '{{.SchemaName}}'
AND t.relname = '{{.TableName}}'
AND a.attname = '{{.ColumnName}}'
WHERE n.nspname = {{quote .SchemaName}}
AND t.relname = {{quote .TableName}}
AND a.attname = {{quote .ColumnName}}
AND a.attnum > 0
AND NOT a.attisdropped;
IF current_type IS NOT NULL
AND current_type <> ALL(ARRAY[{{.EquivalentTypes}}]) THEN
BEGIN
ALTER TABLE {{qual_table .SchemaName .TableName}}
ALTER COLUMN {{quote_ident .ColumnName}} TYPE {{.NewType}}{{if .UsingExpr}} USING {{.UsingExpr}}{{end}};
EXCEPTION WHEN OTHERS THEN
renamed_column := '{{.ColumnName}}_' || trim(both '_' from regexp_replace(lower(current_type), '[^a-z0-9]+', '_', 'g'));
EXECUTE format('ALTER TABLE {{qual_table .SchemaName .TableName}} RENAME COLUMN {{quote_ident .ColumnName}} TO %I', renamed_column);
ALTER TABLE {{qual_table .SchemaName .TableName}}
ADD COLUMN {{quote_ident .ColumnName}} {{.NewType}};
END;
ALTER TABLE {{qual_table .SchemaName .TableName}}
ALTER COLUMN {{quote_ident .ColumnName}} TYPE {{.NewType}}{{if .UsingExpr}} USING {{.UsingExpr}}{{end}};
END IF;
END;
$$;
{{dollar_delimiter}};
+16 -16
View File
@@ -1,8 +1,8 @@
CREATE OR REPLACE FUNCTION {{qual_table_raw .SchemaName .FunctionName}}()
CREATE OR REPLACE FUNCTION {{qual_table .SchemaName .FunctionName}}()
RETURNS trigger AS
$body$
{{dollar_delimiter}}
DECLARE
m_funcname text = '{{.FunctionName}}';
m_funcname text = {{quote .FunctionName}};
m_user text;
m_atevent integer;
BEGIN
@@ -23,8 +23,8 @@ BEGIN
{{- if .AuditInsert}}
IF TG_OP = 'INSERT' THEN
-- Record INSERT
INSERT INTO {{.AuditSchema}}.atevent (tablename, tableprefix, rid_parent, changeuser, changedate, changetime, actionx)
VALUES ('{{.TableName}}', {{.TablePrefix}}, new.{{.PrimaryKey}}, m_user, CURRENT_DATE, CURRENT_TIME, 1)
INSERT INTO {{quote_ident .AuditSchema}}.atevent (tablename, tableprefix, rid_parent, changeuser, changedate, changetime, actionx)
VALUES ({{quote .TableName}}, {{.TablePrefix}}, new.{{quote_ident .PrimaryKey}}, m_user, CURRENT_DATE, CURRENT_TIME, 1)
RETURNING rid_atevent INTO m_atevent;
{{- end}}
@@ -32,15 +32,15 @@ BEGIN
ELSIF TG_OP = 'UPDATE' THEN
-- Check if any audited columns changed
IF ({{.UpdateCondition}}) THEN
INSERT INTO {{.AuditSchema}}.atevent (tablename, tableprefix, rid_parent, changeuser, changedate, changetime, actionx)
VALUES ('{{.TableName}}', {{.TablePrefix}}, new.{{.PrimaryKey}}, m_user, CURRENT_DATE, CURRENT_TIME, 2)
INSERT INTO {{quote_ident .AuditSchema}}.atevent (tablename, tableprefix, rid_parent, changeuser, changedate, changetime, actionx)
VALUES ({{quote .TableName}}, {{.TablePrefix}}, new.{{quote_ident .PrimaryKey}}, m_user, CURRENT_DATE, CURRENT_TIME, 2)
RETURNING rid_atevent INTO m_atevent;
-- Record column changes
{{- range .UpdateColumns}}
IF (old.{{.Name}} IS DISTINCT FROM new.{{.Name}}) THEN
INSERT INTO {{$.AuditSchema}}.atdetail(rid_atevent, datacolumn, changedfrom, changedto)
VALUES (m_atevent, '{{.Name}}', substr({{.OldValue}}, 1, 1000), substr({{.NewValue}}, 1, 1000));
IF (old.{{quote_ident .Name}} IS DISTINCT FROM new.{{quote_ident .Name}}) THEN
INSERT INTO {{quote_ident $.AuditSchema}}.atdetail(rid_atevent, datacolumn, changedfrom, changedto)
VALUES (m_atevent, {{quote .Name}}, substr({{.OldValue}}, 1, 1000), substr({{.NewValue}}, 1, 1000));
END IF;
{{- end}}
END IF;
@@ -49,14 +49,14 @@ BEGIN
{{- if .AuditDelete}}
ELSIF TG_OP = 'DELETE' THEN
-- Record DELETE
INSERT INTO {{.AuditSchema}}.atevent (tablename, tableprefix, rid_parent, rid_deletedparent, changeuser, changedate, changetime, actionx)
VALUES ('{{.TableName}}', {{.TablePrefix}}, old.{{.PrimaryKey}}, old.{{.PrimaryKey}}, m_user, CURRENT_DATE, CURRENT_TIME, 3)
INSERT INTO {{quote_ident .AuditSchema}}.atevent (tablename, tableprefix, rid_parent, rid_deletedparent, changeuser, changedate, changetime, actionx)
VALUES ({{quote .TableName}}, {{.TablePrefix}}, old.{{quote_ident .PrimaryKey}}, old.{{quote_ident .PrimaryKey}}, m_user, CURRENT_DATE, CURRENT_TIME, 3)
RETURNING rid_atevent INTO m_atevent;
-- Record deleted column values
{{- range .DeleteColumns}}
INSERT INTO {{$.AuditSchema}}.atdetail(rid_atevent, datacolumn, changedfrom, changedto)
VALUES (m_atevent, '{{.Name}}', substr({{.OldValue}}, 1, 1000), NULL);
INSERT INTO {{quote_ident $.AuditSchema}}.atdetail(rid_atevent, datacolumn, changedfrom, changedto)
VALUES (m_atevent, {{quote .Name}}, substr({{.OldValue}}, 1, 1000), NULL);
{{- end}}
{{- end}}
END IF;
@@ -76,9 +76,9 @@ EXCEPTION
RAISE WARNING 'Audit function % failed: %', m_funcname, SQLERRM;
RETURN NULL;
END;
$body$
{{dollar_delimiter}}
LANGUAGE plpgsql
VOLATILE
SECURITY DEFINER;
COMMENT ON FUNCTION {{qual_table_raw .SchemaName .FunctionName}}() IS 'Audit trigger function for table {{qual_table_raw .SchemaName .TableName}}';
COMMENT ON FUNCTION {{qual_table .SchemaName .FunctionName}}() IS {{quote (printf "Audit trigger function for table %s" (qual_table .SchemaName .TableName))}};
+24 -24
View File
@@ -1,5 +1,5 @@
-- Audit Event Header Table
CREATE TABLE IF NOT EXISTS {{.AuditSchema}}.atevent (
CREATE TABLE IF NOT EXISTS {{quote_ident .AuditSchema}}.atevent (
rid_atevent serial PRIMARY KEY,
tablename text NOT NULL,
tableprefix text,
@@ -12,38 +12,38 @@ CREATE TABLE IF NOT EXISTS {{.AuditSchema}}.atevent (
CONSTRAINT ck_atevent_action CHECK (actionx IN (1, 2, 3))
);
CREATE INDEX IF NOT EXISTS idx_atevent_tablename ON {{.AuditSchema}}.atevent(tablename);
CREATE INDEX IF NOT EXISTS idx_atevent_rid_parent ON {{.AuditSchema}}.atevent(rid_parent);
CREATE INDEX IF NOT EXISTS idx_atevent_changedate ON {{.AuditSchema}}.atevent(changedate);
CREATE INDEX IF NOT EXISTS idx_atevent_changeuser ON {{.AuditSchema}}.atevent(changeuser);
CREATE INDEX IF NOT EXISTS idx_atevent_tablename ON {{quote_ident .AuditSchema}}.atevent(tablename);
CREATE INDEX IF NOT EXISTS idx_atevent_rid_parent ON {{quote_ident .AuditSchema}}.atevent(rid_parent);
CREATE INDEX IF NOT EXISTS idx_atevent_changedate ON {{quote_ident .AuditSchema}}.atevent(changedate);
CREATE INDEX IF NOT EXISTS idx_atevent_changeuser ON {{quote_ident .AuditSchema}}.atevent(changeuser);
COMMENT ON TABLE {{.AuditSchema}}.atevent IS 'Audit trail header table - tracks all data changes';
COMMENT ON COLUMN {{.AuditSchema}}.atevent.rid_atevent IS 'Audit event ID';
COMMENT ON COLUMN {{.AuditSchema}}.atevent.tablename IS 'Name of the table that was modified';
COMMENT ON COLUMN {{.AuditSchema}}.atevent.rid_parent IS 'Primary key value of the modified record';
COMMENT ON COLUMN {{.AuditSchema}}.atevent.rid_deletedparent IS 'Parent reference for deleted records';
COMMENT ON COLUMN {{.AuditSchema}}.atevent.changeuser IS 'User who made the change';
COMMENT ON COLUMN {{.AuditSchema}}.atevent.changedate IS 'Date of change';
COMMENT ON COLUMN {{.AuditSchema}}.atevent.changetime IS 'Time of change';
COMMENT ON COLUMN {{.AuditSchema}}.atevent.actionx IS 'Action type: 1=INSERT, 2=UPDATE, 3=DELETE';
COMMENT ON TABLE {{quote_ident .AuditSchema}}.atevent IS 'Audit trail header table - tracks all data changes';
COMMENT ON COLUMN {{quote_ident .AuditSchema}}.atevent.rid_atevent IS 'Audit event ID';
COMMENT ON COLUMN {{quote_ident .AuditSchema}}.atevent.tablename IS 'Name of the table that was modified';
COMMENT ON COLUMN {{quote_ident .AuditSchema}}.atevent.rid_parent IS 'Primary key value of the modified record';
COMMENT ON COLUMN {{quote_ident .AuditSchema}}.atevent.rid_deletedparent IS 'Parent reference for deleted records';
COMMENT ON COLUMN {{quote_ident .AuditSchema}}.atevent.changeuser IS 'User who made the change';
COMMENT ON COLUMN {{quote_ident .AuditSchema}}.atevent.changedate IS 'Date of change';
COMMENT ON COLUMN {{quote_ident .AuditSchema}}.atevent.changetime IS 'Time of change';
COMMENT ON COLUMN {{quote_ident .AuditSchema}}.atevent.actionx IS 'Action type: 1=INSERT, 2=UPDATE, 3=DELETE';
-- Audit Event Detail Table
CREATE TABLE IF NOT EXISTS {{.AuditSchema}}.atdetail (
CREATE TABLE IF NOT EXISTS {{quote_ident .AuditSchema}}.atdetail (
rid_atdetail serial PRIMARY KEY,
rid_atevent integer NOT NULL,
datacolumn text NOT NULL,
changedfrom text,
changedto text,
CONSTRAINT fk_atdetail_atevent FOREIGN KEY (rid_atevent)
REFERENCES {{.AuditSchema}}.atevent(rid_atevent) ON DELETE CASCADE
REFERENCES {{quote_ident .AuditSchema}}.atevent(rid_atevent) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_atdetail_rid_atevent ON {{.AuditSchema}}.atdetail(rid_atevent);
CREATE INDEX IF NOT EXISTS idx_atdetail_datacolumn ON {{.AuditSchema}}.atdetail(datacolumn);
CREATE INDEX IF NOT EXISTS idx_atdetail_rid_atevent ON {{quote_ident .AuditSchema}}.atdetail(rid_atevent);
CREATE INDEX IF NOT EXISTS idx_atdetail_datacolumn ON {{quote_ident .AuditSchema}}.atdetail(datacolumn);
COMMENT ON TABLE {{.AuditSchema}}.atdetail IS 'Audit trail detail table - stores individual column changes';
COMMENT ON COLUMN {{.AuditSchema}}.atdetail.rid_atdetail IS 'Audit detail ID';
COMMENT ON COLUMN {{.AuditSchema}}.atdetail.rid_atevent IS 'Reference to audit event';
COMMENT ON COLUMN {{.AuditSchema}}.atdetail.datacolumn IS 'Name of the column that changed';
COMMENT ON COLUMN {{.AuditSchema}}.atdetail.changedfrom IS 'Old value before change';
COMMENT ON COLUMN {{.AuditSchema}}.atdetail.changedto IS 'New value after change';
COMMENT ON TABLE {{quote_ident .AuditSchema}}.atdetail IS 'Audit trail detail table - stores individual column changes';
COMMENT ON COLUMN {{quote_ident .AuditSchema}}.atdetail.rid_atdetail IS 'Audit detail ID';
COMMENT ON COLUMN {{quote_ident .AuditSchema}}.atdetail.rid_atevent IS 'Reference to audit event';
COMMENT ON COLUMN {{quote_ident .AuditSchema}}.atdetail.datacolumn IS 'Name of the column that changed';
COMMENT ON COLUMN {{quote_ident .AuditSchema}}.atdetail.changedfrom IS 'Old value before change';
COMMENT ON COLUMN {{quote_ident .AuditSchema}}.atdetail.changedto IS 'New value after change';
@@ -1,16 +1,16 @@
DO $$
DO {{dollar_delimiter}}
BEGIN
IF NOT EXISTS (
SELECT 1
FROM pg_trigger
WHERE tgname = '{{.TriggerName}}'
AND tgrelid = '{{qual_table_raw .SchemaName .TableName}}'::regclass
WHERE tgname = {{quote .TriggerName}}
AND tgrelid = {{quote (qual_table .SchemaName .TableName)}}::regclass
) THEN
CREATE TRIGGER {{.TriggerName}}
CREATE TRIGGER {{quote_ident .TriggerName}}
AFTER {{.Events}}
ON {{qual_table_raw .SchemaName .TableName}}
ON {{qual_table .SchemaName .TableName}}
FOR EACH ROW
EXECUTE FUNCTION {{qual_table_raw .SchemaName .FunctionName}}();
EXECUTE FUNCTION {{qual_table .SchemaName .FunctionName}}();
END IF;
END;
$$;
{{dollar_delimiter}};
@@ -1,39 +1,39 @@
{{/* Base constraint template */}}
{{- define "constraint_base" -}}
ALTER TABLE {{qual_table_raw .SchemaName .TableName}}
ADD CONSTRAINT {{.ConstraintName}}
ALTER TABLE {{qual_table .SchemaName .TableName}}
ADD CONSTRAINT {{quote_ident .ConstraintName}}
{{block "constraint_definition" .}}{{end}};
{{- end -}}
{{/* Drop constraint with check */}}
{{- define "drop_constraint_safe" -}}
DO $$
DO {{dollar_delimiter}}
BEGIN
IF EXISTS (
SELECT 1 FROM information_schema.table_constraints
WHERE table_schema = '{{.SchemaName}}'
AND table_name = '{{.TableName}}'
AND constraint_name = '{{.ConstraintName}}'
WHERE table_schema = {{quote .SchemaName}}
AND table_name = {{quote .TableName}}
AND constraint_name = {{quote .ConstraintName}}
) THEN
ALTER TABLE {{qual_table_raw .SchemaName .TableName}}
DROP CONSTRAINT {{.ConstraintName}};
ALTER TABLE {{qual_table .SchemaName .TableName}}
DROP CONSTRAINT {{quote_ident .ConstraintName}};
END IF;
END;
$$;
{{dollar_delimiter}};
{{- end -}}
{{/* Add constraint with existence check */}}
{{- define "add_constraint_safe" -}}
DO $$
DO {{dollar_delimiter}}
BEGIN
IF NOT EXISTS (
SELECT 1 FROM information_schema.table_constraints
WHERE table_schema = '{{.SchemaName}}'
AND table_name = '{{.TableName}}'
AND constraint_name = '{{.ConstraintName}}'
WHERE table_schema = {{quote .SchemaName}}
AND table_name = {{quote .TableName}}
AND constraint_name = {{quote .ConstraintName}}
) THEN
{{template "constraint_base" .}}
END IF;
END;
$$;
{{dollar_delimiter}};
{{- end -}}
+8 -8
View File
@@ -1,23 +1,23 @@
{{/* Base DDL template with common structure */}}
{{- define "ddl_header" -}}
-- DDL Operation: {{.Operation}}
-- Schema: {{.Schema}}
-- Object: {{.ObjectName}}
-- DDL Operation: {{comment_text .Operation}}
-- Schema: {{comment_text .Schema}}
-- Object: {{comment_text .ObjectName}}
{{- end -}}
{{- define "ddl_footer" -}}
-- End of {{.Operation}}
-- End of {{comment_text .Operation}}
{{- end -}}
{{/* Base ALTER TABLE structure */}}
{{- define "alter_table_base" -}}
ALTER TABLE {{qual_table_raw .SchemaName .TableName}}
ALTER TABLE {{qual_table .SchemaName .TableName}}
{{block "alter_operation" .}}{{end}};
{{- end -}}
{{/* Common existence check pattern */}}
{{- define "exists_check" -}}
DO $$
DO {{dollar_delimiter}}
BEGIN
IF NOT EXISTS (
{{block "exists_query" .}}{{end}}
@@ -25,10 +25,10 @@ BEGIN
{{block "create_statement" .}}{{end}}
END IF;
END;
$$;
{{dollar_delimiter}};
{{- end -}}
{{/* Common drop pattern */}}
{{- define "drop_if_exists" -}}
{{block "drop_type" .}}{{end}} IF EXISTS {{qual_table_raw .SchemaName .ObjectName}};
{{block "drop_type" .}}{{end}} IF EXISTS {{qual_table .SchemaName .ObjectName}};
{{- end -}}
@@ -1 +1 @@
COMMENT ON COLUMN {{qual_table .SchemaName .TableName}}.{{quote_ident .ColumnName}} IS '{{.Comment}}';
COMMENT ON COLUMN {{qual_table .SchemaName .TableName}}.{{quote_ident .ColumnName}} IS {{quote .Comment}};
@@ -1 +1 @@
COMMENT ON TABLE {{qual_table .SchemaName .TableName}} IS '{{.Comment}}';
COMMENT ON TABLE {{qual_table .SchemaName .TableName}} IS {{quote .Comment}};
@@ -1,12 +1,12 @@
DO $$
DO {{dollar_delimiter}}
BEGIN
IF NOT EXISTS (
SELECT 1 FROM information_schema.table_constraints
WHERE table_schema = '{{.SchemaName}}'
AND table_name = '{{.TableName}}'
AND constraint_name = '{{.ConstraintName}}'
WHERE table_schema = {{quote .SchemaName}}
AND table_name = {{quote .TableName}}
AND constraint_name = {{quote .ConstraintName}}
) THEN
ALTER TABLE {{qual_table .SchemaName .TableName}} ADD CONSTRAINT {{quote_ident .ConstraintName}} CHECK ({{.Expression}});
END IF;
END;
$$;
{{dollar_delimiter}};
@@ -7,4 +7,4 @@ ALTER TABLE {{qual_table .SchemaName .TableName}}
REFERENCES {{qual_table .TargetSchema .TargetTable}} ({{.TargetColumns}})
ON DELETE {{.OnDelete}}
ON UPDATE {{.OnUpdate}}
DEFERRABLE;
{{.Deferral}};
@@ -1,18 +1,18 @@
DO $$
DO {{dollar_delimiter}}
BEGIN
IF NOT EXISTS (
SELECT 1 FROM information_schema.table_constraints
WHERE table_schema = '{{.SchemaName}}'
AND table_name = '{{.TableName}}'
AND constraint_name = '{{.ConstraintName}}'
WHERE table_schema = {{quote .SchemaName}}
AND table_name = {{quote .TableName}}
AND constraint_name = {{quote .ConstraintName}}
) THEN
ALTER TABLE {{qual_table .SchemaName .TableName}}
ADD CONSTRAINT {{quote_ident .ConstraintName}}
FOREIGN KEY ({{.SourceColumns}})
REFERENCES {{qual_table .TargetSchema .TargetTable}} ({{.TargetColumns}})
ON DELETE {{.OnDelete}}
ON UPDATE {{.OnUpdate}}{{if .Deferrable}}
DEFERRABLE{{end}};
ON UPDATE {{.OnUpdate}}{{if .Deferral}}
{{.Deferral}}{{end}};
END IF;
END;
$$;
{{dollar_delimiter}};
@@ -1,13 +1,13 @@
DO $$
DO {{dollar_delimiter}}
BEGIN
IF NOT EXISTS (
SELECT 1 FROM information_schema.table_constraints
WHERE table_schema = '{{.SchemaName}}'
AND table_name = '{{.TableName}}'
AND constraint_name = '{{.ConstraintName}}'
WHERE table_schema = {{quote .SchemaName}}
AND table_name = {{quote .TableName}}
AND constraint_name = {{quote .ConstraintName}}
) THEN
ALTER TABLE {{qual_table .SchemaName .TableName}}
ADD CONSTRAINT {{quote_ident .ConstraintName}} PRIMARY KEY ({{.Columns}});
END IF;
END;
$$;
{{dollar_delimiter}};
@@ -1,4 +1,4 @@
DO $$
DO {{dollar_delimiter}}
DECLARE
current_pk_name text;
current_pk_matches boolean := false;
@@ -16,22 +16,22 @@ BEGIN
ON a.attrelid = t.oid
AND a.attnum = cols.attnum
WHERE c.contype = 'p'
AND n.nspname = '{{.SchemaName}}'
AND t.relname = '{{.TableName}}'
AND n.nspname = {{quote .SchemaName}}
AND t.relname = {{quote .TableName}}
ORDER BY cols.ord
),
ARRAY[]::text[]
) = ARRAY[{{.ColumnNames}}]
INTO current_pk_name, current_pk_matches
FROM information_schema.table_constraints tc
WHERE tc.table_schema = '{{.SchemaName}}'
AND tc.table_name = '{{.TableName}}'
WHERE tc.table_schema = {{quote .SchemaName}}
AND tc.table_name = {{quote .TableName}}
AND tc.constraint_type = 'PRIMARY KEY';
IF current_pk_name IS NOT NULL
AND NOT current_pk_matches
AND current_pk_name IN ({{.AutoGenNames}}) THEN
EXECUTE 'ALTER TABLE {{qual_table .SchemaName .TableName}} DROP CONSTRAINT ' || quote_ident(current_pk_name) || ' CASCADE';
EXECUTE {{quote (printf "ALTER TABLE %s DROP CONSTRAINT " (qual_table .SchemaName .TableName))}} || quote_ident(current_pk_name) || ' CASCADE';
END IF;
-- Add the desired primary key only when no matching primary key already exists.
@@ -40,4 +40,4 @@ BEGIN
ALTER TABLE {{qual_table .SchemaName .TableName}} ADD CONSTRAINT {{quote_ident .ConstraintName}} PRIMARY KEY ({{.Columns}});
END IF;
END;
$$;
{{dollar_delimiter}};
@@ -1,12 +1,12 @@
DO $$
DO {{dollar_delimiter}}
BEGIN
IF NOT EXISTS (
SELECT 1 FROM information_schema.table_constraints
WHERE table_schema = '{{.SchemaName}}'
AND table_name = '{{.TableName}}'
AND constraint_name = '{{.ConstraintName}}'
WHERE table_schema = {{quote .SchemaName}}
AND table_name = {{quote .TableName}}
AND constraint_name = {{quote .ConstraintName}}
) THEN
ALTER TABLE {{qual_table .SchemaName .TableName}} ADD CONSTRAINT {{quote_ident .ConstraintName}} UNIQUE ({{.Columns}});
ALTER TABLE {{qual_table .SchemaName .TableName}} ADD CONSTRAINT {{quote_ident .ConstraintName}} UNIQUE {{if .UsingIndex}}USING INDEX {{quote_ident .UsingIndex}}{{else}}({{.Columns}}){{end}};
END IF;
END;
$$;
{{dollar_delimiter}};

Some files were not shown because too many files have changed in this diff Show More