Compare commits

..
6 Commits
Author SHA1 Message Date
Hein e3bc6cf491 chore(go): require Go 1.26.9 for stdlib vulnerability fixes
Release / test (push) Successful in 2m10s
Release / release (push) Successful in 2m48s
Release / pkg-aur (push) Successful in 33m34s
Release / pkg-deb (push) Successful in 1m35s
Release / pkg-rpm (push) Successful in 2m0s
go 1.26.0 made CI's setup-go install a toolchain with 27 known stdlib
vulnerabilities (fixed in 1.26.1-1.26.9).
2026-10-12 09:48:05 +02:00
Hein fa6a5d0339 chore(release): update package version to 1.0.89
Release / test (push) Failing after 2m37s
Release / release (push) Skipped
Release / pkg-aur (push) Skipped
Release / pkg-deb (push) Skipped
Release / pkg-rpm (push) Skipped
2026-10-12 09:33:34 +02:00
Hein 337af294f3 chore(deps): update all dependencies
Fixes GO-2026-6629 (golang.org/x/text precis panic). Bumps bun and
bun/dialect/pgdialect together to v1.3.0; go directive now 1.26.0.
2026-10-12 09:31:55 +02:00
Hein a4c68e5661 ci(release): attach source archives to release and use them for AUR package
Gitea's /archive/ endpoint is disabled, which broke the AUR source download.
Build source tar.gz/zip with git archive, upload as release assets, and point
the PKGBUILD and checksum step at the release asset.
2026-10-12 09:28:52 +02:00
warkanum 9f0269f591 chore(release): update package version to 1.0.88
Release / test (push) Failing after 1m36s
Release / release (push) Skipped
Release / pkg-aur (push) Skipped
Release / pkg-deb (push) Skipped
Release / pkg-rpm (push) Skipped
2026-10-08 00:08:26 +02:00
warkanum b4f18f088b feat(pgsql): unique constraints on apply, matching unique index names, FK deferral
- migration writer now emits unique constraints (direct apply and live diff
  previously dropped them); idempotent via the existing DO block
- unique constraint adopts a same-named model unique index with
  ADD CONSTRAINT ... UNIQUE USING INDEX instead of building a second index
- dbml reader names unnamed unique indexes ukey_<table>_<cols> to match the
  unique constraint naming
- foreign key deferral: Constraint.Deferrable/InitiallyDeferred are *bool
  (nil = unspecified, never diffed); writers emit DEFERRABLE, DEFERRABLE
  INITIALLY DEFERRED and NOT DEFERRABLE; live reader reads condeferrable and
  condeferred; dbml reader/writer support deferrable and initially deferred
- tests and README/naming docs updated
2026-10-08 00:07:49 +02:00
820 changed files with 1386837 additions and 4172424 deletions
+8 -1
View File
@@ -85,6 +85,13 @@ jobs:
VERSION="${{ github.event.inputs.tag || github.ref_name }}"
makensis -DVERSION="${VERSION#v}" -DEXE="$PWD/relspec-windows-amd64.exe" -DOUT="$PWD/relspec-setup-windows-amd64.exe" windows/installer.nsi
- name: Build source archives
run: |
TAG="${{ github.event.inputs.tag || github.ref_name }}"
PREFIX="relspec-${TAG#v}"
git archive --format=tar.gz --prefix="${PREFIX}/" -o "relspec-${TAG}-source.tar.gz" "$TAG"
git archive --format=zip --prefix="${PREFIX}/" -o "relspec-${TAG}-source.zip" "$TAG"
- name: Create release and upload assets
run: |
TAG="${{ github.event.inputs.tag || github.ref_name }}"
@@ -198,7 +205,7 @@ jobs:
echo "Publishing AUR package version ${PKGVER}-${PKGREL}"
# Compute SHA256 of the source archive from the same URL the PKGBUILD will download.
SHA=$(curl -fsSL "https://git.warky.dev/wdevs/relspecgo/archive/v${PKGVER}.zip" | sha256sum | cut -d' ' -f1)
SHA=$(curl -fsSL "https://git.warky.dev/wdevs/relspecgo/releases/download/v${PKGVER}/relspec-v${PKGVER}-source.zip" | sha256sum | cut -d' ' -f1)
# Update PKGBUILD — keep remote source URL, bump version/checksum, and increment pkgrel for same-version rebuilds.
sed -e "s/^pkgver=.*/pkgver=${PKGVER}/" \
+11 -2
View File
@@ -16,7 +16,7 @@ GOCLEAN=$(GOCMD) clean
# Tool versions (compiled on demand via `go run` so they match the local toolchain)
GOLANGCI_LINT = go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest
STATICCHECK = go run honnef.co/go/tools/cmd/staticcheck@latest
STATICCHECK = build/tools/staticcheck
GOVULNCHECK = go run golang.org/x/vuln/cmd/govulncheck@latest
# Version information
@@ -63,7 +63,16 @@ fmt-check: ## Check formatting (gofumpt + goimports via golangci-lint)
@diff=$$($(GOLANGCI_LINT) fmt --diff --config=.golangci.json); \
if [ -n "$$diff" ]; then echo "$$diff"; echo "Run: make fmt"; exit 1; fi
staticcheck: ## Run staticcheck
# staticcheck is built against the latest golang.org/x/tools so it can read export
# data from the newest Go toolchain (`go run ...@latest` uses an older, pinned x/tools).
$(STATICCHECK):
@echo "Building staticcheck..."
@mkdir -p build/tools/src
@cd build/tools/src && { [ -f go.mod ] || go mod init staticcheck-build >/dev/null 2>&1; } && \
go get honnef.co/go/tools/cmd/staticcheck@latest golang.org/x/tools@latest && \
go build -o ../staticcheck honnef.co/go/tools/cmd/staticcheck
staticcheck: $(STATICCHECK) ## Run staticcheck
@echo "Running staticcheck..."
$(STATICCHECK) ./...
+1 -1
View File
@@ -1 +1 @@
1.0.87
1.0.89
+23 -25
View File
@@ -1,27 +1,26 @@
module git.warky.dev/wdevs/relspecgo
go 1.25.13
go 1.26.9
require (
github.com/gdamore/tcell/v2 v2.13.9
github.com/go-sql-driver/mysql v1.9.3
github.com/gdamore/tcell/v2 v2.13.10
github.com/go-sql-driver/mysql v1.10.1
github.com/google/uuid v1.6.0
github.com/jackc/pgx/v5 v5.9.2
github.com/microsoft/go-mssqldb v1.10.0
github.com/jackc/pgx/v5 v5.11.0
github.com/microsoft/go-mssqldb v1.11.2
github.com/rivo/tview v0.42.0
github.com/spf13/cobra v1.10.2
github.com/stretchr/testify v1.11.1
github.com/uptrace/bun v1.2.18
github.com/uptrace/bun/dialect/pgdialect v1.2.18
golang.org/x/text v0.39.0
github.com/stretchr/testify v1.12.1
github.com/uptrace/bun v1.3.0
github.com/uptrace/bun/dialect/pgdialect v1.3.0
golang.org/x/text v0.43.0
gopkg.in/yaml.v3 v3.0.1
modernc.org/sqlite v1.50.1
modernc.org/sqlite v1.60.1
)
require (
filippo.io/edwards25519 v1.1.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
filippo.io/edwards25519 v1.2.0 // indirect
github.com/dustin/go-humanize v1.1.0 // indirect
github.com/gdamore/encoding v1.0.1 // indirect
github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9 // indirect
github.com/golang-sql/sqlexp v0.1.0 // indirect
@@ -31,25 +30,24 @@ require (
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/jinzhu/inflection v1.0.0 // indirect
github.com/kr/pretty v0.3.1 // indirect
github.com/lucasb-eyer/go-colorful v1.4.0 // indirect
github.com/mattn/go-isatty v0.0.22 // indirect
github.com/ncruces/go-strftime v1.0.0 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/lucasb-eyer/go-colorful v1.4.1 // indirect
github.com/mattn/go-isatty v0.0.24 // indirect
github.com/ncruces/go-strftime v1.1.0 // indirect
github.com/puzpuzpuz/xsync/v3 v3.5.1 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/rogpeppe/go-internal v1.14.1 // indirect
github.com/shopspring/decimal v1.4.0 // indirect
github.com/shopspring/decimal v1.5.0 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc // indirect
github.com/vmihailenco/msgpack/v5 v5.4.1 // indirect
github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
golang.org/x/crypto v0.53.0 // indirect
golang.org/x/net v0.56.0 // indirect
golang.org/x/sync v0.21.0 // indirect
golang.org/x/sys v0.46.0 // indirect
golang.org/x/term v0.44.0 // indirect
modernc.org/libc v1.72.3 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
golang.org/x/crypto v0.58.0 // indirect
golang.org/x/sync v0.24.0 // indirect
golang.org/x/sys v0.49.0 // indirect
golang.org/x/term v0.47.0 // indirect
modernc.org/libc v1.78.0 // indirect
modernc.org/mathutil v1.7.1 // indirect
modernc.org/memory v1.11.0 // indirect
modernc.org/memory v1.12.1 // indirect
)
+66 -67
View File
@@ -1,38 +1,36 @@
filippo.io/edwards25519 v1.1.0 h1:FNf4tywRC1HmFuKW5xopWpigGjJKiJSV0Cqo0cJWDaA=
filippo.io/edwards25519 v1.1.0/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.1 h1:jHb/wfvRikGdxMXYV3QG/SzUOPYN9KEUUuC0Yd0/vC0=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.1/go.mod h1:pzBXCYn05zvYIrwLgtK8Ap8QcjRg+0i76tMQdWN6wOk=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.13.1 h1:Hk5QBxZQC1jb2Fwj6mpzme37xbCDdNTxU7O9eb5+LB4=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.13.1/go.mod h1:IYus9qsFobWIc2YVwe/WPjcnyCkPKtnHAqUYeebc8z0=
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.1 h1:zvXfGJCWvywnCA814d8ZiVyt+fm9nnTE8xSb99zRyfo=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.1/go.mod h1:iptorS+VYKFL2N6PnebpS91dubG35eAOEERnT4PJbQU=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.1 h1:u93s+zU2JD62im61Bm5CZIc1ZrOJaIAWEg0WOrMVkEo=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.1/go.mod h1:oXtinPO4OLj9d1DOTrqrL1oRwGhcqadvAmrl6wTeGlk=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0 h1:fhqpLE3UEXi9lPaBRpQ6XuRW0nU7hgg4zlmZZa+a9q4=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0/go.mod h1:7dCRMLwisfRH3dBupKeNCioWYUZ4SS09Z14H+7i8ZoY=
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azkeys v1.4.0 h1:E4MgwLBGeVB5f2MdcIVD3ELVAWpr+WD6MUe1i+tM/PA=
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azkeys v1.4.0/go.mod h1:Y2b/1clN4zsAoUd/pgNAQHjLDnTis/6ROkUfyob6psM=
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azkeys v1.5.0 h1:MaKvxE6D0KkjOg6Wd9M00iqP5PR0kUxCfiezes4JweM=
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azkeys v1.5.0/go.mod h1:i2h9fsTFKZorh8RdV2IcSUf/Qj98GlTkrTvUbX/s8as=
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/internal v1.2.0 h1:nCYfgcSyHZXJI8J0IWE5MsCGlb2xp9fJiXyxWgmOFg4=
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/internal v1.2.0/go.mod h1:ucUjca2JtSZboY8IoUqyQyuuXvwbMBVwFOm0vdQPNhA=
github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0 h1:XRzhVemXdgvJqCH0sFfrBUTnUJSBrBf7++ypk+twtRs=
github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0/go.mod h1:HKpQxkWaGLJ+D/5H8QRpyQXA1eKjxkFlOMwck5+33Jk=
github.com/AzureAD/microsoft-authentication-library-for-go v1.8.0 h1:Nljr4q1GRA/5vCrMONS+g4u4LRHNgOXVSh3O43J2CnI=
github.com/AzureAD/microsoft-authentication-library-for-go v1.8.0/go.mod h1:Y33QHnf0FfdVewFFISOGe20mkZbxX4H839o955/PoeI=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/dustin/go-humanize v1.1.0 h1:dbKTrvD0klcbBV/h4AWJdMuZogJACoMlvWIWZ5b2xWg=
github.com/dustin/go-humanize v1.1.0/go.mod h1:hc1CvRkJMsgxqjmjMQF3QNRAZBwY8AXBAzKYoSX9sFI=
github.com/gdamore/encoding v1.0.1 h1:YzKZckdBL6jVt2Gc+5p82qhrGiqMdG/eNs6Wy0u3Uhw=
github.com/gdamore/encoding v1.0.1/go.mod h1:0Z0cMFinngz9kS1QfMjCP8TY7em3bZYeeklsSDPivEo=
github.com/gdamore/tcell/v2 v2.13.9 h1:uI5l3DYPcFvHINKlGft+en23evOKL+dwtD21QR8ejVA=
github.com/gdamore/tcell/v2 v2.13.9/go.mod h1:+Wfe208WDdB7INEtCsNrAN6O2m+wsTPk1RAovjaILlo=
github.com/go-sql-driver/mysql v1.9.3 h1:U/N249h2WzJ3Ukj8SowVFjdtZKfu9vlLZxjPXV1aweo=
github.com/go-sql-driver/mysql v1.9.3/go.mod h1:qn46aNg1333BRMNU69Lq93t8du/dwxI64Gl8i5p1WMU=
github.com/gdamore/tcell/v2 v2.13.10 h1:Afs3JKt83HnhuUKdZ3MnxUgOqQRWftj5JyDqv1LLynA=
github.com/gdamore/tcell/v2 v2.13.10/go.mod h1:+Wfe208WDdB7INEtCsNrAN6O2m+wsTPk1RAovjaILlo=
github.com/go-sql-driver/mysql v1.10.1 h1:arlSnNLq6a5yxGxV7qg9lF4j0C+KwD6NbQyKr9QL6ME=
github.com/go-sql-driver/mysql v1.10.1/go.mod h1:M+cqaI7+xxXGG9swrdeUIoPG3Y3KCkF0pZej+SK+nWk=
github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY=
github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9 h1:au07oEsX2xN0ktxqI+Sida1w446QrXBRJ0nee3SNZlA=
github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9/go.mod h1:8vg3r2VgvsThLBIFL93Qb5yWzgyZWhEmBwUJWevAkK0=
github.com/golang-sql/sqlexp v0.1.0 h1:ZCD6MBpcuOVfGVqsEmY5/4FtYiKz6tSyUv9LPEDei6A=
github.com/golang-sql/sqlexp v0.1.0/go.mod h1:J4ad9Vo8ZCWQ2GMrC4UCQy1JpCbwU9m3EOqtpKwwwHI=
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs=
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA=
github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3 h1:LMLX+LgTNWpfvCBdFebv6EsYotImrt/Ppc5cXIriCSo=
github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3/go.mod h1:jl5iWTm0/hd5PjEYEOuwAJ57L/CibdZfrqZ5XA5GrCk=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
@@ -43,8 +41,8 @@ github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsI
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgx/v5 v5.9.2 h1:3ZhOzMWnR4yJ+RW1XImIPsD1aNSz4T4fyP7zlQb56hw=
github.com/jackc/pgx/v5 v5.9.2/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/pgx/v5 v5.11.0 h1:IzBBtyK9AHqf98cctWFifYSci2hgQR/cd56wB4p+ogg=
github.com/jackc/pgx/v5 v5.11.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/jinzhu/inflection v1.0.0 h1:K317FqzuhWc8YvSVlFMCCUb36O/S9MCKRDI7QkRKD/E=
@@ -55,18 +53,17 @@ github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/lucasb-eyer/go-colorful v1.4.0 h1:UtrWVfLdarDgc44HcS7pYloGHJUjHV/4FwW4TvVgFr4=
github.com/lucasb-eyer/go-colorful v1.4.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
github.com/mattn/go-isatty v0.0.22 h1:j8l17JJ9i6VGPUFUYoTUKPSgKe/83EYU2zBC7YNKMw4=
github.com/mattn/go-isatty v0.0.22/go.mod h1:ZXfXG4SQHsB/w3ZeOYbR0PrPwLy+n6xiMrJlRFqopa4=
github.com/microsoft/go-mssqldb v1.10.0 h1:pHEt+Qz6YFPWqREq10mqSE524QQo+/QremwTCQht7TY=
github.com/microsoft/go-mssqldb v1.10.0/go.mod h1:mnG7lGa9iYJbzJqGCXyuQCegStKMr3kogDLD6+bmggg=
github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
github.com/lucasb-eyer/go-colorful v1.4.1 h1:1EO+WB73+EH8EVbzlrG3KLAfEypQWVHIBqlTf+2hNss=
github.com/lucasb-eyer/go-colorful v1.4.1/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI=
github.com/mattn/go-isatty v0.0.24/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A=
github.com/microsoft/go-mssqldb v1.11.2 h1:FCgeBIK8um2+X4tbun6Q71N1KsfyCDPKY41e1yGVjSE=
github.com/microsoft/go-mssqldb v1.11.2/go.mod h1:CYgwG5AMXFojbjTg+GNP5G/y6uz1BhTyZaPqQWzkGnQ=
github.com/ncruces/go-strftime v1.1.0 h1:vtPmPWMkb539RGV9WpWT9EwJGvrOCKvMNI4obWL47Hk=
github.com/ncruces/go-strftime v1.1.0/go.mod h1:yVaq7iwCN09vHRrKBrrflXT0M9tTMG2KmZ9Fj7xeG/g=
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c h1:+mdjkGKdHQG3305AYmdv1U2eRNDiU2ErMBj1gwrq8eQ=
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c/go.mod h1:7rwL4CYBLnjLxUqIJNnCWiEdr3bn6IUYi15bNlnbCCU=
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/puzpuzpuz/xsync/v3 v3.5.1 h1:GJYJZwO6IdxN/IKbneznS6yPkVC+c3zyY/j19c++5Fg=
github.com/puzpuzpuz/xsync/v3 v3.5.1/go.mod h1:VjzYrABPabuM4KyBh1Ftq6u8nhwY5tBPKP9jpmh0nnA=
@@ -80,8 +77,8 @@ github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/f
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k=
github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
github.com/shopspring/decimal v1.5.0 h1:HxSWk31RmJv0EyY4oLiGQB9u7WPludUwPBdvQZnYFPc=
github.com/shopspring/decimal v1.5.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
@@ -90,65 +87,67 @@ github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3A
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc h1:9lRDQMhESg+zvGYmW5DyG0UqvY96Bu5QYsTLvCHdrgo=
github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc/go.mod h1:bciPuU6GHm1iF1pBvUfxfsH0Wmnc2VbpgvbI9ZWuIRs=
github.com/uptrace/bun v1.2.18 h1:3HnRcMfS6OBPMG1eSOzlbFJ/X/AyMEJb7rMxE6VQvDU=
github.com/uptrace/bun v1.2.18/go.mod h1:wNltaKJk4JtOt4SG5I5zmA7v0/Mzjh1+/S906Rayd3Y=
github.com/uptrace/bun/dialect/pgdialect v1.2.18 h1:IZ6nM2+OYrL8lkEAy7UkSEZvoa3vluTAUlZfPtlRB2k=
github.com/uptrace/bun/dialect/pgdialect v1.2.18/go.mod h1:Tqdf4QP1okrGYpXfodXvCOK6Ob1OOTwSaoAzCgBB3IU=
github.com/uptrace/bun v1.3.0 h1:hUcbfRpbQk4/upFSI+uZf6jSJw5A0lkIKt0XdpvbUZo=
github.com/uptrace/bun v1.3.0/go.mod h1:wNltaKJk4JtOt4SG5I5zmA7v0/Mzjh1+/S906Rayd3Y=
github.com/uptrace/bun/dialect/pgdialect v1.3.0 h1:0ZkGqGugxaPCxko4DV2f4d7i+FnM2Qj3rNpChfhiv54=
github.com/uptrace/bun/dialect/pgdialect v1.3.0/go.mod h1:kP5HGEpAX8iJCqj3WHm04XnoeBPacdCN1P2HiwMtHpg=
github.com/vmihailenco/msgpack/v5 v5.4.1 h1:cQriyiUvjTwOHg8QZaPihLWeRAAVoCpE00IUPn0Bjt8=
github.com/vmihailenco/msgpack/v5 v5.4.1/go.mod h1:GaZTsDaehaPpQVyxrf5mtQlH+pc21PIudVV/E3rRQok=
github.com/vmihailenco/tagparser/v2 v2.0.0 h1:y09buUbR+b5aycVFQs/g70pqKVZNBmxwAhO7/IwNM9g=
github.com/vmihailenco/tagparser/v2 v2.0.0/go.mod h1:Wri+At7QHww0WTrCBeu4J6bNtoV6mEfg5OIWRZA9qds=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto=
golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54=
golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c=
golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/net v0.60.0 h1:79p50tfZlm0J9YfoDsSi639qSXNGVwEzOPLCxM2FsYU=
golang.org/x/net v0.60.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.44.0 h1:0rLvDRCtNj0gZkyIXhCyOb2OAzEhLVqc4B+hrsBhrmc=
golang.org/x/term v0.44.0/go.mod h1:7ze4MdzUzLXpSAoFP1H0bOI9aXDqveSvatT5vKcFh2Y=
golang.org/x/term v0.47.0 h1:cpyOwQkr2Nzxe2p6KnrTG7Ei2wtzjsuSjhpXXepZ/BM=
golang.org/x/term v0.47.0/go.mod h1:efAfAuG6RKABvg1ArGebRkvX6LXul14SM67o2e4mUCY=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.39.0 h1:UbZz4pLOvn600D6Oh6GGEI6VAmndrEBLv8/6BEXzyus=
golang.org/x/text v0.39.0/go.mod h1:3UwRclnC2g0TU9x8PZiyfOajCd1zaUNHF9cvqcQZ+ZM=
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
golang.org/x/tools v0.51.0 h1:k4Xc/1Om9jwkBJBo4NVLMSARBoWtK10mx+W5BnXCeAI=
golang.org/x/tools v0.51.0/go.mod h1:9eEncMayCV6zRMGhR5eZEC2iBx98qWcF1HZ9Z7wJOoA=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
@@ -156,30 +155,30 @@ gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EV
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
modernc.org/cc/v4 v4.28.2 h1:3tQ0lf2ADtoby2EtSP+J7IE2SHwEJdP8ioR59wx7XpY=
modernc.org/cc/v4 v4.28.2/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=
modernc.org/ccgo/v4 v4.34.0 h1:yRLPFZieg532OT4rp4JFNIVcquwalMX26G95WQDqwCQ=
modernc.org/ccgo/v4 v4.34.0/go.mod h1:AS5WYMyBakQ+fhsHhtP8mWB82KTGPkNNJDGfGQCe0/A=
modernc.org/cc/v4 v4.30.1 h1:ZXInYH2CD24GPWSwOFPwB6JrnFt4/6d7ceW1Ukd30Bk=
modernc.org/cc/v4 v4.30.1/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=
modernc.org/ccgo/v4 v4.36.2 h1:j6d5HuXtBxF00vEcI5RE0O6EIrG6RyRpgMiGV2IK9ys=
modernc.org/ccgo/v4 v4.36.2/go.mod h1:Wh8QlPZq/US2DrsCc31FxJVBOEEcN2BBfla6OQDHydA=
modernc.org/fileutil v1.4.0 h1:j6ZzNTftVS054gi281TyLjHPp6CPHr2KCxEXjEbD6SM=
modernc.org/fileutil v1.4.0/go.mod h1:EqdKFDxiByqxLk8ozOxObDSfcVOv/54xDs/DUHdvCUU=
modernc.org/gc/v2 v2.6.5 h1:nyqdV8q46KvTpZlsw66kWqwXRHdjIlJOhG6kxiV/9xI=
modernc.org/gc/v2 v2.6.5/go.mod h1:YgIahr1ypgfe7chRuJi2gD7DBQiKSLMPgBQe9oIiito=
modernc.org/gc/v3 v3.1.2 h1:ZtDCnhonXSZexk/AYsegNRV1lJGgaNZJuKjJSWKyEqo=
modernc.org/gc/v3 v3.1.2/go.mod h1:HFK/6AGESC7Ex+EZJhJ2Gni6cTaYpSMmU/cT9RmlfYY=
modernc.org/gc/v3 v3.2.0 h1:/Y3LGDMQaoYf55PebP1qsZ5FDGgAZyG1RYKEHeQzh1o=
modernc.org/gc/v3 v3.2.0/go.mod h1:HFK/6AGESC7Ex+EZJhJ2Gni6cTaYpSMmU/cT9RmlfYY=
modernc.org/goabi0 v0.2.0 h1:HvEowk7LxcPd0eq6mVOAEMai46V+i7Jrj13t4AzuNks=
modernc.org/goabi0 v0.2.0/go.mod h1:CEFRnnJhKvWT1c1JTI3Avm+tgOWbkOu5oPA8eH8LnMI=
modernc.org/libc v1.72.3 h1:ZnDF4tXn4NBXFutMMQC4vtbTFSXhhKzR73fv0beZEAU=
modernc.org/libc v1.72.3/go.mod h1:dn0dZNnnn1clLyvRxLxYExxiKRZIRENOfqQ8XEeg4Qs=
modernc.org/libc v1.78.0 h1:Lpm+e9K6ruVYEGwWfuBO1xXtYEaXDouFOG1SuWjFDP4=
modernc.org/libc v1.78.0/go.mod h1:wvxR55nOF2/66NkJoNV7/coXBTCvT3HR22rcqW6lOuA=
modernc.org/mathutil v1.7.1 h1:GCZVGXdaN8gTqB1Mf/usp1Y/hSqgI2vAGGP4jZMCxOU=
modernc.org/mathutil v1.7.1/go.mod h1:4p5IwJITfppl0G4sUEDtCr4DthTaT47/N3aT6MhfgJg=
modernc.org/memory v1.11.0 h1:o4QC8aMQzmcwCK3t3Ux/ZHmwFPzE6hf2Y5LbkRs+hbI=
modernc.org/memory v1.11.0/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw=
modernc.org/memory v1.12.1 h1:nFMiWrpStgZczNl6XI9GnIk/rWhYIyHGUaR04pGbp9g=
modernc.org/memory v1.12.1/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw=
modernc.org/opt v0.2.0 h1:tGyef5ApycA7FSEOMraay9SaTk5zmbx7Tu+cJs4QKZg=
modernc.org/opt v0.2.0/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns=
modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w=
modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE=
modernc.org/sqlite v1.50.1 h1:l+cQvn0sd0zJJtfygGHuQJ5AjlrwXmWPw4KP3ZMwr9w=
modernc.org/sqlite v1.50.1/go.mod h1:tcNzv5p84E0skkmJn038y+hWJbLQXQqEnQfeh5r2JLM=
modernc.org/sqlite v1.60.1 h1:/blz53O951KWFOso4QQvEs/Fq6cDBKLtMVrYNSeJVKw=
modernc.org/sqlite v1.60.1/go.mod h1:1dIoEagfDE72QytD5scH1lxARtaUgKgHC/NuApA27r0=
modernc.org/strutil v1.2.1 h1:UneZBkQA+DX2Rp35KcM69cSsNES9ly8mQWD71HKlOA0=
modernc.org/strutil v1.2.1/go.mod h1:EHkiggD70koQxjVdSBM3JKM7k6L0FbGE5eymy9i3B9A=
modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
+5 -5
View File
@@ -1,17 +1,17 @@
# Maintainer: Hein (Warky Devs) <hein@warky.dev>
pkgname=relspec
pkgver=1.0.87
pkgver=1.0.89
pkgrel=1
pkgdesc="RelSpec is a comprehensive database relations management tool that reads, transforms, and writes database table specifications across multiple formats and ORMs."
arch=('x86_64' 'aarch64')
url="https://git.warky.dev/wdevs/relspecgo"
license=('MIT')
makedepends=('go')
source=("$pkgname-$pkgver.zip::$url/archive/v$pkgver.zip")
source=("$pkgname-$pkgver.zip::$url/releases/download/v$pkgver/relspec-v$pkgver-source.zip")
sha256sums=('SKIP')
build() {
cd "relspecgo"
cd "$pkgname-$pkgver"
export CGO_ENABLED=0
go build \
-trimpath \
@@ -20,12 +20,12 @@ build() {
}
check() {
cd "relspecgo"
cd "$pkgname-$pkgver"
go test ./...
}
package() {
cd "relspecgo"
cd "$pkgname-$pkgver"
# Binary
install -Dm755 "$pkgname" "$pkgdir/usr/bin/$pkgname"
+1 -1
View File
@@ -1,5 +1,5 @@
Name: relspec
Version: 1.0.87
Version: 1.0.89
Release: 1%{?dist}
Summary: RelSpec is a comprehensive database relations management tool that reads, transforms, and writes database table specifications across multiple formats and ORMs.
+3
View File
@@ -504,6 +504,9 @@ func compareConstraintDetails(source, target *models.Constraint) map[string]any
if normalizeConstraintAction(source.OnUpdate) != normalizeConstraintAction(target.OnUpdate) {
changes["on_update"] = map[string]string{"source": source.OnUpdate, "target": target.OnUpdate}
}
if !models.DeferralMatches(source, target) {
changes["deferral"] = map[string]string{"source": source.DeferralClause(), "target": target.DeferralClause()}
}
return changes
}
+10 -2
View File
@@ -605,8 +605,8 @@ func cloneConstraint(constraint *models.Constraint) *models.Constraint {
OnDelete: constraint.OnDelete,
Expression: constraint.Expression,
Name: constraint.Name,
Deferrable: constraint.Deferrable,
InitiallyDeferred: constraint.InitiallyDeferred,
Deferrable: copyBoolPtr(constraint.Deferrable),
InitiallyDeferred: copyBoolPtr(constraint.InitiallyDeferred),
Sequence: constraint.Sequence,
}
copy(newConstraint.Columns, constraint.Columns)
@@ -823,3 +823,11 @@ func min(a, b int) int {
}
return b
}
func copyBoolPtr(b *bool) *bool {
if b == nil {
return nil
}
v := *b
return &v
}
+48
View File
@@ -0,0 +1,48 @@
package models
// DeferralClause returns the SQL deferral clause for a foreign key constraint
// ("DEFERRABLE", "DEFERRABLE INITIALLY DEFERRED" or "NOT DEFERRABLE"), or ""
// when the model does not specify one. InitiallyDeferred without Deferrable
// implies DEFERRABLE; InitiallyDeferred=false alone specifies nothing.
func (d *Constraint) DeferralClause() string {
deferrable, initially, set := d.effectiveDeferral()
switch {
case !set:
return ""
case !deferrable:
return "NOT DEFERRABLE"
case initially != nil && *initially:
return "DEFERRABLE INITIALLY DEFERRED"
default:
return "DEFERRABLE"
}
}
// effectiveDeferral resolves the optional flags. set is false when the model says nothing
// about deferral; initially is nil when the initial mode is unspecified.
func (d *Constraint) effectiveDeferral() (deferrable bool, initially *bool, set bool) {
switch {
case d.Deferrable != nil:
return *d.Deferrable, d.InitiallyDeferred, true
case d.InitiallyDeferred != nil && *d.InitiallyDeferred:
return true, d.InitiallyDeferred, true
}
return false, nil, false
}
// DeferralMatches reports whether two constraints agree on deferral. An unspecified side
// matches anything, and the initial mode is only compared when both sides specify it.
func DeferralMatches(a, b *Constraint) bool {
da, ia, aSet := a.effectiveDeferral()
db, ib, bSet := b.effectiveDeferral()
if !aSet || !bSet {
return true
}
if da != db {
return false
}
if !da || ia == nil || ib == nil {
return true
}
return *ia == *ib
}
+59
View File
@@ -0,0 +1,59 @@
package models
import "testing"
func boolPtr(b bool) *bool { return &b }
func TestConstraint_DeferralClause(t *testing.T) {
tests := []struct {
name string
c Constraint
want string
}{
{"unspecified", Constraint{}, ""},
{"deferrable", Constraint{Deferrable: boolPtr(true)}, "DEFERRABLE"},
{"deferrable initially deferred", Constraint{Deferrable: boolPtr(true), InitiallyDeferred: boolPtr(true)}, "DEFERRABLE INITIALLY DEFERRED"},
{"deferrable initially immediate", Constraint{Deferrable: boolPtr(true), InitiallyDeferred: boolPtr(false)}, "DEFERRABLE"},
{"initially deferred implies deferrable", Constraint{InitiallyDeferred: boolPtr(true)}, "DEFERRABLE INITIALLY DEFERRED"},
{"initially immediate alone is unspecified", Constraint{InitiallyDeferred: boolPtr(false)}, ""},
{"explicitly not deferrable", Constraint{Deferrable: boolPtr(false)}, "NOT DEFERRABLE"},
{"not deferrable ignores initially deferred", Constraint{Deferrable: boolPtr(false), InitiallyDeferred: boolPtr(true)}, "NOT DEFERRABLE"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := tt.c.DeferralClause(); got != tt.want {
t.Errorf("DeferralClause() = %q, want %q", got, tt.want)
}
})
}
}
func TestDeferralMatches(t *testing.T) {
live := func(def, init bool) *Constraint {
return &Constraint{Deferrable: boolPtr(def), InitiallyDeferred: boolPtr(init)}
}
tests := []struct {
name string
model, other *Constraint
want bool
}{
{"model unspecified vs deferrable live", &Constraint{}, live(true, false), true},
{"model unspecified vs not deferrable live", &Constraint{}, live(false, false), true},
{"deferrable vs deferrable live", &Constraint{Deferrable: boolPtr(true)}, live(true, true), true},
{"deferrable vs not deferrable live", &Constraint{Deferrable: boolPtr(true)}, live(false, false), false},
{"not deferrable vs deferrable live", &Constraint{Deferrable: boolPtr(false)}, live(true, false), false},
{"initially deferred vs initially immediate live", &Constraint{InitiallyDeferred: boolPtr(true)}, live(true, false), false},
{"initially deferred vs initially deferred live", &Constraint{InitiallyDeferred: boolPtr(true)}, live(true, true), true},
{"explicit initially immediate vs initially deferred live", &Constraint{Deferrable: boolPtr(true), InitiallyDeferred: boolPtr(false)}, live(true, true), false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := DeferralMatches(tt.model, tt.other); got != tt.want {
t.Errorf("DeferralMatches() = %v, want %v", got, tt.want)
}
if got := DeferralMatches(tt.other, tt.model); got != tt.want {
t.Errorf("DeferralMatches() reversed = %v, want %v", got, tt.want)
}
})
}
}
+2 -2
View File
@@ -329,8 +329,8 @@ type Constraint struct {
ReferencedColumns []string `json:"referenced_columns" yaml:"referenced_columns" xml:"referenced_columns"`
OnDelete string `json:"on_delete" yaml:"on_delete" xml:"on_delete"` // CASCADE, SET NULL, RESTRICT, etc.
OnUpdate string `json:"on_update" yaml:"on_update" xml:"on_update"`
Deferrable bool `json:"deferrable,omitempty" yaml:"deferrable,omitempty" xml:"deferrable,omitempty"`
InitiallyDeferred bool `json:"initially_deferred,omitempty" yaml:"initially_deferred,omitempty" xml:"initially_deferred,omitempty"`
Deferrable *bool `json:"deferrable,omitempty" yaml:"deferrable,omitempty" xml:"deferrable,omitempty"` // nil: unspecified, left as is
InitiallyDeferred *bool `json:"initially_deferred,omitempty" yaml:"initially_deferred,omitempty" xml:"initially_deferred,omitempty"` // nil: unspecified, left as is
Sequence uint `json:"sequence,omitempty" yaml:"sequence,omitempty" xml:"sequence,omitempty"`
GUID string `json:"guid" yaml:"guid" xml:"guid"`
}
+25 -1
View File
@@ -86,16 +86,40 @@ Ref: posts.user_id > users.id [delete: cascade]
- Table definitions with columns
- Primary keys (`pk`)
- Not null constraints (`not null`)
- Unique constraints (`unique`)
- Unique constraints (`unique`) → constraint `ukey_<table>_<col>`
- Default values (`default`)
- Inline references (`ref`)
- Standalone `Ref` blocks
- FK deferral: `deferrable`, `initially deferred` (see below)
- Commented cross-file refs (`// Ref:` — see below)
- Indexes and composite indexes
- Table notes and column notes
- Enums
- Dialect directives (`@postgres:` / `@sqlite:` — see below)
## Naming
| Source | Generated name |
| ------ | -------------- |
| Column `[unique]` | constraint `ukey_<table>_<col>` |
| Unnamed unique index | index `ukey_<table>_<cols>` (same name as the constraint on the same columns) |
| Unnamed index | `idx_<table>_<cols>` |
Explicit `name:` is never changed. Leading `_` in the table name is stripped.
## FK deferral
```
col int [ref: > t.id, deferrable, initially deferred]
Ref: a.x > b.id [delete: cascade, deferrable]
```
| Setting | SQL |
| ------- | --- |
| `deferrable` | `DEFERRABLE` |
| `initially deferred` | `DEFERRABLE INITIALLY DEFERRED` |
| neither | unspecified (existing FK left as is) |
## Commented cross-file refs
`// Ref:` / `// ref:` lines (ignored by dbdiagram) become FKs + relationships once both ends are loaded.
+26 -3
View File
@@ -848,6 +848,13 @@ func (r *Reader) parseColumn(line, tableName, schemaName string) (*models.Column
}
}
// "deferrable" / "initially deferred" column settings apply to the inline foreign key
if constraint != nil && constraint.Type == models.ForeignKeyConstraint && attrs != "" {
for _, attr := range splitColumnAttrs(attrs) {
applyDeferralSetting(constraint, attr)
}
}
// Parse inline comment
if inlineComment != "" {
column.Comment = inlineComment
@@ -1081,11 +1088,13 @@ func (r *Reader) parseIndex(line, tableName, schemaName string) *models.Index {
// Generate name if not provided
if index.Name == "" {
prefix := "idx"
// Unique indexes share the ukey_ naming of unique constraints, so an index and a
// constraint on the same columns get the same name and can be merged on write.
prefix, name := "idx", tableName
if index.Unique {
prefix = "uidx"
prefix, name = "ukey", strings.TrimLeft(tableName, "_")
}
index.Name = fmt.Sprintf("%s_%s_%s", prefix, tableName, strings.Join(columns, "_"))
index.Name = fmt.Sprintf("%s_%s_%s", prefix, name, strings.Join(columns, "_"))
}
return index
@@ -1182,6 +1191,8 @@ func (r *Reader) parseRef(refStr string) *models.Constraint {
constraint.OnDelete = strings.TrimSpace(strings.TrimPrefix(action, "ondelete:"))
} else if strings.HasPrefix(action, "onupdate:") {
constraint.OnUpdate = strings.TrimSpace(strings.TrimPrefix(action, "onupdate:"))
} else {
applyDeferralSetting(constraint, action)
}
}
}
@@ -1190,6 +1201,18 @@ func (r *Reader) parseRef(refStr string) *models.Constraint {
return constraint
}
// applyDeferralSetting applies a "deferrable" or "initially deferred" Ref setting to a
// foreign key constraint. Other settings are ignored.
func applyDeferralSetting(constraint *models.Constraint, setting string) {
yes := true
switch strings.Join(strings.Fields(strings.ToLower(setting)), " ") {
case "deferrable":
constraint.Deferrable = &yes
case "initially deferred", "initially_deferred":
constraint.InitiallyDeferred = &yes
}
}
// parseTableRef parses a table reference like "schema.table.(column1, column2)" or "schema"."table"."column"
func (r *Reader) parseTableRef(ref string) (schema, table string, columns []string) {
// Extract columns if present in parentheses format
+70 -3
View File
@@ -3,6 +3,7 @@ package dbml
import (
"os"
"path/filepath"
"strings"
"testing"
"git.warky.dev/wdevs/relspecgo/pkg/models"
@@ -842,9 +843,9 @@ func TestConstraintNaming(t *testing.T) {
t.Logf("Available constraints: %v", getKeys(postsTable.Constraints))
}
// Test unique index naming: uidx_table_columns
if _, exists := postsTable.Indexes["uidx_posts_slug"]; !exists {
t.Error("Expected unique index 'uidx_posts_slug' not found")
// Test unique index naming: ukey_table_columns, matching the unique constraint name
if _, exists := postsTable.Indexes["ukey_posts_slug"]; !exists {
t.Error("Expected unique index 'ukey_posts_slug' not found")
t.Logf("Available indexes: %v", getKeys(postsTable.Indexes))
}
@@ -1121,3 +1122,69 @@ func TestReader_DuplicateIndexNameInTableKept(t *testing.T) {
}
}
}
func TestReader_ForeignKeyDeferral(t *testing.T) {
dbml := `Table users {
id integer [pk]
}
Table posts {
id integer [pk]
user_id integer [ref: > users.id, deferrable, initially deferred]
editor_id integer
plain_id integer [ref: > users.id]
}
Table comments {
id integer [pk]
post_id integer
author_id integer
}
Ref: comments.post_id > posts.id [delete: cascade, deferrable]
Ref: comments.author_id > users.id [deferrable, initially deferred]
`
path := filepath.Join(t.TempDir(), "deferral.dbml")
if err := os.WriteFile(path, []byte(dbml), 0o600); err != nil {
t.Fatal(err)
}
db, err := NewReader(&readers.ReaderOptions{FilePath: path}).ReadDatabase()
if err != nil {
t.Fatalf("ReadDatabase() error = %v", err)
}
find := func(table, column string) *models.Constraint {
for _, s := range db.Schemas {
for _, tb := range s.Tables {
if tb.Name != table {
continue
}
for _, c := range tb.Constraints {
if c.Type == models.ForeignKeyConstraint && len(c.Columns) == 1 && c.Columns[0] == column {
return c
}
}
}
}
t.Fatalf("FK %s.%s not found", table, column)
return nil
}
tests := []struct {
table, column string
want string
}{
{"posts", "user_id", "DEFERRABLE INITIALLY DEFERRED"},
{"posts", "plain_id", ""},
{"comments", "post_id", "DEFERRABLE"},
{"comments", "author_id", "DEFERRABLE INITIALLY DEFERRED"},
}
for _, tt := range tests {
if got := find(tt.table, tt.column).DeferralClause(); got != tt.want {
t.Errorf("%s.%s deferral = %q, want %q", tt.table, tt.column, got, tt.want)
}
}
if got := find("comments", "post_id").OnDelete; !strings.EqualFold(got, "cascade") {
t.Errorf("delete action lost, got %q", got)
}
}
+12 -2
View File
@@ -439,7 +439,9 @@ func (r *Reader) queryForeignKeys(schemaName string) (map[string][]*models.Const
fcl.relname AS referenced_table_name,
fatt.attname AS referenced_column,
` + actionCase("con.confupdtype") + ` AS update_rule,
` + actionCase("con.confdeltype") + ` AS delete_rule
` + actionCase("con.confdeltype") + ` AS delete_rule,
con.condeferrable AS deferrable,
con.condeferred AS initially_deferred
FROM pg_catalog.pg_constraint con
JOIN pg_catalog.pg_class cl ON cl.oid = con.conrelid
JOIN pg_catalog.pg_namespace ns ON ns.oid = cl.relnamespace
@@ -470,6 +472,8 @@ func (r *Reader) queryForeignKeys(schemaName string) (map[string][]*models.Const
referencedColumns []string
updateRule string
deleteRule string
deferrable bool
initiallyDeferred bool
}
fkMap := make(map[string]*fkData)
@@ -479,8 +483,9 @@ func (r *Reader) queryForeignKeys(schemaName string) (map[string][]*models.Const
var foreignSchema, foreignTable, foreignColumn string
var referencedSchema, referencedTable, referencedColumn string
var updateRule, deleteRule string
var deferrable, initiallyDeferred bool
if err := rows.Scan(&schema, &tableName, &constraintName, &foreignSchema, &foreignTable, &foreignColumn, &referencedSchema, &referencedTable, &referencedColumn, &updateRule, &deleteRule); err != nil {
if err := rows.Scan(&schema, &tableName, &constraintName, &foreignSchema, &foreignTable, &foreignColumn, &referencedSchema, &referencedTable, &referencedColumn, &updateRule, &deleteRule, &deferrable, &initiallyDeferred); err != nil {
return nil, err
}
@@ -497,6 +502,8 @@ func (r *Reader) queryForeignKeys(schemaName string) (map[string][]*models.Const
referencedColumns: []string{},
updateRule: updateRule,
deleteRule: deleteRule,
deferrable: deferrable,
initiallyDeferred: initiallyDeferred,
}
}
@@ -517,6 +524,9 @@ func (r *Reader) queryForeignKeys(schemaName string) (map[string][]*models.Const
constraint.ReferencedColumns = fk.referencedColumns
constraint.OnUpdate = fk.updateRule
constraint.OnDelete = fk.deleteRule
deferrable, initiallyDeferred := fk.deferrable, fk.initiallyDeferred
constraint.Deferrable = &deferrable
constraint.InitiallyDeferred = &initiallyDeferred
tableKey := fk.schema + "." + fk.tableName
foreignKeys[tableKey] = append(foreignKeys[tableKey], constraint)
+7
View File
@@ -273,6 +273,13 @@ func (w *Writer) constraintToDBML(c *models.Constraint, t *models.Table) string
actions = append(actions, fmt.Sprintf("update: %s", c.OnUpdate))
}
if c.Deferrable != nil && *c.Deferrable {
actions = append(actions, "deferrable")
}
if c.InitiallyDeferred != nil && *c.InitiallyDeferred {
actions = append(actions, "initially deferred")
}
refLine := fmt.Sprintf("Ref: %s %s %s", fromRef, relationship, toRef)
if len(actions) > 0 {
refLine += fmt.Sprintf(" [%s]", strings.Join(actions, ", "))
+28
View File
@@ -171,3 +171,31 @@ func TestWriter_WriteDatabase_OneToOneRelationship(t *testing.T) {
assert.Contains(t, output, "Ref: public.profiles.user_id - public.users.id")
}
func TestWriter_WriteDatabase_ForeignKeyDeferral(t *testing.T) {
yes := true
db := models.InitDatabase("test_db")
schema := models.InitSchema("public")
users := models.InitTable("users", "public")
users.Columns["id"] = models.InitColumn("id", "users", "public")
users.Columns["id"].Type = "bigint"
posts := models.InitTable("posts", "public")
posts.Columns["user_id"] = models.InitColumn("user_id", "posts", "public")
posts.Columns["user_id"].Type = "bigint"
fk := models.InitConstraint("fk_posts_user", models.ForeignKeyConstraint)
fk.Table, fk.Schema = "posts", "public"
fk.Columns = []string{"user_id"}
fk.ReferencedTable, fk.ReferencedSchema = "users", "public"
fk.ReferencedColumns = []string{"id"}
fk.Deferrable = &yes
fk.InitiallyDeferred = &yes
posts.Constraints[fk.Name] = fk
schema.Tables = append(schema.Tables, users, posts)
db.Schemas = append(db.Schemas, schema)
outputPath := filepath.Join(t.TempDir(), "test.dbml")
assert.NoError(t, NewWriter(&writers.WriterOptions{OutputPath: outputPath}).WriteDatabase(db))
content, err := os.ReadFile(outputPath)
assert.NoError(t, err)
assert.Contains(t, string(content), "Ref: public.posts.user_id > public.users.id [deferrable, initially deferred]")
}
+13 -13
View File
@@ -8,8 +8,8 @@ Standardized naming rules for all database objects in RelSpec PostgreSQL output.
| ----------------- | ----------- | ---------------------------------- | ------------------------ |
| Primary Key | `pk_` | `pk_<schema>_<table>` | `pk_public_users` |
| Foreign Key | `fk_` | `fk_<table>_<referenced_table>` | `fk_posts_users` |
| Unique Constraint | `uk_` | `uk_<table>_<column>` | `uk_users_email` |
| Unique Index | `uidx_` | `uidx_<table>_<column>` | `uidx_users_email` |
| Unique Constraint | `ukey_` | `ukey_<table>_<column>` | `ukey_users_email` |
| Unique Index | `ukey_` | `ukey_<table>_<column>` | `ukey_users_email` |
| Regular Index | `idx_` | `idx_<table>_<column>` | `idx_posts_user_id` |
| Check Constraint | `chk_` | `chk_<table>_<constraint_purpose>` | `chk_users_age_positive` |
| Sequence | `identity_` | `identity_<table>_<column>` | `identity_users_id` |
@@ -43,25 +43,25 @@ Standardized naming rules for all database objects in RelSpec PostgreSQL output.
### Unique Constraints
**Pattern:** `uk_<table>_<column>`
**Pattern:** `ukey_<table>_<column>`
- Use `uk_` prefix strictly for database constraints (CONSTRAINT type)
- Generated by the DBML reader for column `[unique]`
- Include column name for clarity
- Examples:
- `uk_users_email`
- `uk_users_username`
- `uk_products_sku`
- `ukey_users_email`
- `ukey_users_username`
- `ukey_products_sku`
### Unique Indexes
**Pattern:** `uidx_<table>_<column>`
**Pattern:** `ukey_<table>_<column>`
- Use `uidx_` prefix strictly for index type objects
- Distinguished from constraints for clarity and implementation flexibility
- Unnamed unique indexes use the same name as a unique constraint on the same columns
- Writers then build the constraint on the index (`ADD CONSTRAINT ... UNIQUE USING INDEX`), so no duplicate index is created
- Examples:
- `uidx_users_email`
- `uidx_sessions_token`
- `uidx_api_keys_key`
- `ukey_users_email`
- `ukey_sessions_token`
- `ukey_api_keys_key`
### Regular Indexes
+23
View File
@@ -154,6 +154,29 @@ When `migration_mode` is enabled, generates migration scripts with:
- Transactional DDL
- Rollback support
### Unique constraints and indexes
- Unique constraint → `ADD CONSTRAINT <name> UNIQUE (cols)`, idempotent (checked by name)
- Migration mode and direct apply both emit them (not only `convert`)
- Same-named model unique index → `UNIQUE USING INDEX` (no duplicate index). Requires: same name, same columns, btree, non-partial, no `INCLUDE`
- Otherwise a plain `ADD CONSTRAINT` (second index is built)
- Naming: unique constraint and unique index both `ukey_<table>_<cols>`; see `NAMING_CONVENTIONS.md`
- Renaming `uidx_…` → `ukey_…` drops the old index and re-adds the constraint on the next diff
### FK deferral
| Model (`Deferrable` / `InitiallyDeferred`) | SQL |
| ------------------------------------------ | --- |
| unset | create: `DEFERRABLE` (migration/apply), none (full DDL constraints); existing FK never diffed |
| `true` / unset or `false` | `DEFERRABLE` |
| `true` / `true`, or unset / `true` | `DEFERRABLE INITIALLY DEFERRED` |
| `false` | `NOT DEFERRABLE` |
- Both fields are `*bool`; `nil` = unspecified
- Specified value differing from the live DB → FK dropped and re-added
- Live reader reads `condeferrable` / `condeferred`
- Initial mode compared only when both sides state it
### Audit Triggers
When `include_audit` is enabled, adds:
+103
View File
@@ -222,6 +222,13 @@ func (w *MigrationWriter) generateSchemaScripts(model, current *models.Schema) (
}
scripts = append(scripts, indexScripts...)
// Phase 4.5: Create unique constraints (Priority 185)
uniqueScripts, err := w.generateUniqueConstraintScripts(model, current)
if err != nil {
return nil, fmt.Errorf("failed to generate unique constraint scripts: %w", err)
}
scripts = append(scripts, uniqueScripts...)
// Phase 5: Create foreign keys (Priority 195)
fkScripts, err := w.generateForeignKeyScripts(model, current, droppedFKs)
if err != nil {
@@ -751,6 +758,92 @@ func buildIndexColumnExpressionsFiltered(table *models.Table, index *models.Inde
return columnExprs
}
// generateUniqueConstraintScripts generates ADD CONSTRAINT UNIQUE scripts for model unique
// constraints that are missing from, or differ in, the live schema.
func (w *MigrationWriter) generateUniqueConstraintScripts(model, current *models.Schema) ([]MigrationScript, error) {
scripts := make([]MigrationScript, 0)
currentTables := make(map[string]*models.Table)
if current != nil {
for _, table := range current.Tables {
currentTables[strings.ToLower(table.Name)] = table
}
}
for _, modelTable := range model.Tables {
currentTable := currentTables[strings.ToLower(modelTable.Name)]
for _, constraint := range sortConstraints(modelTable.Constraints) {
if constraint.Type != models.UniqueConstraint {
continue
}
if currentTable != nil {
if currentConstraint, exists := lookupConstraint(currentTable.Constraints, constraint.Name); exists && constraintsEqual(constraint, currentConstraint) {
continue
}
}
columnExprs := make([]string, 0, len(constraint.Columns))
for _, colName := range constraint.Columns {
if col, ok := modelTable.Columns[colName]; ok {
columnExprs = append(columnExprs, col.SQLName())
}
}
if len(columnExprs) == 0 {
continue
}
sql, err := w.executor.ExecuteCreateUniqueConstraint(CreateUniqueConstraintData{
SchemaName: model.Name,
TableName: modelTable.Name,
ConstraintName: constraint.Name,
Columns: strings.Join(columnExprs, ", "),
UsingIndex: adoptableUniqueIndex(modelTable, constraint),
})
if err != nil {
return nil, err
}
scripts = append(scripts, MigrationScript{
ObjectName: fmt.Sprintf("%s.%s.%s", model.Name, modelTable.Name, constraint.Name),
ObjectType: "create unique constraint",
Schema: model.Name,
Priority: 185,
Sequence: len(scripts),
Body: sql,
})
}
}
return scripts, nil
}
// adoptableUniqueIndex returns the name of the model's unique index that the given unique
// constraint can be built on with ADD CONSTRAINT ... UNIQUE USING INDEX, so the constraint does
// not create a second index (or collide with the index's name). The index must carry the
// constraint's name, be a plain non-partial btree index and cover exactly the constraint's
// columns. Index creation runs before constraint creation, so this holds whether or not the
// live database already has the index. Returns "" when none qualifies.
func adoptableUniqueIndex(table *models.Table, constraint *models.Constraint) string {
index, ok := table.Indexes[constraint.Name]
if !ok || !index.Unique || index.Where != "" || len(index.Include) > 0 {
return ""
}
if index.Type != "" && !strings.EqualFold(index.Type, "btree") {
return ""
}
if len(index.Columns) != len(constraint.Columns) {
return ""
}
for i, col := range index.Columns {
if !strings.EqualFold(col, constraint.Columns[i]) {
return ""
}
}
return index.Name
}
// generateForeignKeyScripts generates ADD CONSTRAINT FOREIGN KEY scripts using templates.
// forceRecreate is a set of FK constraint keys (schema.table.name) that must be recreated
// even if unchanged, because their referenced PK was dropped and recreated.
@@ -800,6 +893,12 @@ func (w *MigrationWriter) generateForeignKeyScripts(model, current *models.Schem
onUpdate = strings.ToUpper(constraint.OnUpdate)
}
// An unspecified deferral keeps the historical default of DEFERRABLE.
deferral := constraint.DeferralClause()
if deferral == "" {
deferral = "DEFERRABLE"
}
sql, err := w.executor.ExecuteCreateForeignKey(CreateForeignKeyData{
SchemaName: model.Name,
TableName: modelTable.Name,
@@ -810,6 +909,7 @@ func (w *MigrationWriter) generateForeignKeyScripts(model, current *models.Schem
TargetColumns: strings.Join(constraint.ReferencedColumns, ", "),
OnDelete: onDelete,
OnUpdate: onUpdate,
Deferral: deferral,
})
if err != nil {
return nil, err
@@ -1149,6 +1249,9 @@ func constraintsEqual(c1, c2 *models.Constraint) bool {
if !fkActionsEqual(c1.OnDelete, c2.OnDelete) || !fkActionsEqual(c1.OnUpdate, c2.OnUpdate) {
return false
}
if !models.DeferralMatches(c1, c2) {
return false
}
}
return true
+313
View File
@@ -1027,3 +1027,316 @@ func TestIndexesEqual_OperatorClassAndStorageParameters(t *testing.T) {
})
}
}
func uniqueConstraintTestModel() *models.Database {
model := models.InitDatabase("testdb")
schema := models.InitSchema("public")
table := models.InitTable("login", "public")
idCol := models.InitColumn("id", "login", "public")
idCol.Type = "integer"
idCol.NotNull = true
table.Columns["id"] = idCol
guidCol := models.InitColumn("guid", "login", "public")
guidCol.Type = "uuid"
table.Columns["guid"] = guidCol
uc := models.InitConstraint("ukey_login_guid", models.UniqueConstraint)
uc.Schema = "public"
uc.Table = "login"
uc.Columns = []string{"guid"}
table.Constraints[uc.Name] = uc
schema.Tables = append(schema.Tables, table)
model.Schemas = append(model.Schemas, schema)
return model
}
func TestWriteMigration_UniqueConstraint(t *testing.T) {
tests := []struct {
name string
current func() *models.Database
wantConstrnt bool
}{
{
name: "new table",
current: func() *models.Database {
c := models.InitDatabase("testdb")
c.Schemas = append(c.Schemas, models.InitSchema("public"))
return c
},
wantConstrnt: true,
},
{
name: "existing table without constraint",
current: func() *models.Database {
c := uniqueConstraintTestModel()
delete(c.Schemas[0].Tables[0].Constraints, "ukey_login_guid")
return c
},
wantConstrnt: true,
},
{
name: "existing table with constraint",
current: uniqueConstraintTestModel,
wantConstrnt: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var buf bytes.Buffer
writer, err := NewMigrationWriter(&writers.WriterOptions{})
if err != nil {
t.Fatalf("Failed to create writer: %v", err)
}
writer.writer = &buf
if err := writer.WriteMigration(uniqueConstraintTestModel(), tt.current()); err != nil {
t.Fatalf("WriteMigration failed: %v", err)
}
output := buf.String()
has := strings.Contains(output, "ADD CONSTRAINT") && strings.Contains(output, "ukey_login_guid") && strings.Contains(output, "UNIQUE")
if has != tt.wantConstrnt {
t.Fatalf("unique constraint present = %v, want %v; output:\n%s", has, tt.wantConstrnt, output)
}
if strings.Contains(output, "DROP CONSTRAINT") {
t.Fatalf("unexpected DROP CONSTRAINT; output:\n%s", output)
}
})
}
}
func TestWriteMigration_UniqueConstraintAdoptsSameNamedModelIndex(t *testing.T) {
newIndex := func(name string, unique bool, where string) *models.Index {
idx := models.InitIndex(name, "login", "public")
idx.Columns = []string{"guid"}
idx.Unique = unique
idx.Type = "btree"
idx.Where = where
return idx
}
tests := []struct {
name string
modelIdx *models.Index
liveIdx *models.Index // nil: fresh table
wantAdopt bool
wantDrop bool
}{
{
name: "fresh table: same-named unique index is promoted",
modelIdx: newIndex("ukey_login_guid", true, ""),
wantAdopt: true,
},
{
name: "existing live unique index is promoted",
modelIdx: newIndex("ukey_login_guid", true, ""),
liveIdx: newIndex("ukey_login_guid", true, ""),
wantAdopt: true,
},
{
name: "live unique index not in model is dropped and constraint built fresh",
liveIdx: newIndex("ukey_login_guid", true, ""),
wantDrop: true,
},
{
name: "partial model index is not promoted",
modelIdx: newIndex("ukey_login_guid", true, "guid IS NOT NULL"),
},
{
name: "non-unique same-named index is not promoted",
modelIdx: newIndex("ukey_login_guid", false, ""),
},
{
name: "differently named index is not promoted",
modelIdx: newIndex("idx_login_guid", true, ""),
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
model := uniqueConstraintTestModel()
if tt.modelIdx != nil {
model.Schemas[0].Tables[0].Indexes[tt.modelIdx.Name] = tt.modelIdx
}
current := models.InitDatabase("testdb")
current.Schemas = append(current.Schemas, models.InitSchema("public"))
if tt.liveIdx != nil {
current = uniqueConstraintTestModel()
curTable := current.Schemas[0].Tables[0]
delete(curTable.Constraints, "ukey_login_guid")
curTable.Indexes[tt.liveIdx.Name] = tt.liveIdx
}
var buf bytes.Buffer
writer, err := NewMigrationWriter(&writers.WriterOptions{})
if err != nil {
t.Fatalf("Failed to create writer: %v", err)
}
writer.writer = &buf
if err := writer.WriteMigration(model, current); err != nil {
t.Fatalf("WriteMigration failed: %v", err)
}
output := buf.String()
if got := strings.Contains(output, "UNIQUE USING INDEX"); got != tt.wantAdopt {
t.Errorf("USING INDEX present = %v, want %v; output:\n%s", got, tt.wantAdopt, output)
}
if got := strings.Contains(output, "UNIQUE (guid)") || strings.Contains(output, "UNIQUE (\"guid\")"); got == tt.wantAdopt {
t.Errorf("column-list ADD CONSTRAINT present = %v, want %v; output:\n%s", got, !tt.wantAdopt, output)
}
if got := strings.Contains(output, "DROP INDEX"); got != tt.wantDrop {
t.Errorf("DROP INDEX present = %v, want %v; output:\n%s", got, tt.wantDrop, output)
}
})
}
}
func TestWriteDatabase_UniqueConstraintAdoptsSameNamedIndex(t *testing.T) {
db := uniqueConstraintTestModel()
idx := models.InitIndex("ukey_login_guid", "login", "public")
idx.Columns = []string{"guid"}
idx.Unique = true
db.Schemas[0].Tables[0].Indexes[idx.Name] = idx
var buf bytes.Buffer
w := NewWriter(&writers.WriterOptions{})
w.writer = &buf
if err := w.WriteDatabase(db); err != nil {
t.Fatalf("WriteDatabase failed: %v", err)
}
output := buf.String()
if !strings.Contains(output, "UNIQUE USING INDEX") {
t.Fatalf("expected UNIQUE USING INDEX in output:\n%s", output)
}
if strings.Index(output, "CREATE UNIQUE INDEX") > strings.Index(output, "UNIQUE USING INDEX") {
t.Fatalf("index must be created before the constraint adopts it:\n%s", output)
}
}
func fkDeferralTestModel(def, init *bool) *models.Database {
db := models.InitDatabase("testdb")
schema := models.InitSchema("public")
users := models.InitTable("users", "public")
uid := models.InitColumn("id", "users", "public")
uid.Type = "integer"
uid.NotNull = true
users.Columns["id"] = uid
posts := models.InitTable("posts", "public")
pid := models.InitColumn("id", "posts", "public")
pid.Type = "integer"
pid.NotNull = true
posts.Columns["id"] = pid
puid := models.InitColumn("user_id", "posts", "public")
puid.Type = "integer"
posts.Columns["user_id"] = puid
fk := models.InitConstraint("fk_posts_user_id", models.ForeignKeyConstraint)
fk.Schema = "public"
fk.Table = "posts"
fk.Columns = []string{"user_id"}
fk.ReferencedSchema = "public"
fk.ReferencedTable = "users"
fk.ReferencedColumns = []string{"id"}
fk.Deferrable = def
fk.InitiallyDeferred = init
posts.Constraints[fk.Name] = fk
schema.Tables = append(schema.Tables, users, posts)
db.Schemas = append(db.Schemas, schema)
return db
}
func migrationOutput(t *testing.T, model, current *models.Database) string {
t.Helper()
var buf bytes.Buffer
writer, err := NewMigrationWriter(&writers.WriterOptions{})
if err != nil {
t.Fatalf("Failed to create writer: %v", err)
}
writer.writer = &buf
if err := writer.WriteMigration(model, current); err != nil {
t.Fatalf("WriteMigration failed: %v", err)
}
return buf.String()
}
func TestWriteMigration_ForeignKeyDeferral(t *testing.T) {
yes, no := true, false
empty := func() *models.Database {
c := models.InitDatabase("testdb")
c.Schemas = append(c.Schemas, models.InitSchema("public"))
return c
}
tests := []struct {
name string
def *bool
init *bool
wantFrag string
}{
{"unspecified keeps DEFERRABLE default", nil, nil, "ON UPDATE NO ACTION\n DEFERRABLE;"},
{"deferrable", &yes, nil, "DEFERRABLE;"},
{"initially deferred", &yes, &yes, "DEFERRABLE INITIALLY DEFERRED;"},
{"not deferrable", &no, nil, "NOT DEFERRABLE;"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
out := migrationOutput(t, fkDeferralTestModel(tt.def, tt.init), empty())
if !strings.Contains(out, tt.wantFrag) {
t.Fatalf("expected %q in output:\n%s", tt.wantFrag, out)
}
})
}
}
func TestWriteMigration_ForeignKeyDeferralDiff(t *testing.T) {
yes, no := true, false
live := func(def, init bool) *models.Database {
db := fkDeferralTestModel(&def, &init)
return db
}
tests := []struct {
name string
def, init *bool
liveDef bool
liveInit bool
wantRecreate bool
}{
{"unspecified vs deferrable live: untouched", nil, nil, true, false, false},
{"unspecified vs not deferrable live: untouched", nil, nil, false, false, false},
{"deferrable matches live", &yes, nil, true, true, false},
{"deferrable vs not deferrable live: recreated", &yes, nil, false, false, true},
{"not deferrable vs deferrable live: recreated", &no, nil, true, false, true},
{"initially deferred vs initially immediate live: recreated", &yes, &yes, true, false, true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
out := migrationOutput(t, fkDeferralTestModel(tt.def, tt.init), live(tt.liveDef, tt.liveInit))
got := strings.Contains(out, "ADD CONSTRAINT") && strings.Contains(out, "FOREIGN KEY")
if got != tt.wantRecreate {
t.Fatalf("FK recreated = %v, want %v; output:\n%s", got, tt.wantRecreate, out)
}
})
}
}
func TestWriteDatabase_ForeignKeyDeferral(t *testing.T) {
yes := true
var buf bytes.Buffer
w := NewWriter(&writers.WriterOptions{})
w.writer = &buf
if err := w.WriteDatabase(fkDeferralTestModel(&yes, &yes)); err != nil {
t.Fatalf("WriteDatabase failed: %v", err)
}
if !strings.Contains(buf.String(), "DEFERRABLE INITIALLY DEFERRED;") {
t.Fatalf("expected DEFERRABLE INITIALLY DEFERRED in output:\n%s", buf.String())
}
}
+3 -1
View File
@@ -164,6 +164,7 @@ type CreateForeignKeyData struct {
TargetColumns string
OnDelete string
OnUpdate string
Deferral string // DEFERRABLE / DEFERRABLE INITIALLY DEFERRED / NOT DEFERRABLE, or empty
}
// DropConstraintData contains data for drop constraint template
@@ -238,6 +239,7 @@ type CreateUniqueConstraintData struct {
TableName string
ConstraintName string
Columns string
UsingIndex string // when set, promotes this existing unique index instead of building a new one
}
// CreateCheckConstraintData contains data for create check constraint template
@@ -259,7 +261,7 @@ type CreateForeignKeyWithCheckData struct {
TargetColumns string
OnDelete string
OnUpdate string
Deferrable bool
Deferral string // DEFERRABLE / DEFERRABLE INITIALLY DEFERRED / NOT DEFERRABLE, or empty
}
// SetSequenceValueData contains data for set sequence value template
@@ -7,4 +7,4 @@ ALTER TABLE {{qual_table .SchemaName .TableName}}
REFERENCES {{qual_table .TargetSchema .TargetTable}} ({{.TargetColumns}})
ON DELETE {{.OnDelete}}
ON UPDATE {{.OnUpdate}}
DEFERRABLE;
{{.Deferral}};
@@ -11,8 +11,8 @@ BEGIN
FOREIGN KEY ({{.SourceColumns}})
REFERENCES {{qual_table .TargetSchema .TargetTable}} ({{.TargetColumns}})
ON DELETE {{.OnDelete}}
ON UPDATE {{.OnUpdate}}{{if .Deferrable}}
DEFERRABLE{{end}};
ON UPDATE {{.OnUpdate}}{{if .Deferral}}
{{.Deferral}}{{end}};
END IF;
END;
$$;
@@ -6,7 +6,7 @@ BEGIN
AND table_name = '{{.TableName}}'
AND constraint_name = '{{.ConstraintName}}'
) THEN
ALTER TABLE {{qual_table .SchemaName .TableName}} ADD CONSTRAINT {{quote_ident .ConstraintName}} UNIQUE ({{.Columns}});
ALTER TABLE {{qual_table .SchemaName .TableName}} ADD CONSTRAINT {{quote_ident .ConstraintName}} UNIQUE {{if .UsingIndex}}USING INDEX {{quote_ident .UsingIndex}}{{else}}({{.Columns}}){{end}};
END IF;
END;
$$;
+6 -4
View File
@@ -345,6 +345,7 @@ func (w *Writer) GenerateSchemaStatements(schema *models.Schema) ([]string, erro
TableName: table.Name,
ConstraintName: constraint.Name,
Columns: strings.Join(constraint.Columns, ", "),
UsingIndex: adoptableUniqueIndex(table, constraint),
}
stmt, err := w.executor.ExecuteCreateUniqueConstraint(data)
@@ -411,7 +412,7 @@ func (w *Writer) GenerateSchemaStatements(schema *models.Schema) ([]string, erro
TargetColumns: strings.Join(constraint.ReferencedColumns, ", "),
OnDelete: onDelete,
OnUpdate: onUpdate,
Deferrable: false,
Deferral: constraint.DeferralClause(),
}
stmt, err := w.executor.ExecuteCreateForeignKeyWithCheck(data)
@@ -1150,7 +1151,7 @@ func (w *Writer) writeIndexes(schema *models.Schema) error {
if indexName == "" {
indexType := "idx"
if index.Unique {
indexType = "uidx"
indexType = "ukey"
}
columnSuffix := strings.Join(index.Columns, "_")
indexName = fmt.Sprintf("%s_%s_%s", indexType, table.SQLName(), strings.ToLower(columnSuffix))
@@ -1236,6 +1237,7 @@ func (w *Writer) writeUniqueConstraints(schema *models.Schema) error {
TableName: table.Name,
ConstraintName: constraint.Name,
Columns: strings.Join(columnExprs, ", "),
UsingIndex: adoptableUniqueIndex(table, constraint),
})
if err != nil {
return fmt.Errorf("failed to generate unique constraint: %w", err)
@@ -1380,7 +1382,7 @@ func (w *Writer) writeForeignKeys(schema *models.Schema) error {
TargetColumns: strings.Join(targetColumns, ", "),
OnDelete: onDelete,
OnUpdate: onUpdate,
Deferrable: true,
Deferral: "DEFERRABLE",
}
sql, err := w.executor.ExecuteCreateForeignKeyWithCheck(data)
if err != nil {
@@ -1459,7 +1461,7 @@ func (w *Writer) writeForeignKeys(schema *models.Schema) error {
TargetColumns: strings.Join(targetColumns, ", "),
OnDelete: onDelete,
OnUpdate: onUpdate,
Deferrable: false,
Deferral: constraint.DeferralClause(),
}
sql, err := w.executor.ExecuteCreateForeignKeyWithCheck(data)
if err != nil {
+4 -2
View File
@@ -7,8 +7,10 @@ import "filippo.io/edwards25519"
This library implements the edwards25519 elliptic curve, exposing the necessary APIs to build a wide array of higher-level primitives.
Read the docs at [pkg.go.dev/filippo.io/edwards25519](https://pkg.go.dev/filippo.io/edwards25519).
The code is originally derived from Adam Langley's internal implementation in the Go standard library, and includes George Tankersley's [performance improvements](https://golang.org/cl/71950). It was then further developed by Henry de Valence for use in ristretto255, and was finally [merged back into the Go standard library](https://golang.org/cl/276272) as of Go 1.17. It now tracks the upstream codebase and extends it with additional functionality.
The package tracks the upstream standard library package `crypto/internal/fips140/edwards25519` and extends it with additional functionality.
Most users don't need this package, and should instead use `crypto/ed25519` for signatures, `golang.org/x/crypto/curve25519` for Diffie-Hellman, or `github.com/gtank/ristretto255` for prime order group logic. However, for anyone currently using a fork of `crypto/internal/edwards25519`/`crypto/ed25519/internal/edwards25519` or `github.com/agl/edwards25519`, this package should be a safer, faster, and more powerful alternative.
The code is originally derived from Adam Langley's internal implementation in the Go standard library, and includes George Tankersley's [performance improvements](https://golang.org/cl/71950). It was then further developed by Henry de Valence for use in ristretto255, and was finally [merged back into the Go standard library](https://golang.org/cl/276272) as of Go 1.17.
Most users don't need this package, and should instead use `crypto/ed25519` for signatures, `crypto/ecdh` for Diffie-Hellman, or `github.com/gtank/ristretto255` for prime order group logic. However, for anyone currently using a fork of the internal `edwards25519` package or of `github.com/agl/edwards25519`, this package should be a safer, faster, and more powerful alternative.
Since this package is meant to curb proliferation of edwards25519 implementations in the Go ecosystem, it welcomes requests for new APIs or reviewable performance improvements.
+3 -3
View File
@@ -10,11 +10,11 @@
// the curve used by the Ed25519 signature scheme.
//
// Most users don't need this package, and should instead use crypto/ed25519 for
// signatures, golang.org/x/crypto/curve25519 for Diffie-Hellman, or
// github.com/gtank/ristretto255 for prime order group logic.
// signatures, crypto/ecdh for Diffie-Hellman, or github.com/gtank/ristretto255
// for prime order group logic.
//
// However, developers who do need to interact with low-level edwards25519
// operations can use this package, which is an extended version of
// crypto/internal/edwards25519 from the standard library repackaged as
// crypto/internal/fips140/edwards25519 from the standard library repackaged as
// an importable module.
package edwards25519
+60 -8
View File
@@ -9,6 +9,7 @@ package edwards25519
import (
"errors"
"slices"
"filippo.io/edwards25519/field"
)
@@ -100,13 +101,15 @@ func (v *Point) bytesMontgomery(buf *[32]byte) []byte {
//
// u = (1 + y) / (1 - y)
//
// where y = Y / Z.
// where y = Y / Z and therefore
//
// u = (Z + Y) / (Z - Y)
var y, recip, u field.Element
var n, r, u field.Element
y.Multiply(&v.y, y.Invert(&v.z)) // y = Y / Z
recip.Invert(recip.Subtract(feOne, &y)) // r = 1/(1 - y)
u.Multiply(u.Add(feOne, &y), &recip) // u = (1 + y)*r
n.Add(&v.z, &v.y) // n = Z + Y
r.Invert(r.Subtract(&v.z, &v.y)) // r = 1 / (Z - Y)
u.Multiply(&n, &r) // u = n * r
return copyFieldElement(buf, &u)
}
@@ -124,7 +127,7 @@ func (v *Point) MultByCofactor(p *Point) *Point {
return v.fromP1xP1(&result)
}
// Given k > 0, set s = s**(2*i).
// Given k > 0, set s = s**(2*k).
func (s *Scalar) pow2k(k int) {
for i := 0; i < k; i++ {
s.Multiply(s, s)
@@ -250,12 +253,14 @@ func (v *Point) MultiScalarMult(scalars []*Scalar, points []*Point) *Point {
// between each point in the multiscalar equation.
// Build lookup tables for each point
tables := make([]projLookupTable, len(points))
tables := make([]projLookupTable, 0, 2) // avoid allocation for small sizes
tables = slices.Grow(tables, len(points))[:len(points)]
for i := range tables {
tables[i].FromP3(points[i])
}
// Compute signed radix-16 digits for each scalar
digits := make([][64]int8, len(scalars))
digits := make([][64]int8, 0, 2) // avoid allocation for small sizes
digits = slices.Grow(digits, len(scalars))[:len(scalars)]
for i := range digits {
digits[i] = scalars[i].signedRadix16()
}
@@ -265,6 +270,7 @@ func (v *Point) MultiScalarMult(scalars []*Scalar, points []*Point) *Point {
tmp1 := &projP1xP1{}
tmp2 := &projP2{}
// Lookup-and-add the appropriate multiple of each input point
v.Set(NewIdentityPoint())
for j := range tables {
tables[j].SelectInto(multiple, digits[j][63])
tmp1.Add(v, multiple) // tmp1 = v + x_(j,63)*Q in P1xP1 coords
@@ -347,3 +353,49 @@ func (v *Point) VarTimeMultiScalarMult(scalars []*Scalar, points []*Point) *Poin
v.fromP2(tmp2)
return v
}
// Select sets v to a if cond == 1 and to b if cond == 0.
func (v *Point) Select(a, b *Point, cond int) *Point {
checkInitialized(a, b)
v.x.Select(&a.x, &b.x, cond)
v.y.Select(&a.y, &b.y, cond)
v.z.Select(&a.z, &b.z, cond)
v.t.Select(&a.t, &b.t, cond)
return v
}
// Double sets v = p + p, and returns v.
func (v *Point) Double(p *Point) *Point {
checkInitialized(p)
pp := new(projP2).FromP3(p)
p1 := new(projP1xP1).Double(pp)
return v.fromP1xP1(p1)
}
func (v *Point) addCached(p *Point, qCached *projCached) *Point {
result := new(projP1xP1).Add(p, qCached)
return v.fromP1xP1(result)
}
// ScalarMultSlow sets v = x * q, and returns v. It doesn't precompute a large
// table, so it is considerably slower, but requires less memory.
//
// The scalar multiplication is done in constant time.
func (v *Point) ScalarMultSlow(x *Scalar, q *Point) *Point {
checkInitialized(q)
s := x.Bytes()
qCached := new(projCached).FromP3(q)
v.Set(NewIdentityPoint())
t := new(Point)
for i := 255; i >= 0; i-- {
v.Double(v)
t.addCached(v, qCached)
cond := (s[i/8] >> (i % 8)) & 1
v.Select(t, v, int(cond))
}
return v
}
+17 -17
View File
@@ -90,11 +90,7 @@ func (v *Element) Add(a, b *Element) *Element {
v.l2 = a.l2 + b.l2
v.l3 = a.l3 + b.l3
v.l4 = a.l4 + b.l4
// Using the generic implementation here is actually faster than the
// assembly. Probably because the body of this function is so simple that
// the compiler can figure out better optimizations by inlining the carry
// propagation.
return v.carryPropagateGeneric()
return v.carryPropagate()
}
// Subtract sets v = a - b, and returns v.
@@ -232,18 +228,22 @@ func (v *Element) bytes(out *[32]byte) []byte {
t := *v
t.reduce()
var buf [8]byte
for i, l := range [5]uint64{t.l0, t.l1, t.l2, t.l3, t.l4} {
bitsOffset := i * 51
binary.LittleEndian.PutUint64(buf[:], l<<uint(bitsOffset%8))
for i, bb := range buf {
off := bitsOffset/8 + i
if off >= len(out) {
break
}
out[off] |= bb
}
}
// Pack five 51-bit limbs into four 64-bit words:
//
// 255 204 153 102 51 0
// ├──l4──┼──l3──┼──l2──┼──l1──┼──l0──┤
// ├───u3───┼───u2───┼───u1───┼───u0───┤
// 256 192 128 64 0
u0 := t.l1<<51 | t.l0
u1 := t.l2<<(102-64) | t.l1>>(64-51)
u2 := t.l3<<(153-128) | t.l2>>(128-102)
u3 := t.l4<<(204-192) | t.l3>>(192-153)
binary.LittleEndian.PutUint64(out[0*8:], u0)
binary.LittleEndian.PutUint64(out[1*8:], u1)
binary.LittleEndian.PutUint64(out[2*8:], u2)
binary.LittleEndian.PutUint64(out[3*8:], u3)
return out[:]
}
+1 -2
View File
@@ -1,7 +1,6 @@
// Code generated by command: go run fe_amd64_asm.go -out ../fe_amd64.s -stubs ../fe_amd64.go -pkg field. DO NOT EDIT.
//go:build amd64 && gc && !purego
// +build amd64,gc,!purego
//go:build !purego
package field
+111 -92
View File
@@ -1,7 +1,6 @@
// Code generated by command: go run fe_amd64_asm.go -out ../fe_amd64.s -stubs ../fe_amd64.go -pkg field. DO NOT EDIT.
//go:build amd64 && gc && !purego
// +build amd64,gc,!purego
//go:build !purego
#include "textflag.h"
@@ -17,32 +16,36 @@ TEXT ·feMul(SB), NOSPLIT, $0-24
MOVQ DX, SI
// r0 += 19×a1×b4
MOVQ 8(CX), AX
IMUL3Q $0x13, AX, AX
MULQ 32(BX)
ADDQ AX, DI
ADCQ DX, SI
MOVQ 8(CX), DX
LEAQ (DX)(DX*8), AX
LEAQ (DX)(AX*2), AX
MULQ 32(BX)
ADDQ AX, DI
ADCQ DX, SI
// r0 += 19×a2×b3
MOVQ 16(CX), AX
IMUL3Q $0x13, AX, AX
MULQ 24(BX)
ADDQ AX, DI
ADCQ DX, SI
MOVQ 16(CX), DX
LEAQ (DX)(DX*8), AX
LEAQ (DX)(AX*2), AX
MULQ 24(BX)
ADDQ AX, DI
ADCQ DX, SI
// r0 += 19×a3×b2
MOVQ 24(CX), AX
IMUL3Q $0x13, AX, AX
MULQ 16(BX)
ADDQ AX, DI
ADCQ DX, SI
MOVQ 24(CX), DX
LEAQ (DX)(DX*8), AX
LEAQ (DX)(AX*2), AX
MULQ 16(BX)
ADDQ AX, DI
ADCQ DX, SI
// r0 += 19×a4×b1
MOVQ 32(CX), AX
IMUL3Q $0x13, AX, AX
MULQ 8(BX)
ADDQ AX, DI
ADCQ DX, SI
MOVQ 32(CX), DX
LEAQ (DX)(DX*8), AX
LEAQ (DX)(AX*2), AX
MULQ 8(BX)
ADDQ AX, DI
ADCQ DX, SI
// r1 = a0×b1
MOVQ (CX), AX
@@ -57,25 +60,28 @@ TEXT ·feMul(SB), NOSPLIT, $0-24
ADCQ DX, R8
// r1 += 19×a2×b4
MOVQ 16(CX), AX
IMUL3Q $0x13, AX, AX
MULQ 32(BX)
ADDQ AX, R9
ADCQ DX, R8
MOVQ 16(CX), DX
LEAQ (DX)(DX*8), AX
LEAQ (DX)(AX*2), AX
MULQ 32(BX)
ADDQ AX, R9
ADCQ DX, R8
// r1 += 19×a3×b3
MOVQ 24(CX), AX
IMUL3Q $0x13, AX, AX
MULQ 24(BX)
ADDQ AX, R9
ADCQ DX, R8
MOVQ 24(CX), DX
LEAQ (DX)(DX*8), AX
LEAQ (DX)(AX*2), AX
MULQ 24(BX)
ADDQ AX, R9
ADCQ DX, R8
// r1 += 19×a4×b2
MOVQ 32(CX), AX
IMUL3Q $0x13, AX, AX
MULQ 16(BX)
ADDQ AX, R9
ADCQ DX, R8
MOVQ 32(CX), DX
LEAQ (DX)(DX*8), AX
LEAQ (DX)(AX*2), AX
MULQ 16(BX)
ADDQ AX, R9
ADCQ DX, R8
// r2 = a0×b2
MOVQ (CX), AX
@@ -96,18 +102,20 @@ TEXT ·feMul(SB), NOSPLIT, $0-24
ADCQ DX, R10
// r2 += 19×a3×b4
MOVQ 24(CX), AX
IMUL3Q $0x13, AX, AX
MULQ 32(BX)
ADDQ AX, R11
ADCQ DX, R10
MOVQ 24(CX), DX
LEAQ (DX)(DX*8), AX
LEAQ (DX)(AX*2), AX
MULQ 32(BX)
ADDQ AX, R11
ADCQ DX, R10
// r2 += 19×a4×b3
MOVQ 32(CX), AX
IMUL3Q $0x13, AX, AX
MULQ 24(BX)
ADDQ AX, R11
ADCQ DX, R10
MOVQ 32(CX), DX
LEAQ (DX)(DX*8), AX
LEAQ (DX)(AX*2), AX
MULQ 24(BX)
ADDQ AX, R11
ADCQ DX, R10
// r3 = a0×b3
MOVQ (CX), AX
@@ -134,11 +142,12 @@ TEXT ·feMul(SB), NOSPLIT, $0-24
ADCQ DX, R12
// r3 += 19×a4×b4
MOVQ 32(CX), AX
IMUL3Q $0x13, AX, AX
MULQ 32(BX)
ADDQ AX, R13
ADCQ DX, R12
MOVQ 32(CX), DX
LEAQ (DX)(DX*8), AX
LEAQ (DX)(AX*2), AX
MULQ 32(BX)
ADDQ AX, R13
ADCQ DX, R12
// r4 = a0×b4
MOVQ (CX), AX
@@ -232,18 +241,22 @@ TEXT ·feSquare(SB), NOSPLIT, $0-16
MOVQ DX, BX
// r0 += 38×l1×l4
MOVQ 8(CX), AX
IMUL3Q $0x26, AX, AX
MULQ 32(CX)
ADDQ AX, SI
ADCQ DX, BX
MOVQ 8(CX), DX
LEAQ (DX)(DX*8), AX
LEAQ (DX)(AX*2), AX
SHLQ $0x01, AX
MULQ 32(CX)
ADDQ AX, SI
ADCQ DX, BX
// r0 += 38×l2×l3
MOVQ 16(CX), AX
IMUL3Q $0x26, AX, AX
MULQ 24(CX)
ADDQ AX, SI
ADCQ DX, BX
MOVQ 16(CX), DX
LEAQ (DX)(DX*8), AX
LEAQ (DX)(AX*2), AX
SHLQ $0x01, AX
MULQ 24(CX)
ADDQ AX, SI
ADCQ DX, BX
// r1 = 2×l0×l1
MOVQ (CX), AX
@@ -253,18 +266,21 @@ TEXT ·feSquare(SB), NOSPLIT, $0-16
MOVQ DX, DI
// r1 += 38×l2×l4
MOVQ 16(CX), AX
IMUL3Q $0x26, AX, AX
MULQ 32(CX)
ADDQ AX, R8
ADCQ DX, DI
MOVQ 16(CX), DX
LEAQ (DX)(DX*8), AX
LEAQ (DX)(AX*2), AX
SHLQ $0x01, AX
MULQ 32(CX)
ADDQ AX, R8
ADCQ DX, DI
// r1 += 19×l3×l3
MOVQ 24(CX), AX
IMUL3Q $0x13, AX, AX
MULQ 24(CX)
ADDQ AX, R8
ADCQ DX, DI
MOVQ 24(CX), DX
LEAQ (DX)(DX*8), AX
LEAQ (DX)(AX*2), AX
MULQ 24(CX)
ADDQ AX, R8
ADCQ DX, DI
// r2 = 2×l0×l2
MOVQ (CX), AX
@@ -280,11 +296,13 @@ TEXT ·feSquare(SB), NOSPLIT, $0-16
ADCQ DX, R9
// r2 += 38×l3×l4
MOVQ 24(CX), AX
IMUL3Q $0x26, AX, AX
MULQ 32(CX)
ADDQ AX, R10
ADCQ DX, R9
MOVQ 24(CX), DX
LEAQ (DX)(DX*8), AX
LEAQ (DX)(AX*2), AX
SHLQ $0x01, AX
MULQ 32(CX)
ADDQ AX, R10
ADCQ DX, R9
// r3 = 2×l0×l3
MOVQ (CX), AX
@@ -294,18 +312,19 @@ TEXT ·feSquare(SB), NOSPLIT, $0-16
MOVQ DX, R11
// r3 += 2×l1×l2
MOVQ 8(CX), AX
IMUL3Q $0x02, AX, AX
MULQ 16(CX)
ADDQ AX, R12
ADCQ DX, R11
MOVQ 8(CX), AX
SHLQ $0x01, AX
MULQ 16(CX)
ADDQ AX, R12
ADCQ DX, R11
// r3 += 19×l4×l4
MOVQ 32(CX), AX
IMUL3Q $0x13, AX, AX
MULQ 32(CX)
ADDQ AX, R12
ADCQ DX, R11
MOVQ 32(CX), DX
LEAQ (DX)(DX*8), AX
LEAQ (DX)(AX*2), AX
MULQ 32(CX)
ADDQ AX, R12
ADCQ DX, R11
// r4 = 2×l0×l4
MOVQ (CX), AX
@@ -315,11 +334,11 @@ TEXT ·feSquare(SB), NOSPLIT, $0-16
MOVQ DX, R13
// r4 += 2×l1×l3
MOVQ 8(CX), AX
IMUL3Q $0x02, AX, AX
MULQ 24(CX)
ADDQ AX, R14
ADCQ DX, R13
MOVQ 8(CX), AX
SHLQ $0x01, AX
MULQ 24(CX)
ADDQ AX, R14
ADCQ DX, R13
// r4 += l2×l2
MOVQ 16(CX), AX
+1 -2
View File
@@ -2,8 +2,7 @@
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//go:build !amd64 || !gc || purego
// +build !amd64 !gc purego
//go:build !amd64 || purego
package field
-16
View File
@@ -1,16 +0,0 @@
// Copyright (c) 2020 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//go:build arm64 && gc && !purego
// +build arm64,gc,!purego
package field
//go:noescape
func carryPropagate(v *Element)
func (v *Element) carryPropagate() *Element {
carryPropagate(v)
return v
}
-42
View File
@@ -1,42 +0,0 @@
// Copyright (c) 2020 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//go:build arm64 && gc && !purego
#include "textflag.h"
// carryPropagate works exactly like carryPropagateGeneric and uses the
// same AND, ADD, and LSR+MADD instructions emitted by the compiler, but
// avoids loading R0-R4 twice and uses LDP and STP.
//
// See https://golang.org/issues/43145 for the main compiler issue.
//
// func carryPropagate(v *Element)
TEXT ·carryPropagate(SB),NOFRAME|NOSPLIT,$0-8
MOVD v+0(FP), R20
LDP 0(R20), (R0, R1)
LDP 16(R20), (R2, R3)
MOVD 32(R20), R4
AND $0x7ffffffffffff, R0, R10
AND $0x7ffffffffffff, R1, R11
AND $0x7ffffffffffff, R2, R12
AND $0x7ffffffffffff, R3, R13
AND $0x7ffffffffffff, R4, R14
ADD R0>>51, R11, R11
ADD R1>>51, R12, R12
ADD R2>>51, R13, R13
ADD R3>>51, R14, R14
// R4>>51 * 19 + R10 -> R10
LSR $51, R4, R21
MOVD $19, R22
MADD R22, R10, R21, R10
STP (R10, R11), 0(R20)
STP (R12, R13), 16(R20)
MOVD R14, 32(R20)
RET
-12
View File
@@ -1,12 +0,0 @@
// Copyright (c) 2021 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//go:build !arm64 || !gc || purego
// +build !arm64 !gc purego
package field
func (v *Element) carryPropagate() *Element {
return v.carryPropagateGeneric()
}
+88 -82
View File
@@ -12,20 +12,42 @@ type uint128 struct {
lo, hi uint64
}
// mul64 returns a * b.
func mul64(a, b uint64) uint128 {
// mul returns a * b.
func mul(a, b uint64) uint128 {
hi, lo := bits.Mul64(a, b)
return uint128{lo, hi}
}
// addMul64 returns v + a * b.
func addMul64(v uint128, a, b uint64) uint128 {
// addMul returns v + a * b.
func addMul(v uint128, a, b uint64) uint128 {
hi, lo := bits.Mul64(a, b)
lo, c := bits.Add64(lo, v.lo, 0)
hi, _ = bits.Add64(hi, v.hi, c)
return uint128{lo, hi}
}
// mul19 returns v * 19.
func mul19(v uint64) uint64 {
// Using this approach seems to yield better optimizations than *19.
return v + (v+v<<3)<<1
}
// addMul19 returns v + 19 * a * b, where a and b are at most 52 bits.
func addMul19(v uint128, a, b uint64) uint128 {
hi, lo := bits.Mul64(mul19(a), b)
lo, c := bits.Add64(lo, v.lo, 0)
hi, _ = bits.Add64(hi, v.hi, c)
return uint128{lo, hi}
}
// addMul38 returns v + 38 * a * b, where a and b are at most 52 bits.
func addMul38(v uint128, a, b uint64) uint128 {
hi, lo := bits.Mul64(mul19(a), b*2)
lo, c := bits.Add64(lo, v.lo, 0)
hi, _ = bits.Add64(hi, v.hi, c)
return uint128{lo, hi}
}
// shiftRightBy51 returns a >> 51. a is assumed to be at most 115 bits.
func shiftRightBy51(a uint128) uint64 {
return (a.hi << (64 - 51)) | (a.lo >> 51)
@@ -76,45 +98,40 @@ func feMulGeneric(v, a, b *Element) {
//
// Finally we add up the columns into wide, overlapping limbs.
a1_19 := a1 * 19
a2_19 := a2 * 19
a3_19 := a3 * 19
a4_19 := a4 * 19
// r0 = a0×b0 + 19×(a1×b4 + a2×b3 + a3×b2 + a4×b1)
r0 := mul64(a0, b0)
r0 = addMul64(r0, a1_19, b4)
r0 = addMul64(r0, a2_19, b3)
r0 = addMul64(r0, a3_19, b2)
r0 = addMul64(r0, a4_19, b1)
r0 := mul(a0, b0)
r0 = addMul19(r0, a1, b4)
r0 = addMul19(r0, a2, b3)
r0 = addMul19(r0, a3, b2)
r0 = addMul19(r0, a4, b1)
// r1 = a0×b1 + a1×b0 + 19×(a2×b4 + a3×b3 + a4×b2)
r1 := mul64(a0, b1)
r1 = addMul64(r1, a1, b0)
r1 = addMul64(r1, a2_19, b4)
r1 = addMul64(r1, a3_19, b3)
r1 = addMul64(r1, a4_19, b2)
r1 := mul(a0, b1)
r1 = addMul(r1, a1, b0)
r1 = addMul19(r1, a2, b4)
r1 = addMul19(r1, a3, b3)
r1 = addMul19(r1, a4, b2)
// r2 = a0×b2 + a1×b1 + a2×b0 + 19×(a3×b4 + a4×b3)
r2 := mul64(a0, b2)
r2 = addMul64(r2, a1, b1)
r2 = addMul64(r2, a2, b0)
r2 = addMul64(r2, a3_19, b4)
r2 = addMul64(r2, a4_19, b3)
r2 := mul(a0, b2)
r2 = addMul(r2, a1, b1)
r2 = addMul(r2, a2, b0)
r2 = addMul19(r2, a3, b4)
r2 = addMul19(r2, a4, b3)
// r3 = a0×b3 + a1×b2 + a2×b1 + a3×b0 + 19×a4×b4
r3 := mul64(a0, b3)
r3 = addMul64(r3, a1, b2)
r3 = addMul64(r3, a2, b1)
r3 = addMul64(r3, a3, b0)
r3 = addMul64(r3, a4_19, b4)
r3 := mul(a0, b3)
r3 = addMul(r3, a1, b2)
r3 = addMul(r3, a2, b1)
r3 = addMul(r3, a3, b0)
r3 = addMul19(r3, a4, b4)
// r4 = a0×b4 + a1×b3 + a2×b2 + a3×b1 + a4×b0
r4 := mul64(a0, b4)
r4 = addMul64(r4, a1, b3)
r4 = addMul64(r4, a2, b2)
r4 = addMul64(r4, a3, b1)
r4 = addMul64(r4, a4, b0)
r4 := mul(a0, b4)
r4 = addMul(r4, a1, b3)
r4 = addMul(r4, a2, b2)
r4 = addMul(r4, a3, b1)
r4 = addMul(r4, a4, b0)
// After the multiplication, we need to reduce (carry) the five coefficients
// to obtain a result with limbs that are at most slightly larger than 2⁵¹,
@@ -149,7 +166,7 @@ func feMulGeneric(v, a, b *Element) {
c3 := shiftRightBy51(r3)
c4 := shiftRightBy51(r4)
rr0 := r0.lo&maskLow51Bits + c4*19
rr0 := r0.lo&maskLow51Bits + mul19(c4)
rr1 := r1.lo&maskLow51Bits + c0
rr2 := r2.lo&maskLow51Bits + c1
rr3 := r3.lo&maskLow51Bits + c2
@@ -158,8 +175,12 @@ func feMulGeneric(v, a, b *Element) {
// Now all coefficients fit into 64-bit registers but are still too large to
// be passed around as an Element. We therefore do one last carry chain,
// where the carries will be small enough to fit in the wiggle room above 2⁵¹.
*v = Element{rr0, rr1, rr2, rr3, rr4}
v.carryPropagate()
v.l0 = rr0&maskLow51Bits + mul19(rr4>>51)
v.l1 = rr1&maskLow51Bits + rr0>>51
v.l2 = rr2&maskLow51Bits + rr1>>51
v.l3 = rr3&maskLow51Bits + rr2>>51
v.l4 = rr4&maskLow51Bits + rr3>>51
}
func feSquareGeneric(v, a *Element) {
@@ -190,44 +211,31 @@ func feSquareGeneric(v, a *Element) {
// l0l4 19×l4l4 19×l3l4 19×l2l4 19×l1l4 =
// --------------------------------------
// r4 r3 r2 r1 r0
//
// With precomputed 2×, 19×, and 2×19× terms, we can compute each limb with
// only three Mul64 and four Add64, instead of five and eight.
l0_2 := l0 * 2
l1_2 := l1 * 2
l1_38 := l1 * 38
l2_38 := l2 * 38
l3_38 := l3 * 38
l3_19 := l3 * 19
l4_19 := l4 * 19
// r0 = l0×l0 + 19×(l1×l4 + l2×l3 + l3×l2 + l4×l1) = l0×l0 + 19×2×(l1×l4 + l2×l3)
r0 := mul64(l0, l0)
r0 = addMul64(r0, l1_38, l4)
r0 = addMul64(r0, l2_38, l3)
r0 := mul(l0, l0)
r0 = addMul38(r0, l1, l4)
r0 = addMul38(r0, l2, l3)
// r1 = l0×l1 + l1×l0 + 19×(l2×l4 + l3×l3 + l4×l2) = 2×l0×l1 + 19×2×l2×l4 + 19×l3×l3
r1 := mul64(l0_2, l1)
r1 = addMul64(r1, l2_38, l4)
r1 = addMul64(r1, l3_19, l3)
r1 := mul(l0*2, l1)
r1 = addMul38(r1, l2, l4)
r1 = addMul19(r1, l3, l3)
// r2 = l0×l2 + l1×l1 + l2×l0 + 19×(l3×l4 + l4×l3) = 2×l0×l2 + l1×l1 + 19×2×l3×l4
r2 := mul64(l0_2, l2)
r2 = addMul64(r2, l1, l1)
r2 = addMul64(r2, l3_38, l4)
r2 := mul(l0*2, l2)
r2 = addMul(r2, l1, l1)
r2 = addMul38(r2, l3, l4)
// r3 = l0×l3 + l1×l2 + l2×l1 + l3×l0 + 19×l4×l4 = 2×l0×l3 + 2×l1×l2 + 19×l4×l4
r3 := mul64(l0_2, l3)
r3 = addMul64(r3, l1_2, l2)
r3 = addMul64(r3, l4_19, l4)
r3 := mul(l0*2, l3)
r3 = addMul(r3, l1*2, l2)
r3 = addMul19(r3, l4, l4)
// r4 = l0×l4 + l1×l3 + l2×l2 + l3×l1 + l4×l0 = 2×l0×l4 + 2×l1×l3 + l2×l2
r4 := mul64(l0_2, l4)
r4 = addMul64(r4, l1_2, l3)
r4 = addMul64(r4, l2, l2)
r4 := mul(l0*2, l4)
r4 = addMul(r4, l1*2, l3)
r4 = addMul(r4, l2, l2)
c0 := shiftRightBy51(r0)
c1 := shiftRightBy51(r1)
@@ -235,32 +243,30 @@ func feSquareGeneric(v, a *Element) {
c3 := shiftRightBy51(r3)
c4 := shiftRightBy51(r4)
rr0 := r0.lo&maskLow51Bits + c4*19
rr0 := r0.lo&maskLow51Bits + mul19(c4)
rr1 := r1.lo&maskLow51Bits + c0
rr2 := r2.lo&maskLow51Bits + c1
rr3 := r3.lo&maskLow51Bits + c2
rr4 := r4.lo&maskLow51Bits + c3
*v = Element{rr0, rr1, rr2, rr3, rr4}
v.carryPropagate()
v.l0 = rr0&maskLow51Bits + mul19(rr4>>51)
v.l1 = rr1&maskLow51Bits + rr0>>51
v.l2 = rr2&maskLow51Bits + rr1>>51
v.l3 = rr3&maskLow51Bits + rr2>>51
v.l4 = rr4&maskLow51Bits + rr3>>51
}
// carryPropagateGeneric brings the limbs below 52 bits by applying the reduction
// carryPropagate brings the limbs below 52 bits by applying the reduction
// identity (a * 2²⁵⁵ + b = a * 19 + b) to the l4 carry.
func (v *Element) carryPropagateGeneric() *Element {
c0 := v.l0 >> 51
c1 := v.l1 >> 51
c2 := v.l2 >> 51
c3 := v.l3 >> 51
c4 := v.l4 >> 51
// c4 is at most 64 - 51 = 13 bits, so c4*19 is at most 18 bits, and
func (v *Element) carryPropagate() *Element {
// (l4>>51) is at most 64 - 51 = 13 bits, so (l4>>51)*19 is at most 18 bits, and
// the final l0 will be at most 52 bits. Similarly for the rest.
v.l0 = v.l0&maskLow51Bits + c4*19
v.l1 = v.l1&maskLow51Bits + c0
v.l2 = v.l2&maskLow51Bits + c1
v.l3 = v.l3&maskLow51Bits + c2
v.l4 = v.l4&maskLow51Bits + c3
l0 := v.l0
v.l0 = v.l0&maskLow51Bits + mul19(v.l4>>51)
v.l4 = v.l4&maskLow51Bits + v.l3>>51
v.l3 = v.l3&maskLow51Bits + v.l2>>51
v.l2 = v.l2&maskLow51Bits + v.l1>>51
v.l1 = v.l1&maskLow51Bits + l0>>51
return v
}
+53
View File
@@ -0,0 +1,53 @@
#!/usr/bin/env bash
set -euo pipefail
if [ "$#" -ne 1 ]; then
echo "Usage: $0 <tag>"
exit 1
fi
TAG="$1"
TMPDIR="$(mktemp -d)"
cleanup() {
rm -rf "$TMPDIR"
}
trap cleanup EXIT
command -v git >/dev/null
command -v git-filter-repo >/dev/null
if [ -d "$HOME/go/.git" ]; then
REFERENCE=(--reference "$HOME/go" --dissociate)
else
REFERENCE=()
fi
git -c advice.detachedHead=false clone --no-checkout "${REFERENCE[@]}" \
-b "$TAG" https://go.googlesource.com/go.git "$TMPDIR"
# Simplify the history graph by removing the dev.boringcrypto branches, whose
# merges end up empty after grafting anyway. This also fixes a weird quirk
# (maybe a git-filter-repo bug?) where only one file from an old path,
# src/crypto/ed25519/internal/edwards25519/const.go, would still exist in the
# filtered repo.
git -C "$TMPDIR" replace --graft f771edd7f9 99f1bf54eb
git -C "$TMPDIR" replace --graft 109c13b64f c2f96e686f
git -C "$TMPDIR" replace --graft aa4da4f189 912f075047
git -C "$TMPDIR" filter-repo --force \
--paths-from-file /dev/stdin \
--prune-empty always \
--prune-degenerate always \
--tag-callback 'tag.skip()' <<'EOF'
src/crypto/internal/fips140/edwards25519
src/crypto/internal/edwards25519
src/crypto/ed25519/internal/edwards25519
EOF
git fetch "$TMPDIR"
git update-ref "refs/heads/upstream/$TAG" FETCH_HEAD
echo
echo "Fetched upstream history up to $TAG. Merge with:"
echo -e "\tgit merge --no-ff --no-commit --allow-unrelated-histories upstream/$TAG"
+18 -9
View File
@@ -7,6 +7,7 @@ package edwards25519
import (
"encoding/binary"
"errors"
"math/bits"
)
// A Scalar is an integer modulo
@@ -179,15 +180,23 @@ func isReduced(s []byte) bool {
return false
}
for i := len(s) - 1; i >= 0; i-- {
switch {
case s[i] > scalarMinusOneBytes[i]:
return false
case s[i] < scalarMinusOneBytes[i]:
return true
}
}
return true
s0 := binary.LittleEndian.Uint64(s[:8])
s1 := binary.LittleEndian.Uint64(s[8:16])
s2 := binary.LittleEndian.Uint64(s[16:24])
s3 := binary.LittleEndian.Uint64(s[24:])
l0 := binary.LittleEndian.Uint64(scalarMinusOneBytes[:8])
l1 := binary.LittleEndian.Uint64(scalarMinusOneBytes[8:16])
l2 := binary.LittleEndian.Uint64(scalarMinusOneBytes[16:24])
l3 := binary.LittleEndian.Uint64(scalarMinusOneBytes[24:])
// Do a constant time subtraction chain scalarMinusOneBytes - s. If there is
// a borrow at the end, then s > scalarMinusOneBytes.
_, b := bits.Sub64(l0, s0, 0)
_, b = bits.Sub64(l1, s1, b)
_, b = bits.Sub64(l2, s2, b)
_, b = bits.Sub64(l3, s3, b)
return b == 0
}
// SetBytesWithClamping applies the buffer pruning described in RFC 8032,
+1 -3
View File
@@ -4,9 +4,7 @@
package edwards25519
import (
"crypto/subtle"
)
import "crypto/subtle"
// A dynamic lookup table for variable-base, constant-time scalar muls.
type projLookupTable struct {
+4
View File
@@ -0,0 +1,4 @@
# shellcheck shell=bash
set -eu
use flake
+1
View File
@@ -0,0 +1 @@
.direnv
-21
View File
@@ -1,21 +0,0 @@
sudo: false
language: go
go_import_path: github.com/dustin/go-humanize
go:
- 1.13.x
- 1.14.x
- 1.15.x
- 1.16.x
- stable
- master
matrix:
allow_failures:
- go: master
fast_finish: true
install:
- # Do nothing. This is needed to prevent default install action "go get -t -v ./..." from happening here (we want it to happen inside script step).
script:
- diff -u <(echo -n) <(gofmt -d -s .)
- go vet .
- go install -v -race ./...
- go test -v -race ./...
+4 -1
View File
@@ -1,4 +1,4 @@
# Humane Units [![Build Status](https://travis-ci.org/dustin/go-humanize.svg?branch=master)](https://travis-ci.org/dustin/go-humanize) [![GoDoc](https://godoc.org/github.com/dustin/go-humanize?status.svg)](https://godoc.org/github.com/dustin/go-humanize)
# Humane Units [![Build Status](https://travis-ci.org/dustin/go-humanize.svg?branch=master)](https://travis-ci.org/dustin/go-humanize) [![GoDoc](https://godoc.org/github.com/dustin/go-humanize?status.svg)](https://godoc.org/github.com/dustin/go-humanize) [![Coverage](https://img.shields.io/badge/coverage-report-brightgreen)](https://dustin.github.io/go-humanize/)
Just a few functions for helping humanize times and sizes.
@@ -8,6 +8,9 @@ Just a few functions for helping humanize times and sizes.
See [godoc](https://pkg.go.dev/github.com/dustin/go-humanize) for
complete documentation.
See the [test coverage report](https://dustin.github.io/go-humanize/) for
coverage details.
## Sizes
This lets you take numbers like `82854982` and convert them to useful
+5
View File
@@ -0,0 +1,5 @@
# Security Policy
To report a security issue, please disclose it at [security advisory](https://github.com/dustin/go-humanize/security/advisories/new).
We will respond within 7 working days of your submission. If the issue is confirmed as a vulnerability, we will open a Security Advisory and acknowledge your contributions as part of it. This project follows a 90 day disclosure timeline.
+73 -9
View File
@@ -3,6 +3,7 @@ package humanize
import (
"fmt"
"math"
"math/bits"
"strconv"
"strings"
"unicode"
@@ -65,39 +66,90 @@ func logn(n, b float64) float64 {
return math.Log(n) / math.Log(b)
}
func humanateBytes(s uint64, base float64, sizes []string) string {
func countDigits(n int64) int {
digits := 0
for n != 0 {
n /= 10
digits += 1
}
return digits
}
func humanateBytes(s uint64, base float64, minDigits int, sizes []string) string {
if s < 10 {
return fmt.Sprintf("%d B", s)
}
e := math.Floor(logn(float64(s), base))
suffix := sizes[int(e)]
val := math.Floor(float64(s)/math.Pow(base, e)*10+0.5) / 10
f := "%.0f %s"
if val < 10 {
f = "%.1f %s"
val := float64(s) / math.Pow(base, e)
// Determine how many decimal places to keep based on the unrounded
// value's integer-digit count, then round exactly once to that
// precision. Rounding first to a fixed precision and then again when
// formatting (double rounding) can flip the result at a boundary due
// to floating point error (see #103).
intDigits := countDigits(int64(val))
digits := max(minDigits-intDigits, 0)
rounding := math.Pow10(digits)
val = math.Floor(val*rounding+0.5) / rounding
// Rounding may have carried into an additional integer digit (e.g.
// 9.996 -> 10.0), which means fewer decimal places should be shown
// than originally computed.
if newIntDigits := countDigits(int64(val)); newIntDigits > intDigits {
digits = max(digits-(newIntDigits-intDigits), 0)
}
ff := "%%.%df %%s"
f := fmt.Sprintf(ff, digits)
return fmt.Sprintf(f, val, suffix)
}
// Bytes produces a human readable representation of an SI size.
// Bytes produces a human-readable representation of an SI size.
//
// See also: ParseBytes.
//
// Bytes(82854982) -> 83 MB
func Bytes(s uint64) string {
sizes := []string{"B", "kB", "MB", "GB", "TB", "PB", "EB"}
return humanateBytes(s, 1000, sizes)
return humanateBytes(s, 1000, 2, sizes)
}
// IBytes produces a human readable representation of an IEC size.
// BytesN produces a human-readable representation of an SI size.
// n specifies the total number of digits to output, including the decimal part.
// If n is less than or equal to the number of digits in the integer part, the decimal part will be omitted.
//
// See also: ParseBytes.
//
// BytesN(82854982, 3) -> 82.9 MB
// BytesN(82854982, 4) -> 82.85 MB
func BytesN(s uint64, n int) string {
sizes := []string{"B", "kB", "MB", "GB", "TB", "PB", "EB"}
return humanateBytes(s, 1000, n, sizes)
}
// IBytes produces a human-readable representation of an IEC size.
//
// See also: ParseBytes.
//
// IBytes(82854982) -> 79 MiB
func IBytes(s uint64) string {
sizes := []string{"B", "KiB", "MiB", "GiB", "TiB", "PiB", "EiB"}
return humanateBytes(s, 1024, sizes)
return humanateBytes(s, 1024, 2, sizes)
}
// IBytesN produces a human-readable representation of an IEC size.
// n specifies the total number of digits to output, including the decimal part.
// If n is less than or equal to the number of digits in the integer part, the decimal part will be omitted.
//
// See also: ParseBytes.
//
// IBytesN(82854982, 4) -> 79.02 MiB
// IBytesN(123456789, 3) -> 118 MiB
// IBytesN(123456789, 6) -> 117.738 MiB
func IBytesN(s uint64, n int) string {
sizes := []string{"B", "KiB", "MiB", "GiB", "TiB", "PiB", "EiB"}
return humanateBytes(s, 1024, n, sizes)
}
// ParseBytes parses a string representation of bytes into the number
@@ -132,6 +184,18 @@ func ParseBytes(s string) (uint64, error) {
extra := strings.ToLower(strings.TrimSpace(s[lastDigit:]))
if m, ok := bytesSizeTable[extra]; ok {
// Parse whole-number inputs exactly. A float64 cannot represent
// integers above 2^53, and the multiplication below would also reject
// otherwise-valid values near math.MaxUint64, so route integers through
// uint64 arithmetic with an overflow check.
if !strings.ContainsRune(num, '.') {
if v, perr := strconv.ParseUint(num, 10, 64); perr == nil {
if hi, lo := bits.Mul64(v, m); hi == 0 {
return lo, nil
}
return 0, fmt.Errorf("too large: %v", s)
}
}
f *= float64(m)
if f >= math.MaxUint64 {
return 0, fmt.Errorf("too large: %v", s)
+73 -20
View File
@@ -13,34 +13,48 @@ import (
//
// e.g. Comma(834142) -> 834,142
func Comma(v int64) string {
sign := ""
// Shortcut for [0, 7]
if v&^0b111 == 0 {
return string([]byte{byte(v) + 48})
}
// Min int64 can't be negated to a usable value, so it has to be special cased.
if v == math.MinInt64 {
return "-9,223,372,036,854,775,808"
}
// Counting the number of digits.
var count byte = 0
for n := v; n != 0; n = n / 10 {
count++
}
count += (count - 1) / 3
if v < 0 {
sign = "-"
v = 0 - v
count++
}
output := make([]byte, count)
j := len(output) - 1
parts := []string{"", "", "", "", "", "", ""}
j := len(parts) - 1
for v > 999 {
parts[j] = strconv.FormatInt(v%1000, 10)
switch len(parts[j]) {
case 2:
parts[j] = "0" + parts[j]
case 1:
parts[j] = "00" + parts[j]
}
v = v / 1000
var counter byte = 0
for v > 9 {
output[j] = byte(v%10) + 48
v = v / 10
j--
if counter == 2 {
counter = 0
output[j] = ','
j--
} else {
counter++
}
}
parts[j] = strconv.Itoa(int(v))
return sign + strings.Join(parts[j:], ",")
output[j] = byte(v) + 48
if j == 1 {
output[0] = '-'
}
return string(output)
}
// Commaf produces a string form of the given number in base 10 with
@@ -48,6 +62,16 @@ func Comma(v int64) string {
//
// e.g. Commaf(834142.32) -> 834,142.32
func Commaf(v float64) string {
if math.IsNaN(v) {
return "NaN"
}
if math.IsInf(v, 1) {
return "+Inf"
}
if math.IsInf(v, -1) {
return "-Inf"
}
buf := &bytes.Buffer{}
if v < 0 {
buf.Write([]byte{'-'})
@@ -76,17 +100,30 @@ func Commaf(v float64) string {
return buf.String()
}
// CommafWithDigits works like the Commaf but limits the resulting
// string to the given number of decimal places.
// CommafWithDigits works like the Commaf but always formats the
// resulting string to exactly the given number of decimal places,
// truncating extra digits or padding with zeros as needed.
//
// e.g. CommafWithDigits(834142.32, 1) -> 834,142.3
// e.g. CommafWithDigits(1000, 2) -> 1,000.00
func CommafWithDigits(f float64, decimals int) string {
return stripTrailingDigits(Commaf(f), decimals)
s := stripTrailingDigits(Commaf(f), decimals)
if decimals <= 0 {
return s
}
if dot := strings.IndexByte(s, '.'); dot < 0 {
s += "." + strings.Repeat("0", decimals)
} else if have := len(s) - dot - 1; have < decimals {
s += strings.Repeat("0", decimals-have)
}
return s
}
// BigComma produces a string form of the given big.Int in base 10
// with commas after every three orders of magnitude.
func BigComma(b *big.Int) string {
func BigComma(bin *big.Int) string {
b := new(big.Int).Set(bin)
sign := ""
if b.Sign() < 0 {
sign = "-"
@@ -114,3 +151,19 @@ func BigComma(b *big.Int) string {
parts[j] = strconv.Itoa(int(b.Int64()))
return sign + strings.Join(parts[j:], ",")
}
// ParseComma parses a string produced by Comma.
// It strips commas and returns the int64 value, or an error if the
// string cannot be parsed.
func ParseComma(s string) (int64, error) {
s = strings.ReplaceAll(s, ",", "")
return strconv.ParseInt(s, 10, 64)
}
// ParseCommaf parses a string produced by Commaf.
// It strips commas and returns the float64 value, or an error if the
// string cannot be parsed.
func ParseCommaf(s string) (float64, error) {
s = strings.ReplaceAll(s, ",", "")
return strconv.ParseFloat(s, 64)
}
+7
View File
@@ -12,6 +12,13 @@ import (
// BigCommaf produces a string form of the given big.Float in base 10
// with commas after every three orders of magnitude.
func BigCommaf(v *big.Float) string {
if v.IsInf() {
if v.Sign() < 0 {
return "-Inf"
}
return "+Inf"
}
buf := &bytes.Buffer{}
if v.Sign() < 0 {
buf.Write([]byte{'-'})
+58
View File
@@ -0,0 +1,58 @@
{
"nodes": {
"flake-utils": {
"inputs": {
"systems": "systems"
},
"locked": {
"lastModified": 1731533236,
"narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=",
"owner": "numtide",
"repo": "flake-utils",
"rev": "11707dc2f618dd54ca8739b309ec4fc024de578b",
"type": "github"
},
"original": {
"owner": "numtide",
"repo": "flake-utils",
"type": "github"
}
},
"nixpkgs": {
"locked": {
"lastModified": 1789370336,
"narHash": "sha256-obwZbWHP7htCIoq2Z0g/uTMcEgN6yqcMNIcOMe6P2qI=",
"rev": "c7def046b9a883d46974757852106483d741586f",
"type": "tarball",
"url": "https://releases.nixos.org/nixpkgs/nixpkgs-26.11pre1073483.c7def046b9a8/nixexprs.tar.zst"
},
"original": {
"id": "nixpkgs",
"type": "indirect"
}
},
"root": {
"inputs": {
"flake-utils": "flake-utils",
"nixpkgs": "nixpkgs"
}
},
"systems": {
"locked": {
"lastModified": 1681028828,
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
"owner": "nix-systems",
"repo": "default",
"rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
"type": "github"
},
"original": {
"owner": "nix-systems",
"repo": "default",
"type": "github"
}
}
},
"root": "root",
"version": 7
}
+15
View File
@@ -0,0 +1,15 @@
{
description = "go-humanize";
nixConfig.bash-prompt = "\[go-humanize\]$ ";
inputs.flake-utils.url = "github:numtide/flake-utils";
outputs = { self, nixpkgs, flake-utils }:
flake-utils.lib.eachDefaultSystem
(system:
let pkgs = nixpkgs.legacyPackages.${system}; in
{
devShells.default = import ./shell.nix { inherit pkgs; };
}
);
}
+1 -1
View File
@@ -29,7 +29,7 @@ func stripTrailingDigits(s string, digits int) string {
return s[:i]
}
i++
if i+digits >= len(s) {
if digits >= len(s)-i {
return s
}
return s[:i+digits]
+5 -4
View File
@@ -42,11 +42,12 @@ var (
)
// FormatFloat produces a formatted number as string based on the following user-specified criteria:
//
// * thousands separator
// * decimal separator
// * decimal precision
//
// Usage: s := RenderFloat(format, n)
// Usage: s := FormatFloat(format, n)
// The format parameter tells how to render the number n.
//
// See examples: http://play.golang.org/p/LXc1Ddm1lJ
@@ -111,7 +112,7 @@ func FormatFloat(format string, n float64) string {
// +0000
if formatIndx[0] == 0 {
if format[formatIndx[0]] != '+' {
panic("RenderFloat(): invalid positive sign directive")
panic("FormatFloat(): invalid positive sign directive")
}
positiveStr = "+"
formatIndx = formatIndx[1:]
@@ -125,7 +126,7 @@ func FormatFloat(format string, n float64) string {
// 000,000.00
if len(formatIndx) == 2 {
if (formatIndx[1] - formatIndx[0]) != 4 {
panic("RenderFloat(): thousands separator directive must be followed by 3 digit-specifiers")
panic("FormatFloat(): thousands separator directive must be followed by 3 digit-specifiers")
}
thousandStr = string(format[formatIndx[0]])
formatIndx = formatIndx[1:]
@@ -160,7 +161,7 @@ func FormatFloat(format string, n float64) string {
intf, fracf := math.Modf(n + renderFloatPrecisionRounders[precision])
// generate integer part string
intStr := strconv.FormatInt(int64(intf), 10)
intStr := strconv.FormatFloat(intf, 'f', 0, 64)
// add thousand separator if required
if len(thousandStr) > 0 {
+11
View File
@@ -0,0 +1,11 @@
{ pkgs ? import <nixpkgs> { } }:
with pkgs;
mkShell {
buildInputs = [
go
gotools
];
shellHook = ''
'';
}
+43 -1
View File
@@ -33,12 +33,39 @@ var siPrefixTable = map[float64]string{
var revSIPrefixTable = revfmap(siPrefixTable)
// minSIExponent and maxSIExponent are the smallest and largest exponents
// siPrefixTable can name. They are derived from the table rather than written
// down so that adding a prefix (as ronto/quecto/ronna/quetta were added) keeps
// them correct.
var minSIExponent, maxSIExponent = func() (float64, float64) {
min, max := math.Inf(1), math.Inf(-1)
for k := range siPrefixTable {
if k < min {
min = k
}
if k > max {
max = k
}
}
return min, max
}()
// siParseAliases lets ParseSI accept extra spellings of prefixes that look
// identical to the canonical ones in siPrefixTable. Output still uses the
// canonical character; this is input-side only.
var siParseAliases = map[string]float64{
"μ": 1e-6, // GREEK SMALL LETTER MU, often typed instead of µ (U+00B5)
}
// revfmap reverses the map and precomputes the power multiplier
func revfmap(in map[float64]string) map[string]float64 {
rv := map[string]float64{}
for k, v := range in {
rv[v] = math.Pow(10, k)
}
for a, v := range siParseAliases {
rv[a] = v
}
return rv
}
@@ -49,6 +76,9 @@ func init() {
for _, v := range siPrefixTable {
ri += v
}
for a := range siParseAliases {
ri += a
}
ri += `]?)(.*)`
riParseRegex = regexp.MustCompile(ri)
@@ -69,11 +99,23 @@ func ComputeSI(input float64) (float64, string) {
exponent := math.Floor(logn(mag, 10))
exponent = math.Floor(exponent/3) * 3
// Outside the table there is no prefix that can name the magnitude, so
// clamp to the nearest one there is and let the returned value carry the
// remaining orders of magnitude. Without this the lookup below misses,
// the prefix comes back as the empty string, and the value is left scaled
// for a prefix that was never returned - so the caller silently gets a
// number off by as much as 10^300 with no error.
if exponent < minSIExponent {
exponent = minSIExponent
} else if exponent > maxSIExponent {
exponent = maxSIExponent
}
value := mag / math.Pow(10, exponent)
// Handle special case where value is exactly 1000.0
// Should return 1 M instead of 1000 k
if value == 1000.0 {
if value == 1000.0 && exponent+3 <= maxSIExponent {
exponent += 3
value = mag / math.Pow(10, exponent)
}
+9 -1
View File
@@ -4,6 +4,7 @@ import (
"fmt"
"math"
"sort"
"strings"
"time"
)
@@ -113,5 +114,12 @@ func CustomRelTime(a, b time.Time, albl, blbl string, magnitudes []RelTimeMagnit
escaped = ch == '%'
}
}
return fmt.Sprintf(mag.Format, args...)
s := fmt.Sprintf(mag.Format, args...)
if lbl == "" {
// When the label is empty, trim the whitespace its "%s"
// placeholder would otherwise leave behind (e.g. a trailing
// space in "3 weeks %s").
s = strings.TrimRight(s, " ")
}
return s
}
+4 -1
View File
@@ -248,9 +248,12 @@ func (cb *CellBuffer) Resize(w, h int) {
// If either the foreground or background are ColorNone, then the respective
// color is unchanged.
func (cb *CellBuffer) Fill(r rune, style Style) {
// PERF: convert the rune to a string once. all cells share the same
// underlying value so a single allocation suffices.
s := string(r)
for i := range cb.cells {
c := &cb.cells[i]
c.currStr = string(r)
c.currStr = s
cs := style
if cs.fg == ColorNone {
cs.fg = c.currStyle.fg
+4
View File
@@ -150,7 +150,11 @@ func (w *winTty) getConsoleInput() error {
rv, _, er := procGetNumberOfConsoleInputEvents.Call(
uintptr(w.in),
uintptr(unsafe.Pointer(&nrec)))
if rv == 0 {
return er
}
rec := make([]inputRecord, nrec)
rv, _, er = procReadConsoleInput.Call(
uintptr(w.in),
uintptr(unsafe.Pointer(&rec[0])),
+8 -3
View File
@@ -18,8 +18,8 @@ Alex Snast <alexsn at fb.com>
Alexey Palazhchenko <alexey.palazhchenko at gmail.com>
Andrew Reid <andrew.reid at tixtrack.com>
Animesh Ray <mail.rayanimesh at gmail.com>
Arne Hormann <arnehormann at gmail.com>
Ariel Mashraki <ariel at mashraki.co.il>
Arne Hormann <arnehormann at gmail.com>
Artur Melanchyk <artur.melanchyk@gmail.com>
Asta Xie <xiemengjun at gmail.com>
B Lamarche <blam413 at gmail.com>
@@ -38,6 +38,7 @@ Daniel Montoya <dsmontoyam at gmail.com>
Daniel Nichter <nil at codenode.com>
Daniël van Eeden <git at myname.nl>
Dave Protasowski <dprotaso at gmail.com>
Demouth <yuya at demouth.net>
Diego Dupin <diego.dupin at gmail.com>
Dirkjan Bussink <d.bussink at gmail.com>
DisposaBoy <disposaboy at dby.me>
@@ -66,6 +67,7 @@ Jeff Hodges <jeff at somethingsimilar.com>
Jeffrey Charles <jeffreycharles at gmail.com>
Jennifer Purevsuren <jennifer at dolthub.com>
Jerome Meyer <jxmeyer at gmail.com>
Jiabin Zhang <jiabin.z at qq.com>
Jiajia Zhong <zhong2plus at gmail.com>
Jian Zhen <zhenjl at gmail.com>
Joe Mann <contact at joemann.co.uk>
@@ -85,10 +87,12 @@ Linh Tran Tuan <linhduonggnu at gmail.com>
Lion Yang <lion at aosc.xyz>
Luca Looz <luca.looz92 at gmail.com>
Lucas Liu <extrafliu at gmail.com>
Lunny Xiao <xiaolunwen at gmail.com>
Luke Scott <luke at webconnex.com>
Lunny Xiao <xiaolunwen at gmail.com>
Maciej Zimnoch <maciej.zimnoch at codilime.com>
Michael Woolnough <michael.woolnough at gmail.com>
Minh Quang <minhquang4334 at gmail.com>
Morgan Tocker <tocker at gmail.com>
Nao Yokotsuka <yokotukanao at gmail.com>
Nathanial Murphy <nathanial.murphy at gmail.com>
Nicola Peduzzi <thenikso at gmail.com>
@@ -99,7 +103,6 @@ Paul Bonser <misterpib at gmail.com>
Paulius Lozys <pauliuslozys at gmail.com>
Peter Schultz <peter.schultz at classmarkets.com>
Phil Porada <philporada at gmail.com>
Minh Quang <minhquang4334 at gmail.com>
Rebecca Chin <rchin at pivotal.io>
Reed Allman <rdallman10 at gmail.com>
Richard Wilkes <wilkes at me.com>
@@ -126,6 +129,7 @@ Xiangyu Hu <xiangyu.hu at outlook.com>
Xiaobing Jiang <s7v7nislands at gmail.com>
Xiuming Chen <cc at cxm.cc>
Xuehong Chan <chanxuehong at gmail.com>
Yoshito Ohata <yoshito.dev at gmail.com>
Zhang Xiang <angwerzx at 126.com>
Zhenye Xie <xiezhenye at gmail.com>
Zhixin Wen <john.wenzhixin at gmail.com>
@@ -134,6 +138,7 @@ Ziheng Lyu <zihenglv at gmail.com>
# Organizations
Barracuda Networks, Inc.
Block, Inc.
Counting Ltd.
Defined Networking Inc.
DigitalOcean Inc.
+26 -3
View File
@@ -1,13 +1,36 @@
# Changelog
## v1.10.1 (2026-09-02)
* Fix `Config.FormatDSN()` dropping `Addr` when `Net` is empty.
It now uses the default `tcp` network so configs with only `Addr` round-trip correctly. (#1770)
* Fix typed-nil `json.RawMessage` with `interpolateParams=true` being interpolated as an empty string.
It is now interpolated as SQL `NULL`, matching server-side prepared statements. (#1782)
* Add MariaDB 11.8 and 12.3 to the test matrix. (#1774)
## v1.10.0 (2026-04-28)
* Fix `getSystemVar("max_allowed_packet")` potentially returned wrong value. (#1754)
This affects only when `maxAllowedPacket=0` is set.
* Bump filippo.io/edwards25519 from 1.1.1 to 1.2.0. (#1756)
While older versions have reported CVEs, they do not affect go-mysql.
* Update Go versions to 1.24-1.26. (#1763)
* Enhance interpolateParams to correctly handle placeholders. (#1732)
The question mark (?) within strings and comments will no longer be treated as a placeholder.
## v1.9.3 (2025-06-13)
* `tx.Commit()` and `tx.Rollback()` returned `ErrInvalidConn` always.
Now they return cached real error if present. (#1690)
* Optimize reading small resultsets to fix performance regression
introduced by compression protocol support. (#1707)
* Optimize reading small result sets to fix a performance regression
introduced by compression protocol support. (`#1707`)
* Fix `db.Ping()` on compressed connection. (#1723)
+9 -3
View File
@@ -1,5 +1,8 @@
# Go-MySQL-Driver
[![DeepWiki](https://img.shields.io/badge/DeepWiki-go--sql--driver%2Fmysql-blue.svg?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACwAAAAyCAYAAAAnWDnqAAAAAXNSR0IArs4c6QAAA05JREFUaEPtmUtyEzEQhtWTQyQLHNak2AB7ZnyXZMEjXMGeK/AIi+QuHrMnbChYY7MIh8g01fJoopFb0uhhEqqcbWTp06/uv1saEDv4O3n3dV60RfP947Mm9/SQc0ICFQgzfc4CYZoTPAswgSJCCUJUnAAoRHOAUOcATwbmVLWdGoH//PB8mnKqScAhsD0kYP3j/Yt5LPQe2KvcXmGvRHcDnpxfL2zOYJ1mFwrryWTz0advv1Ut4CJgf5uhDuDj5eUcAUoahrdY/56ebRWeraTjMt/00Sh3UDtjgHtQNHwcRGOC98BJEAEymycmYcWwOprTgcB6VZ5JK5TAJ+fXGLBm3FDAmn6oPPjR4rKCAoJCal2eAiQp2x0vxTPB3ALO2CRkwmDy5WohzBDwSEFKRwPbknEggCPB/imwrycgxX2NzoMCHhPkDwqYMr9tRcP5qNrMZHkVnOjRMWwLCcr8ohBVb1OMjxLwGCvjTikrsBOiA6fNyCrm8V1rP93iVPpwaE+gO0SsWmPiXB+jikdf6SizrT5qKasx5j8ABbHpFTx+vFXp9EnYQmLx02h1QTTrl6eDqxLnGjporxl3NL3agEvXdT0WmEost648sQOYAeJS9Q7bfUVoMGnjo4AZdUMQku50McDcMWcBPvr0SzbTAFDfvJqwLzgxwATnCgnp4wDl6Aa+Ax283gghmj+vj7feE2KBBRMW3FzOpLOADl0Isb5587h/U4gGvkt5v60Z1VLG8BhYjbzRwyQZemwAd6cCR5/XFWLYZRIMpX39AR0tjaGGiGzLVyhse5C9RKC6ai42ppWPKiBagOvaYk8lO7DajerabOZP46Lby5wKjw1HCRx7p9sVMOWGzb/vA1hwiWc6jm3MvQDTogQkiqIhJV0nBQBTU+3okKCFDy9WwferkHjtxib7t3xIUQtHxnIwtx4mpg26/HfwVNVDb4oI9RHmx5WGelRVlrtiw43zboCLaxv46AZeB3IlTkwouebTr1y2NjSpHz68WNFjHvupy3q8TFn3Hos2IAk4Ju5dCo8B3wP7VPr/FGaKiG+T+v+TQqIrOqMTL1VdWV1DdmcbO8KXBz6esmYWYKPwDL5b5FA1a0hwapHiom0r/cKaoqr+27/XcrS5UwSMbQAAAABJRU5ErkJggg==)](https://deepwiki.com/go-sql-driver/mysql)
A MySQL-Driver for Go's [database/sql](https://golang.org/pkg/database/sql/) package
![Go-MySQL-Driver logo](https://raw.github.com/wiki/go-sql-driver/mysql/gomysql_m.png "Golang Gopher holding the MySQL Dolphin")
@@ -42,8 +45,8 @@ A MySQL-Driver for Go's [database/sql](https://golang.org/pkg/database/sql/) pac
## Requirements
* Go 1.21 or higher. We aim to support the 3 latest versions of Go.
* MySQL (5.7+) and MariaDB (10.5+) are supported.
* Go 1.24 or higher. We aim to support the 3 latest versions of Go.
* MySQL (5.7+) and MariaDB (10.5+) are supported by maintainers.
* [TiDB](https://github.com/pingcap/tidb) is supported by PingCAP.
* Do not ask questions about TiDB in our issue tracker or forum.
* [Document](https://docs.pingcap.com/tidb/v6.1/dev-guide-sample-application-golang)
@@ -311,7 +314,10 @@ Type: duration
Default: 0
```
[Truncate time values](https://pkg.go.dev/time#Duration.Truncate) to the specified duration. The value must be a decimal number with a unit suffix (*"ms"*, *"s"*, *"m"*, *"h"*), such as *"30s"*, *"0.5m"* or *"1m30s"*.
[Truncate time values](https://pkg.go.dev/time#Duration.Truncate) in query arguments to the specified duration. The value must be a decimal number with a unit suffix (*"ns"*, *"us"*, *"ms"*, etc...), such as "1us", "1ms", or "10ns".
> [!NOTE]
> `time.Time` arguments are sent with up to nanosecond precision, so a value from `time.Now()` usually has more fractional-second digits than a `DATETIME(N)` or `TIMESTAMP(N)` column stores. On MariaDB, comparing such a value against an indexed column can prevent an index range scan, turning it into a full index scan. Truncating to the column's precision (`1us` for `DATETIME(6)`) avoids this. Only arguments sent to the server are truncated; values read from the server are not affected.
##### `maxAllowedPacket`
```
+1 -1
View File
@@ -305,7 +305,7 @@ func (mc *mysqlConn) auth(authData []byte, plugin string) ([]byte, error) {
if !mc.cfg.AllowNativePasswords {
return nil, ErrNativePassword
}
// https://dev.mysql.com/doc/internals/en/secure-password-authentication.html
// https://dev.mysql.com/doc/dev/mysql-server/8.4.5/page_protocol_connection_phase_authentication_methods_native_password_authentication.html
// Native password authentication only need and will need 20-byte challenge.
authResp := scramblePassword(authData[:20], mc.cfg.Passwd)
return authResp, nil
-1
View File
@@ -7,7 +7,6 @@
// You can obtain one at http://mozilla.org/MPL/2.0/.
//go:build linux || darwin || dragonfly || freebsd || netbsd || openbsd || solaris || illumos
// +build linux darwin dragonfly freebsd netbsd openbsd solaris illumos
package mysql
-1
View File
@@ -7,7 +7,6 @@
// You can obtain one at http://mozilla.org/MPL/2.0/.
//go:build !linux && !darwin && !dragonfly && !freebsd && !netbsd && !openbsd && !solaris && !illumos
// +build !linux,!darwin,!dragonfly,!freebsd,!netbsd,!openbsd,!solaris,!illumos
package mysql
+187 -90
View File
@@ -33,7 +33,8 @@ type mysqlConn struct {
connector *connector
maxAllowedPacket int
maxWriteSize int
flags clientFlag
capabilities capabilityFlag
extCapabilities extendedCapabilityFlag
status statusFlag
sequence uint8
compressSequence uint8
@@ -171,7 +172,7 @@ func (mc *mysqlConn) close() {
}
// Closes the network connection and unsets internal variables. Do not call this
// function after successfully authentication, call Close instead. This function
// function after successful authentication, call Close instead. This function
// is called before auth or on auth failure because MySQL will have already
// closed the network connection.
func (mc *mysqlConn) cleanup() {
@@ -223,13 +224,21 @@ func (mc *mysqlConn) Prepare(query string) (driver.Stmt, error) {
columnCount, err := stmt.readPrepareResultPacket()
if err == nil {
if stmt.paramCount > 0 {
if err = mc.readUntilEOF(); err != nil {
if err = mc.skipColumns(stmt.paramCount); err != nil {
return nil, err
}
}
if columnCount > 0 {
err = mc.readUntilEOF()
if mc.extCapabilities&clientCacheMetadata != 0 {
if stmt.columns, err = mc.readColumns(int(columnCount), nil); err != nil {
return nil, err
}
} else {
if err = mc.skipColumns(int(columnCount)); err != nil {
return nil, err
}
}
}
}
@@ -237,100 +246,186 @@ func (mc *mysqlConn) Prepare(query string) (driver.Stmt, error) {
}
func (mc *mysqlConn) interpolateParams(query string, args []driver.Value) (string, error) {
// Number of ? should be same to len(args)
if strings.Count(query, "?") != len(args) {
return "", driver.ErrSkip
}
noBackslashEscapes := (mc.status & statusNoBackslashEscapes) != 0
const (
stateNormal = iota
stateString
stateEscape
stateEOLComment
stateSlashStarComment
stateBacktick
)
const (
QUOTE_BYTE = byte('\'')
DBL_QUOTE_BYTE = byte('"')
BACKSLASH_BYTE = byte('\\')
QUESTION_MARK_BYTE = byte('?')
SLASH_BYTE = byte('/')
STAR_BYTE = byte('*')
HASH_BYTE = byte('#')
MINUS_BYTE = byte('-')
LINE_FEED_BYTE = byte('\n')
BACKTICK_BYTE = byte('`')
)
buf, err := mc.buf.takeCompleteBuffer()
if err != nil {
// can not take the buffer. Something must be wrong with the connection
mc.cleanup()
// interpolateParams would be called before sending any query.
// So its safe to retry.
return "", driver.ErrBadConn
}
buf = buf[:0]
state := stateNormal
singleQuotes := false
lastChar := byte(0)
argPos := 0
lenQuery := len(query)
lastIdx := 0
for i := 0; i < len(query); i++ {
q := strings.IndexByte(query[i:], '?')
if q == -1 {
buf = append(buf, query[i:]...)
break
}
buf = append(buf, query[i:i+q]...)
i += q
arg := args[argPos]
argPos++
if arg == nil {
buf = append(buf, "NULL"...)
for i := range lenQuery {
currentChar := query[i]
if state == stateEscape && !((currentChar == QUOTE_BYTE && singleQuotes) || (currentChar == DBL_QUOTE_BYTE && !singleQuotes)) {
state = stateString
lastChar = currentChar
continue
}
switch v := arg.(type) {
case int64:
buf = strconv.AppendInt(buf, v, 10)
case uint64:
// Handle uint64 explicitly because our custom ConvertValue emits unsigned values
buf = strconv.AppendUint(buf, v, 10)
case float64:
buf = strconv.AppendFloat(buf, v, 'g', -1, 64)
case bool:
if v {
buf = append(buf, '1')
} else {
buf = append(buf, '0')
switch currentChar {
case STAR_BYTE:
if state == stateNormal && lastChar == SLASH_BYTE {
state = stateSlashStarComment
}
case time.Time:
if v.IsZero() {
buf = append(buf, "'0000-00-00'"...)
} else {
buf = append(buf, '\'')
buf, err = appendDateTime(buf, v.In(mc.cfg.Loc), mc.cfg.timeTruncate)
if err != nil {
return "", err
}
buf = append(buf, '\'')
case SLASH_BYTE:
if state == stateSlashStarComment && lastChar == STAR_BYTE {
state = stateNormal
// Clear lastChar so the '/' that closed the comment isn't
// reused to start a new comment with a following '*'.
lastChar = 0
continue
}
case json.RawMessage:
buf = append(buf, '\'')
if mc.status&statusNoBackslashEscapes == 0 {
buf = escapeBytesBackslash(buf, v)
} else {
buf = escapeBytesQuotes(buf, v)
case HASH_BYTE:
if state == stateNormal {
state = stateEOLComment
}
buf = append(buf, '\'')
case []byte:
if v == nil {
buf = append(buf, "NULL"...)
} else {
buf = append(buf, "_binary'"...)
if mc.status&statusNoBackslashEscapes == 0 {
buf = escapeBytesBackslash(buf, v)
case MINUS_BYTE:
if state == stateNormal && lastChar == MINUS_BYTE {
// -- only starts a comment if followed by whitespace or control char
if i+1 < lenQuery {
nextChar := query[i+1]
if nextChar == ' ' || nextChar == '\t' || nextChar == '\n' || nextChar == '\r' {
state = stateEOLComment
}
} else {
buf = escapeBytesQuotes(buf, v)
state = stateEOLComment
}
buf = append(buf, '\'')
}
case string:
buf = append(buf, '\'')
if mc.status&statusNoBackslashEscapes == 0 {
buf = escapeStringBackslash(buf, v)
} else {
buf = escapeStringQuotes(buf, v)
case LINE_FEED_BYTE:
if state == stateEOLComment {
state = stateNormal
}
buf = append(buf, '\'')
default:
return "", driver.ErrSkip
}
case DBL_QUOTE_BYTE:
if state == stateNormal {
state = stateString
singleQuotes = false
} else if state == stateString && !singleQuotes {
state = stateNormal
} else if state == stateEscape {
state = stateString
}
case QUOTE_BYTE:
if state == stateNormal {
state = stateString
singleQuotes = true
} else if state == stateString && singleQuotes {
state = stateNormal
} else if state == stateEscape {
state = stateString
}
case BACKSLASH_BYTE:
if state == stateString && !noBackslashEscapes {
state = stateEscape
}
case QUESTION_MARK_BYTE:
if state == stateNormal {
if argPos >= len(args) {
return "", driver.ErrSkip
}
buf = append(buf, query[lastIdx:i]...)
arg := args[argPos]
argPos++
if len(buf)+4 > mc.maxAllowedPacket {
return "", driver.ErrSkip
if arg == nil {
buf = append(buf, "NULL"...)
lastIdx = i + 1
break
}
switch v := arg.(type) {
case int64:
buf = strconv.AppendInt(buf, v, 10)
case uint64:
buf = strconv.AppendUint(buf, v, 10)
case float64:
buf = strconv.AppendFloat(buf, v, 'g', -1, 64)
case bool:
if v {
buf = append(buf, '1')
} else {
buf = append(buf, '0')
}
case time.Time:
if v.IsZero() {
buf = append(buf, "'0000-00-00'"...)
} else {
buf = append(buf, '\'')
buf, err = appendDateTime(buf, v.In(mc.cfg.Loc), mc.cfg.timeTruncate)
if err != nil {
return "", err
}
buf = append(buf, '\'')
}
case json.RawMessage:
if v == nil {
buf = append(buf, "NULL"...)
} else if noBackslashEscapes {
buf = escapeBytesQuotes(buf, v, false)
} else {
buf = escapeBytesBackslash(buf, v, false)
}
case []byte:
if v == nil {
buf = append(buf, "NULL"...)
} else {
if noBackslashEscapes {
buf = escapeBytesQuotes(buf, v, true)
} else {
buf = escapeBytesBackslash(buf, v, true)
}
}
case string:
if noBackslashEscapes {
buf = escapeStringQuotes(buf, v)
} else {
buf = escapeStringBackslash(buf, v)
}
default:
return "", driver.ErrSkip
}
if len(buf)+4 > mc.maxAllowedPacket {
return "", driver.ErrSkip
}
lastIdx = i + 1
}
case BACKTICK_BYTE:
if state == stateBacktick {
state = stateNormal
} else if state == stateNormal {
state = stateBacktick
}
}
lastChar = currentChar
}
buf = append(buf, query[lastIdx:]...)
if argPos != len(args) {
return "", driver.ErrSkip
}
@@ -370,19 +465,19 @@ func (mc *mysqlConn) exec(query string) error {
}
// Read Result
resLen, err := handleOk.readResultSetHeaderPacket()
resLen, _, err := handleOk.readResultSetHeaderPacket()
if err != nil {
return err
}
if resLen > 0 {
// columns
if err := mc.readUntilEOF(); err != nil {
if err := mc.skipColumns(resLen); err != nil {
return err
}
// rows
if err := mc.readUntilEOF(); err != nil {
if err := mc.skipRows(); err != nil {
return err
}
}
@@ -419,7 +514,7 @@ func (mc *mysqlConn) query(query string, args []driver.Value) (*textRows, error)
// Read Result
var resLen int
resLen, err = handleOk.readResultSetHeaderPacket()
resLen, _, err = handleOk.readResultSetHeaderPacket()
if err != nil {
return nil, err
}
@@ -439,21 +534,20 @@ func (mc *mysqlConn) query(query string, args []driver.Value) (*textRows, error)
}
// Columns
rows.rs.columns, err = mc.readColumns(resLen)
rows.rs.columns, err = mc.readColumns(resLen, nil)
return rows, err
}
// Gets the value of the given MySQL System Variable
// The returned byte slice is only valid until the next read
func (mc *mysqlConn) getSystemVar(name string) ([]byte, error) {
func (mc *mysqlConn) getSystemVar(name string) (string, error) {
// Send command
handleOk := mc.clearResult()
if err := mc.writeCommandPacketStr(comQuery, "SELECT @@"+name); err != nil {
return nil, err
return "", err
}
// Read Result
resLen, err := handleOk.readResultSetHeaderPacket()
resLen, _, err := handleOk.readResultSetHeaderPacket()
if err == nil {
rows := new(textRows)
rows.mc = mc
@@ -461,17 +555,20 @@ func (mc *mysqlConn) getSystemVar(name string) ([]byte, error) {
if resLen > 0 {
// Columns
if err := mc.readUntilEOF(); err != nil {
return nil, err
if err := mc.skipColumns(resLen); err != nil {
return "", err
}
}
dest := make([]driver.Value, resLen)
if err = rows.readRow(dest); err == nil {
return dest[0].([]byte), mc.readUntilEOF()
// Convert to string before skipRows, which may
// overwrite the read buffer that dest[0] points into.
val := string(dest[0].([]byte))
return val, mc.skipRows()
}
}
return nil, err
return "", err
}
// cancel is called when the query has canceled.
+7 -5
View File
@@ -42,7 +42,7 @@ func encodeConnectionAttributes(cfg *Config) string {
}
// user-defined connection attributes
for _, connAttr := range strings.Split(cfg.ConnectionAttributes, ",") {
for connAttr := range strings.SplitSeq(cfg.ConnectionAttributes, ",") {
k, v, found := strings.Cut(connAttr, ":")
if !found {
continue
@@ -131,7 +131,7 @@ func (c *connector) Connect(ctx context.Context) (driver.Conn, error) {
mc.buf = newBuffer()
// Reading Handshake Initialization Packet
authData, plugin, err := mc.readHandshakePacket()
authData, serverCapabilities, serverExtCapabilities, plugin, err := mc.readHandshakePacket()
if err != nil {
mc.cleanup()
return nil, err
@@ -153,6 +153,7 @@ func (c *connector) Connect(ctx context.Context) (driver.Conn, error) {
return nil, err
}
}
mc.initCapabilities(serverCapabilities, serverExtCapabilities, mc.cfg)
if err = mc.writeHandshakeResponsePacket(authResp, plugin); err != nil {
mc.cleanup()
return nil, err
@@ -161,13 +162,14 @@ func (c *connector) Connect(ctx context.Context) (driver.Conn, error) {
// Handle response to auth packet, switch methods if possible
if err = mc.handleAuthResult(authData, plugin); err != nil {
// Authentication failed and MySQL has already closed the connection
// (https://dev.mysql.com/doc/internals/en/authentication-fails.html).
// (https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase.html#sect_protocol_connection_phase_fast_path_fails).
// Do not send COM_QUIT, just cleanup and return the error.
mc.cleanup()
return nil, err
}
if mc.cfg.compress && mc.flags&clientCompress == clientCompress {
// compression is enabled after auth, not right after sending handshake response.
if mc.capabilities&clientCompress > 0 {
mc.compress = true
mc.compIO = newCompIO(mc)
}
@@ -180,7 +182,7 @@ func (c *connector) Connect(ctx context.Context) (driver.Conn, error) {
mc.Close()
return nil, err
}
n, err := strconv.Atoi(string(maxap))
n, err := strconv.Atoi(maxap)
if err != nil {
mc.Close()
return nil, fmt.Errorf("invalid max_allowed_packet value (%q): %w", maxap, err)
+17 -4
View File
@@ -32,7 +32,7 @@ const (
)
// MySQL constants documentation:
// http://dev.mysql.com/doc/internals/en/client-server-protocol.html
// https://dev.mysql.com/doc/dev/mysql-server/latest/PAGE_PROTOCOL.html
const (
iOK byte = 0x00
@@ -42,11 +42,12 @@ const (
iERR byte = 0xff
)
// https://dev.mysql.com/doc/internals/en/capability-flags.html#packet-Protocol::CapabilityFlags
type clientFlag uint32
// https://dev.mysql.com/doc/dev/mysql-server/latest/group__group__cs__capabilities__flags.html
// https://mariadb.com/kb/en/connection/#capabilities
type capabilityFlag uint32
const (
clientLongPassword clientFlag = 1 << iota
clientMySQL capabilityFlag = 1 << iota
clientFoundRows
clientLongFlag
clientConnectWithDB
@@ -73,6 +74,18 @@ const (
clientDeprecateEOF
)
// https://mariadb.com/kb/en/connection/#capabilities
type extendedCapabilityFlag uint32
const (
progressIndicator extendedCapabilityFlag = 1 << iota
clientComMulti
clientStmtBulkOperations
clientExtendedMetadata
clientCacheMetadata
clientUnitBulkResult
)
const (
comQuit byte = iota + 1
comInitDB
+16 -11
View File
@@ -15,6 +15,7 @@ import (
"crypto/tls"
"errors"
"fmt"
"maps"
"math/big"
"net"
"net/url"
@@ -157,9 +158,7 @@ func (cfg *Config) Clone() *Config {
}
if len(cp.Params) > 0 {
cp.Params = make(map[string]string, len(cfg.Params))
for k, v := range cfg.Params {
cp.Params[k] = v
}
maps.Copy(cp.Params, cfg.Params)
}
if cfg.pubKey != nil {
cp.pubKey = &rsa.PublicKey{
@@ -265,13 +264,19 @@ func (cfg *Config) FormatDSN() string {
}
// [protocol[(address)]]
if len(cfg.Net) > 0 {
buf.WriteString(cfg.Net)
if len(cfg.Addr) > 0 {
buf.WriteByte('(')
buf.WriteString(cfg.Addr)
buf.WriteByte(')')
if len(cfg.Addr) > 0 {
net := cfg.Net
if net == "" {
net = "tcp"
}
buf.WriteString(net)
buf.WriteByte('(')
buf.WriteString(cfg.Addr)
buf.WriteByte(')')
} else if cfg.Net != "" && cfg.Net != "tcp" {
// Preserve an explicit non-default protocol when there's no
// address, so e.g. Net="unix" still round-trips.
buf.WriteString(cfg.Net)
}
// /dbname
@@ -414,7 +419,7 @@ func ParseDSN(dsn string) (cfg *Config, err error) {
if dsn[j] == '@' {
// username[:password]
// Find the first ':' in dsn[:j]
for k = 0; k < j; k++ {
for k = 0; k < j; k++ { // We cannot use k = range j here, because we use dsn[:k] below
if dsn[k] == ':' {
cfg.Passwd = dsn[k+1 : j]
break
@@ -477,7 +482,7 @@ func ParseDSN(dsn string) (cfg *Config, err error) {
// parseDSNParams parses the DSN "query string"
// Values must be url.QueryEscape'ed
func parseDSNParams(cfg *Config, params string) (err error) {
for _, v := range strings.Split(params, "&") {
for v := range strings.SplitSeq(params, "&") {
key, value, found := strings.Cut(v, "=")
if !found {
continue
+21 -17
View File
@@ -120,23 +120,24 @@ func (mf *mysqlField) typeDatabaseName() string {
}
var (
scanTypeFloat32 = reflect.TypeOf(float32(0))
scanTypeFloat64 = reflect.TypeOf(float64(0))
scanTypeInt8 = reflect.TypeOf(int8(0))
scanTypeInt16 = reflect.TypeOf(int16(0))
scanTypeInt32 = reflect.TypeOf(int32(0))
scanTypeInt64 = reflect.TypeOf(int64(0))
scanTypeNullFloat = reflect.TypeOf(sql.NullFloat64{})
scanTypeNullInt = reflect.TypeOf(sql.NullInt64{})
scanTypeNullTime = reflect.TypeOf(sql.NullTime{})
scanTypeUint8 = reflect.TypeOf(uint8(0))
scanTypeUint16 = reflect.TypeOf(uint16(0))
scanTypeUint32 = reflect.TypeOf(uint32(0))
scanTypeUint64 = reflect.TypeOf(uint64(0))
scanTypeString = reflect.TypeOf("")
scanTypeNullString = reflect.TypeOf(sql.NullString{})
scanTypeBytes = reflect.TypeOf([]byte{})
scanTypeUnknown = reflect.TypeOf(new(any))
scanTypeFloat32 = reflect.TypeFor[float32]()
scanTypeFloat64 = reflect.TypeFor[float64]()
scanTypeInt8 = reflect.TypeFor[int8]()
scanTypeInt16 = reflect.TypeFor[int16]()
scanTypeInt32 = reflect.TypeFor[int32]()
scanTypeInt64 = reflect.TypeFor[int64]()
scanTypeNullFloat = reflect.TypeFor[sql.NullFloat64]()
scanTypeNullInt = reflect.TypeFor[sql.NullInt64]()
scanTypeNullUint = reflect.TypeFor[sql.Null[uint64]]()
scanTypeNullTime = reflect.TypeFor[sql.NullTime]()
scanTypeUint8 = reflect.TypeFor[uint8]()
scanTypeUint16 = reflect.TypeFor[uint16]()
scanTypeUint32 = reflect.TypeFor[uint32]()
scanTypeUint64 = reflect.TypeFor[uint64]()
scanTypeString = reflect.TypeFor[string]()
scanTypeNullString = reflect.TypeFor[sql.NullString]()
scanTypeBytes = reflect.TypeFor[[]byte]()
scanTypeUnknown = reflect.TypeFor[*any]()
)
type mysqlField struct {
@@ -185,6 +186,9 @@ func (mf *mysqlField) scanType() reflect.Type {
}
return scanTypeInt64
}
if mf.flags&flagUnsigned != 0 {
return scanTypeNullUint
}
return scanTypeNullInt
case fieldTypeFloat:
+1 -4
View File
@@ -95,10 +95,7 @@ const defaultPacketSize = 16 * 1024 // 16KB is small enough for disk readahead a
func (mc *okHandler) handleInFileRequest(name string) (err error) {
var rdr io.Reader
packetSize := defaultPacketSize
if mc.maxWriteSize < packetSize {
packetSize = mc.maxWriteSize
}
packetSize := min(mc.maxWriteSize, defaultPacketSize)
if idx := strings.Index(name, "Reader::"); idx == 0 || (idx > 0 && name[idx-1] == '/') { // io.Reader
// The server might return an an absolute path. See issue #355.
+205 -138
View File
@@ -179,20 +179,22 @@ func (mc *mysqlConn) writePacket(data []byte) error {
******************************************************************************/
// Handshake Initialization Packet
// http://dev.mysql.com/doc/internals/en/connection-phase-packets.html#packet-Protocol::Handshake
func (mc *mysqlConn) readHandshakePacket() (data []byte, plugin string, err error) {
// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_packets_protocol_handshake_v10.html
// https://mariadb.com/kb/en/connection/#initial-handshake-packet
func (mc *mysqlConn) readHandshakePacket() (data []byte, capabilities capabilityFlag, extendedCapabilities extendedCapabilityFlag, plugin string, err error) {
data, err = mc.readPacket()
if err != nil {
return
}
if data[0] == iERR {
return nil, "", mc.handleErrorPacket(data)
err = mc.handleErrorPacket(data)
return
}
// protocol version [1 byte]
if data[0] < minProtocolVersion {
return nil, "", fmt.Errorf(
return nil, 0, 0, "", fmt.Errorf(
"unsupported protocol version %d. Version %d or higher is required",
data[0],
minProtocolVersion,
@@ -210,15 +212,15 @@ func (mc *mysqlConn) readHandshakePacket() (data []byte, plugin string, err erro
pos += 8 + 1
// capability flags (lower 2 bytes) [2 bytes]
mc.flags = clientFlag(binary.LittleEndian.Uint16(data[pos : pos+2]))
if mc.flags&clientProtocol41 == 0 {
return nil, "", ErrOldProtocol
capabilities = capabilityFlag(binary.LittleEndian.Uint16(data[pos : pos+2]))
if capabilities&clientProtocol41 == 0 {
return nil, capabilities, 0, "", ErrOldProtocol
}
if mc.flags&clientSSL == 0 && mc.cfg.TLS != nil {
if capabilities&clientSSL == 0 && mc.cfg.TLS != nil {
if mc.cfg.AllowFallbackToPlaintext {
mc.cfg.TLS = nil
} else {
return nil, "", ErrNoTLS
return nil, capabilities, 0, "", ErrNoTLS
}
}
pos += 2
@@ -228,11 +230,16 @@ func (mc *mysqlConn) readHandshakePacket() (data []byte, plugin string, err erro
// status flags [2 bytes]
pos += 3
// capability flags (upper 2 bytes) [2 bytes]
mc.flags |= clientFlag(binary.LittleEndian.Uint16(data[pos:pos+2])) << 16
capabilities |= capabilityFlag(binary.LittleEndian.Uint16(data[pos:pos+2])) << 16
pos += 2
// length of auth-plugin-data [1 byte]
// reserved (all [00]) [10 bytes]
pos += 11
// reserved (all [00]) [6 bytes]
pos += 7
if capabilities&clientMySQL == 0 {
// MariaDB server extended flag
extendedCapabilities = extendedCapabilityFlag(binary.LittleEndian.Uint32(data[pos : pos+4]))
}
pos += 4
// second part of the password cipher [minimum 13 bytes],
// where len=MAX(13, length of auth-plugin-data - 8)
@@ -260,82 +267,72 @@ func (mc *mysqlConn) readHandshakePacket() (data []byte, plugin string, err erro
// make a memory safe copy of the cipher slice
var b [20]byte
copy(b[:], authData)
return b[:], plugin, nil
return b[:], capabilities, extendedCapabilities, plugin, nil
}
// make a memory safe copy of the cipher slice
var b [8]byte
copy(b[:], authData)
return b[:], plugin, nil
return b[:], capabilities, 0, plugin, nil
}
// Client Authentication Packet
// http://dev.mysql.com/doc/internals/en/connection-phase-packets.html#packet-Protocol::HandshakeResponse
func (mc *mysqlConn) writeHandshakeResponsePacket(authResp []byte, plugin string) error {
// Adjust client flags based on server support
clientFlags := clientProtocol41 |
clientSecureConn |
clientLongPassword |
clientTransactions |
clientLocalFiles |
clientPluginAuth |
clientMultiResults |
mc.flags&clientConnectAttrs |
mc.flags&clientLongFlag
// initCapabilities initializes the capabilities based on server support and configuration
func (mc *mysqlConn) initCapabilities(serverCapabilities capabilityFlag, serverExtCapabilities extendedCapabilityFlag, cfg *Config) {
clientCapabilities :=
clientMySQL |
clientLongFlag |
clientProtocol41 |
clientSecureConn |
clientTransactions |
clientPluginAuthLenEncClientData |
clientLocalFiles |
clientPluginAuth |
clientMultiResults |
clientConnectAttrs |
clientDeprecateEOF
sendConnectAttrs := mc.flags&clientConnectAttrs != 0
if mc.cfg.ClientFoundRows {
clientFlags |= clientFoundRows
if cfg.ClientFoundRows {
clientCapabilities |= clientFoundRows
}
if mc.cfg.compress && mc.flags&clientCompress == clientCompress {
clientFlags |= clientCompress
if cfg.compress {
clientCapabilities |= clientCompress
}
// To enable TLS / SSL
if mc.cfg.TLS != nil {
clientFlags |= clientSSL
clientCapabilities |= clientSSL
}
if mc.cfg.MultiStatements {
clientFlags |= clientMultiStatements
clientCapabilities |= clientMultiStatements
}
if n := len(cfg.DBName); n > 0 {
clientCapabilities |= clientConnectWithDB
}
// encode length of the auth plugin data
var authRespLEIBuf [9]byte
authRespLen := len(authResp)
authRespLEI := appendLengthEncodedInteger(authRespLEIBuf[:0], uint64(authRespLen))
if len(authRespLEI) > 1 {
// if the length can not be written in 1 byte, it must be written as a
// length encoded integer
clientFlags |= clientPluginAuthLenEncClientData
}
// only keep client capabilities that server have
mc.capabilities = clientCapabilities & serverCapabilities
pktLen := 4 + 4 + 1 + 23 + len(mc.cfg.User) + 1 + len(authRespLEI) + len(authResp) + 21 + 1
// set MariaDB extended clientCacheMetadata capability if server support it
mc.extCapabilities = clientCacheMetadata & serverExtCapabilities
}
// To specify a db name
if n := len(mc.cfg.DBName); n > 0 {
clientFlags |= clientConnectWithDB
pktLen += n + 1
}
// encode length of the connection attributes
var connAttrsLEI []byte
if sendConnectAttrs {
var connAttrsLEIBuf [9]byte
connAttrsLen := len(mc.connector.encodedAttributes)
connAttrsLEI = appendLengthEncodedInteger(connAttrsLEIBuf[:0], uint64(connAttrsLen))
pktLen += len(connAttrsLEI) + len(mc.connector.encodedAttributes)
}
// Calculate packet length and get buffer with that size
data, err := mc.buf.takeBuffer(pktLen + 4)
// Client Authentication Packet
// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_packets_protocol_handshake_response.html
func (mc *mysqlConn) writeHandshakeResponsePacket(authResp []byte, plugin string) error {
// packet header 4
// capabilities 4
// maxPacketSize 4
// collation id 1
// filler 23
data, err := mc.buf.takeSmallBuffer(4*3 + 24)
if err != nil {
mc.cleanup()
return err
}
_ = data[4*3+23] // boundery check
// ClientFlags [32 bit]
binary.LittleEndian.PutUint32(data[4:], uint32(clientFlags))
// clientCapabilities [32 bit]
binary.LittleEndian.PutUint32(data[4:], uint32(mc.capabilities))
// MaxPacketSize [32 bit] (none)
binary.LittleEndian.PutUint32(data[8:], 0)
@@ -353,16 +350,26 @@ func (mc *mysqlConn) writeHandshakeResponsePacket(authResp []byte, plugin string
}
// Filler [23 bytes] (all 0x00)
// or filler 19bytes + mariadb extCapabilities
pos := 13
for ; pos < 13+23; pos++ {
data[pos] = 0
if mc.capabilities&clientMySQL == 0 {
for ; pos < 13+19; pos++ {
data[pos] = 0
}
// MariaDB Extended Capabilities
binary.LittleEndian.PutUint32(data[13+19:], uint32(mc.extCapabilities))
} else {
for ; pos < 13+23; pos++ {
data[pos] = 0
}
}
// SSL Connection Request Packet
// http://dev.mysql.com/doc/internals/en/connection-phase-packets.html#packet-Protocol::SSLRequest
// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_packets_protocol_ssl_request.html
// https://mariadb.com/kb/en/connection/#sslrequest-packet
if mc.cfg.TLS != nil {
// Send TLS / SSL request packet
if err := mc.writePacket(data[:(4+4+1+23)+4]); err != nil {
if err := mc.writePacket(data); err != nil {
return err
}
@@ -379,37 +386,35 @@ func (mc *mysqlConn) writeHandshakeResponsePacket(authResp []byte, plugin string
// User [null terminated string]
if len(mc.cfg.User) > 0 {
pos += copy(data[pos:], mc.cfg.User)
data = append(data, mc.cfg.User...)
}
data[pos] = 0x00
pos++
data = append(data, 0)
// Auth Data [length encoded integer]
pos += copy(data[pos:], authRespLEI)
pos += copy(data[pos:], authResp)
data = appendLengthEncodedInteger(data, uint64(len(authResp)))
data = append(data, authResp...)
// Databasename [null terminated string]
if len(mc.cfg.DBName) > 0 {
pos += copy(data[pos:], mc.cfg.DBName)
data[pos] = 0x00
pos++
// Database name [null terminated string]
if mc.capabilities&clientConnectWithDB != 0 {
data = append(data, mc.cfg.DBName...)
data = append(data, 0)
}
pos += copy(data[pos:], plugin)
data[pos] = 0x00
pos++
data = append(data, plugin...)
data = append(data, 0)
// Connection Attributes
if sendConnectAttrs {
pos += copy(data[pos:], connAttrsLEI)
pos += copy(data[pos:], []byte(mc.connector.encodedAttributes))
if mc.capabilities&clientConnectAttrs != 0 {
connAttrsLen := len(mc.connector.encodedAttributes)
data = appendLengthEncodedInteger(data, uint64(connAttrsLen))
data = append(data, mc.connector.encodedAttributes...)
}
// Send Auth packet
return mc.writePacket(data[:pos])
return mc.writePacket(data)
}
// http://dev.mysql.com/doc/internals/en/connection-phase-packets.html#packet-Protocol::AuthSwitchResponse
// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_packets_protocol_auth_switch_response.html
func (mc *mysqlConn) writeAuthSwitchPacket(authData []byte) error {
pktLen := 4 + len(authData)
data, err := mc.buf.takeBuffer(pktLen)
@@ -511,7 +516,7 @@ func (mc *mysqlConn) readAuthResult() ([]byte, string, error) {
case iEOF:
if len(data) == 1 {
// https://dev.mysql.com/doc/internals/en/connection-phase-packets.html#packet-Protocol::OldAuthSwitchRequest
// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_packets_protocol_old_auth_switch_request.html
return nil, "mysql_old_password", nil
}
pluginEndIndex := bytes.IndexByte(data, 0x00)
@@ -545,36 +550,41 @@ func (mc *okHandler) readResultOK() error {
// Result Set Header Packet
// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_query_response.html
func (mc *okHandler) readResultSetHeaderPacket() (int, error) {
func (mc *okHandler) readResultSetHeaderPacket() (int, bool, error) {
// handleOkPacket replaces both values; other cases leave the values unchanged.
mc.result.affectedRows = append(mc.result.affectedRows, 0)
mc.result.insertIds = append(mc.result.insertIds, 0)
data, err := mc.conn().readPacket()
if err != nil {
return 0, err
return 0, false, err
}
switch data[0] {
case iOK:
return 0, mc.handleOkPacket(data)
return 0, false, mc.handleOkPacket(data)
case iERR:
return 0, mc.conn().handleErrorPacket(data)
return 0, false, mc.conn().handleErrorPacket(data)
case iLocalInFile:
return 0, mc.handleInFileRequest(string(data[1:]))
return 0, false, mc.handleInFileRequest(string(data[1:]))
}
// column count
// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_query_response_text_resultset.html
num, _, _ := readLengthEncodedInteger(data)
// https://mariadb.com/kb/en/result-set-packets/#column-count-packet
num, _, len := readLengthEncodedInteger(data)
if mc.extCapabilities&clientCacheMetadata != 0 {
return int(num), data[len] == 0x01, nil
}
// ignore remaining data in the packet. see #1478.
return int(num), nil
return int(num), true, nil
}
// Error Packet
// http://dev.mysql.com/doc/internals/en/generic-response-packets.html#packet-ERR_Packet
// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_err_packet.html
func (mc *mysqlConn) handleErrorPacket(data []byte) error {
if data[0] != iERR {
return ErrMalformPkt
@@ -656,7 +666,7 @@ func (mc *mysqlConn) clearResult() *okHandler {
}
// Ok Packet
// http://dev.mysql.com/doc/internals/en/generic-response-packets.html#packet-OK_Packet
// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_ok_packet.html
func (mc *okHandler) handleOkPacket(data []byte) error {
var n, m int
var affectedRows, insertId uint64
@@ -690,24 +700,19 @@ func (mc *okHandler) handleOkPacket(data []byte) error {
}
// Read Packets as Field Packets until EOF-Packet or an Error appears
// http://dev.mysql.com/doc/internals/en/com-query-response.html#packet-Protocol::ColumnDefinition41
func (mc *mysqlConn) readColumns(count int) ([]mysqlField, error) {
// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_query_response_text_resultset_column_definition.html#sect_protocol_com_query_response_text_resultset_column_definition_41
func (mc *mysqlConn) readColumns(count int, old []mysqlField) ([]mysqlField, error) {
columns := make([]mysqlField, count)
if len(old) != count {
old = nil
}
for i := 0; ; i++ {
for i := range count {
data, err := mc.readPacket()
if err != nil {
return nil, err
}
// EOF Packet
if data[0] == iEOF && (len(data) == 5 || len(data) == 1) {
if i == count {
return columns, nil
}
return nil, fmt.Errorf("column count mismatch n:%d len:%d", count, len(columns))
}
// Catalog
pos, err := skipLengthEncodedString(data)
if err != nil {
@@ -728,7 +733,12 @@ func (mc *mysqlConn) readColumns(count int) ([]mysqlField, error) {
return nil, err
}
pos += n
columns[i].tableName = string(tableName)
if old != nil && old[i].tableName == string(tableName) {
// avoid allocating new string
columns[i].tableName = old[i].tableName
} else {
columns[i].tableName = string(tableName)
}
} else {
n, err = skipLengthEncodedString(data[pos:])
if err != nil {
@@ -749,7 +759,12 @@ func (mc *mysqlConn) readColumns(count int) ([]mysqlField, error) {
if err != nil {
return nil, err
}
columns[i].name = string(name)
if old != nil && old[i].name == string(name) {
// avoid allocating new string
columns[i].name = old[i].name
} else {
columns[i].name = string(name)
}
pos += n
// Original name [len coded string]
@@ -780,17 +795,17 @@ func (mc *mysqlConn) readColumns(count int) ([]mysqlField, error) {
// Decimals [uint8]
columns[i].decimals = data[pos]
//pos++
// Default value [len coded binary]
//if pos < len(data) {
// defaultVal, _, err = bytesToLengthCodedBinary(data[pos:])
//}
}
// skip EOF packet if client does not support deprecateEOF
if err := mc.skipEof(); err != nil {
return nil, err
}
return columns, nil
}
// Read Packets as Field Packets until EOF-Packet or an Error appears
// http://dev.mysql.com/doc/internals/en/com-query-response.html#packet-ProtocolText::ResultsetRow
// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_query_response_text_resultset_row.html
func (rows *textRows) readRow(dest []driver.Value) error {
mc := rows.mc
@@ -804,9 +819,20 @@ func (rows *textRows) readRow(dest []driver.Value) error {
}
// EOF Packet
if data[0] == iEOF && len(data) == 5 {
// server_status [2 bytes]
rows.mc.status = readStatus(data[3:])
// text row packets may starts with LengthEncodedString.
// In such case, 0xFE can mean string larger than 0xffffff.
// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_dt_integers.html#sect_protocol_basic_dt_int_le
if data[0] == iEOF && len(data) <= 0xffffff {
if mc.capabilities&clientDeprecateEOF == 0 {
// Deprecated EOF packet
// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_eof_packet.html
mc.status = readStatus(data[3:])
} else {
// Ok Packet with an 0xFE header
_, _, n := readLengthEncodedInteger(data[1:]) // affected_rows
_, _, m := readLengthEncodedInteger(data[1+n:]) // last_insert_id
mc.status = readStatus(data[1+n+m:])
}
rows.rs.done = true
if !rows.HasNextResultSet() {
rows.mc = nil
@@ -880,8 +906,34 @@ func (rows *textRows) readRow(dest []driver.Value) error {
return nil
}
// Reads Packets until EOF-Packet or an Error appears. Returns count of Packets read
func (mc *mysqlConn) readUntilEOF() error {
func (mc *mysqlConn) skipPackets(n int) error {
for range n {
if _, err := mc.readPacket(); err != nil {
return err
}
}
return nil
}
// skips EOF packet after n * ColumnDefinition packets when clientDeprecateEOF is not set
func (mc *mysqlConn) skipEof() error {
if mc.capabilities&clientDeprecateEOF == 0 {
if _, err := mc.readPacket(); err != nil {
return err
}
}
return nil
}
func (mc *mysqlConn) skipColumns(n int) error {
if err := mc.skipPackets(n); err != nil {
return err
}
return mc.skipEof()
}
// Reads Packets until EOF-Packet or an Error appears.
func (mc *mysqlConn) skipRows() error {
for {
data, err := mc.readPacket()
if err != nil {
@@ -892,10 +944,20 @@ func (mc *mysqlConn) readUntilEOF() error {
case iERR:
return mc.handleErrorPacket(data)
case iEOF:
if len(data) == 5 {
mc.status = readStatus(data[3:])
// text row packets may starts with LengthEncodedString.
// In such case, 0xFE can mean string larger than 0xffffff.
if len(data) <= 0xffffff {
if mc.capabilities&clientDeprecateEOF == 0 {
// EOF packet
mc.status = readStatus(data[3:])
} else {
// OK packet with an 0xFE header
_, _, n := readLengthEncodedInteger(data[1:]) // affected_rows
_, _, m := readLengthEncodedInteger(data[1+n:]) // last_insert_id
mc.status = readStatus(data[1+n+m:])
}
return nil
}
return nil
}
}
}
@@ -905,7 +967,7 @@ func (mc *mysqlConn) readUntilEOF() error {
******************************************************************************/
// Prepare Result Packets
// http://dev.mysql.com/doc/internals/en/com-stmt-prepare-response.html
// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_stmt_prepare.html#sect_protocol_com_stmt_prepare_response
func (stmt *mysqlStmt) readPrepareResultPacket() (uint16, error) {
data, err := stmt.mc.readPacket()
if err == nil {
@@ -932,7 +994,7 @@ func (stmt *mysqlStmt) readPrepareResultPacket() (uint16, error) {
return 0, err
}
// http://dev.mysql.com/doc/internals/en/com-stmt-send-long-data.html
// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_stmt_send_long_data.html
func (stmt *mysqlStmt) writeCommandLongData(paramID int, arg []byte) error {
maxLen := stmt.mc.maxAllowedPacket - 1
pktLen := maxLen
@@ -979,7 +1041,7 @@ func (stmt *mysqlStmt) writeCommandLongData(paramID int, arg []byte) error {
}
// Execute Prepared Statement
// http://dev.mysql.com/doc/internals/en/com-stmt-execute.html
// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_stmt_execute.html
func (stmt *mysqlStmt) writeExecutePacket(args []driver.Value) error {
if len(args) != stmt.paramCount {
return fmt.Errorf(
@@ -993,10 +1055,7 @@ func (stmt *mysqlStmt) writeExecutePacket(args []driver.Value) error {
mc := stmt.mc
// Determine threshold dynamically to avoid packet size shortage.
longDataSize := mc.maxAllowedPacket / (stmt.paramCount + 1)
if longDataSize < 64 {
longDataSize = 64
}
longDataSize := max(mc.maxAllowedPacket/(stmt.paramCount+1), 64)
// Reset packet-sequence
mc.resetSequence()
@@ -1185,17 +1244,17 @@ func (stmt *mysqlStmt) writeExecutePacket(args []driver.Value) error {
// mc.affectedRows and mc.insertIds.
func (mc *okHandler) discardResults() error {
for mc.status&statusMoreResultsExists != 0 {
resLen, err := mc.readResultSetHeaderPacket()
resLen, _, err := mc.readResultSetHeaderPacket()
if err != nil {
return err
}
if resLen > 0 {
// columns
if err := mc.conn().readUntilEOF(); err != nil {
if err := mc.conn().skipColumns(resLen); err != nil {
return err
}
// rows
if err := mc.conn().readUntilEOF(); err != nil {
if err := mc.conn().skipRows(); err != nil {
return err
}
}
@@ -1203,7 +1262,7 @@ func (mc *okHandler) discardResults() error {
return nil
}
// http://dev.mysql.com/doc/internals/en/binary-protocol-resultset-row.html
// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_binary_resultset.html#sect_protocol_binary_resultset_row
func (rows *binaryRows) readRow(dest []driver.Value) error {
data, err := rows.mc.readPacket()
if err != nil {
@@ -1212,9 +1271,17 @@ func (rows *binaryRows) readRow(dest []driver.Value) error {
// packet indicator [1 byte]
if data[0] != iOK {
// EOF Packet
if data[0] == iEOF && len(data) == 5 {
rows.mc.status = readStatus(data[3:])
// EOF/OK Packet
if data[0] == iEOF {
if rows.mc.capabilities&clientDeprecateEOF == 0 {
// EOF packet
rows.mc.status = readStatus(data[3:])
} else {
// OK Packet with an 0xFE header
_, _, n := readLengthEncodedInteger(data[1:])
_, _, m := readLengthEncodedInteger(data[1+n:])
rows.mc.status = readStatus(data[1+n+m:])
}
rows.rs.done = true
if !rows.HasNextResultSet() {
rows.mc = nil
+4 -2
View File
@@ -8,6 +8,8 @@
package mysql
import "slices"
import "database/sql/driver"
// Result exposes data not available through *connection.Result.
@@ -42,9 +44,9 @@ func (res *mysqlResult) RowsAffected() (int64, error) {
}
func (res *mysqlResult) AllLastInsertIds() []int64 {
return append([]int64{}, res.insertIds...) // defensive copy
return slices.Clone(res.insertIds) // defensive copy
}
func (res *mysqlResult) AllRowsAffected() []int64 {
return append([]int64{}, res.affectedRows...) // defensive copy
return slices.Clone(res.affectedRows) // defensive copy
}
+5 -5
View File
@@ -113,7 +113,7 @@ func (rows *mysqlRows) Close() (err error) {
// Remove unread packets from stream
if !rows.rs.done {
err = mc.readUntilEOF()
err = mc.skipRows()
}
if err == nil {
handleOk := mc.clearResult()
@@ -143,7 +143,7 @@ func (rows *mysqlRows) nextResultSet() (int, error) {
// Remove unread packets from stream
if !rows.rs.done {
if err := rows.mc.readUntilEOF(); err != nil {
if err := rows.mc.skipRows(); err != nil {
return 0, err
}
rows.rs.done = true
@@ -156,7 +156,7 @@ func (rows *mysqlRows) nextResultSet() (int, error) {
rows.rs = resultSet{}
// rows.mc.affectedRows and rows.mc.insertIds accumulate on each call to
// nextResultSet.
resLen, err := rows.mc.resultUnchanged().readResultSetHeaderPacket()
resLen, _, err := rows.mc.resultUnchanged().readResultSetHeaderPacket()
if err != nil {
// Clean up about multi-results flag
rows.rs.done = true
@@ -186,7 +186,7 @@ func (rows *binaryRows) NextResultSet() error {
return err
}
rows.rs.columns, err = rows.mc.readColumns(resLen)
rows.rs.columns, err = rows.mc.readColumns(resLen, nil)
return err
}
@@ -208,7 +208,7 @@ func (rows *textRows) NextResultSet() (err error) {
return err
}
rows.rs.columns, err = rows.mc.readColumns(resLen)
rows.rs.columns, err = rows.mc.readColumns(resLen, nil)
return err
}
+26 -8
View File
@@ -20,6 +20,7 @@ type mysqlStmt struct {
mc *mysqlConn
id uint32
paramCount int
columns []mysqlField
}
func (stmt *mysqlStmt) Close() error {
@@ -64,19 +65,26 @@ func (stmt *mysqlStmt) Exec(args []driver.Value) (driver.Result, error) {
handleOk := stmt.mc.clearResult()
// Read Result
resLen, err := handleOk.readResultSetHeaderPacket()
resLen, metadataFollows, err := handleOk.readResultSetHeaderPacket()
if err != nil {
return nil, err
}
if resLen > 0 {
// Columns
if err = mc.readUntilEOF(); err != nil {
return nil, err
if metadataFollows && stmt.mc.extCapabilities&clientCacheMetadata != 0 {
// we can not skip column metadata because next stmt.Query() may use it.
if stmt.columns, err = mc.readColumns(resLen, stmt.columns); err != nil {
return nil, err
}
} else {
if err = mc.skipColumns(resLen); err != nil {
return nil, err
}
}
// Rows
if err := mc.readUntilEOF(); err != nil {
if err = mc.skipRows(); err != nil {
return nil, err
}
}
@@ -107,7 +115,7 @@ func (stmt *mysqlStmt) query(args []driver.Value) (*binaryRows, error) {
// Read Result
handleOk := stmt.mc.clearResult()
resLen, err := handleOk.readResultSetHeaderPacket()
resLen, metadataFollows, err := handleOk.readResultSetHeaderPacket()
if err != nil {
return nil, err
}
@@ -116,7 +124,17 @@ func (stmt *mysqlStmt) query(args []driver.Value) (*binaryRows, error) {
if resLen > 0 {
rows.mc = mc
rows.rs.columns, err = mc.readColumns(resLen)
if metadataFollows {
if rows.rs.columns, err = mc.readColumns(resLen, stmt.columns); err != nil {
return nil, err
}
stmt.columns = rows.rs.columns
} else {
if err = mc.skipEof(); err != nil {
return nil, err
}
rows.rs.columns = stmt.columns
}
} else {
rows.rs.done = true
@@ -131,7 +149,7 @@ func (stmt *mysqlStmt) query(args []driver.Value) (*binaryRows, error) {
return rows, err
}
var jsonType = reflect.TypeOf(json.RawMessage{})
var jsonType = reflect.TypeFor[json.RawMessage]()
type converter struct{}
@@ -193,7 +211,7 @@ func (c converter) ConvertValue(v any) (driver.Value, error) {
return nil, fmt.Errorf("unsupported type %T, a %s", v, rv.Kind())
}
var valuerReflectType = reflect.TypeOf((*driver.Valuer)(nil)).Elem()
var valuerReflectType = reflect.TypeFor[driver.Valuer]()
// callValuerValue returns vr.Value(), with one exception:
// If vr.Value is an auto-generated method on a pointer type and the
+65 -90
View File
@@ -182,7 +182,7 @@ func parseDateTime(b []byte, loc *time.Location) (time.Time, error) {
func parseByteYear(b []byte) (int, error) {
year, n := 0, 1000
for i := 0; i < 4; i++ {
for i := range 4 {
v, err := bToi(b[i])
if err != nil {
return 0, err
@@ -207,7 +207,7 @@ func parseByte2Digits(b1, b2 byte) (int, error) {
func parseByteNanoSec(b []byte) (int, error) {
ns, digit := 0, 100000 // max is 6-digits
for i := 0; i < len(b); i++ {
for i := range b {
v, err := bToi(b[i])
if err != nil {
return 0, err
@@ -625,108 +625,80 @@ func reserveBuffer(buf []byte, appendSize int) []byte {
return buf[:newSize]
}
// escapeBytesBackslash escapes []byte with backslashes (\)
// This escapes the contents of a string (provided as []byte) by adding backslashes before special
// characters, and turning others into specific escape sequences, such as
// turning newlines into \n and null bytes into \0.
// https://github.com/mysql/mysql-server/blob/mysql-5.7.5/mysys/charset.c#L823-L932
func escapeBytesBackslash(buf, v []byte) []byte {
pos := len(buf)
buf = reserveBuffer(buf, len(v)*2)
// Lookup table for backslash escapes (used for both string and bytes)
var backslashEscapeTable [256]byte
for _, c := range v {
switch c {
case '\x00':
buf[pos+1] = '0'
buf[pos] = '\\'
pos += 2
case '\n':
buf[pos+1] = 'n'
buf[pos] = '\\'
pos += 2
case '\r':
buf[pos+1] = 'r'
buf[pos] = '\\'
pos += 2
case '\x1a':
buf[pos+1] = 'Z'
buf[pos] = '\\'
pos += 2
case '\'':
buf[pos+1] = '\''
buf[pos] = '\\'
pos += 2
case '"':
buf[pos+1] = '"'
buf[pos] = '\\'
pos += 2
case '\\':
buf[pos+1] = '\\'
buf[pos] = '\\'
pos += 2
default:
buf[pos] = c
pos++
}
}
return buf[:pos]
func init() {
backslashEscapeTable['\x00'] = '0'
backslashEscapeTable['\n'] = 'n'
backslashEscapeTable['\r'] = 'r'
backslashEscapeTable['\x1a'] = 'Z'
backslashEscapeTable['\''] = '\''
backslashEscapeTable['"'] = '"'
backslashEscapeTable['\\'] = '\\'
}
// escapeStringBackslash is similar to escapeBytesBackslash but for string.
func escapeStringBackslash(buf []byte, v string) []byte {
pos := len(buf)
buf = reserveBuffer(buf, len(v)*2)
buf = reserveBuffer(buf, len(v)*2+2)
buf[pos] = '\''
pos++
for i := 0; i < len(v); i++ {
c := v[i]
switch c {
case '\x00':
buf[pos+1] = '0'
if esc := backslashEscapeTable[c]; esc != 0 {
buf[pos+1] = esc
buf[pos] = '\\'
pos += 2
case '\n':
buf[pos+1] = 'n'
buf[pos] = '\\'
pos += 2
case '\r':
buf[pos+1] = 'r'
buf[pos] = '\\'
pos += 2
case '\x1a':
buf[pos+1] = 'Z'
buf[pos] = '\\'
pos += 2
case '\'':
buf[pos+1] = '\''
buf[pos] = '\\'
pos += 2
case '"':
buf[pos+1] = '"'
buf[pos] = '\\'
pos += 2
case '\\':
buf[pos+1] = '\\'
buf[pos] = '\\'
pos += 2
default:
} else {
buf[pos] = c
pos++
}
}
buf[pos] = '\''
pos++
return buf[:pos]
}
// escapeBytesQuotes escapes apostrophes in []byte by doubling them up.
// This escapes the contents of a string by doubling up any apostrophes that
// it contains. This is used when the NO_BACKSLASH_ESCAPES SQL_MODE is in
// effect on the server.
// https://github.com/mysql/mysql-server/blob/mysql-5.7.5/mysys/charset.c#L963-L1038
func escapeBytesQuotes(buf, v []byte) []byte {
// escapeBytesBackslash appends _binary'...' or '...' with backslash escaping for bytes.
func escapeBytesBackslash(buf, v []byte, binary bool) []byte {
pos := len(buf)
buf = reserveBuffer(buf, len(v)*2)
if binary {
buf = reserveBuffer(buf, len(v)*2+9)
copy(buf[pos:], []byte("_binary'"))
pos += 8
} else {
buf = reserveBuffer(buf, len(v)*2+2)
buf[pos] = '\''
pos++
}
for _, c := range v {
if esc := backslashEscapeTable[c]; esc != 0 {
buf[pos+1] = esc
buf[pos] = '\\'
pos += 2
} else {
buf[pos] = c
pos++
}
}
buf[pos] = '\''
pos++
return buf[:pos]
}
// escapeBytesQuotes appends _binary'...' or '...' with single-quote escaping for bytes.
func escapeBytesQuotes(buf, v []byte, binary bool) []byte {
pos := len(buf)
if binary {
buf = reserveBuffer(buf, len(v)*2+9)
copy(buf[pos:], []byte("_binary'"))
pos += 8
} else {
buf = reserveBuffer(buf, len(v)*2+2)
buf[pos] = '\''
pos++
}
for _, c := range v {
if c == '\'' {
buf[pos+1] = '\''
@@ -737,16 +709,18 @@ func escapeBytesQuotes(buf, v []byte) []byte {
pos++
}
}
buf[pos] = '\''
pos++
return buf[:pos]
}
// escapeStringQuotes is similar to escapeBytesQuotes but for string.
func escapeStringQuotes(buf []byte, v string) []byte {
pos := len(buf)
buf = reserveBuffer(buf, len(v)*2)
for i := 0; i < len(v); i++ {
buf = reserveBuffer(buf, len(v)*2+2)
buf[pos] = '\''
pos++
for i := range len(v) {
c := v[i]
if c == '\'' {
buf[pos+1] = '\''
@@ -757,7 +731,8 @@ func escapeStringQuotes(buf []byte, v string) []byte {
pos++
}
}
buf[pos] = '\''
pos++
return buf[:pos]
}
+21 -1
View File
@@ -22,6 +22,26 @@ _testmain.go
*.exe
.envrc
/.testdb
# Per-checkout development state: this checkout's allocated TCP ports, the values derived from
# them, its decoded client certificates, and its own PostgreSQL/CockroachDB clusters. A git
# worktree is the native equivalent of a second devcontainer instance, so this is instance-local
# by definition -- never shared, never committed. See DEVELOPMENT.md.
/.dev/
# port-tamer writes this checkout's allocation to .dev/ports.env (the --state-file every task
# passes). These are where a bare `port-tamer` run at the repo root would put a stray second
# allocation instead; ignored so it cannot be committed by accident.
/.port-tamer.env
/.port-tamer.env.lock
# Personal, machine-local mise overrides (mise.toml IS committed/shared). Git has no knowledge of
# mise's conventions, so this must be ignored explicitly.
/mise.local.toml
# Provisioned on demand with `rake references:setup`; never committed.
/references/
.DS_Store
/.vscode
+2
View File
@@ -6,6 +6,8 @@ linters:
enable:
- govet
- ineffassign
- unconvert
- gocritic
# See: https://golangci-lint.run/usage/formatters/
formatters:
+15
View File
@@ -0,0 +1,15 @@
# Host dependencies only. Project tools and language runtimes belong in
# mise.toml; scripts/setup-host installs mise itself when necessary.
# Server and client binaries for the checkout-local test matrix.
brew "postgresql@14"
brew "postgresql@15"
brew "postgresql@16"
brew "postgresql@17"
brew "postgresql@18"
# mise compiles Ruby from source and links against these.
brew "openssl@3"
brew "readline"
brew "libyaml"
brew "gmp"
+227
View File
@@ -1,3 +1,230 @@
# 5.11.0 (September 7, 2026)
This release adds direct PostgreSQL type scanning through `database/sql` on Go 1.27, improves compatibility with
libpq connection strings and PostgreSQL date/time values, and includes further decoder hardening. See Changes for
connection-string and date/time behavior changes that may affect existing applications.
## Features
* stdlib: support Go 1.27's `driver.RowsColumnScanner`, allowing PostgreSQL types such as arrays and ranges to be
scanned directly into Go values without `pgtype.Map.SQLScanner`. Existing `database/sql` scalar conversions and
`sql.Scanner` behavior are preserved. The minimum supported Go version remains 1.25.
* Add `Rows.TypeMap` to expose the type map used to decode rows, including rows created by `RowsFromResultReader`
that have no underlying `Conn`. Custom implementations of `Rows`, including mocks, must add this method.
* pgconn: add `Config.MaxProtocolMessageBodyLen` to configure the maximum incoming protocol message body size
(carter-ya)
* pgconn: add `ErrReadOnlyConnection`, `ErrReadWriteConnection`, `ErrPrimaryConnection`, and `ErrStandbyConnection`
sentinel errors for `target_session_attrs` validation, allowing callers to use `errors.Is` (Adrian-Stefan Mares)
* pgxpool: accept `pool_ping_timeout` in connection strings to configure `Config.PingTimeout`. The default is zero;
zero and negative durations mean no timeout (1991santhu)
## Changes
* Name-based row-to-struct mapping now matches explicit `db` tags case-insensitively, with exact matches taking
precedence so tags can still distinguish quoted column names that differ only by case (AlisinaDevelo)
* pgconn: resolve the OS user account only when no user is supplied by the connection string, environment, or service
file, avoiding unnecessary account lookups and crashes in some restricted container environments. Home-directory
defaults for password, service, and TLS files remain available independently of the account lookup. On Unix these
now use `$HOME` rather than the OS account's home directory (Mohamed MAACHE)
* pgtype: `date`, `timestamp` and `timestamptz` text values are now parsed and written by a hand-written parser and
encoder for PostgreSQL's ISO date/time format instead of `time.Parse` and `time.Format`. Go's layout language cannot
express a variable-width year or the BC era, which is the root of the bugs below. The text scan path is roughly 2.5x
faster for `timestamp` and `timestamptz`. Bug fixes:
* `timestamp` and `timestamptz` no longer silently move February 29 of a BC leap year to March 1 when encoding.
`time.Date(-4712, 2, 29, ...)` was written as `4713-03-01 BC` and is now written as `4713-02-29 BC`. This
affected ordinary four-digit BC years, not only extended-range ones. `date` was never affected.
* `timestamp` and `timestamptz` can now scan BC leap days. `4713-02-29 BC` previously failed with
`day out of range`. `date` could already scan them.
* Years past 9999 can now be scanned. `10000-01-02 03:04:05` previously failed to parse, so `timestamp` and
`timestamptz` values at the high end of PostgreSQL's range were unreadable over the simple protocol and in any
other text-format result.
* `time.Time` arguments in the simple protocol now encode BC dates correctly, using the same timestamp encoder.
* Fractional seconds beyond microsecond precision are rounded the way the server rounds them (round half to even,
carrying into the rest of the value) instead of being kept at full precision. PostgreSQL never sends more than six
fractional digits, so this only affects values from other sources.
Behavior changes:
* `date` now rejects impossible dates instead of normalizing them. `2024-02-30` returned `2024-03-01` and
`2024-13-01` returned `2025-01-01`; both are now errors. `timestamp` and `timestamptz` already rejected them.
* All three types now reject values outside PostgreSQL's range for that type, in the binary format as well as the
text format. PostgreSQL never sends out-of-range dates, so this only affects corrupt or hand-built input; the range
is checked in both formats so that whether a value is accepted does not depend on `QueryExecMode`.
`timestamptz` also rejects time zone displacements outside PostgreSQL's signed 32-bit seconds range, while accepting
the wider offsets emitted for POSIX time zones, such as `+16`.
* `timestamptz` values scanned from the text format are now returned in `time.Local`, or in `ScanLocation` when it is
set, matching what the binary format has always returned. Previously the text path kept whatever location
`time.Parse` derived from the offset the server sent, so the same value scanned in the two formats could report a
different `Location()` and `Zone()`. The instant is unchanged, but everything that renders the location changes
with it: `Timestamptz.MarshalJSON` now writes the client's offset rather than the server's, so a value the server
sent as `+05:30` marshals as `2024-01-01T13:34:05-08:00` on a UTC-8 client instead of `2024-01-02T03:04:05+05:30`,
and `DecodeDatabaseSQLValue` hands `database/sql` a `time.Time` in that same location. Set the codec's
`ScanLocation` to `time.UTC` to pin the location regardless of the client's zone.
* Error messages from these paths have changed.
* pgconn: connection URIs (`postgres://...`) are now parsed by a new parser designed to exactly match libpq's URI
parser behavior instead of `net/url`,
making pgx accept and reject exactly the same URIs as libpq (verified by differential fuzzing against libpq itself).
Most connection strings are unaffected. Edge-case behavior changes, all matching libpq:
* `+` in query values is literal, no longer decoded as a space.
* Malformed percent-encoding is a parse error instead of the parameter being silently dropped. `%00` is rejected.
* Leading/trailing spaces in URI components are trimmed; interior spaces are a parse error (encode them as `%20`).
* `#` is ordinary data, not a fragment delimiter.
* The userinfo terminator is the first `@` before any `/` (previously the last `@`).
* When a query parameter is repeated, the last occurrence wins (previously the first).
* `ssl=true` is accepted as an alias for `sslmode=require` in URIs (JDBC compatibility). A repeated `ssl` key
follows the same last-occurrence-wins rule as other repeated parameters, even across the rewrite to `sslmode`. If
the final `ssl` value is not `true`, an independent explicit `sslmode` remains in effect.
* Multiple hosts with mixed port specs are positionally aligned: `postgres://h1,h2:5433/db` now means h1:5432 and
h2:5433 (previously both hosts got port 5433). A port list that is neither a single port nor exactly one port per
host is an error (`could not match N port numbers to M hosts`), also for keyword/value connection strings.
* An IPv6 address in a URI must be enclosed in brackets. A bare `postgres://::1/db` was previously accepted as host
`::1`; it is now read as an empty host followed by port `:1` and fails with an invalid port error. Write it as
`postgres://[::1]/db`.
* Empty host list elements (e.g. `h1,,h2`) get the default host instead of being dropped. Likewise, an empty host in
a keyword/value string (`host=`) now means the default host -- typically the Unix socket directory -- where it
previously meant a TCP connection to an empty hostname.
* An empty port (`?port=` in a URI or `port=` in a keyword/value string) now means the default port 5432 for the
affected hosts; previously it was an invalid port error. Like any connection-string port, a present-but-empty port
takes precedence over `PGPORT`.
* ASCII control characters (tab, newline, ...) in a URI are ordinary data bytes, as they are to libpq; `net/url`
rejected any URI containing one. The exception is a literal NUL byte, which is still rejected, as `net/url` did.
(libpq never sees one -- C strings end at the first NUL -- but in Go a raw NUL could otherwise pass through into
the NUL-delimited startup message and inject extra parameters.)
Unlike libpq, unrecognized URI query parameters are still accepted (they become runtime parameters or pgx-specific
options). Parse error messages avoid quoting the unredacted connection string and redact recognizable password
fields on a best-effort basis. Invalid connection strings can be structurally ambiguous, so password redaction
cannot be guaranteed for every malformed input.
* pgconn: keyword/value connection strings (`host=... user=...`) now match libpq's parser exactly, the same treatment
the URI parser received above and verified the same way, by differential fuzzing against libpq itself. Most
connection strings are unaffected. Behavior changes, all matching libpq:
* A backslash escapes whatever character follows it and is dropped, where previously only `\\` and `\'` were
unescaped and every other backslash was kept. A value containing a backslash must now escape it, as libpq
requires: `sslcert=C:\path\to\cert` reads as `C:pathtocert` and has to be written `sslcert=C:\\path\\to\\cert`.
This mainly affects Windows certificate and key paths, which previously came through intact without doubling.
* A trailing backslash in an unquoted value escapes the end of the string, so it is dropped and the value ends
there; it was previously rejected with `invalid backslash`. Inside a quoted value the escaped terminator leaves
the string unterminated, which is still an error.
* Whitespace inside a keyword is an error (`missing "=" after "us" in connection info string`) instead of becoming
part of the key. Whitespace around the `=` is unaffected. This most often shows up with an unquoted value
containing a space: `application_name=my app host=x` previously set neither parameter and sent `app host` to the
server as a runtime parameter, and now fails to parse.
As with URIs, unrecognized keywords are still accepted where libpq rejects them, and an empty `user=` is still
dropped so that `PGUSER` and the OS user still apply.
## Fixes
* Keep the connection open after a recoverable PostgreSQL error from `Begin` or `BeginTx`
(Victor Alejandro Sanz Ararat)
* Call `TraceQueryEnd` when `Exec` fails while deallocating invalidated cached statements (Chris Bandy)
* Deallocate a failed prepare using the statement name actually sent to the server, and skip cleanup if Parse never
completed, avoiding leaked prepared statements and unnecessary cleanup errors (Eliran Ben-Zikri)
* Fix `LoadTypes` overwriting scalar codecs such as `box` and `point` with an incorrect `ArrayCodec` (Arsen Ozhetov)
* pgconn: retrieve field descriptions when cached descriptions are empty, such as for cursor `FETCH` statements,
including batch and pipeline execution (water)
* pgconn: keep batch statement descriptions and result formats aligned when commands return no rows or when
`Batch.ExecStatement` is mixed with other batch commands; preserve field descriptions for empty results
* pgconn: handle empty and comment-only queries in pipeline mode, discard stale statement data after bind errors,
and return a nil result from `Pipeline.GetResults` on error
* pgconn: skip reading the password file when a password is already set (Jared Fowkes)
* pgxpool: treat non-positive `MaxConnLifetime` values as unlimited instead of immediately expiring connections
(Aurelien Pillevesse)
* pgtype: support non-comma text array delimiters through `ArrayCodec.Delimiter`, including the semicolon delimiter
used by `box[]`. `LoadType` and `LoadTypes` now load the delimiter from PostgreSQL (Sueun Cho)
* pgtype: quote text array elements containing internal whitespace (Louisa Huang)
* pgtype: quote and escape text range bounds containing delimiters, quotes, or backslashes, and distinguish empty
string bounds from unbounded ranges (Sueun Cho)
* pgtype: preserve decimal precision in `Numeric.ScanScientific` and accept scientific notation in
`Numeric.UnmarshalJSON`; reject out-of-range scientific exponents and preserve the original input in parse errors
(Sueun Cho)
* pgtype: encode and decode numeric infinity in JSON as `"Infinity"` and `"-Infinity"` instead of encoding it as zero
(Vladimir Saraikin)
* pgtype: treat a valid `Numeric` with a nil `Int` as zero in `Int64Value`, and return errors when converting NaN or
infinity to an integer instead of panicking (Vladimir Saraikin)
* pgtype: fix an infinite loop when decoding binary numeric zero with a nonzero digit count (Vladimir Saraikin)
* pgtype: fix binary numeric digit-count overflow and trailing-byte handling. Binary encoding now rejects values
whose digit count, weight, or scale cannot fit the wire format, while accepting the full unsigned digit-count range.
* pgtype: fix scanning through multiple pointer levels, including SQL NULL and XML values, and return an error
instead of panicking when a pointer-to-pointer scan destination is nil (Rangel Reale)
* pgtype: use bounds-checked binary reads throughout the codecs and reject malformed lengths, counts, and trailing
data. This includes fixes for panics on malformed records and truncated multiranges (Vladimir Saraikin), and
validation of `bit` / `varbit` bit lengths against the actual data (g3m0sis).
* pgtype: return errors instead of panicking on malformed interval text (greymoth-jp), unterminated composite text
fields, and text arrays whose dimensions and element counts disagree
* pgtype: cap the initial allocation estimate when parsing hstore text to avoid excessive allocation from unvalidated
separator counts; valid hstores may still contain any number of pairs (AshSgDe29071999)
* pgtype: correct reversed bounds in integer scan error messages
* pgconn: a backslash as the last byte of a quoted value in a keyword/value connection string no longer panics with
`slice bounds out of range`. `host='a\` -- and the shorter `='\`, reachable through `pgx.ParseConfig` and
`pgxpool.ParseConfig` -- now return `unterminated quoted string in connection info string`, libpq's own message for
the same input. The unquoted branch has been guarded since be69c1c1; the quoted branch carried the same unguarded
increment since the parser was ported from pgx v3. Found by fuzzing (Maxim Korotkov)
* pgconn: error messages that embed the connection string now also redact `password` and `sslpassword` values supplied
as URI query parameters; previously only the userinfo password was redacted. Redaction matches keys the way the
parser does -- percent-encoded spellings such as `pass%77ord=` are recognized -- and masks the entire raw value, so
a password containing a space cannot leak its tail into the error message. Credentials stranded outside the
userinfo by a malformed URI are masked whole, and invalid-port errors no longer embed the offending text (which in
a malformed URI can be a mislaid password). Redaction of invalid connection strings is necessarily best effort:
their structure may be ambiguous, so some malformed inputs can still expose password text in an error.
* pgconn: `ParseConfigOptions.ConnStringAllowedKeys` no longer exempts an explicitly supplied empty port (`?port=` in
a URI or `port=` in a keyword/value string) from the allow-list. Only the implied all-empty port list of a
multi-host URI without ports (`postgres://h1,h2/db`) is exempt. An explicit empty port shadows `PGPORT` even though
it is empty, so it must be allowed like any other user-supplied key. The URI-only `ssl=true` alias is accepted when
either `ssl` or `sslmode` is allowed, and every `ssl`/`sslmode` spelling written in the URI is validated --
including occurrences superseded by later repeated parameters.
* pgconn: drain socket before close in `asyncClose` so context cancellation produces a TCP FIN instead of RST, avoiding "connection reset by peer" on the server / proxy (Sean Chittenden at CrowdStrike, Inc.)
* pgproto3: `StartupMessage.Encode` rejects a NUL byte in any parameter name or value instead of writing it. The
startup message body is a run of NUL-delimited strings whose length is data-driven, so a NUL in a value ends that
parameter and everything after it is read by the server as further parameters -- an `application_name` of
`x\x00user\x00admin` changed the role the connection logged in as. libpq cannot reach this state because its
parameters are NUL-terminated C strings. `Connect` now fails with nothing written to the wire, which covers
settings that bypass connection string parsing: service files and direct assignment to `Config.RuntimeParams`,
`Config.User`, or `Config.Database`.
* pgconn: keyword/value connection strings containing a NUL byte are rejected by `ParseConfig`, as URIs already were.
# 5.10.0 (June 3, 2026)
This release includes a significant amount of hardening against malicious or compromised PostgreSQL servers,
contributed by Sean Chittenden at CrowdStrike, Inc. This work bounds binary decoders against attacker-controlled
message sizes, caps server-supplied SCRAM iteration counts, adds `require_auth` to restrict which authentication
methods a server may use (mitigating downgrade attacks under `sslmode=prefer`), and ensures cancellation requests are
sent over TLS when the original connection used TLS.
## Features
* Add `require_auth` to restrict accepted server authentication methods (Sean Chittenden at CrowdStrike, Inc.)
* Add `ParseConfigOptions.ConnStringAllowedKeys` to restrict allowed connection string keys (Sean Chittenden at CrowdStrike, Inc.)
* Add `StructArgs` and `StrictStructArgs` for `@`-named queries (Tubelight30)
* Add `ErrConnClosed` sentinel error and unwrap it from `connLockError` (Charlie Tonneslan)
* pgxpool: check if connection is expired before acquire (arthurdotwork)
## Security Hardening
* Encrypt `CancelRequest` connection when the primary connection used TLS (Sean Chittenden at CrowdStrike, Inc.)
* Cap server-supplied SCRAM iteration count (Sean Chittenden at CrowdStrike, Inc.)
* Default Frontend max message body length to ~1 GiB (Sean Chittenden at CrowdStrike, Inc.)
* Bound hstore binary decode against malicious server input (Sean Chittenden at CrowdStrike, Inc.)
* Bound array binary decode element length against remaining message bytes (Sean Chittenden at CrowdStrike, Inc.)
* Bound array element count against remaining message bytes (Sean Chittenden at CrowdStrike, Inc.)
* Bound range, multirange, and tsvector binary decoders (Sean Chittenden at CrowdStrike, Inc.)
* Document secure connection configuration (Sean Chittenden at CrowdStrike, Inc.)
* Fix panic on malformed geometric text; return an error instead (MaIII)
## Fixes
* Fix scanning `"char"` (OID 18) into `*string` in binary format (luongs3)
* Fix handling of typed-nil `driver.Valuer` in array and composite codecs (Donncha Fahy)
* Fix `CopyData.Data` hex decoding in `UnmarshalJSON` (Charlie Tonneslan)
* Fix data race when context is cancelled during connect
* Fix `parseKeywordValueSettings` rejecting trailing whitespace (alliasgher)
* pgconn: preserve full error chain in `normalizeTimeoutError` (Charlie Tonneslan)
* pgconn: use a fresh context for the fallback connection in `connectPreferred` (Charlie Tonneslan)
* pgxpool: fix `MaxLifetimeDestroyCount` and ping order for acquire-time expiry check
* Add missing error check of `rows.Err` to load types (Jen Altavilla)
# 5.9.2 (April 18, 2026)
Fix SQL Injection via placeholder confusion with dollar quoted string literals (GHSA-j88v-2chj-qfwx)
+43 -25
View File
@@ -8,41 +8,59 @@ pgx is a PostgreSQL driver and toolkit for Go (`github.com/jackc/pgx/v5`). It pr
## Build & Test Commands
Every checkout has its own PostgreSQL 14-18 and CockroachDB instances, supervised by
process-compose. `mise run dev` starts PostgreSQL 18; tests start other targets on demand and stop
them afterwards unless they were explicitly prewarmed. See DEVELOPMENT.md.
```bash
# Run all tests (requires PGX_TEST_DATABASE to be set)
go test ./...
mise run dev # start PostgreSQL 18 and the database supervisor
mise run dev:all # eagerly start every available database
mise run dev -- -D # ... detached; then `mise run dev:wait`, and
# `mise run dev:down` when finished. Agents must do this.
# Run a specific test
go test -run TestFunctionName ./...
./test.sh # Full suite against PostgreSQL 18 (the default target)
./test.sh pg16 # Against PostgreSQL 16
./test.sh crdb # Against CockroachDB
./test.sh all # Every target (pg14-18 + crdb)
./test.sh pg16 -run TestConnect # Trailing arguments are passed to `go test`
# Run tests for a specific package
go test ./pgconn/...
go test ./... # Also works: mise loads the default target's PGX_TEST_*
go test -race ./... # With the race detector
# Run tests with race detector
go test -race ./...
goimports -w . # Format (always run after making changes)
golangci-lint run ./... # Lint
# DevContainer: run tests against specific PostgreSQL versions
./test.sh pg18 # Default: PostgreSQL 18
./test.sh pg16 -run TestConnect # Specific test against PG16
./test.sh crdb # CockroachDB
./test.sh all # All targets (pg14-18 + crdb)
# Format (always run after making changes)
goimports -w .
# Lint
golangci-lint run ./...
mise run dev:ports # This checkout's ports and where each server's data lives
mise run db:start pg16 crdb # Prewarm targets; tests then leave them running
mise run db:stop pg16 crdb # Stop prewarmed targets
mise run db:psql # psql against PostgreSQL 18; `mise run db:psql 16` for another
process-compose process logs pg16 # One server's output
```
Do not hardcode database ports. They are allocated per checkout by port-tamer and read from the
environment (`PGPORT`, `PGPORT_16`, `CRDB_PORT`) or `.dev/ports.env`; 5432 and 26257 mean nothing
here.
The `PGX_TEST_*` connection strings have one definition, `scripts/lib/test_targets.rb`. Add or
change a target there, never in a second copy.
## Test Database Setup
Tests require `PGX_TEST_DATABASE` environment variable. In the devcontainer, `test.sh` handles this. For local development:
The lifecycle scripts handle setup: a PostgreSQL server initializes its cluster on first start and
creates `pgx_test` with the extensions and auth roles from `testsetup/postgresql_setup.sql`.
CockroachDB recreates `pgx_test` after each in-memory restart. Nothing needs to be set up by hand.
```bash
export PGX_TEST_DATABASE="host=localhost user=postgres password=postgres dbname=pgx_test"
```
Contributors who would rather point pgx at a PostgreSQL server they already have can set
`PGX_TEST_DATABASE` themselves; see CONTRIBUTING.md. Many tests are skipped unless additional
`PGX_TEST_*` variables are set (for TLS, SCRAM, MD5, unix socket, PgBouncer testing).
The test database needs extensions: `hstore`, `ltree`, and a `uint64` domain. See `testsetup/postgresql_setup.sql` for full setup. Many tests are skipped unless additional `PGX_TEST_*` env vars are set (for TLS, SCRAM, MD5, unix socket, PgBouncer testing).
## Reference Material
`references/` holds read-only reference checkouts used when building pgx — currently the PostgreSQL source tree pinned to `REL_18_STABLE`. It is gitignored and provisioned on demand: bare mirrors are cached at a machine-level path (`/persist/shared/references` in a devcontainer, `~/.local/share/pgx/references` natively; `REFERENCES_MIRROR_DIR` overrides) and lightweight local checkouts are created in `references/` with `rake references:setup`. Each checkout has per-instance Git metadata while borrowing the shared mirror's object store. Related tasks: `rake references:update`, `rake references:status`, `rake references:clean`.
- Do not automatically provision or update `references/`.
- Never run `rake references:setup`, `rake references:update`, or any large download on your own initiative.
- If reference sources are missing, work without them or ask the user.
## Architecture
@@ -69,5 +87,5 @@ Supporting packages:
- **Context-based** — all blocking operations take `context.Context`.
- **Tracer interfaces** — observability via `QueryTracer`, `BatchTracer`, `CopyFromTracer`, `PrepareTracer` on `ConnConfig.Tracer`.
- **Formatting** — always run `goimports -w .` after making changes to ensure code is properly formatted. CI checks formatting via `gofmt -l -s -w . && git diff --exit-code`. `gofumpt` with extra rules is also enforced via `golangci-lint`.
- **Linters** — `govet` and `ineffassign` only (configured in `.golangci.yml`).
- **Linters** — `govet`, `ineffassign`, and `unconvert` only (configured in `.golangci.yml`).
- **CI matrix** — tests run against Go 1.25/1.26 × PostgreSQL 14-18 + CockroachDB, on Linux and Windows. Race detector enabled on Linux only.
+29 -76
View File
@@ -24,21 +24,30 @@ Using AI is acceptable (not that it can really be stopped) under one the followi
## Development Environment Setup
pgx tests naturally require a PostgreSQL database. It will connect to the database specified in the `PGX_TEST_DATABASE`
environment variable. The `PGX_TEST_DATABASE` environment variable can either be a URL or key-value pairs. In addition,
the standard `PG*` environment variables will be respected. Consider using [direnv](https://github.com/direnv/direnv) to
simplify environment variable handling.
pgx tests naturally require a PostgreSQL database. It will connect to the database specified in the
`PGX_TEST_DATABASE` environment variable. The `PGX_TEST_DATABASE` environment variable can either be
a URL or key-value pairs. In addition, the standard `PG*` environment variables will be respected.
### Devcontainer
### The full environment
The easiest way to start development is with the included devcontainer. It includes containers for each supported
PostgreSQL version as well as CockroachDB. `./test.sh all` will run the tests against all database types.
[DEVELOPMENT.md](DEVELOPMENT.md) describes the maintained setup: [mise](https://mise.jdx.dev)
installs the toolchain, and each checkout has its own PostgreSQL 14-18 clusters and CockroachDB
node, so the whole test matrix is available locally. PostgreSQL 18 stays running; other servers
start and stop around their tests. It works natively on macOS and Linux, and in the included
devcontainer.
### Using an Existing PostgreSQL Cluster Outside of a Devcontainer
```
mise install
mise run dev:init
mise run dev
./test.sh all
```
If you already have a PostgreSQL development server this is the quickest way to start and run the majority of the pgx
test suite. Some tests will be skipped that require server configuration changes (e.g. those testing different
authentication methods).
### Using an existing PostgreSQL cluster
If you already have a PostgreSQL development server this is the quickest way to run the majority of
the pgx test suite, and it needs nothing from the section above. Some tests will be skipped that
require server configuration changes (e.g. those testing different authentication methods).
Create and setup a test database:
@@ -50,86 +59,30 @@ psql -c 'create extension ltree;'
psql -c 'create domain uint64 as numeric(20,0);'
```
Ensure a `postgres` user exists. This happens by default in normal PostgreSQL installs, but some installation methods
such as Homebrew do not.
Ensure a `postgres` user exists. This happens by default in normal PostgreSQL installs, but some
installation methods such as Homebrew do not.
```
createuser -s postgres
```
Ensure your `PGX_TEST_DATABASE` environment variable points to the database you just created and run the tests.
Ensure your `PGX_TEST_DATABASE` environment variable points to the database you just created and run
the tests.
```
export PGX_TEST_DATABASE="host=/private/tmp database=pgx_test"
go test ./...
```
This will run the vast majority of the tests, but some tests will be skipped (e.g. those testing different connection methods).
### Creating a New PostgreSQL Cluster Exclusively for Testing Outside of a Devcontainer
The following environment variables need to be set both for initial setup and whenever the tests are run. (direnv is
highly recommended). Depending on your platform, you may need to change the host for `PGX_TEST_UNIX_SOCKET_CONN_STRING`.
```
export PGPORT=5015
export PGUSER=postgres
export PGDATABASE=pgx_test
export POSTGRESQL_DATA_DIR=postgresql
export PGX_TEST_DATABASE="host=127.0.0.1 database=pgx_test user=pgx_md5 password=secret"
export PGX_TEST_UNIX_SOCKET_CONN_STRING="host=/private/tmp database=pgx_test"
export PGX_TEST_TCP_CONN_STRING="host=127.0.0.1 database=pgx_test user=pgx_md5 password=secret"
export PGX_TEST_SCRAM_PASSWORD_CONN_STRING="host=127.0.0.1 user=pgx_scram password=secret database=pgx_test channel_binding=disable"
export PGX_TEST_SCRAM_PLUS_CONN_STRING="host=localhost user=pgx_ssl password=secret sslmode=verify-full sslrootcert=`pwd`/.testdb/ca.pem database=pgx_test channel_binding=require"
export PGX_TEST_MD5_PASSWORD_CONN_STRING="host=127.0.0.1 database=pgx_test user=pgx_md5 password=secret"
export PGX_TEST_PLAIN_PASSWORD_CONN_STRING="host=127.0.0.1 user=pgx_pw password=secret"
export PGX_TEST_TLS_CONN_STRING="host=localhost user=pgx_ssl password=secret sslmode=verify-full sslrootcert=`pwd`/.testdb/ca.pem channel_binding=disable"
export PGX_SSL_PASSWORD=certpw
export PGX_TEST_TLS_CLIENT_CONN_STRING="host=localhost user=pgx_sslcert sslmode=verify-full sslrootcert=`pwd`/.testdb/ca.pem database=pgx_test sslcert=`pwd`/.testdb/pgx_sslcert.crt sslkey=`pwd`/.testdb/pgx_sslcert.key"
```
Create a new database cluster.
```
initdb --locale=en_US -E UTF-8 --username=postgres .testdb/$POSTGRESQL_DATA_DIR
echo "listen_addresses = '127.0.0.1'" >> .testdb/$POSTGRESQL_DATA_DIR/postgresql.conf
echo "port = $PGPORT" >> .testdb/$POSTGRESQL_DATA_DIR/postgresql.conf
cat testsetup/postgresql_ssl.conf >> .testdb/$POSTGRESQL_DATA_DIR/postgresql.conf
cp testsetup/pg_hba.conf .testdb/$POSTGRESQL_DATA_DIR/pg_hba.conf
cd .testdb
# Generate CA, server, and encrypted client certificates.
go run ../testsetup/generate_certs.go
# Copy certificates to server directory and set permissions.
cp ca.pem $POSTGRESQL_DATA_DIR/root.crt
cp localhost.key $POSTGRESQL_DATA_DIR/server.key
chmod 600 $POSTGRESQL_DATA_DIR/server.key
cp localhost.crt $POSTGRESQL_DATA_DIR/server.crt
cd ..
```
Start the new cluster. This will be necessary whenever you are running pgx tests.
```
postgres -D .testdb/$POSTGRESQL_DATA_DIR
```
Setup the test database in the new cluster.
```
createdb
psql --no-psqlrc -f testsetup/postgresql_setup.sql
```
This will run the vast majority of the tests, but some tests will be skipped (e.g. those testing
different connection methods).
### PgBouncer
There are tests specific for PgBouncer that will be executed if `PGX_TEST_PGBOUNCER_CONN_STRING` is set.
The test PgBouncer must be version 1.21.0 or newer, use transaction pooling, and have `max_prepared_statements` set to a
non-zero value. This ensures the tests cover PgBouncer's protocol-level named prepared statement support in addition to
the pgx query modes that do not use named prepared statements.
### Optional Tests
+239
View File
@@ -0,0 +1,239 @@
# Developing pgx
Two supported environments, one command set. Native macOS or Linux is the fast inner loop; the
devcontainer is a Linux compatibility and isolation option. Both run the same mise tasks against
the same `process-compose.yaml`, so nothing here is specific to one of them except the
prerequisites in §1.
```sh
scripts/setup-host # native macOS or Ubuntu host prerequisites (once per machine)
export PATH="$HOME/.local/bin:$PATH" # if mise was just installed
mise trust # trust this checkout's configuration
mise install # tool versions from mise.toml
mise run dev:init # this checkout's ports and certificates
mise run dev # start PostgreSQL 18 and the on-demand database supervisor
./test.sh # the suite against PostgreSQL 18
./test.sh all # every target; non-default servers start and stop around their tests
```
---
## 1. Prerequisites
### Native
[mise](https://mise.jdx.dev) provides Go, Ruby, CockroachDB, process-compose, and port-tamer.
PostgreSQL itself is the one thing it does not: pgx tests against five major versions, and they
come from the system package manager.
```sh
scripts/setup-host
```
On macOS, install [Homebrew](https://brew.sh) first. The script uses `Brewfile` to install
PostgreSQL 14-18 and Ruby build dependencies. On Ubuntu, it uses sudo when needed, configures
the official PostgreSQL apt repository, and installs the same server versions and build
dependencies. Other Linux distributions require manual prerequisite installation.
The Ubuntu installer temporarily disables automatic `main` cluster creation during server
installation and removes that override on exit, leaving the package-owned cluster configuration
unchanged. The macOS installer does not start Homebrew services.
The dispatcher installs mise if it is missing and prints its executable path. It does not run
`mise trust`, install project tools, allocate ports, initialize databases, or start servers. Run
the remaining commands above explicitly as your regular development user.
- **Do not** `brew services start` any of them, and on Debian do not let the packages create a
machine-wide cluster. pgx runs its own clusters per checkout.
- The Homebrew formulas are *keg-only*, so `psql` and `pg_isready` are not on your `PATH` after
install. `mise.toml` adds PostgreSQL 18's `bin` to `PATH` for this project — one client serves
every server, since libpq is backward compatible.
- Only the majors you actually test against need to be installed. `mise run dev` looks for each
one's server binaries and names any that are missing; `./test.sh pg15` reports a direct install
hint rather than trying to launch a missing server. `PGBIN_16` and friends override the search
for a major built or installed somewhere unusual.
- Clusters are created with the `en_US.UTF-8` locale where the system has it (matching CI and the
container images this replaced), falling back to `C.UTF-8` and then `C`. A stock Debian or Ubuntu
has only `C.UTF-8` unless you have run `locale-gen en_US.UTF-8`.
### Devcontainer
Reopen in the container; `.devcontainer/` handles the rest. It installs the same five PostgreSQL
servers and runs the same per-checkout clusters — it is a Linux shell around this same setup, not a
second architecture.
---
## 2. Checkouts are the unit of isolation
A git worktree is the native equivalent of a second devcontainer instance. Each one gets:
```
.dev/ # gitignored, per-checkout runtime state
ports.env # this checkout's TCP ports (port-tamer's state file)
derived.env # PG* defaults and the default target's PGX_TEST_* set
certs/ # the client certificates the TLS tests use
logs/ # one log per service
<os>-<arch>/postgres/run # one socket directory, shared by all five servers (mode 0700)
<os>-<arch>/postgres/<major>/data
<os>-<arch>/crdb/
```
The last three — the clusters, the CockroachDB store and the socket directory — move when
`PGX_DEV_RUNTIME_DIR` is set. The devcontainer sets it to `/persist/local/dev`, on its own named
volume: `.dev/` is inside the `/workspaces/pgx` bind mount, and `initdb` and Unix sockets do not
work reliably on Docker Desktop's macOS/Windows file sharing. (The per-version PostgreSQL
containers this setup replaced used named volumes for their data and socket directories for the
same reason.) Everything else stays in the checkout.
```sh
git worktree add ../pgx-feature-x feature-x
cd ../pgx-feature-x
mise run dev:init && mise run dev
```
Both checkouts run simultaneously: different ports, independent database state, independent
process-compose instances. `mise run dev:ports` prints the allocation.
Ports are allocated once per checkout by [port-tamer](https://github.com/jackc/port-tamer) and then
persisted. `port-tamer.toml` declares which ports a checkout needs — **append new entries at the
end**, since inserting or reordering renumbers the existing ones. A listening port never moves an
existing allocation, because it may well belong to this checkout's own running services; when two
checkouts genuinely collide, stop one and run `mise run dev:ports:overwrite`.
The data directories are keyed by platform because a cluster `initdb`'d by the Linux devcontainer
cannot be read by a native macOS server, and one checkout may be opened both ways. After switching
a checkout between the two, re-run `mise run dev:ports:ensure` so the derived paths follow.
**Reference sources** (`references/`) are provisioned per checkout with `rake references:setup`,
sharing one machine-level mirror (`/persist/shared` in a container, `~/.local/share/pgx` natively;
`REFERENCES_MIRROR_DIR` overrides). They are a multi-GB download; nothing provisions them
automatically.
---
## 3. The databases
`mise run dev` starts PostgreSQL 18 under
[process-compose](https://github.com/F1bonacc1/process-compose). PostgreSQL 14-17 and a single-node
in-memory CockroachDB are registered with the same supervisor but disabled initially, ready to be
started by a test or an explicit `db:start`. `mise run dev:all` eagerly starts every available
server. These are the same services the devcontainer used to run as containers; they are now
ordinary processes against this checkout's own clusters.
Each PostgreSQL server initializes its cluster on first start — `initdb`, this project's
`pg_hba.conf`, and the TLS certificates — then creates `pgx_test` with its extensions and its
`pgx_md5` / `pgx_scram` / `pgx_pw` / `pgx_ssl` / `pgx_sslcert` roles. The default server uses its
follow-on `pg18-setup` process; the lazy lifecycle performs the same idempotent setup directly for
an on-demand target. Restarts are free once the cluster exists. If setup fails partway it drops the
half-built database so the next start retries rather than reporting it as present.
CockroachDB has no separate setup process: its store is in memory, so every restart is an empty
cluster and its readiness probe creates `pgx_test` itself.
```sh
process-compose process list # status, scriptable
process-compose process logs pg16 # one server's output
mise run db:start pg16 crdb # prewarm one or more targets
mise run db:stop pg16 crdb # stop them again
mise run db:start all # prewarm every target
process-compose down # stop this checkout's stack only
mise run db:psql # psql against the already-running PostgreSQL 18
mise run db:start pg16 # prewarm PostgreSQL 16 before interactive use
mise run db:psql 16 # psql against PostgreSQL 16
mise run db:psql 16 -c 'select 1' # arguments after the major go to psql
```
The `process-compose` commands need no flags: `PC_PORT_NUM` is part of this checkout's
environment, so they never reach another checkout's stack.
`rake db:psql[16]` is the same as `mise run db:psql 16`. Note the brackets — a bare
`rake db:psql 16` is rake asking for a *task* named `16`, not an argument, so the `db:*` tasks
reject it rather than quietly acting on every cluster.
`mise run db:reset` destroys and re-creates clusters and refuses to run while the selected servers
are up: removing a data directory under a live postmaster corrupts it. Stop selected targets with
`mise run db:stop`, or stop the whole supervisor with `mise run dev:down`.
All five PostgreSQL servers share **one** Unix socket directory. Sockets are named
`.s.PGSQL.<port>`, so the port picks the server — which is what lets a single
`PGX_TEST_UNIX_SOCKET_CONN_STRING` work for every major, exactly as the container's shared
`/var/run/postgresql` volume did.
Tests preserve explicit choices: if `db:start` prewarmed a target, tests leave it running. If a
test had to start the target, it stops it in an `ensure` block whether the suite passes, fails, or
is interrupted. A per-target lock prevents concurrent test commands from stopping a server the
other is using.
---
## 4. Running tests
```sh
./test.sh # PostgreSQL 18, the default target
./test.sh pg14 # PostgreSQL 14
./test.sh crdb # CockroachDB
./test.sh all # every target, sequentially
./test.sh pg16 -run TestConnect # trailing arguments go to `go test`
```
`mise run test` and `mise run test:all` are equivalent. All of them require the stack to be
running, but only their selected database target needs to be up. A stopped target is started,
bootstrapped, and stopped automatically. `./test.sh all` does this sequentially, so at most the
default PostgreSQL 18 plus one additional database is normally resident.
A bare `go test ./...` also works: mise loads `.dev/derived.env`, which carries the default
target's full `PGX_TEST_*` set, so an activated shell is already pointed at PostgreSQL 18.
Those connection strings have exactly one definition, `scripts/lib/test_targets.rb`. Both consumers
— the generated `.dev/derived.env` and `./test.sh <target>` — read it, so there is no second copy
to drift.
Some tests only run when their environment variable is set. `go test ./... -v | grep SKIP` shows
what is being skipped; on a healthy stack that is the PgBouncer, OAuth, CrateDB, and libpq-oracle
tests, which are CI-only or manual.
---
## 5. Everyday commands
| Command | What it does |
|---|---|
| `mise run dev` | start PostgreSQL 18 and the on-demand database supervisor |
| `mise run dev:all` | eagerly start every available database |
| `mise run dev -- -D` | start the default stack detached, for CI and agents |
| `mise run dev:wait` / `dev:down` | wait for a detached stack / stop it |
| `mise run dev:init` | allocate this checkout's ports, decode its certificates |
| `mise run dev:ports` | the allocation, and where each server's data lives |
| `mise run test [target]` | the suite against one target |
| `mise run test:all` | every target |
| `mise run db:start [targets...]` / `db:stop` | prewarm or stop targets (`pg16`, `crdb`, or `all`) |
| `mise run db:init [major]` / `db:psql [major]` / `db:reset [major]` | create / open / rebuild the clusters |
| `mise run fmt` | `goimports -w .` |
| `mise run generate` | regenerate the ERB-templated sources |
`rake <task>` remains equally valid and is where some of the logic lives; `rake -T` lists
everything, including tasks with no mise wrapper (`references:*`, `db:setup`).
---
## 6. Notes
- **`PGHOST` and `PGPORT` are a pair.** They come from `.dev/` via mise — `PGPORT` from the
allocation, `PGHOST` derived from it. Setting one without the other names a real port on the
wrong server, and the error will point at a socket nothing ever created. `mise run dev` asserts
the two agree with the cluster before starting anything.
- **`.dev/derived.env` is generated.** If it is ever hand-edited into something mise's dotenv
parser rejects, every `mise` command in the directory fails — including the one that would
rewrite it. Recover with `rm .dev/derived.env && mise run dev:ports:ensure`.
- **A `~/.psqlrc` that changes session defaults** can break the database bootstrap while leaving
interactive use fine. The scripts pass `--no-psqlrc` for that reason.
- **`./test.sh` needs Ruby 3.0+**, which `mise install` provides. If mise is installed but not yet
activated in your shell, `./test.sh` re-runs itself through `mise exec` rather than failing on
the system interpreter.
- **CI does not use any of this.** `.github/workflows/ci.yml` installs system PostgreSQL through
`ci/setup_test.bash` and carries its own copy of the connection strings. Local runs and CI
exercise the same auth paths — `PGX_TEST_DATABASE` connects as `pgx_md5`, and one
`testsetup/pg_hba.conf` serves both — but the two setups are still independent.
- **PgBouncer and the PG18 OAuth validator module** are exercised only in CI. Neither is part of
the local stack, and both were absent from the devcontainer too.
+44 -4
View File
@@ -12,7 +12,15 @@ The toolkit component is a related set of packages that implement PostgreSQL fun
and type mapping between PostgreSQL and Go. These underlying packages can be used to implement alternative drivers,
proxies, load balancers, logical replication clients, etc.
## Example Usage
## Quick Start
### Installation
```bash
go get github.com/jackc/pgx/v5
```
### Example Usage
```go
package main
@@ -46,7 +54,18 @@ func main() {
}
```
See the [getting started guide](https://github.com/jackc/pgx/wiki/Getting-started-with-pgx) for more information.
### Connection Configuration
`pgx.Connect` and `pgxpool.New` accept PostgreSQL connection URLs (such as `postgres://user:pass@host:5432/db?sslmode=verify-full`) as well as `key=value` strings. See [`pgconn.ParseConfig`](https://pkg.go.dev/github.com/jackc/pgx/v5/pgconn#ParseConfig) and the [PostgreSQL connection string documentation](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING) for supported options and environment variables.
For a step-by-step walkthrough, see the [getting started guide](https://github.com/jackc/pgx/wiki/Getting-started-with-pgx).
## Documentation
Package documentation and API reference are available on [pkg.go.dev](https://pkg.go.dev/github.com/jackc/pgx/v5):
* [`pgx`](https://pkg.go.dev/github.com/jackc/pgx/v5) — base PostgreSQL driver
* [`pgxpool`](https://pkg.go.dev/github.com/jackc/pgx/v5/pgxpool) — concurrency-safe connection pool
* [`stdlib`](https://pkg.go.dev/github.com/jackc/pgx/v5/stdlib) — `database/sql` compatibility adapter
## Features
@@ -82,9 +101,30 @@ The pgx interface is recommended when:
It is also possible to use the `database/sql` interface and convert a connection to the lower-level pgx interface as needed.
## Testing
## Development and Testing
See [CONTRIBUTING.md](./CONTRIBUTING.md) for setup instructions.
Each checkout has its own PostgreSQL 14-18 clusters and CockroachDB node, so the whole test matrix
is available locally on macOS, on Linux, or in the included devcontainer. Only PostgreSQL 18 stays
running by default; other targets start and stop around their tests:
```sh
scripts/setup-host # native macOS (Homebrew required) or Ubuntu: host packages and mise
export PATH="$HOME/.local/bin:$PATH" # if mise was just installed
mise trust
mise install # project tools
mise run dev:init # checkout ports and certificates
mise run dev # start PostgreSQL 18 and the on-demand database supervisor
./test.sh # the suite against PostgreSQL 18
./test.sh all # every target, starting and stopping each server as needed
```
Host setup installs PostgreSQL 14-18 and Ruby build dependencies. It does not install project
tools or initialize the checkout; those remain separate steps above. The devcontainer already
provides the host prerequisites.
See [DEVELOPMENT.md](./DEVELOPMENT.md) for the full setup, and
[CONTRIBUTING.md](./CONTRIBUTING.md) for how to contribute — including how to run the tests against
a PostgreSQL server you already have, without any of the above.
## Architecture
+291
View File
@@ -1,4 +1,5 @@
require "erb"
require "fileutils"
rule '.go' => '.go.erb' do |task|
erb = ERB.new(File.read(task.source))
@@ -16,3 +17,293 @@ generated_code_files = [
desc "Generate code"
task generate: generated_code_files
# dev -- per-checkout development state. A git worktree is the native equivalent of a second
# devcontainer instance: same machine, same network stack, so the five PostgreSQL servers and
# CockroachDB that each owned a well-known port in their own container namespace now need distinct
# ones. `dev:ports:*` drives port-tamer (port-tamer.toml declares the names; the allocation lands
# in the gitignored .dev/ports.env) and writes the values derived from it -- scripts/devenv.rb.
namespace :dev do
namespace :ports do
desc "Allocate this checkout's TCP ports if it has none (idempotent)"
task :ensure do
sh RbConfig.ruby, "scripts/devenv.rb", "ensure"
end
desc "Move this checkout to a different port group (stop its services first)"
task :overwrite do
sh RbConfig.ruby, "scripts/devenv.rb", "overwrite"
end
desc "Print this checkout's port allocation and where each server's data lives"
task :show do
sh RbConfig.ruby, "scripts/devenv.rb", "show"
end
end
end
# db -- this checkout's own PostgreSQL clusters, one per major (scripts/devdb.rb). Clusters per
# checkout rather than databases inside shared servers: destructive resets stay local, two
# checkouts run at once, and the layout matches the per-instance devcontainer model it replaces.
# The SERVERS run in the foreground under process-compose (process-compose.yaml); these are the
# one-shot half. Each task takes an optional major, defaulting to all of them.
#
# rake db:init initialize every cluster
# rake db:init[16] just PostgreSQL 16
# rake db:psql[16] psql against PostgreSQL 16 (default: the newest major)
#
# The mise wrappers (`mise run db:init 16`) call scripts/devdb.rb directly rather than routing
# through these tasks — mise APPENDS its arguments, and `rake db:init 16` means "run task db:init,
# then run task 16", not "run db:init with the argument 16".
namespace :db do
# A bare major on the command line is the natural thing to type and the one rake cannot mean:
# it becomes a second TASK name, so `rake db:reset 16` runs db:reset with no major — every
# cluster — and only then fails on the unknown task, after the data is gone. Catch it first.
def db_major(task, args)
stray = Rake.application.top_level_tasks.find { |t| t =~ /\A\d+\z/ }
if stray
abort "rake: `#{stray}` is being read as a task name, not an argument. " \
"Write `rake #{task.name}[#{stray}]` (or `mise run #{task.name} #{stray}`)."
end
Array(args[:major])
end
desc "Create this checkout's PostgreSQL clusters under .dev (idempotent). Optional: rake db:init[16]"
task :init, [:major] do |task, args|
sh RbConfig.ruby, "scripts/devdb.rb", "init", *db_major(task, args)
end
desc "psql against one of this checkout's clusters. Optional: rake db:psql[16]"
task :psql, [:major] do |task, args|
sh RbConfig.ruby, "scripts/devdb.rb", "psql", *db_major(task, args)
end
desc "Create the pgx_test database and roles in a running cluster (idempotent)"
task :setup, [:major] do |task, args|
sh RbConfig.ruby, "scripts/devdb.rb", "setup", *db_major(task, args)
end
desc "Destroy this checkout's clusters and re-initialize (destructive; CONFIRM=yes)"
task :reset, [:major] do |task, args|
sh RbConfig.ruby, "scripts/devdb.rb", "reset", *db_major(task, args)
end
end
# references:* — provision local, read-only checkouts of reference material used
# when building pgx (currently the PostgreSQL source tree).
#
# Storage model:
# * A bare `--mirror` clone of each repo lives on the devcontainer's shared
# persist volume under MIRROR_ROOT. It holds the full history + all
# branches/tags, is downloaded once, and survives container rebuilds. It is
# the canonical copy, shared across every container for this project.
# * Each container creates a lightweight shared clone in ./references/<name>.
# Its Git metadata is local to that checkout while its object store borrows
# from the mirror. Containers can therefore use the same in-container path
# without sharing Git worktree registrations or downloading history again.
#
# Provisioning a new container is therefore cheap: a local clone from the
# already-present mirror, with no network fetch or duplicate object store.
# Each entry is one reference repo. `ref` is the branch/tag checked out into the
# local checkout. PostgreSQL is pinned to REL_18_STABLE to match the devcontainer's
# default PG18 test target.
REFERENCE_REPOS = [
{ name: "postgres", url: "https://github.com/postgres/postgres.git", ref: "REL_18_STABLE", license: "PostgreSQL License" },
].freeze
# Where the canonical bare mirrors live. They are machine-level state, not checkout-level:
# multiple GB, read-only, and identical for every checkout -- so they belong outside the tree, and
# every checkout on the machine shares one copy.
#
# * devcontainer -- the shared persist volume, so every container for this project reuses one
# download and a rebuild costs nothing.
# * native -- an XDG data directory under $HOME, where git worktrees of this repo share it the
# same way containers share the volume.
#
# REFERENCES_MIRROR_DIR overrides both (a second disk, a scratch location, a test).
def default_mirror_root
return "/persist/shared/references" if File.directory?("/persist/shared")
xdg = ENV["XDG_DATA_HOME"]
base = xdg.nil? || xdg.empty? ? File.join(Dir.home, ".local", "share") : xdg
File.join(base, "pgx", "references")
end
MIRROR_ROOT = ENV.fetch("REFERENCES_MIRROR_DIR") { default_mirror_root }
CHECKOUT_ROOT = File.join(__dir__, "references")
def mirror_path(repo) = File.expand_path(File.join(MIRROR_ROOT, "#{repo[:name]}.git"))
def checkout_path(repo) = File.join(CHECKOUT_ROOT, repo[:name])
# File.exist? is false for dangling symlinks, which are still leftovers setup
# must replace (and clean must remove).
def path_present?(path) = File.exist?(path) || File.symlink?(path)
# Run a git command against a bare mirror. Names the gitdir explicitly and lifts
# any safe.bareRepository=explicit guard. Raises on failure.
def git_bare(repo, *args)
sh "git", "-c", "safe.bareRepository=all", "--git-dir", mirror_path(repo), *args
end
# Capture stdout of a command given as an argv array (no shell parsing, no
# quoting pitfalls). Returns [stdout_string, success_boolean].
def capture(*args)
out = IO.popen(args, err: File::NULL, &:read)
[out.to_s, $?.success?]
end
# A managed checkout has its own repository metadata, uses this exact mirror as
# its origin, and borrows that mirror's object database. This rejects ordinary
# clones, linked worktrees left by the old implementation, and dangling gitfiles.
def checkout_valid?(repo)
wp = checkout_path(repo)
git_dir = File.join(wp, ".git")
objects_dir = File.join(git_dir, "objects")
alternates_path = File.join(objects_dir, "info", "alternates")
return false unless File.directory?(git_dir) && File.file?(alternates_path)
inside, inside_ok = capture("git", "-C", wp, "rev-parse", "--is-inside-work-tree")
origin, origin_ok = capture("git", "-C", wp, "remote", "get-url", "origin")
return false unless inside_ok && inside.strip == "true" && origin_ok
return false unless File.identical?(File.expand_path(origin.strip, wp), mirror_path(repo))
File.foreach(alternates_path).any? do |alternate|
alternate = alternate.strip
next false if alternate.empty?
File.identical?(File.expand_path(alternate, objects_dir), File.join(mirror_path(repo), "objects"))
end
rescue SystemCallError
false
end
# A mirror is valid only if it exists AND has at least one ref — this rejects a
# directory left behind by an interrupted clone (which exists but is incomplete).
def mirror_valid?(repo)
return false unless File.directory?(mirror_path(repo))
out, ok = capture("git", "-c", "safe.bareRepository=all", "--git-dir", mirror_path(repo), "for-each-ref", "--count=1")
ok && !out.strip.empty?
end
# Clone the bare mirror onto the persist volume if it is missing or broken.
def ensure_mirror(repo)
if mirror_valid?(repo)
puts " mirror cached: #{mirror_path(repo)}"
else
FileUtils.rm_rf(mirror_path(repo)) # clear any partial/broken clone
puts " cloning mirror (full history): #{repo[:url]}"
sh "git", "clone", "--mirror", repo[:url], mirror_path(repo)
end
end
# Serialize mirror creation and updates across devcontainer instances. Local
# checkouts do not share metadata, but they all read from the same object store.
def with_mirror_lock(repo)
FileUtils.mkdir_p(MIRROR_ROOT)
File.open(File.join(MIRROR_ROOT, ".#{repo[:name]}.lock"), File::RDWR | File::CREAT, 0o644) do |lock|
lock.flock(File::LOCK_EX)
yield
end
end
# Check out (or re-point) the local clone at the configured ref. Resolve the
# commit in the freshly updated mirror so an existing checkout cannot remain
# stale merely because only the mirror was fetched.
def ensure_checkout(repo)
wp = checkout_path(repo)
ref = repo[:ref]
commit, ok = capture("git", "-c", "safe.bareRepository=all", "--git-dir", mirror_path(repo),
"rev-parse", "--verify", "#{ref}^{commit}")
raise "ref #{ref.inspect} not found in #{mirror_path(repo)}" unless ok && !commit.strip.empty?
if checkout_valid?(repo)
puts " checkout present: #{wp} -> #{ref}"
else
puts " replacing invalid checkout: #{wp}" if path_present?(wp)
FileUtils.rm_rf(wp) if path_present?(wp)
puts " cloning checkout: #{wp} -> #{ref}"
sh "git", "clone", "--shared", "--no-checkout", mirror_path(repo), wp
raise "checkout at #{wp} is not linked to #{mirror_path(repo)}" unless checkout_valid?(repo)
end
sh "git", "-C", wp, "checkout", "--detach", commit.strip
end
# Run a block per repo, collecting failures so one bad repo does not abort the rest.
def for_each_repo
failures = []
REFERENCE_REPOS.each do |repo|
puts "#{repo[:name]}:"
begin
yield repo
rescue => e
warn " FAILED: #{e.message}"
failures << repo[:name]
end
end
abort "references: failed for #{failures.join(', ')}" unless failures.empty?
end
namespace :references do
desc "Clone/refresh reference mirrors on persist and create checkouts in references/"
task :setup do
FileUtils.mkdir_p(MIRROR_ROOT)
FileUtils.mkdir_p(CHECKOUT_ROOT)
for_each_repo do |repo|
with_mirror_lock(repo) do
ensure_mirror(repo)
ensure_checkout(repo)
end
end
puts
puts "Done. Reference sources are in #{CHECKOUT_ROOT}"
Rake::Task["references:status"].invoke
end
desc "Fetch latest upstream for all mirrors and re-point checkouts"
task :update do
for_each_repo do |repo|
with_mirror_lock(repo) do
abort "mirror missing for #{repo[:name]}; run `rake references:setup`" unless mirror_valid?(repo)
git_bare(repo, "remote", "update", "--prune")
ensure_checkout(repo)
end
end
end
desc "Show provisioned reference repos, their pinned ref, and current HEAD"
task :status do
puts
puts format(" %-16s %-14s %-14s %s", "REPO", "REF", "HEAD", "LICENSE")
REFERENCE_REPOS.each do |repo|
wp = checkout_path(repo)
state =
if checkout_valid?(repo)
head, ok = capture("git", "-C", wp, "rev-parse", "--short", "HEAD")
ok && !head.strip.empty? ? head.strip : "(invalid)"
elsif path_present?(wp)
"(invalid)"
else
"(not set up)"
end
puts format(" %-16s %-14s %-14s %s", repo[:name], repo[:ref], state, repo[:license])
end
puts
puts " mirrors: #{MIRROR_ROOT}"
puts " checkouts: #{CHECKOUT_ROOT}"
end
desc "Remove checkouts from references/ (keeps the cached mirrors on persist)"
task :clean do
REFERENCE_REPOS.each do |repo|
wp = checkout_path(repo)
next unless path_present?(wp)
puts "removing checkout: #{wp}"
FileUtils.rm_rf(wp)
end
end
end
+28 -17
View File
@@ -36,9 +36,10 @@ type ConnConfig struct {
DescriptionCacheCapacity int
// DefaultQueryExecMode controls the default mode for executing queries. By default pgx uses the extended protocol
// and automatically prepares and caches prepared statements. However, this may be incompatible with proxies such as
// PGBouncer. In this case it may be preferable to use [QueryExecModeExec] or [QueryExecModeSimpleProtocol]. The same
// functionality can be controlled on a per query basis by passing a [QueryExecMode] as the first query argument.
// and automatically prepares and caches prepared statements. This may be incompatible with proxies such as PgBouncer
// unless they are configured to support protocol-level prepared statements. In an incompatible configuration it may
// be preferable to use [QueryExecModeExec] or [QueryExecModeSimpleProtocol]. The same functionality can be controlled
// on a per query basis by passing a [QueryExecMode] as the first query argument.
DefaultQueryExecMode QueryExecMode
createdByParseConfig bool // Used to enforce created by ParseConfig rule.
@@ -358,8 +359,13 @@ func (c *Conn) Prepare(ctx context.Context, name, sql string) (sd *pgconn.Statem
sd, err = c.pgConn.Prepare(ctx, psName, sql, nil)
if err != nil {
var pErr *pgconn.PrepareError
if errors.As(err, &pErr) {
c.failedDescribeStatement = psKey
if errors.As(err, &pErr) && pErr.ParseComplete {
// The server-side statement was created under psName — the name sent in
// Parse. In the name == sql case psKey is the SQL text, and deallocating
// by it would close a nonexistent statement while leaking the real one.
// When Parse never completed no statement was created at all, so there
// is nothing to clean up.
c.failedDescribeStatement = psName
}
return nil, err
}
@@ -475,6 +481,9 @@ func (c *Conn) Exec(ctx context.Context, sql string, arguments ...any) (pgconn.C
}
if err := c.deallocateInvalidatedCachedStatements(ctx); err != nil {
if c.queryTracer != nil {
c.queryTracer.TraceQueryEnd(ctx, c, TraceQueryEndData{Err: err})
}
return pgconn.CommandTag{}, err
}
@@ -742,7 +751,7 @@ type QueryRewriter interface {
// collected before processing rather than processed while receiving each row. This avoids the possibility of the
// application processing rows from a query that the server rejected. The CollectRows function is useful here.
//
// An implementor of QueryRewriter may be passed as the first element of args. It can rewrite the sql and change or
// An implementer of QueryRewriter may be passed as the first element of args. It can rewrite the sql and change or
// replace args. For example, NamedArgs is QueryRewriter that implements named arguments.
//
// For extra control over how the query is executed, the types QueryExecMode, QueryResultFormats, and
@@ -808,7 +817,8 @@ optionLoop:
var err error
sd, explicitPreparedStatement := c.preparedStatements[sql]
if sd != nil || mode == QueryExecModeCacheStatement || mode == QueryExecModeCacheDescribe || mode == QueryExecModeDescribeExec {
switch {
case sd != nil || mode == QueryExecModeCacheStatement || mode == QueryExecModeCacheDescribe || mode == QueryExecModeDescribeExec:
if sd == nil {
sd, err = c.getStatementDescription(ctx, mode, sql)
if err != nil {
@@ -833,7 +843,7 @@ optionLoop:
if resultFormatsByOID != nil {
resultFormats = make([]int16, len(sd.Fields))
for i := range resultFormats {
resultFormats[i] = resultFormatsByOID[uint32(sd.Fields[i].DataTypeOID)]
resultFormats[i] = resultFormatsByOID[sd.Fields[i].DataTypeOID]
}
}
@@ -846,7 +856,7 @@ optionLoop:
} else {
rows.resultReader = c.pgConn.ExecStatement(ctx, sd, c.eqb.ParamValues, c.eqb.ParamFormats, resultFormats)
}
} else if mode == QueryExecModeExec {
case mode == QueryExecModeExec:
err := c.eqb.Build(c.typeMap, nil, args)
if err != nil {
rows.fatal(err)
@@ -854,7 +864,7 @@ optionLoop:
}
rows.resultReader = c.pgConn.ExecParams(ctx, sql, c.eqb.ParamValues, nil, c.eqb.ParamFormats, c.eqb.ResultFormats)
} else if mode == QueryExecModeSimpleProtocol {
case mode == QueryExecModeSimpleProtocol:
sql, err = c.sanitizeForSimpleQuery(sql, args...)
if err != nil {
rows.fatal(err)
@@ -872,7 +882,7 @@ optionLoop:
}
return rows, nil
} else {
default:
err = fmt.Errorf("unknown QueryExecMode: %v", mode)
rows.fatal(err)
return rows, rows.err
@@ -1304,7 +1314,7 @@ func (c *Conn) LoadType(ctx context.Context, typeName string) (*pgtype.Type, err
switch typtype {
case "b": // array
elementOID, err := c.getArrayElementOID(ctx, oid)
elementOID, delimiter, err := c.getArrayElementOIDAndDelimiter(ctx, oid)
if err != nil {
return nil, err
}
@@ -1314,7 +1324,7 @@ func (c *Conn) LoadType(ctx context.Context, typeName string) (*pgtype.Type, err
return nil, errors.New("array element OID not registered")
}
return &pgtype.Type{Name: typeName, OID: oid, Codec: &pgtype.ArrayCodec{ElementType: dt}}, nil
return &pgtype.Type{Name: typeName, OID: oid, Codec: &pgtype.ArrayCodec{ElementType: dt, Delimiter: delimiter}}, nil
case "c": // composite
fields, err := c.getCompositeFields(ctx, oid)
if err != nil {
@@ -1360,15 +1370,16 @@ func (c *Conn) LoadType(ctx context.Context, typeName string) (*pgtype.Type, err
}
}
func (c *Conn) getArrayElementOID(ctx context.Context, oid uint32) (uint32, error) {
func (c *Conn) getArrayElementOIDAndDelimiter(ctx context.Context, oid uint32) (uint32, byte, error) {
var typelem uint32
var typdelim string
err := c.QueryRow(ctx, "select typelem from pg_type where oid=$1", oid).Scan(&typelem)
err := c.QueryRow(ctx, "select typelem, typdelim::text from pg_type where oid=$1", oid).Scan(&typelem, &typdelim)
if err != nil {
return 0, err
return 0, 0, err
}
return typelem, nil
return typelem, parseTypeDelimiter(typdelim), nil
}
func (c *Conn) getRangeElementOID(ctx context.Context, oid uint32) (uint32, error) {
+26 -6
View File
@@ -64,9 +64,12 @@ UNION ALL
-- As can be seen, there are 3 ways this can occur (the last of which
-- is due to being a composite class, where the composite fields are children)
pc(parent, child) AS (
-- typtype = 'b' AND typelem != 0 is not sufficient to identify an array type: some
-- scalar types (box, point, line, lseg, name, ...) also set typelem to describe their
-- internal C representation. typcategory = 'A' is the reliable "is an array" signal.
SELECT parent.oid, parent.typelem
FROM pg_type parent
WHERE parent.typtype = 'b' AND parent.typelem != 0
WHERE parent.typtype = 'b' AND parent.typelem != 0 AND parent.typcategory = 'A'
UNION ALL
SELECT parent.oid, parent.typbasetype
FROM pg_type parent
@@ -113,6 +116,7 @@ SELECT typname,
typtype,
typbasetype,
typelem,
typdelim,
pg_type.oid,`)
if supportsMultirange {
parts = append(parts, `
@@ -135,9 +139,13 @@ SELECT typname,
parts = append(parts, `
LEFT OUTER JOIN composite USING (oid)
LEFT OUTER JOIN pg_namespace ON (pg_type.typnamespace = pg_namespace.oid)
WHERE NOT (typtype = 'b' AND typelem = 0)`)
-- Only emit typtype = 'b' rows that are true arrays (typcategory = 'A'). Other base
-- types, including ones with a non-zero typelem for non-array reasons (box, point,
-- line, lseg, name, ...), already have a codec registered and must not be re-emitted,
-- or LoadTypes will overwrite their correct codec with a bogus ArrayCodec.
WHERE NOT (typtype = 'b' AND NOT (typelem != 0 AND typcategory = 'A'))`)
parts = append(parts, `
GROUP BY typname, pg_namespace.nspname, typtype, typbasetype, typelem, pg_type.oid, pg_range.rngsubtype,`)
GROUP BY typname, pg_namespace.nspname, typtype, typbasetype, typelem, typdelim, pg_type.oid, pg_range.rngsubtype,`)
if supportsMultirange {
parts = append(parts, `
multirange.rngtypid,`)
@@ -150,11 +158,18 @@ SELECT typname,
type derivedTypeInfo struct {
Oid, Typbasetype, Typelem, Rngsubtype, Rngtypid uint32
TypeName, Typtype, NspName string
TypeName, Typtype, NspName, Typdelim string
Attnames []string
Atttypids []uint32
}
func parseTypeDelimiter(typdelim string) byte {
if typdelim == "" {
return 0
}
return typdelim[0]
}
// LoadTypes performs a single (complex) query, returning all the required
// information to register the named types, as well as any other types directly
// or indirectly required to complete the registration.
@@ -177,7 +192,7 @@ func (c *Conn) LoadTypes(ctx context.Context, typeNames []string) ([]*pgtype.Typ
result := make([]*pgtype.Type, 0, 100)
for rows.Next() {
ti := derivedTypeInfo{}
err = rows.Scan(&ti.TypeName, &ti.NspName, &ti.Typtype, &ti.Typbasetype, &ti.Typelem, &ti.Oid, &ti.Rngtypid, &ti.Rngsubtype, &ti.Attnames, &ti.Atttypids)
err = rows.Scan(&ti.TypeName, &ti.NspName, &ti.Typtype, &ti.Typbasetype, &ti.Typelem, &ti.Typdelim, &ti.Oid, &ti.Rngtypid, &ti.Rngsubtype, &ti.Attnames, &ti.Atttypids)
if err != nil {
return nil, fmt.Errorf("While scanning type information: %w", err)
}
@@ -188,7 +203,7 @@ func (c *Conn) LoadTypes(ctx context.Context, typeNames []string) ([]*pgtype.Typ
if !ok {
return nil, fmt.Errorf("Array element OID %v not registered while loading pgtype %q", ti.Typelem, ti.TypeName)
}
type_ = &pgtype.Type{Name: ti.TypeName, OID: ti.Oid, Codec: &pgtype.ArrayCodec{ElementType: dt}}
type_ = &pgtype.Type{Name: ti.TypeName, OID: ti.Oid, Codec: &pgtype.ArrayCodec{ElementType: dt, Delimiter: parseTypeDelimiter(ti.Typdelim)}}
case "c": // composite
var fields []pgtype.CompositeCodecField
for i, fieldName := range ti.Attnames {
@@ -236,6 +251,11 @@ func (c *Conn) LoadTypes(ctx context.Context, typeNames []string) ([]*pgtype.Typ
}
result = append(result, type_)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("While processing rows: %w", err)
}
return result, nil
}
+33 -2
View File
@@ -74,6 +74,27 @@ pgx uses the [pgtype] package to converting Go values to and from PostgreSQL val
directly and is customizable and extendable. User defined data types such as enums, domains, and composite types may
require type registration. See that package's documentation for details.
PostgreSQL arrays (including results from set-returning aggregates such as array_agg)
can be scanned directly into a matching Go slice. For scalar columns, pass the slice
as the scan destination:
var ids []int64
err := conn.QueryRow(ctx, "select array_agg(id) from things").Scan(&ids)
For a column that is part of a row, combine the slice with the usual row-to-struct
helpers. A struct field of slice type will pick up the array_agg column when
collected via [CollectRows] and [RowToStructByName] (or
[RowToAddrOfStructByPos]):
type ThingEntry struct {
GroupID int64
ThingIDs []int64 `db:"thing_ids"`
}
rows, _ := conn.Query(ctx,
"select group_id, array_agg(thing_id) as thing_ids from things group by group_id")
entries, err := pgx.CollectRows(rows, pgx.RowToStructByName[ThingEntry])
Transactions
Transactions are started by calling [Conn.Begin].
@@ -189,8 +210,18 @@ implemented on top of [pgconn.PgConn]. The [Conn.PgConn] method can be used to a
PgBouncer
By default pgx automatically uses prepared statements. Prepared statements are incompatible with PgBouncer. This can be
disabled by setting a different [QueryExecMode] in [ConnConfig.DefaultQueryExecMode].
By default pgx automatically uses protocol-level named prepared statements. PgBouncer 1.21.0 and newer can use these
prepared statements in transaction and statement pooling modes when its max_prepared_statements setting is greater than
zero. In this configuration the default [QueryExecModeCacheStatement] mode may be used. See the PgBouncer documentation
for limitations and configuration details: https://www.pgbouncer.org/config.html#max_prepared_statements.
When using an older PgBouncer version or when prepared statement support is disabled, set
[ConnConfig.DefaultQueryExecMode] to [QueryExecModeExec]. [QueryExecModeCacheDescribe] and
[QueryExecModeSimpleProtocol] are also compatible with PgBouncer. Prefer [QueryExecModeExec] over
[QueryExecModeSimpleProtocol] whenever possible. Do not use [QueryExecModeDescribeExec] with transaction pooling because
PgBouncer may assign a different server connection between the describe and execute steps.
SQL-level prepared statements created with PREPARE are not supported by PgBouncer in transaction pooling mode.
*/
package pgx
+1 -1
View File
@@ -33,7 +33,7 @@ func Get(size int) *[]byte {
return &buf
}
ptrBuf := (pools[i].Get().(*[]byte))
ptrBuf := pools[i].Get().(*[]byte)
*ptrBuf = (*ptrBuf)[:size]
return ptrBuf
+118
View File
@@ -0,0 +1,118 @@
// Package pgdatetime writes PostgreSQL's ISO date/time text format.
//
// The format is the one EncodeDateTime and AppendSeconds produce in
// src/backend/utils/adt/datetime.c under the default ISO DateStyle:
//
// stamp = date [ SP time [ zone ] ] [ " BC" ]
// date = year "-" 2DIGIT "-" 2DIGIT
// time = 2DIGIT ":" 2DIGIT ":" 2DIGIT [ "." 1*6DIGIT ]
//
// The year is zero-padded to four digits and then grows as wide as it needs, the
// fractional part is omitted when it is zero and never carries trailing zeros, and " BC"
// comes last, after any time zone. Go's layout language can express none of those, which
// is why the format is written out here rather than left to time.Format.
//
// This is the only implementation of the format in pgx. pgtype's date, timestamp and
// timestamptz text encoders and the simple protocol's query sanitizer all go through it,
// so they cannot drift apart.
package pgdatetime
import (
"strconv"
"time"
)
// AppendDate appends t's date. The fields are read from t in its own location.
func AppendDate(buf []byte, t time.Time) []byte {
year, bc := splitBCYear(t.Year())
buf = appendDate(buf, year, int(t.Month()), t.Day())
return appendEra(buf, bc)
}
// AppendTimestamp appends t's date and time of day followed by zone, which goes before
// the era suffix because PostgreSQL writes " BC" last. zone is "Z" for a value already
// converted to UTC and "" for a type that carries no time zone at all.
//
// The fields are read from t in its own location, so a caller that wants the UTC instant
// must convert first. Sub-microsecond precision is discarded, matching the resolution of
// the wire format.
func AppendTimestamp(buf []byte, t time.Time, zone string) []byte {
year, bc := splitBCYear(t.Year())
buf = appendDate(buf, year, int(t.Month()), t.Day())
buf = append(buf, ' ')
buf = appendTime(buf, t.Hour(), t.Minute(), t.Second(), t.Nanosecond())
buf = append(buf, zone...)
return appendEra(buf, bc)
}
// splitBCYear converts an astronomical year, in which 1 BC is 0 and 2 BC is -1, to the
// year and era PostgreSQL writes.
func splitBCYear(year int) (displayYear int, bc bool) {
if year <= 0 {
return 1 - year, true
}
return year, false
}
func appendEra(buf []byte, bc bool) []byte {
if bc {
buf = append(buf, " BC"...)
}
return buf
}
// appendDate appends year-month-day. year is a displayed year from splitBCYear rather
// than an astronomical one.
func appendDate(buf []byte, year, month, day int) []byte {
buf = appendYear(buf, year)
buf = append(buf, '-')
buf = append2Digits(buf, month)
buf = append(buf, '-')
return append2Digits(buf, day)
}
func appendTime(buf []byte, hour, min, sec, nsec int) []byte {
buf = append2Digits(buf, hour)
buf = append(buf, ':')
buf = append2Digits(buf, min)
buf = append(buf, ':')
buf = append2Digits(buf, sec)
usec := nsec / 1000
if usec == 0 {
return buf
}
var frac [6]byte
for i := 5; i >= 0; i-- {
frac[i] = byte('0' + usec%10)
usec /= 10
}
// The server trims trailing zeros, so ".5" rather than ".500000".
end := len(frac)
for end > 1 && frac[end-1] == '0' {
end--
}
buf = append(buf, '.')
return append(buf, frac[:end]...)
}
// append2Digits appends v as exactly two digits. Every field but the year is fixed width
// and comes from a time.Time accessor, so v is always in [0, 99] and the general routine
// below is not needed.
func append2Digits(buf []byte, v int) []byte {
return append(buf, byte('0'+v/10), byte('0'+v%10))
}
// appendYear appends v zero-padded to four digits, or wider when the year needs it. v is
// a displayed year from splitBCYear and so is always positive.
func appendYear(buf []byte, v int) []byte {
var tmp [20]byte
s := strconv.AppendInt(tmp[:0], int64(v), 10)
for i := len(s); i < 4; i++ {
buf = append(buf, '0')
}
return append(buf, s...)
}
+4 -2
View File
@@ -1,6 +1,8 @@
// Package pgio is a low-level toolkit building messages in the PostgreSQL wire protocol.
// Package pgio is a low-level toolkit for building and parsing messages in the
// PostgreSQL wire protocol.
/*
pgio provides functions for appending integers to a []byte while doing byte
order conversion.
order conversion, and a bounds-checked Reader for parsing binary values from
untrusted input.
*/
package pgio
+250
View File
@@ -0,0 +1,250 @@
package pgio
import (
"bytes"
"encoding/binary"
"errors"
"fmt"
)
// ErrInsufficientBytes is wrapped by all Reader errors caused by a read past
// the end of the source.
var ErrInsufficientBytes = errors.New("insufficient bytes")
// Reader is a bounds-checked reader for the PostgreSQL binary format. It is
// designed so that decoders of untrusted input cannot forget a length check:
// every read validates against the remaining bytes, and the first failure
// sticks. After a failure all subsequent reads return zero values, so a
// decoder can read an entire structure without intermediate error checks and
// inspect Err or Finish once at the end.
//
// Reads never panic. A decoder that branches on a value it just read (e.g. an
// element count used to size an allocation) should check Err before acting on
// the value.
type Reader struct {
s []byte
rp int
err error
}
func NewReader(s []byte) *Reader {
return &Reader{s: s}
}
func (r *Reader) fail(err error) {
if r.err == nil {
r.err = err
}
}
// need reports whether n more bytes are available, recording an error if not.
//
// The error is built by failShort rather than here so that need stays within
// the inliner's budget. Reads of fixed-size values are the hottest path in the
// driver, and an un-inlined bounds check costs more than the read itself.
func (r *Reader) need(n int) bool {
if r.err != nil {
return false
}
if len(r.s)-r.rp < n {
r.failShort(n)
return false
}
return true
}
func (r *Reader) failShort(n int) {
r.fail(fmt.Errorf("%w: %d needed at offset %d, %d remain", ErrInsufficientBytes, n, r.rp, len(r.s)-r.rp))
}
// Err returns the first error encountered, if any.
func (r *Reader) Err() error {
return r.err
}
// Remaining returns the number of unread bytes.
func (r *Reader) Remaining() int {
return len(r.s) - r.rp
}
func (r *Reader) Byte() byte {
if !r.need(1) {
return 0
}
b := r.s[r.rp]
r.rp += 1
return b
}
func (r *Reader) Uint16() uint16 {
if !r.need(2) {
return 0
}
n := binary.BigEndian.Uint16(r.s[r.rp:])
r.rp += 2
return n
}
func (r *Reader) Int16() int16 {
return int16(r.Uint16())
}
func (r *Reader) Uint32() uint32 {
if !r.need(4) {
return 0
}
n := binary.BigEndian.Uint32(r.s[r.rp:])
r.rp += 4
return n
}
func (r *Reader) Int32() int32 {
return int32(r.Uint32())
}
func (r *Reader) Uint64() uint64 {
if !r.need(8) {
return 0
}
n := binary.BigEndian.Uint64(r.s[r.rp:])
r.rp += 8
return n
}
func (r *Reader) Int64() int64 {
return int64(r.Uint64())
}
// Bytes reads the next n bytes. The returned slice aliases the source; it is
// not a copy.
func (r *Reader) Bytes(n int) []byte {
if n < 0 {
r.fail(fmt.Errorf("invalid byte count %d at offset %d", n, r.rp))
return nil
}
if !r.need(n) {
return nil
}
b := r.s[r.rp : r.rp+n]
r.rp += n
return b
}
// CString reads a NUL-terminated string, returning the bytes before the
// terminator and consuming the terminator. The returned slice aliases the
// source; it is not a copy.
func (r *Reader) CString() []byte {
if r.err != nil {
return nil
}
i := bytes.IndexByte(r.s[r.rp:], 0)
if i < 0 {
r.fail(fmt.Errorf("%w: unterminated string at offset %d", ErrInsufficientBytes, r.rp))
return nil
}
b := r.s[r.rp : r.rp+i]
r.rp += i + 1
return b
}
// Count reads an int32 element count and validates it against the remaining
// bytes: the count must be non-negative, and since each element occupies at
// least minElemSize bytes, count*minElemSize must not exceed the remaining
// message. This bounds allocations sized from the count against a malicious
// or corrupt message claiming a huge count. Returns 0 on any failure.
func (r *Reader) Count(minElemSize int) int {
offset := r.rp
count := int(r.Int32())
if r.err != nil {
return 0
}
if count < 0 {
r.fail(fmt.Errorf("invalid element count %d at offset %d", count, offset))
return 0
}
if minElemSize < 1 {
minElemSize = 1
}
if count > r.Remaining()/minElemSize {
r.fail(fmt.Errorf("element count %d at offset %d exceeds %d remaining bytes", count, offset, r.Remaining()))
return 0
}
return count
}
// Value reads an int32 length followed by that many bytes — the standard
// PostgreSQL binary representation of a value. A length of -1 means NULL and
// returns (nil, true). Any other negative length is an error. The returned
// slice aliases the source; null is only meaningful if Err returns nil.
func (r *Reader) Value() (data []byte, null bool) {
offset := r.rp
length := r.Int32()
if r.err != nil {
return nil, false
}
if length == -1 {
return nil, true
}
if length < 0 {
r.fail(fmt.Errorf("invalid value length %d at offset %d", length, offset))
return nil, false
}
return r.Bytes(int(length)), false
}
// Finish returns the first error encountered, or an error if unread bytes
// remain. Decoders that must consume the entire source should end with Finish.
func (r *Reader) Finish() error {
if r.err != nil {
return r.err
}
if r.rp != len(r.s) {
return r.errTrailing()
}
return nil
}
func (r *Reader) errTrailing() error {
return fmt.Errorf("%d unexpected trailing bytes at offset %d", len(r.s)-r.rp, r.rp)
}
// ErrInvalidLength is wrapped by the errors returned from the exact-length
// read functions below.
var ErrInvalidLength = errors.New("invalid length")
func errLength(want, got int) error {
return fmt.Errorf("%w: expected %d bytes, got %d", ErrInvalidLength, want, got)
}
// The Uint*Exact functions read a single fixed-size value that makes up an
// entire message, which is what the scan plans for the fixed-size PostgreSQL
// types receive. They are the counterpart to Reader for values that have no
// internal structure: there is no position to track and no error to make
// sticky, just an exact-length assertion the caller cannot skip. Keeping them
// separate from Reader is deliberate — these are the hottest decode paths in
// the driver and they are small enough for the compiler to inline, which a
// Reader method carrying a bounds check and a read pointer is not.
// Uint16Exact returns the big-endian uint16 in src, which must be exactly 2 bytes.
func Uint16Exact(src []byte) (uint16, error) {
if len(src) != 2 {
return 0, errLength(2, len(src))
}
return binary.BigEndian.Uint16(src), nil
}
// Uint32Exact returns the big-endian uint32 in src, which must be exactly 4 bytes.
func Uint32Exact(src []byte) (uint32, error) {
if len(src) != 4 {
return 0, errLength(4, len(src))
}
return binary.BigEndian.Uint32(src), nil
}
// Uint64Exact returns the big-endian uint64 in src, which must be exactly 8 bytes.
func Uint64Exact(src []byte) (uint64, error) {
if len(src) != 8 {
return 0, errLength(8, len(src))
}
return binary.BigEndian.Uint64(src), nil
}
+8 -2
View File
@@ -11,6 +11,8 @@ import (
"sync"
"time"
"unicode/utf8"
"github.com/jackc/pgx/v5/internal/pgdatetime"
)
// Part is either a string or an int. A string is raw SQL. An int is a
@@ -81,8 +83,12 @@ func (q *Query) Sanitize(args ...any) (string, error) {
case string:
p = QuoteString(buf.AvailableBuffer(), arg)
case time.Time:
p = arg.Truncate(time.Microsecond).
AppendFormat(buf.AvailableBuffer(), "'2006-01-02 15:04:05.999999999Z07:00:00'")
// pgtype's timestamptz text encoder writes the same format. Going through
// the same code keeps the two from drifting apart, and time.Format cannot
// express PostgreSQL's BC era or its variable width year anyway.
p = append(buf.AvailableBuffer(), '\'')
p = pgdatetime.AppendTimestamp(p, arg.UTC(), "Z")
p = append(p, '\'')
default:
return "", fmt.Errorf("invalid arg type: %T", arg)
}
+1 -1
View File
@@ -62,7 +62,7 @@ func (c *LRUCache) Put(sd *pgconn.StatementDescription) {
return
}
// The statement may have been invalidated but not yet handled. Do not readd it to the cache.
// The statement may have been invalidated but not yet handled. Do not re-add it to the cache.
if _, invalidated := c.invalidSet[sd.SQL]; invalidated {
return
}
+173
View File
@@ -0,0 +1,173 @@
# mise is this project's single development entry point: the same `mise run <task>` works on
# native macOS, on native Linux, and inside the devcontainer. See DEVELOPMENT.md.
#
# The division of labour:
# mise tool versions, this checkout's environment, one-shot tasks
# process-compose long-running services (process-compose.yaml) — five PostgreSQL clusters + CRDB
# port-tamer this checkout's TCP ports (port-tamer.toml -> .dev/ports.env)
#
# PostgreSQL itself is the one prerequisite mise does not provide; DEVELOPMENT.md §1 lists it.
[tools]
go = '1.27.0'
ruby = '4.0.4'
"go:github.com/go-critic/go-critic/cmd/gocritic" = "latest"
"go:github.com/gordonklaus/ineffassign" = "latest"
"go:github.com/mdempsky/unconvert" = "latest"
"go:golang.org/x/tools/cmd/goimports" = "latest"
"go:mvdan.cc/gofumpt" = "latest"
# The linter .golangci.yml configures and CLAUDE.md documents. It was never actually provisioned
# by the devcontainer, so `golangci-lint run ./...` only worked if you had installed it yourself.
golangci-lint = '2.13.2'
# The process supervisor for the long-running development services. Pinned like every other tool
# so macOS, Linux, and the devcontainer run the same version.
process-compose = '1.122.0'
# Per-checkout TCP port allocation. The `github:` backend takes the prebuilt release binary, so it
# does not depend on the Go toolchain this repo pins.
"github:jackc/port-tamer" = '0.1.0'
# The CockroachDB test target. Previously a container image; the version tracks the one CI installs
# in ci/setup_test.bash.
cockroach = '25.4.4'
# --- Environment ---------------------------------------------------------------------------------
#
# This checkout's own service endpoints. Loading them here is what makes the per-checkout clusters
# the DEFAULT: a bare `psql` and a bare `go test ./...` read PG*/PGX_TEST_* from the environment,
# so pointing those at .dev/ in one place moves every consumer at once.
#
# Two files with two owners, and the order matters — the derived values are read after the
# allocation they come from:
#
# .dev/ports.env port-tamer's state file (port-tamer.toml declares the names). NAME=<port>
# lines only; port-tamer rewrites it canonically and rejects anything else.
# .dev/derived.env PGHOST/PGPORT/PGUSER/PGDATABASE plus the default target's PGX_TEST_* set —
# scripts/devenv.rb, via scripts/lib/test_targets.rb.
#
# Both are generated per checkout and gitignored; `mise run dev:init` creates them. Their absence
# is not an error — mise skips a missing file and PG* falls back to whatever the shell already had.
[env]
_.file = ['.dev/ports.env', '.dev/derived.env']
# Homebrew's postgresql@<major> formulas are KEG-ONLY: nothing is linked into the prefix bin, so
# psql and pg_isready are not on PATH after `brew install`. The scripts that need the SERVER
# binaries resolve them per major explicitly (scripts/lib/pg_bin.rb), but the CLIENT tools are
# invoked by name from the readiness probes, `rake db:psql`, and the database bootstrap.
#
# The newest major's client is used for every server: libpq is backward compatible, so one psql
# talks to all five clusters. Both standard Homebrew prefixes are listed (Apple silicon, then
# Intel). Neither exists on Linux, where a nonexistent PATH entry is simply ignored, so this is
# inert in the devcontainer.
_.path = [
'/opt/homebrew/opt/postgresql@18/bin',
'/usr/local/opt/postgresql@18/bin',
]
# process-compose's control API address. The port comes from this checkout's allocation
# (PC_PORT_NUM in .dev/ports.env), so every `process-compose ...` command reaches THIS checkout's
# stack with no flags.
PC_ADDRESS = "127.0.0.1"
# Kept separate deliberately: `mise run dev` writes the supervisor log, and a client command such
# as `process-compose process list` would otherwise truncate the log of the stack it is inspecting.
PC_SERVER_LOG_FILE = "{{config_root}}/.dev/process-compose.log"
PC_LOG_FILE = "{{config_root}}/.dev/process-compose-client.log"
# --- Tasks: the project's development interface ---------------------------------------------------
#
# Thin wrappers. The logic lives in scripts/ and the Rakefile; these exist so a workflow is
# reachable without knowing what is underneath, and so the same command works in every environment.
[tasks."dev:init"]
description = "Bootstrap this checkout: allocate its ports, decode its certificates"
run = "ruby scripts/devenv.rb ensure"
[tasks.dev]
description = "Start PostgreSQL 18 and this checkout's on-demand database supervisor"
run = "ruby scripts/dev.rb"
[tasks."dev:all"]
description = "Start PostgreSQL 14-18 and CockroachDB eagerly under process-compose"
run = "ruby scripts/dev.rb all"
[tasks."dev:down"]
description = "Stop this checkout's development stack"
run = "ruby scripts/dev.rb down"
[tasks."dev:wait"]
description = "Wait for a detached development stack to become ready"
run = "ruby scripts/dev_ready.rb --wait"
[tasks."dev:ready"]
description = "Require the development stack to be running"
run = "ruby scripts/dev_ready.rb"
hide = true
[tasks."dev:ports"]
description = "Print this checkout's port allocation and where each server's data lives"
run = "rake dev:ports:show"
[tasks."dev:ports:ensure"]
description = "Allocate this checkout's ports if it has none (idempotent)"
run = "rake dev:ports:ensure"
[tasks."dev:ports:overwrite"]
description = "Move this checkout to a different port group (stop its services first)"
run = "rake dev:ports:overwrite"
# --- Databases -----------------------------------------------------------------------------------
[tasks."db:start"]
description = "Prewarm database targets (default: pg18); e.g. mise run db:start pg16 crdb"
run = "ruby scripts/dev_services.rb start"
[tasks."db:stop"]
description = "Stop database targets (default: pg18); e.g. mise run db:stop pg16 crdb"
run = "ruby scripts/dev_services.rb stop"
# These call scripts/devdb.rb rather than the equivalent rake tasks. mise APPENDS a task's
# arguments to its `run` line, and `rake db:reset 16` does not mean "db:reset with the argument
# 16" — rake reads the 16 as a SECOND TASK, runs db:reset with no major (which is every cluster),
# and only then fails on the unknown task. devdb.rb takes the major as an ordinary argument, so
# `mise run db:reset 16` reaches it as written. `rake db:reset[16]` is still the rake spelling.
[tasks."db:init"]
description = "Create this checkout's PostgreSQL clusters under .dev (idempotent); e.g. mise run db:init 16"
run = "ruby scripts/devdb.rb init"
[tasks."db:psql"]
description = "psql against one of this checkout's clusters (default: the newest major)"
run = "ruby scripts/devdb.rb psql"
[tasks."db:setup"]
description = "Create pgx_test and its roles in a running cluster (idempotent)"
run = "ruby scripts/devdb.rb setup"
[tasks."db:reset"]
description = "Destroy and recreate this checkout's clusters (destructive; CONFIRM=yes, stack down)"
run = "ruby scripts/devdb.rb reset"
# --- Test, format, generate ------------------------------------------------------------------------
[tasks.test]
description = "Run the test suite against one target (default pg18); e.g. mise run test pg16"
run = "./test.sh"
[tasks."test:all"]
description = "Run the test suite against every target: PostgreSQL 14-18 and CockroachDB"
run = "./test.sh all"
[tasks.fmt]
description = "Format all Go sources with goimports"
run = "goimports -w ."
[tasks.lint]
description = "Run the linters configured in .golangci.yml"
run = "golangci-lint run ./..."
[tasks.generate]
description = "Regenerate the ERB-templated Go sources"
run = "rake generate"
+116 -1
View File
@@ -3,6 +3,7 @@ package pgx
import (
"context"
"fmt"
"reflect"
"strconv"
"strings"
"unicode/utf8"
@@ -18,6 +19,7 @@ import (
//
// Named placeholders are case sensitive and must start with a letter or underscore. Subsequent characters can be
// letters, numbers, or underscores.
type NamedArgs map[string]any
// RewriteQuery implements the QueryRewriter interface.
@@ -34,6 +36,119 @@ func (sna StrictNamedArgs) RewriteQuery(ctx context.Context, conn *Conn, sql str
return rewriteQuery(sna, sql, true)
}
type errorQueryRewriter struct {
err error
}
func (r errorQueryRewriter) RewriteQuery(ctx context.Context, conn *Conn, sql string, args []any) (newSQL string, newArgs []any, err error) {
return "", nil, r.err
}
// StructArgs converts exported fields of a struct into a QueryRewriter so it can
// be used as the first argument to a query method (e.g. "where id=@id").
//
// Field names are taken from the `db` struct tag if present. Tag values may
// include comma-separated options (e.g. `db:"id,omitempty"`). A `db:"-"` field is
// ignored. If no `db` tag is present, the Go field name is used.
//
// sa may be a struct or a pointer to a struct.
func StructArgs(sa any) QueryRewriter {
args, err := structArgs(sa)
if err != nil {
return errorQueryRewriter{err: err}
}
return NamedArgs(args)
}
// StrictStructArgs is like StructArgs but uses StrictNamedArgs rewriting
// semantics (i.e. errors if the SQL query references missing arguments or if
// extra arguments are provided).
func StrictStructArgs(sa any) QueryRewriter {
args, err := structArgs(sa)
if err != nil {
return errorQueryRewriter{err: err}
}
return StrictNamedArgs(args)
}
func structArgs(sa any) (map[string]any, error) {
if sa == nil {
return nil, fmt.Errorf("StructArgs requires a struct or pointer to struct, got nil")
}
v := reflect.ValueOf(sa)
t := v.Type()
if t.Kind() == reflect.Pointer {
if v.IsNil() {
return nil, fmt.Errorf("StructArgs requires a non-nil pointer to struct")
}
v = v.Elem()
t = v.Type()
}
if t.Kind() != reflect.Struct {
return nil, fmt.Errorf("StructArgs requires a struct or pointer to struct, got %s", t)
}
out := make(map[string]any, t.NumField())
for i := 0; i < t.NumField(); i++ {
sf := t.Field(i)
// Ignore unexported fields.
if sf.PkgPath != "" {
continue
}
key, ok, err := dbTagKey(sf)
if err != nil {
return nil, err
}
if !ok {
continue
}
if _, exists := out[key]; exists {
return nil, fmt.Errorf("duplicate StructArgs key %q", key)
}
out[key] = v.Field(i).Interface()
}
return out, nil
}
// dbTagKey derives the named-argument key for a struct field. Tag parsing matches
// RowToStructByName* in rows.go (structTagKey, Lookup, comma options, db:"-").
// Anonymous embedded structs are skipped without flattening (unlike row scanning).
func dbTagKey(sf reflect.StructField) (key string, ok bool, err error) {
if sf.Anonymous {
ft := sf.Type
if ft.Kind() == reflect.Pointer {
ft = ft.Elem()
}
if ft.Kind() == reflect.Struct {
return "", false, nil
}
}
dbTag, dbTagPresent := sf.Tag.Lookup(structTagKey)
if dbTagPresent {
dbTag, _, _ = strings.Cut(dbTag, ",")
}
if dbTag == "-" {
return "", false, nil
}
if dbTagPresent {
if dbTag == "" {
return "", false, fmt.Errorf("field %s has empty `%s` tag", sf.Name, structTagKey)
}
return dbTag, true, nil
}
return sf.Name, true, nil
}
type namedArg string
type sqlLexer struct {
@@ -163,7 +278,7 @@ func namedArgState(l *sqlLexer) stateFn {
if _, found := l.nameToOrdinal[na]; !found {
l.nameToOrdinal[na] = len(l.nameToOrdinal) + 1
}
l.parts = append(l.parts, namedArg(na))
l.parts = append(l.parts, na)
l.start = l.pos
return rawState
}
+11 -3
View File
@@ -229,11 +229,12 @@ func (sc *scramClient) clientFirstMessage() []byte {
sc.clientFirstMessageBare = fmt.Appendf(nil, "n=,r=%s", sc.clientNonce)
if sc.authMechanism == scramSHA256PlusName {
switch {
case sc.authMechanism == scramSHA256PlusName:
sc.clientGS2Header = []byte("p=tls-server-end-point,,")
} else if sc.hasTLS {
case sc.hasTLS:
sc.clientGS2Header = []byte("y,,")
} else {
default:
sc.clientGS2Header = []byte("n,,")
}
@@ -283,6 +284,13 @@ func (sc *scramClient) recvServerFirstMessage(serverFirstMessage []byte) error {
if err != nil || sc.iterations <= 0 {
return fmt.Errorf("invalid SCRAM iteration count received from server: %w", err)
}
// Bound server-supplied iteration count to prevent a malicious server from forcing the client
// to spend unbounded CPU in PBKDF2. PostgreSQL's scram_iterations defaults to 4096; this ceiling
// is ~2500x that.
const maxScramIterations = 10_000_000
if sc.iterations > maxScramIterations {
return fmt.Errorf("SCRAM iteration count from server too high: %d (max %d)", sc.iterations, maxScramIterations)
}
if !bytes.HasPrefix(sc.clientAndServerNonce, sc.clientNonce) {
return errors.New("invalid SCRAM nonce: did not start with client nonce")
+288 -125
View File
@@ -7,11 +7,9 @@ import (
"encoding/pem"
"errors"
"fmt"
"io"
"maps"
"math"
"net"
"net/url"
"os"
"path/filepath"
"strconv"
@@ -44,6 +42,11 @@ type Config struct {
LookupFunc LookupFunc // e.g. net.Resolver.LookupHost
BuildFrontend BuildFrontendFunc
// MaxProtocolMessageBodyLen is the maximum length of a PostgreSQL wire protocol message body in octets. If a
// message body exceeds this length, reading the message will fail with pgproto3.ExceededMaxBodyLenErr. The default
// value is 0, which means no maximum is enforced.
MaxProtocolMessageBodyLen int
// BuildContextWatcherHandler is called to create a ContextWatcherHandler for a connection. The handler is called
// when a context passed to a PgConn method is canceled.
BuildContextWatcherHandler func(*PgConn) ctxwatch.Handler
@@ -100,14 +103,64 @@ type Config struct {
// Valid values: "disable", "prefer", "require". Defaults to "prefer".
ChannelBinding string
// RequireAuth restricts which authentication methods the client will accept from the server,
// matching libpq's require_auth parameter. It is a comma-separated list of method names
// (password, md5, gss, sspi, scram-sha-256, oauth, none). A leading "!" on every entry negates
// the list (forbid these methods, allow all others). Empty (the default) means all methods are
// accepted.
RequireAuth string
createdByParseConfig bool // Used to enforce created by ParseConfig rule.
}
// defaultPort is the port used when neither the connection string, the environment, nor the
// service file supplies one. It is also the per-element fallback for empty entries in a
// multi-host port list.
const defaultPort = "5432"
// connStringKeyAliases maps libpq parameter keywords to the canonical key names this package
// uses internally in the parsed-settings map. Most keywords are already canonical; this map
// holds only those whose pgx-internal name differs from the libpq spelling.
var connStringKeyAliases = map[string]string{
"dbname": "database",
}
// canonicalConnStringKey returns the canonical settings-map key for a libpq parameter keyword.
func canonicalConnStringKey(k string) string {
if c, ok := connStringKeyAliases[k]; ok {
return c
}
return k
}
// ParseConfigOptions contains options that control how a config is built such as GetSSLPassword.
type ParseConfigOptions struct {
// GetSSLPassword gets the password to decrypt a SSL client certificate. This is analogous to the libpq function
// PQsetSSLKeyPassHook_OpenSSL.
GetSSLPassword GetSSLPasswordFunc
// ConnStringAllowedKeys, if non-nil, restricts which parameter keys may appear in connString
// itself. Any other key (whether connString is in keyword/value or URL form) causes
// ParseConfigWithOptions to return an error before any filesystem access or network
// resolution is attempted. Environment variables (PGHOST, PGSERVICEFILE, ...) and built-in
// defaults are not checked: only keys that originate from the connString argument.
//
// Keys may be given in either their libpq spelling ("dbname") or pgx-internal spelling
// ("database"); both are accepted. The URI-only ssl=true alias for sslmode=require is
// accepted when either "ssl" or "sslmode" is allowed; an explicit sslmode key or a
// non-"true" ssl value only matches its own spelling. Every ssl/sslmode occurrence in
// the connection string is validated, including occurrences superseded by a later
// repeated parameter.
//
// A nil slice (the default) applies no restriction and matches libpq behaviour. An empty
// non-nil slice rejects every key, i.e. connString must be empty.
//
// Use this when any part of connString is built from input the application does not fully
// control (tenant configuration, RPC parameters, admin UI fields). List only the keys that
// input is expected to supply. This fails closed: a future libpq parameter that pgconn learns
// to parse will be rejected unless the application has explicitly allowed it, rather than
// silently passing through.
ConnStringAllowedKeys []string
}
// Copy returns a deep copy of the config that is safe to use and modify.
@@ -192,10 +245,10 @@ func NetworkAddress(host string, port uint16) (network, address string) {
// to only read from the environment. If a password is not supplied it will attempt to read the .pgpass file.
//
// # Example Keyword/Value
// user=jack password=secret host=pg.example.com port=5432 dbname=mydb sslmode=verify-ca
// user=jack password=secret host=pg.example.com port=5432 dbname=mydb sslmode=verify-full
//
// # Example URL
// postgres://jack:secret@pg.example.com:5432/mydb?sslmode=verify-ca
// postgres://jack:secret@pg.example.com:5432/mydb?sslmode=verify-full
//
// The returned *Config may be modified. However, it is strongly recommended that any configuration that can be done
// through the connection string be done there. In particular the fields Host, Port, TLSConfig, and Fallbacks can be
@@ -253,11 +306,35 @@ func NetworkAddress(host string, port uint16) (network, address string) {
// changed later but the unencrypted fallback is present. Ensure there are no stale fallbacks when manually setting
// TLSConfig.
//
// Several connection parameters cause ParseConfig to read files from the local filesystem: servicefile, passfile,
// sslkey, sslcert, and sslrootcert. Applications that build connection strings from untrusted input must not allow
// these keys to be set by that input. In particular, servicefile (which pgconn accepts in the connection string;
// libpq does not) is read as an INI file whose entries override other connection settings including host, port, and
// sslmode, so an attacker who controls servicefile and service can redirect the connection. If any portion of the
// connection string is externally supplied, use ParseConfigWithOptions and set ParseConfigOptions.ConnStringAllowedKeys
// to an allow-list of the keys that input is expected to supply; any other key in the connection string is then
// rejected before any filesystem access occurs.
//
// Other known differences with libpq:
//
// When multiple hosts are specified, libpq allows them to have different passwords set via the .pgpass file. pgconn
// does not.
//
// URL query parameters that libpq does not recognize cause libpq to fail with an "invalid URI query parameter" error.
// ParseConfig accepts them: they become runtime parameters or pgx-specific options (e.g. pool_max_conns).
//
// Connection strings containing a NUL byte are rejected, in both URI and keyword/value form. libpq cannot encounter
// one because its conninfo strings are NUL-terminated C strings, but a Go string can carry a NUL into the startup
// packet, where it delimits parameters rather than being data. Settings reaching Config by other routes (a service
// file, or direct assignment to Config.RuntimeParams, User, or Database) are not checked here; a NUL in those is
// caught when the startup message is encoded, and Connect fails rather than sending it.
//
// Error messages from ParseConfig avoid quoting the unredacted connection string and attempt to redact recognizable
// password fields, while libpq quotes the failing input verbatim. This redaction is best effort. An invalid connection
// string can be structurally ambiguous, so pgconn cannot guarantee that every password in malformed input will be
// identified or redacted. Applications should not assume that parse errors are safe to expose when connection strings
// may contain secrets.
//
// In addition, ParseConfig accepts the following options:
//
// - servicefile.
@@ -276,11 +353,12 @@ func ParseConfigWithOptions(connString string, options ParseConfigOptions) (*Con
envSettings := parseEnvSettings()
connStringSettings := make(map[string]string)
var urlMeta parseURLMeta
if connString != "" {
var err error
// connString may be a database URL or in PostgreSQL keyword/value format
if strings.HasPrefix(connString, "postgres://") || strings.HasPrefix(connString, "postgresql://") {
connStringSettings, err = parseURLSettings(connString)
connStringSettings, urlMeta, err = parseURLSettings(connString)
if err != nil {
return nil, &ParseConfigError{ConnString: connString, msg: "failed to parse as URL", err: err}
}
@@ -292,7 +370,82 @@ func ParseConfigWithOptions(connString string, options ParseConfigOptions) (*Con
}
}
if options.ConnStringAllowedKeys != nil {
allowed := make(map[string]struct{}, len(options.ConnStringAllowedKeys))
for _, k := range options.ConnStringAllowedKeys {
allowed[canonicalConnStringKey(k)] = struct{}{}
}
notAllowed := func(k string) error {
return &ParseConfigError{ConnString: connString, msg: fmt.Sprintf("connection string key %q is not in ConnStringAllowedKeys", k)}
}
_, sslAllowed := allowed["ssl"]
_, sslmodeAllowed := allowed["sslmode"]
// Repeated-key handling and the URI ssl=true alias rewrite can remove
// ssl/sslmode occurrences from the final settings map, so those two keys
// are validated from what the user actually wrote -- every occurrence,
// fail closed -- rather than from what survived. The alias itself is
// accepted under either spelling, the same way dbname/database are
// interchangeable; an explicit sslmode key or a non-"true" ssl value
// only matches its own spelling.
if urlMeta.sawRawSSLKey && !sslAllowed {
return nil, notAllowed("ssl")
}
if urlMeta.sawExplicitSSLModeKey && !sslmodeAllowed {
return nil, notAllowed("sslmode")
}
if urlMeta.sawSSLTrueAlias && !sslAllowed && !sslmodeAllowed {
return nil, notAllowed("ssl")
}
for k := range connStringSettings {
if _, ok := allowed[k]; ok {
continue
}
// A multi-host URI with no explicit ports produces an implied
// all-empty port list (e.g. ","), matching libpq. Only that list
// -- identified by the parser from the raw syntax, not inferred
// from the value -- is exempt from the allow-list: it carries no
// user-supplied port text. An explicit empty port (?port= in a
// URI or port= in keyword/value form) is user-supplied, and
// because a present-but-empty port still shadows PGPORT it must
// pass the allow-list like any other key.
if k == "port" && urlMeta.impliedEmptyPortList {
continue
}
// A surviving ssl or sslmode entry from a URI was already
// validated above against every spelling the user wrote.
if k == "ssl" && urlMeta.sawRawSSLKey {
continue
}
if k == "sslmode" && (urlMeta.sawSSLTrueAlias || urlMeta.sawExplicitSSLModeKey) {
continue
}
return nil, notAllowed(k)
}
}
settings := mergeSettings(defaultSettings, envSettings, connStringSettings)
// The home-directory-derived defaults (passfile, servicefile, sslcert,
// sslkey, sslrootcert) are already present in settings at this point:
// defaultSettings resolves them via the user's home directory, which is
// safe and cheap to look up (see defaults.go / defaults_windows.go).
//
// The default PostgreSQL user name is different: resolving it requires
// looking up the OS user account, which can be slow or, in some
// restricted container environments, crash the process. So that lookup
// is memoized and only performed lazily below, the first time it is
// actually needed -- i.e. only when a connection string or environment
// does not already supply a user.
var cachedOSUserSettings map[string]string
lazyOSUserSettings := func() map[string]string {
if cachedOSUserSettings == nil {
cachedOSUserSettings = osUserSettings()
}
return cachedOSUserSettings
}
if service, present := settings["service"]; present {
serviceSettings, err := parseServiceSettings(settings["servicefile"], service)
if err != nil {
@@ -302,15 +455,20 @@ func ParseConfigWithOptions(connString string, options ParseConfigOptions) (*Con
settings = mergeSettings(defaultSettings, envSettings, serviceSettings, connStringSettings)
}
// Only fall back to the OS user account for the default PostgreSQL user
// name when it was not already supplied by the connection string,
// environment, or service file.
if settings["user"] == "" {
settings = mergeSettings(lazyOSUserSettings(), settings)
}
config := &Config{
createdByParseConfig: true,
Database: settings["database"],
User: settings["user"],
Password: settings["password"],
RuntimeParams: make(map[string]string),
BuildFrontend: func(r io.Reader, w io.Writer) *pgproto3.Frontend {
return pgproto3.NewFrontend(r, w)
},
BuildFrontend: pgproto3.NewFrontend,
BuildContextWatcherHandler: func(pgConn *PgConn) ctxwatch.Handler {
return &DeadlineContextWatcherHandler{Conn: pgConn.conn}
},
@@ -360,6 +518,7 @@ func ParseConfigWithOptions(connString string, options ParseConfigOptions) (*Con
"min_protocol_version": {},
"max_protocol_version": {},
"channel_binding": {},
"require_auth": {},
}
// Adding kerberos configuration
@@ -382,17 +541,41 @@ func ParseConfigWithOptions(connString string, options ParseConfigOptions) (*Con
hosts := strings.Split(settings["host"], ",")
ports := strings.Split(settings["port"], ",")
// Like libpq, if exactly one port is given it applies to all hosts;
// otherwise there must be exactly one port per host. Empty list elements
// mean "use the default".
if len(ports) > 1 && len(ports) != len(hosts) {
return nil, &ParseConfigError{ConnString: connString, msg: fmt.Sprintf("could not match %d port numbers to %d hosts", len(ports), len(hosts))}
}
// defaultHost stats candidate socket directories, so resolve it at most
// once even when several host list elements are empty. It never returns "".
resolvedDefaultHost := ""
for i, host := range hosts {
var portStr string
if i < len(ports) {
portStr = ports[i]
} else {
portStr = ports[0]
if host == "" {
if resolvedDefaultHost == "" {
resolvedDefaultHost = defaultHost()
}
host = resolvedDefaultHost
}
portStr := ports[0]
if len(ports) > 1 {
portStr = ports[i]
}
if portStr == "" {
portStr = defaultPort
}
// The strconv error is deliberately not wrapped: it quotes the
// offending text, and in a malformed URI the bytes that land in the
// port position can be a mislaid password (postgres://u:sec:ret@h
// parses "sec:ret@h" as the port). The best-effort-redacted connection
// string in ParseConfigError provides the available context without
// deliberately quoting the offending port text again.
port, err := parsePort(portStr)
if err != nil {
return nil, &ParseConfigError{ConnString: connString, msg: "invalid port", err: err}
return nil, &ParseConfigError{ConnString: connString, msg: "invalid port"}
}
var tlsConfigs []*tls.Config
@@ -423,14 +606,13 @@ func ParseConfigWithOptions(connString string, options ParseConfigOptions) (*Con
config.Fallbacks = fallbacks[1:]
config.SSLNegotiation = settings["sslnegotiation"]
passfile, err := pgpassfile.ReadPassfile(settings["passfile"])
if err == nil {
if config.Password == "" {
if config.Password == "" {
passfile, err := pgpassfile.ReadPassfile(settings["passfile"])
if err == nil {
host := config.Host
if network, _ := NetworkAddress(config.Host, config.Port); network == "unix" {
host = "localhost"
}
config.Password = passfile.FindPassword(host, strconv.Itoa(int(config.Port)), config.Database, config.User)
}
}
@@ -498,6 +680,11 @@ func ParseConfigWithOptions(connString string, options ParseConfigOptions) (*Con
return nil, &ParseConfigError{ConnString: connString, msg: fmt.Sprintf("unknown channel_binding value: %v", channelBinding)}
}
config.RequireAuth = settings["require_auth"]
if _, err := parseRequireAuth(config.RequireAuth); err != nil {
return nil, &ParseConfigError{ConnString: connString, msg: "invalid require_auth", err: err}
}
return config, nil
}
@@ -537,6 +724,8 @@ func parseEnvSettings() map[string]string {
"PGOPTIONS": "options",
"PGMINPROTOCOLVERSION": "min_protocol_version",
"PGMAXPROTOCOLVERSION": "max_protocol_version",
"PGCHANNELBINDING": "channel_binding",
"PGREQUIREAUTH": "require_auth",
}
for envname, realname := range nameMap {
@@ -549,88 +738,49 @@ func parseEnvSettings() map[string]string {
return settings
}
func parseURLSettings(connString string) (map[string]string, error) {
settings := make(map[string]string)
parsedURL, err := url.Parse(connString)
if err != nil {
if urlErr := new(url.Error); errors.As(err, &urlErr) {
return nil, urlErr.Err
}
return nil, err
}
if parsedURL.User != nil {
if u := parsedURL.User.Username(); u != "" {
settings["user"] = u
}
if password, present := parsedURL.User.Password(); present {
settings["password"] = password
}
}
// Handle multiple host:port's in url.Host by splitting them into host,host,host and port,port,port.
var hosts []string
var ports []string
for host := range strings.SplitSeq(parsedURL.Host, ",") {
if host == "" {
continue
}
if isIPOnly(host) {
hosts = append(hosts, strings.Trim(host, "[]"))
continue
}
h, p, err := net.SplitHostPort(host)
if err != nil {
return nil, fmt.Errorf("failed to split host:port in '%s', err: %w", host, err)
}
if h != "" {
hosts = append(hosts, h)
}
if p != "" {
ports = append(ports, p)
}
}
if len(hosts) > 0 {
settings["host"] = strings.Join(hosts, ",")
}
if len(ports) > 0 {
settings["port"] = strings.Join(ports, ",")
}
database := strings.TrimLeft(parsedURL.Path, "/")
if database != "" {
settings["database"] = database
}
nameMap := map[string]string{
"dbname": "database",
}
for k, v := range parsedURL.Query() {
if k2, present := nameMap[k]; present {
k = k2
}
settings[k] = v[0]
}
return settings, nil
}
func isIPOnly(host string) bool {
return net.ParseIP(strings.Trim(host, "[]")) != nil || !strings.Contains(host, ":")
}
var asciiSpace = [256]uint8{'\t': 1, '\n': 1, '\v': 1, '\f': 1, '\r': 1, ' ': 1}
// unescapeKeywordValue applies libpq's backslash rule to the raw text of a
// keyword/value value: a backslash is dropped and whatever follows it is taken
// literally, whatever that character is. A backslash at the very end escapes
// the end of the string, so it is dropped and contributes nothing -- which is
// how libpq accepts `host=a\`. Only the unquoted branch can reach that case;
// inside quotes the escaped terminator leaves the string unterminated and the
// caller has already rejected it.
func unescapeKeywordValue(s string) string {
if !strings.ContainsRune(s, '\\') {
return s
}
var sb strings.Builder
sb.Grow(len(s))
for i := 0; i < len(s); i++ {
if s[i] == '\\' {
i++
if i == len(s) {
break
}
}
sb.WriteByte(s[i])
}
return sb.String()
}
func parseKeywordValueSettings(s string) (map[string]string, error) {
settings := make(map[string]string)
nameMap := map[string]string{
"dbname": "database",
// Reject NUL bytes up front, as parseURLSettings does. libpq never sees one
// because its conninfo strings are NUL-terminated C strings; a Go string can
// carry a NUL through to the startup packet, where it acts as a parameter
// delimiter rather than data. StartupMessage.Encode refuses such parameters,
// but failing here reports the problem against the input that caused it.
if strings.IndexByte(s, 0) >= 0 {
return nil, errors.New("forbidden NUL byte in connection string")
}
// Trim any leading whitespace so that the loop exits cleanly when only
// spaces remain (e.g. trailing spaces after the last value).
s = strings.TrimLeft(s, " \t\n\r\v\f")
for len(s) > 0 {
var key, val string
eqIdx := strings.IndexRune(s, '=')
@@ -639,9 +789,19 @@ func parseKeywordValueSettings(s string) (map[string]string, error) {
}
key = strings.Trim(s[:eqIdx], " \t\n\r\v\f")
// libpq reads a keyword as a run of non-space characters and then
// requires the next non-space character to be '=', so whitespace
// inside a keyword terminates it and is an error. Trimming alone would
// accept the space as part of the key and turn a typo into a bogus
// RuntimeParam that only the server rejects. Report it as libpq does,
// naming the keyword it had read.
if i := strings.IndexAny(key, " \t\n\r\v\f"); i >= 0 {
return nil, fmt.Errorf(`missing "=" after %q in connection info string`, key[:i])
}
s = strings.TrimLeft(s[eqIdx+1:], " \t\n\r\v\f")
if len(s) == 0 {
} else if s[0] != '\'' {
switch {
case len(s) == 0:
case s[0] != '\'':
end := 0
for ; end < len(s); end++ {
if asciiSpace[s[end]] == 1 {
@@ -649,18 +809,19 @@ func parseKeywordValueSettings(s string) (map[string]string, error) {
}
if s[end] == '\\' {
end++
// A trailing backslash escapes the end of the string.
// libpq drops it and ends the value there. Break rather
// than let the loop's post-increment push end past len(s).
if end == len(s) {
return nil, errors.New("invalid backslash")
break
}
}
}
val = strings.Replace(strings.Replace(s[:end], "\\\\", "\\", -1), "\\'", "'", -1)
if end == len(s) {
s = ""
} else {
s = s[end+1:]
}
} else { // quoted string
val = unescapeKeywordValue(s[:end])
// Consume the value and trim any subsequent whitespace so that
// multiple trailing spaces don't cause a spurious parse failure.
s = strings.TrimLeft(s[end:], " \t\n\r\v\f")
default: // quoted string
s = s[1:]
end := 0
for ; end < len(s); end++ {
@@ -669,22 +830,20 @@ func parseKeywordValueSettings(s string) (map[string]string, error) {
}
if s[end] == '\\' {
end++
if end == len(s) {
return nil, errors.New("unterminated quoted string in connection info string")
}
}
}
if end == len(s) {
return nil, errors.New("unterminated quoted string in connection info string")
}
val = strings.Replace(strings.Replace(s[:end], "\\\\", "\\", -1), "\\'", "'", -1)
if end == len(s) {
s = ""
} else {
s = s[end+1:]
}
val = unescapeKeywordValue(s[:end])
// Consume the closing quote and any subsequent whitespace.
s = strings.TrimLeft(s[end+1:], " \t\n\r\v\f")
}
if k, ok := nameMap[key]; ok {
key = k
}
key = canonicalConnStringKey(key)
if key == "" {
return nil, errors.New("invalid keyword/value")
@@ -710,16 +869,9 @@ func parseServiceSettings(servicefilePath, serviceName string) (map[string]strin
return nil, fmt.Errorf("unable to find service: %v", serviceName)
}
nameMap := map[string]string{
"dbname": "database",
}
settings := make(map[string]string, len(service.Settings))
for k, v := range service.Settings {
if k2, present := nameMap[k]; present {
k = k2
}
settings[k] = v
settings[canonicalConnStringKey(k)] = v
}
return settings, nil
@@ -956,6 +1108,17 @@ func makeConnectTimeoutDialFunc(timeout time.Duration) DialFunc {
return d.DialContext
}
var (
// ErrReadOnlyConnection is returned when a read-write connection is required but the connection is read-only.
ErrReadOnlyConnection = errors.New("read only connection")
// ErrReadWriteConnection is returned when a read-only connection is required but the connection is read-write.
ErrReadWriteConnection = errors.New("connection is not read only")
// ErrPrimaryConnection is returned when a standby connection is required but the server is primary.
ErrPrimaryConnection = errors.New("server is not in hot standby mode")
// ErrStandbyConnection is returned when a primary connection is required but the server is in standby mode.
ErrStandbyConnection = errors.New("server is in standby mode")
)
// ValidateConnectTargetSessionAttrsReadWrite is a ValidateConnectFunc that implements libpq compatible
// target_session_attrs=read-write.
func ValidateConnectTargetSessionAttrsReadWrite(ctx context.Context, pgConn *PgConn) error {
@@ -965,7 +1128,7 @@ func ValidateConnectTargetSessionAttrsReadWrite(ctx context.Context, pgConn *PgC
}
if string(result[0].Rows[0][0]) == "on" {
return errors.New("read only connection")
return ErrReadOnlyConnection
}
return nil
@@ -980,7 +1143,7 @@ func ValidateConnectTargetSessionAttrsReadOnly(ctx context.Context, pgConn *PgCo
}
if string(result[0].Rows[0][0]) != "on" {
return errors.New("connection is not read only")
return ErrReadWriteConnection
}
return nil
@@ -995,7 +1158,7 @@ func ValidateConnectTargetSessionAttrsStandby(ctx context.Context, pgConn *PgCon
}
if string(result[0].Rows[0][0]) != "t" {
return errors.New("server is not in hot standby mode")
return ErrPrimaryConnection
}
return nil
@@ -1010,7 +1173,7 @@ func ValidateConnectTargetSessionAttrsPrimary(ctx context.Context, pgConn *PgCon
}
if string(result[0].Rows[0][0]) == "t" {
return errors.New("server is in standby mode")
return ErrStandbyConnection
}
return nil
@@ -1025,7 +1188,7 @@ func ValidateConnectTargetSessionAttrsPreferStandby(ctx context.Context, pgConn
}
if string(result[0].Rows[0][0]) != "t" {
return &NotPreferredError{err: errors.New("server is not in hot standby mode")}
return &NotPreferredError{err: ErrPrimaryConnection}
}
return nil
+57 -14
View File
@@ -1,5 +1,4 @@
//go:build !windows
// +build !windows
package pgconn
@@ -9,22 +8,46 @@ import (
"path/filepath"
)
// currentOSUser resolves the current OS user account. It is a seam so tests
// can simulate os/user.Current failing -- including the unrecoverable
// crashes reported in some restricted/broken-NSS container environments --
// without needing such an environment.
var currentOSUser = user.Current
// userHomeDir resolves the current user's home directory. It is a seam so
// tests can simulate os.UserHomeDir failing without needing to unset $HOME
// for the whole process.
//
// os.UserHomeDir only reads the $HOME environment variable on this platform;
// it never calls into cgo or NSS, so unlike os/user.Current it cannot crash
// in restricted/broken container environments. That also means it can
// resolve a different directory than the OS account's passwd-file home
// directory if $HOME has been overridden -- this is an accepted tradeoff so
// that home-directory-derived defaults (pgpass, pg_service.conf, client SSL
// certificate/key/root files) keep resolving even when the OS user account
// lookup itself is unavailable or unsafe to call.
var userHomeDir = os.UserHomeDir
func defaultSettings() map[string]string {
settings := make(map[string]string)
settings["host"] = defaultHost()
settings["port"] = "5432"
settings["port"] = defaultPort
settings["target_session_attrs"] = "any"
// Default to the OS user name. Purposely ignoring err getting user name from
// OS. The client application will simply have to specify the user in that
// case (which they typically will be doing anyway).
user, err := user.Current()
if err == nil {
settings["user"] = user.Username
settings["passfile"] = filepath.Join(user.HomeDir, ".pgpass")
settings["servicefile"] = filepath.Join(user.HomeDir, ".pg_service.conf")
sslcert := filepath.Join(user.HomeDir, ".postgresql", "postgresql.crt")
sslkey := filepath.Join(user.HomeDir, ".postgresql", "postgresql.key")
// The home-directory-derived defaults (~/.pgpass, ~/.pg_service.conf,
// and the client SSL certificate/key/root files under ~/.postgresql)
// only need the user's home directory, not the full OS user account, so
// they are resolved unconditionally here via os.UserHomeDir. This keeps
// them available even when looking up the OS user account (needed only
// for the default PostgreSQL user name, see osUserSettings) is slow,
// fails, or -- in some restricted container environments with a broken
// NSS/CGO setup -- would crash the process.
if homeDir, err := userHomeDir(); err == nil {
settings["passfile"] = filepath.Join(homeDir, ".pgpass")
settings["servicefile"] = filepath.Join(homeDir, ".pg_service.conf")
sslcert := filepath.Join(homeDir, ".postgresql", "postgresql.crt")
sslkey := filepath.Join(homeDir, ".postgresql", "postgresql.key")
if _, err := os.Stat(sslcert); err == nil {
if _, err := os.Stat(sslkey); err == nil {
// Both the cert and key must be present to use them, or do not use either
@@ -32,13 +55,33 @@ func defaultSettings() map[string]string {
settings["sslkey"] = sslkey
}
}
sslrootcert := filepath.Join(user.HomeDir, ".postgresql", "root.crt")
sslrootcert := filepath.Join(homeDir, ".postgresql", "root.crt")
if _, err := os.Stat(sslrootcert); err == nil {
settings["sslrootcert"] = sslrootcert
}
}
settings["target_session_attrs"] = "any"
return settings
}
// osUserSettings returns the default PostgreSQL user name derived from the
// current OS user account.
//
// Resolving this requires looking up the OS user account, which can be slow
// or, in some restricted container environments (e.g. distroless images
// with a broken NSS/CGO setup), crash the process. Callers must only call
// osUserSettings when the default user name is not already supplied by the
// connection string, environment, or service file.
func osUserSettings() map[string]string {
settings := make(map[string]string)
// Default to the OS user name. Purposely ignoring err getting user name from
// OS. The client application will simply have to specify the user in that
// case (which they typically will be doing anyway).
user, err := currentOSUser()
if err == nil {
settings["user"] = user.Username
}
return settings
}
+67 -17
View File
@@ -7,28 +7,44 @@ import (
"strings"
)
// currentOSUser resolves the current OS user account. It is a seam so tests
// can simulate os/user.Current failing -- including the unrecoverable
// crashes reported in some restricted/broken-NSS container environments --
// without needing such an environment.
var currentOSUser = user.Current
// userHomeDir resolves the current user's home directory. It is a seam so
// tests can simulate os.UserHomeDir failing without needing to unset the
// relevant environment variables for the whole process.
//
// os.UserHomeDir only reads %USERPROFILE% (falling back to
// %HOMEDRIVE%+%HOMEPATH%) on this platform; it never calls into the Windows
// user account APIs, so unlike os/user.Current it cannot crash in
// restricted/broken container environments. That also means it can resolve
// a different directory than the OS account API's reported home directory
// if those environment variables have been overridden -- this is an
// accepted tradeoff so that home-directory-derived defaults (pg_service.conf)
// keep resolving even when the OS user account lookup itself is unavailable
// or unsafe to call.
var userHomeDir = os.UserHomeDir
func defaultSettings() map[string]string {
settings := make(map[string]string)
settings["host"] = defaultHost()
settings["port"] = "5432"
settings["port"] = defaultPort
settings["target_session_attrs"] = "any"
// Default to the OS user name. Purposely ignoring err getting user name from
// OS. The client application will simply have to specify the user in that
// case (which they typically will be doing anyway).
user, err := user.Current()
appData := os.Getenv("APPDATA")
if err == nil {
// Windows gives us the username here as `DOMAIN\user` or `LOCALPCNAME\user`,
// but the libpq default is just the `user` portion, so we strip off the first part.
username := user.Username
if strings.Contains(username, "\\") {
username = username[strings.LastIndex(username, "\\")+1:]
}
settings["user"] = username
// The %APPDATA%\postgresql-derived defaults (pgpass.conf and the client
// SSL certificate/key/root files) only need the APPDATA environment
// variable, not the full OS user account, so they are resolved
// unconditionally here. This keeps them available even when looking up
// the OS user account (needed only for the default PostgreSQL user
// name, see osUserSettings) is slow, fails, or -- in some restricted
// container environments with a broken NSS/CGO setup -- would crash the
// process.
if appData := os.Getenv("APPDATA"); appData != "" {
settings["passfile"] = filepath.Join(appData, "postgresql", "pgpass.conf")
settings["servicefile"] = filepath.Join(user.HomeDir, ".pg_service.conf")
sslcert := filepath.Join(appData, "postgresql", "postgresql.crt")
sslkey := filepath.Join(appData, "postgresql", "postgresql.key")
if _, err := os.Stat(sslcert); err == nil {
@@ -44,7 +60,41 @@ func defaultSettings() map[string]string {
}
}
settings["target_session_attrs"] = "any"
// The default ~/.pg_service.conf location is derived from the user's
// home directory, resolved the same crash-safe way as on other
// platforms (see defaults.go).
if homeDir, err := userHomeDir(); err == nil {
settings["servicefile"] = filepath.Join(homeDir, ".pg_service.conf")
}
return settings
}
// osUserSettings returns the default PostgreSQL user name derived from the
// current OS user account.
//
// Resolving this requires looking up the OS user account, which can be slow
// or, in some restricted container environments (e.g. distroless images
// with a broken NSS/CGO setup), crash the process. Callers must only call
// osUserSettings when the default user name is not already supplied by the
// connection string, environment, or service file.
func osUserSettings() map[string]string {
settings := make(map[string]string)
// Default to the OS user name. Purposely ignoring err getting user name from
// OS. The client application will simply have to specify the user in that
// case (which they typically will be doing anyway).
user, err := currentOSUser()
if err == nil {
// Windows gives us the username here as `DOMAIN\user` or `LOCALPCNAME\user`,
// but the libpq default is just the `user` portion, so we strip off the first part.
username := user.Username
if strings.Contains(username, "\\") {
username = username[strings.LastIndex(username, "\\")+1:]
}
settings["user"] = username
}
return settings
}
+26
View File
@@ -8,6 +8,32 @@ Establishing a Connection
Use Connect to establish a connection. It accepts a connection string in URL or keyword/value format and will read the
environment for libpq style environment variables.
Connecting Securely
By default ParseConfig matches libpq and uses sslmode=prefer, which silently falls back to an unencrypted connection
if the server does not offer TLS. For connections that traverse an untrusted network, set the following parameters
explicitly:
# URL form
postgres://user@db.example.com/mydb?sslmode=verify-full&sslrootcert=/path/to/root.crt&channel_binding=require&require_auth=scram-sha-256
# keyword/value form
host=db.example.com user=user dbname=mydb sslmode=verify-full sslrootcert=/path/to/root.crt channel_binding=require require_auth=scram-sha-256
sslmode=verify-full Require TLS, verify the server certificate against sslrootcert, and verify that the
certificate's host name matches the host being connected to. Weaker modes (disable,
allow, prefer, require) either permit plaintext fallback or skip certificate
verification, allowing a network attacker to impersonate the server.
channel_binding=require Require SCRAM-SHA-256-PLUS, which binds the authentication exchange to the TLS channel
so that a TLS-terminating intermediary cannot relay credentials to the real server.
require_auth=scram-sha-256
Refuse to respond to AuthenticationCleartextPassword or AuthenticationMD5Password
requests from the server. Without this, a server (or interceptor) can request the
password in cleartext and the client will send it.
These parameters may also be set via the PGSSLMODE, PGSSLROOTCERT, PGCHANNELBINDING, and PGREQUIREAUTH environment
variables.
Executing a Query
ExecParams and ExecPrepared execute a single query. They return readers that iterate over each row. The Read method

Some files were not shown because too many files have changed in this diff Show More