* Add handling for pgvector and PostGIS extensions in migration scripts * Implement operator class and storage parameters for vector indexes * Update tests to validate new index behaviors and extension creation
300 lines
9.0 KiB
Markdown
300 lines
9.0 KiB
Markdown
# PostgreSQL Writer
|
|
|
|
Generates PostgreSQL DDL (Data Definition Language) SQL scripts from database schema information.
|
|
|
|
## Overview
|
|
|
|
The PostgreSQL Writer converts RelSpec's internal database model representation into PostgreSQL-compatible SQL DDL scripts, including CREATE TABLE statements, constraints, indexes, views, and sequences.
|
|
|
|
## Features
|
|
|
|
- Generates complete PostgreSQL DDL
|
|
- Creates schemas, tables, columns
|
|
- Defines constraints (PK, FK, unique, check)
|
|
- Creates indexes
|
|
- Generates views and sequences
|
|
- Supports migration scripts
|
|
- Includes audit triggers (optional)
|
|
- Handles PostgreSQL-specific data types
|
|
|
|
## Usage
|
|
|
|
### Basic Example
|
|
|
|
```go
|
|
package main
|
|
|
|
import (
|
|
"git.warky.dev/wdevs/relspecgo/pkg/models"
|
|
"git.warky.dev/wdevs/relspecgo/pkg/writers"
|
|
"git.warky.dev/wdevs/relspecgo/pkg/writers/pgsql"
|
|
)
|
|
|
|
func main() {
|
|
options := &writers.WriterOptions{
|
|
OutputPath: "schema.sql",
|
|
}
|
|
|
|
writer := pgsql.NewWriter(options)
|
|
err := writer.WriteDatabase(db)
|
|
if err != nil {
|
|
panic(err)
|
|
}
|
|
}
|
|
```
|
|
|
|
### CLI Examples
|
|
|
|
```bash
|
|
# Generate PostgreSQL DDL from JSON schema
|
|
relspec --input json \
|
|
--in-file schema.json \
|
|
--output pgsql \
|
|
--out-file schema.sql
|
|
|
|
# Convert GORM models to PostgreSQL DDL
|
|
relspec --input gorm \
|
|
--in-file models.go \
|
|
--output pgsql \
|
|
--out-file create_tables.sql
|
|
|
|
# Export live database schema to SQL
|
|
relspec --input pgsql \
|
|
--conn "postgres://localhost/source_db" \
|
|
--output pgsql \
|
|
--out-file backup_schema.sql
|
|
```
|
|
|
|
## Generated SQL Example
|
|
|
|
```sql
|
|
-- Schema: public
|
|
|
|
CREATE SCHEMA IF NOT EXISTS public;
|
|
|
|
-- Table: public.users
|
|
|
|
CREATE TABLE IF NOT EXISTS public.users (
|
|
id BIGSERIAL PRIMARY KEY,
|
|
username VARCHAR(50) NOT NULL,
|
|
email VARCHAR(100) NOT NULL,
|
|
bio TEXT,
|
|
created_at TIMESTAMP NOT NULL DEFAULT NOW()
|
|
);
|
|
|
|
-- Constraints for public.users
|
|
|
|
ALTER TABLE public.users
|
|
ADD CONSTRAINT uq_users_username UNIQUE (username);
|
|
|
|
-- Indexes for public.users
|
|
|
|
CREATE INDEX idx_users_email ON public.users (email);
|
|
|
|
-- Table: public.posts
|
|
|
|
CREATE TABLE IF NOT EXISTS public.posts (
|
|
id BIGSERIAL PRIMARY KEY,
|
|
user_id BIGINT NOT NULL,
|
|
title VARCHAR(200) NOT NULL,
|
|
content TEXT,
|
|
created_at TIMESTAMP DEFAULT NOW()
|
|
);
|
|
|
|
-- Foreign Keys for public.posts
|
|
|
|
ALTER TABLE public.posts
|
|
ADD CONSTRAINT fk_posts_user_id
|
|
FOREIGN KEY (user_id)
|
|
REFERENCES public.users (id)
|
|
ON DELETE CASCADE
|
|
ON UPDATE NO ACTION;
|
|
|
|
-- Indexes for public.posts
|
|
|
|
CREATE INDEX idx_posts_user_id ON public.posts (user_id);
|
|
```
|
|
|
|
## Writer Options
|
|
|
|
### Metadata Options
|
|
|
|
```go
|
|
options := &writers.WriterOptions{
|
|
OutputPath: "schema.sql",
|
|
Metadata: map[string]interface{}{
|
|
"include_drop": true, // Include DROP statements
|
|
"include_audit": true, // Include audit triggers
|
|
"if_not_exists": true, // Use IF NOT EXISTS
|
|
"migration_mode": false, // Generate migration script
|
|
},
|
|
}
|
|
```
|
|
|
|
## Features
|
|
|
|
### Full DDL Generation
|
|
|
|
Generates complete database structure:
|
|
- CREATE SCHEMA statements
|
|
- CREATE TABLE with all columns and types
|
|
- PRIMARY KEY constraints
|
|
- FOREIGN KEY constraints with actions
|
|
- UNIQUE constraints
|
|
- CHECK constraints
|
|
- CREATE INDEX statements
|
|
- CREATE VIEW statements
|
|
- CREATE SEQUENCE statements
|
|
|
|
### Migration Mode
|
|
|
|
When `migration_mode` is enabled, generates migration scripts with:
|
|
- Version tracking
|
|
- Up/down migrations
|
|
- Transactional DDL
|
|
- Rollback support
|
|
|
|
### Audit Triggers
|
|
|
|
When `include_audit` is enabled, adds:
|
|
- Created/updated timestamp triggers
|
|
- Audit logging functionality
|
|
- Change tracking
|
|
|
|
## PostgreSQL-Specific Features
|
|
|
|
- Serial types (SERIAL, BIGSERIAL)
|
|
- Advanced types (UUID, JSONB, ARRAY)
|
|
- Schema-qualified names
|
|
- Constraint actions (CASCADE, RESTRICT, SET NULL)
|
|
- Partial indexes
|
|
- Function-based indexes
|
|
- Concurrent index creation (`CREATE INDEX CONCURRENTLY`) via `Index.Concurrent`
|
|
- Check constraints with expressions
|
|
- Extension types and indexes: PostGIS, pgvector, citext, hstore, ltree (see below)
|
|
|
|
## Data Types
|
|
|
|
Supports all PostgreSQL data types:
|
|
- Integer types: SMALLINT, INTEGER, BIGINT, SERIAL, BIGSERIAL
|
|
- Numeric types: NUMERIC, DECIMAL, REAL, DOUBLE PRECISION
|
|
- String types: VARCHAR, CHAR, TEXT
|
|
- Date/Time: DATE, TIME, TIMESTAMP, TIMESTAMPTZ, INTERVAL
|
|
- Boolean: BOOLEAN
|
|
- Binary: BYTEA
|
|
- JSON: JSON, JSONB
|
|
- UUID: UUID
|
|
- Network: INET, CIDR, MACADDR
|
|
- Special: ARRAY, HSTORE
|
|
|
|
## Extension Types (PostGIS, pgvector)
|
|
|
|
Extension column types are preserved verbatim, including their type modifier:
|
|
|
|
| Type | Example column type | Extension |
|
|
|------|---------------------|-----------|
|
|
| PostGIS | `geometry(Point,4326)`, `geography(Point)`, `box2d`, `raster` | `postgis`, `postgis_raster`, `postgis_topology` |
|
|
| pgvector | `vector(1536)`, `halfvec(768)`, `sparsevec(1000)` | `vector` |
|
|
| Other | `citext`, `hstore`, `ltree` | `citext`, `hstore`, `ltree` |
|
|
|
|
`CREATE EXTENSION IF NOT EXISTS <ext>;` is emitted automatically for every extension the
|
|
schema needs. See [Extensions](#extensions).
|
|
|
|
### Extension Indexes
|
|
|
|
`Index.Type` selects the access method: `gist`, `spgist`, `brin` (PostGIS), `hnsw`, `ivfflat`
|
|
(pgvector), `vchordrq`, `vchordg` (VectorChord), `bm25` (pg_search).
|
|
|
|
Operator class and access-method parameters ride in `Index.Comment`:
|
|
|
|
```
|
|
opclass=vector_l2_ops; with (lists=100)
|
|
```
|
|
|
|
- `opclass=<name>` — used only when compatible with the column type; otherwise ignored.
|
|
Bare operator class names in the comment (e.g. `gin_trgm_ops`) are also recognized.
|
|
- `with (k=v, …)` — rendered as `WITH (k = v, …)`. Only well-formed `key = value` pairs are
|
|
kept, so comment prose never reaches the DDL. Values may be bare (`lists=100`), quoted
|
|
(`key_field='id'`), or dollar-quoted (`options=$$[build.internal]$$`).
|
|
|
|
Defaults when no operator class is requested:
|
|
|
|
| Access method | Column type | Emitted operator class |
|
|
|---------------|-------------|------------------------|
|
|
| `hnsw`, `ivfflat`, `vchordrq`, `vchordg` | `vector` / `halfvec` / `sparsevec` / `bit` | `vector_cosine_ops` / `halfvec_cosine_ops` / `sparsevec_cosine_ops` / `bit_hamming_ops` |
|
|
| `gist`, `spgist`, `brin` | `geometry`, `geography` | none (PostGIS default operator class) |
|
|
| `gin` | text / `jsonb` / array | `gin_trgm_ops` / `jsonb_ops` / `array_ops` |
|
|
|
|
pgvector defines no default operator class, so a vector index always names one.
|
|
|
|
```sql
|
|
CREATE INDEX IF NOT EXISTS idx_documents_embedding
|
|
ON public.documents USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
|
|
CREATE INDEX IF NOT EXISTS idx_documents_location
|
|
ON public.documents USING gist (location);
|
|
```
|
|
|
|
Migrations only recreate an index when both sides specify a hint and they differ, so a model
|
|
without hints does not churn against a live database.
|
|
|
|
## Extensions
|
|
|
|
`CREATE EXTENSION IF NOT EXISTS <ext>;` is emitted per schema, deduplicated and ordered so
|
|
dependencies come first (`postgis` before `postgis_topology`/`postgis_raster`/`pgrouting`,
|
|
`vector` before `vchord`). Names needing quoting are quoted: `CREATE EXTENSION IF NOT EXISTS "uuid-ossp";`
|
|
|
|
### Detection
|
|
|
|
| Source | Example | Extension |
|
|
|--------|---------|-----------|
|
|
| Column type | `vector(1536)`, `geometry(Point,4326)`, `citext`, `ltree` | `vector`, `postgis`, `citext`, `ltree` |
|
|
| Index access method | `hnsw`, `ivfflat` / `vchordrq`, `vchordg` / `bm25` | `vector` / `vchord` / `pg_search` |
|
|
| Operator class | `gin_trgm_ops`, `gist_ltree_ops` | `pg_trgm`, `ltree` |
|
|
| GIN/GiST on a scalar type | `USING gin (views)` | `btree_gin` / `btree_gist` |
|
|
| Function in a default, CHECK, index `WHERE`, or view body | `uuid_generate_v4()`, `crypt()`, `ST_Area()`, `unaccent()`, `json_matches_schema()` | `uuid-ossp`, `pgcrypto`, `postgis`, `unaccent`, `pg_jsonschema` |
|
|
|
|
`gen_random_uuid()` is built in since PostgreSQL 13 and does not pull in `pgcrypto`.
|
|
|
|
### Declaring extensions explicitly
|
|
|
|
Extensions that leave no trace in the schema go in `schema.Metadata["extensions"]`, as a list
|
|
or a comma-separated string. Dependencies are pulled in automatically; unknown names are kept
|
|
as given. The PostgreSQL reader populates this from `pg_extension` for the schemas it reads.
|
|
|
|
```yaml
|
|
metadata:
|
|
extensions: [pg_cron, timescaledb, pg_stat_statements]
|
|
```
|
|
|
|
### Recognized extensions
|
|
|
|
| Category | Extensions |
|
|
|----------|------------|
|
|
| ai/search | `vector`, `vchord` |
|
|
| document | `hstore`, `ltree` |
|
|
| federation | `postgres_fdw` |
|
|
| geospatial | `postgis`, `postgis_raster`, `postgis_topology`, `pgrouting` |
|
|
| indexing | `btree_gin`, `btree_gist` |
|
|
| integration | `http` |
|
|
| integrity | `amcheck` |
|
|
| jobs / scheduling | `pg_background`, `pg_cron` |
|
|
| maintenance | `pg_repack`, `pgstattuple` |
|
|
| observability | `pg_qualstats`, `pg_stat_statements` |
|
|
| partitioning | `pg_partman` |
|
|
| procedural | `plpython3u` |
|
|
| search | `pg_search`, `pg_textsearch` |
|
|
| security | `pgcrypto` |
|
|
| text | `citext`, `fuzzystrmatch`, `pg_trgm`, `unaccent` |
|
|
| time-series | `timescaledb` |
|
|
| utility | `uuid-ossp` |
|
|
| validation | `pg_jsonschema` |
|
|
|
|
## Notes
|
|
|
|
- Generated SQL is formatted and readable
|
|
- Comments are preserved from source schema
|
|
- Schema names are fully qualified
|
|
- Default values are properly quoted
|
|
- Constraint names follow PostgreSQL conventions
|
|
- Compatible with PostgreSQL 12+
|