feat(pgsql): support vector and PostGIS indexes with extensions
* Add handling for pgvector and PostGIS extensions in migration scripts * Implement operator class and storage parameters for vector indexes * Update tests to validate new index behaviors and extension creation
This commit is contained in:
@@ -171,6 +171,7 @@ When `include_audit` is enabled, adds:
|
||||
- Function-based indexes
|
||||
- Concurrent index creation (`CREATE INDEX CONCURRENTLY`) via `Index.Concurrent`
|
||||
- Check constraints with expressions
|
||||
- Extension types and indexes: PostGIS, pgvector, citext, hstore, ltree (see below)
|
||||
|
||||
## Data Types
|
||||
|
||||
@@ -186,6 +187,108 @@ Supports all PostgreSQL data types:
|
||||
- Network: INET, CIDR, MACADDR
|
||||
- Special: ARRAY, HSTORE
|
||||
|
||||
## Extension Types (PostGIS, pgvector)
|
||||
|
||||
Extension column types are preserved verbatim, including their type modifier:
|
||||
|
||||
| Type | Example column type | Extension |
|
||||
|------|---------------------|-----------|
|
||||
| PostGIS | `geometry(Point,4326)`, `geography(Point)`, `box2d`, `raster` | `postgis`, `postgis_raster`, `postgis_topology` |
|
||||
| pgvector | `vector(1536)`, `halfvec(768)`, `sparsevec(1000)` | `vector` |
|
||||
| Other | `citext`, `hstore`, `ltree` | `citext`, `hstore`, `ltree` |
|
||||
|
||||
`CREATE EXTENSION IF NOT EXISTS <ext>;` is emitted automatically for every extension the
|
||||
schema needs. See [Extensions](#extensions).
|
||||
|
||||
### Extension Indexes
|
||||
|
||||
`Index.Type` selects the access method: `gist`, `spgist`, `brin` (PostGIS), `hnsw`, `ivfflat`
|
||||
(pgvector), `vchordrq`, `vchordg` (VectorChord), `bm25` (pg_search).
|
||||
|
||||
Operator class and access-method parameters ride in `Index.Comment`:
|
||||
|
||||
```
|
||||
opclass=vector_l2_ops; with (lists=100)
|
||||
```
|
||||
|
||||
- `opclass=<name>` — used only when compatible with the column type; otherwise ignored.
|
||||
Bare operator class names in the comment (e.g. `gin_trgm_ops`) are also recognized.
|
||||
- `with (k=v, …)` — rendered as `WITH (k = v, …)`. Only well-formed `key = value` pairs are
|
||||
kept, so comment prose never reaches the DDL. Values may be bare (`lists=100`), quoted
|
||||
(`key_field='id'`), or dollar-quoted (`options=$$[build.internal]$$`).
|
||||
|
||||
Defaults when no operator class is requested:
|
||||
|
||||
| Access method | Column type | Emitted operator class |
|
||||
|---------------|-------------|------------------------|
|
||||
| `hnsw`, `ivfflat`, `vchordrq`, `vchordg` | `vector` / `halfvec` / `sparsevec` / `bit` | `vector_cosine_ops` / `halfvec_cosine_ops` / `sparsevec_cosine_ops` / `bit_hamming_ops` |
|
||||
| `gist`, `spgist`, `brin` | `geometry`, `geography` | none (PostGIS default operator class) |
|
||||
| `gin` | text / `jsonb` / array | `gin_trgm_ops` / `jsonb_ops` / `array_ops` |
|
||||
|
||||
pgvector defines no default operator class, so a vector index always names one.
|
||||
|
||||
```sql
|
||||
CREATE INDEX IF NOT EXISTS idx_documents_embedding
|
||||
ON public.documents USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
|
||||
CREATE INDEX IF NOT EXISTS idx_documents_location
|
||||
ON public.documents USING gist (location);
|
||||
```
|
||||
|
||||
Migrations only recreate an index when both sides specify a hint and they differ, so a model
|
||||
without hints does not churn against a live database.
|
||||
|
||||
## Extensions
|
||||
|
||||
`CREATE EXTENSION IF NOT EXISTS <ext>;` is emitted per schema, deduplicated and ordered so
|
||||
dependencies come first (`postgis` before `postgis_topology`/`postgis_raster`/`pgrouting`,
|
||||
`vector` before `vchord`). Names needing quoting are quoted: `CREATE EXTENSION IF NOT EXISTS "uuid-ossp";`
|
||||
|
||||
### Detection
|
||||
|
||||
| Source | Example | Extension |
|
||||
|--------|---------|-----------|
|
||||
| Column type | `vector(1536)`, `geometry(Point,4326)`, `citext`, `ltree` | `vector`, `postgis`, `citext`, `ltree` |
|
||||
| Index access method | `hnsw`, `ivfflat` / `vchordrq`, `vchordg` / `bm25` | `vector` / `vchord` / `pg_search` |
|
||||
| Operator class | `gin_trgm_ops`, `gist_ltree_ops` | `pg_trgm`, `ltree` |
|
||||
| GIN/GiST on a scalar type | `USING gin (views)` | `btree_gin` / `btree_gist` |
|
||||
| Function in a default, CHECK, index `WHERE`, or view body | `uuid_generate_v4()`, `crypt()`, `ST_Area()`, `unaccent()`, `json_matches_schema()` | `uuid-ossp`, `pgcrypto`, `postgis`, `unaccent`, `pg_jsonschema` |
|
||||
|
||||
`gen_random_uuid()` is built in since PostgreSQL 13 and does not pull in `pgcrypto`.
|
||||
|
||||
### Declaring extensions explicitly
|
||||
|
||||
Extensions that leave no trace in the schema go in `schema.Metadata["extensions"]`, as a list
|
||||
or a comma-separated string. Dependencies are pulled in automatically; unknown names are kept
|
||||
as given. The PostgreSQL reader populates this from `pg_extension` for the schemas it reads.
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
extensions: [pg_cron, timescaledb, pg_stat_statements]
|
||||
```
|
||||
|
||||
### Recognized extensions
|
||||
|
||||
| Category | Extensions |
|
||||
|----------|------------|
|
||||
| ai/search | `vector`, `vchord` |
|
||||
| document | `hstore`, `ltree` |
|
||||
| federation | `postgres_fdw` |
|
||||
| geospatial | `postgis`, `postgis_raster`, `postgis_topology`, `pgrouting` |
|
||||
| indexing | `btree_gin`, `btree_gist` |
|
||||
| integration | `http` |
|
||||
| integrity | `amcheck` |
|
||||
| jobs / scheduling | `pg_background`, `pg_cron` |
|
||||
| maintenance | `pg_repack`, `pgstattuple` |
|
||||
| observability | `pg_qualstats`, `pg_stat_statements` |
|
||||
| partitioning | `pg_partman` |
|
||||
| procedural | `plpython3u` |
|
||||
| search | `pg_search`, `pg_textsearch` |
|
||||
| security | `pgcrypto` |
|
||||
| text | `citext`, `fuzzystrmatch`, `pg_trgm`, `unaccent` |
|
||||
| time-series | `timescaledb` |
|
||||
| utility | `uuid-ossp` |
|
||||
| validation | `pg_jsonschema` |
|
||||
|
||||
## Notes
|
||||
|
||||
- Generated SQL is formatted and readable
|
||||
|
||||
@@ -0,0 +1,260 @@
|
||||
package pgsql
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.warky.dev/wdevs/relspecgo/pkg/models"
|
||||
"git.warky.dev/wdevs/relspecgo/pkg/writers"
|
||||
)
|
||||
|
||||
// buildExtensionSchema returns a single-table schema the extension detection tests mutate.
|
||||
func buildExtensionSchema(t *testing.T) (*models.Schema, *models.Table) {
|
||||
t.Helper()
|
||||
|
||||
schema := models.InitSchema("public")
|
||||
table := models.InitTable("documents", "public")
|
||||
schema.Tables = append(schema.Tables, table)
|
||||
return schema, table
|
||||
}
|
||||
|
||||
func addColumn(table *models.Table, name, sqlType string) *models.Column {
|
||||
col := models.InitColumn(name, table.Name, table.Schema)
|
||||
col.Type = sqlType
|
||||
table.Columns[name] = col
|
||||
return col
|
||||
}
|
||||
|
||||
func TestRequiredExtensions_Detection(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
build func(schema *models.Schema, table *models.Table)
|
||||
want []string
|
||||
}{
|
||||
{
|
||||
name: "no extensions",
|
||||
build: func(_ *models.Schema, table *models.Table) { addColumn(table, "id", "integer") },
|
||||
want: nil,
|
||||
},
|
||||
{
|
||||
name: "column type",
|
||||
build: func(_ *models.Schema, table *models.Table) {
|
||||
addColumn(table, "embedding", "vector(1536)")
|
||||
addColumn(table, "name", "citext")
|
||||
},
|
||||
want: []string{"citext", "vector"},
|
||||
},
|
||||
{
|
||||
name: "column default function",
|
||||
build: func(_ *models.Schema, table *models.Table) {
|
||||
addColumn(table, "id", "uuid").Default = "uuid_generate_v4()"
|
||||
},
|
||||
want: []string{"uuid-ossp"},
|
||||
},
|
||||
{
|
||||
name: "check constraint expression",
|
||||
build: func(_ *models.Schema, table *models.Table) {
|
||||
addColumn(table, "geom", "geometry")
|
||||
table.Constraints["chk_geom"] = &models.Constraint{
|
||||
Name: "chk_geom",
|
||||
Type: models.CheckConstraint,
|
||||
Expression: "ST_IsValid(geom)",
|
||||
}
|
||||
},
|
||||
want: []string{"postgis"},
|
||||
},
|
||||
{
|
||||
name: "partial index predicate",
|
||||
build: func(_ *models.Schema, table *models.Table) {
|
||||
addColumn(table, "title", "text")
|
||||
table.Indexes["idx_title"] = &models.Index{
|
||||
Name: "idx_title",
|
||||
Type: "btree",
|
||||
Columns: []string{"title"},
|
||||
Where: "similarity(title, 'x') > 0.3",
|
||||
}
|
||||
},
|
||||
want: []string{"pg_trgm"},
|
||||
},
|
||||
{
|
||||
name: "view definition",
|
||||
build: func(schema *models.Schema, table *models.Table) {
|
||||
addColumn(table, "title", "text")
|
||||
schema.Views = append(schema.Views, &models.View{
|
||||
Name: "v_documents",
|
||||
Schema: "public",
|
||||
Definition: "SELECT unaccent(title) FROM documents",
|
||||
})
|
||||
},
|
||||
want: []string{"unaccent"},
|
||||
},
|
||||
{
|
||||
name: "index access method",
|
||||
build: func(_ *models.Schema, table *models.Table) {
|
||||
addColumn(table, "body", "text")
|
||||
table.Indexes["idx_body"] = &models.Index{
|
||||
Name: "idx_body",
|
||||
Type: "bm25",
|
||||
Columns: []string{"body"},
|
||||
Comment: "with (key_field='id')",
|
||||
}
|
||||
},
|
||||
want: []string{"pg_search"},
|
||||
},
|
||||
{
|
||||
name: "vchord depends on vector",
|
||||
build: func(_ *models.Schema, table *models.Table) {
|
||||
addColumn(table, "embedding", "vector(3)")
|
||||
table.Indexes["idx_embedding"] = &models.Index{
|
||||
Name: "idx_embedding",
|
||||
Type: "vchordrq",
|
||||
Columns: []string{"embedding"},
|
||||
}
|
||||
},
|
||||
want: []string{"vector", "vchord"},
|
||||
},
|
||||
{
|
||||
name: "gin on scalar needs btree_gin",
|
||||
build: func(_ *models.Schema, table *models.Table) {
|
||||
addColumn(table, "views", "integer")
|
||||
table.Indexes["idx_views"] = &models.Index{
|
||||
Name: "idx_views",
|
||||
Type: "gin",
|
||||
Columns: []string{"views"},
|
||||
}
|
||||
},
|
||||
want: []string{"btree_gin"},
|
||||
},
|
||||
{
|
||||
name: "gist on scalar needs btree_gist",
|
||||
build: func(_ *models.Schema, table *models.Table) {
|
||||
addColumn(table, "views", "integer")
|
||||
table.Indexes["idx_views"] = &models.Index{
|
||||
Name: "idx_views",
|
||||
Type: "gist",
|
||||
Columns: []string{"views"},
|
||||
}
|
||||
},
|
||||
want: []string{"btree_gist"},
|
||||
},
|
||||
{
|
||||
name: "gist on geometry uses postgis operator classes",
|
||||
build: func(_ *models.Schema, table *models.Table) {
|
||||
addColumn(table, "location", "geometry(Point,4326)")
|
||||
table.Indexes["idx_location"] = &models.Index{
|
||||
Name: "idx_location",
|
||||
Type: "gist",
|
||||
Columns: []string{"location"},
|
||||
}
|
||||
},
|
||||
want: []string{"postgis"},
|
||||
},
|
||||
{
|
||||
name: "gin on jsonb needs no companion",
|
||||
build: func(_ *models.Schema, table *models.Table) {
|
||||
addColumn(table, "payload", "jsonb")
|
||||
table.Indexes["idx_payload"] = &models.Index{
|
||||
Name: "idx_payload",
|
||||
Type: "gin",
|
||||
Columns: []string{"payload"},
|
||||
}
|
||||
},
|
||||
want: nil,
|
||||
},
|
||||
{
|
||||
name: "gin on text uses pg_trgm",
|
||||
build: func(_ *models.Schema, table *models.Table) {
|
||||
addColumn(table, "title", "text")
|
||||
table.Indexes["idx_title"] = &models.Index{
|
||||
Name: "idx_title",
|
||||
Type: "gin",
|
||||
Columns: []string{"title"},
|
||||
}
|
||||
},
|
||||
want: []string{"pg_trgm"},
|
||||
},
|
||||
{
|
||||
name: "gin on array needs no companion",
|
||||
build: func(_ *models.Schema, table *models.Table) {
|
||||
addColumn(table, "tags", "text[]")
|
||||
table.Indexes["idx_tags"] = &models.Index{
|
||||
Name: "idx_tags",
|
||||
Type: "gin",
|
||||
Columns: []string{"tags"},
|
||||
}
|
||||
},
|
||||
want: nil,
|
||||
},
|
||||
{
|
||||
name: "declared in metadata as string",
|
||||
build: func(schema *models.Schema, _ *models.Table) {
|
||||
schema.Metadata = map[string]any{"extensions": "pg_cron, timescaledb"}
|
||||
},
|
||||
want: []string{"pg_cron", "timescaledb"},
|
||||
},
|
||||
{
|
||||
name: "declared in metadata as list",
|
||||
build: func(schema *models.Schema, _ *models.Table) {
|
||||
schema.Metadata = map[string]any{"extensions": []any{"postgis_topology", "pg_stat_statements"}}
|
||||
},
|
||||
// postgis is pulled in as a dependency of postgis_topology and emitted first.
|
||||
want: []string{"pg_stat_statements", "postgis", "postgis_topology"},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
schema, table := buildExtensionSchema(t)
|
||||
tt.build(schema, table)
|
||||
|
||||
if got := requiredExtensions(schema); !reflect.DeepEqual(got, tt.want) {
|
||||
t.Errorf("requiredExtensions() = %v, want %v", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRequiredExtensions_NilSchema(t *testing.T) {
|
||||
if got := requiredExtensions(nil); got != nil {
|
||||
t.Errorf("requiredExtensions(nil) = %v, want nil", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteDatabase_QuotesExtensionNames(t *testing.T) {
|
||||
db := models.InitDatabase("testdb")
|
||||
schema, table := buildExtensionSchema(t)
|
||||
addColumn(table, "id", "uuid").Default = "uuid_generate_v4()"
|
||||
db.Schemas = append(db.Schemas, schema)
|
||||
|
||||
output := writeDatabaseOutput(t, db)
|
||||
if !strings.Contains(output, `CREATE EXTENSION IF NOT EXISTS "uuid-ossp";`) {
|
||||
t.Fatalf("expected quoted extension name, got:\n%s", output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateSchemaStatements_ExtensionDependencyOrder(t *testing.T) {
|
||||
schema, table := buildExtensionSchema(t)
|
||||
addColumn(table, "embedding", "vector(3)")
|
||||
table.Indexes["idx_embedding"] = &models.Index{
|
||||
Name: "idx_embedding",
|
||||
Type: "vchordrq",
|
||||
Columns: []string{"embedding"},
|
||||
}
|
||||
|
||||
writer := NewWriter(&writers.WriterOptions{})
|
||||
statements, err := writer.GenerateSchemaStatements(schema)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateSchemaStatements failed: %v", err)
|
||||
}
|
||||
|
||||
joined := strings.Join(statements, "\n")
|
||||
vector := strings.Index(joined, "CREATE EXTENSION IF NOT EXISTS vector")
|
||||
vchord := strings.Index(joined, "CREATE EXTENSION IF NOT EXISTS vchord")
|
||||
if vector < 0 || vchord < 0 {
|
||||
t.Fatalf("expected vector and vchord extensions, got:\n%s", joined)
|
||||
}
|
||||
if vector > vchord {
|
||||
t.Fatalf("expected vector to be created before vchord, got:\n%s", joined)
|
||||
}
|
||||
}
|
||||
@@ -164,14 +164,14 @@ func (w *MigrationWriter) WriteMigration(model *models.Database, current *models
|
||||
func (w *MigrationWriter) generateSchemaScripts(model *models.Schema, current *models.Schema) ([]MigrationScript, error) {
|
||||
scripts := make([]MigrationScript, 0)
|
||||
|
||||
if schemaRequiresPGTrgm(model) {
|
||||
for _, extension := range requiredExtensions(model) {
|
||||
scripts = append(scripts, MigrationScript{
|
||||
ObjectName: "extension.pg_trgm",
|
||||
ObjectName: "extension." + extension,
|
||||
ObjectType: "create extension",
|
||||
Schema: model.Name,
|
||||
Priority: 80,
|
||||
Sequence: len(scripts),
|
||||
Body: "CREATE EXTENSION IF NOT EXISTS pg_trgm;",
|
||||
Body: fmt.Sprintf("CREATE EXTENSION IF NOT EXISTS %s;", pgsql.QuoteExtensionName(extension)),
|
||||
})
|
||||
}
|
||||
|
||||
@@ -646,13 +646,14 @@ func (w *MigrationWriter) generateIndexScripts(model *models.Schema, current *mo
|
||||
}
|
||||
|
||||
sql, err := w.executor.ExecuteCreateIndex(CreateIndexData{
|
||||
SchemaName: model.Name,
|
||||
TableName: modelTable.Name,
|
||||
IndexName: indexName,
|
||||
IndexType: indexType,
|
||||
Columns: strings.Join(columnExprs, ", "),
|
||||
Unique: modelIndex.Unique,
|
||||
Concurrent: modelIndex.Concurrent,
|
||||
SchemaName: model.Name,
|
||||
TableName: modelTable.Name,
|
||||
IndexName: indexName,
|
||||
IndexType: indexType,
|
||||
Columns: strings.Join(columnExprs, ", "),
|
||||
Unique: modelIndex.Unique,
|
||||
Concurrent: modelIndex.Concurrent,
|
||||
StorageParameters: indexStorageParameters(modelIndex.Comment),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -674,20 +675,31 @@ func (w *MigrationWriter) generateIndexScripts(model *models.Schema, current *mo
|
||||
return scripts, nil
|
||||
}
|
||||
|
||||
// buildIndexColumnExpressions renders the column list of an index, appending the operator
|
||||
// class each column needs for the access method (GIN opclasses, pgvector distance ops,
|
||||
// explicitly requested PostGIS opclasses). Columns that cannot be resolved on the table are
|
||||
// emitted verbatim.
|
||||
func buildIndexColumnExpressions(table *models.Table, index *models.Index, indexType string) []string {
|
||||
return buildIndexColumnExpressionsFiltered(table, index, indexType, false)
|
||||
}
|
||||
|
||||
// buildIndexColumnExpressionsFiltered is buildIndexColumnExpressions with the option to drop
|
||||
// columns that do not exist on the table instead of emitting them verbatim.
|
||||
func buildIndexColumnExpressionsFiltered(table *models.Table, index *models.Index, indexType string, skipUnresolved bool) []string {
|
||||
columnExprs := make([]string, 0, len(index.Columns))
|
||||
for _, colName := range index.Columns {
|
||||
colExpr := colName
|
||||
if table != nil {
|
||||
if col, ok := resolveIndexColumn(table, colName); ok && col != nil {
|
||||
colExpr = col.SQLName()
|
||||
if strings.EqualFold(indexType, "gin") {
|
||||
opClass := ginOperatorClassForColumn(col, index.Comment)
|
||||
if opClass != "" {
|
||||
colExpr = fmt.Sprintf("%s %s", col.SQLName(), opClass)
|
||||
}
|
||||
}
|
||||
col, ok := resolveIndexColumn(table, colName)
|
||||
if !ok || col == nil {
|
||||
if skipUnresolved {
|
||||
continue
|
||||
}
|
||||
columnExprs = append(columnExprs, colName)
|
||||
continue
|
||||
}
|
||||
|
||||
colExpr := col.SQLName()
|
||||
if opClass := indexOperatorClassForColumn(col, indexType, index.Comment); opClass != "" {
|
||||
colExpr = fmt.Sprintf("%s %s", colExpr, opClass)
|
||||
}
|
||||
columnExprs = append(columnExprs, colExpr)
|
||||
}
|
||||
@@ -1046,5 +1058,19 @@ func indexesEqual(idx1, idx2 *models.Index) bool {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
// Operator class and storage parameters ride along in the index comment. They only
|
||||
// signal a difference when both sides specify one, so an index whose model side omits
|
||||
// the hint is not recreated on every migration.
|
||||
if !indexHintsEqual(extractOperatorClass(idx1.Comment), extractOperatorClass(idx2.Comment)) {
|
||||
return false
|
||||
}
|
||||
return indexHintsEqual(indexStorageParameters(idx1.Comment), indexStorageParameters(idx2.Comment))
|
||||
}
|
||||
|
||||
// indexHintsEqual compares two optional index hints, treating an unspecified hint as a match.
|
||||
func indexHintsEqual(hint1, hint2 string) bool {
|
||||
if hint1 == "" || hint2 == "" {
|
||||
return true
|
||||
}
|
||||
return strings.EqualFold(hint1, hint2)
|
||||
}
|
||||
|
||||
@@ -852,3 +852,93 @@ func TestWriteMigration_NilCurrentTreatsDatabaseAsEmpty(t *testing.T) {
|
||||
t.Fatalf("expected CREATE TABLE in migration output, got:\n%s", output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteMigration_VectorAndPostGISIndexes(t *testing.T) {
|
||||
current := models.InitDatabase("testdb")
|
||||
current.Schemas = append(current.Schemas, models.InitSchema("public"))
|
||||
|
||||
model := models.InitDatabase("testdb")
|
||||
modelSchema := models.InitSchema("public")
|
||||
|
||||
table := models.InitTable("documents", "public")
|
||||
|
||||
embedding := models.InitColumn("embedding", "documents", "public")
|
||||
embedding.Type = "vector(1536)"
|
||||
table.Columns["embedding"] = embedding
|
||||
|
||||
location := models.InitColumn("location", "documents", "public")
|
||||
location.Type = "geometry(Point,4326)"
|
||||
table.Columns["location"] = location
|
||||
|
||||
table.Indexes["idx_documents_embedding"] = &models.Index{
|
||||
Name: "idx_documents_embedding",
|
||||
Type: "ivfflat",
|
||||
Columns: []string{"embedding"},
|
||||
Comment: "opclass=vector_cosine_ops; with (lists=100)",
|
||||
}
|
||||
table.Indexes["idx_documents_location"] = &models.Index{
|
||||
Name: "idx_documents_location",
|
||||
Type: "gist",
|
||||
Columns: []string{"location"},
|
||||
}
|
||||
|
||||
modelSchema.Tables = append(modelSchema.Tables, table)
|
||||
model.Schemas = append(model.Schemas, modelSchema)
|
||||
|
||||
var buf bytes.Buffer
|
||||
writer, err := NewMigrationWriter(&writers.WriterOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create writer: %v", err)
|
||||
}
|
||||
writer.writer = &buf
|
||||
|
||||
if err := writer.WriteMigration(model, current); err != nil {
|
||||
t.Fatalf("WriteMigration failed: %v", err)
|
||||
}
|
||||
|
||||
output := buf.String()
|
||||
for _, want := range []string{
|
||||
"CREATE EXTENSION IF NOT EXISTS postgis;",
|
||||
"CREATE EXTENSION IF NOT EXISTS vector;",
|
||||
"vector(1536)",
|
||||
"geometry(Point,4326)",
|
||||
"USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100)",
|
||||
"USING gist (location)",
|
||||
} {
|
||||
if !strings.Contains(output, want) {
|
||||
t.Fatalf("expected migration to contain %q, got:\n%s", want, output)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestIndexesEqual_OperatorClassAndStorageParameters(t *testing.T) {
|
||||
newIndex := func(comment string) *models.Index {
|
||||
return &models.Index{
|
||||
Name: "idx_documents_embedding",
|
||||
Type: "hnsw",
|
||||
Columns: []string{"embedding"},
|
||||
Comment: comment,
|
||||
}
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
comment1 string
|
||||
comment2 string
|
||||
wantEqual bool
|
||||
}{
|
||||
{"identical hints", "opclass=vector_l2_ops", "opclass=vector_l2_ops", true},
|
||||
{"different operator class", "opclass=vector_l2_ops", "opclass=vector_cosine_ops", false},
|
||||
{"different storage parameters", "with (m=16)", "with (m=32)", false},
|
||||
{"unspecified hint on one side", "", "opclass=vector_l2_ops; with (m=16)", true},
|
||||
{"unrelated comments", "primary lookup index", "primary lookup index", true},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if got := indexesEqual(newIndex(tt.comment1), newIndex(tt.comment2)); got != tt.wantEqual {
|
||||
t.Errorf("indexesEqual() = %v, want %v", got, tt.wantEqual)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -140,6 +140,9 @@ type CreateIndexData struct {
|
||||
Columns string
|
||||
Unique bool
|
||||
Concurrent bool
|
||||
// StorageParameters holds access-method parameters rendered as WITH (...),
|
||||
// e.g. "lists = 100" for ivfflat or "m = 16, ef_construction = 64" for hnsw.
|
||||
StorageParameters string
|
||||
}
|
||||
|
||||
// CreateForeignKeyData contains data for create foreign key template
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
CREATE {{if .Unique}}UNIQUE {{end}}INDEX {{if .Concurrent}}CONCURRENTLY {{end}}IF NOT EXISTS {{quote_ident .IndexName}}
|
||||
ON {{qual_table .SchemaName .TableName}} USING {{.IndexType}} ({{.Columns}});
|
||||
ON {{qual_table .SchemaName .TableName}} USING {{.IndexType}} ({{.Columns}}){{if .StorageParameters}} WITH ({{.StorageParameters}}){{end}};
|
||||
+337
-58
@@ -6,8 +6,10 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"git.warky.dev/wdevs/relspecgo/pkg/models"
|
||||
@@ -147,8 +149,8 @@ func (w *Writer) GenerateSchemaStatements(schema *models.Schema) ([]string, erro
|
||||
statements = append(statements, fmt.Sprintf("CREATE SCHEMA IF NOT EXISTS %s", schema.SQLName()))
|
||||
}
|
||||
|
||||
if schemaRequiresPGTrgm(schema) {
|
||||
statements = append(statements, `CREATE EXTENSION IF NOT EXISTS pg_trgm`)
|
||||
for _, extension := range requiredExtensions(schema) {
|
||||
statements = append(statements, fmt.Sprintf("CREATE EXTENSION IF NOT EXISTS %s", pgsql.QuoteExtensionName(extension)))
|
||||
}
|
||||
|
||||
// Phase 2: Create sequences
|
||||
@@ -271,18 +273,12 @@ func (w *Writer) GenerateSchemaStatements(schema *models.Schema) ([]string, erro
|
||||
indexType = "btree"
|
||||
}
|
||||
|
||||
// Build column expressions with operator class support for GIN indexes
|
||||
columnExprs := make([]string, 0, len(index.Columns))
|
||||
for _, colName := range index.Columns {
|
||||
colExpr := colName
|
||||
if col, ok := resolveIndexColumn(table, colName); ok {
|
||||
if strings.EqualFold(indexType, "gin") {
|
||||
if opClass := ginOperatorClassForColumn(col, index.Comment); opClass != "" {
|
||||
colExpr = fmt.Sprintf("%s %s", colName, opClass)
|
||||
}
|
||||
}
|
||||
}
|
||||
columnExprs = append(columnExprs, colExpr)
|
||||
// Build column expressions with operator class support (GIN, pgvector, PostGIS)
|
||||
columnExprs := buildIndexColumnExpressions(table, index, indexType)
|
||||
|
||||
withClause := ""
|
||||
if params := indexStorageParameters(index.Comment); params != "" {
|
||||
withClause = fmt.Sprintf(" WITH (%s)", params)
|
||||
}
|
||||
|
||||
whereClause := ""
|
||||
@@ -290,8 +286,8 @@ func (w *Writer) GenerateSchemaStatements(schema *models.Schema) ([]string, erro
|
||||
whereClause = fmt.Sprintf(" WHERE %s", index.Where)
|
||||
}
|
||||
|
||||
stmt := fmt.Sprintf("CREATE %sINDEX IF NOT EXISTS %s ON %s USING %s (%s)%s",
|
||||
uniqueStr, quoteIdentifier(index.Name), w.qualTable(schema.SQLName(), table.SQLName()), indexType, strings.Join(columnExprs, ", "), whereClause)
|
||||
stmt := fmt.Sprintf("CREATE %sINDEX IF NOT EXISTS %s ON %s USING %s (%s)%s%s",
|
||||
uniqueStr, quoteIdentifier(index.Name), w.qualTable(schema.SQLName(), table.SQLName()), indexType, strings.Join(columnExprs, ", "), withClause, whereClause)
|
||||
statements = append(statements, stmt)
|
||||
}
|
||||
}
|
||||
@@ -819,11 +815,14 @@ func (w *Writer) writeCreateSchema(schema *models.Schema) error {
|
||||
}
|
||||
|
||||
func (w *Writer) writeRequiredExtensions(schema *models.Schema) error {
|
||||
if !schemaRequiresPGTrgm(schema) {
|
||||
extensions := requiredExtensions(schema)
|
||||
if len(extensions) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
fmt.Fprintln(w.writer, "CREATE EXTENSION IF NOT EXISTS pg_trgm;")
|
||||
for _, extension := range extensions {
|
||||
fmt.Fprintf(w.writer, "CREATE EXTENSION IF NOT EXISTS %s;\n", pgsql.QuoteExtensionName(extension))
|
||||
}
|
||||
fmt.Fprintln(w.writer)
|
||||
return nil
|
||||
}
|
||||
@@ -1063,21 +1062,13 @@ func (w *Writer) writeIndexes(schema *models.Schema) error {
|
||||
indexName = fmt.Sprintf("%s_%s_%s", indexType, table.SQLName(), strings.ToLower(columnSuffix))
|
||||
}
|
||||
|
||||
// Build column list with operator class support for GIN indexes
|
||||
columnExprs := make([]string, 0, len(index.Columns))
|
||||
for _, colName := range index.Columns {
|
||||
if col, ok := resolveIndexColumn(table, colName); ok {
|
||||
colExpr := col.SQLName()
|
||||
if strings.EqualFold(index.Type, "gin") {
|
||||
opClass := ginOperatorClassForColumn(col, index.Comment)
|
||||
if opClass != "" {
|
||||
colExpr = fmt.Sprintf("%s %s", col.SQLName(), opClass)
|
||||
}
|
||||
}
|
||||
columnExprs = append(columnExprs, colExpr)
|
||||
}
|
||||
indexType := index.Type
|
||||
if indexType == "" {
|
||||
indexType = "btree"
|
||||
}
|
||||
|
||||
// Build column list with operator class support (GIN, pgvector, PostGIS)
|
||||
columnExprs := buildIndexColumnExpressionsFiltered(table, index, indexType, true)
|
||||
if len(columnExprs) == 0 {
|
||||
continue
|
||||
}
|
||||
@@ -1087,9 +1078,9 @@ func (w *Writer) writeIndexes(schema *models.Schema) error {
|
||||
unique = "UNIQUE "
|
||||
}
|
||||
|
||||
indexType := index.Type
|
||||
if indexType == "" {
|
||||
indexType = "btree"
|
||||
withClause := ""
|
||||
if params := indexStorageParameters(index.Comment); params != "" {
|
||||
withClause = fmt.Sprintf(" WITH (%s)", params)
|
||||
}
|
||||
|
||||
whereClause := ""
|
||||
@@ -1104,8 +1095,8 @@ func (w *Writer) writeIndexes(schema *models.Schema) error {
|
||||
|
||||
fmt.Fprintf(w.writer, "CREATE %sINDEX %sIF NOT EXISTS %s\n",
|
||||
unique, concurrently, indexName)
|
||||
fmt.Fprintf(w.writer, " ON %s USING %s (%s)%s;\n\n",
|
||||
w.qualTable(schema.SQLName(), table.SQLName()), indexType, strings.Join(columnExprs, ", "), whereClause)
|
||||
fmt.Fprintf(w.writer, " ON %s USING %s (%s)%s%s;\n\n",
|
||||
w.qualTable(schema.SQLName(), table.SQLName()), indexType, strings.Join(columnExprs, ", "), withClause, whereClause)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1483,7 +1474,69 @@ func isTextTypeWithoutLength(colType string) bool {
|
||||
return strings.EqualFold(colType, "text")
|
||||
}
|
||||
|
||||
func ginOperatorClassForColumn(col *models.Column, comment string) string {
|
||||
// vectorOperatorClasses maps pgvector operator classes to the column base type they
|
||||
// apply to. pgvector defines no default operator class, so an hnsw/ivfflat index must
|
||||
// always name one explicitly.
|
||||
var vectorOperatorClasses = map[string]string{
|
||||
"vector_l2_ops": "vector",
|
||||
"vector_ip_ops": "vector",
|
||||
"vector_cosine_ops": "vector",
|
||||
"vector_l1_ops": "vector",
|
||||
"halfvec_l2_ops": "halfvec",
|
||||
"halfvec_ip_ops": "halfvec",
|
||||
"halfvec_cosine_ops": "halfvec",
|
||||
"halfvec_l1_ops": "halfvec",
|
||||
"sparsevec_l2_ops": "sparsevec",
|
||||
"sparsevec_ip_ops": "sparsevec",
|
||||
"sparsevec_cosine_ops": "sparsevec",
|
||||
"sparsevec_l1_ops": "sparsevec",
|
||||
"bit_hamming_ops": "bit",
|
||||
"bit_jaccard_ops": "bit",
|
||||
}
|
||||
|
||||
// defaultVectorOperatorClasses is the operator class used for an hnsw/ivfflat index when
|
||||
// the index comment does not request one. Cosine distance is the common default for
|
||||
// embedding columns; override it with an "opclass" hint in the index comment.
|
||||
var defaultVectorOperatorClasses = map[string]string{
|
||||
"vector": "vector_cosine_ops",
|
||||
"halfvec": "halfvec_cosine_ops",
|
||||
"sparsevec": "sparsevec_cosine_ops",
|
||||
"bit": "bit_hamming_ops",
|
||||
}
|
||||
|
||||
// spatialOperatorClasses are the PostGIS operator classes recognized in index comments.
|
||||
// PostGIS installs default operator classes for gist/spgist/brin, so these are only
|
||||
// emitted when explicitly requested (e.g. the 3D/nD variants).
|
||||
var spatialOperatorClasses = map[string]bool{
|
||||
"gist_geometry_ops_2d": true,
|
||||
"gist_geometry_ops_nd": true,
|
||||
"gist_geography_ops": true,
|
||||
"spgist_geometry_ops_2d": true,
|
||||
"spgist_geometry_ops_3d": true,
|
||||
"spgist_geometry_ops_nd": true,
|
||||
"brin_geometry_inclusion_ops_2d": true,
|
||||
"brin_geometry_inclusion_ops_3d": true,
|
||||
"brin_geometry_inclusion_ops_4d": true,
|
||||
"brin_geography_inclusion_ops_2d": true,
|
||||
"btree_geometry_ops": true,
|
||||
"btree_geography_ops": true,
|
||||
}
|
||||
|
||||
// isVectorIndexMethod reports whether the access method indexes pgvector types, which
|
||||
// covers both pgvector itself (hnsw, ivfflat) and VectorChord (vchordrq, vchordg).
|
||||
func isVectorIndexMethod(method string) bool {
|
||||
switch strings.ToLower(strings.TrimSpace(method)) {
|
||||
case "hnsw", "ivfflat", "vchordrq", "vchordg":
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// indexOperatorClassForColumn returns the operator class to emit for a column in an index
|
||||
// of the given access method, honouring an explicit request from the index comment when it
|
||||
// is compatible with the column type.
|
||||
func indexOperatorClassForColumn(col *models.Column, indexType, comment string) string {
|
||||
if col == nil {
|
||||
return ""
|
||||
}
|
||||
@@ -1492,26 +1545,53 @@ func ginOperatorClassForColumn(col *models.Column, comment string) string {
|
||||
baseType := pgsql.CanonicalizeBaseType(pgsql.ExtractBaseTypeLower(sqlType))
|
||||
isArray := pgsql.IsArrayType(sqlType)
|
||||
requested := extractOperatorClass(comment)
|
||||
|
||||
if requested != "" && ginOperatorClassCompatible(baseType, isArray, requested) {
|
||||
return requested
|
||||
method := strings.ToLower(strings.TrimSpace(indexType))
|
||||
if method == "" {
|
||||
method = "btree"
|
||||
}
|
||||
|
||||
if isArray {
|
||||
return "array_ops"
|
||||
if requested != "" && operatorClassCompatible(method, baseType, isArray, requested) {
|
||||
return requested
|
||||
}
|
||||
|
||||
switch {
|
||||
case isTextGinBaseType(baseType):
|
||||
return "gin_trgm_ops"
|
||||
case baseType == "jsonb":
|
||||
return "jsonb_ops"
|
||||
case method == "gin":
|
||||
if isArray {
|
||||
return "array_ops"
|
||||
}
|
||||
switch {
|
||||
case isTextGinBaseType(baseType):
|
||||
return "gin_trgm_ops"
|
||||
case baseType == "jsonb":
|
||||
return "jsonb_ops"
|
||||
default:
|
||||
return requested
|
||||
}
|
||||
case isVectorIndexMethod(method):
|
||||
if isArray {
|
||||
return ""
|
||||
}
|
||||
return defaultVectorOperatorClasses[baseType]
|
||||
default:
|
||||
return requested
|
||||
// gist/spgist/brin/btree have default operator classes (PostGIS included),
|
||||
// so nothing is emitted unless the comment requested a compatible class.
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
func ginOperatorClassCompatible(baseType string, isArray bool, opClass string) bool {
|
||||
// ginOperatorClassForColumn is the GIN-specific form of indexOperatorClassForColumn.
|
||||
func ginOperatorClassForColumn(col *models.Column, comment string) string {
|
||||
return indexOperatorClassForColumn(col, "gin", comment)
|
||||
}
|
||||
|
||||
func operatorClassCompatible(method, baseType string, isArray bool, opClass string) bool {
|
||||
if vectorType, ok := vectorOperatorClasses[opClass]; ok {
|
||||
return !isArray && baseType == vectorType && isVectorIndexMethod(method)
|
||||
}
|
||||
if spatialOperatorClasses[opClass] {
|
||||
return !isArray && pgsql.IsSpatialType(baseType)
|
||||
}
|
||||
|
||||
switch opClass {
|
||||
case "gin_trgm_ops", "gin_bigm_ops":
|
||||
return !isArray && isTextGinBaseType(baseType)
|
||||
@@ -1524,6 +1604,10 @@ func ginOperatorClassCompatible(baseType string, isArray bool, opClass string) b
|
||||
}
|
||||
}
|
||||
|
||||
func ginOperatorClassCompatible(baseType string, isArray bool, opClass string) bool {
|
||||
return operatorClassCompatible("gin", baseType, isArray, opClass)
|
||||
}
|
||||
|
||||
func isTextGinBaseType(baseType string) bool {
|
||||
switch baseType {
|
||||
case "text", "varchar", "character varying", "char", "character", "string", "citext", "bpchar":
|
||||
@@ -1533,29 +1617,188 @@ func isTextGinBaseType(baseType string) bool {
|
||||
}
|
||||
}
|
||||
|
||||
func schemaRequiresPGTrgm(schema *models.Schema) bool {
|
||||
// requiredExtensions returns the PostgreSQL extensions a schema depends on, ordered so
|
||||
// that dependencies are created first (postgis before postgis_topology, vector before
|
||||
// vchord). Extensions are detected from column types, index access methods, resolved
|
||||
// operator classes, and function calls in defaults, check constraints, partial index
|
||||
// predicates and view definitions. Extensions that leave no trace in the model (pg_cron,
|
||||
// timescaledb, postgres_fdw, …) can be declared in schema.Metadata["extensions"].
|
||||
func requiredExtensions(schema *models.Schema) []string {
|
||||
if schema == nil {
|
||||
return false
|
||||
return nil
|
||||
}
|
||||
|
||||
required := make(map[string]bool)
|
||||
add := func(names ...string) {
|
||||
for _, name := range names {
|
||||
if name != "" {
|
||||
required[name] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
add(declaredExtensions(schema)...)
|
||||
|
||||
for _, view := range schema.Views {
|
||||
if view == nil {
|
||||
continue
|
||||
}
|
||||
add(pgsql.ExtensionsForExpression(view.Definition)...)
|
||||
}
|
||||
|
||||
for _, table := range schema.Tables {
|
||||
if table == nil {
|
||||
continue
|
||||
}
|
||||
for _, index := range table.Indexes {
|
||||
if index == nil || !strings.EqualFold(index.Type, "gin") {
|
||||
|
||||
for _, col := range table.Columns {
|
||||
if col == nil {
|
||||
continue
|
||||
}
|
||||
add(pgsql.TypeExtension(effectiveColumnSQLType(col)))
|
||||
if def, ok := col.Default.(string); ok {
|
||||
add(pgsql.ExtensionsForExpression(def)...)
|
||||
}
|
||||
}
|
||||
|
||||
for _, constraint := range table.Constraints {
|
||||
if constraint == nil {
|
||||
continue
|
||||
}
|
||||
add(pgsql.ExtensionsForExpression(constraint.Expression)...)
|
||||
}
|
||||
|
||||
for _, index := range table.Indexes {
|
||||
if index == nil {
|
||||
continue
|
||||
}
|
||||
add(pgsql.IndexMethodExtension(index.Type))
|
||||
add(pgsql.ExtensionsForExpression(index.Where)...)
|
||||
|
||||
for _, colName := range index.Columns {
|
||||
col, ok := resolveIndexColumn(table, colName)
|
||||
if !ok || col == nil {
|
||||
continue
|
||||
}
|
||||
if ginOperatorClassForColumn(col, index.Comment) == "gin_trgm_ops" {
|
||||
return true
|
||||
}
|
||||
opClass := indexOperatorClassForColumn(col, index.Type, index.Comment)
|
||||
add(pgsql.OperatorClassExtension(opClass))
|
||||
add(btreeCompanionExtension(index.Type, col, opClass))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
extensions := make([]string, 0, len(required))
|
||||
for ext := range required {
|
||||
extensions = append(extensions, ext)
|
||||
}
|
||||
|
||||
// Pull in dependencies, so a declared postgis_topology also creates postgis.
|
||||
for i := 0; i < len(extensions); i++ {
|
||||
for _, dependency := range pgsql.ExtensionDependencies(extensions[i]) {
|
||||
if !required[dependency] {
|
||||
required[dependency] = true
|
||||
extensions = append(extensions, dependency)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return pgsql.SortExtensions(extensions)
|
||||
}
|
||||
|
||||
// declaredExtensions reads schema.Metadata["extensions"], which accepts either a list or a
|
||||
// comma-separated string. Unknown names are kept: the metadata is an explicit instruction.
|
||||
func declaredExtensions(schema *models.Schema) []string {
|
||||
value, ok := schema.Metadata["extensions"]
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
|
||||
var names []string
|
||||
switch declared := value.(type) {
|
||||
case string:
|
||||
names = strings.Split(declared, ",")
|
||||
case []string:
|
||||
names = declared
|
||||
case []any:
|
||||
for _, item := range declared {
|
||||
if name, ok := item.(string); ok {
|
||||
names = append(names, name)
|
||||
}
|
||||
}
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
|
||||
cleaned := make([]string, 0, len(names))
|
||||
for _, name := range names {
|
||||
if name = strings.TrimSpace(name); name != "" {
|
||||
cleaned = append(cleaned, name)
|
||||
}
|
||||
}
|
||||
return cleaned
|
||||
}
|
||||
|
||||
// btreeCompanionExtension returns btree_gin or btree_gist when a GIN/GiST index covers a
|
||||
// scalar type that neither access method has a built-in operator class for. Without the
|
||||
// companion extension PostgreSQL rejects the CREATE INDEX outright.
|
||||
func btreeCompanionExtension(indexType string, col *models.Column, opClass string) string {
|
||||
if opClass != "" {
|
||||
return ""
|
||||
}
|
||||
|
||||
method := strings.ToLower(strings.TrimSpace(indexType))
|
||||
if method != "gin" && method != "gist" {
|
||||
return ""
|
||||
}
|
||||
|
||||
sqlType := effectiveColumnSQLType(col)
|
||||
if pgsql.IsArrayType(sqlType) {
|
||||
return ""
|
||||
}
|
||||
|
||||
baseType := pgsql.CanonicalizeBaseType(pgsql.ExtractBaseTypeLower(sqlType))
|
||||
if pgsql.TypeExtension(baseType) != "" {
|
||||
// Extension types (geometry, vector, citext, …) ship their own operator classes.
|
||||
return ""
|
||||
}
|
||||
|
||||
if method == "gin" {
|
||||
if nativeGinBaseType(baseType) {
|
||||
return ""
|
||||
}
|
||||
return "btree_gin"
|
||||
}
|
||||
if nativeGistBaseType(baseType) {
|
||||
return ""
|
||||
}
|
||||
return "btree_gist"
|
||||
}
|
||||
|
||||
// nativeGinBaseType reports whether core PostgreSQL provides a GIN operator class.
|
||||
func nativeGinBaseType(baseType string) bool {
|
||||
switch baseType {
|
||||
case "jsonb", "json", "tsvector", "tsquery":
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// nativeGistBaseType reports whether core PostgreSQL provides a GiST operator class.
|
||||
func nativeGistBaseType(baseType string) bool {
|
||||
switch baseType {
|
||||
case "tsvector", "tsquery", "point", "box", "circle", "polygon", "line", "lseg", "path", "inet", "cidr":
|
||||
return true
|
||||
}
|
||||
return strings.HasSuffix(baseType, "range") || strings.HasSuffix(baseType, "multirange")
|
||||
}
|
||||
|
||||
func schemaRequiresPGTrgm(schema *models.Schema) bool {
|
||||
for _, ext := range requiredExtensions(schema) {
|
||||
if ext == "pg_trgm" {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
@@ -1642,14 +1885,21 @@ func formatStringList(items []string) string {
|
||||
|
||||
// extractOperatorClass extracts operator class from index comment/note
|
||||
// Looks for common operator classes like gin_trgm_ops, gist_trgm_ops, etc.
|
||||
// explicitOperatorClassPattern matches an "opclass=<name>" hint, the form the PostgreSQL
|
||||
// reader uses to carry an index's operator class through the model.
|
||||
var explicitOperatorClassPattern = regexp.MustCompile(`(?i)\bopclass\s*=\s*([a-z_][a-z0-9_]*)\b`)
|
||||
|
||||
func extractOperatorClass(comment string) string {
|
||||
if comment == "" {
|
||||
return ""
|
||||
}
|
||||
|
||||
lowerComment := strings.ToLower(comment)
|
||||
// Common GIN/GiST operator classes
|
||||
opClasses := []string{"gin_trgm_ops", "gist_trgm_ops", "gin_bigm_ops", "jsonb_ops", "jsonb_path_ops", "array_ops"}
|
||||
for _, op := range opClasses {
|
||||
if matches := explicitOperatorClassPattern.FindStringSubmatch(lowerComment); len(matches) > 1 {
|
||||
return matches[1]
|
||||
}
|
||||
|
||||
for _, op := range knownOperatorClasses() {
|
||||
if strings.Contains(lowerComment, op) {
|
||||
return op
|
||||
}
|
||||
@@ -1657,6 +1907,35 @@ func extractOperatorClass(comment string) string {
|
||||
return ""
|
||||
}
|
||||
|
||||
// knownOperatorClasses lists every operator class recognized in an index comment,
|
||||
// longest name first so that e.g. gist_geometry_ops_nd wins over a shorter prefix.
|
||||
var knownOperatorClasses = sync.OnceValue(func() []string {
|
||||
names := []string{"gin_trgm_ops", "gist_trgm_ops", "gin_bigm_ops", "jsonb_ops", "jsonb_path_ops", "array_ops"}
|
||||
for name := range vectorOperatorClasses {
|
||||
names = append(names, name)
|
||||
}
|
||||
for name := range spatialOperatorClasses {
|
||||
names = append(names, name)
|
||||
}
|
||||
sort.Slice(names, func(i, j int) bool {
|
||||
if len(names[i]) != len(names[j]) {
|
||||
return len(names[i]) > len(names[j])
|
||||
}
|
||||
return names[i] < names[j]
|
||||
})
|
||||
return names
|
||||
})
|
||||
|
||||
// indexStorageParameters extracts access-method storage parameters from an index comment.
|
||||
// Only well-formed "key = value" pairs are kept, so comment prose cannot leak into DDL.
|
||||
// Example: "opclass=vector_cosine_ops with (m=16, ef_construction=64)" -> "m = 16, ef_construction = 64".
|
||||
func indexStorageParameters(comment string) string {
|
||||
if comment == "" {
|
||||
return ""
|
||||
}
|
||||
return pgsql.FormatStorageParameters(pgsql.ExtractWithClause(comment))
|
||||
}
|
||||
|
||||
// escapeQuote escapes single quotes in strings for SQL
|
||||
func escapeQuote(s string) string {
|
||||
return strings.ReplaceAll(s, "'", "''")
|
||||
|
||||
@@ -1310,3 +1310,203 @@ func TestWriteSchema_UsesStorageTypeForSerialAlterStatements(t *testing.T) {
|
||||
t.Fatalf("expected serial alter to include USING cast, got:\n%s", output)
|
||||
}
|
||||
}
|
||||
|
||||
// buildVectorSpatialSchema returns a database with a pgvector column and a PostGIS column.
|
||||
func buildVectorSpatialSchema(indexType, indexComment string) *models.Database {
|
||||
db := models.InitDatabase("testdb")
|
||||
schema := models.InitSchema("public")
|
||||
|
||||
table := models.InitTable("documents", "public")
|
||||
|
||||
embedding := models.InitColumn("embedding", "documents", "public")
|
||||
embedding.Type = "vector(1536)"
|
||||
table.Columns["embedding"] = embedding
|
||||
|
||||
location := models.InitColumn("location", "documents", "public")
|
||||
location.Type = "geometry(Point,4326)"
|
||||
table.Columns["location"] = location
|
||||
|
||||
if indexType != "" {
|
||||
index := &models.Index{
|
||||
Name: "idx_documents_embedding",
|
||||
Type: indexType,
|
||||
Columns: []string{"embedding"},
|
||||
Comment: indexComment,
|
||||
}
|
||||
table.Indexes[index.Name] = index
|
||||
}
|
||||
|
||||
schema.Tables = append(schema.Tables, table)
|
||||
db.Schemas = append(db.Schemas, schema)
|
||||
return db
|
||||
}
|
||||
|
||||
func writeDatabaseOutput(t *testing.T, db *models.Database) string {
|
||||
t.Helper()
|
||||
|
||||
var buf bytes.Buffer
|
||||
writer := NewWriter(&writers.WriterOptions{})
|
||||
writer.writer = &buf
|
||||
|
||||
if err := writer.WriteDatabase(db); err != nil {
|
||||
t.Fatalf("WriteDatabase failed: %v", err)
|
||||
}
|
||||
return buf.String()
|
||||
}
|
||||
|
||||
func TestWriteDatabase_VectorAndPostGISColumnsCreateExtensions(t *testing.T) {
|
||||
output := writeDatabaseOutput(t, buildVectorSpatialSchema("", ""))
|
||||
|
||||
for _, want := range []string{
|
||||
"CREATE EXTENSION IF NOT EXISTS postgis;",
|
||||
"CREATE EXTENSION IF NOT EXISTS vector;",
|
||||
"vector(1536)",
|
||||
"geometry(Point,4326)",
|
||||
} {
|
||||
if !strings.Contains(output, want) {
|
||||
t.Fatalf("expected output to contain %q, got:\n%s", want, output)
|
||||
}
|
||||
}
|
||||
|
||||
// postgis must be created before postgis-dependent extensions and stay deterministic
|
||||
if strings.Index(output, "EXISTS postgis;") > strings.Index(output, "EXISTS vector;") {
|
||||
t.Fatalf("expected extensions to be emitted in sorted order, got:\n%s", output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteDatabase_HNSWIndexUsesDefaultVectorOperatorClass(t *testing.T) {
|
||||
output := writeDatabaseOutput(t, buildVectorSpatialSchema("hnsw", ""))
|
||||
|
||||
if !strings.Contains(output, "USING hnsw (embedding vector_cosine_ops)") {
|
||||
t.Fatalf("expected hnsw index with default vector operator class, got:\n%s", output)
|
||||
}
|
||||
if !strings.Contains(output, "CREATE EXTENSION IF NOT EXISTS vector;") {
|
||||
t.Fatalf("expected pgvector extension, got:\n%s", output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteDatabase_VectorIndexHonoursRequestedOperatorClassAndStorageParameters(t *testing.T) {
|
||||
output := writeDatabaseOutput(t, buildVectorSpatialSchema("ivfflat", "opclass=vector_l2_ops; with (lists=100)"))
|
||||
|
||||
if !strings.Contains(output, "USING ivfflat (embedding vector_l2_ops) WITH (lists = 100)") {
|
||||
t.Fatalf("expected ivfflat index with requested opclass and storage parameters, got:\n%s", output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteDatabase_VectorIndexIgnoresIncompatibleOperatorClass(t *testing.T) {
|
||||
output := writeDatabaseOutput(t, buildVectorSpatialSchema("hnsw", "opclass=halfvec_l2_ops"))
|
||||
|
||||
if !strings.Contains(output, "USING hnsw (embedding vector_cosine_ops)") {
|
||||
t.Fatalf("expected halfvec operator class to be rejected for a vector column, got:\n%s", output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteDatabase_VectorIndexIgnoresCommentProseInStorageParameters(t *testing.T) {
|
||||
output := writeDatabaseOutput(t, buildVectorSpatialSchema("hnsw", "tuned with (m=16, ef_construction=64, drop table foo)"))
|
||||
|
||||
if !strings.Contains(output, "WITH (m = 16, ef_construction = 64)") {
|
||||
t.Fatalf("expected only well-formed storage parameters, got:\n%s", output)
|
||||
}
|
||||
if strings.Contains(output, "drop table") {
|
||||
t.Fatalf("expected prose to be dropped from storage parameters, got:\n%s", output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteDatabase_GistIndexOnGeometryUsesDefaultOperatorClass(t *testing.T) {
|
||||
db := models.InitDatabase("testdb")
|
||||
schema := models.InitSchema("public")
|
||||
|
||||
table := models.InitTable("places", "public")
|
||||
geom := models.InitColumn("geom", "places", "public")
|
||||
geom.Type = "geometry(Point,4326)"
|
||||
table.Columns["geom"] = geom
|
||||
|
||||
table.Indexes["idx_places_geom"] = &models.Index{
|
||||
Name: "idx_places_geom",
|
||||
Type: "gist",
|
||||
Columns: []string{"geom"},
|
||||
}
|
||||
|
||||
schema.Tables = append(schema.Tables, table)
|
||||
db.Schemas = append(db.Schemas, schema)
|
||||
|
||||
output := writeDatabaseOutput(t, db)
|
||||
|
||||
if !strings.Contains(output, "USING gist (geom)") {
|
||||
t.Fatalf("expected gist index to rely on the PostGIS default operator class, got:\n%s", output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteDatabase_GistIndexHonoursRequestedSpatialOperatorClass(t *testing.T) {
|
||||
db := models.InitDatabase("testdb")
|
||||
schema := models.InitSchema("public")
|
||||
|
||||
table := models.InitTable("places", "public")
|
||||
geom := models.InitColumn("geom", "places", "public")
|
||||
geom.Type = "geometry(PointZ,4326)"
|
||||
table.Columns["geom"] = geom
|
||||
|
||||
table.Indexes["idx_places_geom_nd"] = &models.Index{
|
||||
Name: "idx_places_geom_nd",
|
||||
Type: "gist",
|
||||
Columns: []string{"geom"},
|
||||
Comment: "opclass=gist_geometry_ops_nd",
|
||||
}
|
||||
|
||||
schema.Tables = append(schema.Tables, table)
|
||||
db.Schemas = append(db.Schemas, schema)
|
||||
|
||||
output := writeDatabaseOutput(t, db)
|
||||
|
||||
if !strings.Contains(output, "USING gist (geom gist_geometry_ops_nd)") {
|
||||
t.Fatalf("expected requested spatial operator class, got:\n%s", output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateDatabaseStatements_VectorIndexIncludesOperatorClassAndParameters(t *testing.T) {
|
||||
db := buildVectorSpatialSchema("hnsw", "opclass=vector_ip_ops; with (m=16)")
|
||||
|
||||
writer := NewWriter(&writers.WriterOptions{})
|
||||
statements, err := writer.GenerateDatabaseStatements(db)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateDatabaseStatements failed: %v", err)
|
||||
}
|
||||
|
||||
joined := strings.Join(statements, "\n")
|
||||
for _, want := range []string{
|
||||
"CREATE EXTENSION IF NOT EXISTS vector",
|
||||
"CREATE EXTENSION IF NOT EXISTS postgis",
|
||||
"USING hnsw (embedding vector_ip_ops) WITH (m = 16)",
|
||||
} {
|
||||
if !strings.Contains(joined, want) {
|
||||
t.Fatalf("expected statements to contain %q, got:\n%s", want, joined)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestIndexStorageParameters(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
comment string
|
||||
want string
|
||||
}{
|
||||
{"empty", "", ""},
|
||||
{"no with clause", "opclass=vector_cosine_ops", ""},
|
||||
{"single parameter", "with (lists=100)", "lists = 100"},
|
||||
{"multiple parameters", "WITH (m = 16, ef_construction = 64)", "m = 16, ef_construction = 64"},
|
||||
{"quoted value kept", "with (fillfactor='90')", "fillfactor = '90'"},
|
||||
{"bm25 key field", "with (key_field='id')", "key_field = 'id'"},
|
||||
{"dollar quoted value", "with (options = $$[build.internal]\nlists = [4096]$$)", "options = $$[build.internal]\nlists = [4096]$$"},
|
||||
{"dollar quoted value with parens", "with (options = $$f(x)$$, m = 16)", "options = $$f(x)$$, m = 16"},
|
||||
{"prose dropped", "with (lists=100, please drop everything)", "lists = 100"},
|
||||
{"unterminated quote dropped", "with (key_field='id)", ""},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if got := indexStorageParameters(tt.comment); got != tt.want {
|
||||
t.Errorf("indexStorageParameters(%q) = %q, want %q", tt.comment, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user