ResolveSpec/audit/pkg/metrics.audit.md

Audit: pkg/metrics

Package	github.com/bitechdev/ResolveSpec/pkg/metrics
Files	interfaces.go (98), config.go (64), prometheus.go (312), example_test.go (64)
Audit date	2026-09-29
Axes	thread locking/waiting, slowness, security, panic handling & logging
Threat model	hostile internet client; request bodies, headers, query params, schema/table/column names all attacker-controlled
Depth	lighter (small, mostly-declarative package) — but reachability is traced into every consumer

Summary

pkg/metrics is the best-behaved global-state package in this repository: the provider singleton is properly mutex-guarded and returns a working NoOpProvider instead of nil (interfaces.go:50-72), which is the pattern pkg/logger and pkg/cache should copy (see _CROSS-CUTTING.audit.md finding X5). Prometheus client types are goroutine-safe, so there are no locking or data-race defects in this package.

The dominant finding is not a defect in any one function but the state of the subsystem as a whole: no provider is ever installed. metrics.SetProvider is called only from _test.go files, and NewPrometheusProvider has no non-test caller anywhere in the repository. Every GetProvider() therefore returns &NoOpProvider{}, so all 39 database instrumentation points, the panic counter, and the event-broker metrics discard their observations, and /metrics — if it were routed — answers 404 Metrics provider not configured. The instrumentation is written, tested and wired, and then goes nowhere.

That fact reclassifies most of the rest. Two genuine label-cardinality hazards exist (findings 2 and 5), but neither is live today: the HTTP middleware that would create them has no callers, and the database labels are in practice bounded by the model registry, because pkg/restheadspec returns early for an unregistered schema.entity before any query is built (handler.go:130-141). They are recorded here as latent — the traps that spring on whoever first installs a real provider, which is exactly what fixing finding 1 requires. Fix finding 1 without findings 2 and 5 and the result is a remotely triggerable memory leak.

Secondary themes: a duplicate-registration panic at construction, an unauthenticated handler that exposes the default registry, a config struct whose Enabled and Provider fields are never read and have no section in pkg/config to bind to, and a Pushgateway goroutine that discards its errors under a comment admitting it should log.

Findings

#	Severity	Axis	Finding
1	High	correctness / logging	No provider is ever installed: all instrumentation is silently a no-op and /metrics 404s
2	Medium (latent)	security / slowness	path label is r.URL.Path — unbounded cardinality from any client, once Middleware is mounted
3	Medium	panic handling	promauto panics on duplicate registration; NewPrometheusProvider cannot be called twice and returns no error
4	Medium (latent)	security	Handler() returns promhttp.Handler() — no auth, and exposes the default registry including Go/process collectors
5	Medium (latent)	security / slowness	DB label values have no length or charset bound, and the fallback puts a 120-char SQL fragment in the entity label
6	Medium	correctness	Config.Enabled and Config.Provider are never read; there is no provider factory
7	Medium	logging	startAutoPush discards every push error with _ = err; the comment says a logger is needed; no recover()
8	Medium	panic handling	StopAutoPush panics on a second call and does not wait for the goroutine
9	Medium	correctness	tableFromRawQuery labels a subquery-in-FROM with the literal table name SELECT
10	Low	correctness	ResponseWriter drops http.Flusher/http.Hijacker/io.ReaderFrom, breaking SSE and WebSocket upgrade
11	Low	correctness	ResponseWriter.statusCode stays 200 when the handler never calls WriteHeader
12	Low	correctness	eventDuration reuses DBQueryBuckets, so a db_query_buckets change silently reshapes event histograms
13	Low	correctness	PushgatewayInterval is an untyped int of seconds, inconsistent with the duration strings used elsewhere
14	Low	correctness	Push() returns nil when Pushgateway is unconfigured, so a caller cannot tell "pushed" from "no-op"
15	Low	correctness	Pusher uses prometheus.DefaultGatherer, ignoring which registry the metrics went to
16	Low	security	NoOpProvider.Handler() returns a 404 body that fingerprints the metrics subsystem
17	Low	correctness	metrics.Config has no section in pkg/config, so its mapstructure tags bind to nothing
18	Low	slowness	WithLabelValues resolves a child by string on every observation; no pre-curried handles

---

1. High — the metrics subsystem is never activated

GetProvider() falls back to a no-op when no provider has been set (interfaces.go:63-72):

func GetProvider() Provider {
	globalProviderMu.RLock()
	p := globalProvider
	globalProviderMu.RUnlock()
	if p == nil {
		return &NoOpProvider{}
	}
	return p
}

That fallback is sound design. The problem is that nothing ever replaces it. Searching the whole module for the two functions that would install a provider:

Symbol	Non-test callers	Test-only callers
metrics.SetProvider	0	pkg/middleware/panic_test.go:32-33, pkg/metrics/example_test.go:13, :32, pkg/common/adapters/database/query_metrics_test.go (9 pairs, :106 … :347)
metrics.NewPrometheusProvider	0	pkg/metrics/example_test.go:12, :31

Only three files in the repository import pkg/metrics outside of tests — pkg/eventbroker/metrics.go, pkg/middleware/panic.go and pkg/common/adapters/database/query_metrics.go — and all three are producers that call GetProvider(). No cmd/, no pkg/server, and no example installs a provider.

Failure scenario. Everything below is instrumented and none of it records anything:

• 39 database call sites (bun.go 12, pgsql.go 17, gorm.go 10) route through recordQueryMetrics (query_metrics.go:15), which calls metrics.GetProvider().RecordDBQuery(...) (:20-27) — into the no-op. Note these are additionally gated on a per-connection metricsEnabled flag (:16-18) whose config default is false (pkg/config/manager.go:246), so there are two independent switches, both off.
• Panic counting. pkg/middleware/panic.go:19 calls metrics.GetProvider().RecordPanic(...), so panics_total never increments. A service can panic on every request and the metric an operator would alert on stays at zero — the failure is invisible in precisely the way metrics exist to prevent.
• Event broker. pkg/eventbroker/metrics.go:11, :18, :25 likewise.
• /metrics. NoOpProvider.Handler() (interfaces.go:90-98) returns 404 Metrics provider not configured. A Prometheus scrape job configured against this service fails every scrape, and up == 0 is the only signal.

The severity is in the silence. Nothing logs "metrics disabled", no startup warning fires, and NoOpProvider satisfies the interface perfectly — so the code reads as fully instrumented at every call site. An operator who enables enable_metrics: true on a connection, expecting query metrics, gets nothing and has no diagnostic to explain why. The cost already paid — ~39 instrumented call sites, 350+ lines of query_metrics.go, a tested PrometheusProvider — delivers zero observability.

Recommendation. Install a provider during startup, from config, and say so:

// in server bootstrap, after config load
provider, err := metrics.NewProvider(&cfg.Metrics)   // see finding 6
if err != nil {
	return fmt.Errorf("metrics: %w", err)
}
metrics.SetProvider(provider)
logger.Info("Metrics initialized: provider=%s namespace=%q", cfg.Metrics.Provider, cfg.Metrics.Namespace)

and route the handler on an internal listener (finding 4). Order matters: SetProvider must run before the first request is served, since the DB adapters capture nothing before it. Fix findings 2, 4 and 5 in the same change — they are dormant only because this one is unfixed. Add a startup logger.Warn when enable_metrics is true on any connection while the global provider is still a no-op; that single line turns this class of silent misconfiguration into a log message.

---

2. Medium (latent) — path label is the raw request path

prometheus.go:258-278:

func (p *PrometheusProvider) Middleware(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		start := time.Now()

		// Increment in-flight requests
		p.IncRequestsInFlight()
		defer p.DecRequestsInFlight()

		// Wrap response writer to capture status code
		rw := NewResponseWriter(w)

		// Call next handler
		next.ServeHTTP(rw, r)

		// Record metrics
		duration := time.Since(start)
		status := strconv.Itoa(rw.statusCode)

		p.RecordHTTPRequest(r.Method, r.URL.Path, status, duration)
	})
}

feeding metrics declared with []string{"method", "path", "status"} (prometheus.go:64, :71). RecordHTTPRequest calls WithLabelValues(method, path, status) (:192-193), which creates and permanently retains a child series per unseen tuple. prometheus.HistogramVec has no eviction and no cardinality cap.

Why this is latent, not live. Middleware has no callers in the repository, and RecordHTTPRequest/IncRequestsInFlight have none either — so the HTTP metrics are dead code today, on top of finding 1. This is a trap, not a live vulnerability.

Failure scenario, once mounted. An attacker requests GET /api/public/orders/1, /2, /3, … or simply GET /<random> in a loop. Every distinct path creates one http_request_duration_seconds child — with the default 11 buckets (config.go:43) that is 11 counters plus _sum, _count and the retained label strings, on the order of 400–600 bytes of live heap — plus one http_requests_total child. A million distinct paths, reachable in minutes at a few thousand requests per second and requiring no valid credential (the label is recorded whatever the status), is several hundred megabytes of permanently retained heap. The process is OOM-killed, and because the series live in the default registry they cannot be dropped without a restart. The same requests inflate every scrape into a multi-megabyte response, so the monitoring system amplifies the attack and Prometheus's own ingestion suffers.

No hostile client is even needed: a REST API with ID-bearing paths — which this project is — generates unbounded cardinality from ordinary traffic. method is a second attacker-controlled axis, since an arbitrary request method string becomes a label value unchecked.

Recommendation. Label with the matched route pattern, never the URL. Because the pattern is only known after ServeHTTP, resolve it there:

next.ServeHTTP(rw, r)

route := "unmatched"
if cr := mux.CurrentRoute(r); cr != nil {
	if tpl, err := cr.GetPathTemplate(); err == nil {
		route = tpl
	}
}
p.RecordHTTPRequest(normalizeMethod(r.Method), route, status, duration)

(chi.RouteContext(r.Context()).RoutePattern() for chi.) Normalize method against the known verbs, mapping anything else to "OTHER". Add a hard backstop independent of the router: track distinct label tuples per metric and collapse anything past a few hundred to "other", so no future consumer can reintroduce this.

---

3. Medium — promauto panics on duplicate registration

Every metric is built with promauto.New* (prometheus.go:58–:150), which registers into prometheus.DefaultRegisterer and panics on an AlreadyRegisteredError. The constructor has no recover() and returns no error:

func NewPrometheusProvider(cfg *Config) *PrometheusProvider {

Failure scenario. Calling NewPrometheusProvider twice in one process — two config sections, a reload path, a test constructing a provider per case, or an embedding application that also configures metrics — panics with duplicate metrics collector registration attempted. From init or TestMain that is a hard abort; from a config-reload handler it takes down a running server. Since the signature has no error, a caller cannot defend against it without wrapping the call in its own recover().

This is reachable today in tests: example_test.go:12 and :31 both call NewPrometheusProvider in the same package. Those pass only because ExampleNewPrometheusProvider_custom sets Namespace — metricName (prometheus.go:50-55) prefixes names only when cfg.Namespace != "", so the two providers register under different names. Drop that namespace and the test binary panics. The failure depends on configuration, not on code, which makes it easy to ship.

Recommendation. Return an error and register against an injectable registry:

func NewPrometheusProvider(cfg *Config) (*PrometheusProvider, error) {
	...
	reg := cfg.Registry
	if reg == nil {
		reg = prometheus.DefaultRegisterer
	}
	p := &PrometheusProvider{...}   // prometheus.NewHistogramVec, not promauto
	for _, c := range p.collectors() {
		if err := reg.Register(c); err != nil {
			var are prometheus.AlreadyRegisteredError
			if errors.As(err, &are) {
				continue   // or adopt are.ExistingCollector
			}
			return nil, fmt.Errorf("metrics: register %T: %w", c, err)
		}
	}
	return p, nil
}

An owned *prometheus.Registry also fixes findings 4 and 15.

---

4. Medium (latent) — the metrics handler has no access control

prometheus.go:252-255:

func (p *PrometheusProvider) Handler() http.Handler {
	return promhttp.Handler()
}

promhttp.Handler() serves prometheus.DefaultGatherer — which, because the default registry is used, includes the Go collector (go_*: goroutine count, heap sizes, GC pauses, Go version) and the process collector (process_*: resident memory, open file descriptors, start time, CPU seconds) alongside the application metrics. There is no authentication, no IP restriction, and no timeout or in-flight limit.

Latent for two reasons: no provider is installed (finding 1), and nothing routes Handler() today. It becomes live the moment finding 1 is fixed in the obvious way.

Failure scenario. Mounted on the public listener — the natural thing to do, since Handler() is the package's only HTTP surface and the interface comment says "e.g., /metrics endpoint" (interfaces.go:46) — any client learns the Go version and build (fingerprinting against known CVEs), in-flight request counts and traffic volume, cache hit/miss ratios, per-table database query latencies, panic counts, and — via findings 2 and 5 — the set of paths and SQL shapes the application uses. process_open_fds and go_goroutines also make the effectiveness of a resource-exhaustion attack directly observable, letting an attacker tune it in real time.

Recommendation. Serve metrics on a separate listener bound to a private interface. pkg/config already supports multiple server instances (servers.instances.*, pkg/config/manager.go:182-186), so this is configuration plus a documented pattern. If it must share the public listener, require authentication and bound the handler:

func (p *PrometheusProvider) Handler() http.Handler {
	return promhttp.HandlerFor(p.registry, promhttp.HandlerOpts{
		ErrorHandling:       promhttp.HTTPErrorOnError,
		Timeout:             10 * time.Second,
		MaxRequestsInFlight: 3,
	})
}

Using an owned registry (finding 3) also stops go_*/process_* from leaking; register collectors.NewGoCollector() deliberately if those are wanted.

---

5. Medium (latent) — DB label values are unbounded by design, and the fallback puts SQL text in a label

db_query_duration_seconds uses []string{"operation", "schema", "entity", "table"} and db_queries_total adds "status" (prometheus.go:86, :93) — a four- and five-dimensional label space. The values are produced by pkg/common/adapters/database/query_metrics.go and normalized by four helpers (:30-66) whose only sanitizer is cleanMetricIdentifier (:330-335):

func cleanMetricIdentifier(value string) string {
	value = strings.TrimSpace(value)
	value = strings.Trim(value, "\"'`[]")
	value = strings.TrimRight(value, ";")
	return value
}

That trims quotes and a trailing semicolon. It does not bound length, does not restrict the character set, and does not check against any allowlist.

What bounds this today. Ten of the 39 recordQueryMetrics call sites parse their labels out of raw SQL (bun.go:195, :215, :1731, :1739; pgsql.go:128, :154, :1092, :1106; gorm.go:144, :165). The rest take them from the model-derived helpers schemaAndTableFromModel / entityNameFromModel — 27 call sites, nine in each of bun.go, pgsql.go and gorm.go — and schemaAndTableFromModel (:89-96) returns parseTableName(provider.TableName(), driverName), the model's own declared name rather than request input. Critically, pkg/restheadspec resolves the model before building any query and returns early when it does not exist (handler.go:130-141):

model, err := h.registry.GetModelByEntity(schema, entity)
if err != nil {
	// Model not found - call fallback handler if set, otherwise pass through
	logger.Debug("Model not found for %s.%s", schema, entity)
	...
	return
}

So GET /api/<random>/<random> never reaches the database, and the label space is in practice bounded by the number of registered models. The cardinality risk is latent, resting on a guarantee enforced elsewhere and nowhere documented here.

Failure scenario. Two things remain wrong regardless:

• SQL text as a label value. When no table can be parsed, fallbackMetricEntityFromQuery (:149-160) puts up to 120 characters of the statement (maxMetricFallbackEntityLength, :13) into entity. The fallback triggers when tableFromRawQuery returns "" (:290-308), i.e. whenever the first keyword is not SELECT/INSERT/UPDATE/DELETE — a WITH CTE, EXPLAIN, CALL, or a leading comment. Since pkg/restheadspec builds raw SQL from request-supplied filters, sort and custom_sql_where/custom_sql_join fragments (see FetchRowNumber, handler.go:3198), the label then varies with placeholder arity: IN (?), IN (?,?), IN (?,?,?) are three distinct shapes, so one client varying a filter list length mints a new permanent series per length. On an unauthenticated /metrics (finding 4), that same label discloses table names, join structure, and the presence of soft-delete and tenancy predicates. sanitizeMetricQueryShape (:162-234) is careful and does strip quoted literals and ?/$n placeholders, so parameter values do not leak — the structure does.
• No defensive bound. cleanMetricIdentifier is the single choke point for every identifier label, and it enforces nothing. The safety property lives in a different package, so any new caller — a migration tool, an admin endpoint, a fallback handler that does reach the DB — silently reintroduces unbounded cardinality with no review signal.

Recommendation.

1. Bound every label value at the choke point, so the guarantee is local:

var metricIdentRe = regexp.MustCompile(`^[A-Za-z0-9_]{1,32}$`)

func cleanMetricIdentifier(value string) string {
	value = strings.Trim(strings.TrimSpace(value), "\"'`[]")
	value = strings.TrimRight(value, ";")
	if !metricIdentRe.MatchString(value) {
		return ""      // callers already map "" to "default"/"unknown"
	}
	return value
}

2. Delete the query-shape fallback — return "unknown". A truncated SQL statement is not a label value; if the shape is wanted for debugging, log it at debug level or attach it to a trace span, which has no cardinality contract.
3. Resolve identifiers against the model registry in query_metrics.go rather than trusting SQL parsing, so the bound is explicit rather than inherited.
4. Collapse entity and table. They are identical whenever a table is parsed (:145 sets entity = cleanMetricIdentifier(table)), so the fourth dimension buys nothing and widens every tuple.
5. Add the cardinality backstop from finding 2 in pkg/metrics itself.

---

6. Medium — Config.Enabled and Config.Provider are never read

config.go:4-6 declares the flag:

type Config struct {
	// Enabled determines whether metrics collection is enabled
	Enabled bool `mapstructure:"enabled"`

DefaultConfig() sets it to true (:40); ApplyDefaults() (:50-64) never touches it; NewPrometheusProvider never reads it. The same holds for Config.Provider (:9), defaulted in two places (:41, :52) and never read — nothing in the package dispatches on it to choose between prometheus and noop, because no factory exists.

Failure scenario. An operator sets enabled: false to stop collection — perhaps because finding 2 or 5 is consuming memory. Nothing happens. NewPrometheusProvider still registers every collector and records every observation; the only thing that decides is which constructor the caller hard-coded. Setting provider: noop fails identically. The configuration lies, and silently: no warning, no log line. Combined with finding 17 (no metrics section exists to set these in) the config surface is entirely decorative.

Recommendation. Add the factory the config implies, and make it the only supported entry point:

func NewProvider(cfg *Config) (Provider, error) {
	if cfg == nil {
		cfg = DefaultConfig()
	}
	cfg.ApplyDefaults()
	if !cfg.Enabled {
		return &NoOpProvider{}, nil
	}
	switch cfg.Provider {
	case "prometheus":
		return NewPrometheusProvider(cfg)   // per finding 3
	case "noop":
		return &NoOpProvider{}, nil
	default:
		return nil, fmt.Errorf("metrics: unknown provider %q", cfg.Provider)
	}
}

Erroring on an unknown name matters — today a typo would fall through to whatever the caller chose.

---

7. Medium — the Pushgateway goroutine discards every error

prometheus.go:290-304:

func (p *PrometheusProvider) startAutoPush() {
	for {
		select {
		case <-p.pushTicker.C:
			if err := p.Push(); err != nil {
				// Log error but continue pushing
				// Note: In production, you might want to use a proper logger
				_ = err
			}
		case <-p.pushStop:
			p.pushTicker.Stop()
			return
		}
	}
}

The comment states the requirement and the code does the opposite. There is also no recover() in this goroutine.

Failure scenario. The Pushgateway URL is wrong, DNS fails, or the gateway rejects the payload. Every tick fails, forever, in complete silence — no log line, no counter, nothing. Pushgateway is chosen for short-lived or non-scrapable workloads, so those are exactly the deployments where there is no other path for metrics to arrive: the observability system is dark and nothing says so. Discovery happens when someone notices a dashboard has been empty for a week.

The missing recover() compounds it: push.Pusher.Push gathers from DefaultGatherer, so a panicking custom collector would crash the whole process from a background goroutine with no handler in the stack — the pattern flagged in _CROSS-CUTTING.audit.md finding X7.

Recommendation. pkg/metrics already imports pkg/logger (interfaces.go:8), so this costs nothing:

func (p *PrometheusProvider) startAutoPush() {
	defer logger.CatchPanicCallback("metrics.startAutoPush", nil)()
	failures := 0
	for {
		select {
		case <-p.pushTicker.C:
			if err := p.Push(); err != nil {
				failures++
				if failures == 1 || failures%60 == 0 {
					logger.Warn("metrics: pushgateway push failed (%d consecutive): %v", failures, err)
				}
				continue
			}
			failures = 0
		case <-p.pushStop:
			p.pushTicker.Stop()
			return
		}
	}
}

The backoff matters: an unconditional Warn per tick forwards a message to Sentry per tick (_CROSS-CUTTING.audit.md finding X8), turning a broken gateway into a second incident. Add a metrics_push_failures_total counter so the failure is visible in whatever monitoring is working.

---

8. Medium — StopAutoPush panics on a second call and does not wait

prometheus.go:308-312:

func (p *PrometheusProvider) StopAutoPush() {
	if p.pushStop != nil {
		close(p.pushStop)
	}
}

close of an already-closed channel panics with close of closed channel. The check guards only nil, not "already closed", and the method is documented as a shutdown hook — the code most likely to run twice. (pushStop is declared chan bool at prometheus.go:35 and created at :163; the value sent is never read, so chan struct{} is the more honest type.)

Failure scenario. A shutdown path calls StopAutoPush and a deferred cleanup or signal handler calls it again — or two signals arrive. The second close panics during shutdown, where pkg/middleware/panic.go is not in the stack because this is not a request. The process aborts mid-drain, losing the in-flight requests that the 25-second servers.drain_timeout (pkg/config/manager.go:176) exists to protect.

Separately, the method returns without waiting for startAutoPush to observe the close. A push in progress is abandoned and the process's final interval of metrics is lost — precisely the interval a Pushgateway deployment most wants.

Recommendation. sync.Once plus a done channel, making shutdown idempotent and synchronous:

type PrometheusProvider struct {
	...
	pushStop     chan struct{}
	pushStopOnce sync.Once
	pushDone     chan struct{}
}

func (p *PrometheusProvider) StopAutoPush() {
	if p.pushStop == nil {
		return
	}
	p.pushStopOnce.Do(func() { close(p.pushStop) })
	<-p.pushDone
}

with defer close(p.pushDone) in startAutoPush and a final p.Push() before it returns. Consider StopAutoPush(ctx) so the wait is bounded.

---

9. Medium — a subquery in FROM is labelled as a table named SELECT

tableFromRawQuery (query_metrics.go:290-308) takes the first token after the first FROM, and tokenizeQuery (:319-328) replaces ( and ) with spaces:

func tokenizeQuery(query string) []string {
	replacer := strings.NewReplacer(
		"\n", " ", "\t", " ", "(", " ", ")", " ", ",", " ",
	)
	return strings.Fields(replacer.Replace(query))
}

so for SELECT ... FROM ( SELECT ... ) the token after FROM is the inner SELECT. parseTableName("SELECT", driver) then yields table = "SELECT", and metricTargetFromRawQuery sets entity = cleanMetricIdentifier(table) (:145), so both identifier labels become the keyword.

Failure scenario. FetchRowNumber (pkg/restheadspec/handler.go:3114) builds exactly that shape — an outer select over a ROW_NUMBER() OVER (...) subquery, assembled with fmt.Sprintf at :3173-3189 — and executes it through the raw path (db.Query(ctx, &result, queryStr, pkValue), :3198). Its metrics are recorded with entity = "SELECT" and table = "SELECT". Every such query across every table collapses into one series named after a keyword. An operator investigating slow cursor pagination sees a db_query_duration_seconds{table="SELECT"} bucket with no indication of which table is slow, while the real table's own series omits this traffic entirely — the metric is not merely missing but actively misleading, and it silently aggregates unrelated tables into one latency distribution.

The same tokenizer means tokenAfter matches a FROM appearing anywhere, including inside a subquery or a CASE expression, so the first FROM is not reliably the outer one.

Recommendation. Treat an unparseable target as unknown rather than guessing. Reject any candidate that is a SQL keyword before accepting it:

var sqlKeywords = map[string]struct{}{
	"select": {}, "insert": {}, "update": {}, "delete": {}, "with": {},
	"values": {}, "set": {}, "where": {}, "from": {}, "join": {}, "as": {},
}

func tokenAfter(tokens []string, keyword string) string {
	for idx, token := range tokens {
		if strings.EqualFold(token, keyword) && idx+1 < len(tokens) {
			cand := cleanMetricIdentifier(tokens[idx+1])
			if _, isKw := sqlKeywords[strings.ToLower(cand)]; isKw {
				return ""
			}
			return cand
		}
	}
	return ""
}

Better: have the callers that already know the target pass it explicitly. FetchRowNumber knows tableName — threading it into the metric call is more accurate than any amount of SQL parsing, and removes the guesswork for the highest-traffic raw query in the codebase. Note that with finding 5's recommendation applied, "" here means entity/table become "unknown" rather than falling back to a SQL fragment.

---

10. Low — ResponseWriter drops optional interfaces

prometheus.go:172-176:

type ResponseWriter struct {
	http.ResponseWriter
	statusCode int
}

Embedding promotes only Header, Write and WriteHeader (the latter overridden at :185-188). The wrapper does not implement http.Flusher, http.Hijacker, http.Pusher or io.ReaderFrom, so a handler's type assertion for any of them fails once this middleware is in the chain.

Failure scenario. This repository has pkg/websocketspec. A WebSocket upgrade requires w.(http.Hijacker); with this middleware in front, the assertion fails and gorilla/websocket returns websocket: response does not implement http.Hijacker. Every WebSocket connection fails, and the cause is a metrics wrapper several layers up — a hard bug to locate, since removing the middleware "fixes" it and the code looks unrelated. The same applies to SSE, chunked streaming and long-polling: without Flusher, output buffers until the handler returns. Losing io.ReaderFrom silently disables sendfile for large bodies.

Recommendation. Implement the optional interfaces with pass-through and a capability check, and add Unwrap:

func (rw *ResponseWriter) Flush() {
	if f, ok := rw.ResponseWriter.(http.Flusher); ok {
		f.Flush()
	}
}

func (rw *ResponseWriter) Hijack() (net.Conn, *bufio.ReadWriter, error) {
	if h, ok := rw.ResponseWriter.(http.Hijacker); ok {
		return h.Hijack()
	}
	return nil, nil, fmt.Errorf("metrics: ResponseWriter does not support Hijack")
}

func (rw *ResponseWriter) Unwrap() http.ResponseWriter { return rw.ResponseWriter }

Unwrap is what Go 1.20+'s http.ResponseController uses, making the wrapper transparent to any handler written against that API. Better still, use httpsnoop or ResponseController and delete the wrapper. Audit the other response wrappers in pkg/middleware and pkg/server for the same defect.

---

11. Low — statusCode stays 200 when nothing was written

NewResponseWriter seeds statusCode: http.StatusOK (prometheus.go:178-183), correct for Go's implicit-200 behaviour, and the middleware records it unconditionally after ServeHTTP returns (:273-276).

Failure scenario. A handler panics. If PanicRecovery is inside the metrics middleware it writes a 500 and the label is right; if it is outside — or if the panic escapes to net/http's own recovery, which writes nothing — statusCode is still 200 and a failed request is counted as a success. Identically, a client that disconnects mid-response, or a handler that returns without writing, is recorded as status="200". Error-rate alerts built on http_requests_total{status=~"5.."} under-report exactly the failures they exist to catch.

Recommendation. Track whether the header was written and label accordingly:

type ResponseWriter struct {
	http.ResponseWriter
	statusCode  int
	wroteHeader bool
}

func (rw *ResponseWriter) WriteHeader(code int) {
	if rw.wroteHeader {
		return          // also suppresses the "superfluous WriteHeader" warning
	}
	rw.statusCode, rw.wroteHeader = code, true
	rw.ResponseWriter.WriteHeader(code)
}

func (rw *ResponseWriter) Write(b []byte) (int, error) {
	if !rw.wroteHeader {
		rw.wroteHeader = true      // implicit 200
	}
	return rw.ResponseWriter.Write(b)
}

Use a distinct "none" status when !wroteHeader, and document that PanicRecovery must be mounted inside the metrics middleware so panics are observed as 500s. Ordering is covered in middleware.audit.md and server.audit.md.

---

12. Low — event histograms reuse the DB query buckets

prometheus.go:130-137:

eventDuration: promauto.NewHistogramVec(
	prometheus.HistogramOpts{
		Name:    metricName("event_processing_duration_seconds"),
		Help:    "Event processing duration in seconds",
		Buckets: cfg.DBQueryBuckets, // Events are typically fast like DB queries
	},
	[]string{"source", "event_type"},
),

There is no EventBuckets field in Config.

Failure scenario. An operator tunes db_query_buckets for their database — say narrowing the top bucket to 1s because queries are fast — and silently changes the resolution of event-processing histograms too. Event handlers do arbitrary work, including outbound HTTP, so their latency distribution has no reason to match a query's. Observations above the last bucket land only in +Inf, so histogram_quantile returns the last boundary and a slow consumer is hidden. The coupling is invisible from the config file.

Recommendation. Add EventBuckets []float64 to Config with its own default in DefaultConfig/ApplyDefaults, defaulting to the DB buckets for compatibility. Consider a native histogram (NativeHistogramBucketFactor), which removes bucket tuning entirely on a Prometheus that supports it.

---

13. Low — PushgatewayInterval is an untyped seconds int

config.go:34 declares PushgatewayInterval int and prometheus.go:164 reads it as seconds:

p.pushTicker = time.NewTicker(time.Duration(cfg.PushgatewayInterval) * time.Second)

Everywhere else in this project, intervals are duration strings parsed by viper — servers.shutdown_timeout: "30s", dbmanager.health_check_interval: "30s", cache.memcache.timeout: "100ms" (pkg/config/manager.go:175, :231, :202).

Failure scenario. An operator follows the surrounding convention and writes pushgateway_interval: "30s". viper's mapstructure decode into an int fails or yields 0, and 0 is the documented "automatic pushing is disabled" value (config.go:32-33) — so pushing is silently off and, as in finding 7, nothing logs it. A value of 1 instead pushes every second, hammering the gateway.

Recommendation. Change the field to time.Duration; viper decodes duration strings natively via StringToTimeDurationHookFunc. Validate in ApplyDefaults that a configured interval is at least a second, and log the effective value once at startup.

---

14. Low — Push() cannot report "no-op"

prometheus.go:282-287:

func (p *PrometheusProvider) Push() error {
	if p.pusher == nil {
		return nil // Pushgateway not configured, silently skip
	}
	return p.pusher.Push()
}

Failure scenario. A short-lived job — a migration, a cron task — calls Push() before exiting, checks the error, sees nil, and exits believing its metrics were delivered. PushgatewayURL was empty (a typo, an unset environment variable), so nothing was sent and nothing will ever scrape a process that has already exited. The metrics are lost with an explicit success report, which is worse than an error.

Recommendation. Return a sentinel the caller can distinguish:

var ErrPushgatewayNotConfigured = errors.New("metrics: pushgateway not configured")

func (p *PrometheusProvider) Push() error {
	if p.pusher == nil {
		return ErrPushgatewayNotConfigured
	}
	return p.pusher.Push()
}

startAutoPush never reaches Push with a nil pusher — the goroutine starts only inside the cfg.PushgatewayURL != "" branch (:157-167) — so the sentinel needs no special handling there, but treat it as non-failing if that changes.

---

15. Low — the pusher gathers from the default gatherer

prometheus.go:158-159:

p.pusher = push.New(cfg.PushgatewayURL, cfg.PushgatewayJobName).
	Gatherer(prometheus.DefaultGatherer)

Failure scenario. Consistent with promauto registering into the default registerer, so it works today — but it silently couples the pusher to global state. Once findings 3 and 4 move the provider to an owned registry, the pusher keeps gathering from the default one and pushes the wrong set of metrics (Go and process collectors, plus whatever any other library registered) while omitting the application metrics it was configured for. Nothing errors; the gateway receives a plausible payload.

It also means the pushed payload currently includes go_* and process_* from every instance, the Pushgateway anti-pattern — those are per-process and the gateway has no instance label to separate them, so instances overwrite each other.

Recommendation. Gather from the provider's own registry and add an instance grouping:

p.pusher = push.New(cfg.PushgatewayURL, cfg.PushgatewayJobName).
	Gatherer(p.registry).
	Grouping("instance", instanceID)

pkg/config already has event_broker.instance_id (pkg/config/manager.go:256) that could supply the value.

---

16. Low — the no-op handler fingerprints the subsystem

interfaces.go:90-98:

func (n *NoOpProvider) Handler() http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusNotFound)
		_, err := w.Write([]byte("Metrics provider not configured"))
		if err != nil {
			logger.Warn("Failed to write. %v", err)
		}
	})
}

Failure scenario. Minor, but distinguishable: a client probing /metrics gets 404 Metrics provider not configured rather than the router's generic 404, confirming the route is registered and metrics merely disabled. That tells an attacker a configuration change or restart may expose the data in finding 4, and distinguishes this application from others behind the same proxy. Given finding 1, this is also the response a real Prometheus scrape gets today.

The logger.Warn on a failed write is the wrong level: a client disconnecting mid-response is routine, and per _CROSS-CUTTING.audit.md finding X8 every Warn is forwarded to Sentry — so a closed connection generates a third-party error event.

Recommendation. Return a bare http.NotFound(w, r) so the response is indistinguishable from any unregistered path, and drop the log line or make it Debug.

---

17. Low — metrics.Config is not wired into pkg/config

pkg/config/config.go has sections for tracing, cache, logger, error tracking, middleware, CORS, event broker, dbmanager, paths and extensions, and setDefaults (pkg/config/manager.go:170-293) sets defaults for all of them. There is no metrics section and no metrics.* default, and Manager.SetConfig (:112-134) likewise sets every section but metrics. metrics.Config's mapstructure tags (config.go:6, :9, :12, …) therefore have nothing to bind to.

Failure scenario. An operator cannot configure metrics through the normal mechanism — RESOLVESPEC_METRICS_ENABLED does nothing, and neither does a metrics: block in config.yaml. The struct's tags imply otherwise, so the natural conclusion is that configuration is broken rather than absent, and the only way to learn the truth is to read setDefaults. Together with findings 1 and 6, metrics are unconfigurable and uninstallable without writing Go.

Recommendation. Add Metrics metrics.Config \mapstructure:"metrics"`toconfig.Configand defaults tosetDefaults`:

v.SetDefault("metrics.enabled", false)   // see findings 2 and 5 before defaulting to true
v.SetDefault("metrics.provider", "prometheus")
v.SetDefault("metrics.namespace", "resolvespec")

Check the import direction first: pkg/metrics imports pkg/logger (interfaces.go:8), and pkg/logger imports pkg/errortracking (logger.go:12), so pkg/config → pkg/metrics → pkg/logger → pkg/errortracking must not close back on pkg/config (errortracking.audit.md finding 8 flags this coupling). If it does, duplicate the fields in pkg/config rather than importing.

---

18. Low — WithLabelValues on every observation

Each record method resolves its child from strings on every call, e.g. prometheus.go:192-193:

p.requestDuration.WithLabelValues(method, path, status).Observe(duration.Seconds())
p.requestTotal.WithLabelValues(method, path, status).Inc()

WithLabelValues hashes the label values and takes the MetricVec's internal RWMutex to look up or create the child — twice here, twice in RecordDBQuery (:212-213) and twice in RecordEventProcessed (:238-239).

Failure scenario. A small constant cost, not a defect, and the normal way to use the API. It becomes measurable only when the label space is large — which findings 2 and 5 would guarantee. A map with millions of entries has poor cache locality, so the two lookups per request grow from tens of nanoseconds to microseconds, and the MetricVec mutex becomes a contention point across all request-serving goroutines.

Recommendation. Fix findings 2 and 5 first; the cardinality bound is the real remedy. If the DB path proves hot afterwards, pre-curry per-table children at model-registration time and cache the prometheus.Observer/Counter handles rather than resolving by string per query — but only with a bounded, known label set.

---

What looks right

• The global provider is correctly synchronized (interfaces.go:50-72) — sync.RWMutex, read under RLock, written under Lock, with the pointer copied to a local before the lock is released. This is the only global in pkg/ that gets this right, and GetProvider returning &NoOpProvider{} instead of nil means no caller needs a nil check. pkg/logger, pkg/cache, pkg/config and pkg/tracing should adopt this shape (_CROSS-CUTTING.audit.md finding X5).
• NoOpProvider implements the full interface (interfaces.go:74-98) with genuine no-ops, so disabling metrics costs one interface dispatch.
• Prometheus client types are goroutine-safe by contract, so the record methods need no locking of their own — and correctly have none.
• IncRequestsInFlight/DecRequestsInFlight are paired with defer in the middleware (prometheus.go:263-264), so the gauge cannot leak on a panic.
• Bucket defaults are sensible and distinct for HTTP versus DB (config.go:43-45), with the reasoning in comments.
• ApplyDefaults (config.go:50-64) is idempotent and only fills empty values.
• RecordDBQuery derives status from err != nil (prometheus.go:208-211) rather than taking a caller-supplied string, so that label is bounded to two values by construction — the right instinct, applied to one label out of five.
• In the consumers: sanitizeMetricQueryShape (query_metrics.go:162-234) correctly strips single-quoted literals (including '' escapes) and ?/$n placeholders before query text is used as a label, so parameter values do not reach /metrics. tableNameProviderType is cached at package level (:99) and tableNameProviderFromModel checks Implements before reflect.New (:121-125) to avoid an allocation — careful, deliberate work.
• The adapter raw-query methods wrap themselves in recover() + logger.HandlePanic (e.g. bun.go:189-193, :209-213) and still call recordQueryMetrics on the error path (:204, :222), so a failed query is counted rather than dropped.
• pkg/common/adapters/database/query_metrics_test.go has 11 test functions, nine of which install a recording provider via SetProvider and restore the previous one with defer — the correct pattern for a global, and the best-tested metrics code in the repo.
• pkg/eventbroker/metrics.go:11, :18, :25 nil-check GetProvider() defensively even though it cannot return nil. Harmless belt-and-braces.
• example_test.go is correctly a _test.go file in package metrics_test with // Output: assertions — unlike pkg/cache/example_usage.go, which ships log.Fatal in the library (cache.audit.md finding 25).

Suggested follow-up

1. Decide whether metrics are a supported feature. If yes, install a provider at startup from config (finding 1) and add the factory and config section (findings 6, 17). If no, delete PrometheusProvider and the 39 instrumented call sites rather than carrying dead weight that reads as working instrumentation. The current middle state is the worst of both.
2. Fix findings 2, 4 and 5 in the same change as finding 1, not after. Route templates plus a cardinality backstop, an internal-only listener, and a bounded cleanMetricIdentifier. Activating metrics without these converts three latent defects into a remotely triggerable memory leak and an unauthenticated information leak on the same day.
3. Make NewPrometheusProvider return an error against an owned registry (finding 3), which also fixes findings 4 and 15 and makes the package testable.
4. Fix the SELECT-as-table-name mislabelling (finding 9) — have FetchRowNumber and the other raw-query callers pass the target they already know.
5. Fix the ResponseWriter wrapper (findings 10, 11) before anything depends on streaming or a WebSocket upgrade through this middleware. Add Unwrap().
6. Log Pushgateway failures with backoff and make StopAutoPush idempotent (findings 7, 8).
7. Add go test -race ./pkg/metrics/... to CI. The package's 64 test lines are four Example* functions with no assertions beyond // Output:, and it is not in the tested package set (_CROSS-CUTTING.audit.md findings X1, X2, X9). A test that constructs two providers with the same namespace would catch finding 3 immediately; one asserting GetProvider() is not a *NoOpProvider after bootstrap would catch finding 1.

Cross-references

• audit/pkg/_CROSS-CUTTING.audit.md — X10 (declared-but-never-installed subsystems) generalizes finding 1 of this audit: the same defect shape appears in pkg/middleware and in the ignored CORS config, and X10 carries the combined remediation order. Finding X5 cites this package's globalProviderMu as the pattern to copy; X1/X2/X9 cover the absent tests; X8 is why finding 7's log line needs rate limiting and why finding 16's Warn is the wrong level.
• audit/pkg/cache.audit.md — the Provider interface here has RecordCacheHit/RecordCacheMiss/UpdateCacheSize but no cache-error counter, and pkg/cache calls none of the three (finding 17 there). Those three methods have zero callers repo-wide, so cache observability is not merely a no-op but entirely unwritten.
• audit/pkg/middleware.audit.md — PanicRecovery (pkg/middleware/panic.go:14-33) is the only caller of RecordPanic, and its ordering relative to this package's Middleware decides whether finding 11 misreports panics as 200s. panic.go:28 also returns the panic value to the client, covered there.
• audit/pkg/common.audit.md — query_metrics.go produces findings 5 and 9's label values; cleanMetricIdentifier (:330) is the choke point, and tokenizeQuery (:319) the parser at fault.
• audit/pkg/restheadspec.audit.md — handler.go:130-141 is the early return that currently bounds finding 5; FetchRowNumber (:3175-3198) is the raw-query path behind findings 5 and 9 (FetchRowNumber begins at :3114).
• audit/pkg/config.audit.md — finding 17 (no metrics section); dbmanager.connections.*.enable_metrics defaults to false (manager.go:246), the second of the two switches keeping finding 5 dormant.
• audit/pkg/eventbroker.audit.md — UpdateEventQueueSize is an unlabelled gauge, so it is cardinality-safe; source and event_type (prometheus.go:121, :128, :136) are bounded only if event types are.
• audit/pkg/server.audit.md — where a separate metrics listener would be configured, where SetProvider should be called, and where middleware order is decided.