ResolveSpec/pkg/metrics/README.md

Metrics Package

A pluggable metrics collection system with Prometheus implementation.

Quick Start

import "github.com/bitechdev/ResolveSpec/pkg/metrics"

// Initialize Prometheus provider with default config
provider := metrics.NewPrometheusProvider(nil)
metrics.SetProvider(provider)

// Apply middleware to your router
router.Use(provider.Middleware)

// Expose metrics endpoint
http.Handle("/metrics", provider.Handler())

Configuration

You can customize the metrics provider using a configuration struct:

import "github.com/bitechdev/ResolveSpec/pkg/metrics"

// Create custom configuration
config := &metrics.Config{
    Enabled:  true,
    Provider: "prometheus",
    Namespace: "myapp", // Prefix all metrics with "myapp_"
    HTTPRequestBuckets: []float64{0.01, 0.05, 0.1, 0.5, 1, 2, 5},
    DBQueryBuckets: []float64{0.001, 0.01, 0.05, 0.1, 0.5, 1},
}

// Initialize with custom config
provider := metrics.NewPrometheusProvider(config)
metrics.SetProvider(provider)

Configuration Options

Field	Type	Default	Description
Enabled	bool	true	Enable/disable metrics collection
Provider	string	"prometheus"	Metrics provider type
Namespace	string	""	Prefix for all metric names
HTTPRequestBuckets	[]float64	See below	Histogram buckets for HTTP duration (seconds)
DBQueryBuckets	[]float64	See below	Histogram buckets for DB query duration (seconds)
HTTPMaxPaths	int	1024	Max distinct path label values; extras become "other" (negative disables)
HTTPPathNormalizer	func(*http.Request) string	nil	Custom request → path label mapping (return "" to use the default)

HTTP path label: the middleware uses, in order: HTTPPathNormalizer, the matched http.ServeMux pattern (r.Pattern, e.g. /users/{id}), then the raw path with numeric/UUID/hex/opaque-token segments replaced by :id. For routers other than ServeMux, supply HTTPPathNormalizer with your route template. The HTTPMaxPaths cap applies on top.

Default HTTP Request Buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]

Default DB Query Buckets: [0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5]

Enabled flag and JSON pull

Config.Enabled is honoured: a disabled provider records nothing, Middleware passes requests straight through, Handler()/JSONHandler() answer 404, and push loops are not started (manual pushes return an error). Note a &metrics.Config{} literal has Enabled: false; use DefaultConfig() or set Enabled: true. NewPrometheusProvider(nil) is enabled.

provider.JSONHandler() serves the same JSON as the push json format on GET/HEAD:

http.Handle("/metrics", provider.Handler())          // Prometheus text
http.Handle("/metrics.json", provider.JSONHandler()) // JSON

Resetting Stats

• provider.Reset() clears counters, histograms and the cache-size gauge (live gauges such as in-flight requests are kept). Package-level metrics.Reset() does the same for the current provider if it implements metrics.Resetter.
• provider.PushAndReset() pushes to the Pushgateway and resets only if the push succeeded (errors if no Pushgateway is configured).
• Config.PushgatewayResetOnPush: true makes the automatic push loop do this on every tick.
• provider.ResetHandler() is a POST-only endpoint (?push=true to push first). It has no auth: mount it on an internal route.

http.Handle("/metrics/reset", provider.ResetHandler())

Note: the normal /metrics scrape is read-only and never clears anything. Observations recorded between a push and its reset are lost. Prometheus handles the counter drop as a reset, but if you reset often, prefer increase()/rate() over raw counter values.

Custom Push Endpoint (Optional)

POST metrics to your own server, optionally clearing local stats after a 2xx reply:

provider := metrics.NewPrometheusProvider(&metrics.Config{
    PushEndpointURL:            "https://collector.example.com/metrics",
    PushEndpointFormat:         "json", // or "text" (Prometheus exposition, default)
    PushEndpointHeaders:        map[string]string{"Authorization": "Bearer token"},
    PushEndpointInterval:       30,     // seconds; 0 = manual only
    PushEndpointTimeout:        10,     // seconds (default 10)
    PushEndpointResetOnSuccess: true,   // clear local stats after a 2xx
})

err := provider.PushToEndpoint(ctx) // manual push; also honours ResetOnSuccess
provider.StopAutoPush()             // stops the Pushgateway and endpoint loops

The json body is a list of {name, help, type, metrics:[{labels, value | count, sum, buckets}]}. Failures (non-2xx, network, timeout) are logged and never reset stats, so the next tick retries with the accumulated data. The payload covers everything in the default Prometheus registry, including Go runtime metrics.

Pushgateway Configuration (Optional)

For batch jobs, cron tasks, or short-lived processes, you can push metrics to Prometheus Pushgateway:

Field	Type	Default	Description
PushgatewayURL	string	""	URL of Pushgateway (e.g., "http://pushgateway:9091")
PushgatewayJobName	string	"resolvespec"	Job name for pushed metrics
PushgatewayInterval	int	0	Auto-push interval in seconds (0 = disabled)

config := &metrics.Config{
    PushgatewayURL:      "http://pushgateway:9091",
    PushgatewayJobName:  "batch-job",
    PushgatewayInterval: 30, // Push every 30 seconds
}

Provider Interface

The package uses a provider interface, allowing you to plug in different metric systems:

type Provider interface {
    RecordHTTPRequest(method, path, status string, duration time.Duration)
    IncRequestsInFlight()
    DecRequestsInFlight()
    RecordDBQuery(operation, table string, duration time.Duration, err error)
    RecordCacheHit(provider string)
    RecordCacheMiss(provider string)
    UpdateCacheSize(provider string, size int64)
    Handler() http.Handler
}

Recording Metrics

HTTP Metrics (Automatic)

When using the middleware, HTTP metrics are recorded automatically:

router.Use(provider.Middleware)

Collected:

• Request duration (histogram)
• Request count by method, path, and status
• Requests in flight (gauge)

Database Metrics

start := time.Now()
rows, err := db.Query("SELECT * FROM users WHERE id = ?", userID)
duration := time.Since(start)

metrics.GetProvider().RecordDBQuery("SELECT", "users", duration, err)

Cache Metrics

// Record cache hit
metrics.GetProvider().RecordCacheHit("memory")

// Record cache miss
metrics.GetProvider().RecordCacheMiss("memory")

// Update cache size
metrics.GetProvider().UpdateCacheSize("memory", 1024)

Prometheus Metrics

When using PrometheusProvider, the following metrics are available:

Metric Name	Type	Labels	Description
http_request_duration_seconds	Histogram	method, path, status	HTTP request duration
http_requests_total	Counter	method, path, status	Total HTTP requests
http_requests_in_flight	Gauge	-	Current in-flight requests
db_query_duration_seconds	Histogram	operation, table	Database query duration
db_queries_total	Counter	operation, table, status	Total database queries
cache_hits_total	Counter	provider	Total cache hits
cache_misses_total	Counter	provider	Total cache misses
cache_size_items	Gauge	provider	Current cache size
events_published_total	Counter	source, event_type	Total events published
events_processed_total	Counter	source, event_type, status	Total events processed
event_processing_duration_seconds	Histogram	source, event_type	Event processing duration
event_queue_size	Gauge	-	Current event queue size
panics_total	Counter	method	Total panics recovered

Note: If a custom Namespace is configured, all metric names will be prefixed with {namespace}_.

Prometheus Queries

HTTP Request Rate

rate(http_requests_total[5m])

HTTP Request Duration (95th percentile)

histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

Database Query Error Rate

rate(db_queries_total{status="error"}[5m])

Cache Hit Rate

rate(cache_hits_total[5m]) / (rate(cache_hits_total[5m]) + rate(cache_misses_total[5m]))

No-Op Provider

If metrics are disabled:

// No provider set - uses no-op provider automatically
metrics.GetProvider().RecordHTTPRequest(...) // Does nothing

Custom Provider

Implement your own metrics provider:

type CustomProvider struct{}

func (c *CustomProvider) RecordHTTPRequest(method, path, status string, duration time.Duration) {
    // Send to your metrics system
}

// Implement other Provider interface methods...

func (c *CustomProvider) Handler() http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // Return your metrics format
    })
}

// Use it
metrics.SetProvider(&CustomProvider{})

Pushgateway Usage

Automatic Push (Batch Jobs)

For jobs that run periodically, use automatic pushing:

package main

import (
    "time"

    "github.com/bitechdev/ResolveSpec/pkg/metrics"
)

func main() {
    // Configure with automatic pushing every 30 seconds
    config := &metrics.Config{
        Enabled:             true,
        Provider:            "prometheus",
        Namespace:           "batch_job",
        PushgatewayURL:      "http://pushgateway:9091",
        PushgatewayJobName:  "data-processor",
        PushgatewayInterval: 30, // Push every 30 seconds
    }

    provider := metrics.NewPrometheusProvider(config)
    metrics.SetProvider(provider)

    // Ensure cleanup on exit
    defer provider.StopAutoPush()

    // Your batch job logic here
    processBatchData()
}

Manual Push (Short-lived Processes)

For one-time jobs or when you want manual control:

package main

import (
    "log"

    "github.com/bitechdev/ResolveSpec/pkg/metrics"
)

func main() {
    // Configure without automatic pushing
    config := &metrics.Config{
        Enabled:            true,
        Provider:           "prometheus",
        PushgatewayURL:     "http://pushgateway:9091",
        PushgatewayJobName: "migration-job",
        // PushgatewayInterval: 0 (default - no auto-push)
    }

    provider := metrics.NewPrometheusProvider(config)
    metrics.SetProvider(provider)

    // Run your job
    err := runMigration()

    // Push metrics at the end
    if pushErr := provider.Push(); pushErr != nil {
        log.Printf("Failed to push metrics: %v", pushErr)
    }

    if err != nil {
        log.Fatal(err)
    }
}

Docker Compose with Pushgateway

version: '3'
services:
  batch-job:
    build: .
    environment:
      PUSHGATEWAY_URL: "http://pushgateway:9091"

  pushgateway:
    image: prom/pushgateway
    ports:
      - "9091:9091"

  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'

prometheus.yml for Pushgateway:

global:
  scrape_interval: 15s

scrape_configs:
  # Scrape the pushgateway
  - job_name: 'pushgateway'
    honor_labels: true  # Important: preserve job labels from pushed metrics
    static_configs:
      - targets: ['pushgateway:9091']

Complete Example

Basic Usage

package main

import (
    "database/sql"
    "log"
    "net/http"
    "time"

    "github.com/bitechdev/ResolveSpec/pkg/metrics"
    "github.com/gorilla/mux"
)

func main() {
    // Initialize metrics with default config
    provider := metrics.NewPrometheusProvider(nil)
    metrics.SetProvider(provider)

    // Create router
    router := mux.NewRouter()

    // Apply metrics middleware
    router.Use(provider.Middleware)

    // Expose metrics endpoint
    router.Handle("/metrics", provider.Handler())

    // Your API routes
    router.HandleFunc("/api/users", getUsersHandler)

    log.Fatal(http.ListenAndServe(":8080", router))
}

func getUsersHandler(w http.ResponseWriter, r *http.Request) {
    // Record database query
    start := time.Now()
    users, err := fetchUsers()
    duration := time.Since(start)

    metrics.GetProvider().RecordDBQuery("SELECT", "users", duration, err)

    if err != nil {
        http.Error(w, "Internal Server Error", 500)
        return
    }

    // Return users...
}

With Custom Configuration

package main

import (
    "log"
    "net/http"

    "github.com/bitechdev/ResolveSpec/pkg/metrics"
    "github.com/gorilla/mux"
)

func main() {
    // Custom metrics configuration
    metricsConfig := &metrics.Config{
        Enabled:  true,
        Provider: "prometheus",
        Namespace: "myapp",
        // Custom buckets optimized for your application
        HTTPRequestBuckets: []float64{0.01, 0.05, 0.1, 0.5, 1, 2, 5, 10},
        DBQueryBuckets: []float64{0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1},
    }

    // Initialize with custom config
    provider := metrics.NewPrometheusProvider(metricsConfig)
    metrics.SetProvider(provider)

    router := mux.NewRouter()
    router.Use(provider.Middleware)
    router.Handle("/metrics", provider.Handler())

    log.Fatal(http.ListenAndServe(":8080", router))
}

Docker Compose Example

version: '3'
services:
  app:
    build: .
    ports:
      - "8080:8080"

  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'

  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    depends_on:
      - prometheus

prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'resolvespec'
    static_configs:
      - targets: ['app:8080']

Best Practices

1. Label Cardinality: Keep labels low-cardinality

   • ✅ Good: method, status_code
   • ❌ Bad: user_id, timestamp

2. Path Normalization: Done automatically for the path label (see Configuration Options)

   // /api/users/123 is recorded as /api/users/:id
   

3. Metric Naming: Follow Prometheus conventions

   • Use _total suffix for counters
   • Use _seconds suffix for durations
   • Use base units (seconds, not milliseconds)

4. Performance: Metrics collection is lock-free and highly performant

   • Safe for high-throughput applications
   • Minimal overhead (<1% in most cases)

5. Pull vs Push:

   • Use Pull (default): Long-running services, web servers, microservices
   • Use Push (Pushgateway): Batch jobs, cron tasks, short-lived processes, serverless functions
   • Pull is preferred for most applications as it allows Prometheus to detect if your service is down