ResolveSpec/pkg/server/README.md

Server Package

Graceful HTTP server with request draining and shutdown coordination.

Quick Start

import "github.com/bitechdev/ResolveSpec/pkg/server"

// Create server
srv := server.NewGracefulServer(server.Config{
    Addr:    ":8080",
    Handler: router,
})

// Start server (blocks until shutdown signal)
if err := srv.ListenAndServe(); err != nil {
    log.Fatal(err)
}

Features

✅ Graceful shutdown on SIGINT/SIGTERM ✅ Request draining (waits for in-flight requests) ✅ Automatic request rejection during shutdown ✅ Health and readiness endpoints ✅ Shutdown callbacks for cleanup ✅ Configurable timeouts

Configuration

config := server.Config{
    // Server address
    Addr: ":8080",

    // HTTP handler
    Handler: myRouter,

    // Maximum time for graceful shutdown (default: 30s)
    ShutdownTimeout: 30 * time.Second,

    // Time to wait for in-flight requests (default: 25s)
    DrainTimeout: 25 * time.Second,

    // Request read timeout (default: 10s)
    ReadTimeout: 10 * time.Second,

    // Response write timeout (default: 10s)
    WriteTimeout: 10 * time.Second,

    // Idle connection timeout (default: 120s)
    IdleTimeout: 120 * time.Second,
}

srv := server.NewGracefulServer(config)

Shutdown Behavior

Signal received (SIGINT/SIGTERM):

1. Mark as shutting down - New requests get 503
2. Drain requests - Wait up to DrainTimeout for in-flight requests
3. Shutdown server - Close listeners and connections
4. Execute callbacks - Run registered cleanup functions

Time   Event
─────────────────────────────────────────
0s     Signal received: SIGTERM
       ├─ Mark as shutting down
       ├─ Reject new requests (503)
       └─ Start draining...

1s     In-flight: 50 requests
2s     In-flight: 32 requests
3s     In-flight: 12 requests
4s     In-flight: 3 requests
5s     In-flight: 0 requests ✓
       └─ All requests drained

5s     Execute shutdown callbacks
6s     Shutdown complete

Health Checks

Health Endpoint

Returns 200 when healthy, 503 when shutting down:

router.HandleFunc("/health", srv.HealthCheckHandler())

Response (healthy):

{"status":"healthy"}

Response (shutting down):

{"status":"shutting_down"}

Readiness Endpoint

Includes in-flight request count:

router.HandleFunc("/ready", srv.ReadinessHandler())

Response:

{"ready":true,"in_flight_requests":12}

During shutdown:

{"ready":false,"reason":"shutting_down"}

Shutdown Callbacks

Register cleanup functions to run during shutdown:

// Close database
server.RegisterShutdownCallback(func(ctx context.Context) error {
    logger.Info("Closing database connection...")
    return db.Close()
})

// Flush metrics
server.RegisterShutdownCallback(func(ctx context.Context) error {
    logger.Info("Flushing metrics...")
    return metricsProvider.Flush(ctx)
})

// Close cache
server.RegisterShutdownCallback(func(ctx context.Context) error {
    logger.Info("Closing cache...")
    return cache.Close()
})

Complete Example

package main

import (
    "context"
    "log"
    "net/http"
    "time"

    "github.com/bitechdev/ResolveSpec/pkg/middleware"
    "github.com/bitechdev/ResolveSpec/pkg/metrics"
    "github.com/bitechdev/ResolveSpec/pkg/server"
    "github.com/gorilla/mux"
)

func main() {
    // Initialize metrics
    metricsProvider := metrics.NewPrometheusProvider()
    metrics.SetProvider(metricsProvider)

    // Create router
    router := mux.NewRouter()

    // Apply middleware
    rateLimiter := middleware.NewRateLimiter(100, 20)
    sizeLimiter := middleware.NewRequestSizeLimiter(middleware.Size10MB)
    sanitizer := middleware.DefaultSanitizer()

    router.Use(rateLimiter.Middleware)
    router.Use(sizeLimiter.Middleware)
    router.Use(sanitizer.Middleware)
    router.Use(metricsProvider.Middleware)

    // API routes
    router.HandleFunc("/api/data", dataHandler)

    // Create graceful server
    srv := server.NewGracefulServer(server.Config{
        Addr:            ":8080",
        Handler:         router,
        ShutdownTimeout: 30 * time.Second,
        DrainTimeout:    25 * time.Second,
    })

    // Health checks
    router.HandleFunc("/health", srv.HealthCheckHandler())
    router.HandleFunc("/ready", srv.ReadinessHandler())

    // Metrics endpoint
    router.Handle("/metrics", metricsProvider.Handler())

    // Register shutdown callbacks
    server.RegisterShutdownCallback(func(ctx context.Context) error {
        log.Println("Cleanup: Flushing metrics...")
        return nil
    })

    server.RegisterShutdownCallback(func(ctx context.Context) error {
        log.Println("Cleanup: Closing database...")
        // return db.Close()
        return nil
    })

    // Start server (blocks until shutdown)
    log.Printf("Starting server on :8080")
    if err := srv.ListenAndServe(); err != nil {
        log.Fatal(err)
    }

    // Wait for shutdown to complete
    srv.Wait()
    log.Println("Server stopped")
}

func dataHandler(w http.ResponseWriter, r *http.Request) {
    // Your handler logic
    time.Sleep(100 * time.Millisecond) // Simulate work
    w.WriteHeader(http.StatusOK)
    w.Write([]byte(`{"message":"success"}`))
}

Kubernetes Integration

Deployment with Probes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: app
        image: myapp:latest
        ports:
        - containerPort: 8080

        # Liveness probe - is app running?
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 5

        # Readiness probe - can app handle traffic?
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          timeoutSeconds: 3

        # Graceful shutdown
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 5"]

        # Environment
        env:
        - name: SHUTDOWN_TIMEOUT
          value: "30"

Service

apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp
  ports:
  - port: 80
    targetPort: 8080
  type: LoadBalancer

Docker Compose

version: '3.8'
services:
  app:
    build: .
    ports:
      - "8080:8080"
    environment:
      - SHUTDOWN_TIMEOUT=30
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 10s
    stop_grace_period: 35s  # Slightly longer than shutdown timeout

Testing Graceful Shutdown

Test Script

#!/bin/bash

# Start server in background
./myapp &
SERVER_PID=$!

# Wait for server to start
sleep 2

# Send some requests
for i in {1..10}; do
    curl http://localhost:8080/api/data &
done

# Wait a bit
sleep 1

# Send shutdown signal
kill -TERM $SERVER_PID

# Try to send more requests (should get 503)
curl -v http://localhost:8080/api/data

# Wait for server to stop
wait $SERVER_PID
echo "Server stopped gracefully"

Expected Output

Starting server on :8080
Received signal: terminated, initiating graceful shutdown
Starting graceful shutdown...
Waiting for 8 in-flight requests to complete...
Waiting for 4 in-flight requests to complete...
Waiting for 1 in-flight requests to complete...
All requests drained in 2.3s
Cleanup: Flushing metrics...
Cleanup: Closing database...
Shutting down HTTP server...
Graceful shutdown complete
Server stopped

Monitoring In-Flight Requests

// Get current in-flight count
count := srv.InFlightRequests()
fmt.Printf("In-flight requests: %d\n", count)

// Check if shutting down
if srv.IsShuttingDown() {
    fmt.Println("Server is shutting down")
}

Advanced Usage

Custom Shutdown Logic

// Implement custom shutdown
go func() {
    sigChan := make(chan os.Signal, 1)
    signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM)

    <-sigChan
    log.Println("Shutdown signal received")

    // Custom pre-shutdown logic
    log.Println("Running custom cleanup...")

    // Shutdown with callbacks
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    if err := srv.ShutdownWithCallbacks(ctx); err != nil {
        log.Printf("Shutdown error: %v", err)
    }
}()

// Start server
srv.server.ListenAndServe()

Multiple Servers

// HTTP server
httpSrv := server.NewGracefulServer(server.Config{
    Addr:    ":8080",
    Handler: httpRouter,
})

// HTTPS server
httpsSrv := server.NewGracefulServer(server.Config{
    Addr:    ":8443",
    Handler: httpsRouter,
})

// Start both
go httpSrv.ListenAndServe()
go httpsSrv.ListenAndServe()

// Shutdown both on signal
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, os.Interrupt)
<-sigChan

ctx := context.Background()
httpSrv.Shutdown(ctx)
httpsSrv.Shutdown(ctx)

Best Practices

1. Set appropriate timeouts

   • DrainTimeout < ShutdownTimeout
   • ShutdownTimeout < Kubernetes terminationGracePeriodSeconds

2. Register cleanup callbacks for:

   • Database connections
   • Message queues
   • Metrics flushing
   • Cache shutdown
   • Background workers

3. Health checks

   • Use /health for liveness (is app alive?)
   • Use /ready for readiness (can app serve traffic?)

4. Load balancer considerations

   • Set preStop hook in Kubernetes (5-10s delay)
   • Allows load balancer to deregister before shutdown

5. Monitoring

   • Track in-flight requests in metrics
   • Alert on slow drains
   • Monitor shutdown duration

Troubleshooting

Shutdown Takes Too Long

// Increase drain timeout
config.DrainTimeout = 60 * time.Second

Requests Still Timing Out

// Increase write timeout
config.WriteTimeout = 30 * time.Second

Force Shutdown Not Working

The server will force shutdown after ShutdownTimeout even if requests are still in-flight. Adjust timeouts as needed.

Debugging Shutdown

// Enable debug logging
import "github.com/bitechdev/ResolveSpec/pkg/logger"

logger.SetLevel("debug")