feat(cache): 🎉 add message caching functionality

* Implement MessageCache to store events when no webhooks are available. * Add configuration options for enabling cache, setting data path, max age, and max events. * Create API endpoints for managing cached events, including listing, replaying, and deleting. * Integrate caching into the hooks manager to store events when no active webhooks are found. * Enhance logging for better traceability of cached events and operations.
2026-01-30 16:00:34 +02:00
parent 3901bbb668
commit c4d974d6ce
9 changed files with 1535 additions and 30 deletions
--- a/WHATSAPP_BUSINESS.md
+++ b/WHATSAPP_BUSINESS.md
@@ -169,7 +169,8 @@ Update your `config.json` with the Business API configuration:
        "phone_number_id": "123456789012345",
        "access_token": "EAAxxxxxxxxxxxx_your_permanent_token_here",
        "business_account_id": "987654321098765",
-        "api_version": "v21.0"
+        "api_version": "v21.0",
+        "verify_token": "your_secure_random_token_here"
      }
    }
  ]
@@ -187,6 +188,21 @@ Update your `config.json` with the Business API configuration:
 | `access_token` | Yes | Permanent access token (from Step 4) |
 | `business_account_id` | No | WhatsApp Business Account ID (optional, for reference) |
 | `api_version` | No | Graph API version (defaults to `"v21.0"`) |
+| `verify_token` | Yes | Random string for webhook verification (see Step 8a) |
+
+### Step 8a: Generate Verify Token
+
+The verify token is used by Meta to verify your webhook endpoint. Generate a secure random string:
+
+```bash
+# Generate a random token
+openssl rand -hex 32
+
+# Or use any secure random string like:
+# "my_secure_verify_token_abc123xyz789"
+```
+
+Add this token to your `config.json` (see above) and save it - you'll need it for webhook configuration in Step 10.

 ## Step 9: Start WhatsHooked

@@ -197,6 +213,7 @@ Update your `config.json` with the Business API configuration:
 You should see:
 ```
 INFO Business API client connected account_id=business phone=+1234567890
+INFO Hook manager started and subscribed to events event_types=13
 ```

 If you see `Failed to connect client`, check the error message and verify:
@@ -205,8 +222,647 @@ If you see `Failed to connect client`, check the error message and verify:
 3. Access token hasn't expired
 4. Business Account has WhatsApp API access enabled

+## Step 10: Configure Webhook in Meta Developer Console
+
+WhatsHooked provides a webhook endpoint to receive incoming messages and status updates from WhatsApp.
+
+### 10.1: Webhook URL Format
+
+Your webhook URL should be:
+```
+https://your-domain.com/webhooks/whatsapp/{account_id}
+```
+
+Where `{account_id}` matches the `id` field in your config (e.g., "business").
+
+**Example**: If your domain is `api.example.com` and account ID is `business`:
+```
+https://api.example.com/webhooks/whatsapp/business
+```
+
+### 10.2: Configure in Meta Developer Console
+
+1. Go to [Meta Developers](https://developers.facebook.com/)
+2. Select your app
+3. Navigate to **WhatsApp** → **Configuration**
+4. Under "Webhook", click **Edit**
+5. Enter:
+   - **Callback URL**: `https://your-domain.com/webhooks/whatsapp/business`
+   - **Verify Token**: The same token from your `config.json` (`verify_token` field)
+6. Click **Verify and Save**
+
+Meta will send a GET request to verify your endpoint. If verification succeeds, you'll see a green checkmark.
+
+### 10.3: Subscribe to Webhook Events
+
+After verification, subscribe to these webhook fields:
+- ✅ **messages** - Incoming messages and message status updates
+- ✅ **message_template_status_update** - Template approval/rejection (optional)
+- ✅ **account_update** - Account changes (optional)
+- ✅ **phone_number_quality_update** - Quality rating changes (optional)
+
+Click **Subscribe** for each field you want to receive.
+
+## Supported Webhook Events
+
+WhatsHooked supports all WhatsApp Business API webhook events and message types:
+
+### Message Types
+
+| Type | Supported | Downloads Media | Description |
+|------|-----------|-----------------|-------------|
+| `text` | ✅ | N/A | Text messages |
+| `image` | ✅ | ✅ | Images with optional caption |
+| `video` | ✅ | ✅ | Videos with optional caption |
+| `document` | ✅ | ✅ | PDFs, docs, etc. with filename |
+| `audio` | ✅ | ✅ | Voice messages and audio files |
+| `sticker` | ✅ | ✅ | Animated and static stickers |
+| `location` | ✅ | N/A | GPS coordinates with name/address |
+| `contacts` | ✅ | N/A | Shared contact cards (vCard) |
+| `interactive` | ✅ | N/A | Button/list/flow replies |
+| `button` | ✅ | N/A | Quick reply button responses |
+| `reaction` | ✅ | N/A | Emoji reactions to messages |
+| `order` | ✅ | N/A | Catalog/commerce orders |
+| `system` | ✅ | N/A | System notifications |
+
+### Status Updates
+
+| Status | Event | Description |
+|--------|-------|-------------|
+| `sent` | `message.sent` | Message sent from your number |
+| `delivered` | `message.delivered` | Message delivered to recipient |
+| `read` | `message.read` | Message read by recipient |
+| `failed` | `message.failed` | Message delivery failed |
+
+### Webhook Notification Types
+
+| Field | Description | Events Published |
+|-------|-------------|------------------|
+| `messages` | Message events | `message.received`, message status updates |
+| `message_template_status_update` | Template changes | Logged to console |
+| `account_update` | Account config changes | Logged to console |
+| `phone_number_quality_update` | Quality rating changes | Logged to console |
+| `phone_number_name_update` | Display name changes | Logged to console |
+| `account_alerts` | Important alerts | Logged to console |
+
+## Webhook Security
+
+WhatsHooked implements proper webhook security:
+
+1. **Verification**: Uses the `verify_token` to verify Meta's webhook setup request
+2. **Account isolation**: Each account has its own webhook endpoint path
+3. **No authentication required**: Meta's webhooks don't support custom auth headers
+4. **Validation**: Verifies webhook payload structure
+
+### Webhook Verification Flow
+
+```
+Meta sends: GET /webhooks/whatsapp/business?hub.mode=subscribe&hub.verify_token=YOUR_TOKEN&hub.challenge=CHALLENGE
+                                              ↓
+                                    WhatsHooked verifies token
+                                              ↓
+                                 Returns CHALLENGE (200 OK) if valid
+                                       403 Forbidden if invalid
+```
+
+### Receiving Messages
+
+```
+Meta sends: POST /webhooks/whatsapp/business
+                                              ↓
+                                    WhatsHooked processes webhook
+                                              ↓
+                                    Downloads media (if present)
+                                              ↓
+                                    Publishes to event bus
+                                              ↓
+                                    Triggers your configured hooks
+                                              ↓
+                                    Returns 200 OK
+```
+
+## Testing Webhooks
+
+### Test with Meta's Test Button
+
+1. In WhatsApp Configuration → Webhooks
+2. Click **Test** next to "messages"
+3. Select a sample event (e.g., "Text Message")
+4. Click **Send to My Server**
+5. Check WhatsHooked logs for the received event
+
+### Test with Real Messages
+
+1. Send a message to your WhatsApp Business number
+2. Check WhatsHooked logs (set `"log_level": "debug"` for details):
+
+```
+DEBUG Publishing message received event account_id=business message_id=wamid.xxx from=1234567890 type=text
+DEBUG Hook manager received event event_type=message.received
+DEBUG Hook matches event hook_id=message_hook event_type=message.received
+DEBUG Found relevant hooks for event event_type=message.received hook_count=1
+DEBUG Sending to hook hook_id=message_hook url=https://your-webhook.com/messages
+```
+
+3. Your webhook should receive the payload
+
+### Webhook Payload Example
+
+```json
+{
+  "account_id": "business",
+  "message_id": "wamid.HBgNMTIzNDU2Nzg5MAUCABEYEjQyMzRGRDhENzk5MkY5OUFBMQA",
+  "from": "1234567890",
+  "to": "1234567890",
+  "text": "Hello World",
+  "timestamp": "2026-01-30T12:00:00Z",
+  "is_group": false,
+  "sender_name": "John Doe",
+  "message_type": "text"
+}
+```
+
+## Step 11: Configure Your Webhooks
+
+## Step 11: Configure Your Webhooks
+
+WhatsHooked forwards events to your own webhook URLs. Configure them in `config.json`:
+
+```json
+{
+  "hooks": [
+    {
+      "id": "message_hook",
+      "name": "Message Handler",
+      "url": "https://your-app.com/api/whatsapp/messages",
+      "method": "POST",
+      "headers": {
+        "Authorization": "Bearer your-app-token"
+      },
+      "active": true,
+      "events": [
+        "message.received",
+        "message.sent",
+        "message.delivered",
+        "message.read"
+      ],
+      "description": "Receives all message events"
+    }
+  ]
+}
+```
+
+### Hook Configuration Fields
+
+| Field | Required | Description |
+|-------|----------|-------------|
+| `id` | Yes | Unique identifier for this hook |
+| `name` | Yes | Human-readable name |
+| `url` | Yes | Your webhook URL to receive events |
+| `method` | Yes | HTTP method (usually "POST") |
+| `headers` | No | Custom headers (for authentication, etc.) |
+| `active` | Yes | Enable/disable this hook |
+| `events` | No | Event types to receive (empty = all events) |
+| `description` | No | Description for documentation |
+
+### Available Event Types
+
+**Message Events:**
+- `message.received` - Incoming messages
+- `message.sent` - Outgoing messages
+- `message.delivered` - Delivery confirmations
+- `message.read` - Read receipts
+- `message.failed` - Delivery failures
+
+**Connection Events:**
+- `whatsapp.connected` - Account connected
+- `whatsapp.disconnected` - Account disconnected
+
+**QR Code Events** (whatsmeow only):
+- `whatsapp.qr.code` - QR code for pairing
+- `whatsapp.qr.timeout` - QR code expired
+- `whatsapp.qr.error` - QR code error
+
+**Hook Events:**
+- `hook.triggered` - Hook was called
+- `hook.success` - Hook responded successfully
+- `hook.failed` - Hook call failed
+
+### Query Parameters
+
+WhatsHooked automatically adds query parameters to your webhook URL:
+
+```
+https://your-app.com/api/whatsapp/messages?event=message.received&account_id=business
+```
+
+- `event` - The event type
+- `account_id` - The WhatsApp account that triggered the event
+
+## Message Cache System
+
+WhatsHooked includes a message cache that stores events when no active webhooks are configured. This ensures zero message loss.
+
+### Enable Message Cache
+
+Add to your `config.json`:
+
+```json
+{
+  "message_cache": {
+    "enabled": true,
+    "data_path": "./data/message_cache",
+    "max_age_days": 7,
+    "max_events": 10000
+  }
+}
+```
+
+### When Events Are Cached
+
+Events are automatically cached when:
+- No webhooks are configured for the event type
+- All webhooks are inactive (`"active": false`)
+- No webhooks match the event in their `events` array
+
+### Cache Management API
+
+**List cached events:**
+```bash
+curl -u username:password http://localhost:8080/api/cache
+```
+
+**Get cache statistics:**
+```bash
+curl -u username:password http://localhost:8080/api/cache/stats
+```
+
+**Replay all cached events:**
+```bash
+curl -X POST -u username:password http://localhost:8080/api/cache/replay
+```
+
+**Replay specific event:**
+```bash
+curl -X POST -u username:password \
+  "http://localhost:8080/api/cache/event/replay?id=EVENT_ID"
+```
+
+**Delete cached event:**
+```bash
+curl -X DELETE -u username:password \
+  "http://localhost:8080/api/cache/event/delete?id=EVENT_ID"
+```
+
+**Clear all cache:**
+```bash
+curl -X DELETE -u username:password \
+  "http://localhost:8080/api/cache/clear?confirm=true"
+```
+
+### Cache Workflow Example
+
+1. **Disable webhooks** → New messages get cached
+2. **Configure/enable webhooks** → Future messages delivered immediately
+3. **Call replay API** → Cached messages delivered to webhooks
+4. **Successful delivery** → Events removed from cache automatically
+
 ## Troubleshooting

+### Webhooks Not Receiving Events
+
+**Check these items:**
+
+1. **Verify token is correct** in both `config.json` and Meta Developer Console
+2. **Check webhook is active** in Meta console (green checkmark)
+3. **Verify URL is accessible** from internet (Meta needs to reach it)
+4. **Check logs** with `"log_level": "debug"`:
+   ```
+   DEBUG Publishing message received event account_id=business
+   DEBUG Hook manager received event event_type=message.received
+   DEBUG Hook matches event hook_id=message_hook
+   ```
+5. **Test with curl**:
+   ```bash
+   # Send test message to your WhatsApp Business number
+   # Check if webhook receives it
+   ```
+
+### Webhook Verification Fails
+
+**Error**: "The callback URL or verify token couldn't be validated"
+
+**Causes**:
+- `verify_token` mismatch between config.json and Meta console
+- WhatsHooked server not running
+- Firewall blocking Meta's IP ranges
+- Wrong webhook URL format
+
+**Fix**:
+1. Ensure server is running: `./bin/whatshook-server -config config.json`
+2. Check logs for verification attempt
+3. Verify token matches exactly (case-sensitive)
+4. Test URL is accessible: `curl https://your-domain.com/webhooks/whatsapp/business`
+
+### Messages Not Cached
+
+**Check**:
+1. `message_cache.enabled` is `true` in config
+2. Hooks are actually inactive or not matching events
+3. Check cache stats: `curl -u user:pass http://localhost:8080/api/cache/stats`
+
+### No Hooks Configured Error
+
+If events are being cached but you have hooks configured, check:
+- Hook `"active"` is `true`
+- Hook `"events"` array includes the event type (or is empty for all events)
+- Hook URL is reachable and responding with 2xx status
+
+Enable debug logging to trace the issue:
+```json
+{
+  "log_level": "debug"
+}
+```
+
+## Webhook Payload Examples
+
+### Text Message
+
+```json
+{
+  "account_id": "business",
+  "message_id": "wamid.HBgNMTIzNDU2Nzg5MAUCABEYEjQyMzRGRDhENzk5MkY5OUFBMQA",
+  "from": "1234567890",
+  "to": "1234567890",
+  "text": "Hello, how can I help?",
+  "timestamp": "2026-01-30T12:00:00Z",
+  "is_group": false,
+  "sender_name": "John Doe",
+  "message_type": "text"
+}
+```
+
+### Image Message (with media)
+
+```json
+{
+  "account_id": "business",
+  "message_id": "wamid.xxx",
+  "from": "1234567890",
+  "to": "1234567890",
+  "text": "Check this out!",
+  "timestamp": "2026-01-30T12:00:00Z",
+  "is_group": false,
+  "sender_name": "John Doe",
+  "message_type": "image",
+  "media": {
+    "type": "image",
+    "mime_type": "image/jpeg",
+    "filename": "wamid.xxx_a1b2c3d4.jpg",
+    "url": "http://localhost:8080/api/media/business/wamid.xxx_a1b2c3d4.jpg",
+    "base64": "..." // Only if media.mode is "base64" or "both"
+  }
+}
+```
+
+### Location Message
+
+```json
+{
+  "account_id": "business",
+  "message_id": "wamid.xxx",
+  "from": "1234567890",
+  "to": "1234567890",
+  "text": "Location: Office (123 Main St) - 40.712800, -74.006000",
+  "timestamp": "2026-01-30T12:00:00Z",
+  "is_group": false,
+  "sender_name": "John Doe",
+  "message_type": "location"
+}
+```
+
+### Button Reply (Interactive)
+
+```json
+{
+  "account_id": "business",
+  "message_id": "wamid.xxx",
+  "from": "1234567890",
+  "to": "1234567890",
+  "text": "Yes, I'm interested",
+  "timestamp": "2026-01-30T12:00:00Z",
+  "is_group": false,
+  "sender_name": "John Doe",
+  "message_type": "interactive"
+}
+```
+
+### Delivery Status
+
+```json
+{
+  "event_type": "message.delivered",
+  "timestamp": "2026-01-30T12:00:05Z",
+  "data": {
+    "account_id": "business",
+    "message_id": "wamid.xxx",
+    "from": "1234567890",
+    "timestamp": "2026-01-30T12:00:05Z"
+  }
+}
+```
+
+## Complete Configuration Example
+
+Here's a complete `config.json` with all Business API features:
+
+```json
+{
+  "server": {
+    "host": "0.0.0.0",
+    "port": 8080,
+    "default_country_code": "1",
+    "username": "admin",
+    "password": "secure_password",
+    "auth_key": "optional_api_key"
+  },
+  "whatsapp": [
+    {
+      "id": "business",
+      "type": "business-api",
+      "phone_number": "+1234567890",
+      "business_api": {
+        "phone_number_id": "123456789012345",
+        "access_token": "EAAxxxxxxxxxxxx",
+        "business_account_id": "987654321098765",
+        "api_version": "v21.0",
+        "verify_token": "my_secure_random_token_abc123"
+      }
+    }
+  ],
+  "hooks": [
+    {
+      "id": "message_hook",
+      "name": "Message Handler",
+      "url": "https://your-app.com/api/whatsapp/messages",
+      "method": "POST",
+      "headers": {
+        "Authorization": "Bearer your-app-secret-token",
+        "X-Custom-Header": "value"
+      },
+      "active": true,
+      "events": [
+        "message.received",
+        "message.sent",
+        "message.delivered",
+        "message.read"
+      ],
+      "description": "Handles all message events"
+    },
+    {
+      "id": "status_hook",
+      "name": "Connection Monitor",
+      "url": "https://your-app.com/api/whatsapp/status",
+      "method": "POST",
+      "active": true,
+      "events": [
+        "whatsapp.connected",
+        "whatsapp.disconnected"
+      ],
+      "description": "Monitors connection status"
+    }
+  ],
+  "media": {
+    "data_path": "./data/media",
+    "mode": "link",
+    "base_url": "https://your-domain.com"
+  },
+  "message_cache": {
+    "enabled": true,
+    "data_path": "./data/message_cache",
+    "max_age_days": 7,
+    "max_events": 10000
+  },
+  "event_logger": {
+    "enabled": true,
+    "targets": ["file", "sqlite"],
+    "file_dir": "./data/events",
+    "table_name": "event_logs"
+  },
+  "log_level": "info"
+}
+```
+
+## Advanced Features
+
+### Media Handling Modes
+
+WhatsHooked supports three media delivery modes:
+
+**1. Link Mode** (default, recommended)
+```json
+{
+  "media": {
+    "mode": "link",
+    "base_url": "https://your-domain.com"
+  }
+}
+```
+- Downloads media and stores locally
+- Webhooks receive URL: `https://your-domain.com/api/media/business/filename.jpg`
+- Efficient for large media files
+
+**2. Base64 Mode**
+```json
+{
+  "media": {
+    "mode": "base64"
+  }
+}
+```
+- Encodes media as base64 in webhook payload
+- No separate download needed
+- Good for small files, increases payload size
+
+**3. Both Mode**
+```json
+{
+  "media": {
+    "mode": "both"
+  }
+}
+```
+- Provides both URL and base64
+- Maximum flexibility, largest payloads
+
+### Event Logger
+
+Track all events to file and/or database:
+
+```json
+{
+  "event_logger": {
+    "enabled": true,
+    "targets": ["file", "sqlite", "postgres"],
+    "file_dir": "./data/events",
+    "table_name": "event_logs"
+  },
+  "database": {
+    "type": "postgres",
+    "host": "localhost",
+    "port": 5432,
+    "username": "whatshooked",
+    "password": "password",
+    "database": "whatshooked"
+  }
+}
+```
+
+Logged events include:
+- All message events
+- Connection status changes
+- Hook success/failure
+- Webhook triggers
+
+### Two-Way Communication
+
+Your webhooks can respond to trigger outgoing messages:
+
+**Webhook Response Format:**
+```json
+{
+  "send_message": true,
+  "to": "1234567890",
+  "text": "Thanks for your message!",
+  "account_id": "business"
+}
+```
+
+This sends a reply immediately when your webhook receives an event.
+
+## Production Deployment Checklist
+
+Before going live:
+
+- [ ] Use a System User token (not personal user token)
+- [ ] Set `verify_token` to a secure random string (32+ characters)
+- [ ] Configure webhooks in Meta Developer Console
+- [ ] Subscribe to required webhook fields (messages, etc.)
+- [ ] Test webhook verification succeeds
+- [ ] Enable HTTPS for production (required by Meta)
+- [ ] Set up firewall rules to allow Meta's webhook IPs
+- [ ] Configure authentication (`username`/`password` or `auth_key`)
+- [ ] Enable message cache for reliability
+- [ ] Set up event logging for audit trail
+- [ ] Test sending and receiving messages
+- [ ] Monitor logs for errors
+- [ ] Set up log rotation for production
+- [ ] Document your webhook endpoints
+- [ ] Set up monitoring/alerts for webhook failures
+
+## Troubleshooting Common Issues
+
 ### Error: "Object with ID does not exist" (error_subcode: 33)

 **Cause**: One of the following: