Skip to main content

Webhooks & Event Processing

🔔 Webhook Event Processing​

Email Event Webhook​

Endpoint: POST /v1/integrations/sendgrid/webhook

Purpose: Receive and process email delivery events from Sendgrid

Sendgrid Events: delivered, open, click, bounce, dropped, spamreport, unsubscribe

Request Body (from Sendgrid):

[
  {
    "email": "recipient@example.com",
    "timestamp": 1704844800,
    "smtp-id": "<unique-id@sendgrid.com>",
    "event": "delivered",
    "category": [],
    "sg_event_id": "abc123...",
    "sg_message_id": "msg_id.filter_id.123.456...",
    "account_id": "507f191e810c19729de860ea",
    "uid": "507f191e810c19729de860eb"
  },
  {
    "email": "recipient@example.com",
    "timestamp": 1704844850,
    "event": "open",
    "sg_event_id": "def456...",
    "sg_message_id": "msg_id.filter_id.123.456...",
    "useragent": "Mozilla/5.0...",
    "ip": "192.168.1.1"
  }
]

📊 Event Processing Logic​

Message ID Extraction​

// Extract msg_id from sg_message_id
// Format: msg_id.filter_id.timestamp.sequence
const msgId = event.sg_message_id.split('.')[0];

Database Update​

  1. Find Communication Document:

    const communication = await Communication.findOne({
      msgID: msgId,
    });

  2. Append Event (with deduplication):

    await Communication.updateOne(
      { msgID: msgId },
      {
        $addToSet: {
          events: {
            event: event.event,
            email: event.email,
            timestamp: event.timestamp,
            sg_event_id: event.sg_event_id,
            sg_message_id: event.sg_message_id,
            account_id: event.account_id,
            uid: event.uid,
            useragent: event.useragent,
            ip: event.ip,
            url: event.url, // For click events
          },
        },
      },
    );

  3. Event Deduplication: Uses sg_event_id for uniqueness via $addToSet

Special Event Handling​

Open Events​

Purpose: Mark message as read in conversation UI

Process:

  1. Find message in support_messages collection
  2. Emit supportMessageAck socket event for read receipt

Socket Event:

axios.post(
  `${process.env.CONVERSATION_SOCKET}/emit`,
  {
    event: 'supportMessageAck',
    data: {
      roomId: room._id.toString(),
      messageId: message_id.toString(),
    },
  },
  {
    headers: {
      Authorization: `Bearer ${jwt_token}`,
      'x-account-id': account_id.toString(),
    },
  },
);

Unsubscribe Events​

Purpose: Automatically add email to Do Not Disturb list

Process:

await DND.create({
  account_id: ObjectId(account_id),
  value: event.email,
  type: 'permanent',
  reason: 'Unsubscribed via Sendgrid webhook',
});

DND Collection:

{
  _id: ObjectId,
  account_id: ObjectId,
  value: String,             // Email address
  type: String,              // "permanent"
  reason: String,
  createdAt: Date
}

📨 Inbound Email Webhook​

Inbound Email Parser​

Endpoint: POST /v1/integrations/sendgrid/webhook/inbound

Purpose: Parse incoming email replies and route to conversation system

Request Body (parsed email from Sendgrid):

{
  "headers": "Message-ID: <msg123@example.com>\nIn-Reply-To: <ref456@example.com>\n...",
  "from": "John Doe <john@example.com>",
  "to": "support@mail.dashclicks.com",
  "subject": "Re: Support Request",
  "text": "Plain text content",
  "html": "<p>HTML content</p>",
  "envelope": "{\"to\":[\"support@mail.dashclicks.com\"],\"from\":\"john@example.com\"}"
}

🔄 Inbound Email Processing Flow​

1. Extract Headers​

// Parse headers string
const headerLines = req.body.headers.split('\n');

// Extract In-Reply-To header (for threading)
const inReplyToHeader = headerLines.find(line => line.startsWith('In-Reply-To:'));
const referenceID = inReplyToHeader ? inReplyToHeader.split('<')[1].split('>')[0] : null;

// Extract Message-ID
const messageIdHeader = headerLines.find(line => line.startsWith('Message-ID:'));
const msgID = messageIdHeader ? messageIdHeader.split('<')[1].split('>')[0] : null;

2. Find Original Message​

const originalMessage = await Communication.findOne({
  msgID: referenceID,
  module: 'SENDGRID',
});

Email Threading:

Original email (from DashClicks):
  Message-ID: <abc123@mail.dashclicks.com>
  Stored in: communication.msgID = "abc123"

Reply email (from contact):
  In-Reply-To: <abc123@mail.dashclicks.com>
  Webhook extracts: referenceID = "abc123"
  Finds original: communication.msgID === "abc123"
  Routes to: Same conversation/room

3. Find or Create Contact​

// Extract sender email
const senderEmail = req.body.from.match(/<(.+)>/)?.[1] || req.body.from;

// Find existing contact
let contact = await Contact.findOne({
  email: senderEmail,
  parent_account: account_id,
});

// Create if not exists
if (!contact) {
  contact = await Contact.create({
    email: senderEmail,
    name: req.body.from.split('<')[0].trim(),
    parent_account: account_id,
  });
}

4. Find or Create Support Conversation​

// Check if support conversation exists
let supportConversation = await SupportConversation.findOne({
  contact_id: contact._id,
  account_id: account_id,
  is_closed: false,
});

// Create if not exists
if (!supportConversation) {
  supportConversation = await SupportConversation.create({
    contact_id: contact._id,
    account_id: account_id,
    subject: req.body.subject,
    created_at: new Date(),
  });
}

5. Find or Create Support Room​

// Check for existing thread in message lock
let room = await SupportMessageLock.findOne({
  support_conversation_id: supportConversation._id,
});

// Create room if new conversation
if (!room) {
  room = await SupportRoom.create({
    account_id: account_id,
    conversation_id: supportConversation._id,
    participants: [contact._id],
  });

  // Link to default email inbound inbox
  const defaultInbox = await SupportInbox.findOne({
    account_id: account_id,
    type: 'email_inbound',
    is_default: true,
  });

  if (defaultInbox) {
    await SupportRoom.updateOne({ _id: room._id }, { $set: { inbox_id: defaultInbox._id } });
  }
}

6. Store Messages​

Create Two Support Messages:

  1. Original Outbound Message (if new thread):

    if (originalMessage) {
      await SupportMessage.create({
        room_id: room._id,
        message: originalMessage.html || originalMessage.text,
        sent_by: originalMessage.sent_by,
        message_type: 'OUTGOING',
        created_at: originalMessage.createdAt,
      });
    }

  2. Inbound Reply Message:

    const inboundMessage = await SupportMessage.create({
      room_id: room._id,
      message: req.body.html || req.body.text,
      sent_by: contact._id,
      message_type: 'INCOMING',
      created_at: new Date(),
    });

Create Communication Record:

await Communication.create({
  account_id: account_id,
  contact_id: [contact._id],
  sent_by: contact._id,
  module: 'SENDGRID',
  message_type: 'EMAIL',
  type: 'INCOMING',
  from: senderEmail,
  to: [req.body.to],
  subject: req.body.subject,
  html: req.body.html,
  text: req.body.text,
  msgID: msgID,
  referenceID: referenceID,
  success: true,
});

7. Handle Attachments​

Upload to Wasabi:

if (req.files && req.files.length > 0) {
  const wasabiConfig = await WasabiConfig.findConfig();
  const uploadedAttachments = await wasabiProvider.upload(wasabiConfig, req.files);

  await SupportMessage.updateOne(
    { _id: inboundMessage._id },
    {
      $set: {
        attachments: uploadedAttachments,
      },
    },
  );
}

8. Emit Socket Events​

// Join room event
await axios.post(`${process.env.CONVERSATION_SOCKET}/emit`, {
  event: 'joinRoom',
  data: {
    roomId: room._id.toString(),
    userId: contact._id.toString(),
  },
});

// New message event
await socketProvider.emitMessage(inboundMessage._id);

🛠️ Webhook Configuration​

Create Event Webhook​

Endpoint: POST /v1/integrations/sendgrid/webhook/create

Purpose: Configure Sendgrid to send events to DashClicks

Process:

  1. Get subuser from account
  2. Create or update event webhook via Sendgrid API

Sendgrid API Call:

POST https://api.sendgrid.com/v3/user/webhooks/event/settings
Authorization: Bearer {api_key}
On-Behalf-Of: {subuser_username}
Content-Type: application/json

{
  "enabled": true,
  "url": "https://api.dashclicks.com/v1/e/sendgrid/webhook",
  "group_resubscribe": true,
  "delivered": true,
  "group_unsubscribe": true,
  "spam_report": true,
  "bounce": true,
  "deferred": false,
  "unsubscribe": true,
  "dropped": true,
  "open": true,
  "click": true,
  "processed": false
}

Response:

{
  "success": true,
  "message": "SUCCESS",
  "data": {
    "enabled": true,
    "url": "https://api.dashclicks.com/v1/e/sendgrid/webhook",
    "delivered": true,
    "open": true,
    "click": true
    // ...
  }
}

Create Inbound Parse Webhook​

Purpose: Configure Sendgrid to parse incoming emails

Requirements:

  • Domain/subdomain MX records must point to Sendgrid
  • Example MX record: mx.sendgrid.net with priority 10

Configuration:

POST https://api.sendgrid.com/v3/user/webhooks/parse/settings
Authorization: Bearer {api_key}
On-Behalf-Of: {subuser_username}

{
  "hostname": "mail.yourdomain.com",
  "url": "https://api.dashclicks.com/v1/e/sendgrid/webhook/inbound",
  "spam_check": true,
  "send_raw": false
}

🎛️ WebHookController​

Location: Controllers/WebHookController.js

createWebhook()​

Route: POST /v1/integrations/sendgrid/webhook/create

Logic:

  1. Get subuser from account document
  2. Create or update event webhook
  3. Subscribe to all relevant events

webhook() - Event Listener​

Route: POST /v1/integrations/sendgrid/webhook

Logic:

  1. Extract msg_id from sg_message_id
  2. Find tracker document
  3. Append event to events array (dedupe by sg_event_id)
  4. Special handling for open and unsubscribe events
  5. Return 200 OK (always, even on errors)

Response: Always returns 200 OK to prevent Sendgrid retries

webhookInbound() - Inbound Email Parser​

Route: POST /v1/integrations/sendgrid/webhook/inbound

Logic:

  1. Extract email headers
  2. Find original message
  3. Find or create contact
  4. Find or create support conversation
  5. Create support room
  6. Store messages
  7. Upload attachments
  8. Emit socket events

Response: 200 OK

🧪 Testing​

Test Event Webhook​

curl -X POST http://localhost:5003/v1/e/sendgrid/webhook \
  -H "Content-Type: application/json" \
  -d '[{
    "email": "test@example.com",
    "timestamp": 1704844800,
    "event": "delivered",
    "sg_event_id": "test_event_123",
    "sg_message_id": "msg_id.filter_id.123.456",
    "account_id": "507f191e810c19729de860ea",
    "uid": "507f191e810c19729de860eb"
  }]'

Test Inbound Webhook​

curl -X POST http://localhost:5003/v1/e/sendgrid/webhook/inbound \
  -H "Content-Type: application/json" \
  -d '{
    "headers": "Message-ID: <new@example.com>\nIn-Reply-To: <original@dashclicks.com>",
    "from": "John Doe <john@example.com>",
    "to": "support@mail.dashclicks.com",
    "subject": "Re: Support Request",
    "text": "Thank you for your help!",
    "html": "<p>Thank you for your help!</p>"
  }'

Create Webhook Configuration​

curl -X POST http://localhost:5003/v1/e/sendgrid/webhook/create \
  -H "Authorization: Bearer {jwt_token}" \
  -H "Content-Type: application/json"

📊 Event Types​

Delivery Events​

  • delivered: Email successfully delivered to recipient's server
  • deferred: Recipient's server temporarily rejected (will retry)
  • bounce: Permanent delivery failure
  • dropped: Sendgrid dropped email (invalid recipient, unsubscribed)

Engagement Events​

  • open: Recipient opened the email
  • click: Recipient clicked a link in the email

List Management Events​

  • spamreport: Recipient marked as spam
  • unsubscribe: Recipient unsubscribed
  • group_unsubscribe: Unsubscribed from suppression group
  • group_resubscribe: Resubscribed to suppression group

⚡ Performance Considerations​

Processing Times:

  • Event webhook: ~10-50ms per event
  • DND insertion: ~5-10ms
  • Socket event emission: ~50-100ms
  • Inbound email parsing: ~100-300ms
  • Attachment upload: ~100-500ms per file

Optimization Tips:

  1. Always Return 200 OK: Prevents Sendgrid retries even on error
  2. Async Socket Events: Emit after webhook response
  3. Batch Event Processing: Process multiple events in single request
  4. Deduplicate Events: Use $addToSet with sg_event_id

⚠️ Important Notes​

  • 📨 Always Return 200: Webhook must return 200 OK to prevent retries
  • 🔄 Email Threading: Uses Message-ID and In-Reply-To headers
  • 🚫 Auto-DND: Unsubscribe events automatically add to Do Not Disturb list
  • 📊 Event Deduplication: sg_event_id ensures no duplicate events
  • 🔔 Real-time Updates: Socket events for conversation UI updates
  • 💾 Attachment Storage: Inbound attachments uploaded to Wasabi
  • 🎯 Message Linking: Inbound emails linked to original outbound messages
  • 📧 Support Integration: Inbound emails create support conversations
  • 🏢 Room Creation: Each conversation thread gets dedicated support room
  • 🔍 Contact Lookup: Sender email used to find or create contact

🚨 Troubleshooting​

Webhook Events Not Processing​

Diagnosis:

  1. Check webhook configuration in Sendgrid dashboard
  2. Verify webhook URL is publicly accessible
  3. Check communication collection for message: 
    db.communications.findOne({ msgID: 'sendgrid_message_id' });
  4. Verify sg_message_id format: msg_id.filter_id.timestamp.sequence

Inbound Emails Not Creating Conversations​

Diagnosis:

  1. Check Sendgrid Inbound Parse configuration
  2. Verify MX records point to Sendgrid
  3. Check original message exists: 
    db.communications.findOne({ msgID: 'original_message_id' });
  4. Check contact exists: 
    db.contacts.findOne({ email: 'sender@example.com' });
  5. Verify CONVERSATION_SOCKET environment variable

🔗 Related Documentation​

💬

Documentation Assistant

Ask me anything about the docs

Hi! I'm your documentation assistant. Ask me anything about the docs!

I can help you with:
- Code examples
- Configuration details
- Troubleshooting
- Best practices

Try asking: How do I configure the API?
09:31 AM