Inbound Webhooks Service

📖 Overview

The Webhooks Controller handles incoming webhook requests from 10+ marketing platforms and processes lead data through a unified pipeline. It is the most complex controller in the Inbound module and CRITICAL to the system's operation.

Core Responsibilities

• Multi-Platform Support: Facebook Lead Ads, ClickFunnels, ManyChat, Unbounce, Instapage, Kartra, PhoneSites, CallRail, Call Tracking Metrics
• Lead Capture: Parse webhook payloads and store lead data
• Contact Creation: Auto-create CRM contacts with duplicate detection
• Deal Automation: Create deals in CRM pipelines based on campaign settings
• Notification Routing: Send email/SMS notifications to assigned representatives
• Retry Logic: Handle webhook failures with exponential backoff (3 attempts)
• Field Extraction: Intelligent field mapping from varied webhook formats
• Phone Formatting: International phone number normalization
• Support Conversations: Auto-create support tickets from leads

---

🗄️ Collections Used

📚 Full Schema: See Database Collections Documentation

inbound.leads

• Operations: Read/Write for lead storage and deduplication
• Model: shared/models/leads-data.js
• Usage: Stores all inbound leads with raw webhook data

campaigns

• Operations: Read for campaign configuration
• Model: shared/models/campaign-data.js
• Usage: Integration settings, CRM mapping, webhook routing

facebook.keys

• Operations: Read for Facebook access token retrieval
• Model: shared/models/facebookads-key.js
• Usage: Facebook API authentication and token refresh

crm.contacts

• Operations: Read/Write for contact creation and updates
• Model: shared/models/contact.js
• Usage: Create/update business and person contacts

crm.deals

• Operations: Write for deal creation from leads
• Model: shared/models/deal.js
• Usage: Automatic deal creation when configured

support.conversations

• Operations: Write for support ticket creation
• Model: shared/models/conversation.js
• Usage: Auto-create support conversations from leads

support.rooms

• Operations: Write for conversation rooms
• Model: shared/models/room.js
• Usage: Support conversation thread management

support.messages

• Operations: Write for messages
• Model: shared/models/message.js
• Usage: Support conversation messages

logs

• Operations: Write for retry tracking
• Model: shared/models/logs.js
• Usage: Webhook retry attempts (max 3)

---

🔄 Common Webhook Processing Pattern

All webhook handlers follow a consistent pattern:

flowchart TD
    A[Webhook Received] --> B{Validate Payload}
    B -->|Invalid| C[Return 200 OK]
    B -->|Valid| D{Check Duplicate Lead}

    D -->|Duplicate| C
    D -->|New| E{Find Campaign}

    E -->|Not Found| C
    E -->|Found| F{Campaign Active?}

    F -->|Inactive| C
    F -->|Active| G{Account Active?}

    G -->|Inactive| H[Skip to next campaign]
    G -->|Active| I[Extract Lead Data]

    I --> J{add_to_crm enabled?}

    J -->|Yes| K[Extract Contact Fields]
    K --> L[Format Phone with Country Code]
    L --> M[Create Contact in CRM]
    J -->|No| N[Skip Contact Creation]

    M --> O[Find/Assign Representative]
    N --> O

    O --> P[Create Support Conversation]
    P --> Q[Save Lead to Database]

    Q --> R{Notifications?}

    R -->|Yes| S[Send FCM/Email/SMS]
    R -->|No| T[Skip Notifications]

    S --> U{create_deal enabled?}
    T --> U

    U -->|Yes| V[Create Deal in Pipeline]
    U -->|No| W[Skip Deal Creation]

    V --> X[Update Lead with Errors Array]
    W --> X

    X --> Y[Emit Socket Event: leadadded]
    Y --> Z[Return 200 OK]

---

🎯 Webhook Handlers

1. Facebook Lead Ads - receiveWebhooks

Purpose: Handles Facebook Lead Ads webhook callbacks for lead capture and verification.

HTTP Method: GET (verification) / POST (webhook data)

Authentication: None (uses verification token)

Endpoint Pattern: /v1/inbound/webhooks/facebook

Webhook Verification (GET)

Facebook sends a verification request on webhook setup:

Request:

GET /v1/inbound/webhooks/facebook?hub.verify_token=TOKEN&hub.challenge=CHALLENGE&hub.mode=subscribe

Response:

// Success - Returns challenge
res.send(req.query['hub.challenge']);

// Failure - Invalid token
{
  success: true; // Still returns 200 to prevent webhook issues
}

Webhook Data Processing (POST)

Request Payload:

{
  entry: [
    {
      changes: [
        {
          value: {
            leadgen_id: '123456789', // Facebook lead ID
            form_id: '987654321', // Facebook form ID
            page_id: '555555555', // Facebook page ID
          },
        },
      ],
    },
  ];
}

Processing Flow:

1. Validate Structure: Check for required fields (leadgen_id, form_id, page_id)
2. Duplicate Check: Query leads-data for existing lead_data.id
3. Campaign Lookup: Match campaign by page_id and lead_form_id
4. Account Status: Verify account is active using checkAccountStatus
5. Fetch Lead Data: Call Facebook API to get full lead details
6. Multiple Campaigns: Process ALL matching campaigns (not just first)

Facebook API Lead Data:

// Retrieved from Facebook Graph API
{
  id: "123456789",
  created_time: "2025-01-15T10:30:00+0000",
  ad_id: "9876543210",
  form_id: "987654321",
  field_data: [
    {
      name: "full_name",
      values: ["John Doe"]
    },
    {
      name: "email",
      values: ["john@example.com"]
    },
    {
      name: "phone_number",
      values: ["4155551234"]
    },
    {
      name: "city",
      values: ["San Francisco"]
    }
  ]
}

Key Features:

• Access Token Management: Automatically refreshes expired tokens via checkAccessTokenValidity
• Multi-Campaign Support: Processes same lead for multiple campaigns
• Account Status Check: Skips inactive accounts without breaking loop
• Comprehensive Logging: Detailed logging at every step for debugging

Environment Variables:

• FACEBOOK_VERIFY_TOKEN - Webhook verification token
• FACEBOOK_API_VERSION - Graph API version (e.g., "v18.0")

---

2. ClickFunnels - clickfunnels

Purpose: Handles form submissions from ClickFunnels landing pages.

HTTP Method: POST

Authentication: None (public webhook)

Endpoint Pattern: /v1/inbound/webhooks/clickfunnels/:accountid/:campaignid

Request Payload:

{
  data: {
    attributes: {
      id: "12345",              // ClickFunnels submission ID
      full_name: "John Doe",
      email: "john@example.com",
      phone: "4155551234",
      // ... custom fields from form
    }
  }
}

Key Features:

• URL Parameters: Account and campaign IDs in URL path
• Nested Data: Lead data in data.attributes object
• No Duplicate Check: Uses lead_id for deduplication

---

3. ManyChat - manychat

Purpose: Handles lead captures from ManyChat chatbot conversations.

HTTP Method: POST

Authentication: None (public webhook)

Endpoint Pattern: /v1/inbound/webhooks/manychat/:accountid/:campaignid

Request Payload:

{
  id: "chat_12345",
  custom_fields: {
    email: "john@example.com",
    phone: "4155551234",
    first_name: "John",
    last_name: "Doe",
    // ... other custom fields
  }
}

Special Processing:

• Email Detection: Regex search across all custom fields if email not found
• Phone Detection: Regex search across all custom fields if phone not found
• Flexible Field Mapping: Checks both custom_fields and root level data

Regex Patterns:

// Email regex
const emailRegex =
  /^(([^<>()[\]\\.,;:\s@"]+(\.[^<>()[\]\\.,;:\s@"]+)*)|(".+"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;

// Phone regex
const phoneRegex =
  /^\s*(?:\+?(\d{1,3}))?[-. (]*(\d{3})[-. )]*(\d{3})[-. ]*(\d{4})(?: *x(\d+))?\s*$/;

---

4. Unbounce - unbounce

Purpose: Handles form submissions from Unbounce landing pages.

HTTP Method: POST

Authentication: None (public webhook)

Endpoint Pattern: /v1/inbound/webhooks/unbounce/:accountid/:campaignid

Request Payload:

{
  "data.json": "[{...}]",     // JSON string with lead data
  "data.xml": "<xml>...</xml>" // XML format (not used)
}

Special Processing:

• JSON Parsing: Parses data.json string to object
• Array Conversion: Converts fieldName: ["value"] to fieldName: "value"
• Data Cleanup: Removes empty array values

Processing Steps:

// 1. Parse JSON string
unbounceDataDecoded = JSON.parse(unbounceData['data-json']);

// 2. Convert arrays to single values
for (const property in unbounceDataDecoded) {
  if (unbounceDataDecoded[property][0]) {
    unbounceDataDecoded[property] = unbounceDataDecoded[property][0];
  } else {
    delete unbounceDataDecoded[property];
  }
}

---

5. Instapage - instapage

Purpose: Handles form submissions from Instapage landing pages.

HTTP Method: POST

Authentication: None (public webhook)

Endpoint Pattern: /v1/inbound/webhooks/instapage/:accountid/:campaignid

Request Payload:

{
  id: "submission_12345",
  email: "john@example.com",
  phone: "4155551234",
  first_name: "John",
  last_name: "Doe",
  // ... custom form fields
}

Features: Similar to other form webhooks with standard field extraction.

---

6. Kartra - kartra

Purpose: Handles form submissions from Kartra marketing platform.

HTTP Method: POST

Authentication: None (public webhook)

Endpoint Pattern: /v1/inbound/webhooks/kartra/:accountid/:campaignid

Request Payload:

{
  action: "filled_form",      // Event type
  action_details: {
    form: {
      form_id: "12345"        // Kartra form ID
    }
  },
  lead: {
    id: "lead_12345",
    email: "john@example.com",
    first_name: "John",
    last_name: "Doe",
    phone: "4155551234",
    phone_country_code: "+1"  // May include country code
  }
}

Special Processing:

• Action Validation: Only processes action: "filled_form" events
• Form ID Check: Verifies form_id exists before processing
• Phone Country Code: Uses phone_country_code if provided

---

7. PhoneSites - phonesites

Purpose: Handles form submissions from PhoneSites landing pages.

HTTP Method: POST

Authentication: None (public webhook)

Endpoint Pattern: /v1/inbound/webhooks/phonesites/:accountid/:campaignid

Request Payload:

{
  lead_id: "ps_12345",
  page: "Landing Page Name",   // Required field
  email: "john@example.com",
  phone: "4155551234",
  // ... custom fields
}

Validation: Requires page field to be present.

---

8. CallRail - callrail

Purpose: Handles incoming phone call webhooks from CallRail.

HTTP Method: POST

Authentication: None (public webhook)

Endpoint Pattern: /v1/inbound/webhooks/callrail/:accountid/:campaignid

Request Payload:

{
  resource_id: "call_12345",           // CallRail call ID
  trackingnum: "+14155559999",         // Tracking number called
  formatted_customer_name: "John Doe", // Caller ID name
  customer_phone_number: "+14155551234" // Caller's phone
}

Contact Creation:

• Name: Uses formatted_customer_name
• Phone: Uses customer_phone_number
• Email: Not available from CallRail

Use Case: Phone call tracking and lead attribution.

---

9. Call Tracking Metrics - calltrackingmetrics

Purpose: Handles phone call webhooks from Call Tracking Metrics.

HTTP Method: POST

Authentication: None (public webhook)

Endpoint Pattern: /v1/inbound/webhooks/calltrackingmetrics/:accountid/:campaignid

Request Payload:

{
  id: "call_12345",
  tracking_number: "+14155559999",  // Number called
  contact_number: "+14155551234",   // Caller's number
  cnam: "John Doe"                  // Caller ID name
}

Campaign Matching: Matches campaign by tracking_number in addition to campaign ID.

Special Handling:

• JSON Stringify: Stores lead_data as JSON string (not object)
• Tracking Number Match: Validates tracking number matches campaign

---

🔧 Common Processing Functions

Field Extraction Functions

getName(leadData, contactType)

Extracts name from lead data with intelligent fallback.

Priority Order:

// For Business
1. businessName, companyName, Company Name*
2. name, Name, full_name, Full Name
3. firstName + lastName

// For Person
1. name, full_name, Full Name
2. firstName + lastName
3. first_name + last_name

Returns: String or null

---

getBusinessName(leadData)

Extracts business name from lead data.

Field Priority:

[
  'Company Name*', // Highest priority
  'Business Name*',
  'company_name',
  'companyName',
  'business_name',
  'businessName',
  'company',
  'business',
  // ... 15+ variations
];

---

getEmail(leadData)

Extracts email from lead data.

Field Priority:

[
  'Business Email*', // Highest priority
  'Email*',
  'email',
  'Email',
  'email_address',
  'emailAddress',
  'business_email',
  'Work Email',
  // ... 20+ variations
];

Returns: Valid email string or null

---

getPhone(leadData)

Extracts phone number from lead data.

Field Priority:

[
  'Phone*', // Highest priority
  'Mobile*',
  'mobilePhone.value', // Nested value support
  'phone',
  'Phone',
  'business_number',
  'mobile',
  'cell',
  'telephone',
  // ... 30+ variations
];

Returns: Phone string or null

---

getCountry(leadData)

Extracts country from lead data.

Field Priority:

['Country*', 'Shipping Country', 'country', 'Country', 'country_name', 'shipping_country'];

---

getAddressFields(leadData, contactType)

Extracts complete address object from lead data.

Returns:

{
  street: String,
  unit: String,
  city: String,
  state_province: String,
  postal_code: String,
  country: String
}

Processing Logic:

1. If contactType === 'business': Check business fields first, then personal
2. If contactType === 'person': Check personal fields first, then business
3. Remove undefined fields
4. Return null if no fields found

Business Address Fields:

• Business Street*, shipping_street, address
• Business City*, shipping_city
• Business State*, shipping_state
• Business ZIP*, shipping_zip

Personal Address Fields:

• Street*, street, Street Address
• Unit*, unit, Apartment Number
• City*, city
• State/Province*, state
• Postal Code*, postal_code, zip

---

getCountryCodes(countryName)

Converts country name to country calling code.

Returns:

{
  country: "United States",
  code: "US",
  countryCodes: ["+1"]
}

Matching Strategy:

1. Exact Match: Case-insensitive exact match on country name or code
2. Word Boundary Match: Safe regex with escaped special characters
3. Returns: Country object or null

---

Phone Number Formatting

Pattern:

// If phone doesn't start with '+' and country is provided
if (contact.phone && !contact.phone.match(/\+[0-9,-]+$/gi) && codes) {
  const match = await getCountryCodes(country);
  if (match?.countryCodes?.[0]) {
    contact.phone = '+' + match.countryCodes[0] + contact.phone;
  }
}

Examples:

// Input: phone="4155551234", country="United States"
// Output: "+14155551234"

// Input: phone="+14155551234", country="United States"
// Output: "+14155551234" (no change)

---

Contact Creation Functions

createContact({ contacts, campaignData, returnFullObject })

Universal contact creation function used by all webhooks.

Purpose: Creates CRM contact with duplicate detection and business/person handling.

Parameters:

{
  contacts: [                    // Array of contact objects
    {
      name: String,              // OR first_name + last_name
      email: String,
      phone: String,
      address: Object,
      source: "inbound"          // Auto-set if missing
    }
  ],
  campaignData: Object,          // Full campaign document
  returnFullObject: Boolean      // If true, returns full contact doc
}

Returns:

• If returnFullObject === false: Returns contact_id (ObjectId)
• If returnFullObject === true: Returns full contact document (Object)

Processing Flow:

flowchart TD
    A[createContact Called] --> B[Clean Contact Object]
    B --> C[Format Phone - remove spaces/parentheses]
    C --> D[Build OR Query - email or phone]

    D --> E{Find Existing Contact}
    E -->|Found| F[Update Existing Contact]
    E -->|Not Found| G{Contact Type?}

    G -->|Business| H[createBusinessContact]
    G -->|Person| I[createPersonContact]

    H --> J{Already Exists?}
    I --> J

    J -->|Yes| K[Fetch Existing Contact]
    J -->|No| L[Return New Contact]

    F --> M[Return Updated Contact]
    K --> M
    L --> M
    M --> N[Return contact_id or full object]

Duplicate Handling:

1. Search Criteria: Matches on email OR phone within same account
2. Type Match: Only matches contacts of same type (person/business)
3. Update Strategy: Updates existing contact with new data (preserves email/phone)
4. Error Handling: If CRM returns CONTACT_ALREADY_EXIST, fetches existing contact

Business vs Person:

• Determined by campaignData.add_person_or_business
• For businesses: Creates only business contact (not associated person)
• For persons: Creates only person contact

---

Representative Assignment

findassignee(campaignData)

Determines which representative to assign lead to.

Purpose: Round-robin lead distribution among selected representatives.

Algorithm:

// 1. Get selected_user array from campaign
const selectedUser = campaignData.selected_user;

// 2. Find last lead for this campaign
const lastLead = await leadsDataModel.findOne({ campaign_id });

// 3. Build array excluding last assigned rep (if multiple reps)
let userArray = [];
if (selectedUser.length > 1) {
  // Exclude last assigned rep for better distribution
  userArray = selectedUser.filter(u => u.id !== lastLead.user_id);
} else {
  // Only one rep, assign to them
  userArray = [selectedUser[0].id];
}

// 4. Random selection from remaining reps
const randomIndex = Math.floor(Math.random() * userArray.length);
return userArray[randomIndex];

Distribution Logic:

• Single Rep: Always assigns to that rep
• Multiple Reps: Pseudo-round-robin (excludes last assigned)
• Improves Distribution: Prevents same rep getting consecutive leads

---

Notification Functions

sendNotificationToRep(accountId, repID, leadId)

Sends push and email notification to assigned representative.

Notifications Sent

1. FCM Push Notification

{
  title: "A new Inbound lead has been assigned to you.",
  body: "A new Inbound lead has been assigned to you.",
  data: { lead_id: leadId },
  module: "inbound",
  type: "lead_assigned"
}

2. Email Notification

{
  subject: "A new Inbound lead has been assigned to you.",
  content: [{
    value: "A new Inbound lead has been assigned to you."
  }],
  origin: "lead_assigned"
}

Error Handling: Notifications sent asynchronously - failures don't block webhook processing.

---

sendMailToRep(mailData, accountId, userId)

Sends custom email to representative.

Parameters:

{
  repId: ObjectId,              // Representative to email
  subject: String,              // Email subject
  content: String,              // Email HTML body
  from: String,                 // From email address
  attachments: Array,           // File attachments
  reply_to: Object              // Reply-to address
}

Attachment Processing:

• Fetches file buffers from URLs via getFileBufferData
• Converts to format required by Mail service

Uses: Configured in campaign email_details for custom notifications.

---

sendSMS(accountId, userId, smsData)

Sends SMS to representative.

Parameters:

{
  repId: ObjectId,              // Representative to SMS
  content: String,              // SMS message text
  from: String,                 // Twilio phone number
  custom_number: String         // Custom phone (if set in campaign)
}

Phone Number Logic:

• If custom_number set in campaign selected_user: Use that
• Else: Fetch user's phone from User collection

Uses: Configured in campaign sms_details for SMS notifications.

---

Deal Creation

Automatic Deal Creation (all webhooks):

if (campaignData.pipeline_id && campaignData.stage_id && campaignData.create_deal) {
  const dealInfo = {
    pipeline_id: campaignData.pipeline_id,
    stage_id: campaignData.stage_id,
    name: generateDealName(contact),
    source: 'inbound',
    person: contact_id, // or business: contact_id
    value: campaignData.deal_value || 0,
    currency: campaignData.currency || 'USD',
  };

  await deals(pipeline_id, account_id, owner, dealInfo);
}

Deal Name Generation:

// Priority order
if (contact.name) dealName = contact.name;
else if (contact.first_name || contact.last_name) dealName = `${first_name} ${last_name}`;
else if (contact.email) dealName = contact.email.match(/^.+(?=@)/)[0];
else dealName = campaignData.campaign_name + ' lead';

---

Support Conversation Integration

processSupportConversation({ contact_id, campaignData, userResponse, contactObject })

Creates or updates support conversation for lead.

Purpose: Every webhook lead creates a support conversation thread for follow-up.

Process Flow:

sequenceDiagram
    participant Webhook
    participant SupportConv
    participant SupportRoom
    participant SupportMessage
    participant Socket

    Webhook->>SupportConv: Find or create conversation
    SupportConv->>SupportRoom: Find existing room

    alt Room Exists
        SupportRoom->>Socket: Send message to existing room
        Socket-->>UI: Update conversation
    else No Room
        SupportRoom->>SupportRoom: Create new room
        SupportRoom->>SupportSession: Create session with IP
        SupportSession->>SupportMessage: Create system message
        SupportMessage->>SupportMessage: Create lead message
        SupportMessage->>Socket: Emit joinRoom event
        Socket-->>UI: Show new conversation
    end

Support Conversation:

• One per contact across all campaigns
• Upsert Pattern: Creates if not exists, reuses if exists

Support Room:

• One per conversation/session
• Type: room_type: "inbound"
• Verified: Initially false

Messages Created:

1. System Message: ::{USER} started a new conversation on ::{TIME_STAMP}
2. Lead Message: Lead received for {contact_name}

Metadata Stored:

{
  contact_id: ObjectId,
  origin: "inbound",
  inbound_integration: String, // e.g., "facebook_ads"
  webhook_response: Object // Full webhook payload
}

---

Retry Mechanism

checkLogCount(uniqueId, type)

Implements 3-retry logic for failed webhooks.

Purpose: Allows Facebook/external platforms to retry failed webhooks up to 3 times.

Database: logs collection

Logic:

// 1. Find existing log for this lead
const log = await Logs.findOne({ unique_id: leadId, type: 'inbound' });

if (log) {
  if (log.count >= 3) {
    // Max retries reached
    await Logs.deleteOne({ unique_id: leadId });
    return false; // Stop retrying (return 200 to webhook)
  } else {
    // Increment count
    await Logs.updateOne({ unique_id: leadId }, { $inc: { count: 1 } });
    return true; // Allow retry (return 400 to webhook)
  }
} else {
  // First attempt - create log
  await new Logs({ type: 'inbound', unique_id: leadId, count: 1 }).save();
  return true; // Allow retry (return 400 to webhook)
}

HTTP Response Strategy:

try {
  // ... webhook processing
} catch (error) {
  const shouldRetry = await this.checkLogCount(leadId, 'inbound');

  if (shouldRetry) {
    return res.status(400).json({ success: false }); // Triggers retry
  } else {
    return res.status(200).json({ success: true }); // Stops retries
  }
}

Use Case: Transient errors (database connection, API timeouts) can be retried.

---

📊 Data Models

Lead Data (Extended)

{
  _id: ObjectId,
  lead_id: String,                    // External ID from platform
  lead_data: Object,                  // Raw webhook payload
  campaign_id: ObjectId,
  account_id: ObjectId,
  owner: ObjectId,
  user_id: ObjectId,                  // Assigned representative
  contact_id: ObjectId,               // Created CRM contact

  // Error Tracking
  errors: [
    {
      type: String,                   // Error type
      details: Object                 // Error details
    }
  ],

  // InstaSites/InstaReport
  instasite: ObjectId,                // OPTIONAL - Generated InstaSite
  instareport: ObjectId,              // OPTIONAL - Generated InstaReport
  build_failed: {                     // OPTIONAL - Build failure details
    message: String,
    error: Object,
    timestamp: Date
  },

  // Metadata
  freshness: String,                  // "new" | "read" | "contacted"
  is_deleted: Boolean,
  created_at: Date,
  updated_at: Date
}

---

Logs (Retry Tracking)

{
  _id: ObjectId,
  type: String,                      // "inbound"
  unique_id: String,                 // lead_id from webhook
  count: Number,                     // Retry attempt count (max 3)
  created_at: Date
}

---

🛡️ Error Handling

Error Categorization

All errors stored in lead's errors array:

{
  type: String,                      // Error category
  details: {
    errorJson: String,               // Stringified error
    stack: String,                   // Stack trace
    config: Object                   // Request config (if applicable)
  }
}

Error Types:

• contactCreation - Failed to create CRM contact
• assigneeCreation - Failed to find/assign representative
• emailSendError - Failed to send email notification
• SmsSendError - Failed to send SMS notification
• dealCreateError - Failed to create deal in pipeline

Common Error Scenarios

Scenario	Handling	User Impact
Invalid webhook payload	Return 200, no lead created	None - ignored
Duplicate lead	Return 200, skip processing	None - prevented duplicate
Campaign not found	Return 200, skip processing	None - webhook misconfigured
Inactive campaign	Return 200, skip processing	None - campaign disabled
Inactive account	Skip to next campaign	None - account suspended
Contact creation fails	Store error, continue processing	Lead saved, contact can be created manually
Notification fails	Store error, continue processing	Lead saved, notification can be resent
Deal creation fails	Store error, continue processing	Lead saved, deal can be created manually
Database error (retry <3)	Return 400, allow retry	Platform retries webhook
Database error (retry ≥3)	Return 200, stop retry	Lead may be lost, check logs

---

🔀 Integration Points

Internal Services

• CRM Contact Service:

  • addPeople() - Creates person contacts
  • addBusinesses() - Creates business contacts
  • Duplicate detection by email/phone

• Deals Service:

  • deals() - Creates deals in pipelines
  • Associates with person or business contact

• Mail Service (link removed - file does not exist):

  • Sends email notifications to representatives
  • Supports attachments and reply-to

• SMS Service (link removed - file does not exist):

  • Sends SMS notifications to representatives
  • Custom number override support

• Socket Service (link removed - file does not exist):

  • socketEmit() - Real-time lead notifications
  • Event: 'leadadded' to owner

• Support Conversations:

  • processSupportConversation() - Creates support tickets
  • Links conversation to contact

External Services

• Facebook Graph API:

  • Fetches lead data by leadgen_id
  • Requires valid access_token
  • Automatic token refresh on expiration

• Webhook Providers:

  • Facebook Lead Ads
  • ClickFunnels
  • ManyChat
  • Unbounce
  • Instapage
  • Kartra
  • PhoneSites
  • CallRail
  • Call Tracking Metrics

Utilities

• Country Codes (link removed - file does not exist):

  • Phone number formatting
  • Country code lookup by name

• Field Extractors (link removed - file does not exist):

  • getEmail() - Extract email from varied formats
  • getPhone() - Extract phone with formatting
  • getName() - Extract name (person vs business)
  • getCountry() - Extract country origin
  • getAddressFields() - Extract address components

---

🧪 Edge Cases & Special Handling

Duplicate Lead Detection

Facebook Specific:

// Uses Facebook's leadgen_id
let condition = { 'lead_data.id': leadId };

Other Platforms:

// Uses lead_id from payload or generated UUID
let condition = { lead_id: leadId };

Multiple Campaigns from Same Form

Facebook Lead Ads:

// Multiple campaigns can use same page_id + form_id
for (let campInfo of allCampaignData) {
  // Process each campaign independently
  // Lead saved multiple times (once per campaign)
}

Account Status Mid-Campaign

Account Deactivated:

const accountStatusCheck = await utilities.checkAccountStatus({
  accountId: campaignData.account_id
});

if (!accountStatusCheck.status) {
  continue; // Skip this campaign, continue loop
}

Token Expiration During Webhook

Facebook Token Refresh:

let accessToken = await keysModel.checkAccessTokenValidity({
  req: req,
  expirationTime: existKeys.token.expires_in,
  accessToken: existKeys.token.access_token,
  docId: existKeys.docId,
});
// Automatically refreshes if expired

Field Extraction Variations

Email Field Names:

• 'email', 'Email', 'EMAIL'
• 'email_address', 'emailAddress'
• 'user_email', 'contact_email'
• Regex fallback for unrecognized fields

Phone Field Names:

• 'phone', 'Phone', 'PHONE'
• 'phone_number', 'phoneNumber'
• 'mobile', 'cell', 'telephone'
• Regex fallback for patterns

Name Handling:

// Person: first_name + last_name
// Business: name or company_name
// Fallback: full_name split into first/last

Notification Preferences

Notification Types:

• 'only_send_email' - Email only
• 'only_send_sms' - SMS only
• 'send_email_sms' - Both email and SMS
• undefined - No notifications

Deal Creation Edge Cases

Missing Deal Configuration:

if (campaignData.pipeline_id && campaignData.stage_id && campaignData.create_deal) {
  // Only create deal if all three conditions met
}

Deal Name Generation:

// Priority:
// 1. Contact name (if exists)
// 2. Email local part (before @)
// 3. Campaign name + " lead"

Error Accumulation

Non-Blocking Errors:

let errors = [];

try {
  // Contact creation
} catch (error) {
  errors.push({ type: 'contactCreation', details: errorJson });
}

try {
  // Deal creation
} catch (error) {
  errors.push({ type: 'dealCreateError', details: errorJson });
}

// Lead still saved with errors array
await leadsDataModel.update({ _id: leadId }, { errors: errors });

Retry Logic

Max 3 Attempts:

let resLog = await this.checkLogCount(leadId, 'inbound');

if (resLog) {
  // Attempts 1-2: Return 400 (webhook provider will retry)
  return res.status(400).json({ success: false });
} else {
  // Attempt 3+: Return 200 (stop retrying)
  return res.status(200).json({ success: true });
}

---

⚠️ Important Notes

1. Facebook Verification: GET requests must return challenge string for webhook verification. Use FACEBOOK_VERIFY_TOKEN environment variable to validate verify_token parameter.

2. Duplicate Prevention: Check for duplicate leads before processing. Use platform-specific lead identifiers (Facebook's leadgen_id, ClickFunnels' id, etc.). Return 200 success for duplicates to prevent retries.

3. Multiple Campaigns: Facebook webhooks can trigger multiple campaigns if same form used across campaigns. Each campaign processes lead independently. Lead saved multiple times with different campaign associations.

4. Token Management: Facebook access tokens expire after 60 days. Automatically refresh tokens using checkAccessTokenValidity before API calls. Store absolute expiration timestamps.

5. Error Tolerance: All errors logged in lead document but don't prevent lead capture. Contact creation, deal creation, and notifications can fail without losing lead data. Always save lead first.

6. Notification Routing: Check notification_type before sending. Support email-only, SMS-only, or both. Use custom_number override for SMS if configured per representative.

7. Retry Logic: Return 400 for first 2 attempts (webhook provider retries). Return 200 after 3 attempts to stop infinite retry loops. Track attempts in logs collection.

8. Field Extraction: Use helper functions (getEmail, getPhone, getName) to handle varied field naming. Implement regex fallback for unrecognized field names. Case-insensitive matching.

9. Phone Formatting: Always format phone numbers with country codes. Use phone_country_code if provided by platform. Fall back to country name lookup. Validate regex pattern before adding country code.

10. Account Status: Check account status before processing campaign. Skip inactive accounts within campaign loop. Prevents processing leads for suspended/cancelled accounts.

11. Deal Automation: Only create deals if pipeline_id, stage_id, and create_deal all configured. Use campaign's deal_value and currency. Associate deal with person or business based on add_person_or_business.

12. Support Conversations: Auto-create support tickets from leads for customer service tracking. Links conversation to contact. Includes lead data as initial message.

---

📈 Performance Considerations

Optimization Strategies

1. Parallel Campaign Processing:

   • Facebook webhooks process ALL matching campaigns
   • Non-blocking - one campaign failure doesn't block others

2. Asynchronous Notifications:

   • FCM/Email/SMS sent without await
   • Failures logged but don't block webhook response

3. Database Indexing:

// Recommended indexes
{ "lead_data.id": 1, account_id: 1 }  // Duplicate detection
{ campaign_id: 1, is_deleted: 1 }      // Campaign lookup
{ account_id: 1, page_id: 1, lead_form_id: 1 }  // Facebook campaign match
{ tracking_number: 1, is_deleted: 1 }  // CallRail/CTM lookup

4. Webhook Response Time:

   • Target: less than 2 seconds for Facebook webhooks
   • Actual: ~500ms-1s for simple leads
   • Complex: 2-4s with deal creation and notifications

5. Rate Limiting:

   • External platforms may send bursts of webhooks
   • Consider queue system for high-volume accounts

---

🔒 Security Considerations

Webhook Verification

1. Facebook: Uses hub.verify_token for webhook setup
2. Other Platforms: No verification (public endpoints)
3. Account/Campaign Validation: Ensures IDs in URL are valid

Data Sanitization

1. Phone Number Cleaning:

contact.phone = contact.phone.replace(/[() ]/g, ''); // Remove formatting

2. Empty Field Removal:

// cleanContactObject removes null/undefined/empty strings
const cleanContactObject = obj => {
  // ... removes invalid values recursively
};

3. SQL/NoSQL Injection Prevention:
   • All IDs converted to mongoose.Types.ObjectId
   • No raw query string concatenation

Multi-Tenant Isolation

• All webhook endpoints require accountid in URL
• All queries filtered by account_id
• Campaign validation ensures ownership

---

🧪 Testing Recommendations

Unit Tests

1. Field Extraction Functions:

   • Test all field name variations
   • Test nested value extraction
   • Test null/undefined handling

2. Phone Formatting:

   • Test country code detection
   • Test existing + prefix handling
   • Test invalid country names

3. Contact Creation:

   • Test duplicate detection
   • Test business vs person logic
   • Test field cleaning

4. Retry Logic:

   • Test first attempt
   • Test retry count increment
   • Test max retry termination

Integration Tests

1. Facebook Webhook Flow:

   • Mock Facebook API responses
   • Test token refresh
   • Test multi-campaign processing

2. All Webhook Platforms:

   • Send real webhook payloads
   • Verify lead creation
   • Verify contact creation
   • Verify notifications sent
   • Verify deals created

3. Error Scenarios:

   • Test duplicate leads
   • Test inactive campaigns
   • Test database failures
   • Test notification failures

Performance Tests

• Load Testing: 100 concurrent webhooks
• Token Refresh: Under load conditions
• Database Performance: With 1M+ leads
• Notification Throughput: Email/SMS limits

---

📊 Monitoring & Debugging

Key Metrics to Track

1. Webhook Success Rate:

   • Total webhooks received
   • Successfully processed
   • Failed with errors

2. Processing Time:

   • Average webhook response time
   • 95th percentile
   • Max time

3. Error Rates by Type:

   • Contact creation failures
   • Notification failures
   • Deal creation failures

4. Integration Health:

   • Facebook token refresh frequency
   • External API failures
   • Retry counts

Logging Strategy

All webhooks use extensive logging:

logger.log({
  req,
  initiator: 'internal/inbound/receiveWebhooks',
  message: 'Webhook received',
  leadId,
  campaignId,
  integration: 'facebook_ads',
});

Log Levels:

• logger.log() - Info (normal flow)
• logger.warn() - Warnings (non-critical issues)
• logger.error() - Errors (failures)

Key Log Points:

• Webhook received
• Campaign matched
• Contact created
• Lead saved
• Notifications sent
• Errors occurred

---

🚀 Best Practices

For Platform Administrators

1. Webhook URLs:

   • Use HTTPS only
   • Include accountid and campaignid in path
   • Don't change URLs after configuration

2. Campaign Configuration:

   • Always test webhooks with test leads
   • Verify page_id/form_id match exactly
   • Enable notifications for new campaigns

3. Error Monitoring:

   • Review lead errors arrays regularly
   • Set up alerts for high error rates
   • Investigate notification failures

4. Contact Management:

   • Decide person vs business at campaign level
   • Use consistent phone formatting
   • Verify country code detection

For Developers

1. Adding New Webhook Platform:

   • Follow existing webhook pattern
   • Implement duplicate detection
   • Use common helper functions (getName, getEmail, etc.)
   • Test with real webhook payloads

2. Modifying Existing Webhooks:

   • Preserve backward compatibility
   • Test all existing campaigns
   • Log changes for debugging

3. Performance Optimization:

   • Minimize database queries
   • Use indexes effectively
   • Consider async processing for heavy operations

---

🔗 Related Documentation

• Leads Service - Lead management and retrieval
• Campaigns Service - Campaign configuration
• CRM Contacts - Contact management
• Deals Service - Deal creation
• Mail Service (link removed - file does not exist) - Email notifications
• SMS Service (link removed - file does not exist) - SMS notifications
• Support Conversations - Support ticket creation
• Country Codes Utility (link removed - file does not exist) - Phone formatting
• Socket Utility (link removed - file does not exist) - Real-time updates