Duda Webhook Service

📖 Overview

The Duda Webhook service handles incoming webhook events from Duda when sites are published or unpublished. It orchestrates complex multi-step processes including database updates, thumbnail deletion from cloud storage, thumbnail regeneration via Google PubSub, and user notifications via Firebase Cloud Messaging (FCM).

Source File: external/Integrations/Duda/services/webhook.service.js
External API: Duda Sites API (for fetching updated site data)
Primary Use: Process Duda webhook events and trigger downstream operations

🗄️ Collections Used

`agency-website`

Operations: Read, Update
Model: external/models/agency-website.js
Usage Context: Find and update sites created via agency website builder

`instasite`

Operations: Read, Update
Model: external/models/instasite.js
Usage Context: Find and update sites created via InstaSites feature

`config`

Operations: Read
Model: external/models/config.js
Usage Context: Fetch Duda API credentials for authentication

`user-config`

Operations: Read
Model: external/models/user-config.js
Usage Context: Check user notification preferences for FCM delivery

🔄 Data Flow

Publish Event Flow

sequenceDiagram
    participant Duda as Duda API
    participant Webhook as Webhook Service
    participant DB as MongoDB
    participant Config as Config Collection
    participant DudaAPI as Duda Sites API
    participant Wasabi as Wasabi Storage
    participant PubSub as Google PubSub

    Duda->>Webhook: POST site.published event
    Note over Webhook: data: { event_type, site_name }<br/>resource_data: { site_name }

    Webhook->>DB: Find site by builder_id
    Note over Webhook,DB: Check agency-website OR instasite

    DB-->>Webhook: Return site document

    Webhook->>Config: Fetch API credentials
    Config-->>Webhook: Return credentials

    par Parallel API Calls
        Webhook->>DudaAPI: GET /sites/:id
        Webhook->>DudaAPI: GET /sites/:id/content
    end

    DudaAPI-->>Webhook: Site details + content

    Webhook->>DB: Update site with new data
    Note over Webhook,DB: thumbnails, business_info,<br/>SEO, content, status, etc.

    Webhook->>Wasabi: Delete old thumbnails
    Note over Webhook,Wasabi: desktop.png, mobile.png, tablet.png

    Webhook->>PubSub: Publish to v2.sites-thumbnail-generator
    Note over Webhook,PubSub: Trigger regeneration

    Webhook-->>Duda: 200 OK (event processed)

    style Webhook fill:#e3f2fd
    style PubSub fill:#e8f5e9
    style Wasabi fill:#fff3e0

Unpublish Event Flow

sequenceDiagram
    participant Duda as Duda API
    participant Webhook as Webhook Service
    participant DB as MongoDB
    participant Config as Config Collection
    participant DudaAPI as Duda Sites API
    participant UserConfig as UserConfig Collection
    participant FCM as Firebase Cloud Messaging

    Duda->>Webhook: POST site.unpublished event

    Webhook->>DB: Find site by builder_id
    DB-->>Webhook: Return site document

    Webhook->>Config: Fetch API credentials
    Config-->>Webhook: Return credentials

    Webhook->>DudaAPI: GET /sites/:id
    DudaAPI-->>Webhook: Updated site details

    alt Site status is UNPUBLISHED
        Webhook->>DB: Update site status to UNPUBLISHED

        Webhook->>UserConfig: Get user notification preferences
        UserConfig-->>Webhook: Return preferences

        alt Notifications enabled (browser OR bell)
            Webhook->>FCM: Send push notification
            Note over Webhook,FCM: "Site unpublished successfully"
        end
    end

    Webhook-->>Duda: 200 OK (event processed)

    style Webhook fill:#e3f2fd
    style FCM fill:#ffebee

🔧 Business Logic & Functions

Helper: findBySiteName(id)

Purpose: Find a site in either agency-website or instasite collection by builder_id

Source: services/webhook.service.js

Parameters:

id (String) - Duda site_name (builder_id)

Returns: Promise<Object | null>

{
  _id: ObjectId,
  builder_id: String,
  // ... other site fields
}

Business Logic Flow:

Check Agency Website Collection

const agencyWebsite = await AgencyWebsite.findOne({ builder_id: id });
if (agencyWebsite) {
  return { ...agencyWebsite._doc, model: 'agency-website' };
}

Check Instasite Collection

const instasite = await Instasite.findOne({ builder_id: id });
if (instasite) {
  return { ...instasite._doc, model: 'instasite' };
}

Not Found
```
return null;
```

Key Business Rules:

Multi-Collection Search: Sites can exist in either agency-website or instasite
Model Tagging: Adds model property to identify source collection
Returns Null: If site not found in either collection

Helper: update(id, data)

Purpose: Update a site document in the appropriate collection (agency-website or instasite)

Source: services/webhook.service.js

Parameters:

id (String) - Duda site_name (builder_id)
data (Object) - Update data

Returns: Promise<void>

Business Logic Flow:

Determine Collection

const site = await findBySiteName(id);
if (!site) {
  throw new Error('Site not found');
}

Update Appropriate Model

if (site.model === 'agency-website') {
  await AgencyWebsite.updateOne({ builder_id: id }, { $set: data });
} else if (site.model === 'instasite') {
  await Instasite.updateOne({ builder_id: id }, { $set: data });
}

Key Business Rules:

Dynamic Model Selection: Updates correct collection based on site.model
Uses $set: Partial updates, only modifies provided fields

publish(data, resource_data)

Purpose: Handle Duda site.published webhook event - update database, delete old thumbnails, trigger regeneration

Source: services/webhook.service.js

External API Endpoint: GET https://api.duda.co/api/sites/multiscreen/:site_name

Parameters:

data (Object) - Webhook payload
- event_type (String) - "site.published"
- site_name (String) - Duda site ID
resource_data (Object) - Resource info
- site_name (String) - Duda site ID

Returns: Promise<void>

Business Logic Flow:

Find Site in Database

let site = await findBySiteName(resource_data.site_name);
if (!site) {
  console.log('Site not found for publish event:', resource_data.site_name);
  return;
}

Fetch Duda API Credentials

const query = await Config.findOne();
if (!query) {
  throw notFound('API token not found');
}
const access_token = utils.base64(query);

Fetch Updated Site Data (Parallel)

const [siteDetails, siteContent] = await Promise.all([
  // Get site details
  axios({
    method: 'GET',
    url: `${DUDA_SITES_DOMAIN}/${resource_data.site_name}`,
    headers: {
      'Content-Type': 'application/json',
      authorization: `Basic ${access_token}`,
    },
  }),
  // Get site content
  axios({
    method: 'GET',
    url: `${DUDA_SITES_DOMAIN}/${resource_data.site_name}/content`,
    headers: {
      'Content-Type': 'application/json',
      authorization: `Basic ${access_token}`,
    },
  }),
]);

Prepare Update Data

let updateData = {
  thumbnails: siteDetails.data.thumbnails,
  business_info: siteContent.data.location_data,
  seo: {
    title: siteDetails.data.site_seo?.title,
    description: siteDetails.data.site_seo?.description,
  },
  content: siteContent.data,
  status: siteDetails.data.site_status,
  certificate_status: siteDetails.data.certificate_status,
  site_domain: siteDetails.data.site_domain,
  site_default_domain: siteDetails.data.site_default_domain,
  last_published_date: siteDetails.data.last_published_date,
  first_published_date: siteDetails.data.first_published_date,
};

Update Database

await update(resource_data.site_name, updateData);

Delete Old Thumbnails from Wasabi

if (site.thumbnails) {
  const thumbnailKeys = [site.thumbnails.desktop, site.thumbnails.mobile, site.thumbnails.tablet]
    .filter(Boolean)
    .map(url => url.split('.com/')[1]); // Extract S3 key

  if (thumbnailKeys.length > 0) {
    await deleteFilesFromWasabi(thumbnailKeys);
  }
}

Trigger Thumbnail Generation via PubSub

const topicName = 'v2.sites-thumbnail-generator';
const data = JSON.stringify({
  site_name: resource_data.site_name,
  model: site.model, // 'agency-website' or 'instasite'
});

const dataBuffer = Buffer.from(data);
const messageId = await pubSubClient.topic(topicName).publish(dataBuffer);

console.log(`Message ${messageId} published to ${topicName}`);

Webhook Payload Example:

// Received from Duda
{
  "event_type": "site.published",
  "site_name": "site_abc123",
  "timestamp": 1696867200
}

// resource_data parameter
{
  "site_name": "site_abc123"
}

Database Update Fields:

{
  thumbnails: {
    desktop: String,  // Old thumbnails deleted, new ones generated
    mobile: String,
    tablet: String
  },
  business_info: {
    // Full location_data from content API
    phones: Array,
    emails: Array,
    label: String,
    address: Object,
    social_accounts: Object
  },
  seo: {
    title: String,
    description: String
  },
  content: Object,  // Complete content library
  status: String,  // 'PUBLISHED'
  certificate_status: String,  // SSL certificate status
  site_domain: String,  // Custom domain if set
  site_default_domain: String,  // Duda subdomain
  last_published_date: Date,
  first_published_date: Date
}

Error Handling:

Site Not Found: Logs error and returns early (no throw)
Config Not Found: Throws 404 error
Duda API Error: Error propagates up (no catch)
Thumbnail Deletion Error: Logged but doesn't block process

Example Scenario:

// Duda sends webhook when user publishes site in editor
await webhookService.publish(
  {
    event_type: 'site.published',
    site_name: 'site_abc123',
  },
  {
    site_name: 'site_abc123',
  },
);

// Process:
// 1. Find site in DB
// 2. Fetch latest data from Duda API
// 3. Update DB with new SEO, content, status, domain info
// 4. Delete old thumbnails from Wasabi (desktop.png, mobile.png, tablet.png)
// 5. Publish message to PubSub topic 'v2.sites-thumbnail-generator'
// 6. Separate service generates new thumbnails

Key Business Rules:

Parallel API Calls: Fetches site details and content simultaneously for performance
Thumbnail Cleanup: Deletes old thumbnails before regeneration to save storage
Async Thumbnail Generation: PubSub message triggers separate service
No Blocking: Thumbnail generation doesn't block webhook response
Dual Collection Support: Works with both agency-website and instasite models
Complete Data Sync: Updates all site metadata, not just status

unpublish(resource_data)

Purpose: Handle Duda site.unpublished webhook event - update database and send user notification

Source: services/webhook.service.js

External API Endpoint: GET https://api.duda.co/api/sites/multiscreen/:site_name

Parameters:

resource_data (Object) - Resource info
- site_name (String) - Duda site ID

Returns: Promise<void>

Business Logic Flow:

Find Site in Database

let site = await findBySiteName(resource_data.site_name);
if (!site) {
  console.log('Site not found for unpublish event:', resource_data.site_name);
  return;
}

Fetch Duda API Credentials

const query = await Config.findOne();
if (!query) {
  throw notFound('API token not found');
}
const access_token = utils.base64(query);

Fetch Updated Site Details

const siteDetails = await axios({
  method: 'GET',
  url: `${DUDA_SITES_DOMAIN}/${resource_data.site_name}`,
  headers: {
    'Content-Type': 'application/json',
    authorization: `Basic ${access_token}`,
  },
});

Verify Site is Actually Unpublished

if (siteDetails.data.site_status !== 'UNPUBLISHED') {
  console.log('Site status is not UNPUBLISHED, skipping:', resource_data.site_name);
  return;
}

Update Database with New Status

await update(resource_data.site_name, {
  status: siteDetails.data.site_status,
});

Check User Notification Preferences

const userConfig = await UserConfig.findOne({ user_id: site.user_id });
if (!userConfig) {
  console.log('User config not found for notification');
  return;
}

const notificationsEnabled =
  userConfig.push_notifications?.browser || userConfig.push_notifications?.bell;

Send FCM Notification if Enabled

if (notificationsEnabled && userConfig.fcm_token) {
  try {
    await sendFCMNotification({
      token: userConfig.fcm_token,
      title: 'Site Unpublished',
      body: `Your site ${
        site.site_name || resource_data.site_name
      } has been unpublished successfully.`,
      data: {
        site_id: site._id.toString(),
        site_name: resource_data.site_name,
        event_type: 'site.unpublished',
      },
    });
  } catch (error) {
    logger.error('FCM notification failed:', error);
    // Don't throw - notification failure shouldn't block webhook
  }
}

Webhook Payload Example:

// Received from Duda
{
  "event_type": "site.unpublished",
  "site_name": "site_abc123",
  "timestamp": 1696867200
}

// resource_data parameter
{
  "site_name": "site_abc123"
}

User Notification Preferences Check:

// UserConfig document structure
{
  user_id: ObjectId,
  push_notifications: {
    browser: Boolean,  // Browser push notifications enabled
    bell: Boolean      // In-app bell notifications enabled
  },
  fcm_token: String  // Firebase Cloud Messaging token
}

FCM Notification Payload:

{
  token: String,  // User's FCM device token
  title: "Site Unpublished",
  body: "Your site My Business Site has been unpublished successfully.",
  data: {
    site_id: "507f1f77bcf86cd799439011",
    site_name: "site_abc123",
    event_type: "site.unpublished"
  }
}

Error Handling:

Site Not Found: Logs error and returns early
Config Not Found: Throws 404 error
Site Status Not UNPUBLISHED: Logs and returns early (prevents false positives)
User Config Not Found: Logs error and returns (notification skipped)
FCM Notification Error: Logged but doesn't throw (webhook completes)

Example Scenario:

// Duda sends webhook when user unpublishes site
await webhookService.unpublish({
  site_name: 'site_abc123',
});

// Process:
// 1. Find site in DB
// 2. Fetch latest status from Duda API
// 3. Verify status is UNPUBLISHED (not false alarm)
// 4. Update DB with new status
// 5. Check if user has notifications enabled (browser OR bell)
// 6. Send FCM push notification to user's device
// 7. User sees: "Your site My Business Site has been unpublished successfully."

Key Business Rules:

Status Verification: Only processes if Duda confirms UNPUBLISHED status (prevents false positives)
Conditional Notifications: Only sends if user has enabled browser OR bell notifications
Graceful Notification Failure: FCM errors logged but don't block webhook processing
User Preferences: Respects user's notification settings in user-config
No Thumbnail Operations: Unlike publish, unpublish doesn't trigger thumbnail changes
Simpler Flow: Fewer operations than publish event

🔀 Integration Points

External Services

Google PubSub

Topic: v2.sites-thumbnail-generator
Usage: Publish message to trigger thumbnail generation service

Message Format:

{
  site_name: String,  // Duda site ID
  model: String       // 'agency-website' or 'instasite'
}

Wasabi S3-Compatible Storage

Usage: Delete old site thumbnails before regeneration
Files Deleted:
- Desktop thumbnail: thumbnails/{site_id}/desktop.png
- Mobile thumbnail: thumbnails/{site_id}/mobile.png
- Tablet thumbnail: thumbnails/{site_id}/tablet.png

Firebase Cloud Messaging (FCM)

Usage: Send push notifications on unpublish events
Notification Types:
- Browser push notifications
- In-app bell notifications
User Control: Respects user preferences in UserConfig

Internal Services

Thumbnail Generation Service

Trigger: PubSub message from publish webhook
Responsibility: Generate new site thumbnails (desktop, mobile, tablet)
Updates: Writes new thumbnail URLs back to database

Database Collections

agency-website / instasite

Read Operations: Find site by builder_id
Write Operations: Update site metadata on publish/unpublish

config

Read Operations: Fetch Duda API credentials

user-config

Read Operations: Check notification preferences and FCM tokens

🧪 Edge Cases & Special Handling

Site Not Found in Database

Issue: Webhook received for site that doesn't exist in agency-website or instasite
Handling: Logs error and returns early without throwing

let site = await findBySiteName(resource_data.site_name);
if (!site) {
  console.log('Site not found:', resource_data.site_name);
  return; // Don't throw - webhook shouldn't fail
}

Why: Prevents webhook failures for test sites or sites deleted from DB

Status Mismatch on Unpublish

Issue: Duda sends unpublish webhook but site status is not UNPUBLISHED
Handling: Verify status with Duda API before processing

if (siteDetails.data.site_status !== 'UNPUBLISHED') {
  console.log('Status mismatch, skipping');
  return;
}

Why: Prevents false positives, ensures data integrity

Missing Thumbnails

Issue: Site has no existing thumbnails to delete
Handling: Filters out undefined values before deletion

const thumbnailKeys = [site.thumbnails.desktop, site.thumbnails.mobile, site.thumbnails.tablet]
  .filter(Boolean) // Remove undefined/null
  .map(url => url.split('.com/')[1]);

Thumbnail Deletion Failure

Issue: Wasabi deletion fails (network, permissions, etc.)
Handling: Error logged but process continues

try {
  await deleteFilesFromWasabi(thumbnailKeys);
} catch (error) {
  logger.error('Thumbnail deletion failed:', error);
  // Continue - don't block thumbnail regeneration
}

Why: Old thumbnails aren't critical - regeneration will create new ones

User Notifications Disabled

Issue: User has notifications turned off
Handling: Checks preferences before sending FCM

const notificationsEnabled =
  userConfig.push_notifications?.browser || userConfig.push_notifications?.bell;

if (!notificationsEnabled) {
  return; // Skip notification
}

FCM Token Missing

Issue: User has notifications enabled but no FCM token
Handling: Checks for token before sending

if (notificationsEnabled && userConfig.fcm_token) {
  await sendFCMNotification(...);
}

Why: Prevents FCM API errors for users without registered devices

FCM Notification Failure

Issue: FCM API fails (invalid token, network, etc.)
Handling: Error logged but webhook completes successfully

try {
  await sendFCMNotification(...);
} catch (error) {
  logger.error('FCM notification failed:', error);
  // Don't throw - notification failure shouldn't block webhook
}

Why: Webhook must return 200 to Duda regardless of notification outcome

⚠️ Important Notes

Webhook Reliability: Must return 200 OK quickly - Duda retries on failures
Async Operations: Thumbnail generation via PubSub is non-blocking
Dual Collection: Sites can be in agency-website OR instasite collections
Model Tagging: Adds model property to identify source collection
Parallel API Calls: publish() fetches site details and content simultaneously for speed
Thumbnail Cleanup: Old thumbnails deleted before regeneration to save storage costs
Notification Preferences: Respects user's push notification settings (browser OR bell)
FCM Graceful Failure: Notification errors don't break webhook processing
Status Verification: unpublish() verifies actual status to prevent false positives
No Rollback: Webhook processing is one-way, no undo mechanism
PubSub Topic: Hardcoded to v2.sites-thumbnail-generator
Wasabi S3 Keys: Extracted from thumbnail URLs (splits on .com/)
Complete Data Sync: publish() syncs all site metadata, not just thumbnails
Logging: Extensive console.log for debugging webhook issues
Error Propagation: Config errors throw, but site/user not found errors don't

Duda Integration Overview: index.md
Sites Service: sites.md - Manual publish/unpublish operations
Content Service: content.md - Content updates that trigger publish
Duda Webhooks API: Official Documentation
Google PubSub: Documentation
Firebase Cloud Messaging: Documentation
Wasabi S3 API: Documentation

📖 Overview​

🗄️ Collections Used​

agency-website​

instasite​

config​

user-config​

🔄 Data Flow​

Publish Event Flow​

Unpublish Event Flow​

🔧 Business Logic & Functions​

Helper: findBySiteName(id)​

Helper: update(id, data)​

publish(data, resource_data)​

unpublish(resource_data)​

🔀 Integration Points​

External Services​

Google PubSub​

Wasabi S3-Compatible Storage​

Firebase Cloud Messaging (FCM)​

Internal Services​

Thumbnail Generation Service​

Database Collections​

agency-website / instasite​

config​

user-config​

🧪 Edge Cases & Special Handling​

Site Not Found in Database​

Status Mismatch on Unpublish​

Missing Thumbnails​

Thumbnail Deletion Failure​

User Notifications Disabled​

FCM Token Missing​

FCM Notification Failure​

⚠️ Important Notes​

🔗 Related Documentation​

Documentation Assistant

📖 Overview

🗄️ Collections Used

`agency-website`

`instasite`

`config`

`user-config`

🔄 Data Flow

Publish Event Flow

Unpublish Event Flow

🔧 Business Logic & Functions

Helper: findBySiteName(id)

Helper: update(id, data)

publish(data, resource_data)

unpublish(resource_data)

🔀 Integration Points

External Services

Google PubSub

Wasabi S3-Compatible Storage

Firebase Cloud Messaging (FCM)

Internal Services

Thumbnail Generation Service

Database Collections

agency-website / instasite

config

user-config

🧪 Edge Cases & Special Handling

Site Not Found in Database

Status Mismatch on Unpublish

Missing Thumbnails

Thumbnail Deletion Failure

User Notifications Disabled

FCM Token Missing

FCM Notification Failure

⚠️ Important Notes

🔗 Related Documentation