Funnels Trigger - Funnel Thumbnail Generation

Overview

File: queue-manager/triggers/funnels.js
Function: buildFunnelThumbnails()
Purpose: Flag funnel steps for thumbnail regeneration when content changes

This trigger detects when funnel page HTML is modified and sets flags that signal the cron-based thumbnail processor to regenerate page previews. It demonstrates the Trigger → Flag → Cron pattern commonly used in Queue Manager.

Trigger Configuration

Watched Collection

• Collection: Funnel Steps
• Events: Insert, Update
• Watched Field: raw_html

Match Conditions

const matchConditions = {
  $or: [
    {
      operationType: 'insert',
      'fullDocument.raw_html': { $exists: true },
    },
    {
      operationType: 'update',
      'updateDescription.updatedFields.raw_html': { $exists: true },
    },
  ],
};

Key Feature: Watches for any change to the raw_html field, which contains the page's HTML content.

Data Flow

sequenceDiagram
    participant USER as User/Funnel Editor
    participant STEP as Funnel Steps
    participant TRIGGER as funnels.js Trigger
    participant CRON as Cron: funnels.js
    participant QUEUE as Queue: build-thumbnail.js
    participant S3 as S3 Storage

    USER->>STEP: Update raw_html
    STEP->>TRIGGER: Change stream event
    TRIGGER->>TRIGGER: Validate required fields
    TRIGGER->>STEP: Set step_changed: true
    TRIGGER->>STEP: Clear processing flags

    Note over CRON: */5 * * * * *<br/>Every 5 minutes

    CRON->>STEP: Query step_changed: true
    CRON->>QUEUE: Add thumbnail job
    QUEUE->>STEP: Generate thumbnail
    QUEUE->>S3: Upload image
    STEP->>STEP: Clear step_changed flag

Processing Logic

Step 1: Validate Required Fields

Before setting flags, ensure the document has all required fields:

if (!fullDocument || !fullDocument.funnel_id || !fullDocument._id || !fullDocument.base_path) {
  logger.warn({
    initiator: 'QM/triggers/funnels/build-thumbnail',
    message: 'Missing required fields in document',
    additional_data: { document: fullDocument },
  });
  return;
}

Required Fields:

• funnel_id: Parent funnel reference
• _id: Step's unique identifier
• base_path: File system path for assets

Step 2: Set Flags

Update the funnel step with processing flags:

await FunnelStep.updateOne(
  { _id: fullDocument._id },
  {
    $set: { step_changed: true },
    $unset: {
      processing_queued: '',
      thumbnail_build_in_progress: '',
      thumbnail_process_started_at: '',
    },
  },
);

Flag Meanings:

• step_changed: true - Signals cron to process this step
• processing_queued - Removed (cron will re-queue)
• thumbnail_build_in_progress - Removed (reset progress)
• thumbnail_process_started_at - Removed (clear timestamp)

Complete Code

const FunnelStep = require('../models/funnel.step');
const logger = require('../utilities/logger');
const { startChangeStream } = require('../common/changeStream');

exports.buildFunnelThumbnails = () => {
  const matchConditions = {
    $or: [
      {
        operationType: 'insert',
        'fullDocument.raw_html': { $exists: true },
      },
      {
        operationType: 'update',
        'updateDescription.updatedFields.raw_html': { $exists: true },
      },
    ],
  };

  startChangeStream(FunnelStep, matchConditions, async changeEvent => {
    try {
      const fullDocument = changeEvent.data;

      if (
        !fullDocument ||
        !fullDocument.funnel_id ||
        !fullDocument._id ||
        !fullDocument.base_path
      ) {
        logger.warn({
          initiator: 'QM/triggers/funnels/build-thumbnail',
          message: 'Missing required fields in document',
          additional_data: { document: fullDocument },
        });
        return;
      }

      await FunnelStep.updateOne(
        { _id: fullDocument._id },
        {
          $set: { step_changed: true },
          $unset: {
            processing_queued: '',
            thumbnail_build_in_progress: '',
            thumbnail_process_started_at: '',
          },
        },
      );
    } catch (error) {
      logger.error({
        initiator: 'QM/triggers/funnels/build-thumbnail',
        message: error.message,
        raw_error: error,
        additional_data: { documentId: changeEvent.data?._id || 'unknown' },
      });
    }
  });
};

Trigger → Flag → Cron Pattern

This trigger exemplifies a common Queue Manager pattern:

Pattern Flow

graph LR
    A[User Action] -->|Updates Data| B[Trigger Fires]
    B -->|Sets Flag| C[Database Record]
    C -->|Cron Queries| D[Cron Job]
    D -->|Adds to Queue| E[Bull Queue]
    E -->|Processes| F[Worker]
    F -->|Clears Flag| C

    style B fill:#f9f
    style D fill:#9f9
    style E fill:#99f

Why This Pattern?

Advantages:

1. Separation of Concerns: Trigger handles flagging, cron handles processing
2. Batch Processing: Cron can process multiple flagged items together
3. Rate Limiting: Cron controls processing frequency (every 5 minutes)
4. Retry Logic: Failed processing can be retried by cron
5. Resource Management: Heavy processing happens in scheduled batches

Alternative (Not Used): Trigger could directly add to queue, but this would:

• Process immediately on every change (may be too frequent)
• Bypass rate limiting
• Create queue congestion during bulk edits
• Complicate error handling

Cron Job Integration

Cron Schedule

File: crons/funnels.js
Schedule: */5 * * * * * (every 5 minutes)

Cron Logic

// Simplified cron logic
const flaggedSteps = await FunnelStep.find({
  step_changed: true,
  processing_queued: { $ne: true },
});

for (const step of flaggedSteps) {
  // Mark as queued
  await FunnelStep.updateOne({ _id: step._id }, { $set: { processing_queued: true } });

  // Add to queue
  await buildThumbnailQueue.add({ stepId: step._id });
}

Queue Processor

File: queues/funnels/build-thumbnail.js

// Simplified processor logic
buildThumbnailQueue.process(async job => {
  const { stepId } = job.data;

  // Set in-progress flag
  await FunnelStep.updateOne(
    { _id: stepId },
    {
      $set: {
        thumbnail_build_in_progress: true,
        thumbnail_process_started_at: new Date(),
      },
    },
  );

  // Generate thumbnail (Puppeteer screenshot)
  const thumbnailBuffer = await generateThumbnail(stepId);

  // Upload to S3
  await uploadToS3(thumbnailBuffer);

  // Clear all flags
  await FunnelStep.updateOne(
    { _id: stepId },
    {
      $set: { thumbnail_url: s3Url },
      $unset: {
        step_changed: '',
        processing_queued: '',
        thumbnail_build_in_progress: '',
        thumbnail_process_started_at: '',
      },
    },
  );
});

Flag State Machine

State Transitions

stateDiagram-v2
    [*] --> Idle: Initial state

    Idle --> Flagged: Trigger fires<br/>(step_changed: true)

    Flagged --> Queued: Cron detects<br/>(processing_queued: true)

    Queued --> Processing: Worker picks up<br/>(thumbnail_build_in_progress: true)

    Processing --> Idle: Success<br/>(all flags cleared)

    Processing --> Flagged: Error<br/>(flags cleared, can retry)

    Flagged --> Flagged: Content changes again<br/>(flags reset)

Flag Combinations

step_changed	processing_queued	thumbnail_build_in_progress	Status
false	false	false	✅ Idle (thumbnail up to date)
true	false	false	🔔 Awaiting cron pickup
true	true	false	📋 Queued, waiting for worker
true	true	true	⚙️ Processing in progress
false	false	false	✅ Complete (all flags cleared)

Use Cases

Use Case 1: User Edits Funnel Page

Scenario: Marketing team updates landing page headline

Flow:

1. User edits HTML in funnel editor
2. Frontend saves: PATCH /funnel-steps/:id { raw_html: "..." }
3. Trigger fires immediately (< 100ms)
4. step_changed: true flag set
5. Cron runs in next 5-minute window
6. Thumbnail generated with new headline
7. Users see updated preview in funnel list

Timing: ~5 minutes from edit to new thumbnail

Use Case 2: Bulk Funnel Import

Scenario: User imports 20-page funnel from template

Flow:

1. 20 funnel steps created with raw_html
2. Trigger fires 20 times
3. All 20 steps flagged with step_changed: true
4. Single cron run picks up all 20
5. Queue processes 20 thumbnails (may take 2-3 minutes)
6. All thumbnails ready for preview

Advantage: Batch processing is more efficient than 20 immediate triggers

Use Case 3: Content Rollback

Scenario: User reverts to previous page version

Flow:

1. Old raw_html restored
2. Trigger fires (HTML changed)
3. Existing flags cleared (prevents old thumbnail job from completing)
4. New flag set
5. Thumbnail regenerated with reverted content

Key: Clearing flags prevents race conditions with in-progress jobs

Collections Updated

Direct Updates

• Funnel Steps - Flags updated

Flag Fields

{
    step_changed: Boolean,           // Set by trigger
    processing_queued: Boolean,      // Set by cron, cleared by trigger
    thumbnail_build_in_progress: Boolean,  // Set by worker, cleared by trigger
    thumbnail_process_started_at: Date     // Set by worker, cleared by trigger
}

Error Handling

Missing Required Fields

if (!fullDocument.funnel_id || !fullDocument._id || !fullDocument.base_path) {
  logger.warn({
    initiator: 'QM/triggers/funnels/build-thumbnail',
    message: 'Missing required fields in document',
    additional_data: { document: fullDocument },
  });
  return;
}

Effect: Trigger exits early, no flags set, no processing attempted.

Update Errors

try {
  await FunnelStep.updateOne(/* ... */);
} catch (error) {
  logger.error({
    initiator: 'QM/triggers/funnels/build-thumbnail',
    message: error.message,
    raw_error: error,
    additional_data: { documentId: changeEvent.data?._id || 'unknown' },
  });
}

Effect: Error logged, change stream continues, step may be processed on next HTML change.

Performance Considerations

Optimizations

1. Immediate Flagging: Flag set in < 100ms
2. Deferred Processing: Heavy thumbnail generation happens asynchronously
3. Batch-Friendly: Multiple flagged steps processed together by cron
4. Single Update: One database write per trigger fire

Bottleneck Avoidance

• No Puppeteer in Trigger: Screenshot generation deferred to queue
• No S3 Upload in Trigger: File operations deferred to queue
• No Complex Queries: Simple updateOne with _id

Why Separate Trigger and Cron?

Benefits

1. Instant Feedback: Flag set immediately, user knows change detected
2. Controlled Load: Cron limits processing to once per 5 minutes
3. Batch Efficiency: Process multiple changes together
4. Error Recovery: Cron can retry flagged items
5. Resource Isolation: Heavy processing in queue worker, not trigger thread

Tradeoffs

• Latency: 5-minute delay before thumbnail appears
• Complexity: Two-stage process vs single-stage
• State Management: Multiple flags to track

Decision: Latency acceptable for thumbnail generation (not critical path)

Related Systems

Upstream

• Funnel Editor (Internal API): Creates/updates funnel steps
• Template Import (Internal API): Bulk creates funnel steps

Downstream

• Cron: crons/funnels.js queries flagged steps
• Queue: queues/funnels/build-thumbnail.js generates images
• S3: Stores generated thumbnail images

Complementary

• Funnel Preview: Displays thumbnails in funnel list
• Analytics: Tracks thumbnail generation metrics

Monitoring

Key Metrics

• Trigger Frequency: Funnel steps modified per hour
• Flag Persistence: Time steps remain flagged before cron pickup
• Processing Time: Time from flag set to thumbnail complete
• Error Rate: Failed validations or flag updates

Logging

// Warning: Missing fields
logger.warn({
  initiator: 'QM/triggers/funnels/build-thumbnail',
  message: 'Missing required fields in document',
  additional_data: { document: fullDocument },
});

// Error: Update failed
logger.error({
  initiator: 'QM/triggers/funnels/build-thumbnail',
  message: error.message,
  raw_error: error,
  additional_data: { documentId: documentId },
});

Best Practices

1. Validate Before Processing

if (!fullDocument.funnel_id || ...) {
    return; // Don't set flags for invalid data
}

Prevents invalid jobs from entering queue.

2. Clear Stale Flags

$unset: {
    processing_queued: '',
    thumbnail_build_in_progress: ''
}

Prevents race conditions with in-progress jobs.

3. Comprehensive Error Logging

Include context for debugging:

additional_data: {
  documentId: changeEvent.data?._id || 'unknown';
}

4. Specific Field Watching

Only watch fields that matter:

'updateDescription.updatedFields.raw_html': { $exists: true }

Improvement Opportunities

1. Priority Queue

High-importance funnels could be processed first:

$set: {
    step_changed: true,
    priority: funnel.is_active ? 'high' : 'low'
}

2. Incremental Updates

Detect if HTML change is minor (text only) vs major (layout):

const htmlDiff = calculateDiff(oldHtml, newHtml);
if (htmlDiff.severity === 'minor') {
  // Skip thumbnail regeneration
}

3. Smart Batching

Batch all steps in same funnel together:

// Instead of processing steps individually
// Process entire funnel at once for efficiency

Summary

The funnels trigger implements a clean Trigger → Flag → Cron pattern for thumbnail generation. By immediately setting flags when HTML changes but deferring heavy processing to scheduled cron jobs, it balances responsiveness with resource efficiency. The multi-flag state machine prevents race conditions and enables retry logic, making thumbnail generation reliable and scalable.

Key Characteristics:

• 🚩 Immediate flagging on HTML changes
• ⏱️ Deferred processing via cron (5-minute interval)
• 🔄 Multi-flag state machine prevents race conditions
• 🎯 Simple, focused responsibility (just set flags)
• 📊 Validates required fields before flagging
• 🔗 Integrates with cron and queue systems
• ⚡ Enables batch processing for efficiency