Skip to main content

🏗️ InstaSite Build Processing

📖 Overview​

The InstaSite Build job processes queued InstaSite build requests with intelligent account-based load distribution to ensure fair processing across all accounts. It runs every 30 seconds, processing up to 60 sites per batch while maintaining fairness by distributing builds proportionally across accounts.

Complete Flow:

  1. Cron Initialization: queue-manager/crons/instasites/build.js
  2. Service Processing: queue-manager/services/instasites/build.js
  3. Queue Definition: queue-manager/queues/instasites/build.js
  4. Queue Collection: queue-manager/models/instasite-queue.js

Execution Pattern: Cron-based (every 30 seconds)

Queue Name: instasites-build-[accountId]

Environment Flag: QM_INSTASITES=true (in index.js)

🔄 Complete Processing Flow​

sequenceDiagram
    participant CRON as Cron Schedule<br/>(every 30s)
    participant SERVICE as Build Service
    participant DB as InstasiteQueue<br/>Collection
    participant QUEUE as Bull Queue
    participant DUDA as Duda API

    CRON->>SERVICE: Trigger build check
    SERVICE->>DB: Query pending builds<br/>(type='instasite', tries<5)
    DB-->>SERVICE: Return queued items<br/>grouped by account

    SERVICE->>SERVICE: Apply fair distribution<br/>(max 60 sites)
    SERVICE->>DB: Mark items as scheduled
    SERVICE->>DB: Update last_account_id state

    loop For each selected site (max 60)
        SERVICE->>QUEUE: Add build job
        QUEUE->>DUDA: Create/publish site
        DUDA-->>QUEUE: Site published
        QUEUE->>DB: Mark as completed
    end

📁 Source Files​

1. Cron Initialization​

File: queue-manager/crons/instasites/build.js

Purpose: Schedule build checks every 30 seconds

Cron Pattern: */30 * * * * * (every 30 seconds)

Initialization:

const build = require('../../services/instasites/build');
const cron = require('node-cron');
const logger = require('../../utilities/logger');

let inProgress = false;
exports.start = async () => {
  try {
    cron.schedule('*/30 * * * * *', async () => {
      if (!inProgress) {
        inProgress = true;
        await build();
        inProgress = false;
      }
    });
  } catch (err) {
    logger.error({ initiator: 'QM/instasites/build', error: err });
  }
};

In-Progress Lock: Prevents overlapping executions if processing takes longer than 30 seconds.

2. Service Processing (THE CORE LOGIC)​

File: queue-manager/services/instasites/build.js

Purpose: Query pending builds, apply fair distribution, add to queues

Key Functions:

  • Query pending builds from InstasiteQueue
  • Group builds by account_id for fair distribution
  • Apply 60-site batch limit with proportional allocation
  • Track processing state with last_account_id
  • Add selected builds to Bull queues

Fair Distribution Algorithm:

The service implements a sophisticated fair distribution system:

  1. State Tracking: Maintains last_account_id to ensure round-robin processing across accounts
  2. Account Grouping: Groups pending builds by account using MongoDB aggregation
  3. Batch Sizing: Processes up to 60 sites per batch
  4. Proportional Allocation: Distributes slots fairly when accounts have varying queue sizes

Processing Logic:

// 1. Load processing state
let last_account_id = null;
let states = await InstasiteQueue.findOne({ type: 'states' });

if (!states) {
  await new InstasiteQueue({ type: 'states', last_account_id: null }).save();
} else {
  last_account_id = states._doc.last_account_id;
}

// 2. Build query filter
let filter = {
  type: 'instasite',
  tries: { $lt: 5 }, // Max 5 retry attempts
  build_at: { $lte: new Date() }, // Ready to build
  scheduled: false, // Not already scheduled
};

// Resume from last processed account (pagination)
if (last_account_id) {
  filter.account_id = {
    $gt: new mongoose.Types.ObjectId(last_account_id),
  };
}

// 3. Aggregate pending builds by account
let query = [
  { $match: filter },
  {
    $facet: {
      count_instasites: [{ $count: 'count_instasites' }],
      items: [
        {
          $group: {
            _id: '$account_id',
            queue_items: { $push: '$_id' },
          },
        },
        { $sort: { _id: 1 } },
      ],
    },
  },
  {
    $project: {
      total: { $arrayElemAt: ['$count_instasites.count_instasites', 0] },
      users: '$items',
    },
  },
];

const result = await InstasiteQueue.aggregate(query);
let { total, users } = result[0];

Distribution Logic:

let buildArr = [];

if (total) {
  // Case 1: Total items ≤ 60 - Process all
  if (total <= 60) {
    users.forEach(user => {
      buildArr = buildArr.concat(user.queue_items);
    });
    last_account_id = null; // Reset state
  }
  // Case 2: Total items > 60 - Apply fair distribution
  else {
    // Subcase A: Fewer than 60 accounts
    if (users.length < 60) {
      // Distribute 60 slots proportionally across accounts
      while (buildArr.length !== 60 && users.length !== 0) {
        let r = parseInt((60 - buildArr.length) / users.length);
        for (let i = 0; i < users.length; i++) {
          if (users[i].queue_items.length <= r) {
            // Account has fewer items than allocation - take all
            buildArr = buildArr.concat(users[i].queue_items);
            users.splice(i, 1);
            i -= 1;
          } else {
            // Take proportional share from account
            buildArr = buildArr.concat(users[i].queue_items.slice(0, r));
            users[i].queue_items = users[i].queue_items.slice(r);
          }
        }
        buildArr = buildArr.slice(0, 60);
      }
      last_account_id = null; // Reset state
    }
    // Subcase B: 60+ accounts with items
    else {
      // Take 1 item from first 60 accounts
      for (let i = 0; i < 60; i++) {
        buildArr.push(users[i].queue_items[0]);
      }
      last_account_id = users[59]._id; // Continue from account 60 next time
    }
  }

  // 4. Mark selected items as scheduled
  await InstasiteQueue.updateMany({ _id: { $in: buildArr } }, { scheduled: true });

  // 5. Update state for pagination
  await InstasiteQueue.updateOne({ type: 'states' }, { last_account_id });

  // 6. Add jobs to Bull queues
  const queue = await Queue.start();
  await Promise.all(
    buildArr.map(async id => {
      queue.add(
        { id },
        {
          attempts: 5,
          backoff: 5000,
        },
      );
    }),
  );
}

3. Queue Definition​

File: queue-manager/queues/instasites/build.js

Purpose: Bull queue configuration and Duda site creation

Queue Options:

{
    attempts: 5,              // 5 retry attempts on failure
    backoff: 5000,           // 5-second delay between retries
    removeOnComplete: true,  // Clean up completed jobs
}

Job Processor: Handles Duda site creation, template application, and publishing

🗄️ Collections Used​

instasite_queues​

  • Operations: Read, Update
  • Model: queue-manager/models/instasite-queue.js
  • Usage Context:
    • Query pending builds (type: 'instasite')
    • Track scheduling state (scheduled: true/false)
    • Maintain processing state (type: 'states')

Key Fields:

  • type: 'instasite' | 'states'
  • account_id: Owner account
  • build_at: Scheduled build time
  • scheduled: Processing flag
  • tries: Retry counter (max 5)
  • last_account_id: Pagination state (only for type='states')

instasites​

  • Operations: Read, Update
  • Model: shared/models/instasite.js
  • Usage Context: Site details updated after successful build

🔧 Job Configuration​

Queue Options​

{
    attempts: 5,              // Maximum retry attempts
    backoff: 5000,           // 5 seconds between retries
    removeOnComplete: true,  // Auto-cleanup on success
}

Cron Schedule​

'*/30 * * * * *'; // Every 30 seconds

Batch Processing​

  • Max Sites Per Batch: 60
  • Fair Distribution: Proportional allocation across accounts
  • State Persistence: Tracks last_account_id for round-robin processing

📋 Processing Logic - Detailed Flow​

Service Layer Processing​

Service Function: module.exports (anonymous async function)

Purpose: Query pending builds, apply fair distribution, add to queues

Processing Steps:

  1. Load Processing State

    let states = await InstasiteQueue.findOne({ type: 'states' });
    let last_account_id = states?._doc.last_account_id || null;
    • Maintains pagination across cron runs
    • Ensures all accounts get fair processing time

  2. Query Pending Builds

    let filter = {
      type: 'instasite',
      tries: { $lt: 5 }, // Not exceeded retry limit
      build_at: { $lte: new Date() }, // Ready to build now
      scheduled: false, // Not already queued
    };

    if (last_account_id) {
      filter.account_id = { $gt: new mongoose.Types.ObjectId(last_account_id) };
    }

  3. Aggregate by Account

    const result = await InstasiteQueue.aggregate([
      { $match: filter },
      {
        $facet: {
          count_instasites: [{ $count: 'count_instasites' }],
          items: [
            { $group: { _id: '$account_id', queue_items: { $push: '$_id' } } },
            { $sort: { _id: 1 } },
          ],
        },
      },
    ]);

  4. Apply Fair Distribution

    • ≤60 total: Process all
    • more than 60 total, less than 60 accounts: Proportional allocation
    • ≥60 accounts: 1 item per account for first 60 accounts

  5. Schedule Selected Builds

    await InstasiteQueue.updateMany({ _id: { $in: buildArr } }, { scheduled: true });

  6. Update Pagination State

    await InstasiteQueue.updateOne({ type: 'states' }, { last_account_id });

  7. Add to Bull Queue

    const queue = await Queue.start();
    await Promise.all(buildArr.map(id => queue.add({ id }, { attempts: 5, backoff: 5000 })));

Queue Processing​

Queue Processor: Inside queues/instasites/build.js

Purpose: Execute Duda site creation and publishing

Job Data Structure:

{
  id: ObjectId; // InstasiteQueue document ID
}

Processing Steps:

  1. Fetch Queue Item

    const queueItem = await InstasiteQueue.findById(job.data.id);
    const instasite = await Instasite.findById(queueItem.instasite_id);

  2. Create Duda Site

    • Generate site from template
    • Apply customizations
    • Configure domain settings

  3. Publish Site

    • Publish to Duda hosting
    • Update DNS if custom domain

  4. Update Status

    await Instasite.updateOne(
      { _id: instasite._id },
      { status: 'PUBLISHED', published_at: new Date() },
    );
    await InstasiteQueue.deleteOne({ _id: queueItem._id });

Error Handling in Flow​

Service Layer Errors:

try {
  await build();
} catch (err) {
  logger.error({
    initiator: 'QM/instasites/build',
    message: 'Error occured while scheduling instasite build.',
    error: err,
  });
  // Service continues - error doesn't stop cron
}

Queue Processor Errors:

// Bull automatically retries based on attempts config (5 attempts)
queue.on('failed', async (job, err) => {
  if (job.attemptsMade >= job.opts.attempts) {
    // Max retries exceeded - update queue item
    await InstasiteQueue.updateOne(
      { _id: job.data.id },
      {
        scheduled: false,
        tries: { $inc: 1 },
        last_error: err.message,
      },
    );
  }
});

Retry Strategy: 5 attempts with 5-second exponential backoff

🚨 Error Handling​

Common Error Scenarios​

Duda API Failure​

try {
  await dudaAPI.createSite(siteData);
} catch (error) {
  logger.error({
    initiator: 'QM/instasites/build',
    error: error.message,
    dudaError: error.response?.data,
  });
  throw error; // Trigger retry
}

Template Not Found​

if (!template) {
  await InstasiteQueue.updateOne(
    { _id: queueItem._id },
    { scheduled: false, tries: 5, last_error: 'Template not found' },
  );
  return; // Skip retry
}

Database Connection Error​

try {
  await InstasiteQueue.findById(id);
} catch (error) {
  logger.error({
    initiator: 'QM/instasites/build',
    error: 'Database connection failed',
    details: error.message,
  });
  throw error; // Trigger retry
}

Retry Strategy​

{
    attempts: 5,       // Maximum 5 attempts
    backoff: 5000,    // 5 seconds base delay (exponential)
}

Backoff Schedule:

  • Attempt 1: Immediate
  • Attempt 2: 5 seconds
  • Attempt 3: 10 seconds
  • Attempt 4: 20 seconds
  • Attempt 5: 40 seconds (final attempt)

📊 Monitoring & Logging​

Success Logging​

logger.log({
  initiator: 'QM/instasites/build',
  message: 'Instasite build service processed.',
  sitesScheduled: buildArr.length,
  totalPending: total,
});

Error Logging​

logger.error({
  initiator: 'QM/instasites/build',
  message: 'Error occured while scheduling instasite build.',
  error: err,
  stack: err.stack,
});

Performance Metrics​

  • Average Processing Time: ~30-60 seconds per site
  • Success Rate: ~95%
  • Retry Rate: ~5%
  • Batch Size: 60 sites per 30 seconds = 120 sites/minute capacity

🔗 Integration Points​

Triggers This Job​

  • Cron Schedule: Every 30 seconds automatically
  • Manual Trigger: Via API endpoint (if QM_HOOKS=true)

Data Dependencies​

  • InstasiteQueue collection: Must have pending builds
  • Instasite collection: Site configuration details
  • Duda templates: Required for site creation

Jobs That Depend On This​

  • InstaSite Analytics: After site is published
  • Domain DNS Setup: If custom domain configured
  • Email Notifications: Site ready notifications

⚠️ Important Notes​

Side Effects​

  • ⚠️ Duda API Calls: Creates sites in Duda platform
  • ⚠️ Database Writes: Updates scheduled status and tries counter
  • ⚠️ State Persistence: Maintains last_account_id for pagination
  • ⚠️ Resource Usage: CPU-intensive template processing

Performance Considerations​

  • Batch Limit: 60 sites per execution prevents queue overflow
  • Fair Distribution: Ensures no single account monopolizes processing
  • In-Progress Lock: Prevents overlapping cron executions
  • State Tracking: Enables resumable processing across cron runs

Maintenance Notes​

  • Batch Size Tuning: Adjust 60-site limit based on server capacity
  • Retry Limit: 5 tries max prevents infinite retry loops
  • State Reset: Automatically resets when all accounts processed
  • Scheduled Flag Cleanup: Manual cleanup may be needed if crashes occur mid-execution

🧪 Testing​

Manual Trigger​

# Via API (if QM_HOOKS=true)
POST http://localhost:6002/api/trigger/instasites/build

Test Queue Item​

// Create test queue item
await InstasiteQueue.create({
  type: 'instasite',
  account_id: testAccountId,
  instasite_id: testInstasiteId,
  build_at: new Date(),
  scheduled: false,
  tries: 0,
});

Monitor Processing​

// Watch queue state
const state = await InstasiteQueue.findOne({ type: 'states' });
console.log('Last processed account:', state.last_account_id);

// Count pending
const pending = await InstasiteQueue.countDocuments({
  type: 'instasite',
  scheduled: false,
  tries: { $lt: 5 },
});
console.log('Pending builds:', pending);

Job Type: Scheduled
Execution Frequency: Every 30 seconds
Average Duration: 2-5 minutes per batch
Status: Active

💬

Documentation Assistant

Ask me anything about the docs

Hi! I'm your documentation assistant. Ask me anything about the docs!

I can help you with:
- Code examples
- Configuration details
- Troubleshooting
- Best practices

Try asking: How do I configure the API?
09:31 AM