Skip to main content

🗑️ InstaSite Purge Processing

📖 Overview​

The InstaSite Purge job automatically removes outdated unpublished sites and orphaned sites from both the DashClicks database and Duda platform. It runs every 30 seconds to identify sites that haven't been published within the configured retention period (default 90 days) or sites marked as orphaned.

Complete Flow:

  1. Cron Initialization: queue-manager/crons/instasites/purge.js
  2. Service Processing: queue-manager/services/instasites/purge.js
  3. Queue Definition: queue-manager/queues/instasites/purge.js

Execution Pattern: Cron-based (every 30 seconds)

Queue Name: instasites-purge-[accountId]

Environment Flag: QM_INSTASITES=true (in index.js)

🔄 Complete Processing Flow​

sequenceDiagram
    participant CRON as Cron Schedule<br/>(every 30s)
    participant SERVICE as Purge Service
    participant DB as Instasite<br/>Collection
    participant QUEUE as Bull Queue<br/>(per account)
    participant DUDA as Duda API

    CRON->>SERVICE: Trigger purge check
    SERVICE->>DB: Query outdated sites<br/>(NOT_PUBLISHED_YET > 90 days<br/>OR orphaned=true)
    DB-->>SERVICE: Return sites<br/>grouped by account

    SERVICE->>DB: Mark purge_in_progress=true

    loop For each account
        SERVICE->>QUEUE: Create account queue
        loop For each site in account
            SERVICE->>QUEUE: Add purge job
            QUEUE->>DUDA: Delete site from Duda
            DUDA-->>QUEUE: Site deleted
            QUEUE->>DB: Delete Instasite document
        end
    end

📁 Source Files​

1. Cron Initialization​

File: queue-manager/crons/instasites/purge.js

Purpose: Schedule purge checks every 30 seconds

Cron Pattern: */30 * * * * * (every 30 seconds)

Initialization:

const purge = require('../../services/instasites/purge');
const cron = require('node-cron');
const logger = require('../../utilities/logger');

let inProgress = false;
exports.start = async () => {
  try {
    cron.schedule('*/30 * * * * *', async () => {
      if (!inProgress) {
        inProgress = true;
        await purge();
        inProgress = false;
      }
    });
  } catch (err) {
    logger.error({ initiator: 'QM/instasites/purge', error: err });
  }
};

In-Progress Lock: Prevents overlapping executions.

2. Service Processing (THE CORE LOGIC)​

File: queue-manager/services/instasites/purge.js

Purpose: Identify outdated sites, group by account, add to purge queues

Purge Criteria:

const DAYS_TO_PURGE_NEW =
  parseInt(process.env.DAYS_UNTIL_INSTASITE_PURGE || '90') * 24 * 60 * 60 * 1000;

Default: 90 days (configurable via environment variable)

Processing Logic:

module.exports = async () => {
  try {
    // 1. Query outdated sites
    const outdatedSites = await Instasite.aggregate([
      {
        $match: {
          $or: [
            {
              // Unpublished for > 90 days
              status: 'NOT_PUBLISHED_YET',
              created: {
                $lt: new Date(Date.now() - DAYS_TO_PURGE_NEW),
              },
            },
            {
              // Orphaned sites (no valid account)
              orphaned: true,
            },
          ],
          purge_in_progress: { $ne: true }, // Not already purging
        },
      },
      {
        // Group by account for parallel processing
        $group: {
          _id: '$account_id',
          ids: {
            $push: '$_id',
          },
        },
      },
    ]);

    if (outdatedSites.length > 0) {
      // 2. Collect all site IDs
      const ids = [];
      outdatedSites.forEach(site => {
        site.ids.forEach(id => {
          ids.push(id);
        });
      });

      // 3. Mark all as purge in progress
      await Instasite.updateMany({ _id: { $in: ids } }, { purge_in_progress: true });

      // 4. Create account-specific queues and add jobs
      await Promise.all(
        outdatedSites.map(async site => {
          let queue;
          try {
            // Create queue for this account
            queue = await purge_queue.start(site._id.toString());
          } catch (err) {
            // Reset purge flag on queue creation failure
            await Instasite.updateMany({ account_id: site._id }, { purge_in_progress: false });
            throw err;
          }

          // Add each site to the account's queue
          await Promise.all(
            site.ids.map(async id => {
              try {
                await queue.add(
                  { id },
                  {
                    jobId: id.toString(),
                    attempts: 10,
                    backoff: 5000,
                  },
                );
              } catch (err) {
                // Reset purge flag on job add failure
                await Instasite.updateOne({ _id: id }, { purge_in_progress: false });
                throw err;
              }
            }),
          );
        }),
      );

      console.log('Instasite purge service processed.');
    }
  } catch (err) {
    console.error(
      `Error occured while getting outdated sites.\n` +
        `Path: /queue-manager/services/instasites/purge.js.\n` +
        `Error: `,
      err.message,
      err.stack,
    );
  }
};

3. Queue Definition​

File: queue-manager/queues/instasites/purge.js

Purpose: Bull queue configuration and Duda site deletion

Account-Specific Queues: Creates separate queue per account for parallel processing

Queue Options:

{
    jobId: siteId.toString(),  // Idempotent job IDs
    attempts: 10,              // 10 retry attempts
    backoff: 5000,            // 5-second delay between retries
    removeOnComplete: true,   // Auto-cleanup on success
}

Job Processor: Handles Duda site deletion and database cleanup

🗄️ Collections Used​

instasites​

  • Operations: Read, Update, Delete
  • Model: shared/models/instasite.js
  • Usage Context:
    • Query unpublished sites older than 90 days
    • Query orphaned sites
    • Mark sites as purge_in_progress
    • Delete sites after Duda cleanup

Key Fields:

  • status: 'NOT_PUBLISHED_YET' | 'PUBLISHED'
  • created: Site creation timestamp
  • orphaned: Boolean flag for orphaned sites
  • purge_in_progress: Lock flag to prevent duplicate purging
  • account_id: Owner account

🔧 Job Configuration​

Queue Options​

{
    jobId: siteId.toString(),  // Prevents duplicate jobs
    attempts: 10,              // Maximum retry attempts
    backoff: 5000,            // 5 seconds between retries
    removeOnComplete: true,   // Auto-cleanup on success
}

Cron Schedule​

'*/30 * * * * *'; // Every 30 seconds

Retention Policy​

# Environment variable (default: 90 days)
DAYS_UNTIL_INSTASITE_PURGE=90

📋 Processing Logic - Detailed Flow​

Purge Criteria​

Sites are purged if they meet either condition:

  1. Unpublished Timeout:

    • Status: NOT_PUBLISHED_YET
    • Created: More than 90 days ago (configurable)

  2. Orphaned Site:

    • Flag: orphaned: true
    • Reason: Account deleted or invalid reference

Service Layer Processing​

Service Function: module.exports (anonymous async function)

Purpose: Query outdated sites, group by account, create purge jobs

Processing Steps:

  1. Query Outdated Sites

    const outdatedSites = await Instasite.aggregate([
      {
        $match: {
          $or: [
            {
              status: 'NOT_PUBLISHED_YET',
              created: { $lt: new Date(Date.now() - DAYS_TO_PURGE_NEW) },
            },
            { orphaned: true },
          ],
          purge_in_progress: { $ne: true },
        },
      },
      {
        $group: {
          _id: '$account_id',
          ids: { $push: '$_id' },
        },
      },
    ]);

  2. Mark as Purge In Progress

    const ids = [];
    outdatedSites.forEach(site => {
      site.ids.forEach(id => ids.push(id));
    });

    await Instasite.updateMany({ _id: { $in: ids } }, { purge_in_progress: true });

  3. Create Account-Specific Queues

    await Promise.all(
      outdatedSites.map(async site => {
        const queue = await purge_queue.start(site._id.toString());
        // Account-specific queue for parallel processing
      }),
    );

  4. Add Purge Jobs

    await Promise.all(
      site.ids.map(async id => {
        await queue.add(
          { id },
          {
            jobId: id.toString(), // Idempotent
            attempts: 10,
            backoff: 5000,
          },
        );
      }),
    );

Queue Processing​

Queue Processor: Inside queues/instasites/purge.js

Purpose: Delete site from Duda and database

Job Data Structure:

{
  id: ObjectId; // Instasite document ID
}

Processing Steps:

  1. Fetch Instasite

    const instasite = await Instasite.findById(job.data.id);
    if (!instasite) {
      return { skipped: true, reason: 'Site not found' };
    }

  2. Delete from Duda

    if (instasite.duda_site_name) {
      await dudaAPI.deleteSite(instasite.duda_site_name);
    }

  3. Delete from Database

    await Instasite.deleteOne({ _id: instasite._id });

  4. Return Result

    return {
      success: true,
      deletedSite: instasite.duda_site_name,
      deletedId: instasite._id,
    };

Error Handling in Flow​

Service Layer Errors:

try {
  const queue = await purge_queue.start(site._id.toString());
} catch (err) {
  // Reset purge flag on queue creation failure
  await Instasite.updateMany({ account_id: site._id }, { purge_in_progress: false });
  throw err;
}

Job Add Errors:

try {
  await queue.add({ id }, { jobId: id.toString(), attempts: 10, backoff: 5000 });
} catch (err) {
  // Reset purge flag for specific site
  await Instasite.updateOne({ _id: id }, { purge_in_progress: false });
  throw err;
}

Queue Processor Errors:

// Bull automatically retries based on attempts config (10 attempts)
queue.on('failed', async (job, err) => {
  if (job.attemptsMade >= job.opts.attempts) {
    // Max retries exceeded - reset purge flag
    await Instasite.updateOne(
      { _id: job.data.id },
      {
        purge_in_progress: false,
        purge_error: err.message,
      },
    );
  }
});

Retry Strategy: 10 attempts with 5-second exponential backoff

🚨 Error Handling​

Common Error Scenarios​

Duda API Failure​

try {
  await dudaAPI.deleteSite(siteId);
} catch (error) {
  if (error.response?.status === 404) {
    // Site already deleted in Duda - continue with DB cleanup
    console.log('Site not found in Duda, cleaning up database only');
  } else {
    throw error; // Trigger retry
  }
}

Site Not Found in Database​

const instasite = await Instasite.findById(job.data.id);
if (!instasite) {
  // Already deleted - job is idempotent
  return { skipped: true, reason: 'Site already deleted' };
}

Queue Creation Failure​

try {
  queue = await purge_queue.start(accountId);
} catch (err) {
  // Reset purge_in_progress for all sites in this account
  await Instasite.updateMany({ account_id: accountId }, { purge_in_progress: false });
  throw err;
}

Retry Strategy​

{
    attempts: 10,      // Maximum 10 attempts
    backoff: 5000,    // 5 seconds base delay (exponential)
}

Backoff Schedule:

  • Attempt 1: Immediate
  • Attempt 2: 5 seconds
  • Attempt 3: 10 seconds
  • Attempt 4: 20 seconds
  • Attempt 5-10: Progressive exponential backoff

📊 Monitoring & Logging​

Success Logging​

console.log('Instasite purge service processed.');
console.log(`Purged ${sitesCount} sites across ${accountsCount} accounts`);

Error Logging​

console.error(
  `Error occured while getting outdated sites.\n` +
    `Path: /queue-manager/services/instasites/purge.js.\n` +
    `Error: `,
  err.message,
  err.stack,
);

Performance Metrics​

  • Average Processing Time: ~5-10 seconds per site
  • Success Rate: ~98%
  • Retry Rate: ~2%
  • Daily Volume: Varies based on unpublished site accumulation

🔗 Integration Points​

Triggers This Job​

  • Cron Schedule: Every 30 seconds automatically
  • Manual Trigger: Via API endpoint (if QM_HOOKS=true)

Data Dependencies​

  • Instasite collection: Must have sites matching purge criteria
  • Duda platform: Sites must exist in Duda for deletion
  • InstaSite Build: Creates sites that may later be purged
  • Account Deletion: May mark sites as orphaned

⚠️ Important Notes​

Side Effects​

  • ⚠️ Duda API Calls: Permanently deletes sites from Duda platform
  • ⚠️ Database Deletion: Removes Instasite documents (irreversible)
  • ⚠️ Storage Cleanup: Frees up Duda hosting resources
  • ⚠️ Account-Specific Queues: Creates multiple Bull queues simultaneously

Performance Considerations​

  • Parallel Processing: Separate queues per account for concurrency
  • Idempotent Jobs: Uses jobId to prevent duplicate deletions
  • In-Progress Lock: Prevents re-purging same site
  • Batch Retry: 10 attempts with backoff reduces API rate limit issues

Maintenance Notes​

  • Retention Period: Default 90 days is configurable via DAYS_UNTIL_INSTASITE_PURGE
  • Orphaned Sites: Manually marked by account deletion processes
  • Purge Flag Reset: May need manual intervention if service crashes mid-purge
  • Duda Rate Limits: 10 retry attempts handle temporary API throttling

🧪 Testing​

Manual Trigger​

# Via API (if QM_HOOKS=true)
POST http://localhost:6002/api/trigger/instasites/purge

Create Test Site for Purging​

// Create old unpublished site
await Instasite.create({
  account_id: testAccountId,
  status: 'NOT_PUBLISHED_YET',
  created: new Date(Date.now() - 91 * 24 * 60 * 60 * 1000), // 91 days ago
  duda_site_name: 'test-site-123',
  purge_in_progress: false,
});

// Create orphaned site
await Instasite.create({
  account_id: deletedAccountId,
  status: 'PUBLISHED',
  orphaned: true,
  duda_site_name: 'orphaned-site-456',
  purge_in_progress: false,
});

Monitor Purge Progress​

// Count sites pending purge
const pendingPurge = await Instasite.countDocuments({
  $or: [
    {
      status: 'NOT_PUBLISHED_YET',
      created: { $lt: new Date(Date.now() - DAYS_TO_PURGE_NEW) },
    },
    { orphaned: true },
  ],
  purge_in_progress: false,
});

console.log('Sites pending purge:', pendingPurge);

// Count sites currently purging
const inProgress = await Instasite.countDocuments({
  purge_in_progress: true,
});

console.log('Sites currently purging:', inProgress);

Job Type: Scheduled
Execution Frequency: Every 30 seconds
Average Duration: 5-10 seconds per site
Status: Active

💬

Documentation Assistant

Ask me anything about the docs

Hi! I'm your documentation assistant. Ask me anything about the docs!

I can help you with:
- Code examples
- Configuration details
- Troubleshooting
- Best practices

Try asking: How do I configure the API?
09:31 AM