Skip to main content

❌ Cancel Domains

📖 Overview​

The Cancel Domains job processes Lightning Domain cancellation requests by deleting custom hostnames from Cloudflare, removing database records, updating associated funnels to test mode, and sending cancellation notifications. It runs every 6 hours, identifies domains flagged for cancellation, and processes deletions with automatic retries and error handling.

Complete Flow:

  1. Cron Initialization: queue-manager/crons/domains/cancel.js
  2. Service Processing: queue-manager/services/domains/cancel.js
  3. Queue Definition: queue-manager/queues/domains/cancel.js

Execution Pattern: Scheduled (every 6 hours)

Queue Name: domains_cancel

Environment Flag: QM_DOMAINS_CANCEL=true (in index.js)

🔄 Complete Processing Flow​

sequenceDiagram
    participant CRON as Cron Schedule<br/>(every 6 hours)
    participant SERVICE as Cancel Service
    participant LD_DB as Lightning<br/>Domains DB
    participant QUEUE as Bull Queue
    participant PROCESSOR as Job Processor
    participant CF_API as Cloudflare API
    participant FUNNEL_DB as Funnels DB
    participant NOTIF_SERVICE as Notification<br/>Service

    CRON->>SERVICE: cancel()
    SERVICE->>LD_DB: Find domains:<br/>- cancel = true<br/>- cancel_in_progress ≠ true
    LD_DB-->>SERVICE: Domains flagged for cancellation

    alt Domains found
        SERVICE->>LD_DB: Set cancel_in_progress=true<br/>(batch updateMany)

        loop Each domain
            SERVICE->>QUEUE: Add job: {domain}
        end

        loop Each queued domain
            QUEUE->>PROCESSOR: Process cancellation

            loop Retry up to 3 times
                PROCESSOR->>CF_API: DELETE /custom_hostnames/{cf_id}
                CF_API-->>PROCESSOR: {success: true/false}

                alt API failure
                    PROCESSOR->>PROCESSOR: Wait 1 second, retry
                end
            end

            alt Cloudflare deletion successful
                PROCESSOR->>LD_DB: deleteOne: Remove domain record
                PROCESSOR->>FUNNEL_DB: reviewAndSetStatusForDomain:<br/>Set funnels to test mode
                PROCESSOR->>NOTIF_SERVICE: GET /run-service?type=domains.cancelled
                NOTIF_SERVICE-->>PROCESSOR: Notification sent
            end

            PROCESSOR-->>QUEUE: done()
        end
    end

📁 Source Files​

1. Cron Initialization​

File: queue-manager/crons/domains/cancel.js

Purpose: Schedule domain cancellation processing every 6 hours

Cron Pattern: 0 */6 * * * (every 6 hours at minute 0)

Initialization:

const cancel = require('../../services/domains/cancel');
const cron = require('node-cron');
const logger = require('../../utilities/logger');

let inProgress = false;
exports.start = async () => {
  try {
    cron.schedule('0 */6 * * *', async () => {
      if (!inProgress) {
        inProgress = true;
        await cancel();
        inProgress = false;
      }
    });
  } catch (err) {
    logger.error({ initiator: 'QM/domains/cancel', error: err });
  }
};

In-Progress Lock: Prevents overlapping executions (unlikely with 6-hour interval).

Why Every 6 Hours?

  • Domain cancellations are infrequent (1-10 per day)
  • Not time-critical (can wait a few hours)
  • Reduces Cloudflare API usage

2. Service Processing (THE FLAGGED DOMAINS PATTERN)​

File: queue-manager/services/domains/cancel.js

Purpose: Find domains flagged for cancellation and queue for processing

Key Functions:

  • Query domains with cancel=true flag
  • Set cancel_in_progress flag before queuing
  • Add domains to Bull queue

Main Processing Function:

const { default: axios } = require('axios');
const Queue = require('../../queues/domains/cancel');
const LightningDomain = require('../../models/lightning-domain');
const logger = require('../../utilities/logger');

module.exports = async () => {
  try {
    const domains = await LightningDomain.find({
      cancel_in_progress: { $ne: true }, // Not already processing
      cancel: true, // Flagged for cancellation
    });

    if (domains.length) {
      const ids = domains.map(queue => queue._id);

      // Handle the promise returned by updateMany
      await LightningDomain.updateMany({ _id: { $in: ids } }, { cancel_in_progress: true })
        .then(() => {
          // Log success if the update was successful
          logger.log({
            initiator: 'QM/domains/cancel',
            message: 'Domain cancel status updated successfully',
          });
        })
        .catch(err => {
          // Log the error if the update failed
          logger.error({
            initiator: 'QM/domains/cancel',
            message: 'Error updating domain cancel status',
            error: err,
          });
        });

      const queue = await Queue.start();
      for (const domain of domains) {
        queue.add(
          { domain },
          {
            attempts: 3,
          },
        );
      }
    }

    logger.log({
      initiator: 'QM/domains/cancel',
      message: 'Domain cancel service processed.',
    });
  } catch (err) {
    logger.error({ initiator: 'QM/domains/cancel', error: err });
  }
};

3. Queue Processor (THE CLOUDFLARE DELETION LOGIC)​

File: queue-manager/queues/domains/cancel.js

Purpose: Delete domain from Cloudflare, database, update funnels, send notification

Key Functions:

  • Retry Cloudflare API deletion (3 attempts with 1-second delay)
  • Delete domain from database
  • Review and set funnel status
  • Send cancellation notification

Main Processor:

const QueueWrapper = require('../../common/queue-wrapper');
const LightningDomain = require('../../models/lightning-domain');
const logger = require('../../utilities/logger');
const { reviewAndSetStatusForDomain } = require('../../utilities/funnelDomainStatusUtility');
const axios = require('axios');

const processCb = async (job, done) => {
  try {
    let { domain } = job.data;

    // Retry Cloudflare API call on failure
    let resp = await retry(
      async () => {
        return await axios.delete(
          `https://api.cloudflare.com/client/v4/zones/${process.env.LD_CLOUDFLARE_ZONE_ID}/custom_hostnames/${domain.cf_id}`,
          { headers: { Authorization: 'Bearer ' + process.env.CLOUDFLARE_API_KEY } },
        );
      },
      3, // Number of retries
      1000, // Delay between retries (in milliseconds)
    );

    resp = resp.data;
    if (!resp?.success) {
      throw new Error(resp.errors?.[0]?.message);
    }

    // Handle database errors
    try {
      await LightningDomain.deleteOne({ _id: domain._id });
      await reviewAndSetStatusForDomain({ domain_id: domain._id });
    } catch (dbErr) {
      logger.error({
        initiator: 'QM/domains/cancel',
        message: 'Database error during domain deletion',
        error: dbErr,
      });
      throw dbErr; // Re-throw to fail the job
    }

    // Handle notification service promise
    await axios
      .get(
        `${
          process.env.NOTIFICATION_SERVICE_URL
        }/run-service?type=domains.cancelled&data=${JSON.stringify(domain)}`,
      )
      .then(() => {
        logger.log({
          initiator: 'QM/domains/cancel',
          message: 'Domain cancellation notification sent successfully',
        });
      })
      .catch(err => {
        logger.error({
          initiator: 'QM/domains/cancel',
          message: 'Notification hook error: ',
          error: err,
        });
      });

    done();
    return;
  } catch (err) {
    logger.error({
      initiator: 'QM/domains/cancel',
      error: err,
      api_error: err?.response?.data?.errors?.[0]?.message || undefined,
    });
    done(err);
  }
};

// Helper function for retrying operations
async function retry(fn, retries, delay) {
  let attempt = 0;
  while (attempt < retries) {
    try {
      return await fn();
    } catch (err) {
      attempt++;
      logger.warn({
        initiator: 'QM/domains/cancel',
        message: `Retry attempt ${attempt} for operation`,
        error: err,
      });
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }
  throw new Error(`Operation failed after ${retries} retries`);
}

let queue;

exports.start = async () => {
  try {
    if (!queue) queue = QueueWrapper(`domains_cancel`, 'global', { processCb });
    return Promise.resolve(queue)
      .then(result => {
        logger.log({
          initiator: 'QM/domains/cancel',
          message: 'Queue resolved' + result,
        });
        return result;
      })
      .catch(error => {
        logger.error({
          initiator: 'QM/domains/cancel',
          message: 'Error resolving queue:' + error,
        });
        throw error;
      });
  } catch (err) {
    logger.error({
      initiator: 'QM/domains/cancel',
      message: 'Error while initializing domain validity queue: ' + err.message + err.stack,
    });
  }
};

🗄️ Collections Used​

lightning_domains​

  • Operations: Find (query flagged domains), Update (set cancel flag), Delete (remove record)
  • Model: shared/models/lightning-domain.js
  • Usage Context: Track and process domain cancellations

Query Criteria (Flagged for Cancellation):

{
    cancel_in_progress: { $ne: true },             // Not already processing
    cancel: true,                                  // Flagged for cancellation
}

Batch Flag Update (Before Queuing):

await LightningDomain.updateMany({ _id: { $in: ids } }, { cancel_in_progress: true });

Delete Operation:

await LightningDomain.deleteOne({ _id: domain._id });

Key Fields:

  • cancel: Boolean flag indicating cancellation request
  • cancel_in_progress: Boolean flag preventing duplicate processing
  • cf_id: Cloudflare custom hostname ID
  • hostname: Domain name (e.g., 'custom.example.com')
  • account_id: Reference to account (used in notification)

funnels (Indirect via Utility)​

  • Operations: Update (via reviewAndSetStatusForDomain utility)
  • Model: shared/models/funnel.js
  • Usage Context: Set funnels to test mode when domain is cancelled

Utility Function: utilities/funnelDomainStatusUtility.js

Purpose: Updates all funnels using the cancelled domain to prevent broken live funnels

🔧 Job Configuration​

Cron Schedule​

'0 */6 * * *'; // Every 6 hours at minute 0 (12am, 6am, 12pm, 6pm)

Why 6-Hour Interval?

  • Domain cancellations are infrequent
  • Not time-critical (can wait a few hours)
  • Reduces Cloudflare API usage

Job Settings​

queue.add(
  { domain }, // Full domain document
  {
    attempts: 3, // Retry up to 3 times
  },
);

Job Data:

  • domain: Complete Lightning Domain document

Queue Configuration​

QueueWrapper(`domains_cancel`, 'global', { processCb });

Queue Name: domains_cancel

Redis Scope: global (shared across queue manager instances)

📋 Processing Logic - Detailed Flow​

1. Domain Cancellation Flag​

Set by Internal API:

// User requests domain cancellation via frontend
await LightningDomain.updateOne({ _id: domainId }, { cancel: true });

Queue Manager Query:

const domains = await LightningDomain.find({
  cancel_in_progress: { $ne: true },
  cancel: true,
});

Why $ne: true Instead of false?

  • Handles null/undefined values (defensive querying)
  • Ensures flag doesn't exist or is explicitly false

2. Pre-Queue Flagging​

Batch Flag Update:

const ids = domains.map(queue => queue._id);
await LightningDomain.updateMany({ _id: { $in: ids } }, { cancel_in_progress: true });

Why Flag Before Queuing?

  • Prevents race condition if cron runs again before jobs complete
  • Atomic operation ensures consistency
  • More efficient than individual updates

3. Cloudflare Deletion with Retry Logic​

Custom Retry Function:

async function retry(fn, retries, delay) {
  let attempt = 0;
  while (attempt < retries) {
    try {
      return await fn();
    } catch (err) {
      attempt++;
      logger.warn({
        initiator: 'QM/domains/cancel',
        message: `Retry attempt ${attempt} for operation`,
        error: err,
      });
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }
  throw new Error(`Operation failed after ${retries} retries`);
}

Usage:

let resp = await retry(
  async () => {
    return await axios.delete(
      `https://api.cloudflare.com/client/v4/zones/${process.env.LD_CLOUDFLARE_ZONE_ID}/custom_hostnames/${domain.cf_id}`,
      { headers: { Authorization: 'Bearer ' + process.env.CLOUDFLARE_API_KEY } },
    );
  },
  3, // Number of retries
  1000, // 1-second delay between retries
);

Retry Parameters:

  • Attempts: 3 retries (4 total attempts including initial)
  • Delay: 1 second between retries
  • Total Time: Up to 4 seconds per domain

Why Retry?

  • Transient network errors
  • Cloudflare API rate limits
  • Temporary service outages

4. Cloudflare API Response​

Success Response:

{
    success: true,
    result: {
        id: 'cf123...',
        hostname: 'custom.example.com'
    },
    errors: [],
    messages: []
}

Failure Response:

{
    success: false,
    errors: [
        {
            code: 1003,
            message: 'custom hostname not found'
        }
    ]
}

Error Handling:

resp = resp.data;
if (!resp?.success) {
  throw new Error(resp.errors?.[0]?.message);
}

5. Database Deletion​

Permanent Removal:

await LightningDomain.deleteOne({ _id: domain._id });

Critical Note: This is permanent deletion - no soft delete, no undo.

Why Delete After Cloudflare?

  • Ensures Cloudflare deletion succeeded first
  • Prevents orphaned database records
  • Maintains data consistency

6. Funnel Status Review​

Update Associated Funnels:

await reviewAndSetStatusForDomain({ domain_id: domain._id });

Utility Function (utilities/funnelDomainStatusUtility.js):

Purpose: Set all funnels using this domain to test mode

Logic (inferred):

// Find all funnels using this domain
const funnels = await Funnel.find({ domain_id: domain._id, status: 'live' });

// Set to test mode to prevent broken live funnels
await Funnel.updateMany({ domain_id: domain._id, status: 'live' }, { status: 'test' });

Why Update Funnels?

  • Cancelled domain = no longer accessible
  • Prevents users from accessing broken funnels
  • Maintains data integrity

7. Cancellation Notification​

Notification Service Call:

await axios.get(
  `${process.env.NOTIFICATION_SERVICE_URL}/run-service?type=domains.cancelled&data=${JSON.stringify(
    domain,
  )}`,
);

Request URL Example:

GET http://notification-service:5008/run-service?type=domains.cancelled&data={"_id":"507f...","hostname":"custom.example.com","account_id":"507f..."}

Notification Service Processing:

  • Sends email to account owner
  • Logs cancellation event
  • Updates account activity feed
  • May trigger refund processing

Error Handling:

.then(() => {
    logger.log({ message: 'Domain cancellation notification sent successfully' });
})
.catch(err => {
    logger.error({ message: 'Notification hook error: ', error: err });
});

Non-Blocking: Notification failure doesn't fail the job (logged only).

🚨 Error Handling​

Common Error Scenarios​

Cloudflare API Errors​

Domain Not Found:

{
  errors: [{ code: 1003, message: 'custom hostname not found' }];
}

Result: Job fails after 3 retries, domain remains in database with cancel_in_progress=true.

Rate Limit Exceeded:

{
  errors: [{ code: 10000, message: 'rate limit exceeded' }];
}

Result: Retry logic handles with 1-second delay, likely succeeds on retry.

Database Errors​

Connection Lost:

Scenario: MongoDB connection lost during deletion

Result: Exception thrown, job fails, retries up to 3 times.

Recovery: Mongoose reconnection handles transient failures.

Deletion Conflict:

Scenario: Domain already deleted by concurrent process

Result: deleteOne succeeds (0 documents deleted), no error thrown.

Notification Service Errors​

Service Unavailable:

Scenario: Notification service down or unreachable

Result: Error logged, job completes successfully (notification is non-critical).

Invalid Domain Data:

Scenario: Domain object missing required fields for notification

Result: Notification fails, logged, job completes.

Retry Behavior​

Custom Retry Function:

  • Cloudflare API: 3 retries with 1-second delay (inline retry logic)
  • Bull Queue: 3 attempts (Bull retry mechanism)
  • Total Possible Attempts: 3 (Bull) × 4 (inline retries) = up to 12 Cloudflare API calls

Note: This is likely excessive and could lead to rate limit issues.

📊 Monitoring & Logging​

Success Logging​

Service-Level:

logger.log({
  initiator: 'QM/domains/cancel',
  message: 'Domain cancel status updated successfully',
});

logger.log({
  initiator: 'QM/domains/cancel',
  message: 'Domain cancel service processed.',
});

Notification Success:

logger.log({
  initiator: 'QM/domains/cancel',
  message: 'Domain cancellation notification sent successfully',
});

Error Logging​

Service Query Errors:

logger.error({
  initiator: 'QM/domains/cancel',
  message: 'Error updating domain cancel status',
  error: err,
});

Processor Errors:

logger.error({
  initiator: 'QM/domains/cancel',
  error: err,
  api_error: err?.response?.data?.errors?.[0]?.message || undefined,
});

Database Errors:

logger.error({
  initiator: 'QM/domains/cancel',
  message: 'Database error during domain deletion',
  error: dbErr,
});

Retry Warnings:

logger.warn({
  initiator: 'QM/domains/cancel',
  message: `Retry attempt ${attempt} for operation`,
  error: err,
});

Performance Metrics​

  • Service Query Time: < 100ms (indexed queries)
  • Cloudflare API Latency: 200-500ms per deletion
  • Job Processing Time: 1-5 seconds per domain (with retries)
  • Typical Volume: 1-10 domains per 6-hour interval

🔗 Integration Points​

Triggers This Job​

  • Internal API: Sets cancel=true flag on domain cancellation request
  • Manual Trigger: Via API endpoint (if QM_HOOKS=true)

External Dependencies​

  • Cloudflare API: Custom Hostnames DELETE endpoint

    • Zone ID: process.env.LD_CLOUDFLARE_ZONE_ID
    • API Key: process.env.CLOUDFLARE_API_KEY
    • Rate Limit: 1,200 requests per 5 minutes

  • Notification Service: Domain cancellation notifications

    • URL: process.env.NOTIFICATION_SERVICE_URL
    • Endpoint: /run-service?type=domains.cancelled

Jobs That Depend On This​

  • Domain Cleanup: May clean up orphaned records (see cleanup.md)
  • Billing Adjustments: May trigger refunds for prepaid domains

Utilities Used​

  • reviewAndSetStatusForDomain: Updates funnel status when domain is cancelled

⚠️ Important Notes​

Side Effects​

  • ⚠️ Permanent Deletion: Removes domain from Cloudflare AND database (no undo)
  • ⚠️ Funnel Impact: Sets all funnels using this domain to test mode
  • ⚠️ Notification Sent: Account owner receives cancellation email
  • ⚠️ No Refund Logic: Refunds must be handled separately (if applicable)

Performance Considerations​

  • 6-Hour Polling: Low frequency, suitable for infrequent cancellations
  • Batch Flagging: Uses updateMany for efficient flag management
  • Retry Logic: 3 retries with 1-second delay per attempt
  • Indexed Queries: Ensure indexes on cancel, cancel_in_progress

Business Logic​

Why 6-Hour Interval?

  • Cancellations are infrequent (1-10 per day)
  • Not time-critical (can wait a few hours)
  • Reduces Cloudflare API usage

Why Delete from Cloudflare First?

  • Ensures external state cleaned up before internal
  • Prevents orphaned Cloudflare records
  • Cloudflare deletion is idempotent (safe to retry)

Why Non-Blocking Notifications?

  • Notification failure shouldn't prevent cancellation
  • User can be notified via other channels
  • Reduces job failure rate

Why Permanent Deletion?

  • Cancelled domains don't need recovery
  • Simplifies database management
  • User can recreate if needed

Maintenance Notes​

  • No Failed Callback: Missing failedCb means cancel_in_progress flag never cleared on failure
  • Excessive Retries: 3 Bull retries × 4 inline retries = up to 12 Cloudflare API calls
  • Notification Service: Monitor availability for successful notifications
  • Database Indexes: Ensure composite index on (cancel, cancel_in_progress)

Potential Issues​

Missing Failed Callback:

Problem: No failedCb defined in queue configuration.

Impact: Domains that fail all 3 retry attempts remain stuck with cancel_in_progress=true.

Workaround: Manual database cleanup or wait for next deployment fix.

Recommendation: Add failed callback:

const failedCb = async (job, err) => {
  const id = job.data.domain._id;
  if (job.attemptsMade >= job.opts.attempts) {
    await LightningDomain.updateOne({ _id: id }, { cancel_in_progress: false });
  }
};

queue = QueueWrapper(`domains_cancel`, 'global', { processCb, failedCb });

🧪 Testing​

Manual Trigger​

# Via API (if QM_HOOKS=true)
POST http://localhost:6002/api/trigger/domains/cancel

Simulate Domain Cancellation​

// Flag domain for cancellation
const domain = await LightningDomain.findOne({ hostname: 'test.example.com' });
await LightningDomain.updateOne({ _id: domain._id }, { cancel: true });

// Wait for next cron run (6 hours) or trigger manually

// Verify deletion
const deleted = await LightningDomain.findById(domain._id);
console.log('Domain deleted:', !deleted); // Should be true

Test Cloudflare Deletion​

// Manually call Cloudflare API
const response = await axios.delete(
  `https://api.cloudflare.com/client/v4/zones/${process.env.LD_CLOUDFLARE_ZONE_ID}/custom_hostnames/${domain.cf_id}`,
  { headers: { Authorization: 'Bearer ' + process.env.CLOUDFLARE_API_KEY } },
);

console.log('Cloudflare deletion:', response.data.success); // true
console.log('Errors:', response.data.errors); // []

Test Funnel Status Update​

// Setup: Create funnel with domain
const funnel = await Funnel.create({
  name: 'Test Funnel',
  domain_id: domain._id,
  status: 'live',
});

// Trigger cancellation
await cancel();

// Verify funnel set to test mode
const updatedFunnel = await Funnel.findById(funnel._id);
console.log('Funnel status:', updatedFunnel.status); // Should be 'test'

Test Notification​

// Mock notification service response
const mockServer = express();
mockServer.get('/run-service', (req, res) => {
  console.log('Notification received:', req.query.type); // 'domains.cancelled'
  console.log('Domain data:', JSON.parse(req.query.data));
  res.json({ success: true });
});
mockServer.listen(5008);

// Trigger cancellation
await cancel();

// Verify notification logged
// Expected log: "Domain cancellation notification sent successfully"

Monitor Stuck Cancellations​

// Find domains stuck in cancellation
const stuckDomains = await LightningDomain.find({
  cancel: true,
  cancel_in_progress: true,
});

console.log(`${stuckDomains.length} domains stuck in cancellation`);

// Manual cleanup (if needed)
await LightningDomain.updateMany(
  { _id: { $in: stuckDomains.map(d => d._id) } },
  { cancel_in_progress: false },
);

Test Retry Logic​

// Simulate Cloudflare API failure
const originalDelete = axios.delete;
let attemptCount = 0;

axios.delete = async (...args) => {
  attemptCount++;
  if (attemptCount < 3) {
    throw new Error('Simulated Cloudflare API failure');
  }
  return originalDelete(...args);
};

// Trigger cancellation
await cancel();

// Verify retries
console.log('Total attempts:', attemptCount); // Should be 3
console.log('Final result: Success'); // Third attempt succeeded

// Restore original delete
axios.delete = originalDelete;

Job Type: Scheduled + Queued
Execution Frequency: Every 6 hours
Average Duration: 1-5 seconds per domain
Status: Active

💬

Documentation Assistant

Ask me anything about the docs

Hi! I'm your documentation assistant. Ask me anything about the docs!

I can help you with:
- Code examples
- Configuration details
- Troubleshooting
- Best practices

Try asking: How do I configure the API?
09:31 AM