Store - Draft Invoices Synchronization

Overview

The Draft Invoices Synchronization module automatically syncs draft invoice status from Stripe to the local MongoDB database. It monitors the StoreInvoice collection for invoices with status: 'draft' and fetches updated invoice details from Stripe every 12 hours, ensuring the local database reflects the current state of invoices that may have been finalized or modified in Stripe.

Key Features:

• Scheduled Sync: Runs every 12 hours to check draft invoice status
• Stripe API Integration: Retrieves invoice details directly from Stripe
• Stale Lock Recovery: Handles stuck in-progress flags with 6-hour timeout
• Selective Updates: Only updates invoices that have left draft status
• Bulk Processing: Processes multiple draft invoices concurrently
• Automatic Cleanup: Resets in-progress flags after processing

Critical Business Impact:

• Data Accuracy: Ensures local database matches Stripe invoice state
• Status Synchronization: Captures draft → open/paid/void transitions
• Invoice Finalization: Detects when draft invoices are finalized
• Audit Trail: Maintains up-to-date billing records

---

Architecture

Execution Flow

sequenceDiagram
    participant Cron as Cron Scheduler
    participant Service as Draft Invoice Service
    participant DB as MongoDB
    participant Queue as Bull Queue
    participant Processor as Sync Processor
    participant Stripe as Stripe API

    Note over Cron,Stripe: Every 12 Hours

    Cron->>Service: Trigger sync check
    Service->>DB: Find draft invoices
    Note over DB: status: 'draft'<br/>in_progress: false<br/>OR stale (6h+ ago)
    DB-->>Service: Draft invoices array

    Service->>DB: Mark in_progress = true

    loop For each draft invoice
        Service->>Queue: Add to sync queue
        Note over Queue: 5 attempts, 5s backoff

        Queue->>Processor: Process sync job
        Processor->>Stripe: GET /v1/invoices/{id}
        Note over Stripe: Retrieve invoice details
        Stripe-->>Processor: Invoice data

        alt Invoice Status Changed
            Processor->>DB: Update StoreInvoice
            Note over DB: Replace with Stripe data<br/>Convert stripe_id field
        else Still Draft
            Processor->>Processor: Skip update
            Note over Processor: No changes needed
        end

        Processor->>DB: Set in_progress = false
        Processor->>Processor: Log completion
    end

Component Structure

queue-manager/
├── crons/
│   └── store/
│       └── draftInvoices.js              # Cron scheduler
├── services/
│   └── store/
│       └── stripe/
│           └── draftInvoices.js          # Service logic
└── queues/
    └── store/
        └── stripe/
            └── draft_invoices.js         # Queue processor

Cron Schedule

File: queue-manager/crons/store/draftInvoices.js

'0 */12 * * *'; // Every 12 hours (at minute 0)

Pattern: Low-frequency scheduler for periodic synchronization

• In-Progress Locking: Prevents concurrent executions
• Purpose: Regular sync to catch invoice status changes
• Frequency: Twice daily (0:00 and 12:00)

---

Configuration

Environment Variables

Variable	Type	Required	Description
STRIPE_SECRET_KEY	String	Yes	Stripe API secret key for invoice retrieval

Queue Retry Configuration

Pattern: queue-manager/services/store/stripe/draftInvoices.js

{
    attempts: 5,
    backoff: 5000  // 5 seconds fixed delay
}

Retry Schedule:

Attempt	Delay	Total Wait
1	0s	0s
2	5s	5s
3	5s	10s
4	5s	15s
5	5s	20s

Total: ~20 seconds of retry attempts (fixed backoff, not exponential)

---

Service Implementation

Draft Invoice Detection

File: queue-manager/services/store/stripe/draftInvoices.js

Query Pattern with Stale Lock Recovery

const sixHoursAgo = new Date(Date.now() - 6 * 60 * 60 * 1000);

let getDraftInvoices = await StoreInvoiceModel.find({
  $or: [
    { status: 'draft', in_progress: false },
    {
      status: 'draft',
      in_progress: true,
      updated_at: { $gte: sixHoursAgo.toISOString() },
    },
  ],
});

Conditions:

1. Fresh Drafts: status: 'draft', in_progress: false

   • Normal case: draft invoices not currently syncing

2. Stale Locks: status: 'draft', in_progress: true, updated_at >= 6h ago

   • Fallback: Handles cases where in_progress was never reset
   • Timeout: 6 hours indicates likely process crash or failure

Purpose: Prevents permanently stuck invoices from blocking future syncs

In-Progress Locking

const ids = getDraftInvoices.map(d => d._id);
await StoreInvoiceModel.updateMany({ _id: { $in: ids } }, { in_progress: true });

Pattern: Bulk update sets in-progress flag before queue addition

• Race Condition Prevention: Multiple cron executions won't process same invoices
• Atomic Operation: Single database update for all invoices

Queue Addition

if (getDraftInvoices.length) {
  let Q = await draftInvoiceQueue.start();
  await Promise.all(
    getDraftInvoices.map(async invoiceParams => {
      invoiceParams = invoiceParams.toObject();

      try {
        await Q.add(
          {
            id: invoiceParams.id,
            stripe_id: invoiceParams.stripe_id,
            status: invoiceParams.status,
          },
          {
            attempts: 5,
            backoff: 5000,
          },
        );
      } catch (err) {
        await StoreInvoiceModel.updateMany({ _id: { $in: ids } }, { in_progress: false });
        console.error('Error while scheduling build: ', err.message, err.stack);
      }
    }),
  );
  console.log('fetch from stripe Started');
}

Payload:

• id: MongoDB document ID
• stripe_id: Stripe invoice ID
• status: Current status from MongoDB

Error Handling:

• Resets in_progress: false for all invoices on queue addition failure
• Logs error details for investigation

---

Queue Processor

Stripe Invoice Retrieval

File: queue-manager/queues/store/stripe/draft_invoices.js

Process Callback

const processCb = async (job, done) => {
  const { id, stripe_id, status } = job.data;
  try {
    // Get invoice details from Stripe
    const invoice = await stripe.invoices.retrieve(stripe_id);

    if (invoice) {
      const invoiceStatus = invoice.status;

      // Update only if status is not draft
      if (invoiceStatus !== 'draft') {
        invoice.stripe_id = invoice.id;
        delete invoice.id;

        // Update store.invoices details
        await draftInvoiceModel.updateOne({ _id: new mongoose.Types.ObjectId(id) }, invoice);
      }
    }
    return done();
  } catch (err) {
    done(err);
  }
};

Stripe API Call:

stripe.invoices.retrieve(stripe_id);

Returns: Full invoice object from Stripe with all current data

Selective Update Logic

if (invoiceStatus !== 'draft') {
  // Update invoice in database
}

Conditional Update:

• Still Draft: Skip database update (no changes)
• Status Changed: Update entire invoice document

Rationale: Avoid unnecessary database writes for unchanged invoices

Field Transformation

invoice.stripe_id = invoice.id;
delete invoice.id;

Purpose: Stripe returns id, but MongoDB stores as stripe_id

• Normalization: Consistent field naming across collections
• Preservation: Original Stripe ID maintained as stripe_id

Database Update

await draftInvoiceModel.updateOne({ _id: new mongoose.Types.ObjectId(id) }, invoice);

Operation: Full document replacement with Stripe data

• Overwrites: All fields replaced with Stripe values
• Preserves: MongoDB _id field (query target)

Success Handling

Callback: completedCb

const completedCb = async job => {
  const { id } = job.data;
  try {
    await draftInvoiceModel.updateOne(
      { _id: new mongoose.Types.ObjectId(id) },
      { in_progress: false },
    );
  } catch (err) {
    console.error('Complete status update on invoice', err.message, err.stack);
    logger.error({
      initiator: 'QM/startup/QM_DRAFT_INVOICE',
      message: 'Complete status update on invoice',
      error: err,
      job: job.id,
      job_data: job.data,
    });
  }
};

Action: Reset in_progress: false after successful sync

• Cleanup: Releases lock for future sync cycles
• Error Handling: Logs failure if update fails (doesn't throw)

Failure Handling

Callback: failedCb

const failedCb = async (job, err) => {
  const { id } = job.data;
  try {
    await draftInvoiceModel.updateOne(
      { _id: new mongoose.Types.ObjectId(id) },
      { in_progress: false },
    );
  } catch (err) {
    console.error('Failed to update status', err.message, err.stack);
    logger.error({
      initiator: 'QM/startup/QM_DRAFT_INVOICE',
      message: 'Failed to update status',
      error: err,
      job: job.id,
      job_data: job.data,
    });
  }
};

Action: Reset in_progress: false even on failure

• Recovery: Allows future sync attempts
• No Retry After Max: 5 attempts exhausted, flag cleared

---

Data Models

StoreInvoice Document Structure

Collection: StoreInvoice

{
    _id: ObjectId,
    stripe_id: String,              // Stripe invoice ID (not 'id')
    status: String,                 // 'draft' | 'open' | 'paid' | 'void' | 'uncollectible'
    in_progress: Boolean,           // Sync processing lock flag
    amount_due: Number,             // Amount in cents
    amount_paid: Number,            // Amount paid in cents
    currency: String,               // Currency code
    customer: String,               // Stripe customer ID
    subscription: String,           // Stripe subscription ID (if applicable)
    lines: {                        // Invoice line items
        data: [{
            description: String,
            amount: Number,
            quantity: Number
        }]
    },
    metadata: Object,               // Custom metadata
    created: Number,                // Unix timestamp
    updated_at: String,             // ISO timestamp for stale lock detection
    // ... all other Stripe invoice fields
}

Key Fields:

• status: Current invoice state from Stripe
• in_progress: Prevents concurrent sync operations
• updated_at: Used for stale lock timeout (6 hours)
• stripe_id: Stripe's invoice ID (field name differs from Stripe's id)

Invoice Status Lifecycle

stateDiagram-v2
    [*] --> draft: Invoice Created
    draft --> open: Finalized
    draft --> void: Deleted
    open --> paid: Payment Succeeded
    open --> uncollectible: Payment Failed
    paid --> [*]
    void --> [*]
    uncollectible --> [*]

Status Transitions:

• draft → open: Invoice finalized, sent to customer
• draft → void: Invoice deleted before finalization
• open → paid: Payment successful
• open → uncollectible: Payment failed after retries

Sync Trigger: Any transition away from draft status

---

Error Handling

Common Error Scenarios

1. Stripe API Rate Limiting

Cause: Too many API requests in short period

Response: Stripe returns 429 status code

Handling: Fixed 5-second backoff between retries

{
    attempts: 5,
    backoff: 5000
}

Recovery: Typically succeeds on subsequent attempts

2. Invoice Not Found

Cause: Invoice deleted from Stripe or invalid stripe_id

Response: Stripe throws error with code resource_missing

Impact: Job fails, in_progress flag reset after 5 attempts

Resolution: May require manual database cleanup

3. Network Timeout

Cause: Stripe API temporarily unavailable

Handling: Axios timeout + retry logic

Recovery: Usually resolves with retry

4. Stale Lock Timeout

Cause: Process crash during previous sync

Detection: in_progress: true + updated_at >= 6h ago

Recovery: Automatically included in next sync query

{
    in_progress: true,
    updated_at: { $gte: sixHoursAgo.toISOString() }
}

Prevention: 6-hour timeout prevents permanent lock

Database Update Failures

Scenarios:

1. MongoDB connection lost during update
2. Document deleted between query and update
3. Validation error on Stripe data

Handling:

try {
  await draftInvoiceModel.updateOne({ _id: id }, invoice);
} catch (err) {
  done(err); // Triggers retry
}

Retry: Exponential backoff for transient issues

---

Stripe API Integration

Invoice Retrieve Endpoint

Stripe API: GET /v1/invoices/{invoice_id}

Node.js SDK:

const stripe = require('stripe')(process.env.STRIPE_SECRET_KEY);
const invoice = await stripe.invoices.retrieve('in_...');

Response Structure:

{
  "id": "in_...",
  "object": "invoice",
  "status": "open",
  "amount_due": 5000,
  "amount_paid": 0,
  "currency": "usd",
  "customer": "cus_...",
  "subscription": "sub_...",
  "lines": {
    "data": [
      {
        "description": "Monthly subscription",
        "amount": 5000,
        "quantity": 1
      }
    ]
  },
  "created": 1640995200,
  "metadata": {}
}

Rate Limits:

• Standard: 100 requests/second
• Burst: Higher limits for short periods

Field Mapping

Stripe Field	MongoDB Field	Notes
id	stripe_id	Renamed for consistency
status	status	Direct copy
amount_due	amount_due	Amount in cents
amount_paid	amount_paid	Amount in cents
All others	Same name	Direct copy

---

Testing Scenarios

1. Draft to Open Transition

Setup:

const invoice = await StoreInvoice.create({
  stripe_id: 'in_draft123',
  status: 'draft',
  in_progress: false,
  amount_due: 5000,
  currency: 'usd',
});

// Mock Stripe to return 'open' status
nock('https://api.stripe.com').get('/v1/invoices/in_draft123').reply(200, {
  id: 'in_draft123',
  status: 'open',
  amount_due: 5000,
  currency: 'usd',
});

Expected Flow:

1. Service detects draft invoice
2. Stripe API returns updated invoice with status: 'open'
3. MongoDB document updated with new status
4. in_progress: false set

2. Invoice Still Draft

Setup:

nock('https://api.stripe.com').get('/v1/invoices/in_draft123').reply(200, {
  id: 'in_draft123',
  status: 'draft',
  amount_due: 5000,
});

Expected Flow:

1. Stripe API returns invoice still in draft
2. Update skipped (no database write)
3. in_progress: false set

3. Stale Lock Recovery

Setup:

const sevenHoursAgo = new Date(Date.now() - 7 * 60 * 60 * 1000);

await StoreInvoice.create({
  stripe_id: 'in_stale123',
  status: 'draft',
  in_progress: true,
  updated_at: sevenHoursAgo.toISOString(),
});

Expected Flow:

1. Query includes stale locked invoice (>6 hours old)
2. Invoice synced despite in_progress: true
3. Lock reset after sync

4. Stripe API Failure with Retry

Setup:

nock('https://api.stripe.com')
  .get('/v1/invoices/in_draft123')
  .times(3)
  .reply(500, { error: { message: 'Internal error' } });

nock('https://api.stripe.com')
  .get('/v1/invoices/in_draft123')
  .reply(200, { id: 'in_draft123', status: 'open' });

Expected Flow:

1. First 3 attempts fail with 500 error
2. 5-second backoff between attempts
3. Fourth attempt succeeds
4. Invoice updated

5. Multiple Draft Invoices

Setup:

const invoices = await StoreInvoice.insertMany([
  { stripe_id: 'in_1', status: 'draft', in_progress: false },
  { stripe_id: 'in_2', status: 'draft', in_progress: false },
  { stripe_id: 'in_3', status: 'draft', in_progress: false },
]);

Expected Flow:

1. All three invoices detected in single query
2. Bulk in_progress: true update
3. Three queue jobs created concurrently
4. Each synced independently

---

Performance Considerations

Query Optimization

Index Requirements:

// StoreInvoice collection
{
    status: 1,
    in_progress: 1,
    updated_at: 1
}

Query Pattern: Compound index for draft invoice detection with stale lock recovery

Batch Processing

Concurrency: Parallel queue additions

await Promise.all(
  getDraftInvoices.map(async invoiceParams => {
    await Q.add(invoiceParams, { attempts: 5, backoff: 5000 });
  }),
);

Performance: Multiple invoices added to queue simultaneously

• Throughput: Limited by Bull queue concurrency
• Stripe API: Rate limits apply per request

Sync Frequency

Schedule: Every 12 hours

Rationale:

• Balance: Frequent enough to catch status changes
• API Efficiency: Avoids excessive Stripe API calls
• Cost: Stripe API is free but rate-limited

Alternative: Could trigger sync on webhook events for real-time updates

---

Monitoring & Logging

Log Patterns

Service Logs:

console.log('fetch from stripe Started');
console.error('Error while scheduling build: ', err.message, err.stack);

Processor Logs:

logger.error({
  initiator: 'QM/startup/QM_DRAFT_INVOICE',
  message: 'Complete status update on invoice',
  error: err,
  job: job.id,
  job_data: job.data,
});

Metrics to Monitor

1. Draft Invoice Count: StoreInvoice.countDocuments({ status: 'draft' })
2. Stale Locks: Count of in_progress: true older than 6 hours
3. Sync Success Rate: Completed jobs vs. failed jobs
4. Status Transitions: Count of draft → open/paid/void changes
5. Processing Time: Average job duration
6. Stripe API Errors: Rate of 4xx/5xx responses

Alerting Scenarios

• Growing Draft Count: Invoices not finalizing
• High Stale Lock Count: Frequent process crashes
• Stripe API Errors: Authentication or rate limit issues
• Long Sync Duration: Performance degradation
• Failed Updates: Database connectivity problems

---

Related Documentation

• Store Invoice Pending Subaccount (link removed - file does not exist) - Subaccount invoice charging
• Store Charge Main Account (link removed - file does not exist) - Main account charging
• Store Subscriptions Cancel (link removed - file does not exist) - Subscription cancellation
• Common Billing Utilities (link removed - file does not exist) - Shared billing utilities

---

Summary

The Draft Invoices Synchronization module ensures MongoDB invoice records remain synchronized with Stripe's source of truth. Its 12-hour sync schedule with stale lock recovery provides reliable data consistency while minimizing API calls. The selective update strategy only modifies invoices that have transitioned away from draft status, reducing unnecessary database writes.

Key Strengths:

• Automatic Sync: No manual intervention required
• Stale Lock Recovery: 6-hour timeout prevents permanent locks
• Selective Updates: Only writes when status changes
• Error Handling: 5 retries with fixed backoff
• Bulk Processing: Handles multiple draft invoices efficiently

Critical for:

• Billing data accuracy
• Invoice status tracking
• Draft finalization detection
• Stripe-MongoDB synchronization