📊 InstaReports Processing

📖 Overview

The InstaReports module handles automated report generation including SEO reports, analytics dashboards, and client-facing performance reports.

Environment Flag: QM_INSTAREPORT_BUILD=true

Source Location: queue-manager/crons/instareports/

Processing Pattern: High-frequency polling (every 30 seconds)

🗄️ Collections Used

instareports-queue

• Operations: Read/Write
• Model: shared/models/instareports-queue.js
• Usage: Queue report build jobs

instareports

• Operations: Read/Write
• Model: shared/models/instareports.js
• Usage: Store report configurations and results

instareports-data

• Operations: Create/Update
• Model: shared/models/instareports-data.js
• Usage: Store report metrics and historical data

🔄 Processing Flow

sequenceDiagram
    participant API as Internal API
    participant QUEUE as instareports-queue
    participant CRON as Build Cron (30s)
    participant SERVICE as Build Service
    participant EXTERNAL as External APIs
    participant DB as MongoDB
    participant S3 as File Storage

    API->>QUEUE: Create build job<br/>(status: 'pending')

    loop Every 30 seconds
        CRON->>SERVICE: Execute
        SERVICE->>QUEUE: Query pending builds
        QUEUE-->>SERVICE: Pending jobs
        SERVICE->>EXTERNAL: Fetch analytics data<br/>(Google Analytics, SEMrush, etc.)
        EXTERNAL-->>SERVICE: Metrics data
        SERVICE->>SERVICE: Generate report HTML/PDF
        SERVICE->>S3: Upload report files
        SERVICE->>DB: Store report data
        SERVICE->>QUEUE: Update status<br/>(status: 'completed')
    end

🔧 Job: Build Processing

Overview

Generates InstaReports by fetching data from multiple sources and compiling into client-facing reports.

Cron Schedule: */30 * * * * * (Every 30 seconds)

Source Files:

• Cron: queue-manager/crons/instareports/build.js
• Service: queue-manager/services/instareports/build.js
• Queue: queue-manager/queues/instareports/build.js

Key Features

• Multi-Source Data: Aggregates from Google Analytics, SEMrush, CallRail, etc.
• Template System: Customizable report templates
• PDF Generation: Puppeteer-based PDF rendering
• Data Caching: Caches metrics to reduce API calls
• Scheduled Reports: Automatic monthly/weekly generation
• White Label: Client-branded reports

Report Types

• SEO Reports: Rankings, backlinks, technical SEO
• Analytics Reports: Traffic, conversions, user behavior
• PPC Reports: Ad performance, ROI, keyword metrics
• Social Reports: Engagement, reach, demographics
• Call Tracking: Call analytics and recordings

Processing Steps

1. Queue Query: Check for pending report builds
2. Data Collection: Fetch from all configured data sources
3. Data Processing: Calculate metrics, trends, comparisons
4. Template Rendering: Apply data to HTML template
5. PDF Generation: Convert to PDF with Puppeteer
6. Storage: Upload to S3 and store metadata
7. Notification: Trigger completion email to client

Code Pattern

Cron Initialization (crons/instareports/build.js):

const { scheduleBuild } = require('../../services/instareports');
const cron = require('node-cron');

let inProgress = false;
exports.start = async () => {
  cron.schedule('*/30 * * * * *', async () => {
    if (!inProgress) {
      inProgress = true;
      await scheduleBuild();
      inProgress = false;
    }
  });
};

Data Source Integration

Supported Integrations:

• Google Analytics (GA4, Universal Analytics)
• Google Search Console
• SEMrush
• CallRail
• Facebook Ads
• Google Ads
• Mailchimp

API Rate Limiting:

• Implements exponential backoff
• Caches data with configurable TTL
• Batch requests where possible

Error Handling

Partial Success:

• Report generates with available data
• Missing sections noted in report
• Retry scheduled for failed data sources

Complete Failure:

• Job marked as failed
• Error details stored in queue record
• Admin notification sent

⚙️ Configuration

Required Environment Variables:

QM_INSTAREPORT_BUILD=true          # Enable report building

# Data Source API Keys
GOOGLE_ANALYTICS_KEY=...
SEMRUSH_API_KEY=...
CALLRAIL_API_KEY=...

# Storage
AWS_S3_BUCKET=...
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...

# Dependencies
MONGO_DB_URL=...
REDIS_HOST=localhost
REDIS_PORT=6379

🎨 Template System

Template Structure

Reports use Handlebars templates with:

• Header/Footer: Client branding, logo, date
• Summary Section: Key metrics overview
• Detailed Sections: Per-channel breakdowns
• Charts: Generated with Chart.js
• Tables: Sortable data tables

Customization

• Per-Client Templates: Custom branding and layouts
• White Label: Agency branding override
• Section Toggle: Enable/disable report sections
• Metric Selection: Choose displayed metrics

📊 Performance Optimization

Caching Strategy

• API Responses: 1-hour cache for analytics data
• Report Assets: CDN caching for images, fonts
• Partial Builds: Reuse unchanged sections

Concurrency

• Multiple reports can build simultaneously
• Queue concurrency: 5 concurrent builds
• Resource limits prevent memory overflow

🔗 Related Documentation

• Queue Manager Overview
• Architecture (documentation unavailable)
• Configuration

---

Module Status: Active
Execution Pattern: High-frequency polling
Last Updated: 2025-10-10