📦 InstaSites Module

🎯 Overview

The InstaSites module powers DashClicks' rapid website fulfillment pipeline. It exposes internal REST endpoints used by the dashboard and queue manager to create InstaSites from CRM contacts, manage template catalogs, trigger Duda publishes/unpublishes, send launch notifications, and surface reporting. All business logic lives under internal/api/v1/instasites, with controllers delegating to services and shared utilities.

📁 Directory Structure

internal/api/v1/instasites/
├── README.md
├── controllers/
│   ├── categories.controller.js      # GET categories and their template counts
│   ├── index.js                      # Exports controller functions
│   ├── is-template-import.js         # Seed data used once for bulk template import
│   ├── reporting.controller.js       # Reporting endpoints (status, industries, logs)
│   └── templates.controller.js       # Template list/create handlers
├── routes/
│   ├── categories.route.js           # /categories wiring
│   ├── index.js                      # Primary router mounting sub-routes
│   ├── reporting.route.js            # /reporting wiring
│   └── templates.route.js            # /templates wiring
├── services/
│   ├── categories.service.js         # Category aggregation logic
│   ├── index.js                      # Core InstaSites operations (queue, Duda, notifications)
│   ├── reporting.service.js          # KPI aggregation helpers
│   └── templates.service.js          # Template fetch & creation logic
├── tests/
│   ├── _constants.js
│   ├── categories.test.js
│   ├── instasites.test.js
│   ├── reporting.test.js
│   ├── templates.test.js
│   └── __mocks__/notifications.mock.js
├── utils/
│   ├── create-business.js            # Contact + business creation for CSV imports
│   ├── cvs-to-json.js                # Wasabi CSV parsing helper
│   ├── generateFileBuffer.js         # Fetches Wasabi objects as mail attachments
│   ├── saveQueueData.js              # Background CSV ingestion worker
│   ├── sendNotifications.js          # Email/SMS blast helper consumed by queue manager
│   └── templates.js                  # Puppeteer screenshot + Wasabi upload utilities
└── validations/
    ├── custom.validation.js
    ├── instasite.validation.js       # Queue, analytics, SSO, association validators
    └── templates.validation.js       # Template filters + create payload

🗄️ MongoDB Collections

📚 Full schema specs live in Database Collections Documentation

Primary

• instasites (shared/models/instasite.js) — canonical site record, including builder IDs, publish state, certificate status, and cached Duda content metadata.
• instasites.additional_information (shared/models/instasites-additional-information.js) — per-site notification metadata, recipient lists, email/SMS event IDs, and lifecycle timestamps.
• queue (shared/models/instasite-queue.js) — generalized queue collection; InstaSites writes type: instasite, instasite-csv, and instasite-notifications jobs here.

Supporting collections

• instasites.templates (shared/models/instasite-template.js) — template definitions, preview assets, global/account ownership.
• instasites.templates.categories (shared/models/instasite-template-category.js) — main/sub category taxonomy powering filters.
• crm.contacts — source of businesses/people used to build InstaSites and enforce visibility rules.
• communications — mail/SMS delivery logs used to update resend metadata.
• agency-websites — agency-owned Duda sites included in “active sites” responses.
• _store.products — pricing references for bundling Stripe price IDs alongside template returns.
• _users — lookups for creator details when listing InstaSites.

🏗️ Architecture Overview

Key responsibilities

• Ingest CRM contacts and queue site builds (services/index.js#addToQueue).
• Orchestrate CSV imports into queued builds (addCSVToQueue, saveQueueData).
• Maintain template catalog and categories derived from Duda (templates.service.js, categories.service.js).
• Proxy critical Duda operations (publish, unpublish, plan changes, analytics, SSO) with scoped auth.
• Broadcast launch notifications via the notifications queue (processNotificationQueue, sendNotifications).
• Provide reporting endpoints for build/view/purchase funnels (reporting.service.js).

Design decisions

• Queue-first fulfillment. Site creation is always asynchronous via queue records; synchronous work stops once records are enqueued.
• Contact visibility filtering. Aggregations respect account preferences (owner_followers, entire) to shield restricted contacts when listing sites or statuses.
• Duda caching. Site fetches refresh content from Duda only when a cached refresh_at timestamp expires to avoid hammering the API.
• Shared utilities. Attachments, template thumbnails, and CSV imports delegate to reusable utilities (generateFileBuffer, templates.js, create-business.js) that are shared with queue workers.

🔗 Submodules

Core services

• 📘 Core InstaSites Service — Queuing, notification, Duda publish, analytics, and record retrieval.
• 📗 Template Service — Template listing and creation via Duda, Stripe pricing hydration.

Supporting services

• 📘 Category Service — Category/main-category aggregations and counts.
• 📗 Reporting Service — Daily funnel metrics, industry breakdowns, and CSV import logs.

🔄 Data Flow

flowchart TD
    A["Client request<br/>(REST endpoint)"] --> B{"Validation layer<br/>(Joi schemas)"}
    B -->|valid| C["Service logic"]
    C --> D["Queue mutations<br/>(queue collection)"]
    C --> E[("Instasite docs<br/>+ metadata")]
    D --> F["Queue Manager workers<br/>(saveQueueData & Duda builders)"]
    F --> G["Duda API<br/>publish/forms/analytics"]
    F --> H["Notifications<br/>(Mail/SMS)"]
    style A fill:#e1f5ff
    style G fill:#fff4d2
    style H fill:#ffe9e9

🔐 Configuration & Dependencies

Variable	Description	Required
DUDA_TOKEN	Builder token forwarded to Duda internal API calls and queue jobs	✅
DUDA_INTERNAL_ENDPOINT	Base URL for the internal Duda proxy (e.g. https://duda-proxy/...)	✅
APP_SECRET	Signing key for short-lived JWTs issued in processNotificationQueue	✅
SHORT_URL_SERVICE	URL shortener endpoint used in email/SMS notifications	✅ (for notifications)
SHORT_DOMAIN	Public domain paired with the shortener response code	✅
WASABI_PUBLIC_IMAGE_DOWNLOAD	Base URL to fetch Wasabi-hosted attachments	✅ (when attachments enabled)
WASABI_*	Credentials + region consumed by utils/templates.js for thumbnail capture	❌ (only when capturing thumbnails)

🚀 Quick Start

Typical workflow for an internal consumer:

1. List candidate contacts the dashboard selects as businesses.
2. POST /instasites/queue with business IDs, template ID, and notification payload — queues one job per business.
3. Queue manager picks up type: instasite entries, builds via Duda, sends notifications, and updates InstasitesAdditionalInformation.
4. Publish/unpublish via PUT /instasites/:id/publish or /unpublish once the client approves.
5. Monitor status through /instasites/reporting/status and /instasites/reporting/logs.

All Read endpoints return success, message, and module-specific payloads consistent with other internal services.

🔄 Background Jobs

Queue Type	Producer	Worker	Notes
instasite	addToQueue & CSV worker	Queue Manager (Duda builder)	Builds sites, updates instasites + emits notifications
instasite-csv	addCSVToQueue	saveQueueData (within this module)	Parses uploaded CSV, creates contacts, and enqueues instasite jobs
instasite-notifications	Queue Manager when notifications need retry	processNotificationQueue	Retries Mail/SMS with 5-attempt cap

🧪 Testing Notes

• Unit tests reside in internal/api/v1/instasites/tests. They mock notification delivery and assert controller/service paths.
• Use pnpm test --filter instasites (workspace-aware) to run module-specific suites.

⚠️ Important Notes

• 🚨 Contact visibility enforcement: All aggregations that surface contact-bound data must call getStatuses or getInstasites helpers to keep logic consistent with account preference rules.
• 💡 Refresh cadence: getSiteById refreshes from Duda hourly; avoid redundant manual refreshes unless refresh_at is overdue.
• 🐛 Notification queue: processNotificationQueue increments tries before ensuring a queue item exists. Be mindful when manually inserting jobs.

🔗 Related Documentation

• Queue Manager Overview
• Notifications Service
• Shared Utilities — logger, mail, sms, pagination helpers