🔗 Shared Resources

📖 Overview

The Shared folder contains centralized Mongoose models and utility functions used across all DashClicks backend services (Internal API, External API, Queue Manager, Gateways, Socket services).

Key Characteristics:

• 207 Mongoose Models: Database schemas for all MongoDB collections
• 65+ Utilities: Common functions for auth, validation, communication, payments, etc.
• 4 Middleware Functions: Reusable Express middleware for access control and OAuth
• Copied to Services: Automatically distributed to all services via build process
• Single Source of Truth: Edit only in /shared/, changes propagate automatically

Distribution Method: npm run copySharedFiles copies to:

• internal/api/v1/models/ and internal/api/v1/utilities/
• external/Integrations/models/ and external/Integrations/utilities/
• queue-manager/models/ and queue-manager/utilities/
• conversation-socket/models/ and conversation-socket/utilities/
• general-socket/models/ and general-socket/utilities/
• dashboard-gateway/models/ and dashboard-gateway/utilities/
• tests/models/ and tests/utilities/

⚠️ CRITICAL: Do Not Edit Service Copies

Service-level folders are Git-ignored!

# In .gitignore
*/models/       # All models folders in services
*/utilities/    # All utilities folders in services

❌ NEVER EDIT:

• internal/api/v1/models/user.js
• queue-manager/utilities/auth.js
• Any models/ or utilities/ folder inside services

✅ ALWAYS EDIT:

• shared/models/user.js
• shared/utilities/auth.js
• Then run npm run copySharedFiles

Why: Service copies are regenerated on every build and are excluded from version control.

📂 Folder Structure

shared/
├── 📁 models/                      # 207 Mongoose schemas
│   ├── 📄 _globalSchemaOptions.js  # Global schema configuration
│   ├── 📄 index.js                 # Model exports
│   ├── 📄 toJSON.js                # JSON transformation plugin
│   ├── 📄 [Account models]         # 6 files: account, account-user, etc.
│   ├── 📄 [API/OAuth models]       # 5 files: api-session, api-grant, etc.
│   ├── 📄 [Billing models]         # 7 files: billing-*, stripe-subscription
│   ├── 📄 [CRM models]             # 9 files: contact, deal, pipeline, etc.
│   ├── 📄 [Conversation models]    # 11 files: conversation-*, message, etc.
│   ├── 📄 [Store/E-commerce]       # 14 files: store-*, consumer-customer
│   ├── 📄 [Funnel models]          # 11 files: funnel*, funnels.*
│   ├── 📄 [InstaReports models]    # 4 files: instareports*
│   ├── 📄 [InstaSites models]      # 5 files: instasite*
│   ├── 📄 [Sites models]           # 1 file: sites-images
│   ├── 📄 [Forms models]           # 7 files: forms*
│   ├── 📄 [eDocuments models]      # 14 files: edocs-*
│   ├── 📄 [Calendar/Scheduling]    # 16 files: event-type*, google-calendar*, etc.
│   ├── 📄 [Analytics models]       # 9 files: analytics-*, campaign-data, semrush
│   ├── 📄 [Reviews models]         # 9 files: review*, reviews*
│   ├── 📄 [Projects models]        # 6 files: projects-*
│   ├── 📄 [Integration Keys]       # 25 files: *-key.js, *-token.js
│   ├── 📄 [Lead Finder models]     # 13 files: lead-finder-*
│   ├── 📄 [Support models]         # 7 files: support.*
│   ├── 📄 [Queue models]           # 9 files: *-queue.js, queues.js
│   ├── 📄 [Webhook models]         # 5 files: webhook*, *-webhooks.js
│   ├── 📄 [Notification models]    # 5 files: notification-*, fcm-*, meeting-*
│   ├── 📄 [Affiliate models]       # 3 files: affiliate-*, affiliates-*
│   ├── 📄 [OneBalance models]      # 3 files: onebalance*
│   └── 📄 [Miscellaneous]          # 20+ files: activity, automation, etc.
└── 📁 utilities/                   # 65+ helper functions
    ├── 📄 [Core utilities]         # 12 files: auth, db, logger, request, etc.
    ├── 📄 [Communication]          # 7 files: mail, sms, webhooks, fcm, etc.
    ├── 📄 [Validation/Error]       # 5 files: validate, api-error, catch-*, etc.
    ├── 📄 [Business Logic]         # 15 files: deal, contact, pipeline, etc.
    ├── 📄 [Integration helpers]    # 8 files: stripe, semrush-auth, etc.
    ├── 📄 [Domain/URL utilities]   # 5 files: domains, cleanupDomain, etc.
    ├── 📄 [Data processing]        # 8 files: filter, object, pick, etc.
    └── 📁 middlewares/             # 4 Express middlewares
        ├── conversation.support.js
        ├── onebalance.js
        ├── verify-access-and-status.middleware.js
        └── verifyScope.js

📚 Documentation Structure

This documentation is organized into focused sections for easy navigation:

📦 Models

Complete documentation of all 207 Mongoose models:

• Model Architecture Principles: Flexible schemas, JSON transformation, rich methods
• 24 Model Categories: Account, CRM, Store, Funnels, Analytics, etc.
• Model Relationships & Data Flow: Entity relationships, data patterns
• Usage Examples: Real-world code samples

Key Topics:

• Account Management (6 models)
• API & OAuth (5 models)
• Billing & Payments (7 models)
• CRM (9 models)
• Conversations & Messaging (11 models)
• Store & E-Commerce (14 models)
• Funnels, Forms, Reports, Sites
• Integration Keys (25 models)
• And 15+ more categories

🔧 Utilities

Comprehensive guide to 65+ shared utility functions:

• 12 Core Utilities: auth, db, logger, request, transactions
• Communication Utilities: mail, SMS, webhooks, push notifications
• Business Logic: CRM, deals, contacts, projects
• Integration Helpers: Stripe, SEMrush, Wasabi
• Advanced Patterns: Template system, credit management

Detailed Coverage:

• auth.js - Password hashing, session verification
• db.js - Connection management, pooling, events
• mail.js - Email sending with conversation integration
• catch-async.js - Error handling wrapper
• with-transaction.js - MongoDB transactions

🛡️ Middlewares

Deep dive into 4 critical Express middlewares:

• conversation.support.js: Support conversation access control
• onebalance.js: Credit system validation
• verify-access-and-status.middleware.js: Authentication & account status
• verifyScope.js: OAuth scope validation with derived scopes

Includes:

• Usage patterns and examples
• Configuration options
• Error responses
• Security patterns

🔄 Shared File Copy Process

Build Process

npm run build
# Installs service dependencies
# Calls: npm run copySharedFiles

Manual Copy

npm run copySharedFiles
# Copies shared/models/ → service/models/
# Copies shared/utilities/ → service/utilities/

Target Services

shared/
  ├── models/
  └── utilities/
        ↓
        ↓ npm run copySharedFiles
        ↓
  ┌─────┴─────┬─────────┬──────────┬──────────┬──────────┬──────────┐
  ↓           ↓         ↓          ↓          ↓          ↓          ↓
internal/ external/ queue-     conversation- general-  dashboard- tests/
api/v1/   Integrations/ manager/  socket/     socket/   gateway/

7 Services receive copied files:

1. internal/api/v1/models/ and internal/api/v1/utilities/
2. external/Integrations/models/ and external/Integrations/utilities/
3. queue-manager/models/ and queue-manager/utilities/
4. conversation-socket/models/ and conversation-socket/utilities/
5. general-socket/models/ and general-socket/utilities/
6. dashboard-gateway/models/ and dashboard-gateway/utilities/
7. tests/models/ and tests/utilities/

Git Status: All service-level models/ and utilities/ folders are in .gitignore

📐 Global Schema Options

File: shared/models/_globalSchemaOptions.js

Purpose: Consistent Mongoose schema configuration across all 207 models

Options:

module.exports = {
  timestamps: {
    createdAt: 'created_at', // Auto-generate created_at field
    updatedAt: 'updated_at', // Auto-generate updated_at field
  },
  toJSON: {
    virtuals: true, // Include virtual properties
    versionKey: false, // Exclude __v field
    transform: function (doc, ret, game) {
      delete ret._id; // Remove _id from JSON output
      delete ret.__v; // Remove __v from JSON output
    },
  },
  toObject: {
    virtuals: true, // Include virtual properties
    versionKey: false, // Exclude __v field
    transform: function (doc, ret, game) {
      delete ret._id; // Remove _id from object output
      delete ret.__v; // Remove __v from object output
    },
  },
};

Applied By: All models in shared/models/

Benefits:

• Consistent Timestamps: All models use created_at and updated_at (not createdAt/updatedAt)
• Clean JSON: Virtual id property replaces _id, no __v field
• Virtual Support: Mongoose virtuals included in JSON/Object serialization
• Immutable Pattern: created_at set once, updated_at auto-updates

🔗 Usage Patterns

Importing Models

// In any service (after copySharedFiles)
const Account = require('./models/account');
const User = require('./models/user');
const Contact = require('./models/contact');
const Deal = require('./models/deal');

// Query examples
const account = await Account.findById(accountId);
const users = await User.find({ account_id: accountId });
const deal = await Deal.findOne({ _id: dealId }).populate('contact_id');

Importing Utilities

// Core utilities
const { verifyToken } = require('./utilities/auth');
const logger = require('./utilities/logger');
const db = require('./utilities/db');

// Communication utilities
const { sendEmail } = require('./utilities/mail');
const { sendSMS } = require('./utilities/sms');
const { triggerWebhook } = require('./utilities/webhooks');

// Business logic utilities
const dealUtils = require('./utilities/deal');
const contactUtils = require('./utilities/contact');

// Validation and error handling
const { ApiError } = require('./utilities/api-error');
const catchAsync = require('./utilities/catch-async');

Using Middleware

// Import middlewares
const { verifyScope } = require('./utilities/middlewares/verifyScope');
const verifyAccessAndStatus = require('./utilities/middlewares/verify-access-and-status.middleware');
const oneBalanceMiddleware = require('./utilities/middlewares/onebalance');
const conversationSupportMiddleware = require('./utilities/middlewares/conversation.support');

// Apply to routes
router.post('/v1/accounts', verifyScope(['accounts:write']), verifyAccessAndStatus, controller);

router.post('/v1/credits-action', oneBalanceMiddleware({ cost: 1 }), controller);

router.get('/v1/support/conversations/:id', conversationSupportMiddleware, controller);

Database Transactions

const withTransaction = require('./utilities/with-transaction');

// Wrap operations in transaction
await withTransaction(async session => {
  await Account.updateOne({ _id: accountId }, { balance: 100 }, { session });
  await User.updateOne({ _id: userId }, { credits: 50 }, { session });
  // Both succeed or both rollback
});

🚨 Common Pitfalls

❌ Editing Service Copies

// WRONG - This will be overwritten!
// Editing: internal/api/v1/models/user.js
const User = mongoose.model('User', userSchema);
module.exports = User;

Problem: Changes lost on next build or copySharedFiles

✅ Editing Shared Source

// CORRECT - Edit the source
// Editing: shared/models/user.js
const User = mongoose.model('User', userSchema);
module.exports = User;

// Then run: npm run copySharedFiles

Result: Changes propagated to all 7 services

❌ Forgetting to Copy After Edits

# Made changes to shared/utilities/auth.js
vim shared/utilities/auth.js

# Started service without copying
npm start  # Services use OLD version!

Problem: Services still use old cached version

✅ Proper Workflow

# 1. Edit shared file
vim shared/utilities/auth.js

# 2. Copy to services
npm run copySharedFiles

# 3. Restart services (or use debugger auto-restart)
npm start

# 4. Test changes
npm test

❌ Committing Service Copies

git add internal/api/v1/models/user.js
git commit -m "Updated user model"
# Nothing committed - file is in .gitignore!

Problem: Service copies are Git-ignored, won't be tracked

✅ Committing Shared Source

git add shared/models/user.js
git commit -m "Updated user model"
# Committed successfully

Result: Changes tracked in version control

📊 Statistics

Models: 207 Mongoose schemas
Utilities: 65+ helper functions
Middlewares: 4 Express middlewares
Subdirectories:

• utilities/middlewares/ (4 files)
• utilities/country-codes/ (country data)
• utilities/fsCache/ (caching utilities)
• utilities/stripe/ (Stripe helpers)
• utilities/tests/ (utility tests)
• models/plugins/ (Mongoose plugins)
• models/node_modules/ (model dependencies)

Services Using: 7 backend services (Internal API, External API, Queue Manager, Conversation Socket, General Socket, Dashboard Gateway, Tests)
Total Distribution: 14 target folders (models + utilities × 7 services)
Lines of Code: ~50,000+ (estimated across all models and utilities)
Database Collections: 200+ MongoDB collections
Third-Party Integrations: 25+ (Google, Facebook, Stripe, Twilio, SendGrid, Duda, Yext, Hubspot, Salesforce, etc.)

🔗 Related Documentation

• Models Documentation - All 207 Mongoose models with relationships and data flow
• Utilities Documentation - 65+ helper functions with detailed examples
• Middlewares Documentation - 4 Express middlewares with usage patterns
• Internal API - Uses all shared models and utilities
• External API - Uses integration-related models and utilities
• Queue Manager - Uses queue models and background processing utilities
• Conversation Socket (link removed - file does not exist) - Uses conversation and messaging models
• General Socket - Uses socket utilities and notification models
• Dashboard Gateway - Uses session and authentication models
• Notifications Service - Uses notification and communication utilities

---

Resource Type: Shared Infrastructure
Status: ✅ Core - used by all services
Maintenance: Update via shared/ folder only, then run npm run copySharedFiles
Git Strategy: Only shared/ folder is version controlled; service copies are Git-ignored
Documentation Coverage: 207 models, 65+ utilities, 4 middlewares fully documented with examples
Last Updated: October 2025