Internal API Service

The Internal API Service is the core business logic engine of the DashClicks platform, handling all proprietary features and internal operations. It serves as the primary service for account management, CRM, billing, content creation, and all platform-specific functionality.

Service Overview

🏗️ Architecture

• Service Name: Internal API
• Port: 5002
• Entry Point: internal/app.js
• API Version: v1 (/v1/* routes)
• Debug Configuration: "API Internal" in VS Code launch.json

🔄 Request Flow

Frontend → Dashboard Gateway (5000) → API Router (5001) → Internal API (5002)

All requests to /v1/* routes are proxied to this service by the API Router.

🔗 Special Route Handlers

GET /:code       # URL shortener redirect handler
GET /v1/:code     # Affiliate redirect handler
GET /            # Basic health check
GET /status      # Service status check

📁 Directory Structure

internal/
├── app.js                    # Express application entry point
├── package.json              # Service dependencies
├── jest.config.js           # Jest testing configuration
├── jest.setup.js            # Jest setup configuration
├── jest.setup.after.env.js  # Jest environment setup
├── Dockerfile               # Docker container configuration
├── README.md                # Service documentation
├── .dockerignore            # Docker ignore patterns
└── api/
    ├── v1/                  # API version 1 implementation
    │   ├── index.js         # Main API router with middleware
    │   ├── models/          # Copied from /shared/models (Git-ignored)
    │   ├── utilities/       # Copied from /shared/utilities (Git-ignored)
    │   ├── __mocks__/       # Jest mock files (e.g., stripe.js)
    │   └── [24 modules]/    # Business logic modules
    └── v2/                  # API version 2 (minimal structure)
        ├── models/          # Copied from /shared/models (Git-ignored)
        └── utilities/       # Copied from /shared/utilities (Git-ignored)

Core Business Modules

The Internal API contains 24 specialized modules handling different aspects of the DashClicks platform:

🏢 Account & User Management

• accounts/ - Multi-tenant account management, domain configuration, and billing setup
• users/ - User authentication, profiles, permissions, and workspace management
• auth/ - JWT authentication, session management, and OAuth integrations

👥 Customer Relationship Management

• crm/ - Complete contact management, deal tracking, and pipeline automation
• activities/ - User activity tracking, audit logs, and engagement analytics
• forms/ - Lead capture forms, form builders, and submission handling
• inbound/ - Lead capture automation and inbound marketing workflows

💰 Sales & Revenue Operations

• billing/ - Subscription management, invoicing, and payment processing
• store/ - White-label marketplace for digital marketing products
• onebalance/ - Unified billing and credit management system
• affiliates/ - Partner and affiliate program management

🌐 Website & Content Creation

• sites/ - Agency website builder and hosting platform (route: /agencywebsites)
• instasites/ - Automated website generation and management
• instareports/ - Automated reporting and analytics generation
• templates/ - Marketing template management and customization
• funnels/ - Sales funnel creation and conversion tracking

💬 Communication & Collaboration

• conversations/ - Legacy messaging system (v1)
• conversation-v2/ - Enhanced real-time messaging system (route: /conv)
• notifications-center/ - Centralized notification management (route: /notifications)

🛠️ Project & Task Management

• projects/ - Task management and project collaboration tools
• reviews/ - Online reputation management and review monitoring (route: /reputation)
• url-shortener/ - Branded link shortening and click tracking (route: /url)

🔧 System & Integration Services

• webhooks/ - External service webhook handling and processing
• oauth/ - Third-party application authentication (route: /auth/oauth)
• cors-proxy/ - Cross-origin request handling (route: /corsproxy)
• filters/ - Data filtering and search functionality
• shared/ - Shared components and utilities
• public/ - Public-facing content and landing pages
• mobile-app/ - Mobile application API endpoints

🔐 Administrative Functions

• admin/ - System administration and configuration management (DashClicks internal use only)

Authentication & Authorization

🔑 JWT Token Authentication

The Internal API uses JWT-based authentication with Bearer tokens:

// All requests require Authorization header
Authorization: Bearer <
  jwt_token >
  // Token contains user context:
  {
    userId: 'user_id',
    accountId: 'account_id',
    permissions: ['read', 'write', 'admin'],
    exp: timestamp,
  };

🛡️ Permission Levels

• User: Basic access to own data and assigned resources
• Admin: Account-level administration and management
• Super Admin: Platform-level administration (DashClicks only)
• API Access: Programmatic access with scoped permissions

🔒 Route Protection

// Middleware chain from /api/v1/index.js
router.use('/accounts', validateLimit, _accounts);

router.use(
  '/crm',
  verifyAuthorization(), // JWT verification
  verifyAccessAndStatus({ accountIDRequired: true }), // Account access validation
  validateLimit, // Rate limiting
  _crm,
);

// Special route handling
router.use('/auth/oauth', disableLog, _oauth); // OAuth with logging disabled
router.use('/auth', disableLog, _auth); // Auth with logging disabled

// Global middleware chain for all /v1/* routes:
// 1. setStartTime - Request timing
// 2. utilities.validateRequestMethod - HTTP method validation
// 3. Request/response logging middleware

Database Integration

🗄️ MongoDB Connection

The service connects to MongoDB using shared utilities:

const { connect: connectMongo } = require('./api/v1/utilities/db');

// Connection initialized in app.js
const mongoURL = process.env.MONGO_DB_URL;
await connectMongo({
  uri: mongoURL,
  initiator: 'internal',
});

📊 Data Models

Uses 150+ Mongoose models from /shared/models/:

• Core Models: Account, User, Contact, Deal, Site, Project
• Integration Models: Service-specific tokens and keys
• Analytics Models: Campaign data, performance metrics
• Communication Models: Messages, notifications, conversations
• Billing Models: Subscriptions, invoices, payments

Service Communication

🔗 Inter-Service Communication

The Internal API communicates with other services:

// External API calls
const externalAPI = 'http://localhost:5003';
const response = await axios.get(`${externalAPI}/v1/e/google-ads/campaigns`);

// Socket services
const socketAPI = 'http://localhost:4000';
await axios.post(`${socketAPI}/emit`, { event: 'account_updated', data });

// Queue Manager
const queueAPI = 'http://localhost:6002';
await axios.post(`${queueAPI}/queue/billing`, { action: 'process_subscription' });

🌐 External Service Integration

Coordinates with external services through the External API:

• Google Services: Ads, Analytics, Business profiles
• Facebook/Meta: Advertising and social media
• Payment Processing: Stripe, PayPal integrations
• Communication: Twilio, SendGrid, email providers
• CRM Systems: Salesforce, HubSpot, Pipedrive

Development Workflow

🚀 Running the Service

Always use VS Code debugger - never run directly:

# In VS Code:
# 1. Go to Debug Panel (Ctrl+Shift+D)
# 2. Select "API Internal"
# 3. Press F5 or click green play button
# Service starts on http://localhost:5002

🔧 Development Commands

# Install dependencies
cd internal/
npm install

# Run tests
npm test

# Run specific test suites
npm test -- --testPathPattern=accounts
npm test -- --testPathPattern=crm

# Start development server
npm run dev        # Uses nodemon for auto-restart
npm start         # Production start

# Service available at http://localhost:5002

📝 Adding New Endpoints

Module structure varies by implementation pattern:

// Pattern 1: Lowercase folders (newer modules)
// internal/api/v1/[module]/controllers/
// internal/api/v1/[module]/routes/
// internal/api/v1/[module]/services/
// internal/api/v1/[module]/validations/ or schemas/
// internal/api/v1/[module]/utils/
// internal/api/v1/[module]/tests/

// Pattern 2: Capitalized folders (legacy modules)
// internal/api/v1/[module]/Controllers/
// internal/api/v1/[module]/Routes/
// internal/api/v1/[module]/Models/
// internal/api/v1/[module]/Utils/ or utilities/
// internal/api/v1/[module]/validators/

// Steps to add new endpoint:
// 1. Add route handler in routes/ or Routes/
// 2. Implement controller logic in controllers/ or Controllers/
// 3. Add validation in validations/, schemas/, or validators/
// 4. Add business logic in services/ (if available)
// 5. Export route in module index.js
// 6. Register in main /api/v1/index.js with middleware

🧪 Testing Strategy

Tests are located in module-specific tests/ folders:

// Jest configuration files:
// internal/jest.config.js           # Main Jest configuration
// internal/jest.setup.js            # Global test setup
// internal/jest.setup.after.env.js  # Environment-specific setup

// Module test structure:
// internal/api/v1/[module]/tests/   # Module-specific tests
//   ├── controllers/                # Controller unit tests
//   ├── routes/                     # Route integration tests
//   ├── services/                   # Business logic tests
//   └── utils/                      # Utility function tests

// Mock dependencies available:
// internal/api/v1/__mocks__/stripe.js  # Stripe service mocks

API Endpoints Overview

🔄 Common Patterns

Most modules follow RESTful patterns:

GET    /v1/[module]           # List resources
GET    /v1/[module]/:id       # Get specific resource
POST   /v1/[module]           # Create new resource
PUT    /v1/[module]/:id       # Update resource
DELETE /v1/[module]/:id       # Delete resource

# Advanced patterns
GET    /v1/[module]/search    # Search resources
POST   /v1/[module]/bulk      # Bulk operations
GET    /v1/[module]/export    # Export data
POST   /v1/[module]/import    # Import data

📊 Response Format

Standard API response structure:

{
  "success": true,
  "data": {
    /* resource data */
  },
  "message": "Resource retrieved successfully",
  "pagination": {
    "page": 1,
    "limit": 20,
    "total": 150,
    "totalPages": 8
  },
  "meta": {
    "timestamp": "2025-10-03T10:30:00Z",
    "version": "v1"
  }
}

❌ Error Handling

Centralized error handling middleware in app.js:

// Standard error response format
{
  "success": false,
  "errno": 400, // HTTP status code
  "message": "Error description",
  "additional_info": "Detailed error information",
  "data": null // Optional error-specific data
}

// Special handling for MongoDB duplicate key errors:
// E11000 duplicate key -> "Duplicate document."

// Axios error handling:
// - Uses response status and data from external API calls
// - Strips sensitive config information

// Global error types handled:
// - Validation errors (Joi)
// - MongoDB duplicate key errors
// - Axios HTTP errors
// - Unhandled promise rejections
// - Uncaught exceptions
// - SIGTERM signals for graceful shutdown

Performance & Optimization

⚡ Caching Strategy

• Redis Integration: Session caching and temporary data
• Database Indexing: Optimized queries with proper indexes
• Connection Pooling: Efficient database connection management
• Response Caching: Cached responses for frequently accessed data

📈 Monitoring & Metrics

• Health Checks: /v1/health endpoint for service monitoring
• Performance Logging: Request/response time tracking
• Error Tracking: Comprehensive error logging and alerting
• Resource Monitoring: Memory and CPU usage tracking

🔍 Debugging Features

• Request Logging: Detailed request/response logging
• Error Stack Traces: Full error context in development
• Database Query Logging: MongoDB query performance monitoring
• Service Communication Logging: Inter-service call tracking

Configuration & Environment

🔧 Environment Variables

Key configuration variables for the Internal API:

# Database
MONGO_DB_URL=mongodb://localhost:27017/dashclicks

# Service Configuration
PORT=5002                    # Service port (default: 5002)

# Authentication & Security
JWT_SECRET=your_jwt_secret_key
JWT_EXPIRES_IN=24h

# Third-party Services (from package.json dependencies)
# Stripe integration
STRIPE_SECRET_KEY=sk_test_...

# Twilio communications
TWILIO_ACCOUNT_SID=AC...
TWILIO_AUTH_TOKEN=...

# SendGrid email
SENDGRID_API_KEY=SG...

# AWS services
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...

# Google Cloud services
GOOGLE_CLOUD_PROJECT_ID=...

# Firebase Admin SDK
FIREBASE_ADMIN_SDK_JSON=...

📦 Key Dependencies

From package.json - major integrations and frameworks:

{
  "express": "^4.17.1", // Web framework
  "mongoose": "^8.18.0", // MongoDB ODM
  "joi": "^17.2.1", // Input validation
  "jsonwebtoken": "^8.5.1", // JWT authentication
  "stripe": "^10.0.0", // Payment processing
  "twilio": "^4.21.0", // SMS/Voice communications
  "aws-sdk": "^2.1031.0", // AWS integrations
  "@sendgrid/client": "^7.6.0", // Email services
  "firebase-admin": "^12.7.0", // Firebase integration
  "axios": "^0.21.1", // HTTP client
  "moment": "^2.29.1", // Date/time handling
  "lodash": "^4.17.21", // Utility functions
  "multer": "^1.4.2", // File upload handling
  "puppeteer": "^19.4.0" // Browser automation (dev)
}

🔒 Security Configuration

• CORS Settings: Configurable cross-origin request handling
• Rate Limiting: API request rate limiting and throttling
• Input Validation: Comprehensive request validation
• SQL Injection Protection: Parameterized queries and input sanitization

Deployment & Scaling

🐳 Docker Configuration

# internal/Dockerfile
FROM node:18-alpine
WORKDIR /internal
COPY package.json /internal

RUN npm install

COPY . /internal
CMD node app.js
EXPOSE 8080

Note: Container exposes port 8080, but service runs on port 5002 (configurable via PORT environment variable).

🔄 Health Monitoring

// Basic health check endpoints
GET /                 # Returns: { "status": "ok" }
GET /status          # Returns: { "status": "ok" }

// Dashboard version header
// All responses include X-Dashboard-Version header
// Retrieved from DashboardMeta collection (type: 'version')

📊 Load Balancing

• Horizontal Scaling: Multiple service instances
• Database Connection Pooling: Efficient resource utilization
• Stateless Design: No server-side session storage
• Microservice Architecture: Independent scaling per module

Module Documentation

Each module has detailed documentation covering:

• Architecture & Design: Module structure and patterns
• API Endpoints: Complete endpoint documentation
• Database Models: Data structures and relationships
• Business Logic: Core functionality and workflows
• Integration Points: External service connections
• Testing Strategy: Unit and integration test coverage
• Configuration: Module-specific settings
• Troubleshooting: Common issues and solutions

📚 Available Module Docs

Click on any module below for detailed documentation:

Core Business Modules

• accounts/ - Account management and configuration
• users/ - User authentication and profiles
• auth/ - Authentication and authorization
• crm/ - Customer relationship management
• billing/ - Subscription and payment processing

Content & Website Management

• sites/ - Agency website builder and hosting
• instasites/ - Automated website generation
• instareports/ - Automated reporting
• templates/ - Marketing template management
• funnels/ - Sales funnel creation

Communication & Collaboration

• conversations/ - Legacy messaging system (v1)
• conversation-v2/ - Enhanced messaging (v2)
• forms/ - Lead capture and form management
• notifications-center/ - Notification management

Sales & Marketing

• store/ - Marketplace and e-commerce
• affiliates/ - Partner program management
• reviews/ - Reputation management
• onebalance/ - Unified billing system

System & Integration

• projects/ - Project and task management
• activities/ - User activity tracking
• webhooks/ - Webhook processing
• filters/ - Data filtering and search
• url-shortener/ - Link shortening
• oauth/ - Third-party authentication
• cors-proxy/ - Cross-origin request handling
• inbound/ - Lead capture automation
• shared/ - Shared components
• public/ - Public-facing content
• mobile-app/ - Mobile API endpoints
• admin/ - System administration

Complete coverage of all 24 internal modules available.

---

Support & Resources

🆘 Getting Help

• Documentation: Complete module docs in /docs/service/internal/
• Code Examples: Reference implementations in each module
• Testing: Comprehensive test suites for guidance
• Debugging: VS Code debugger with breakpoint support

🔗 Related Services

• External API - Third-party integrations
• API Router - Request routing and proxying
• Queue Manager - Background job processing
• Socket Services - Real-time communication

📝 Development Guidelines

• Always use VS Code debugger for service development
• Follow shared code management rules (never edit service-level models/utilities)
• Implement comprehensive error handling and validation
• Write tests for new features and endpoints
• Document API changes and new functionality
• Use consistent response formats across modules