RapidAPI SEO Integration

The RapidAPI integration provides Google Search SEO tracking and keyword ranking monitoring through the RapidAPI marketplace. This integration enables DashClicks users to track their website's search engine rankings for specific keywords across different locations and languages.

External Service

This integration uses the Google Search3 API from RapidAPI marketplace to perform search queries and track keyword rankings. An active RapidAPI subscription is required.

Architecture Overview

graph TB
    subgraph "Client Layer"
        A[Frontend Dashboard]
    end

    subgraph "DashClicks External API"
        B[RapidAPI Router]
        C[Config Controller]
        D[Rapid Controller]
        E[Rapid Provider]
    end

    subgraph "Database Layer"
        F[(Config Collection)]
        G[(SEO Results Collection)]
        H[(Slot Collection)]
    end

    subgraph "External Service"
        I[RapidAPI<br/>Google Search3]
    end

    subgraph "Background Processing"
        J[Cron Trigger]
        K[Queue Manager]
    end

    A -->|Configure SEO Tracking| B
    B --> C
    C --> F

    J -->|Daily Trigger| D
    D --> E
    E -->|Search Query| I
    I -->|Rankings| E
    E --> G

    D --> F
    D --> H

    A -->|View Reports| D
    D --> G

    style I fill:#f9f,stroke:#333,stroke-width:2px
    style J fill:#ffa,stroke:#333,stroke-width:2px

Key Features

SEO Tracking Configuration

• Keyword Management: Configure keywords to track with country and location
• Multi-Language Support: Track rankings in different languages
• Multi-Account Support: Separate configurations for child accounts
• Website Tracking: Monitor specific website rankings in search results

Automated Ranking Collection

• Scheduled Execution: Daily automated keyword ranking checks
• Time Slot Management: Distributed execution across time slots
• Error Recovery: Automatic retry logic with error logging
• Rolled Data: Previous rankings used when current check fails

Reporting and Analytics

• Graph Data: Historical ranking trends over time
• Table View: Detailed keyword rankings with pagination
• Position Tracking: Track exact position in search results (top 100)
• Timezone Support: Data reporting in user's timezone

Integration Components

File Structure

external/Integrations/RapidAPI/
├── index.js                    # Main router
├── README.md                   # Integration documentation
├── Controllers/
│   ├── config.js              # Configuration CRUD operations
│   └── rapid.js               # SEO data retrieval and triggers
├── Providers/
│   └── rapid.js               # RapidAPI external calls
├── Models/
│   ├── config.js              # SEO configuration model
│   ├── seo.js                 # SEO ranking results model
│   └── slot.js                # Time slot management model
├── Routes/
│   ├── config.js              # Configuration routes
│   └── rapid.js               # Data retrieval routes
├── Validators/
│   ├── config.js              # Configuration validation
│   └── rapid.js               # Request validation
└── Utils/
    └── index.js               # Utility functions

API Endpoints

Configuration Endpoints

Endpoint	Method	Purpose	Auth Required
/v1/rapidapi/config	POST	Create SEO configuration	Yes
/v1/rapidapi/config	GET	Get SEO configuration	Yes
/v1/rapidapi/config/:configID	PUT	Update SEO configuration	Yes
/v1/rapidapi/config/:configID	DELETE	Delete SEO configuration	Yes

Data Retrieval Endpoints

Endpoint	Method	Purpose	Auth Required
/v1/rapidapi/trigger	POST	Trigger SEO ranking check	No (Internal)
/v1/rapidapi/graph	GET	Get ranking graph data	Yes
/v1/rapidapi/table/:configID	GET	Get ranking table data	Yes

Authentication

API Key Management

The integration requires a RapidAPI subscription key:

// Environment variable
RAPID_GOOGLE_SEARCH_KEY=your_rapidapi_key_here

// Used in API calls
headers: {
  'x-rapidapi-key': process.env.RAPID_GOOGLE_SEARCH_KEY,
  'x-rapidapi-host': 'google-search3.p.rapidapi.com'
}

Key Security:

• Stored in environment variables only
• Never exposed to clients
• Rotated periodically for security
• Access controlled at API gateway level

User Authorization

All configuration endpoints require authorization:

// Middleware chain
router.use(
  '/config',
  utilities.verifyAuthorization(),
  utilities.verifyAccessAndStatus({ accountRequired: true }),
  configRoutes,
);

Data Models

Config Model

{
  _id: ObjectId,
  account: ObjectId,              // Account ID
  keywords: [                      // Keywords to track
    {
      keyword: String,             // Search term
      country: String,             // Country code (e.g., "US")
      near: String                 // Location context (optional)
    }
  ],
  website: String,                 // Website to track
  language: String,                // Language code (e.g., "en")
  slot: ObjectId,                  // Time slot assignment
  is_cron_running: Boolean,        // Execution status
  last_triggred: Date,             // Last execution time
  error_log: {                     // Failed keyword attempts
    "YYYY-MM-DD": [                // Date-based error log
      {
        keyword: String,
        country: String,
        near: String,
        message: String
      }
    ]
  },
  managed_by_dashclicks: Boolean,  // DashClicks account flag
  deleted_on: Date,                // Soft delete timestamp
  createdAt: Date,
  updatedAt: Date
}

SEO Results Model

{
  _id: ObjectId,
  account: ObjectId,
  website: String,
  keyword: String,
  language: String,
  country: String,
  near: String,
  results: [                       // Ranking positions found
    {
      link: String,                // URL found in results
      position: Number             // Position in search (1-100)
    }
  ],
  rolled_from_previous: Boolean,   // Flag for error recovery data
  managed_by_dashclicks: Boolean,
  deleted_on: Date,
  createdAt: Date,
  updatedAt: Date
}

Slot Model

{
  _id: ObjectId,
  time: String,                    // Time slot (e.g., "14:00")
  timezone: String,                // Timezone (e.g., "America/New_York")
  max_capacity: Number,            // Max configs per slot
  current_count: Number,           // Current assigned configs
  active: Boolean,                 // Slot availability
  createdAt: Date,
  updatedAt: Date
}

Usage Examples

Create SEO Configuration

// Request
POST /v1/rapidapi/config?account={childAccountId}
Authorization: Bearer {token}

{
  "keywords": [
    {
      "keyword": "digital marketing agency",
      "country": "US",
      "near": "New York"
    },
    {
      "keyword": "seo services",
      "country": "US",
      "near": "Los Angeles"
    }
  ],
  "website": "https://example.com",
  "language": "en"
}

// Response
{
  "success": true,
  "message": "SUCCESS",
  "data": {
    "_id": "507f1f77bcf86cd799439011",
    "account": "507f191e810c19729de860ea",
    "keywords": [
      {
        "keyword": "digital marketing agency",
        "country": "US",
        "near": "New York"
      },
      {
        "keyword": "seo services",
        "country": "US",
        "near": "Los Angeles"
      }
    ],
    "website": "https://example.com",
    "language": "en",
    "slot": "507f1f77bcf86cd799439012",
    "is_cron_running": false,
    "createdAt": "2024-12-01T10:00:00.000Z"
  }
}

Get SEO Configuration

// Request
GET /v1/rapidapi/config?account={childAccountId}
Authorization: Bearer {token}

// Response
{
  "success": true,
  "message": "SUCCESS",
  "data": {
    "_id": "507f1f77bcf86cd799439011",
    "account": "507f191e810c19729de860ea",
    "keywords": [...],
    "website": "https://example.com",
    "language": "en",
    "slot": "507f1f77bcf86cd799439012",
    "last_triggred": "2024-12-10T14:00:00.000Z",
    "error_log": {
      "2024-12-09": [
        {
          "keyword": "seo services",
          "country": "US",
          "near": "Los Angeles",
          "message": "Rate limit exceeded"
        }
      ]
    }
  }
}

Update SEO Configuration

// Request
PUT /v1/rapidapi/config/:configID?account={childAccountId}
Authorization: Bearer {token}

{
  "keywords": [
    {
      "keyword": "best seo agency",
      "country": "US",
      "near": "Chicago"
    }
  ]
}

// Response
{
  "success": true,
  "message": "SUCCESS",
  "data": {
    "_id": "507f1f77bcf86cd799439011",
    "keywords": [
      {
        "keyword": "best seo agency",
        "country": "US",
        "near": "Chicago"
      }
    ],
    "updatedAt": "2024-12-10T15:30:00.000Z"
  }
}

Get Graph Data

// Request
GET /v1/rapidapi/graph?startDate=2024-12-01&endDate=2024-12-10&account={childAccountId}&tz=America/New_York
Authorization: Bearer {token}

// Response
{
  "success": true,
  "message": "SUCCESS",
  "data": [
    {
      "date": "2024-12-01",
      "keywords": [
        {
          "keyword": "digital marketing agency",
          "avgPosition": 5.2,
          "bestPosition": 3,
          "worstPosition": 8
        },
        {
          "keyword": "seo services",
          "avgPosition": 12.5,
          "bestPosition": 9,
          "worstPosition": 15
        }
      ]
    },
    {
      "date": "2024-12-02",
      "keywords": [...]
    }
    // ... more dates
  ]
}

Get Table Data

// Request
GET /v1/rapidapi/table/:configID?account={childAccountId}&startDate=2024-12-01&endDate=2024-12-10&page=1&limit=25&sortField=keyword&sortOrder=asc&tz=America/New_York
Authorization: Bearer {token}

// Response
{
  "success": true,
  "message": "SUCCESS",
  "data": [
    {
      "keyword": "digital marketing agency",
      "country": "US",
      "near": "New York",
      "currentPosition": 5,
      "previousPosition": 7,
      "change": 2,
      "bestPosition": 3,
      "worstPosition": 8,
      "avgPosition": 5.2,
      "url": "https://example.com/services/digital-marketing",
      "lastChecked": "2024-12-10T14:00:00.000Z"
    },
    {
      "keyword": "seo services",
      "country": "US",
      "near": "Los Angeles",
      "currentPosition": 12,
      "previousPosition": 11,
      "change": -1,
      "bestPosition": 9,
      "worstPosition": 15,
      "avgPosition": 12.5,
      "url": "https://example.com/services/seo",
      "lastChecked": "2024-12-10T14:00:00.000Z"
    }
  ],
  "pagination": {
    "current_page": 1,
    "per_page": 25,
    "total": 2,
    "total_pages": 1
  }
}

Automated Ranking Checks

Daily Trigger Process

The /trigger endpoint is called daily by the Queue Manager:

sequenceDiagram
    participant QM as Queue Manager
    participant RC as Rapid Controller
    participant RP as Rapid Provider
    participant RA as RapidAPI
    participant DB as Database

    QM->>RC: POST /trigger (daily cron)
    RC->>DB: Find slots for today
    RC->>DB: Find configs in time slot

    loop For each config
        RC->>DB: Set is_cron_running = true
        loop For each keyword
            RC->>RP: fetch(keyword, country, near)
            RP->>RA: Google Search query
            RA-->>RP: Search results (top 100)
            RP-->>RC: Rankings data
            RC->>DB: Save SEO results
        end
        RC->>DB: Update last_triggred, clear errors
    end

    Note over RC,DB: Error Recovery Phase
    RC->>DB: Find failed keywords
    RC->>DB: Copy previous day's data
    RC->>DB: Update error logs

    RC-->>QM: Success response

Search Query Format

// Query parameters sent to RapidAPI
{
  q: "digital+marketing+agency",  // Keyword (space-separated with +)
  num: 100,                        // Number of results (top 100)
  lr: "lang_en",                   // Language restriction
  hl: "en",                        // Interface language
  cr: "US",                        // Country restriction
  near: "New York"                 // Location context (optional)
}

// API endpoint
GET https://google-search3.p.rapidapi.com/api/v1/search/{encodedParams}

Ranking Extraction Logic

// For each search result
const websiteToCheck = website.replace('https://', '').replace('http://', '').replace('www.', '');

responseData.forEach((response, index) => {
  if (response.link.indexOf(websiteToCheck) !== -1) {
    // Found website in results
    results.push({
      link: response.link,
      position: index + 1, // 1-based position
    });
  }
});

Error Handling and Recovery

Standard Error Response

{
  "status": false,
  "errno": 400,
  "message": "Error description",
  "additional_info": "Additional error context"
}

Configuration Error Handling

Missing Configuration

// Request
GET /v1/rapidapi/config?account={nonExistentAccount}

// Response
{
  "status": false,
  "errno": 400,
  "message": "Missing configuration."
}

Duplicate Configuration

// Request
POST /v1/rapidapi/config?account={existingAccount}

// Response
{
  "status": false,
  "errno": 400,
  "message": "Configuration already exist!"
}

No Available Slots

// Request
POST /v1/rapidapi/config

// Response
{
  "status": false,
  "errno": 400,
  "message": "No slots available."
}

RapidAPI Error Handling

Rate Limit Exceeded

// Error logged to config.error_log
{
  "2024-12-10": [
    {
      "keyword": "seo services",
      "country": "US",
      "near": "Los Angeles",
      "message": "Request failed with status code 429"
    }
  ]
}

Invalid API Key

{
  "status": false,
  "errno": 401,
  "message": "Invalid API key",
  "additional_info": {
    "response": {
      "status": 401,
      "data": {
        "message": "Invalid API key. Go to https://docs.rapidapi.com/docs/keys for more info."
      }
    }
  }
}

Error Recovery Mechanism

When keyword tracking fails, the system implements automatic recovery:

1. Error Logging: Failed keywords are logged with date and reason
2. Rolled Data: Previous day's rankings are copied if available
3. Retry Logic: Failed keywords are attempted again next day
4. Cleanup: Successfully recovered keywords are removed from error log

// Recovery process for failed keywords
const lastDoc = await seoModel.findLast({
  account,
  website,
  language,
  keyword: failed.keyword,
  country: failed.country,
  near: failed.near,
  updatedAt: { $lt: lastTriggeredOn },
});

if (lastDoc) {
  // Copy previous day's data
  await seoModel.insertOne({
    ...lastDoc,
    rolled_from_previous: true,
    _id: undefined,
    createdAt: undefined,
    updatedAt: undefined,
  });
}

Time Slot Management

Slot Assignment

// Find available slot during configuration
const availableSlot = await slotModel.findAvailableSlot();

// Slot criteria
{
  active: true,
  current_count: { $lt: max_capacity }
}

Slot-Based Execution

// Daily trigger finds configs for current time slot
const slots = await slotModel.findSlotsByTime();
const slotIds = slots.map(s => s._id);

const configs = await configModel.findAll({
  slot: { $in: slotIds },
  is_cron_running: false,
  $or: [{ last_triggred: { $exists: false } }, { last_triggred: { $lt: startOfDay } }],
});

Benefits of Slot System

• Load Distribution: Spreads API calls throughout the day
• Rate Limit Management: Avoids hitting RapidAPI rate limits
• Resource Optimization: Prevents server overload
• Fair Execution: Ensures all configs get daily updates

Notifications

Configuration Added

// Notification
{
  title: "SEO integration added",
  body: "A new SEO integration has been added.",
  module: "projects",
  type: "project_added",
  recipients: [account_owner]
}

Configuration Deleted

// Email notification
{
  subject: "SEO integration disconnected",
  content: "SEO integration has been disconnected.",
  origin: "analytics",
  recipients: [user]
}

// Browser/Bell notification
{
  title: "SEO integration is Disconnected",
  body: "SEO integration has been disconnected.",
  module: "analytics",
  type: "integration_disconnected"
}

Multi-Account Support

Parent-Child Account Handling

// Create configuration for child account
POST /v1/rapidapi/config?account={childAccountId}

// Configuration stored with child account ID
{
  account: childAccountId,
  managed_by_dashclicks: true, // If parent is DashClicks account
  parent_account: parentAccountId
}

// Notifications sent to both parent and child

DashClicks Account Flags

// Utility function checks account hierarchy
const dashFlags = await findDashclicksFlag(accountId, req, 'add');

// Returns flags
{
  managed_by_dashclicks: Boolean,
  parent_account: ObjectId
}

Performance Considerations

Cron Execution

• Sequential Processing: Configs processed one at a time to avoid overload
• Promise Settlement: Uses Promise.allSettled() for parallel keyword checks
• Error Isolation: One keyword failure doesn't stop others
• Execution Lock: is_cron_running flag prevents concurrent execution

Query Optimization

// Efficient date filtering
const startOfDay = moment().startOf('day');
const options = {
  last_triggred: { $lt: startOfDay.toDate() },
};

// Index usage
// - account + last_triggred
// - slot + is_cron_running
// - account + website + keyword (for lookups)

Data Aggregation

// Graph data uses aggregation pipeline
await seoModel.fetchGraph({
  account,
  startDate,
  endDate,
  website,
  tz,
});

// Table data uses aggregation with pagination
await seoModel.fetchTable({
  account,
  startDate,
  endDate,
  website,
  skip,
  limit,
  sortOrder,
  sortField,
  search,
  tz,
});

Environment Variables

# RapidAPI Configuration (Required)
RAPID_GOOGLE_SEARCH_KEY=your_rapidapi_key_here

# MongoDB Connection (Required)
MONGODB_URI=mongodb://localhost:27017/dashclicks

# Redis (Optional - for caching)
REDIS_URL=redis://localhost:6379

Integration Dependencies

Required Packages

{
  "express": "^4.x",
  "axios": "^1.x",
  "moment-timezone": "^0.5.x",
  "mongoose": "^5.x",
  "querystring": "^0.2.x"
}

Internal Dependencies

• Models: User, Config, Account
• Utilities: Auth, Mail, FCM, Logger
• Middleware: Authorization, Access Control

Testing

Test Structure

tests/
├── rapidapi.test.js          # Main integration tests
└── fixtures/
    ├── config-data.json      # Test configurations
    └── seo-results.json      # Sample SEO data

Test Coverage Areas

• Configuration CRUD operations
• Authorization and access control
• Trigger execution and error handling
• Data aggregation and reporting
• Notification delivery
• Multi-account scenarios

Monitoring and Maintenance

Health Checks

• RapidAPI Availability: Monitor API endpoint status
• Cron Execution: Verify daily trigger completion
• Error Rate: Track failed keyword checks
• Response Times: Monitor query performance
• Slot Utilization: Check slot capacity usage

Maintenance Tasks

• API Key Rotation: Update RapidAPI keys quarterly
• Data Retention: Archive old SEO results (90+ days)
• Error Log Cleanup: Remove old error logs periodically
• Slot Rebalancing: Adjust slot capacities based on usage
• Index Optimization: Review and optimize database indexes

Common Issues and Solutions

Issue: "No slots available"

Cause: All time slots are at maximum capacity
Solution: Increase max_capacity on existing slots or create new slots

// Update slot capacity
await slotModel.updateOne({ _id: slotId }, { $set: { max_capacity: 100 } });

Issue: High error rate for keywords

Cause: RapidAPI rate limiting or service issues
Solution: Adjust time slots to spread load, verify API key limits

Issue: Rolled data persists

Cause: Repeated API failures for specific keywords
Solution: Review error logs, verify keyword/location validity

// Check error log
config.error_log['2024-12-10'].forEach(error => {
  console.log(`Failed: ${error.keyword} - ${error.message}`);
});

Issue: Stale data in reports

Cause: Cron trigger not executing
Solution: Verify Queue Manager is running, check trigger logs

// Manually trigger for testing
POST / v1 / rapidapi / trigger;
// Check last_triggred timestamp in config

Best Practices

Configuration

// ✅ Good: Specific keywords with location
{
  "keywords": [
    {
      "keyword": "seo agency",
      "country": "US",
      "near": "New York"
    }
  ]
}

// ❌ Avoid: Too generic, no location
{
  "keywords": [
    {
      "keyword": "marketing",
      "country": "US"
    }
  ]
}

Keyword Management

• Use specific, targeted keywords (2-5 words)
• Include location context for local SEO
• Limit to 20-30 keywords per configuration
• Review and update keywords quarterly

Data Retrieval

// ✅ Good: Specific date range, reasonable pagination
GET /v1/rapidapi/table/:configID?startDate=2024-12-01&endDate=2024-12-10&limit=25

// ❌ Avoid: Very large date ranges without pagination
GET /v1/rapidapi/table/:configID?startDate=2020-01-01&endDate=2024-12-31&limit=1000

Related Documentation

• FCM Integration - Firebase Cloud Messaging
• Queue Manager - Background job processing
• Notifications Service - Notification delivery

Support Resources

RapidAPI

• Marketplace: https://rapidapi.com/
• Google Search3 API: https://rapidapi.com/apigeek/api/google-search3
• Dashboard: https://rapidapi.com/developer/dashboard
• Support: support@rapidapi.com

DashClicks Support

• Integration Issues: Backend team
• API Key Management: DevOps team
• Feature Requests: Product team
• Billing Questions: Accounting team

---

Last Updated: December 2024
Status: Active
Complexity: MEDIUM
Maintenance: Active monitoring required