📊 Semrush Integration

📖 Overview

Semrush integration providing comprehensive SEO analytics including domain rankings, organic and paid keyword data, backlink analysis, competitor insights, and position tracking. Features intelligent caching with 7-day refresh cycles and subscription-based access control.

Provider: Semrush (https://www.semrush.com)
API Version: v1
Integration Type: API Key-based with automatic data caching

📚 Documentation Structure

This integration is organized into the following sections:

Authentication & Projects - API key setup, project management, and subscription validation
Domain Analytics - Domain overview, rankings, and traffic insights
Keyword Research - Organic and paid keyword analysis
Competitor Analysis - Organic and paid competitor insights
Backlink Analysis - Backlink overview and metrics
Position Tracking - Keyword position tracking with ranking distribution

✨ Features

✅ Domain Overview: Get domain authority, traffic estimates, and ranking metrics
✅ Keyword Research: Analyze organic and paid keywords with search volumes
✅ Competitor Analysis: Discover organic and paid competitors by domain
✅ Backlink Metrics: Track backlinks, referring domains, and authority scores
✅ Position Tracking: Monitor keyword rankings with historical data
✅ Project Management: Save and manage Semrush projects per account
✅ Intelligent Caching: 7-day data cache with force refresh option
✅ Subscription Control: Plan-based keyword limits (Pro: 10, Plus: 20, Platinum: 40)
✅ Country Database Support: Multi-country database selection
✅ CSV Parsing: Automatic CSV-to-JSON conversion

🏗️ Architecture

graph TD
    A[Frontend Request] --> B{Cache Check}
    B -->|Fresh Data < 7 days| C[Return Cached Data]
    B -->|Stale Data > 7 days| D[Fetch from Semrush API]
    B -->|Force Refresh ?new=true| D
    D --> E[Parse CSV Response]
    E --> F[Store in MongoDB]
    F --> G[Return Fresh Data]

    H[Subscription Validation] --> I{Has SEO Plan?}
    I -->|Yes| J[Apply Keyword Limit]
    I -->|No| K[Return 403 Error]

    style A fill:#4CAF50
    style D fill:#2196F3
    style F fill:#FF9800
    style H fill:#9C27B0

🗄️ MongoDB Collections

📘 Detailed Schema: See Database Collections Documentation

`semrush.auth`

Purpose: Store Semrush project configurations per account

Schema:

{
  _id: ObjectId,
  account_id: ObjectId,          // DashClicks account reference
  created_by: ObjectId,          // User who created project
  name: String,                  // Project name
  domain: String,                // Primary domain for project
  project_id: String,            // Semrush project ID
  campaign_id: String,           // Google campaign ID (for position tracking)
  createdAt: Date,
  updatedAt: Date
}

Indexes:

{ account_id: 1, domain: 1 } (unique) - Primary lookup
{ account_id: 1 } - Account-based queries

`semrush.reports`

Purpose: Cache Semrush API responses with 7-day TTL

Schema:

{
  _id: ObjectId,
  uid: ObjectId,                 // User ID
  account_id: ObjectId,          // Account ID
  domain: String,                // Target domain
  type: String,                  // Report type (see below)
  data: Array/Object,            // Parsed API response
  createdAt: Date,
  updatedAt: Date
}

Report Types:

domain_rank - Domain overview metrics
domain_organic - Organic keyword data
domain_organic_organic - Organic competitor data
domain_adwords_adwords - Paid competitor data
backlinks_overview - Backlink metrics
keyword_position_tracking_organic - Position tracking data
projects_list - Global projects list (domain: "global")

Indexes:

{ account_id: 1, domain: 1, type: 1, createdAt: -1 } - Cache lookups

📂 Directory Structure

Source Code Location:

external/Integrations/Semrush/
├── Controllers/
│   └── semrush.js           # Request handlers with caching logic
├── Models/
│   └── semrush.js           # Data access layer and cache queries
├── Providers/
│   └── semrush.js           # Semrush API wrapper
├── Routes/
│   └── semrush.js           # Route definitions
├── Validators/
│   └── index.js             # Joi validation schemas
├── Utils/
│   └── countries.js         # Country code mappings
└── index.js                 # Route registration

Shared Models Used:

shared/models/semrush-auth.js - Project configurations
shared/models/semrush.js - Cached report data
shared/models/store-subscription.js - Subscription validation

⚙️ Environment Variables

# Required
SEMRUSH_API_KEY=your_api_key_here
SEMRUSH_API_URL=https://api.semrush.com/
SEMRUSH_DATABASE=us                    # Default country database

# Optional - For specific countries
# Supported: us, uk, ca, au, de, fr, es, it, nl, br, ru, in, etc.

🚀 Quick Start

1. Save Project Configuration

POST /v1/integrations/semrush/auth/project
Authorization: Bearer {jwt_token}
Content-Type: application/json

{
  "projectName": "Client Website SEO",
  "domain": "example.com",
  "projectId": "12345678"
}

2. Get Domain Overview

GET /v1/integrations/semrush/overview/example.com
Authorization: Bearer {jwt_token}

3. Analyze Organic Keywords

GET /v1/integrations/semrush/keyword/organic/example.com?new=true
Authorization: Bearer {jwt_token}

4. Track Keyword Positions

GET /v1/integrations/semrush/keyword/tracking/example.com?config_id=507f1f77bcf86cd799439011
Authorization: Bearer {jwt_token}

📡 API Endpoints Summary

Method	Endpoint	Description
Projects
POST	`/auth/project`	Save project configuration
DELETE	`/auth`	Delete project and cached data
GET	`/projects`	List all Semrush projects
Domain Analytics
GET	`/overview/:domain`	Domain overview and rankings
Keywords
GET	`/keyword/organic/:domain`	Organic keyword rankings
GET	`/keyword/tracking/:domain`	Position tracking with history
Competitors
GET	`/competitor/organic/:domain`	Organic competitors
GET	`/competitor/paid/:domain`	Paid competitors
Backlinks
GET	`/backlinks/overview/:domain`	Backlink metrics

🔒 Subscription Requirements

SEO Subscription Plans

Access requires an active SEO subscription with plan-based limits:

Plan	Keyword Tracking Limit	Price Point
Pro	10 keywords	Entry-level
Plus	20 keywords	Mid-tier
Platinum	40 keywords	Premium
Internal	Unlimited	DashClicks staff

Validation:

Subscription status: active, trialing, or past_due
Product type: seo
Returns 403 error if no valid subscription found

Note: Internal DashClicks users bypass all limits.

📊 Data Caching Strategy

Cache Lifecycle

First Request: Fetches from Semrush API → Stores in MongoDB
Subsequent Requests: Returns cached data if < 7 days old
Stale Data: Auto-refreshes if > 7 days old
Force Refresh: Use ?new=true to bypass cache

Cache Metadata

Every response includes:

{
  "data": [...],
  "createdAt": "2025-10-03T10:00:00Z",
  "lastUpdate": "2025-10-03T10:00:00Z",
  "nextUpdate": "2025-10-10T10:00:00Z",
  "daysSinceUpdate": 7,
  "daysUntilUpdate": 0,
  "isStale": true
}

🌍 Country Database Support

Semrush supports multiple country-specific databases for localized SEO data:

Supported Countries (partial list):

us - United States (default)
uk - United Kingdom
ca - Canada
au - Australia
de - Germany
fr - France
es - Spain
it - Italy
nl - Netherlands
br - Brazil
ru - Russia
in - India
jp - Japan
cn - China
mx - Mexico

Country Detection: Automatically uses account's country from account.business.address.country

📈 Response Format

Standard Response

{
  "success": true,
  "message": "SUCCESS",
  "data": {
    "data": [
      // Parsed CSV data as JSON array
    ],
    "createdAt": "2025-10-10T08:00:00Z",
    "lastUpdate": "2025-10-10T08:00:00Z",
    "nextUpdate": "2025-10-17T08:00:00Z",
    "daysSinceUpdate": 0,
    "daysUntilUpdate": 7,
    "isStale": false
  }
}

Paginated Response (Projects/Keywords)

{
  "success": true,
  "message": "SUCCESS",
  "data": {
    "data": [...],
    "pagination": {
      "currentPage": 1,
      "totalPages": 3,
      "totalItems": 75,
      "itemsPerPage": 25,
      "hasNextPage": true,
      "hasPreviousPage": false
    }
  }
}

⚠️ Error Handling

Semrush API Errors

Error Code	Meaning	HTTP Status
`ERROR 50`	Invalid API key or subscription required	500
`ERROR 40`	Invalid request or resource not found	400
`ERROR 30`	Rate limit exceeded	429

DashClicks Errors

Error	HTTP Status	Description
`SUBSCRIPTION_REQUIRED`	403	No active SEO subscription found
`PROJECT_NOT_FOUND`	404	Project configuration missing
`INVALID_DOMAIN`	400	Domain format invalid
`TIMEOUT_ERROR`	504	Request timeout (600s limit)

Semrush API: Official Documentation
Semrush Projects: Project Management API
Semrush Reports: Reports API
Semrush Analytics: Analytics API

🎯 Common Use Cases

1. Client SEO Dashboard

Track client website performance with:

Domain authority and rankings
Top organic keywords
Backlink profile
Competitor analysis

2. Keyword Position Tracking

Monitor keyword rankings over time:

Historical position data
Ranking distribution (Top 1, 2-5, 6-10, 10+)
URL-level tracking
Trend analysis

3. Competitive Intelligence

Analyze competitors:

Shared organic keywords
Paid advertising strategies
Traffic estimates
Keyword gaps

4. SEO Reporting

Generate comprehensive SEO reports:

Weekly/monthly position tracking
Keyword performance trends
Backlink growth
Domain authority changes

📝 Important Notes

🕒 Cache Duration: 7 days (configurable via shouldForceRefresh)
🔄 Force Refresh: Use ?new=true to bypass cache
🌍 Country-Specific: Automatically selects database based on account country
📊 CSV Format: API returns CSV; automatically parsed to JSON
🔐 Subscription Required: Position tracking requires active SEO subscription
⏱️ Timeout: 600 seconds (10 minutes) for API requests
🚫 Rate Limiting: Semrush enforces API rate limits (varies by subscription)
📈 Pagination: Projects and keyword tracking support pagination
🔍 Search: Projects endpoint supports search by name, domain, or URL
🎯 Plan Limits: Keyword tracking limited by subscription plan

🆕 Recent Enhancements

✅ Subscription Validation: Added SEO subscription requirement for position tracking
✅ Plan-Based Limits: Keyword limits based on Pro/Plus/Platinum plans
✅ MongoDB Aggregation: Efficient sorting and pagination for large datasets
✅ Keyword Summary: Ranking distribution (rank 1, 2-5, 6-10, 10+)
✅ Enhanced Error Handling: Detailed error context for debugging
✅ Project Campaigns: Auto-fetch Google campaign ID for position tracking
✅ Search Functionality: Search projects by multiple fields
✅ Force Refresh Logic: Improved cache invalidation

Last Updated: October 10, 2025
Integration Status: Production Ready
API Version: Semrush v1

📖 Overview​

📚 Documentation Structure​

✨ Features​

🏗️ Architecture​

🗄️ MongoDB Collections​

semrush.auth​

semrush.reports​

📂 Directory Structure​

⚙️ Environment Variables​

🚀 Quick Start​

1. Save Project Configuration​

2. Get Domain Overview​

3. Analyze Organic Keywords​

4. Track Keyword Positions​

📡 API Endpoints Summary​

🔒 Subscription Requirements​

SEO Subscription Plans​

📊 Data Caching Strategy​

Cache Lifecycle​

Cache Metadata​

🌍 Country Database Support​

📈 Response Format​

Standard Response​

Paginated Response (Projects/Keywords)​

⚠️ Error Handling​

Semrush API Errors​

DashClicks Errors​

🔗 Related Documentation​

🎯 Common Use Cases​

1. Client SEO Dashboard​

2. Keyword Position Tracking​

3. Competitive Intelligence​

4. SEO Reporting​

📝 Important Notes​

🆕 Recent Enhancements​

Documentation Assistant