Skip to main content

🗂️ Add Data Utilities

📖 Overview​

The add_data.js utility provides field mapping and data transformation functions for CSV imports. It handles complex nested object construction (address, social media) and format normalization (date of birth) during contact and account import processes.

Source File: queue-manager/common/add_data.js

🎯 Purpose​

  • Field Mapping: Maps CSV columns to nested database objects
  • Data Normalization: Standardizes date formats and field structures
  • Category Separation: Organizes fields into main, address, social, additional_info
  • Account/Business Mapping: Handles person-company relationship data

📘 Functions​

1. Main Contact Mapping (Default Export)​

module.exports = (mappings, value, maping, cc, address, social, additionalInfo);

Parameters​

  • mappings (Object) - Mapping configuration: { 'CSV Column': 'db.field.path' }
  • value (any) - Current field value from CSV
  • maping (String) - Current CSV column being processed
  • cc (Object) - Accumulator for contact/company fields
  • address (Object) - Accumulator for address fields
  • social (Object) - Accumulator for social media fields
  • additionalInfo (Object) - Accumulator for custom fields

Returns​

{
  cc, address, social, additionalInfo;
}

2. Account/Business Mapping​

module.exports.addAccountData =
  (mappings, value, maping, account, business, address, social, accountInfo, businessInfo);

Parameters​

  • mappings (Object) - Mapping configuration with prefixes (account.*, business.*)
  • value (any) - Current field value
  • maping (String) - Current CSV column
  • account (Object) - Account-specific fields accumulator
  • business (Object) - Business-specific fields accumulator
  • address (Object) - Business address accumulator
  • social (Object) - Business social media accumulator
  • accountInfo (Object) - Final account object accumulator
  • businessInfo (Object) - Final business object accumulator

Returns​

{
  accountInfo, businessInfo;
}

🔧 Implementation Details​

Date of Birth Normalization​

if (mappings[maping] == 'dob') {
  if (moment(value, 'DD-MM-YYYY').isValid()) {
    value = moment(value, 'DD-MM-YYYY').format('YYYY-MM-DD');
  }
  if (moment(value, 'MM-DD-YYYY').isValid()) {
    value = moment(value, 'MM-DD-YYYY').format('YYYY-MM-DD');
  }
}

Logic:

  • Accepts two input formats: DD-MM-YYYY (European) or MM-DD-YYYY (US)
  • Converts to standardized YYYY-MM-DD (ISO 8601)
  • Ensures database consistency for date comparisons
  • Uses moment.js for validation and formatting

Example:

// Input: "15-03-1990" → Output: "1990-03-15"
// Input: "03-15-1990" → Output: "1990-03-15"

Address Field Mapping​

if (
  [
    'address.street',
    'address.unit',
    'address.city',
    'address.state_province',
    'address.postal_code',
    'address.country',
  ].includes(mappings[maping])
) {
  if (value) {
    address[mappings[maping].replace('address.', '')] = value;
  }
  cc['address'] = address;
}

Logic:

  • Detects address.* prefix in mapping
  • Strips prefix and adds to address object
  • Only adds non-empty values (prevents null/undefined pollution)
  • Assigns complete address object to contact

Example:

// Mapping: { 'Street': 'address.street', 'City': 'address.city' }
// CSV Row: { 'Street': '123 Main St', 'City': 'New York' }
// Result: cc.address = { street: '123 Main St', city: 'New York' }

Social Media Field Mapping​

if (
  [
    'social.facebook',
    'social.instagram',
    'social.linkedin',
    'social.twitter',
    'social.youtube',
    'social.yelp',
    'social.pinterest',
    'social.vimeo',
    'social.snapchat',
    'social.reddit',
    'social.tripadvisor',
    'social.foursquare',
    'social.rss',
  ].includes(mappings[maping])
) {
  if (value) {
    social[mappings[maping].replace('social.', '')] = value;
  }
  cc['social'] = social;
}

Supported Platforms:

  • Major Social: Facebook, Instagram, LinkedIn, Twitter, YouTube
  • Review Sites: Yelp, TripAdvisor, Foursquare
  • Media: Pinterest, Vimeo, Snapchat, Reddit
  • Other: RSS feeds

Logic:

  • Same pattern as address mapping
  • Strips social. prefix
  • Creates nested social object
  • Only adds non-empty values

Example:

// Mapping: { 'LinkedIn': 'social.linkedin', 'Twitter': 'social.twitter' }
// CSV: { 'LinkedIn': 'linkedin.com/in/johndoe', 'Twitter': '@johndoe' }
// Result: cc.social = { linkedin: 'linkedin.com/in/johndoe', twitter: '@johndoe' }

Additional Info (Custom Fields)​

else if (mappings[maping].includes('additional_info.')) {
    if (value) {
        additionalInfo[mappings[maping].replace("additional_info.", "")] = value;
    }
    cc["additional_info"] = additionalInfo;
}

Logic:

  • Catches any unmapped custom fields
  • Stores in additional_info object
  • Prevents data loss from custom CSV columns
  • Enables flexible schema extensions

Example:

// Mapping: { 'Custom Field': 'additional_info.custom_field' }
// CSV: { 'Custom Field': 'custom value' }
// Result: cc.additional_info = { custom_field: 'custom value' }

Default Field Mapping​

else {
    if (value) {
        cc[mappings[maping]] = value;
    }
}

Logic:

  • Handles all non-nested fields (name, email, phone, etc.)
  • Direct assignment to contact object
  • Skips empty values to keep object clean

🏢 Account/Business Mapping​

Account Fields​

if (['account.firstname', 'account.lastname', 'account.email'].includes(mappings[maping])) {
  if (value) {
    account[mappings[maping].replace('account.', '')] = value;
  }
  accountInfo = { ...accountInfo, ...account };
}

Logic:

  • Maps account owner information
  • Strips account. prefix
  • Merges into accountInfo object
  • Used for team member imports

Business Fields​

if (['business.name', 'business.email', 'business.phone'].includes(mappings[maping])) {
  if (value) {
    business[mappings[maping].replace('business.', '')] = value;
  }
  businessInfo = { ...businessInfo, ...business };
}

Logic:

  • Maps company/business information
  • Strips business. prefix
  • Separate from personal contact info
  • Enables person-company linking

Business Address​

if (
  [
    'business.address.street',
    'business.address.unit',
    'business.address.city',
    'business.address.state_province',
    'business.address.postal_code',
    'business.address.country',
  ].includes(mappings[maping])
) {
  if (value) {
    address[mappings[maping].replace('business.address.', '')] = value;
  }
  businessInfo['address'] = address;
}

Logic:

  • Two-level prefix: business.address.*
  • Separate from person address
  • Supports business location tracking

Business Social Media​

if (
  [
    'business.social.facebook',
    'business.social.instagram',
    'business.social.linkedin',
    'business.social.twitter',
    'business.social.youtube',
    'business.social.yelp',
    'business.social.pinterest',
    'business.social.vimeo',
    'business.social.snapchat',
    'business.social.reddit',
    'business.social.tripadvisor',
    'business.social.foursquare',
    'business.social.rss',
  ].includes(mappings[maping])
) {
  if (value) {
    social[mappings[maping].replace('business.social.', '')] = value;
  }
  businessInfo['social'] = social;
}

Logic:

  • Two-level prefix: business.social.*
  • Company social media pages (not personal)
  • Same platforms as personal social

🎨 Usage Patterns​

Contact Import Flow​

const addData = require('./common/add_data');

let cc = {};
let address = {};
let social = {};
let additionalInfo = {};

// Iterate through CSV mappings
for (let maping in mappings) {
  const codeRS = maping.trim().split('.');
  let currentMapping;

  // Get value from CSV row using dot notation
  for (let param of codeRS) {
    currentMapping = myFun(param, contact, currentMapping);
  }

  // Apply mapping with add_data utility
  const result = addData(mappings, currentMapping, maping, cc, address, social, additionalInfo);

  // Update accumulators
  cc = result.cc;
  address = result.address;
  social = result.social;
  additionalInfo = result.additionalInfo;
}

// Final contact object
const contactObject = {
  ...cc,
  type: 'person',
  owner: userId,
  // ... other fields
};

Account/Business Import Flow​

const { addAccountData } = require('./common/add_data');

let account = {};
let business = {};
let address = {};
let social = {};
let accountInfo = {};
let businessInfo = {};

for (let maping in mappings) {
  let addType;

  // Determine if field is for account or business
  if (mappings[maping].split('.')[0] === 'account') {
    addType = 'account';
  } else {
    addType = 'business';
  }

  // Get value and apply mapping
  const value = getValueFromCSV(maping, csvRow);

  const result = addAccountData(
    mappings,
    value,
    maping,
    account,
    business,
    address,
    social,
    accountInfo,
    businessInfo,
  );

  accountInfo = result.accountInfo;
  businessInfo = result.businessInfo;
}

📊 Real-World Example​

CSV File​

First Name,Last Name,Email,Phone,Street,City,State,ZIP,LinkedIn,Custom Note
John,Doe,john@example.com,1234567890,123 Main St,New York,NY,10001,linkedin.com/in/johndoe,VIP Client

Mapping Configuration​

const mappings = {
  'First Name': 'first_name',
  'Last Name': 'last_name',
  Email: 'email',
  Phone: 'phone',
  Street: 'address.street',
  City: 'address.city',
  State: 'address.state_province',
  ZIP: 'address.postal_code',
  LinkedIn: 'social.linkedin',
  'Custom Note': 'additional_info.note',
};

Processing Result​

{
    first_name: 'John',
    last_name: 'Doe',
    email: 'john@example.com',
    phone: '1234567890',
    address: {
        street: '123 Main St',
        city: 'New York',
        state_province: 'NY',
        postal_code: '10001'
    },
    social: {
        linkedin: 'linkedin.com/in/johndoe'
    },
    additional_info: {
        note: 'VIP Client'
    }
}

🔄 Data Flow Diagram​

graph TD
    A[CSV Row] --> B{Field Mapping Type}
    B -->|Standard Field| C[Direct Assignment to cc]
    B -->|address.*| D[Add to address Object]
    B -->|social.*| E[Add to social Object]
    B -->|additional_info.*| F[Add to additionalInfo Object]
    B -->|dob| G[Normalize Date Format]

    C --> H[Return Updated Objects]
    D --> I[Assign address to cc.address]
    E --> J[Assign social to cc.social]
    F --> K[Assign additionalInfo to cc.additional_info]
    G --> C

    I --> H
    J --> H
    K --> H

    H --> L[Final Contact Object]

⚙️ Configuration​

Supported Address Fields​

[
  'address.street', // Street address
  'address.unit', // Apartment/suite number
  'address.city', // City name
  'address.state_province', // State or province
  'address.postal_code', // ZIP/postal code
  'address.country', // Country name
];

Supported Social Platforms​

[
  'social.facebook',
  'social.instagram',
  'social.linkedin',
  'social.twitter',
  'social.youtube',
  'social.yelp',
  'social.pinterest',
  'social.vimeo',
  'social.snapchat',
  'social.reddit',
  'social.tripadvisor',
  'social.foursquare',
  'social.rss',
];

🚨 Error Handling​

Empty Value Handling​

if (value) {
  cc[mappings[maping]] = value;
}

Logic:

  • Only adds non-empty values
  • Prevents null, undefined, empty string pollution
  • Keeps objects clean and minimal

Invalid Date Handling​

if (moment(value, 'DD-MM-YYYY').isValid()) {
  value = moment(value, 'DD-MM-YYYY').format('YYYY-MM-DD');
}

Behavior:

  • Invalid dates are skipped (not converted)
  • Original value preserved if neither format matches
  • No error thrown - fails silently

Missing Fields​

  • Unmapped fields are ignored (not added to any object)
  • No validation errors - import continues
  • Can result in incomplete data if mapping is incorrect

📈 Performance Considerations​

Memory Efficiency​

  • Accumulator Pattern: Reuses objects across loop iterations
  • Conditional Addition: Only adds non-empty values
  • No Deep Cloning: Mutates objects in place

Optimization Tips​

  1. Pre-filter Mappings: Remove empty mappings before loop
  2. Batch Processing: Process multiple rows before database insert
  3. Field Validation: Validate required fields before calling addData

🧪 Testing Considerations​

Test Cases​

const addData = require('./common/add_data');

describe('addData', () => {
  test('Maps standard fields', () => {
    const mappings = { Email: 'email' };
    const result = addData(mappings, 'test@example.com', 'Email', {}, {}, {}, {});
    expect(result.cc.email).toBe('test@example.com');
  });

  test('Maps address fields', () => {
    const mappings = { City: 'address.city' };
    const result = addData(mappings, 'New York', 'City', {}, {}, {}, {});
    expect(result.cc.address.city).toBe('New York');
  });

  test('Normalizes date of birth', () => {
    const mappings = { DOB: 'dob' };
    const result = addData(mappings, '15-03-1990', 'DOB', {}, {}, {}, {});
    expect(result.cc.dob).toBe('1990-03-15');
  });

  test('Skips empty values', () => {
    const mappings = { Phone: 'phone' };
    const result = addData(mappings, '', 'Phone', {}, {}, {}, {});
    expect(result.cc.phone).toBeUndefined();
  });
});

🔗 Related Documentation​

📝 Notes​

Mapping Prefix Convention​

  • No prefix: Standard contact field (e.g., email, phone)
  • address.*: Contact address field
  • social.*: Contact social media field
  • additional_info.*: Custom contact field
  • account.*: Account owner field (team member)
  • business.*: Business/company field
  • business.address.*: Business address field
  • business.social.*: Business social media field

Dependencies​

  • moment.js: For date parsing and formatting
  • Consider migrating to dayjs or native Date API for smaller bundle size

Complexity: Medium
Business Impact: High - Critical for import data integrity
Dependencies: moment.js
Last Updated: 2025-10-10

💬

Documentation Assistant

Ask me anything about the docs

Hi! I'm your documentation assistant. Ask me anything about the docs!

I can help you with:
- Code examples
- Configuration details
- Troubleshooting
- Best practices

Try asking: How do I configure the API?
09:31 AM