Destinations

Grouparoo syndicates customer data to many tools using destinations. This type of Plugin defines an App and one more Connections with the export direction. The same Plugin can also make sources, but this document focuses on how to make a Destination.

A Destination will define how Grouparoo should export Record Property values and Group memberships to external tools. What each Record and each Group maps to varies depending on the tool. For example, there can be a Destination that synchronizes all users first name and email address to Mailchimp, tagging them as such if they are in the "High Value" Group.

App

An App represents the ability to communicate with a Destination. For example, this could mean credentials or an API key for a Saas tool.

The properties (PluginApp) of an App are:

• name: The user-facing name of the App. For example, mailchimp or zendesk.

• options: Options needed to configure and connect to your App. This might commonly be an apiKey or username and password. Options have key (string), displayName (string), description (string), placeholder (string), and required (boolean) attributes.

• icon: The path to the App's icon file (SVG or PNG). Icons should be stored in the /public directory of your Plugin.

• App Methods

  These methods are used to configure and interact with the App.

  • test TestPluginMethod This method will be called when an App is created to ensure that the App can be reached. The appOptions and other data will be passed to it. The test method should whether it can connect with those options or not, along with a message. This will often use the connect result to verify success.
  • parallelism AppParallelismMethod If defined, the method will be called to see how many worker processes can be made at a time. The method returns an integer. Many destinations have such a number. For example, Mailchimp can only be contacted 10 times in parallel. More dynamic rate limiting is handled by reacting to the responses while sending data.

Connections

Your Plugin can provide multiple types of Connections for use within Grouparoo. Connections use Apps to import or export data. In the Destination case, this is about the export direction.

The properties (PluginConnection) of a Destination Connection are:

• name: The user-facing name of the connection, like mailchimp-export or zendesk-export.

• direction: Use export for destinations.

• description: The user-facing description of the connection.

• app: The name of the App created before. For example, your mailchimp-export connection might require the mailchimp App, also defined by your Plugin.

• options: Options needed to configure this connection. This differs from the options of an App, and is likely dependent on the App options. For example, a mailchimp-export connection requires a listId option to be set by the Grouparoo user, and will use the apiKey options from the mailchimp App that it is used to get choices for a Mailchimp list.

• Destination Methods

  These methods are used in the creation of a Destination.

  • destinationOptions DestinationOptionsMethod This allows more dynamic options to be specified. For example, it could also use code to provide a set of choices for a list selection. For example, the mailchimp-export lets the user pick from existing Mailchimp lists.
  • destinationMappingOptions DestinationMappingOptionsMethod Connect to the Destination and return the properties that are "required" (often email or external_id) and those that are "known" (often firstName or custom fields). For example, in the Mailchimp case, email_address is required. Asking it via API about the fields returns FNAME, ADDRESS, and any other custom fields that exist. Grouparoo then allows Mapping of these to its properties.

• Data Methods

  One of these should be defined. The choice is whether or not the Destination can export many records at once or it is done one at a time.

  • recordProperty ExportRecordPluginMethod Given the Property values and Group memberships to export for a single Record, send it to the Destination. The singular case is much easier to reason about, particularly when there is an error with the Destination.
  • recordProperties ExportRecordsPluginMethod Given the Property values and Group memberships to export for many records, send it to the Destination. The extra semantics can be more complicated because there can be partial failures. However, if the Destination has a "batch" API, it's generally better to use this for performance reasons.

When defining methods, link to the method directly, do not use a string name.

Research

The first thing to do is create a document that discusses how the Grouparoo model (Records with properties, Group memberships) map to the Destination system. Here are some examples from Salesforce and Intercom.

Here are the main things to figure out (and the answer for the Intercom case).

• What do records map to? Contacts

  • What will the primary key be to look up a Record? email or externalId
  • What does deleting a Record mean? Archive or permanently delete
  • API to look up Record by primary key(s)
  • APIs to create, update (ideally upsert)
  • API to delete

• Understand properties

  • What do they call the other fields? Can they be set at runtime or they predefined? Predefined (but updateable) data attributes
  • Find a list of the valid types of fields
  • API to get the list of known fields

• What will Group membership map to? Tags

  • API to create a Group
  • API to add records to groups

• Do the APIs exist to do these things in bulk? Not anymore?

• Rate limiting

  • Is there a known parallelism limit on concurrent requests? No
  • Is there a number per time period rate limit? Yes. X requests per 10 minutes.
  • Is there a special API response for a rate limit? 429 code with next time header

• Is there a supported/maintained Node API client? Maybe, but it seems out of date

• Anything to learn from other integrations?

  • How does Segment integrate with this Destination? Only handles Users
  • How does Lytics integrate with this Destination? No info

Destination Strategy

We first need to decide if there is anything that should be configured or it's all a simple Mapping. Here are some examples and how we have handled them. Feedback of people already using those systems is critical.

• Salesforce

  There are conceptually unlimited mappings with some primary core "user" Object models: Lead and Contact. They could also be literally anything. A common Group Mapping is to Campaigns, but it can also be very generic. We decided to start with the generic case. There is a highly configurable Destination where you pick the Salesforce Object that you want to sync with. We will likely make more targeted destinations (Leads/Contacts in Campaigns) as we learn more.

• Intercom

  Contacts can be either a concept called "Leads" or "Users" - the difference is if they have signed up or not. Any Contact cannot exist with the same externalId/email combination. In this case, we decided to update whatever was already there and provide options as to how to handle the creation case (as a Lead or a User) because we found people using either exclusively. We also implemented the semantic of "signed up or not" by giving an option to "upgrade" a Lead to a User when they got a externalId.

• Mailchimp

  We had customers asking for different ways to update people in the Mailchimp system. In the most typical case, they had the email address and wants to make Contacts. The required field is email_address. In another case, they actually only had their Mailchimp ID and wanted to update them. This is a different required field and had different semantics (only updating existing users and no creation), so we decided the Mailchimp Plugin could have two different destinations.

• Zendesk

  Many are simple. There is a single Destination and no configuration for Zendesk export. The User fields get filled out and they are tagged based on Group membership.

Batch Strategy

A key thing to understand is if there is a bulk API or not. In general, it's better to use one if it exists, especially with the helpers that we have created to handle the challenges of more than one Record syncing at a time.

The easiest way to get started is to pick the approach and copy an existing Plugin. Pick the ones that is closest. For example if you are making another ad platform Destination that does batching, facebook might be a good bet. Otherwise, here are some simple defaults.

• Single Record at a time (exportRecord): hubspot
• Batch multiple records (exportRecords): marketo

After copying, change the names in the plugin.ts, package.json, and the directory structure to the one you are making. Look for an icon with a license that we can use and note the attribution in the README.md. The icon should be square and 256+ pixels. Transparent backgrounds are preferred.

To keep things building, leave the current package.json dependencies alone, but add the node packages you need. Then go to the root and pnpm install.

Now, it's more or less a copy of the existing one with new names and ids.

Connect the App

The first step generally is to get the App connection going.

The first stop is the plugin.ts file to define the appOptions needed. This is often an apiKey like in hubspot or credentials like in salesforce. Name the options the same thing that the node API client takes in, so it can be passed straight through.

Most destinations have a file called connect.ts that take in the appOptions and return back a client to do API requests. There are many examples: marketo, mailchimp, hubspot.

Make this change and then look to the test.ts file near connect.ts. This is probably referenced from the plugin.ts file and implements the test of the App client. Pass the appOptions to connect.ts to get a client and test it out by doing an API request that returns something that might assure the user things are working. For example, zendesk says who is logged in and mailchimp says who many lists there are available.

Once this is done, the UI should actually work. Create the App with valid credentials. Click on "Check Connection" with good and bad credentials to see is the message is working as expected. Iterate as needed. Write a test if you like.

If there is a parallelism requirement, this is implemented in a parallelism.ts file and noted in plugin.ts like in mailchimp here and here.

Destination Mapping

The next step is to get the Mapping going. This means understanding the equivalent of properties in the Destination system. Common names are "fields," "attributes," "variables", or "properties."

The most common case is that one of these fields are the primary key. This is often "email" or "external_id" or something like that would also be unique in the Grouparoo system. When filling out the Destination options, these would be the required properties like in mailchimp.

Sometimes it's the Destination system dynamically determining what is required. For example, salesforce fetches the fields of the object and sees that some of them are needed to create an Object. These become required fields.

For the rest of the fields, there are three kinds of systems.

• Static

  There are a known set of fields and the user (of the Destination system) can not add more. In this case, the destinationMappingOptions.ts code usually hard codes the known fields. For example, facebook has a set of known fields.

• Defined

  A common case is that the system has a static (or default) set of fields, but the user can add more in the system settings. In this case, an API call is needed to fetch the list. In these cases, they are then Mapping to Grouparoo types from the type given in the API request like in marketo.

  The destinationMappingOptions method allows some known fields to be marked as important. This is done for the most likely ones that a Grouparoo user would want (first name, for example). These are presented immediately in the UI. The rest show up in a typeahead. The marketo Mapping does this by name.

• Dynamic

  A few Destination systems allow completely arbitrary data to be stored. It will essentially make a new field the first time the data is sent. It often already has existing fields. In this case, the Plugin should set allowOptionalFromProperties to true like in sailthru. This gives a new section in the UI for these dynamic mappings.

Especially when there are API requests involved, it's important to create a test suite for the Destination Mapping code like in salesforce, zendesk and others.

At this point, the UI for the Destination Mapping should work. Don't save the Mapping because exportRecords is not ready. However, the Mapping fields can be verified.

Export Records

The core code of the Destination Plugin is the exportRecord (single) or exportRecords (batch) method. It is given the (old and new) properties and the (old and new) Group membership. If the Record is to be deleted, that is noted. The goal is to sync that information to the Destination system.

In both approaches (single or batch), the pattern is generally the same.

• Connect

  Use the appOptions and the connect.ts code to get a client

• Verify Data

  Create an error if a required data (in newRecordProperties is not present). For example, zendesk throws an error if there is no external id.

• Find existing users

  See if the user(s) already exist in the system. This often uses a combination of the old and new Record properties. For example, intercom uses all the primary key data to search for a user and then prioritizes the response.

• Delete if requested

  If being asked to toDelete, the method often returns early. If there was no user found, then it can skip it like in mailchimp.

• Create or Update the user

  Build a payload from the given Property values. Use the oldRecordProperties that are not in newRecordProperties to known which values should be cleared. This sometimes means setting it to null or undefined like in zendesk or an empty string like in malichimp. Sometimes, it means setting a default value like in salesforce.

  Each likely has some formatting constraints. For example, Dates vary between destinations. There is epoch time like in intercom and ISO strings like in marketo.

  If the user was not found, create it via API. If the user was found, update it via API. In either case, there should be a id in the Destination by the end of this step. If it exists, it's best to use an "upsert" API which will use the primary key to create it if necessary. This is more reliable by removing timing risks.

  While creating the payload, sometimes you need more information about the remote fields. If needed while exporting records, cache remote fields because they don't change very often. Share cache but force update in destinationMappingOptions code so they always get the newest. See code from intercom for an example.

• Assign Groups

  Group memberships often become "lists" or "tags" in the Destination system.

  In some destinations, like zendesk, this is done in the payload while updating or creating the user. This is particularly common in the "tag" case.

  In other cases, there is a object (a "list") to possibly create and assign a user to. Caching and mutual exclusion become important in these cases. Several plugins have a listMethods.ts file that handle these cases. The objectCache method can assist in making sure two lists aren't created with the same name and that API calls are minimized. For example, marketo looks for the list and creates it if it does not exist. Wrapping this in objectCache and having the Group name in the cacheKey means that only one thread is doing that at a time and the resulting id is cached.

  Sometimes the API only allows the full list. This gets a bit more complicated because it involves cache management of the whole list, but use examples like hubspot.

  When you get the id of the Group equivalent, then add the user to it. For many destinations, this will take one API request per list. To minimize API calls, if the data of their current membership is on the user Record or can be fetched, only update where needed. See code from intercom.

  When removing from a list, it is ok if the user was never in it to begin with. Try to catch this specific error.

• Return

  The return value for a single Record is an object with a success key as true or false. To help with rate limiting, the method can also return a retryIn key with a number of milliseconds to wait before trying again.

  In the single Record case, an error can be returned and should be a Javascript Error. If the error has a Property called errorLevel (set to error or info), that will be taken into consideration. An info error will not retry, will not show up in Resque errors, but the message will be shown. A regular error (returned or thrown) will have the whole method be retried later, with exponential backoff.

  The case is similar for the batch case, except that it an errors array with a required recordGuid key that notes which Record had the issue.

There should be a significant amount of tests related to exporting records.

Single

The single case makes things fairly straightforward. The system knows which Record it's dealing with, so any error thrown or returned relates to retrying that later. It's already fairly inefficient, but that makes searching much easier because there is no correlation to be done between results and records.

Most of the examples above were from the single Record case. They tend to be a single function or one level deep with a method for each of the above steps. If there is an error in any of them, it does a throw.

Batch

If the API supports batching, it's better to try and use that, but it comes with complications now that there are N records at play. To help with these challenges, Grouparoo has the app-examples/batch helper to codify the above process for the batch case.

The best example is the marketo Plugin.

There are several methods to implement that more or less represent the Destination-specific pieces of the above.

• getClient Return a client to use for the rest of the interaction
• findAndSetDestinationIds Fetch using the given keys to associate a destinationId for each Record
• deleteByDestinationIds Delete the given destinationId values
• updateByDestinationIds Update the given records with their destinationId values
• createByForeignKeyAndSetDestinationIds Create the given records and associate a destinationId
• addToGroups Given a set of groups, add some records
• removeFromGroups Given a set of groups, remove some records
• normalizeForeignKeyValue Gives the opportunity to fix up the email or externalId
• normalizeGroupName Gives the opportunity to fix up the Group name (e.g. lowercase and remove tag spaces)

Within any of these methods, an error or skipMessage (info error) can be associated with a Record. That Record will be omitted from subsequent steps.

Iteration and Testing

Read more about Plugin development to understand the best way to work on your new Plugin.

Plugins are tested using jest. The primary way of testing them is to test the individual methods (exportRecord, destinationMappingOptions, etc) in their own file/suite. Several plugins have multiple suites for exporting records when there are destinationOptions that make them work differently.

Nock

A key piece of these suites is using nock to Record and mock API requests.

Each Plugin tends to have a .env file (along with and .env.example). For example, marketo has the necessary ENV variables as well as setup instructions for the Destination system.

The nockHelper.ts file reads these environment variables to be able to get usable appOptions depending on if the test is recording or not. They also fix up the recorded requests so it doesn't have any sensitive material and can otherwise be replayed.

This can be very tricky. Everything has to be the same when it runs again, so anything based on time or randomness can break it. For example, in sailthru, the library creates a unique request signature, which had to be mocked to be more consistent.

Process

At this point, it has been easiest to start with the copied test and iterate, but one at a time (skipping all the rest). Make sure a Record can be written, then updated, then groups added, ad then deleted.

It's important to be sure to clean up written records so that the first one in the test really is a new user in the Destination system. For example marketo cleans up all test users and lists at the start and end of every run of the suite.

Here is a list of things that are important to test related to exporting records.

• creating users
• updating users
• deleting users
• adding new users to a list
• adding existing users to a list
• handling multiple lists at a time
• removing users from lists
• user change primary key
• edge cases around incorrect values
• known error cases

Use the current tests as your guide.

Example

Here is an example Destination plugin.ts file.

import { Initializer } from "actionhero";
import { plugin } from "@grouparoo/core";

import { test } from "./../lib/test";

import { exportRecord } from "../lib/export/exportRecord";
import { destinationOptions } from "../lib/export/destinationOptions";
import { destinationMappingOptions } from "../lib/export/destinationMappingOptions";
import { exportArrayProperties } from "../lib/export/exportArrayProperties";

const packageJSON = require("./../../package.json");

export class Plugins extends Initializer {
  constructor() {
    super();
    this.name = packageJSON.name;
  }

  async initialize() {
    plugin.registerPlugin({
      name: packageJSON.name,
      icon: "/public/@grouparoo/zendesk/zendesk.png",
      apps: [
        {
          name: "zendesk",
          options: [
            {
              key: "subdomain",
              displayName: "Zendesk Subdomain",
              required: true,
              description: "The `companyname` in companyname.zendesk.com.",
            },
            {
              key: "username",
              displayName: "User Name",
              required: true,
              description:
                "Zendesk username, often the email address of an admin user.",
            },
            {
              key: "token",
              displayName: "API Token",
              required: true,
              description: "Zendesk api token for the admin user.",
            },
          ],
          methods: { test },
        },
      ],
      connections: [
        {
          name: "zendesk-export",
          direction: "export",
          description: "Export Records to a Zendesk account.",
          app: "zendesk",
          options: [],
          methods: {
            exportRecord,
            destinationOptions,
            destinationMappingOptions,
            exportArrayProperties,
          },
        },
      ],
    });
  }

  async start() {
    plugin.mountModels();
  }
}