Connector Design Considerations
When designing a Connector there are many factors to take into consideration. You can design a very simple Connector that just queries data or you can design one that is as complex as the API allows. When developing our own TIBCO Scribe ® Connectors or providing guidance to other developers, we take the following things into Consideration.
What problems does this Connector solve?
Specifically outlining the problem you are trying to solve or the benefits that the Connector provides can be helpful from both a design perspective and a marketing perspective. Outline the goals for your Connector and keep those in mind as you work. Using our HubSpot Connector as an example, your list of benefits might include something similar to the following:
- Integrate your HubSpot marketing contacts with other contact management systems in your organization.
- Update HubSpot contact life cycle stages with input from other business applications.
- Export marketing lists from HubSpot to any other business application or data endpoint.
- Integrate HubSpot company and associated contacts into your CRM application.
- Report on Company revenue based on associated HubSpot closed won Deals.
- Create a client in a project management system when a HubSpot Deal is marked as closed won.
- Synchronize HubSpot Deals with your CRM application.
- Update calendar events in another application when a HubSpot Task is created, updated, or deleted.
Supported TIBCO Scribe ® Options
Which TIBCO Scribe ® options do you plan to support?
At a high level, determine which features you want to support from the following list:
- On-Premise Agent
- Cloud Agent
- Replication Services As Source
- Replication Services As Target
- Integration Services As Source
- Integration Services As Target
- Migration Services As Source
- Migration Services As Target
- Integration Maps
- Request-Reply Maps
- Message Maps
Private or Public Connector
Will this Connector be publicly available or available upon request?
If you release your Connector in the TIBCO Scribe ® Marketplace you can choose to manage it and only give it to selected users or to make it publicly available to all users. In addition, you can choose to charge a fee for using the Connector.
How does the Connector select entities for Replication?
If replication is supported for this Connector, determine how entities that are recommended for replication are selected, such as they have a modified date field. If any entities meet the criteria for recommended but are not included in the recommended list, provide the names of the ignored entities for the documentation.
How does the Connector track deleted source records for Replication?
Typically, records are never deleted from a replication they are just marked as deleted with the datetime of deletion. If the source itself does not have a mechanism for tracking deleted records, then TIBCO Scribe ® must compare the source records to the target records and determine which ones are missing. This can be very time consuming and may cause performance issues.
See Managing TIBCO Scribe ® RS Solutions for an overview of how Replication Services work.
Supported Entities And Operations
Which entities and operations do you want to support?
Review the API information available for the system you want to access. Some systems are very large and you may not want to support all entities and operations. Some entities may not be very useful, may be difficult to use, or may not fit with the use cases you outlined at the beginning of the design process. If the system allows users to create custom entities, those may need to be supported also.
Not all entities support all operations. For example, you may be able to query or create Users, but not update them. Some API's have special operations outside the usual Query, Create, Update, Delete, and Upsert. For example, HubSpot has a special operation called Submit that is used with the Form entity to submit data that would normally be entered in a form on a web site. You need to determine whether or not you want to support those special operations.
What are the setup requirements to allow external Connections?
Most systems have specific configuration requirements that must be met when accessing their API. Consider the following:
- Versions/editions of the software supported by this Connector. For example, the Connector for Microsoft Dynamics 365 can only connect to certain versions of Microsoft Dynamics CRM and Microsoft Dynamics 365.
- Setup requirements, such as, Administrator Access or API Key.
- Any special permissions required in the datastore to access the data.
- An Ignore User to prevent passing the same data back and forth multiple times. See Ignore User in the TIBCO Scribe ® Help.
- Any special local or external configuration files.
- When there is a user account involved, determine whether more than one Connection can use the same account or not. Some APIs restrict the number of connections from the same account.
- Consider what might happen if the user is logged into the source/target application and then a TIBCO Scribe ® Solution runs using the same user account.
What fields do you need on the Connection dialog in TIBCO Scribe ®?
The Connection dialog typically contains fields for the credentials required to connect to the system you are accessing. However, it can also contain fields for global settings or other optional parameters that can be sent back to the API when making a call to that API. Consider the following:
- URL for the system you are accessing. This could be hard-coded, a field with a default setting, or a free form field. For example, for Microsoft Dynamics 365, the URL varies depending on your location in the world, but for Salesforce the URL varies based on the version of the API you want to access.
- Credential fields required and whether or not they need to be masked, such as a Password field.
- Authentication method, such as OAuth or basic.
- Filter fields that reduce the amount of metadata returned. For example, the Microsoft SQL Server Connection dialog allows you to limit the tables accessed by entering a Connection Parameter in the an Additional Parameters field. This can improve performance in large systems.
- Batch and thread settings that control how many records are sent at the same time. The Microsoft Dynamics 365 Connection dialog allows you to set the batch size when bundling records to be sent in a single API call.
- If the Connector is creating or retrieving files, such as JSON or XML files, indicate the directory where the data is stored.
What retry logic do you plan to use if the Connection or one of the operations fail?
Retry logic varies by Connector. Here are some examples of retry logic for some of our existing Connectors.
- Timeout exceptions are caught and retried.
- Connector retries 3 times, for 2 minutes each time with a second, two seconds, and three seconds between each try.
- When a timeout occurs, the Connector retries five times. Each time a retry fails, the time between retries is increased by 10 minutes.
- The Connector waits a second after every call, regardless of success or failure. If the call has a rate limit failure, the Connector waits up to an additional five seconds before retrying. With each successive rate limit failure, the wait time is increased up to an additional 20 seconds. Only after 15 successive rate limit failures will the connector return an error and fail the row.
Does the Connector support relationships?
In many systems there are related entities, such as a SalesOrder parent entity and its associated SalesOrderItems child entity. Depending on the configuration of the API these may be easy or difficult to support. Support can be on a per entity basis and not involve all entities.
Does the Connector support Hierarchical Data?
Many datastores and file types use hierarchical datasets, such as XML or JSON, which contain multiple levels of data within a single datastore. For example, a single file can represent a record which has a hierarchy of Country, State, and City or parent, child, grandchild.
Are there entities that need special considerations?
In some cases entities are really for internal use only and are not helpful to a user. They may not be easy to integrate and work with outside of their home systems because they contain less common data types. Analyze the metadata and determine whether or not some entities should be blacklisted to make sure they are excluded.
Which data types does the Connector support?
Review the data types included in the API you are accessing to make sure you can support them or convert them to a different data type. For example, Oracle includes data type BFILE, which the Oracle Connector does not support. This means that anyone using the Oracle Connector must skip that information when developing a Map. The XML Connector converts XML data types to data types that can be used with TIBCO Scribe ®. For example, the XML Connector converts SByte to Int16.
How does the Connector handle DateTime data?
When the Net Change feature is enabled, TIBCO Scribe ® uses DateTime data as a filter to determine when a new record is created or an existing record is modified. DateTime data can often be used as a filter on the Filter tab of the Query operation Block. The more precise the data is the fewer records are processed, which improves overall performance. Whenever possible, DateTime that includes milliseconds is best.
Most TIBCO Scribe ® Connectors convert source DateTime data to UTC as the data is retrieved and convert data being written to a target system from UTC to the DateTime that system expects. For example, the Pardot Connector converts DateTime data to the time zone setting of the Pardot user account used in the Connection when it writes data to Pardot. Consider the following for DateTime data:
- Review how DateTime data is stored in the system you are accessing, determine whether it is in UTC or another time zone, and whether it includes milliseconds.
- Can you support the Net Change feature? This feature requires that the entity have DateTime fields that are updated when a record is added or changed, such as a LastModifiedOn or CreatedOn field. These fields must be designated as Net Change fields. See Most Recent Record Processed in the TIBCO Scribe ® Help for more information.
How does the Connector handle case sensitivity?
Depending on the system you are accessing either the entire datastore or specific entities could be case sensitive. Wherever possible, case insensitivity is preferable.
How does the Connector handle fields that are pick lists?
- Pick lists may require specific values based on the values of other fields in the record. This information should be included in the field descriptions displayed in the Fields tab of the target operation Block. See the Viewing Field Properties section of Block Properties Fields Tab in the TIBCO Scribe ® Help.
- Does the system allow NULL values for pick lists?
- For multi-select pick lists or pick lists that allow the user to select multiple values, consider how the values are delimited and how to handle special characters within a value, such as the single quote in Joe's Restaurant.
- If it is a multi-select pick list, determine whether or not the values can be used for filtering and Match Criteria.
Which Boolean values does the Connector support?
Determine which boolean values are used by the system you are accessing and how you will translate incoming values that do not match. Some examples include:
Other considerations for Boolean values:
- Are the values case sensitive?
- What happens when a NULL value is received or sent?
- What happens when an unsupported value is received or sent?
How does the Connector handle NULL values?
There may be fields in the system that do not accept NULL values. A formula can be used in TIBCO Scribe ® to detect these values and replace them with a valid piece of data. However, it is better if the Connector can handle them gracefully. You should also determine whether empty, NULL, and 0 values are handled differently.
Filtering, Match Criteria, Lookup
What filtering options are included in the Connector?
In TIBCO Scribe ®, filtering is used to reduce the records returned by a Query. Match Criteria is used to match a source record with a target record that you wish to update or delete. The Lookup operation uses a filter to locate data in a datastore. All three are types of filters that are similar even though they are used in different circumstances. See Block Properties Filters Tab in the TIBCO Scribe ® Help for filtering examples and rules. Filtering considerations include:
- Are AND and OR supported?
- Which operators, such as equals, greater than, or less than, are supported?
- Are there any entities that cannot be filtered?
- Are any entities limited to filtering only on certain fields and what are those fields? Be sure to hide the fields that cannot be used as filters.
- Are any filters required to return records?
- Is there any limit to the number of filters on an entity or all entities?
- What is the sort order of the records being returned?
Sort order is particularly important for Net Change. If records are not returned in order by date, the Connector has to retrieve all records, sort them by datetime and then process only the new or updated records.
Keep in mind that the Net Change feature is considered a filter because it filters by date modified or created. If there are limits to the number of filters, you may not be able to use both Net Change and a filter.
Does the Connector support the Preview Tab in the Query and Fetch Blocks?
Both the Query and the Fetch operations offer the opportunity to preview the first 25 records of source data, which can be helpful when troubleshooting. In some cases, there are issues previewing data. For example, some Connectors require at least one filter or they cannot display data in the Preview pane. See Block Properties Preview Tab in the TIBCO Scribe ® Help.
Does the Connector support bulk target operations?
Determine whether or not Bulk Processing is supported in the system you are accessing and for which entities. If it is not supported for all entities, grey out the Batch size setting on the target operation Block for entities that do not support Bulk /Batch processing. For example, when creating records, the application to which you are connecting may accept data in batches of 100 records and treat that batch as a single call to the API. This reduces API calls and improves performance. See Batch Processing in the TIBCO Scribe ® Help.
If Batch Processing is supported, TIBCO recommends testing to determine the ideal batch size and using that as the default batch size displayed in the Block Properties General tab for the operation.
General notes on issues with entities.
For any given entity and/or associated operation there may be limitations that need to be considered. For example, the Marketo SOAP Connector does not execute Upsert operations as expected. It updates existing records correctly, but cannot create a new record when there is no matching record to update. Consider the following:
- Can Virtual fields be used to compensate for a limitation? See the Connector For Salesforce Help for an example of how Virtual fields are used.
- Are there any entity or field names that differ between the API and the application's UI? See the List Entity in the Connector For Pardot Help for an example.
- Are there any read-only fields that may cause a problem?
- Are there any required filters?
- Are there any conditionally required fields?
- Are there any read-only fields?
- How are unmapped fields handled when updating a record? Are the unmapped fields overwritten with null/blank values or left alone during the update process?
- Are there any dependencies between entities? For example, if you create a record for one entity do you need to create a corresponding record for another entity?
- Are there any special operations that apply to this entity?
- Is there any time lag between when the Connector modifies data and the changes display in the UI of the target datastore? For some applications, changes made via the API do not display immediately.