CloverETL is now CloverDX - Learn Why

Back to CloverDX Blog on Data Integration

Integrating Legacy Systems With Salesforce

Posted by Jay Benedetti on Mar 7, 2017 3:55:02 PM

Have you tried integrating data from a 3rd party application into Salesforce to give your sales team a 360-degree view of your customers? If the answer’s yes, my guess is it most likely involved purchasing an extension of Salesforce with a 3rd party application (via the AppExchange) or by creating a custom application. But what if there’s no AppExchange listing for your particular need? Wouldn’t it be useful if you could still hook that system to your Salesforce ecosystem and benefit from seeing that additional data?

If you’re a CloverDX customer (or are thinking about it, especially after this blog), you can use CloverDX in this broker type of role. CloverDX can not only integrate a legacy system that's not natively connectable to Salesforce, but also bring added value by allowing you to massage and adapt the data to best suit your users' needs along the way.

In this blog, my goal is to better acquaint you with Salesforce integrations in general. I’m not going to dive into any technical solution too deeply. Instead, I’ll walk through the value of integrating data from a legacy 3rd party application into Salesforce using CloverDX, share my experience with it, and offer some advice and suggestions that may help you in the future. But first, let me set the stage about where the idea came from.

Migrating Legacy Systems to ERP: Read The Case Study 

Where It All Began (For Us, Anyway)

Here at CloverDX, we recently held a hackathon to brainstorm fresh ideas, solve old problems, and come up with new problems to solve. One topic suggested by our Sales Engineering team stood out, which I want to share here: integrating data from a legacy system with Salesforce.

We had been wanting to improve how we connected our support ticketing system (OTRS) to Salesforce for a more well-rounded customer view without forcing Salesforce users to log into OTRS to look up their contacts manually. As you can guess, OTRS doesn’t come with a ready-made Salesforce component. So, this was a great opportunity to not only address a business need of our own, but also tackle an interesting technical challenge. Two great things for a consultant like me. By the end of the two-day hackathon, we developed a solution that worked! Sure, it wasn’t production ready, but all the essentials were in place. It felt good to build something useful in such a short, intensive timeframe—and it didn’t hurt that we ourselves could benefit from it.

The Value of Integration

Why bother with Salesforce when you have a system that works perfectly on its own? Before I jump into some advice, I want to highlight the value of integrating your legacy system's data with Salesforce. After all, if you don’t get buy-in from management or the business team, your project will not get off the ground.

In our case of integrating data from the support ticketing system (OTRS), benefits were quite clear:

  • A 360-degree view of customers for the Sales team. This should result in more revenue and happy customers.
  • A decrease in the time consultants spend proactively updating the Sales team on the outstanding support tickets. This will allow consultants to be allocated on external, billable projects, rather than focusing on internal initiatives.
  • An increase in the productivity of the support/development team. The integration will automatically handle moving the necessary data into Salesforce, so the team won’t need to get involved.
  • With all of the above, the management team will see an increase in revenue and a decrease in operational costs.

Your list might look slightly different, but the reality is that Salesforce quickly becomes the central piece of software (or “no software” to be precise) within business units, and data tends to gravitate to it. Users of Salesforce demand more reach and more visibility from within the comfort of the familiar Salesforce interface.

What Does This Type of Integration Look Like?

So how is an integration between Salesforce and a legacy system different with CloverDX compared to a typical AppExchange solution? Well, in short, you get much greater flexibility in exchange for investing a little extra time and work.

Typical modern systems come with out-of-the-box integration modules with a fairly simple configuration, yet limited capabilities. CloverDX, on the other hand, is a platform that lets you design and implement arbitrary functionality. In such a scenario, CloverDX pulls data from the legacy system, transforms, joins, and aggregates it based on the business requirements, and pushes the results to Salesforce. You can even go as far as having a tiny custom application sit inside Salesforce (written in APEX, Salesforce’s programming language) that uses CloverDX to get data, then provides some advanced functionality on its own. All that data movement in and out usually happens by regularly polling either system, or is based on triggers. CloverDX typically sits somewhere in between, either hosted in the cloud or on-premise, close to the legacy system.

Reducing Salesforce Data Loading Time - Read The Case Study

Key Considerations During Planning

Once you identify the need for your system’s data integration into Salesforce and get the buy-in from the business stakeholders, you can get started on the project. Here are a few things we considered during the hackathon when we began moving forward with a solution.

Identify the exact business need

The simplest form of integration is to have CloverDX populate some Salesforce objects, standard or custom, without coding. However, whenever you’re uploading data into Salesforce, you want to be selective with it. You need to assess what data you have in your source and what information you actually want to show in Salesforce.

In the case of our support ticketing system, we had to think whether we were fine with just presenting aggregated stats (like open/closed tickets ratio, average response time per customer, etc.) or if we wanted to list individual tickets with all their details and history right in Salesforce. It’s often about finding the balance between providing enough valuable information versus overloading. In my experience, you want to give the sales team the most important information only—just enough that they have a complete, well-rounded view of the customer. Anything else is superfluous.

Also, keep in mind that Salesforce storage does not come free. You’re paying Salesforce based on the amount of data you store (in addition to number of seats), so it wouldn’t be in your best interest to upload anything unnecessary.

Understand the legacy system

There’s no magic prescription I can offer here. Figuring out how to connect CloverDX to your legacy systems takes, well, a technical understanding of your legacy systems. I recommend asking yourself the following questions: Is there a backend database that CloverDX could read from? Is there a Web Service API that CloverDX could request data from? Is everything stored in files (Excel, CSV, etc)? You have to determine how to expose that data before you can start extracting and integrating into Salesforce. CloverDX packs a wide range of connectors, from the ability to read various files and databases, to direct connections to applications via standard API endpoints (REST or SOAP). So I bet it has you covered in this department. Many of the legacy systems aren’t designed with interoperability in mind, so you might need to get creative to find out how to dig the data out.

Get the right view in Salesforce

In cases when you choose not to plainly dump the source data in Salesforce, some kind of processing must happen between the point where the data leaves the legacy system and the point it shows up in Salesforce. This additional processing can also be done with CloverDX (e.g. computing the average response time across all tickets per customer), bringing value to the raw data—something that might be really tough, if not impossible to achieve in either system alone. Apart from aggregations, CloverDX can perform smart filtering or data enrichment to create new views of your data once it’s in Salesforce. This streamlined integration leverages CloverDX from beginning to end, allowing you to transform data from your legacy systems to present the right business view to fit your needs.

Try CloverDX

Building your Solution: Problems, Suggestions, and Timelines

Now you can start building your solution. Here, I outline a couple of problems that you should look out for along the way, as well as offer a few suggestions based on our experience to ease ongoing development, as well make the integration more valuable.

Problems

  1. Matching objects and records from your legacy system to Salesforce will be challenging. This will be your most difficult piece of the integration due to the complexity of matching objects between a legacy system and Salesforce. The object side of the issue is about finding corresponding elements in both systems. For example, the legacy system might carry names as a single value, while Salesforce natively works with first and last names. Some logic needs to be devised to work around that.

  2. On the record level, matching can get even worse. How should the integration logic determine that “Acme” in the legacy system is actually the same company as “ACME Inc.” in Salesforce? It’s unlikely that there’ll be a common key. We use a couple of techniques to work this out: 1-1 matching, 1-1 matching but in lower case, fuzzy string pattern matching, eliminating common substrings (e.g. “Inc.”) etc. Once you have programmatically matched all cases, you can use the output mapping that has been generated and manually edit the mapping for objects that you identify as a match.

  3. Be aware, you have a limited number of Salesforce API calls in order to write your new data into Salesforce, so think about how you can limit your requests. Fun fact: to ease this API limit pain, our development has gone the extra mile to optimize the CloverDX Salesforce connector to be limit-friendly. The CloverDX Salesforce Bulk API is, per its name, for bulk loading to Salesforce in order to minimize API calls.

Some helpful suggestions

  1. Use Salesforce Chatter. We were mostly working towards augmenting Accounts, so in our case, we could alert the account owner of any changes that were made to the account using the Salesforce API. This can be beneficial to the account owners if they have their notifications setup to alert them of any changes to the account as soon as they occur.

  2. Don’t touch production! Use a development sandbox. As soon as you have data sources identified and have made them accessible, you can set up a Salesforce sandbox for development. I would recommend making a fresh copy of your production environment to a development sandbox so you can use real data. You’ll be touching/enhancing essential data that drives your business, and I cannot stress enough that you do not want to have to scrub your production Salesforce environment for erroneous data, if your initial tests go wrong. Believe me, there’s enough junk already, you don’t want to complicate things even further!

  3. Test the solution for a period of time much like you would for any solution. This way, you can identify any bugs in the solution that need to be addressed prior to rolling out into production. My goal during UAT is to try to find anything and everything that could go wrong before the go-live rollout.

How long should it take?

  1. For our integration with Salesforce, we eventually spent about 10 days total for the development of the solution (including the hackathon time) for 1 Salesforce object in order to provide an aggregated view on open/pending and closed tickets with corresponding external links to OTRS. The implementation covered the following tasks: system access, mapping between Salesforce and our legacy system, business logic, loading into Salesforce, and automatically updating Salesforce Chatter.

  2. The good news is if we need to integrate another Salesforce object or legacy system, we’ll only need to tweak the solution for a new mapping, rather than set up a completely new workflow.

A Successful Salesforce Integration

As you can see, there are a number of decisions and tasks that must happen in order to integrate your legacy data into Salesforce, but I think you can see that it’s achievable in a short period of time. I’m glad I can share some of our insight with others looking to integrate legacy data with Salesforce too. If you don’t think you have the technical expertise or know-how for a successful implementation, let us know. Our team would be more than happy to help.

Try CloverDX

Read the case study - Migrating legacy systems into Workday

Data integration software and ETL tools provided by the CloverDX platform (formerly known as CloverETL) offer solutions for data management tasks such as data integration, data migration, or data quality. CloverDX is a vital part of enterprise solutions such as data warehousing, business intelligence (BI) or master data management (MDM). CloverDX Designer (formerly known as CloverETL Designer) is a visual data transformation designer that helps define data flows and transformations in a quick, visual, and intuitive way. CloverDX Server (formerly known as CloverETL Server) is an enterprise ETL and data integration runtime environment. It offers a set of enterprise features such as automation, monitoring, user management, real-time ETL, data API services, clustering, or cloud data integration.