Part of what makes the Kinetic Platform so uniquely powerful is its ability to integrate with nearly anything. The way that we enabled this over the years has taught us lessons as it has evolved. We took the time to look at what some “next generation” features for our workflow engine might look like and a few things were obvious:
- A builder may need to know three languages: Java, Ruby, and Javascript.
- The vast majority of our integrations are REST or similar HTTP-based integrations.
- Slightly more complex integrations, such as LDAP and SQL, are fairly standard but complicated by the way our workflow integrations (called handlers) are written.
- The manner in which we enable integrations has led to a massive amount of essentially duplicative code written for each one.
We then began analyzing these integrations, and indeed most of them are nearly identical and nearly all of their code is the same. But because they’re written as open-ended Ruby programs that report back to the engine a block of XML, their inputs and outputs were not always consistent.
This led us to begin writing a new tenant service called Integrator, which we hope will become the successor to handlers, bridge adapters, and the Kinetic Agent application. In this process we were faced with a myriad of decisions. Bridges are written in Java. Handlers and workflow are written in Ruby. The rest of the application builder code is JavaScript (portals/bundles, expressions and mappings on forms, etc). Ultimately, we ended up choosing JavaScript as our evaluation engine.
Why not Ruby?
Ruby is a powerful language, and it was originally chosen for the workflow engine because it tends to be an expressive language that strives to be “legible.” But modern Ruby begins losing this legibility very quickly, especially in more complex and technical code. Another reason we chose Ruby is because it has a vast ecosystem of third party packages, called Gems, that workflow builders can use in their Handlers. There are a couple key problems with this approach, however.
The first, and potentially largest, is that adding third party packages as dependencies brings with it a lot of other considerations. Does the dependency you want to use conflict with another handler’s dependency (which you may not have written)? How many dependencies does that one package bring with it? Are all of them being scanned by a security scanning application such as Snyk or Fortify? Will upgrading one have an unpredictable impact upon an unrelated handler (since dependencies are “first come first serve”)?
With hundreds of handlers, having a grasp on what dependencies exist in that ecosystem and within the engine itself is, in reality, a tremendous challenge.
The second is that even after adding some helpers to the engine to handle common scenarios, in the end each handler is a tiny little program that requires a life cycle. Compounding the fact that each of these tiny little programs have a lot of the same (in many instances flat out copy and pasted) code means that if you introduce a bug in one handler, you may have propagated it inadvertently to many others.
To put this code maintenance into perspective, there are more than 480 handlers in our current integration library, with each one averaging over 200 lines of code (not including dependencies, test cases, etc.); which means part of our integration library is nearly 100,000 lines of code.
Why Not Java?
While the majority of our applications are written in Java, only a handful of our integrations are written in Java—mainly Bridge Adapters and Filestores, which are used for connecting forms to third party data sources and persisting attachments.
In comparison, while there are 481 handlers available for enabling workflow builders, there are only 49 Bridge Adapters shipped by default with the platform for use in Forms and Bundles. The reality with our Java-based integrations is that it is exceptionally rare for a partner or customer to build their own.
The fact that this side of integration development is rarely seen by our partners and customers is due to a purposeful decision we made to ensure that the majority of work can be accomplished by a technically minded individual but not necessary require a seasoned, experienced Java developer. This is because this ecosystem comes with a whole host of specific knowledge requirements, tooling, and complexity.
This level of development to build your solution is considered a “last resort.” Additionally, choosing Java does not guarantee that we alleviate the code debt problems noted in why we did not choose Ruby: because it will effectively be a “mini-application,” it will have the problem of code management, it will have numerous third party dependencies that will need to be monitored and upgraded, etc.
Why JavaScript?
We ended up choosing JavaScript for a few simple reasons. In fact, we chose JavaScript and a standard for templating. JavaScript is a language that has traditionally gotten a bad rap. However, after surveying how our platform is used, there’s one thing that is abundantly clear: as long as we are a web-based platform, we will never shed JavaScript as a language that we require builders to know.
In this redesign, we have taken a lot of inspiration from other similar tools and from our own tool. Builders will often create large JavaScript events, but more often than not the simple things, such as visible conditions, can be represented as very simple JavaScript expressions. We also use templating with JavaScript, however we have in the past rolled our own solution.
Realistically we could have pivoted our usage of Ruby to work in a similar manner to what we intend, but in the end we’re not confident that Ruby will be the long-term language of choice for our workflows. Reducing the footprint of what builders are required to know in order to be effective is crucial.
What we ended up doing to solve the handler-spawl problem was create a new tenant-based application that handles 80% of the “hard stuff” in facilitating HTTP, SQL, and LDAP requests. This tool, written in Elixir, uses JavaScript to expose dynamically configurable items to builders.
Templates
We’re using a templating technology called Mustache. Many developers will be familiar with Mustache syntax and derivatives such as Handlebars: . There are some tasks that don’t necessarily require the expressiveness of a full JavaScript expression. For example, if you were defining the URI path for requesting a specific form definition from our platform: /app/api/v1/kapps//: the engine can parse this simple template and then inform someone attempting to use this integration that in order to use it, the workflow or form executing must provide a “kappSlug” and a “formSlug”. If this were written as a JavaScript expression, we would have a difficult time parsing the AST in order to determine what it believes to be arguments to the expression.
Expressions
There are going to be situations in which we will need the expressiveness of a full JavaScript expression. A couple example scenarios that came up in our early designs were things such as transforming the output of a call and evaluating dynamically whether a call failed when it may not be “standard” or “obvious.”
Consider, for example, the content of transforming the output of a call. Your API call may be to retrieve the valid zip codes for a specific state. If that API returned a body like this:
{
"stateKey": "MN",
"stateName": "Minnesota",
"zipCodes": [
{
"zipCode": "55904",
"city": "Rochester",
"state": "MN"
},
// and many more...
]
}
Now, this is all interesting data, but if you happen to know the only information that users of this integration will need is the zip code itself, you could write a transform expression like this:
({ data, metadata, errors }) => data.zipCodes.map(zip => zip.zipCode);
Which would return only an array of the zip code strings.
Another example concept we have considered in the past is the idea of dynamically identifying if a REST call failed. You can usually assume that at an “adapter level,” certain status codes are failures such as a 500, representing some sort of error on the remote server that you cannot resolve from the client side. There are other scenarios in which a valid “200” response is actually an error. We presume this because we cannot predict how third-party APIs will behave. In this example scenario, the third-party server may return a 200 but have an object like this: { "error": "failed_to_create", "message": "A widget with this name already exists." } . The expression for identifying this as failed could be as simple as:
({ data, metadata, errors }) => data.error
Another hypothetical example is a successful retrieval of an API that has no results. While to the third-party server this may seem normal, you may highlight this as a failure because it cannot be an empty set without representing something that is improperly configured. The expression for identifying this could be as simple as:
({data, metadata, errors }) => data.widgets.length < 1
TL;DR
While we have integrations in Ruby and Java already, we know that JavaScript is here to stay in our platform, and it makes sense to begin consolidating to make it easier for new developers. The way we are utilizing JavaScript is meant to continue to offer a powerful means of writing custom integrations without requiring seasoned software engineer levels of expertise. It is meant to be able to give the most power and flexibility with the least technical cost.
Cassy Lueneburg