In this blog we take a look at what makes up Azure Integration Services, its history with BizTalk Server, our work in integration and the Ballard Chalmers’ Data Integration Hub Framework.
THE HISTORY OF AZURE INTEGRATION SERVICES
Azure Integration Services is the latest Microsoft Azure toolset for creating data integrations between IT systems. It replaces Biztalk Server which has existed since 2000.
Biztalk has served well, but it can now definitely be categorised as a legacy system. It needs to be installed, configured, patched and managed by professional system administrators. If resilience and scalability are needed, then at least two Biztalk Servers are required and these both need to be Enterprise edition, all of which increases the cost significantly. Finally, BizTalk uses SQL Server as its data store for all message queues and so at least one SQL Server needs to be involved in the deployment and that requires professional system administrator support as well.
Notwithstanding all that; Biztalk does have all the features needed for integrating enterprise systems including:
- Adaptors for just about every data source including: FTP, HTTP, SQL and systems such as Oracle, SAP and EDI including AS2, X12 and Edifact
- Schema management, and mapping
- Message flow orchestration and transaction management
Microsoft briefly toyed with producing a cloud services version of BizTalk called BizTalk Services, but this had greatly reduced functionality and had overlapping functionality with the new and more flexible cloud services already in Azure. So, BizTalk Services was eventually dropped in favour of what is now called Azure Integration Services.
WHAT IS AZURE INTEGRATION SERVICES
Azure Integration Services is not a new Azure service but instead is a collection of other Azure services. This makes a lot of sense, why try and produce a new service that contains functionality that already exists in other services, just add new functionality where needed and take advantage of the whole of Azure for everything else.
The core components of Azure Integration Services are detailed here: Azure Integration Services | Microsoft Azure and consist of:
- Logic Apps: Provides data connection and orchestration features
- Service Bus: Secure message queuing service
- API Management: Provides secure control and throttling features for APIs
- Event Grid: Event routing system
- Azure Functions: Simplifies complex orchestration problems with event-driven serverless compute
- Azure Data Factory: Visually integrates data sources to construct ETL and ELT processes
Plus, you can use any other Azure service that you wish such as SQL Azure, Azure Tables, CosmosDB, Blob Storage and much more.Logic Apps provides the following main features:
- Orchestration/Workflow management of message flows using a graphical user interface. The example Logic Apps orchestration below, shows a simple email approval workflow:
- Connectors allow it to connect to most data sources. A lot of these have been inherited from BizTalk and refactored for the cloud, for example:
- EDI connectors for AS2, X12 and EDIFACT: See B2B enterprise integration workflows – Azure Logic Apps | Microsoft Learn
- Oracle and SAP etc: See: What are connectors – Azure Logic Apps | Microsoft Learn
- Gateways for accessing data on-premises. The gateway is installed on-premises and allows Logic Apps in the cloud to manage data that is stored in on-premises servers, without the need for a VPN
In addition, because Azure Integration Services is built on Azure using Azure services, the system is:
- Fully managed by Microsoft and so the level of system administrator support is minimal, with no need to patch, backup, or anything like that
- New code releases can be automatedly deployed using the release pipeline features built into Azure DevOps
- New deployments can be completely managed using Azure Resource Manager (ARM) templates, which can automate the setup of all the Azure Services
In summary, Azure Integration Services offers a greatly reduced administration overhead, compared with BizTalk and other systems.
TRANSPARITY DATA INTEGRATION HUB FRAMEWORK
Transparity have carried out a considerable number of data integration projects over the last fifteen years. Many of the earlier ones were based on BizTalk, some were just developed in C# as extensions of existing systems, and more recently they are based on Azure Integration Services. Although each of these projects involved different source and target systems and data sources, with different schemas and workflows, there is a common pattern to all of them. Some examples of these are:
- Different systems have different data schemas and there needs to be a mapping between them. Usually, this is done via a central Canonical schema
- Different systems use different IDs for reference data. For example, an order from one system may refer to a particular product using product id 0001 and another system may refer to the same product as product id A021. There needs to be a conversion between these
- There is generally some standard business logic and validation that needs to be implemented against the standard Canonical schema, that it is common regardless of the source of the data, and will not change even if one of the source or target systems is replaced with a different system
- Some systems can call an API or support WebHooks and so a secure API is needed with firewall protection and control over who can call it, and throttling configuration to protect against one system overloading the system at the expense of others
- Other systems are not API centric and so the system needs to be able to source to target data using files, databases, SOAP APIs, or whatever
- Finally, the system needs to able to be deployed and released automatically to Azure using automated ARM Templates and Azure DevOps release pipelines
Having done all of the above a number of times for different clients Transparity decided there was an opportunity to improve the speed and quality of future integrations by standardising the Data Integration Hub Framework that is used for these projects. The architecture of the resulting framework, which is 100 percent based on Azure Integration Services, is shown in the diagram below:
With respect to the framework, it can be observed that:
- Some of the logic is encapsulated in Azure functions and in other cases, it is Azure Logic Apps depending on which is easiest, fastest or cheapest for the particular job at hand
- Web Application Framework (WAF) is used if external systems are accessing the Hub
- Blob storage is used to store messages in addition to Service Bus for messages greater than 256 KB
In addition:
- Standard ARM templates have been written for each of the standard components such as Azure Service Bus to speed up creating new systems.
- Standard release pipelines have been deployed to speed up releasing new versions of the system
- Standard monitoring is added to the system to track when issues occur or if the system is performing slower than expected
SUMMARY
Azure Integration Services is a great step forward and a world-class cloud-based system for integrating systems in the cloud and on-premises.
When used in combination with the Transparity Data Integration Hub Framework, even the most complex enterprise data integrations can be created easily and risk-free. All using a tried and tested architecture and components based on Azure Integration Services backed up with fully automated deployments and monitoring.
