Azure Modern Data Warehouse for Sedgemoor District Council

About Sedgemoor District Council

Sedgemoor District Council in rural Somerset serves its population of 113,000 with services including housing, planning, recycling, rubbish collection and council tax.

The challenge

The Revenues and Benefits area of the council was looking to modernise. This area is made up of council tax, business rates and housing benefits. The existing systems were built in the early 90s and were due for modernisation. Transparity was engaged to assist in developing a new Microsoft Azure-based system.

The primary briefs were to:

1. Assist in forming an Agile delivery team from current members of the organisation.
2. Familiarise the whole team with the Agile ceremonies and the expected output from each.
3. Assist the business analysts with organising business engagement sessions and producing User stories that the developers could work on.
4. Assist the development team with organising stand-ups and incorporating practices such as code reviews into their routine.
5. Provide technical assistance and know-how wherever required in the development lifecycle.
6. Define the overall solution architecture of the new system.

The solution

Our technical lead provided oversite and consultancy for Sedgemoor District Council’s in-house developers. This partnership enabled them to make use of their in-house team, save on costs and have in-depth knowledge of their system, which aids in support or new features that may crop up over the years; whilst simultaneously gaining our custom development experience and Azure expertise

Architectural Overview

Technology

The new Business Rates system is built with a microservice-based architecture for all core business logic, developed in the C# and TypeScript languages on the .NET Core and Angular frameworks and deployed on Azure Serverless technology.

Each microservice is consumed through an Open API defined REST/JSON Web API and controls its own data storage repository. The main UI/UX is developed using Angular.io

Each service is implemented following Clean Architecture (CA), Command Query Responsibility Segregation (CQRS) and Domain-Driven Design (DDD) paradigms.

Azure cloud services are used for all the required infrastructure, are designed to support monitoring, and include at least the following services: App Services, App Insights, Azure Storage, SQL Azure, Azure Functions, Azure Logic Apps, Azure key Vault.

For the production environment, the following cloud services were required: Web Application Firewall, API Management.

All infrastructure is automated and managed programmatically using an Azure Resource Manager (ARM) template. All deployments are managed using Azure DevOps pipelines.

Each microservice is independent of the others and represents a distinct part of the system.

The initial microservices that were needed for the Business Rates system were the core Business Rates solution and a separate solution, which provides address data. The data models were still evolving and at the time of writing Azure storage was used as the baseline default for storing documents, integration with an on-premise  Electronic document and records management system (EDRMS) system was planned as a later enhancement.

The Business Rates API is consumed through its REST/JSON API by the main Web Application and can also be made publicly accessible to other third-party systems. It manages its own data storage or for commonly managed data will delegate its data storage to other microservices, e.g. the microservice for address data.

It may at some point also raise events to a service bus for other interested microservices to listen for and it may call other microservices via that services API client. A further example solution for this scenario, for a People Register, was developed and provided by the Transparity technical lead.

Microservices are deployed independently and automatically through a deployment pipeline.

Each microservice is deployed to Azure App Services and can be set to scale automatically according to the load for the service.

Azure Web Application Firewall was recommended for production environments to security check all calls to the system. This prevents standard security exploits such as SQL Injection. Again, the setup of this component was demonstrated by the Transparity technical lead.

It was expected that Azure API Management will be implemented in production to define and manage all APIs exposed by the system and grant access to third parties. It can manage/enforce security and throttle the number of requests to prevent, for example, a third-party system overloading the main system. All calls would be managed by API Management including calls between micro-services.

All APIs are designed in accordance with the Open API standard. This means a Swagger file is produced to define the schema and methods exposed by the API. This has several benefits including:

• Each API is self-documenting
• It is easy for a developer to work out how to call the API (A client UI, which also describes all inputs and outputs, is autogenerated for interacting directly with the API)
• Client code for interacting with the API can be auto-generated in a number of languages, reducing development time when building the front-end UI to interact with the API
• Tools such as Power Apps and Power Automate can work out how to call the API automatically without writing any code

Azure Application Insights is used for logging and exception handling and can also be configured for monitoring purposes, raising alerts to interested parties.

Azure Service Bus could be used to manage asynchronous message handling between distributed systems using the publish subscribe pattern and was implemented in the sample People Register solution.

Additional coverage of other Azure Services that could be used as part of the system were provided. Depending on the future requirements of the system other services could be used such as:

• Azure Sentinel: For further automated security checking
• Azure Cognitive Search: For text search
• Power BI Embedded: For reporting
• Azure Data Factory: For managing data loads

The whole system is set up with automated deployment pipelines built into Azure DevOps and using Azure Resource Management (ARM) templates to create all the Azure resources for Dev, Test and UAT and LIVE environments.

Example automated regression tests for API and UI/UX, produced from the Gherkin syntax used in the User story scenarios and Specflow for implementation, were included.

Azure Serverless technology is used throughout to avoid the need for any physical or virtual servers. No servers mean that there is no need to patch, backup or manage servers, greatly reducing management overhead. It also makes scalability significantly easier using built-in cloud features such as the scale-up and scale-out features of App Services.

Developers were encouraged to follow a standard Agile development process, which includes adherence to the principles of Security by Design. This is achieved by ensuring that all developers follow OWASP principles (Open Web Application Security Project) see https://owasp.org

The solution is developed using .NET Core 3.1 Long term support (LTS), which is open source. The principle of using open source, unless it absolutely cannot be avoided, should be adhered to. To avoid falling behind and risking obsolescence, every effort should be made to ensure the solutions are upgraded to each iteration of the next LTS version of .NET as early as possible.  The next scheduled LTS release is .NET 6 in November 2021.

GDS API and technical data standards (https://www.gov.uk/guidance/gds-api-technical-and-data-standards) and the Government Design System (https://design-system.service.gov.uk) were recommended to provide a baseline style and design.

Solution Architecture

The Business rates solution is structured in a way that leverages Clean Architecture (CA) and applies Command Query Responsibility Segregation (CQRS) and Domain-Driven Design (DDD) concepts.

This architecture structures the solutions in an ideal way for evolving out aggregate roots into their own future microservices as the system grows.

In CA, the Application and Domain layers are at the centre of the design and represent the ‘Core’ of the application.

The Domain layer contains enterprise logic and types, whilst the Application layer contains business logic and types. The difference is that enterprise logic and types can be shared across many systems, whilst business logic and types would typically only be used in this application.

Rather than have the Core depend on concerns such as data access and infrastructure, we invert those dependencies. So, Infrastructure and Presentation depend on Core but Core has no dependencies.

This is achieved by adding interfaces or abstractions into Core that are then implemented by the outside layers.

The Infrastructure layer contains all external concerns and the Presentation and Infrastructure layers depend only on the Application layer.

With this architecture, all dependencies point inwards. This results in a design that is:

• Independent of frameworks
• Testable
• Independent of UI
• Independent of Database
• Independent of anything external

We start with 3 layers but may include more (such as messaging, integration, etc) as the solution grows, provided we always remember to keep the dependencies flowing inwards.

The outcome

Isolating Core in this manner can mean the difference between an application that quickly becomes monolithic and needs to be completely re-written after 3 years and one that lasts 20 years and can be progressively updated with minimal effort to leverage the latest presentation and database technologies.

Both the new Sedgemoor Business Rates and Address solutions successfully adopted this architecture, which will now make it easier for the organisation to transition to developing the next big products on their roadmap out of these initial products – the Council Tax and Housing Benefits systems.