ING

Reducing Time to Release from 6 Weeks to 34 Minutes

Download the Case Study

Orchestrating Release Pipelines to Optimize Continuous Delivery

Reducing time to release from 6 weeks to 34 minutes

The Challenge

Slow, manual and error-prone processes that offered no visibility into release status and bottlenecks, and no way to collect release metrics.

The Solution

Automate and orchestrate the release pipeline with XL Release in order to accelerate software delivery, ensure that ING development and compliance standards are met, and provide visibility into every step of the process.

Results

  • Reduced release time from test environment into production from 6 weeks to 34 minutes  
  • Deploy twice a day to Dev environment to get fast and early feedback 
  • Predictable results: only 1 incident in 8 months, instead of multiple incidents with every release

The Environment

Primary language is Java with an Angular JS front end. Environment also includes Jenkins, Maven, Artifactory, GitLab (replaced GitHub), Sonar, Fortify SCA, CA Release Automation, Protractor for GUI testing, FindBugs and Docker

Background

ING Bank has more than 33 million customers and locations in more than 40 countries worldwide. The Lease IT team supports 5 major countries: the Netherlands, Belgium, Spain, France and the Czech Republic.

As the banking industry becomes more and more competitive, the IT group has their hands full keeping up with ever-changing and urgent business needs across countries. The team knows they need to work smarter to survive, so they use the most modern processes and tools to stay ahead. Continuous Delivery (CD) and DevOps practices are very high priorities for them.

Automating with Visibility and Control

“Our overarching philosophy is to automate everything that can be automated and get rid of all manual activities,” said IT Lease team Manager Andréas Prins. “Our goal was to have a fully automated release pipeline.”

As their Continuous Deliver process matured beyond basic deployment and test automation, they looked for a release orchestration tool that could integrate into their complex environment and provide automation, control, and visibility for their entire CD pipeline.

“We needed a fully automated pipeline with high test coverage to ensure we can deploy functionality for a speci c country without disturbing the other countries. A highly automated and transparent delivery pipeline is crucial in this situation.”

“We also needed to make sure we enforced ING’s process requirements and stringent standards for compliance and security. And we needed a way to get feedback on our Continuous Delivery process so we could continuously improve.”

"XL Release reduced our release time from test to production from 6 weeks to 34 minutes and pipeline visibility it provides are crucial for us to meet our Continuous Delivery goals."
- Andréas Prins, Manager, Lease IT, ING Bank

Streamlining the Delivery Cycle with XL Release

Using XL Release, the team has now created a pipeline ow to give fast feedback on their fully automated release process. For example, they now automatically:

  • build and deploy stubs
  • build the product
  • perform a security analysis
  • deploy the product
  • execute different types of functional testing 
  • perform smoke tests and notify the team

The results are staggering. “XL Release reduced our release time from test to production from 6 weeks to 34 minutes,” Andréas noted. “The release orchestration and pipeline visibility it provides are crucial for us to meet our Continuous Delivery goals. It helps us see where things are in the process, get up-to-date status, and capture metrics. But improved time to market and better release quality are the tool’s biggest benefits.”

When the business makes a request, the IT Lease team can now meet that need quickly... rst in one country, and then in all 5 of them. This agility translates into accelerated and greater revenue for the bank and better ability to compete... all while keeping risk under control. “We needed the ability to release independently for each country, while having the same code base in production.”

XL Release will also help them to achieve goals on meeting audit and reporting requirements. Like all banks, they need to keep detailed records, so they are looking to XL Release to supply the following information:

  • what changes were made – when and by whom
  • who approved what and when
  • evidence of the change process
  • be able to document their incident process
  • show relationship between a deployment and an incident

“Our releases now have predictable results and are more robust. We’ve only had one incident in the last 8 months, and we used to have them with every release. Now it only takes us minutes to deploy to production. We deploy twice a day to development to ensure the code we develop ts perfectly in the complete code base. And security is now a built-in part of our development,” Andréas said.

“As an extra bonus, we now have highly motivated developers. They can spot and x problems fast, and they don’t waste their time doing dull manual tasks. So they get a big sense of satisfaction from their work.”

Continuous Delivery is a team e ort for the Lease IT group. When you walk in, a dashboard shows if a release is red or green, so everyone can see the current status of all steps at any time. “XL Release gives us a great view into our complete CD pipeline and allows us to see bottlenecks, so we can continuously improve the process,” said Andréas.

This agility translates into accelerated and greater revenue for the bank and better ability to compete... all while keeping risk under control.

Looking Ahead

Going forward, ING Bank is looking to expand its use of XL Release in other areas.

Automating manual run books and reporting on the process:strong> A lot of teams have manual run books they use to manage the release process. While in some cases they cannot abandon these run books entirely, putting the process in XL Release allows teams to easily see status and to capture important metrics so they can continue to improve.

Helping the COTS teams improve e ciency: The COTS development teams support new releases of software towards the di erent countries. They have heavily manual processes, a lot of custom builds, and fewer tools to do the work. XL Release can really help them measure process steps, alleviate manual pain points, and eliminate waste in their software delivery pipeline.

Orchestrating the overall pipeline for all teams: Right now there’s no orchestration for the overall pipeline across the Lease team, so there’s no feedback at the highest level. The goal is for XL Release to manage the whole pipeline, which consists of many pipelines across teams.

Managing risk and improving compliance: ING Bank has extensive risk processes, but they are difficult to enforce. By automating more pipelines with XL Release, the team ensures no one can change the ow and steps cannot be skipped. In addition, XL Release provides the full audit trail of who does what and when,and significantly reduces audit complexity and cost.

Automating sign-off procedures: Right now, managers need to sign off on all releases even though they typically have no more data than the automated tools provide them. The goal is to have a process that checks to ensure all steps are green and then automatically signs off , which will relieve a signi cant bottleneck in the release process... freeing up the manager to focus on more important things than rubber-stamping releases.

“We’re looking forward to continuing to streamline our processes with XL Release,” said Andréas. “ING Bank has so many areas that automation and visibility can help us improve.”