DevOps on scale in multilocation teams – a story worth sharing

Once upon a time there was a development team. The team members were frustrated with inefficiencies, because they had high hopes: they wanted to become the most efficient team in the world. To achieve their aspirations, they rolled up their sleeves and got to work. They defined the responsibilities of each team member, developed a way to communicate closely, and even removed physical barriers in their workspace and drank coffee around the same table at the same time. They learned to know each other very well, learned to trust each other and became very efficient in their work, delivering great results for the business. Everybody was happy (for a while).  

But as the system and development ecosystem grew, also development teams became more distributed and it impacted teams’ collaboration. Developers could no longer interact in the same physical space. Teams with members from different locations needed to focus even more on communication. Ensuring that business and customer knowledge existed in all teams was crucial. Everyone needed to understand what they were contributing to the big picture.  

If the frontend works, so does the backend 

The teams learned that it was highly important to define the frontend and user experience (UX) on a very detailed level, in close interaction with the customer. Business analyst and UX designer roles were critical, really breaking down different parts into high quality stories for developers, all the way to the acceptance criteria. When developers understood the big picture, they were then free to choose how to deliver it.  

Defining front-end in a detailed way with UI prototypes helped to avoid and catch possible faults in the earlier phases.  

By now every member of every team appreciated that if the front-end works as planned, most often so does the backend too.  

Facing different maturity levels between teams in a multivendor environment 

Our story doesn’t end here, for the system grew even more, as it gained more connections to other systems provided by other vendors. Instead of one vendor, there were soon ten vendors and multiple teams serving the business. This uncovered challenges in communication and testing between teams. 

We now faced a key challenge in scaling DevOps: realizing different maturity levels between teams. As there are always dependencies between different parallel development tracks, teams are dependent on getting test environments and test data from each other. Some projects needed to wait for their test environments for months, while some got theirs within a day. Especially legacy systems and integrations were problematic. 

The teams learned that capabilities in all teams need to be on a level that allows deployments in the same rhythm. They grasped that advanced automation enabled committing to 30-40 deployments per week – whereas without automation, the pace would be as little as one release per quarter. 

Lessons we learned 

The DevOps transformation journey that I took with our customer really drove home that the key to success in scaling DevOps lies in lifting process maturity, development environments, and automation capabilities to the same level in all teams. There is no single magic trick that makes this happen, but there are certainly valuable lessons that are worth taking into consideration: 

• Invest in skilled Scrum masters / Release train engineers / Site reliability engineers – and ensure good interaction between them to ease communication between teams. 
• Keep your lead architects close to the business to ensure business understanding. 
• Investigate and develop operational processes throughout the entire house. 
• Focus on test and deployment automation to speed up processes. A shared system team could support all teams to use automation correctly and maintain it. 
• Culture is the key: everyone must feel responsibility not to break or interfere with other’s builds and see the value of common processes and KPIs. 

• Consider setting up a feature team including developers with mandates to make changes. This improves cross-team communication.  
• Identify what are the key performance indicators to follow and on what level, e.g. top 10 Fisma recommendations.  

As the story above shows, DevOps is not about techniques – it is about culture combined with technical capabilities. Change is possible and reachable, but it does not happen easily. In addition to being a cultural change, DevOps is a strategic choice that requires a strong, continuous business commitment.