Swarm Intelligence in an IT org

 Hi all, Today I am following up on my last article, "The DevSecOps Paradox."  

   In the last article, I challenged the DevSecOps patterns and the ability to solve all issues by - just - shifting left and decentralizing everything into small teams. Typically, you will be creating plenty of blind spots but also face many challenges due to the lack of skills. Today, I will dive into some approaches to mitigate these issues and on how to drive maturity in a product-team organizational structure.  

But why product-team organizational structure?  

More and more companies are transitioning their teams into agile teams, refactoring their code landscape into microservices using containers by doing a Cloud Transformation and providing APIs as interfaces.    

So, many companies are building up dozens of product teams who need to work together and are dependant on each other. Typically, you have three major parties, as shown in the picture below:  

• All the central teams, which focus on the topics that should be done for the whole company.

‍

• All decentral teams, which are acting as product teams and delivering features.

‍

• And the customers, which can be your end customers or business stakeholders to make use of the value.    

   These organizational changes are typically also accompanied by pushing agile methods in parallel and driving a huge cultural change which is typically very valuable.  

   But, there is one huge challenge that most companies face. When you have dozens of product teams, you will recognize quickly a very high disparity of maturity levels between your teams in different areas. I mean - it is totally obvious when are having dozens of lean teams (5-12 team members) that even having similar team setups they will not perform the same way.  

   So, what do you do?  

‍

   Maturity  

First, you need to ensure you can drive maturity growth. But what does maturity actually mean? Maturity is typically visualized with pyramids, and this is done out of good reason.  

 There are essential basic skills that every product team should implement and fulfill as the primary fundamentals. On top of the basics, you can then add more advanced topics focusing e.g. on automation to reach the next levels of maturity. But, in the end, there is a simple rule:  

   Without the basics, no maturity.  

   Innovation  

   Another topic in the same area is innovation. I still get confronted that product teams should have their implementations' full autonomy and decision power to drive innovation. In the first place, this sounds like a valid point. But let us switch the point of view and think about it for a second. Put yourself e.g. in the position of the VP of Engineering - is this still true?  

   It is only valid in the point of view of a single product team and not for the whole company.  

   Innovation does not mean that you can test and play with everything in an uncoordinated way. It also means that tools and techniques are used to a level where it makes a difference. Is it innovation when you have fifty product teams, and all are playing with different frameworks and programming languages, but only a small number of product teams are excelling in them? Will you be able to drive maturity for the whole company when you are not ensuring that the product teams are implementing the basic maturity levels? Surely not.  

    You will very likely end in a world of chaos where product teams use totally different tool stacks to not its full potential, spending unnecessary money and effort.  

   I typically love to call this pseudo-innovation. On paper, you are driving innovation for all the product teams, but somehow you are facing issues, as nothing gets properly finished or is rarely providing the quality you expect, and you are having problems with the delivery times.  

   Also, from an upskilling perspective, it is contradictory. Do you want to set up training materials now for everything? How do you create feedback loops and feedback chambers where people challenge solution architectures into higher maturities when you have fully disconnected product teams?  

   Swarm intelligence as the solution  

     To explain to you the "simple" ways of addressing it, we should have a look into biology first. Swarm animals like ants or bees can decentrally allow freedom but use some dedicated rules to achieve as a whole. (Ref. 1) There is no central team telling them what to do, but it is instinct and unconscious behavior. By this, the ants are, for example, able to find the shortest paths.  

   Unfortunately, there is no unconscious behavior with software engineers or product teams.  

   So, we need to set up cultural, procedural, and technical boundaries that imitate this swarm intelligence and can continuously drive all teams' maturity.  

The majority of these "initiatives" are basics that you should already have heard about. Let us start with the most general ones and then dive narrowly into the technical abyss.  

Code of Conduct  

A general recommendation is to start with an overall Code of Conduct where you define general guidelines that should be respected. Typically, these are paraphrased in a very general way and are rarely technically in detail. e.g., use Infrastructure as Code or always respect RBAC and the principle of least privilege. Also, they will often use the verbs like "can", "should", "must", and sometimes also "should not" or even "must not". These are the general basics of how to work and act.  

   Tech Radar  

   The next step is to provide also Governance in the tooling landscape. I typically sell it as the Company will provide a subset of different tools, platforms, tech stacks, and techniques that are valuable and strategic for the whole company, but the single product teams can decide their own decisions out of this subset and still have their own freedom. When new things come up, they should be dedicatedly reviewed and then, if valuable, made available for the whole company. A good initiative to drive this kind of Governance is the Tech Radar.  

   The Tech Radar provides development and technique guidelines to all product teams. There are different Tech Radars available, and I am featuring the latest one from Thoughtworks because I really like this setup for modern product teams.  

 In short, you have four quadrants with four different topic areas:    

• Techniques

‍

• Platforms

‍

• Tools

‍

• Languages & Frameworks  

In addition, you have four different rings which define the adoption state for each Tech Radar item in  

• Assess

‍

• Trial

‍

• Adopt

‍

• Hold

The idea is simple - you provide your product teams guidance on what to focus on from a strategical point of view regarding the four topic areas but still provide the product teams freedom to choose. These are also the ones where you centrally build up the maturity by giving training, best practices, and building up solution architectures.  

Global Central Knowledge Base  

The next large initiative is an obvious but dramatically highly failed one. Create a single source of truth for your knowledge and share it through all of your company. There are two areas you need to focus on:  

   Platform Architecture  

    First, the platform architecture.  

• How is the underlining platform technically set up?

‍

• How are the processes looking like and what are the requirements?

‍

• How are the large topics like networking, identity or security, and compliance addressed?

‍

The platform architecture generally is built by central teams as economies of scale make the difference here. Does it make sense for dozens of product teams to set up the platform and learn these skills? Not really. Also, the cognitive load would be another discussion point, as it is well-known that product teams should have a clear focus to perform well and you should not put too much cognitive load into single teams.  

‍

 I would go even further - It is not only about the platform but everything where standardization is required or can be easily centrally scaled. A good responsibility cut is typically IaaS vs. Cloud-Native, as IaaS can and should be centrally managed to secure it properly. The more product teams go into the area of Cloud-Native, the more responsibility they should own. (remember DevOps?! - "You build it, you run it.")  

   Solution Architecture  

   And this is where solution architecture jumps into the scale, and you should help to standardize even this area. Think of the platform as the technical underlining requirement that every product team needs to address and solution architecture as the environment that some product teams put on top of it. The idea is simple - if you have typical and repetitive solution patterns in your company, it does not make sense to let all product teams design and rebuild it from scratch.  

   Reusable patterns are those you need to gather, increase its maturity, and scale through your company.  

   Architecture Decision Records  

   Having patterns is good, but you will also need some rules, security or compliance requirements, or other rules to drive standardization and maturity. A well-known way to do so is the approach using Architecture Decision Records. ADR is a defined template to track technical architecture decisions, including all information in the point of time of the decision-making and keeping all in a centralized log location for all of your technical decisions. ADRs are a helpful tool to drive authority and connect your smartest people together to make the best decisions.  

   Everything / Policies as Code

   An afterward result of ADRs is that technical decisions should be monitored whether they are respected, or even better, they are technically enforced. As for the enforcement, always think about the lifecycle of Governance and shifting Governance left. (read here for further context)  

     In a simplified view, you can either proactively catch policy violations before creation or later in the lifecycle reactively after creation. When working with highly dynamic Cloud environments that are changing at an inhuman speed, you need to think about how you want to secure and control this exponentially growing environment.  

   The answer is simple - by code, especially using policies as code and moving to IT as Code.  

   Managing something as code has a lot of benefits, like version control, 4 or more-eye principles, no human error rate due to repetition, and being able to scale configurations globally without huge efforts. Policies as code are typical codified rules with filters on validating technical configurations or rules, automatically enforcing/configuring, monitoring, alerting, or even blocking dedicated states.  It is the only way to keep a dynamic environment like a CSP under control. "IT as code" refers to designing everything codified like Infrastructure as code, which can easily be verified using the explained "policies as code".  

   Product Excellence Dashboard  

 Finally, the overall goal is to create a data-centric company to know your current state and continuously grow your maturity throughout the whole company.  

Therefore, one of your initiatives should be to gather qualifiable data, measure it, and review your low- and highlights. Many companies give this into the hands of the DevSecOps teams and let them build small feedback loops for the micro-cosmos of single or a few product teams. But you really need to focus larger than this.  

In the next article, I will introduce you in detail to the Product Excellence Dashboard and how to drive the maturity feedback loops on an organizational scale, which is already teased in the picture above. But we should break here for now and do a quick, final recap.  

   Recap  

• We reviewed the typical issues when moving to a product-team organization and discussed typical ways of setting it up.

‍

• You learned about swarm intelligence and the different initiatives to drive it.

‍

• A strong recommendation is to identify reusable patterns, centralize them because of economies of scale, increase the maturity and scale them throughout your company.

‍

• Be careful to not centralize everything but still provide your product teams the freedom they require but not overload them in the area of cognitive load.

   I hope you liked it and stay tuned for the next article! Please feel free to drop me comments or feedback directly!  

   Best,  

   David  

‍