We have a lot of marketplaces that have to balance the offer with the supply, while creating an added value for either or both sides to ensure people use the platform.

Here is a list of common marketplaces you might be used:

• Airbnb: Short-and-long-term homestays marketplace

• Fiveer: Freelance marketplace

• Amazon: Third party sellers can use Amazon as a platform to sell their products

• Uber: Drivers provide rides to passengers.

• JustEat: Restaurants deliver food to consumers

• LetGo/Wallapop: List and discover second-hand items near you.

And a lot of them have a common business capability, the chat.

Chat: Business capability facilitate the communication between both users to ensure they don’t leave the platform.

Churn is always a problem when people leave the platform and start a direct relationship, and more when the CAC is higher than the LTV for a single transaction within the platform. When you need recurrence, which it is almost always nowadays.

So, the chat has become a critical capability as the whole product experience to ensure we keep the users within the platform, and not bypass it.

The chat: Core or supportive capability?

The chat capability isn’t a business differentiator anymore.

It lives within the supportive, and still quite tailored to the business domain that using a generic off-the-shelf solution might not be an option.

But the chat capability have a huge rival. WhatsApp.

The constant upgrades on features on WhatsApp, like reactions, reading status, last time read, multimedia messages, stickers, the real time capabilities … All those features that people are used to, you don’t have the scale of Meta to invest on them. But you are expected to move into that direction (Red Queen effect on Wardley Maps).

So, you are in a though spot:

• Should we use an off-the-shelf solution that costs a lot of money that doesn’t adapt enough to our user needs?

• Should we invest into a custom solution, acknowledging that we will have to keep investing as the messaging apps keep evolving?

• Is “not developing a chat” an option? People will bypass our platform anyway.

Wardley Mapping a Chat

Based on how I understand the landscape, visualized using a Wardley Map and a Core Domain Chart, suggests to us that custom-building it isn’t a good strategic decision.

Either relying on existing tools, or adopting an off-the-shelf solution, looks like the right decision to make.

Yet, the devil is in the details.

Any B2C marketplace has million of conversations and messages per month. And based on the current pricing models, it can cost millions per month to outsource this to a third party.

So, we have a business capability that’s cost of doing business, it is not a business differentiator, and we are in macroeconomic that makes it tougher to compete on margins that are shrinking.

It is valid to ask:

AI is accelerating development, we can do more with less, our competition is doing it, shouldn’t we internalize this, and save us the cost of this external tool?

Let’s assume we decide to build-it in house

Building an in-house copy of WhatsApp isn’t viable nor realistic. You won’t have the same requirements as WhatsApp in terms of million of concurrent users, for example.

But there is something that we can use in our advantage when we need to build internally a commodity. It is a well known domain:

• Certainty: Commonly understood (in terms of use)

• Market: Mature market

• Market perception: Ordered / trivial

• User perception: Standard / expected

• Understanding: Believed to be well-defined / stable / measurable

On the other side, those properties make this domain to have higher expectations of usage.

So, in this case we aren’t experimenting, and aiming to reduce the cost of learning. We are developing a well understood domain. So, we need to focus on a methodology like 6-sixma or Lean to develop this business capability.

We can “know” how much it will cost us, and often it is a lot.

How do we build it?

We know the chat capability has specific business needs, but we also have a tech stack, and people skilled with certain tools, and methodologies.

So, how do we develop the chat?

• Using existing tech like Ruby on Rails and Postgres, or we opt for a known technology to work for chat-like systems like Elixir?

• Do we invest in new talent to develop this system? Do we train existing talent, and take advantage of AI? Do we leverage existing tech stack instead?

All decisions are about trade-offs.

And on those decisions, we need to add the domain specific knowledge we have about our users behaviors.

For example, in a B2C about selling second-hand items near you. You can have million of messages per month, but you also know that:

• Conversations aren’t long, and they have a short duration.

• Conversations are co-located. A person in Spain won’t be talking to a person in South Korea to sell a second-hand item.

Understanding this domain specific behaviors, we can shape our custom-build solution more tailored to our needs without reaching the level of WhatsApp for example.

My decision-making path

and how AI influenced how I would run experiments to verify the most viable solution.

1. I would externalize as much as possible. There are a lot of hidden costs on internalizing commodities and products beyond the invoice and the end of the month.

2. In case I need to custom-build it. I would:

   1. Do a deeper research of viability using industry known approach.

   2. In case it is needed and viable to be on specific tech, like Elixir in the chat example, I would experiment with this approach to fully understand the consequences. And I mean also in production.

   3. Otherwise, I would default to current tech stack, and do our best to fulfill customer needs.

      1. This will help us to leverage existing talent and keep the tech stack as homogeneous as possible in case it is important for the company.

Why AI is changing this decision-making process?

Because AI is trained on known data, known products, and everything that’s already present in the web. And generic and supportive subdomains, products and commodities fulfill these criteria.

AI can help us a lot on what’s not a business differentiator but cost of doing business. The LLMs can help us to extract those decisions made in the past that are relevant for our use case.

But this will also have consequences:

• Using LLMs will be about the past, not the present, nor the future. We will also be behind of the rest if we only rely on the LLMs.

• Custom-building has a lot of hidden costs, a lot of them aren’t public, therefore LLMs cannot help us to anticipate those.

• We might end up in a sunk-cost fallacy, where we keep investing instead of decide to go for an off-the-shelf solution.

So, for one side, I consider that LLMs might be opening a space where more big companies will internalize those generic external services to reduce external cost. At the same time, this will start to increase all the hidden work to support those services.

Why internalizing everything isn’t viable

I provided this example for the chat, but you might consider applying the same decision to all third party services. That survey tool, that analytics tool, the CRM, …

And the truth is, those domains are huge, and I don’t consider reasonable to expect that a LLMs will be able to:

1. Understand all the domain specific knowledge required.

2. Support you maintaining the systems.

3. Support you on those edge cases, and industry changes that will force you to keep investing on those systems.

So, why do I consider a chat to be in the messy middle of the decision process to consider custom-building it?

Because, even though, from a capability point of view, I do consider it is a supportive one. In terms of the long term product strategy, you aim to reduce people moving to an external provider, and that’s why, in this case, it can make sense to make it internally, with all the consequences.

You might be investing millions, but you also can reduce the churn and increase the LTV enough that it makes sense. Remember that B2C are economies of scale, and at scale, the numbers can work.

Hi, I’m Eugene, Director of Engineering at Flo Health. Today, I want to share how we evolved our onboarding experience from a rigid, release-bound flow into a dynamic, server-driven platform that now powers rapid experimentation across mobile and web. This is a story about breaking Conway’s Law, building internal platforms with product thinking, and enabling teams to move fast—without breaking things.

Introducing the company

Before diving deep into details, I want to give you an overview of Flo’s domain, scale, and challenges. This information can help you understand why our onboarding is so complex and diverse.

Flo is a super app with various user segments: cycle tracking, getting pregnant, pregnancy tracking, sex life, socializing in secret chats, and many more. Moreover, we recently launched a Flo for Partners feature with 1+ million MAU in 2024, meaning we are no longer only a female health app.

What was the high-stakes problem?

At Flo, onboarding isn't just a product feature - it’s a critical growth lever. As a product-led growth (PLG) company, our ability to convert new users into active, engaged members within their first few minutes in the app directly influences our retention, revenue, and impact.

Initially, we approached onboarding as a set of hardcoded screens - a simple solution that sufficed for a time. But as our ambitions grew, so did the complexity of our onboarding journeys and the urgency to iterate on them quickly. To scale experimentation, we introduced a domain-specific language (DSL) that allowed onboarding flows to be described more flexibly. However, this DSL was still embedded in the app codebase, locking every onboarding change to the mobile release cycle.

This created a high-stakes bottleneck.

• Every experiment or fix was gated by a mobile release, making it impossible to react in real time to user behavior or localization issues.

• Errors took days to detect and fix, delaying learnings and negatively impacting conversion metrics.

• The complex onboarding created a cognitive and operational burden on engineers, PMs, and QA to maintain.

• Mobile engineers were disengaged, spending time updating massive DSL configs instead of solving high-impact user problems.

In short, the very process designed to accelerate onboarding innovation became the blocker. And in a PLG company, slow onboarding iteration is not just a developer inconvenience - it’s a growth risk.

Engineering Strategy

As a business it was critical for Flo to grow onboarding revenue and incite higher engagement from our users. So to address the growing complexity, slow iteration cycles, and developer pain points in our onboarding system, we defined a strategy focused on decoupling onboarding logic from mobile releases and empowering product teams to move faster. Inspired by existing server-driven UI patterns used in multiple parts of the app, we began migrating onboarding flows to a server-driven model. This shift enables real-time updates, safer experimentation, and faster hypothesis validation - without waiting for a mobile release. As a second step, we introduced a visual editor, allowing non-technical stakeholders to create, manage, and test onboarding flows autonomously, further reducing engineering overhead and accelerating iteration.

Analysis

What Works Well in the Flow

1. Established DSL & Contracts
   We already have a robust domain-specific language and clear contracts for rendering key onboarding screens, providing a solid foundation for further abstraction.

2. Backend-driven UI
   Our Backend-Driven UI system is mature enough to describe screen layouts and deliver UI configs via the server, reducing the need for mobile releases.

3. Cross-Platform Consistency
   The DSL allows us to maintain consistent onboarding logic across iOS, Android, Web, reducing platform-specific divergence.

4. Experimentation Culture is Strong
   The product team is hungry for faster iteration and has a clear sense of the right hypotheses to test - demand for rapid change is high and well-structured.

What Blocks the Current Flow

1. Slow Mobile Release Cadence
   Mobile releases are on a two-week cycle, delaying onboarding experiments, fixes, and urgent updates (e.g., localization or attribution errors).

2. Error Feedback Loop is Too Long
   It often takes more than 24 hours to detect misconfigurations in experiments due to non-real-time analytics and delayed rollout visibility.

3. Engineering Time Spent on Config Work
   Mobile engineers are blocked doing low-leverage work like editing large onboarding JSON configs, which reduces team engagement and velocity.

4. Mobile-Only Team Structure (Conway’s Law)
   A team composed entirely of mobile engineers naturally gravitates toward mobile-driven solutions, limiting exploration of server-driven or cross-platform strategies.

5. Lack of Visual Flow Management
   Product managers and QA rely on Miro boards and ad hoc tools to track flow logic. There’s no unified source of truth or editor to visualize and modify onboarding journeys safely.

Direction

Things We Want to Do

1. Apply the Conway Maneuver: Add Backend Engineers
   To break out of mobile-centric thinking and enable backend-driven solutions, we expanded the team to include backend engineers. This shift allowed us to rethink onboarding as a distributed, server-powered capability—not just a mobile UI flow—and enabled quick validation of this new architectural approach.

2. Use Download-at-Once Strategy for Config Delivery
   Following the Lean methodology, we selected the simplest and fastest approach: downloading the entire onboarding JSON (up to 300KB gzipped) at app start. In the first iteration, our backend service was a lightweight wrapper serving configs directly from Git. As a fallback, we include an embedded version of the main onboarding flow (without experiments) in the app bundle during each mobile release. This setup balances speed, reliability, and offline coverage.

3. Launch via Controlled Experiments
   We introduced server-driven onboarding behind a feature flag, starting with Android EN users and gradually expanding exposure (5% → 100%). We defined success not as improvement but as no statistically significant negative change across key business and technical metrics—signaling a safe infrastructure shift.

4. Enable Safe, Validated Server Releases
   Our backend performs structural validation of onboarding flows before serving them to clients. It checks for graph consistency, unreachable nodes, duplicate transitions, and cycles. Invalid configurations are blocked from release, ensuring high confidence in every server-side deployment.

Things We Discarded or Don’t Want to Do

1. Introduce a New Team Prematurely
   We intentionally avoided creating a new team or scaling the org structure before validating the value of server-driven onboarding. Instead, we applied a lean approach—testing the hypothesis with minimal changes and leveraging existing team members to deliver the proof of concept.

2. Build a Visual UI or Complex Backend Logic Early
   For the initial phase, we decided against building a visual editor or adding complex backend authoring tools. It was acceptable that all onboarding configurations would be written by engineers using the existing DSL. This kept the scope tight and allowed us to move quickly while focusing on validating the core approach

Coherent Actions

After the successful proof of concept, it became clear that backend-driven onboarding was technically feasible and brought tangible business benefits—faster iteration cycles, more robust experimentation, and reduced engineering overhead on repetitive tasks.

Rather than treating onboarding as a standalone system, we chose to take a deeper, more strategic step: we established the Survey Engine as a dedicated platform team. Onboarding became just one of many internal customers of this platform. This shift allowed us to generalize the infrastructure and scale its impact across multiple user-facing surfaces like in-app quizzes, polls, and dynamic content.

Key coherent actions we took:

• Established Survey Engine as a Platform Team
  The team’s mission expanded beyond onboarding. It now owns core capabilities such as configuration delivery, runtime rendering contracts, and safe rollout mechanisms- serving multiple product teams across the org.

• Provided Visual Tooling for Configuration Management
  The team built internal tools enabling product managers and content teams to visually create and manage onboarding flows, significantly reducing the need for engineering involvement in day-to-day experiments.

• Integrated Existing Capabilities into a Unified System

  • CMS Integration: Enabled safe release and publishing of onboarding content.

  • Experiment Service Integration: Allowed dynamic branching and targeted delivery of flows through experimentation.

  • Localization via translation management system (TMS): Made all experiments fully translatable from Day 1, eliminating a common delay in onboarding launches.

• Introduced a New Role: Content Operations
  To streamline the workflow, we introduced a Content Operations role. This person acts as a bridge between product, design, and analytics - taking ownership of flow configuration, copy updates, and experiment setup.

• Redefined the Role of Engineers
  Engineers are now focused solely on developing new capabilities (e.g., new component types, data logic, validation rules), not maintaining onboarding flows. This shift both reduced cognitive load and increased engineering team engagement.

Together, these actions established onboarding as a high-leverage, product-led capability- no longer a bottleneck, but a dynamic and adaptable system aligned with Flo’s experimentation culture.

Outcomes, Learnings & Next Steps

The migration to the Survey Engine platform delivered clear, measurable impact:

• Significant Uplift in ARPU and Subscription Conversion
  The shift to rapid, low-cost experimentation drove substantial business impact: average revenue per user (ARPU) increased by 13.5%, and the conversion rate to subscription during onboarding rose by 20%. These are critical metrics for PLG companies, highlighting the direct connection between experimentation velocity and monetization outcomes.

• Experiment Velocity Increased by 170%
  After moving onboarding to Survey Engine, we now launch an average of ~30 experiments per month. This made onboarding one of the most iterative and data-informed parts of the user journey.

• 87% of Experiments Launch Without Engineering Involvement
  By enabling visual tooling and empowering product teams, 87% of onboarding experiments are now launched without engineers, dramatically reducing the cost of iteration and freeing up engineering focus for core platform work.

• Cultural Shift in Product Thinking
  Product managers increasingly validate hypotheses directly with real users rather than relying solely on upfront user research. While user research remains valuable, the lower cost of experimentation has shifted the balance toward more hands-on testing in real contexts.

• Survey Engine Gained Broad Adoption Across the Org
  Thinking globally from the start—and building a platform rather than an onboarding-specific solution—allowed us to onboard ~20 internal customers. Several teams have decommissioned legacy tools or stopped using third-party vendors, consolidating efforts around a shared, robust infrastructure.

• Unified Web and Mobile Onboarding Experience
  Completing the migration of web onboarding to Survey Engine enabled the merger of mobile and web onboarding teams. Product managers now focus not on individual platforms but on delivering a cohesive, cross-platform user journey.

Next Steps

• Integrate AI as a Copilot for Content Creation
  To further accelerate time from idea to release, we are embedding AI capabilities into Survey Engine as a creative and operational copilot. This includes assisting with content generation, validation, and layout suggestions—especially important in sensitive domains where medical and legal credibility are paramount.

• Speed Up Experiment Delivery with AI-Powered Workflows
  We’ve already seen early success: in a recent use case, researchers fully created and launched a survey using just a PRD and prompt within Survey Engine—no engineer and designers' involvement required. This demonstrated the potential for AI to radically simplify and speed up the path from concept to live experiment.

• Content Safe and Credible Automation
  As AI takes on more of the content lifecycle, we’ll implement guardrails for automated validation—checking for medical accuracy, localization quality, and compliance with internal and external standards.

Further Reading & Technical Deep Dives

This post focused on the strategic and organizational aspects of our onboarding evolution at Flo Health. For readers interested in the technical implementation details, architectural decisions, and deeper engineering insights, we've published a comprehensive series that covers the full journey:

• Mobile Onboarding Evolution - Part 1: The Technical Foundation Dive deep into the technical architecture that enabled our server-driven onboarding platform.

• Mobile Onboarding Evolution - Part 2: Migration & Scaling Explore the practical challenges of migrating from client-side to server-driven flows at scale.

Fintech Engineering Strategy Post Series

• Post I. A legacy, a deadline, and no team.

• Post II. Building a product vision, and a team, and replacing Ktor with Spring Boot incrementally.

• Post III. Merging two teams for fast flow.

• Post IV. Simplifying a Ports and Adapters architecture and remove anti-corruption layers made sense from a sociotechnical perspective.

• Post V. Going to production killed fast flow. Understanding how company culture materializes on unaligned team dynamics. Part I

• Post VI. Reducing the chaos before addressing the complex socio-technical system

• Post VII. A team grouping (tribe) with local optimization thinking. Making sense of complex socio-technical system. [This post]

If you didn’t read the previous posts yet, I suggest you to because it is a continuation and I assume you have all the context. Check it here:

Moving into the next big thing, how did we end up in this situation in the first place?

Then, it was time to tackle the next big thing. The tribe.

What’s happening at a tribe level that our socio-technical systems is that fragile? Which dynamics do we have that causes this situation over and over again?

It looked like the there was no root cause, but a set of things that were interrelated that caused the system to behave the way it was doing it.

How did we start tackling the complexity? How did we find some signals to tell us what was goining on? Well, it was a mix of things.

I considered that focusing on the social interactions, responsibilities, and employee experience could give us some good insights.

I’m gonna explain here how we combined Team Topologies, Context Mapping, Team Cognitive Load, Core-Domain Chart, and Wardley Mapping to uncover where to do a set of micro interventions to the socio-technical system.

As an outcome of the workshops we did, we came up with this way of describing the architecture that shown business purpose inspired on DDD Subdomain types.

So, let’s start with the engineering strategy that I’m so proud of that I was able to do with a very talented people. It contains:

• A great implementation of a Platform as a Product approach based on Team Topologies.

• How we measured Team Cognitive Load to bring focus and direction.

• Investing in moving non-competitive services to supporting subdomains.

• How a set leadership decisions created success for some teams but created a huge inertia on others to adopt the new approach.

Engineering Strategy

Purpose

We weren’t able to have a sustainable flow of change. Every now and then we had some kind of a crisis that required that kind of hero culture to fix the stuff.

We weren’t a reliable tribe. Even though at team level we improved.

Our goal was to make the tribe reliable/resilient and create sustainable flow of change to support the different business initiatives.

We were able to put into production one product of three. So, how do we accomplish:

1. Keep improving Product B based on customer feedback and needs.

2. Release Product A and C.

Context

As a tribe, we were much exposed to external changes (either higher leadership, other tribe needs, competition moves, or just regulatory changes) that required fast interventions, which usually implied a leadership top-down decision.

Interestingly, I found that even though some teams might be performing at a maturer level, it doesn’t translate directly into a tribe being more mature.

We didn’t have a predictable roadmap, and we were much jumping on what product/business/sales/leadership said it was more important that week.

As middle-leadership team at tribe level, we were overburden, stressed, too involved in the teams day to day to understand what was happening, and we were requiring staff engineers to coordinate tribe level initiatives and firefight all the time.

Stuff was more calm for some teams because they accomplished E2E ownership (remember B team just was supporting end customers and operations team, and they had a easier time now), but when it come to collaborating with other teams within the tribe, we had a lot of back and forth because backlogs were full and supported multiple stakeholders.

As leadership of the tribe, we noticed that change happened from multiple places and it didn’t take into account how we organized the tribe. When some stakeholder needed a change, they were able to talk directly to some PMs to see if they had backlog capacity to deliver a feature, regardless if it was from their teams ownership.

Understanding

When I face a complex socio-technical system, I know that there is no root cause that explains what is happening. So, I would rather find the area that we could impact the most because chaning anything will require a high effort.

I also know that the purpose of a system is what it does, which we had to modify to create the results that we wanted. It meant we needed to change how people worked together with the hope that it will translate into a better situation.

I focused on understanding the next areas:

• How change happen.

• Architecture that shows the business purpose.

• Team Cognitive Load.

• Teams interactions.

with the hope that we could find the right place to introduce change incrementally.

How change happen

I sat down with the VP of Business to understand better the business model, and from there I continued to understand how an initative was created, how it materializes, and it ends on different teams.

The first realization is that the initative and how the teams were structured didn’t match.

For the business, we were a single product, for us, we were three different produts.

🤯 Of course all this chaos!

At somepoint, we evolved from being a single product/team, into different teams and products because the complexity was that high that a single team was unable to deliver all the expectations.

The tribe evolved, but how the business understood the product didn’t.

Remember the architecture? Everything depending and talking to everything else?

Now it makes sense! It is how the business understand the domain and how the different initiatives are funded.

When you are a scale-up, with people rotation, with leadership rotation, and so on, it is normal that this memo got lost, and more in a company that relies a lot on spoken communication instead of writing communication. If a key people left, a lot of context is lost.

I’m sure this was known by someone but I didn’t speak with that person, or that person wasn’t there anymore.

I wrote that down as part of the engineering strategy we were designing.

Even though it was an interesting data point, I didn’t consider that we should start by changing how the whole organization understood our tribe/product. It involves too many people from different layers.

I was sure that we were able to find another point to start introducing change at a smaller scale.

Architecture that shows business purpose

We identified that how the business understood the domain/product and us wasn’t the same. Now, we wanted to understand two things:

• If how we understood the problem and the architecture was the same.

• Untangle the use cases that touched multiple product/services.
  When we know that a user request translates into multiple API calls, we have a potential distributed monolith.

This is the first picture of our architecture that we created.

It provided good insights.

• Everything we do is core.

• Everything else is either supportive (when we run in our cloud) or generic (when we consume it as a SaaS for example).

• It shown the use cases!
  We could see and E2E use case and how it navigated all the infrastructure. We didn’t have traces or any observability tool to tell us that, so we had to rely on people knowledge to build it.

By asking for the architecture, it brought interesting conversations of why things were failing.

We should do Event-Driven Architecture. We should decouple here and there. Let’s do micro-frontends.

People had opinions, and they were founded in their day to day pain. But the problem wasn’t in the architecture per se, but how we interacted with each other, how we introduced change, and so on that caused this problem.

I knew that if we stopped analysing there, a huge refactor or technology migration would happen but it wouldn’t fix our problems.

Team Cognitive Load

People had pain and they were motivated to help reduce it. But which was their real pain? Should we listen to the loudest voices? Are we missing something? Is it a local problem or a tribe level problem? What should we fix first?

Thanks to the retrospectives, we had some data points on what was painful for teams, but each team had their own roadmap and it was hard to create a coordinated initiative.

Is there something that affects all the teams?

We wanted to provide some clarity on what was causing a bad developer experience that could help us to identify a shared problem cross teams.

We used the Team Cognitive Load Assessment survey by Team Topologies, and we adapted it to our context. We didn’t have a tool like Teamperature back then, so it required a manual work for us, the leadership team.

What the Team Cognitive Load Assessment told us was:

• Building, testing, and deploying services was great.

• Understanding the business needs was ok.

• Operating the services was a mess.

Ugh! The conversations were about changing the architecture to improve reliability, but the real problem is that we didn’t have the right observability and monitoring tools in place.

We were paying SaaS observability tool and self-hosted Elastic Search with Kibana… So… let’s dig here.

We asked people how they were able to understand what was going wrong in the system.

- Devs: We use CloudWatch to see the logs, and then we go to the database to see why it is in the wrong state.
- Aleix: For all the services?
- Devs: Yes.
- Aleix: Why you don’t use SaaS observability tool or Kibana for this?
- Devs: SaaS observability tool is hard, we don’t know how to use it. And Kibana doesn’t work.
- Aleix: And when a problem happens with multiple systems?
- Dev: We ask the Staff Eng.

(Aleix goes to talk to the Staff Eng)

- Aleix: Hey! Are you aware of this problem?
- Staff Eng: Yes… we know but we don’t have time for training. The tool was decided by the global platform team but there was never a training.

(Aleix goes to the global platform team)

- Aleix: Hey! Are you aware of this problem?
- Platform dev: Yes, but we are now migrating to another tool because our SaaS observability tool isn’t working.
- Aleix: but the problem is training, why not investing the migration time into training people?
- Platform devs: That’s not our focus now, we expect teams to learn the tool. And the new tool has a good learning materials for free

The problem seems quite obvious, but I guarantee you that it is more common than you might expect. Choose a tool, do not train people on it, blame the tool, change the tool, don’t train people, repeat.

We identified a skill gap.

Team Interactions

By intoducing the Core Team, we also introduced the collaboration between all the teams when anything required a change to the core services.

The Core Team was doing two things:

• Delivering the Product A because no one else was there to do.

• Introducing change on the shared services.

Here I leave an example of how a Stream Aligned could evolve into Platform Team, but it wasn’t how it happened to us.

Enough to start the direction

At this point, I considered that we understood enough about our context and situation at hand. We could start making some hard decisions on how to proceed.

I will go in depth on the next post.

Fintech Engineering Strategy Post Series

• Post I. A legacy, a deadline, and no team.

• Post II. Building a product vision, and a team, and replacing Ktor with Spring Boot incrementally.

• Post III. Merging two teams for fast flow.

• Post IV. Simplifying a Ports and Adapters architecture and remove anti-corruption layers made sense from a sociotechnical perspective.

• Post V. Going to production killed fast flow. Understanding how company culture materializes on unaligned team dynamics.

• Post VI. Reducing the chaos before addressing the complex socio-technical system [This post]

I introduced the whole context in the previous post. In case you didn’t read it, now it a good moment to do so!

Using the Cynefin Framework and identified that the team was in a chaotic domain, we had to act-sense-respond.

So, how do we move away from the loop of death?

When the teams dynamics aren’t helping but just making it worse?

Making though and focused decisions. One thing at a time.

We focused on the area where the chaos is more intense. Operations and B teams.

I could try to align with Customer Success team, but I would had to create a relationship first with the leadership that isn’t part of our tribe, understand their objectives, and align on the purpose.

Too much.

We needed a solution that we could implement ourselves taking advantage of B team domain knowledge.

I joined the team in their day to day supporting activities for customer incident tickets, and I started to pitch the idea that we had to automate that process somehow.

We started with a document listing the steps needed to solve each ticket and we found that:

• Several tickets were similar but solved in different ways depending of which microservice or external provider failed.

• How to fix the issue was API based most of the time + some checks at database level.

• Most of the failure points were outside the team.

Let’s investigate more.

- Aleix: What’s preventing to fix this recurrent problem?
- B Team: This is a responsibility of core team now.

(Aleix goes to Core team)

- Aleix: Are you aware of this is happening?
- Core team: Yes, but we have a plenty things to do, we have one person supporting B team on this.
- Aleix: Cool! Did you find already the problem?
- Core team: Yes, it is an external provider that when it fails, it causes multiple of our microservices to become in an inconsistent state. So, we are helping B team to fix those problems manually until our provider fix their problem in their end.
- Aleix: You are also doing manual work? Did you consider make the service more resilient?
- Core team: Yes! We want to, but we don’t have the time to make it, we have a lot to do. Supporting B team, continuing the development of A product, help C team with some of their needs to go to production. We need more people!
- Aleix: I see! Thank you for your time!

(Aleix goes to operations)

- Aleix: Could you explain me your process to fix those production issues?
- Ops: Sure! A customer comes to us we don’t know why it is failing to them! and we have a policy that we need to reply within 24h to tell them that we are working on their problem.
- Aleix: Telling them which is the problem?
- Ops: Yes, if we know what’s happening, we can inform them how much it will take to fix the problem and help our customer to be calm because we are working on it.
- Aleix: Is there a difference between letting them know and fixing the problem?
- Ops: Oh yes, we have different SLAs depending on the issue. But because we don’t know, we have to ask the developers to share the information as soon as possible in case it is a critical situation.
- Aleix: And which is the normal case? Are they critical?
- Ops: Oh, no! They are usually problems that could take 3 days to be solved without any problem, we let them know and the customer is happy.
- Aleix: Thank you for your time!

I don’t know if you recall before the all layoffs happened that the norm to fix any problem was to add more people 🤦.

This was our norm too, and you could see that any problem we had, we just added more people. Customer success is unable to keep up with the tickets? Add more people! and so on and so forth.

By adding more people, you change the architecture due to Conway’s Law. And when you change people often, the architecture is a mess because it is unable to adapt at the rate that people do.

That’s was happening to us, a lot. Missing communication everywhere due to the fast growth challenges.

I was able to identify a low hanging fruit. Provide access to the operations teams to understand why a customer had a problem.

If we combine a way to inspect the state of a customer account and a little manual explaining which is the scenario they have, they could inform the customer of their problem without involving the dev team.

Ops team is another product user

A principle we introduced to the B team was “we have at least two users, our end customer and operations team. Our service needs to help both needs”.

By understanding operations team as another user persona, we were able to prioritize work for them from a product perspective. It wasn’t anymore a dev thing, but a Product-Eng thing. Product become aware of the operations needs, and we started to include them in our quarterly prioritization.

So, what we need to do?

The company had several internal dashboards in place which they required to talk with those teams because they weren’t self served. You need to talk with the team, share your needs, prioritize the dashboard to our product integration, and … too much time. Yet another collaboration!

We know that collaboration isn’t allowing fast flow, so, an alternative?

Let’s build a dashboard ourselves. Should it be pretty? No. Should it be secure? Yes. Should it do much? No. Is it aligned with the company principles? No, because we were building something that we shouldn’t do in the first place.

Internal crappy dashboard that kind of works

And that’s what we did! An internal crappy dashboard with Vue via CDN to take advantage of the existing APIs to show data to the operations team secured.

What did it do?

You add the customer ID, and it shows you its state. You go to our internal wiki page and you know how to reply to the customer.

At this point, the amount of tickets dropped in criticality. We no longer had to solve them in the same day. Now we had time between the ticket was open and we had to resolve it.

We got some air!

Automating devs knowledge

- Aleix: Should we consider automating what you shared in the Wiki page at the Product B?
- B team: Well, we shouldn’t have to do this here… but it is true that we are suffering a lot.
- Aleix: What’s the problem of adding the logic here?
- B team: Is it duplication? Shouldn’t it go into the other services?
- Aleix: Yes, it is.
- B team: Isn’t it bad?
- Aleix: Well, it is the place that will take us less time to build, the other implies communication and collaboration with other teams. I think the effort is higher than just do it here.
- B team: I think it will just make our code worse.
- Aleix: Probably yes, it will feel wrong until we learn more about the problem. We shouldn’t touch the logic that’s serving our customers, but build it aside. Ready to be deleted at some point, even though that point might never happen because it is outside our team responsibilities. I’m taking care of this, but it will take longer than we would like to.
- B team: OK, we can start this way and see how it evolves.

And that’s how Service B become aware of all the other services problems, but made the product resilient for the customer perspective.

The team had a very interesting idea, let’s create self-healing system by implementing the rules to help the system to reach a coherent state.

But we had a problem, not all the scenarios can be automated. Some requires opening a ticket to our provider support center.

What did we do? Implement the idea in small increments.

The dashboard starts the automation

Instead of a developer calling the REST API directly, we added specific pages into the dashboard to start the most common operations the devs team made, and now they were available for the operations team to do autonomously.

So now, instead of opening Postman, we opened a dashboard with a list of operations we could use for each scenario that our end customers needed support with.

At first, operations didn’t do it by themselves, and it is normal! If you need to support several customers each day, it is not the same opening tickets and a team of at least 2 developers supporting, than do it yourself.

We had to do several meetings, looms, improve our documentation, and improve the dashboard to make it more accessible for the ops team.

We were able to convince them to support the customers directly because we needed to free the developers for that operations work so they could fix the system.

The service self-heals

Lucky us, we got into the point where 2 developers weren’t spending their time to support end customers directly and start implementing the service improvements to make it more resilient.

The team implemented a nice state machine that could self-heal on the scenarios that were automatizable, and let us know via a Slack notification when it wasn’t possible.

Reaching this point was tedious, demanding, energy taking, and demotivating.

But when you reach the point that you can sustain the product, the issues in production that affect end customers drop to almost 0, it is a huge energy and moral boost.

You know you did a good job! And it was a multi-team effort to accomplish it. We celebrated this moment a lot.

Afterwards learnings

If I had to highlight some actions that helped us to move away from the chaotic situation, they are:

• Focusing on what you can control within the team to avoid team collaboration in a chaotic situation. You increase the chaos by involving people. Be sure you need to involve the people yes or yes before doing so.

• Instead of aligning with multiple people about team responsibilities, I took the decision to make it at team level by de-prioritizing a lot of the planned product work in favor of stabilizing the system. I made that work as part of product work and multiple blockers were removed. I had a good and strong relationship with the PM of the B team, and this was key to have a common speech of why we were making those decisions.

• Instead of following the internal approach to building dashboards (which they weren’t self served, and the DevEx wasn’t good enough), we started a crappy Dashboard that did the job. Fixing customer problems > make it good at the first try.

• By providing the dashboard for the operations people, we learned more about our domain and customer needs. By understanding and supporting the customer, we were capable of implement the right level of self-healing mechanisms later. Otherwise, we could had miss-implemented some of the use cases because we missed deeper domain knowledge. Because how a system failed is also part of the domain knowledge due to several regulatory specific policies.

• Going fast on making the decisions and implementing the good enough increment to free up developers was essential.

• I had to adopt a command and control leadership style to make the situation stable, and it helped everyone to focus on the bet.

  • On a chaotic context, the important aspect is acting and moving away from it, that’s why this leadership style made sense. You can see more about adaptative leadership styles here.

Product teams thinking of operations as a user persona impacted how we understood the teams responsibilities

After leadership noted that a product team by itself was providing good support to customers via operations team, they questioned why we needed a specific CX team to build those dashboard in the first place.

This approach become part of how we were doing things from now and on.

Moving into the next big thing, how did we end up in this situation in the first place?

Then, it was time to tackle the next big thing. The tribe.

What’s happening at a tribe level that our socio-technical systems is that fragile? Which dynamics do we have that causes this situation over and over again?

It looked like the there was no root cause, but a set of things that were interrelated that caused the system to behave the way it was doing it.

In the next post I will go in depth on how I analyzed a complex socio-technical system and which bets I took and why.

Thank you for reading!

Fintech Engineering Strategy Post Series

• Post I. A legacy, a deadline, and no team.

• Post II. Building a product vision, and a team, and replacing Ktor with Spring Boot incrementally.

• Post III. Merging two teams for fast flow.

• Post IV. Simplifying a Ports and Adapters architecture and remove anti-corruption layers made sense from a sociotechnical perspective.

• Post V. Going to production killed fast flow. Understanding how company culture materializes on unaligned team dynamics. [This post]

• Post VI. Reducing the chaos before addressing the complex socio-technical system

On the previous post IV, I introduced that we 1) created a core team at tribe level 2) re-purposed the team to be a platform team.

But why did we re-purposed the team to be a platform team? In which context did it happen?

Context

We simplified the architecture, implemented the feedback and insights found by the beta customers. We released the MVP. 🙌

We reached a good team maturity at the same time the B2B2C product started to gain traction. The past decisions on focusing on this product to have a faster feedback loop started to pay-off.

The product value proposition was to help companies attract and retain talent by offering a better employee experience.

We onboarded few clients per month, which could mean between hundreds to thousands on employees. And with thousands of users per month, you start to see the system weaknesses. For the technical aspect, you see the service to start failing given several concurrent users. For the product/design aspect, you start seeing customers being unable to accomplish their main use case.

Customer tickets started to pile-up, developers were supporting the product constantly, and several manual processes that before weren’t painful, now was blocking all the fast flow.

Customer success team and Operations team were under pressure to resolve tickets to gain the customers trust and make sure they don’t churn. They hadn’t the right tools to make it happen, so they mainly created tickets for the Development Team to fix manually.

If the product-tech team is busy solving production issues, we weren’t improving the system. We weren’t able to escape the dynamics we created.

You know this virtuous cycle that deteriorates everything, architecture, code, team morale, customer trust, … which requires a bold decision to stop it.

In this post, which it needs to be spitted in multiple parts, I will go in depth on:

• Understanding how the socio-technical system produced this result.

  • How leadership affected the whole team dynamics and how we had to change the whole tribe culture.

• How using Team Topologies principles with Inverse Conway Maneuver while constantly measuring Team Cognitive Load helped us make sensible decisions.

  • Starting the change from the people and then though tech.

• Understanding the technical landscape with Wardley Mapping and Domain Driven Design with Strategic Patterns.

  • Why the microservices failed and why the current approach didn’t work.

• Directly challenge the company operating model on how it designed and decided the teams responsibilities.

  • Two teams discontinued and one team become a Platform Team.

I’m sharing with you a six month journey on how we changed completely a tribe from low performing to high performing and be a reference within the organization.

I hope you enjoy this in depth engineering strategy at multiple levels as much as I did working with my peers making it a reality.

Start sensing the grouping instead only the team

I shared between the Post I and IV different strategies that worked at team level because it was the high-stake challenge for the organization and required my full attention.

You can see how I kept involved at team level in the next diagram. I was able to sense problems at tribe level but I decided to ignore them to focus on the high-stake problem first.

I don’t know if you remember, but I joined as Engineer Manager, but my attention was fully at team level. I had to start moving into tribe level challenges, and we were facing another fire 🔥 at team level.

With this peace and dynamic we wouldn’t move into a more stable context. We will move from fire to fire, creating an environment that’s tiring and burns out people.

The day to day was chaotic. We have production incidents due to systems being down with several customer tickets due to the system not being available.

At some point, I had to decide to lead things be on fire so we could make the improvements we all needed.

The good thing is that we had already some experience as a team working together, I was exposed enough to the company complexities, and I asked the team to sustain the situation while I had to focus on identifying the root cause. The team was mature enough that they didn’t depend on me on that chaotic and stresful situation.

So, I started sensing what was happening at tribe level.

There was something that seemed off. It looked like teams had problems, not the organization. The attention focused at team level, like the problem was there and so was the solution.

My helping with those fires and fixing them didn’t help. I was part of creating the hero culture. Those people that jump from fire into fire, and get good recognition for it. Who doesn’t like to be recognized to help on the hard moments?

That’s not a sustainable job. I started to think differently about the situation.

What if the teams were signaling deeper organization/company problems? If only one team underperforms is a team problem, if multiple teams underperform, it is an organization problem materialized at team level.

I was to identify that each team had a different problem at first, but they seemed to follow a pattern.

• Teams without products in production.
  They need several months to release a product.

• Teams with products in production.
  They cannot iterate the product and stabilize it because of the operability is high and there is more need to keep adding functionality.

There was something happening again with hand-offs, like I shared in the Merging two teams for fast flow post.

And then, all the sudden another team appeared! Why? When? How? WHAT!?

I started to host calls with every team leader within the tribe and people that depended on us, like Operations teams.

I discovered interesting things:

• Core team was supporting the Product A development because no one else was doing it while teams B and C needed internal systems improvements that were blocking them.

  • Product B was using several of the internal services, some of which were unstable and their failures impacted the customer.

• Supporting the customer happened outside the tribe responsibilities.

  • Product teams were seen still as feature factory outside the tribe (and probably within the tribe too…). Customer Success and Operations teams had the responsibility of providing a good customer experience and closing tickets as fast as possible.

  • They didn’t have the tooling to be autonomous, so they had to open tickets to the next layer until a developer was able to access the logs, the database, and call the right APIs in the right order to fix the customer problem.

    • This happens when you have a non-resilient distributed monolith BTW.

• Because the responsibility was for the Operations division, they had the authority to create teams to support better the customers, so, they didn’t have to ask anyone to add a new team to our tribe with one objective. Help the other Operations teams to do their job faster.

  • BUT, how will they create a dashboard? They don’t have business nor development context. Will they integrate with our APIs that aren’t ready to make that? Will we have another user to support now? Why creating another handover instead of ourselves creating the Dashboard directly? Which we already started… Will they just add code to our repository?

At the beginning I remove handovers between internal teams. We created a team to deliver E2E, but it looks like the organization didn’t understand the same for supporting the customer. That was another team responsibility, you focus on delivering new things.

I naively expected to find an easier answer of what I needed to fix, but no, it was a complex human problem that materialized on the messy architecture we had.

I was seeing Conway’s Law in front of me, and the problematic resided outside the tribe.

I had to involve VP level people to understand better what was going on while the team in a chaotic situation, which they wouldn’t be able to sustain for too long.

So, we had to do two things:

1. Understand the whole complex context to start betting on how to improve the situation.

2. Just act to move away from the chaotic situation.

Following the Cynefin framework, we could say that we were in the next situation:

So… what to do?

Let’s do both starting by reducing the chaos 😄

In the next post I will share the tactical actions we did to reduce the chaos, and how we let things be on fire for a long while.

Fintech Engineering Strategy Post Series

• Post I. A legacy, a deadline, and no team.

• Post II. Building a product vision, and a team, and replacing Ktor with Spring Boot incrementally.

• Post III. Merging two teams for fast flow.

• Post IV. Simplifying a Ports and Adapters architecture and remove anti-corruption layers made sense from a sociotechnical perspective. [This post]

• Post V. Going to production killed fast flow. Understanding how company culture materializes on unaligned team dynamics.

• Post VI. Reducing the chaos before addressing the complex socio-technical system

I’m concerned about the amount of misunderstood ports and adapters architectures (from now on P&A arch.) out there, more specifically on the decision making process behind them.

The intention with this post is to give you tools and arguments to decide properly which architectural choices do you have and, instead, opt for an evolutionary approach when needed.

I briefly introduced the decision we took of replacing Ktor by Spring Boot in this post, yet, I think I could go more into the details and nuances of the decision.

If you want the short version of the full post, you can check this Tweeter thread.

In the thread I say replacing ports and adapters in favour of MVC+S. Indeed, I would say that what we did was a very thin ports&adapters where the ports and adapters were the ones that Spring Boot provides out of the box with spring-boot-web.

The thing is that a ports and adapters leveraging Spring Boot defaults looks like an MVC+Service, or 3'-layered architecture with an Application Service.

TLDR;

Ports and Adapters architecture isn’t a best practice nor a sign of quality by itself. Indeed, it can even show a poor situational awareness and imply a high accidental complexity and incurred cost to the business.

This is due to it is misunderstood and each implementation is different. The possibility of introducing accidental complexity is high due to it relies a lot on the developer criteria. And we have the tendency to complicate things, right?

If an architecture rely on the developers to keep it simple, it will tend to get complicated rather than the oposite.

I don’t mean that we shouldn’t write sustainable code with a good test base and good practices like observability. I mean that applying all the “popular best practices” without understanding how they apply to our context is a receipt for failure. Applying patterns without the situational awareness shows a lack of seniority or a desire for a CV oriented career.

Disclaimer: I did that mistake, and I took a long time to realize that the business is who keeps paying the consequences.

Learning them and applying specifically for the context at hand is what matters.

Today, I’m explaining you why we made some contra-intuitive decisions such as:

• Migrating from ports and adapters full decoupled from the framework to ports and adapters coupled to spring boot.

• Moving from Ktor and Exposed to Spring Boot.

• Moving from anti-corruption layers to decouple from external systems to conformist (accepting the upstream model).

• Coupling to Spring Boot and to upstream systems.

• Introduce a Platform as a Product in the tribe lead by a Platform Team.

Which helped us to achieve stuff like:

• Removing +1.000 LoC cross multiple services.

• Removing several cross layers duplicated tests.

• Reduce the cycle time and improved the lead time for features.

• Improve the reliability of the systems.

• Reduce the cognitive load of teams

• Improve the onboarding time from 4-5 months when people felt productive to less than two months.

As side-effect, we made decisions based on the learnings we had that impacted the performance review, the career leader definition and the hiring process. In specific, it affected the senior+ developers job expectations. I’m also explaining why and what we did here.

Subscribe now

The context

We just merged two teams, and we started to gain team momentum. People were doing pairing, started to know each other, and we were progressing from forming to storming based on Tuckman's model.

When things start to become less chaotic at human level, you can slow things down to understand the technical part.

At tribe level, we had an architecture like this:

Warning: Notice that the diagram now it changed compared with the 2nd post series. A new team was created to handle the shared services. This is team was named core team. Later I share why the name core was bad, and why the approach had a good intentions yet was a bad implementation and why evolving into platform made sense.

If we look at each service, the architecture was ports and adapters. If we make a diagram to show the size of each layer, it would look like this:

We can notice a couple of things:

1. Services have few domain logic.

2. Services have a lot of network logic.

Domain logic poor-network rich logic service. And it made sense, we chosen an off-the-shelf solution that would handle a huge part of our business logic as it wasn’t a business differentiation but needed to run our business. So, we were calling a lot those external systems to handle the majority of use cases.

Two main questions raised:

1. Why ports and adapters if we mainly pass payloads around and we have very specific business logic?

   1. The product complexity isn’t that high. It looks like an “state machine” plus smart component that connects specific.

   2. But we are connecting to a lot of other systems that we don’t know if they will change the API or something. So, we want to protect ourselves from future changes regardless of the system. We apply this to all systems we integrate with.

2. Why that many ACL? From what we are protecting from?

   1. When a team requires a change on the shared services owned by core team, it impacts all the other teams and we need to protect ourselves from those unanticipated changes. That gives us as a team the possibility to be autonomous to change things without breaking the others systems.

   2. Each time we change a property in a system, we need to update the ACL of other systems we control. Usually, a change on one shared service, implies a change at least to two more services, ours and the BFF (backend for frontend), which we also did with Ktor and Ports and Adapters.

The conversation started at architectural level, but the problem wasn’t there. Focusing on the technology ignoring the human aspect of the problem would let us on the wrong direction.

The architecture and approach was a reflect of how the organization introduced change, who made decisions, and different incentives mechanisms.

In order to understand what was causing the outcome, I focused on the people.

I used:

• Context Mapping from Domain-Driven Design strategic patterns.

• Core domain charts to understand the domain complexity.

• Job expectations.

Understanding

Context mapping

Context mapping helps you understand the teams relationship and how flow is handled by teams.

I went to the engineers and asked how they saw the context mapping of the tribe services from a teams perspective.

The diagram makes sense my itself, but misses how change happens.

• Aleix: How do you introduce change to those shared services?

• Team: We share the blocking dependency and then our tech lead and core team tech lead negotiates when the change can be ready.

• Aleix: How often do you need to request changes?

• Team: Weekly, we strongly depend on them due to they being the door to our external partner systems.

• Aleix: And they are unable to make the change?

• Team: We just do it ourselves if we need it with high priority.

Let’s update the diagram with the new findings.

We had a consumer-supplier relationship with the customer facing teams and the internal core team. Which means that the team C was having problems keeping up with the high demands of customer facing teams that caused them to not be able to improve the services.

I added the one box per team, remember that the core team had more than 5 services they owned that customer facing teams consumed and required changes to keep building their products.

Analysis

When you have a lot of anti-corruption layers and consumer-supplier team relationships it can be an smell that something is off.

As engineers, our main tool to protect from change is code. It can manifest as anti-corruption layers for example. Yet, the key problem of why they needed to protect themselves from constant changes is how change happened in the organization.

It happened at product manager and tech lead level.

This situation made each group of people to make their best to keep the delivery forward but making the whole organization slower.

Core domain charts

When people is measured by their impact, they will work on the core business domains.

• Aleix: Let’s list our domains and services that we own as a tribe.

• Team: Sure, here it is!

• Aleix: Thank you! Let’s introduce the DDD subdomain types.

• Aleix: How would you define your services based on those types?

• Team: (summarized) What we do is core, what we don’t do isn’t core.

There is a lot of truth behind those words.

If your performance review is based on your business impact, you need to work on core stuff. Otherwise, how do you demonstrate your contribution to the business success?

• Team: We already did this exercise, we already applied DDD to our services.

• Aleix: Applied DDD to the services? Could you elaborate that please?

• Team: Yes. Our core services need to handle a lot of complexity, and we adopted a ports and adapters architecture to allow fast iteration and high quality software. That’s why we have created a microservice per domain. They focus to do their domain responsibility super well, and then we allow others to interact with each one so that we reuse domain logic.

This is an argument that I heard too often. It looks OK at first, but indeed, from that argument to the reality there is a full world in the middle.

Let’s break the argument and analyse it separately to uncover the consequences.

Our core services need to handle a lot of complexity.

There are no core services, there are core subdomains that might or might not be a service. They can be an application, a module within the monolith, an actual service.

I wouldn’t expect a product that’s forming to handle a huge amount of complexity at first. I assume they need to fulfill two properties first before you start sensing the high complexity you are promised for:

You should validate that the domain is core working with product and business to verify it is a competitive advantage for your organization. In order to verify that’s a competitive advantage, you need to accept a lot of rapid iteration (change often), and when you start sensing the direction, you start seeing how the complexity materializes.

If your understanding of a core domain is complexity, you will force it to be there by adding unnecessary complexity. Some people adds CQRS, ES, and P&A arch. because they read on a blog post that’s the best way to handle complexity.

We adopted a ports and adapters architecture to allow fast iteration and high quality software.

That’s nonsense argument.

Ports and adapters architecture has certain good properties on a specific contexts, but they require training and experience from the team to not create more harm than good by miss applying those patterns blindly and creating unnecessary complexity. Causing more cognitive load on people for simpler use cases that a CRUD model works fine. I don’t even mention onboarding people.

Ports and adapters isn’t a sign of high quality software. You can create Big Ball of Mud with ports and adapters and other architectures for multiple reasons. Quality software is a combination of multiple things, an architectural style doesn’t guarantee quality by itself.

That’s why we have created a microservice per domain

🚩 This sentence shows a low understanding of domain-driven design. A misunderstanding of what a domain is could lead into way to small microservices. How many times you found a microservices with an entity name? User microservice, Product microservice, Payments microservice.

They focus to do their domain responsibility super well, and then we allow others to interact with each one so that we reuse domain logic.

Those aren’t domains but entities, entities that start calling each other and you just created a distributed monolith.

Focus on understanding the domain, its bounded contexts, and its subdomains. Team-sized microservices is a good rule of thumb.

• Aleix: Did you consider alternative architectures like leveraging more Spring Boot out of the box features or bigger microservices to avoid all those network calls?

• Team: Well, indeed no. In order to become senior developers, we need to show that we know how to do microservices and ports and adapters in an autonomous way. So, part of my performance review objectives was to show that I was able to deliver that core domain alone.

Analysis

There were multiple things that caused microservices purification and ports and adapters everywhere.

• People performance review to be senior required to show that they needed to know ports and adapters and microservices.

• People are measured by business impact, therefore everyone wants to work on core domains.

• Core domains that need to handle high level of complexity, therefore they need to follow the senior best practices.

The reason of all the accidental complexity was based on the wrong incentives. Plus, those incentives were deeply established within the engineering culture.

People expected high level of autonomy within the teams based on those architectural choices, but ignored how change happened within the organization and why they weren’t unable to be fully autonomous.

When exists a separation between the developers and who makes decisions, the architecture will be a reflect of that reality. Yet, because we only see our context, we will apply local maxima solutions that misses the overall context.

Direction

We revisit our between teams agreements to optimize for tribe velocity instead of team velocity due to our current nature of work.

We acknowledge that most of our use cases are proxies to an external partner, and we have a very specific use cases that show business complexity advantage. We optimize for this reality.

Coherent Actions

Core team becomes a platform team

The core team will become platform team with a clear purpose of improving the DevEx of integrating with those services by making them more stable and change them less frequently. Instead of big services, we will do a thin platform with well proven and supported use cases.

Removing the name core from the team, allowed us to realize that those services weren’t core subdomains but supportive subdomains from a DDD point of view. Allowing people to change the expectations.

Instead of those services chaining a lot, they needed to be stable and just to a single purpose. Connect with the external provider, and do it in a way that helped others not be exposed to the external provider complexities.

We wanted to reduce cognitive load for the stream-aligned teams consuming from the platform. And in order to do that, we needed them to be sure they can trust the platform.

The way we did it was the platform team to collaborate with the steam-aligned teams to understand their needs and create collaboration agreements that would fulfill both parts and improve trust.

Conformist instead of anti-corruption layer for internals

When the platform team understood the team needs. We moved from a fast changing internal platform APIs to stable ones. When the exposed model didn’t change often, we could take advantage of that stability to improve team velocity adopting that external model.

If we ensure backwards compatibility, stability, and other stuff. We can take advantage of that system to give speed to others.

Adopting a conformist approach to a stable system gives teams speed. The opposite isn’t true.

Replace Ktor and Expose with Spring Boot

Ktor had a lot of custom build stuff that were repeating cross services. Adding a new use case was easier in the old way than in the new way.

Introducing Spring Boot helped us to reduce the inertia to the new approach and embracing the coherent actions we were taking forward. It helped us establish, this is how we do it now.

Ports and Adapters provided by Spring Boot with CRUD models and some domain models

We will adopt transactional model in those use cases that aren’t domain rich but network intensive. We will create a rich domain model only on certain use cases.

We will use extensively Spring Boot build in features to improve resiliency and reduce custom build adapters.

By adopting a conformist approach between our customer facing services and platform services, we will reduce the ACL overhead we had.

Most of the internal models looked the same. We had analysed the DTOs at network, entity, and database and they had very minor changes.

We reduced the amount of DTOs between entity and database as they were CRUD models, and we kept the Response and entity parsing to ensure we didn’t break any of the existing API contracts.

So, we started to move away from ports and adapters for the new use cases in favor of MVC+S.

Evolve the senior+ engineer expectations

We stop promoting people only in case they show they can do a microservice with a ports and adapters architecture. That’s no longer an incentive for promotion. Instead, they need to show contextual awareness and aim for the right amount of complexity solution that will allow it to evolve as the business evolves. Evolutionary architecture skills become more relevant than creating complex systems for future (and unknown) business complexity from the beginning.

We did this on the next performance review to not affect the current ongoing expectations. No body wants to be pushing in one direction and the management changes the expectations at last moment and it affects people performance review. This change needed to be done with caution on when to introduce it.

Learnings

• Replacing Ktor by Spring Boot and coupling to the framework was our way to materialize the coherent actions we were taking. It was our way to signal to the engineering teams, this is the new way of doing things.

• Replacing customized ports and adapters architecture by leveraging Spring Boot components helped people to reduce their cognitive load and learn from the courses available on Internet.

• Acknowledge that not all our services were core, helped to reduce the people’s expectations of this needs to be high quality, and what the company says it is quality is ports and adapters and microservices.

• Changing how you measure people’s performance is hard and it needs to be taking with care and understanding the people fears and needs.

Fintech Engineering Strategy Post Series

• Post I. A legacy, a deadline, and no team.

• Post II. Building a product vision, and a team, and replacing Ktor with Spring Boot incrementally.

• Post III. Merging two teams for fast flow. [This post]

• Post IV. Simplifying a Ports and Adapters architecture and remove anti-corruption layers made sense from a sociotechnical perspective.

• Post V. Going to production killed fast flow. Understanding how company culture materializes on unaligned team dynamics.

• Post VI. Reducing the chaos before addressing the complex socio-technical system

As Team A, we started to perform and deliver at a good pace.

We were proud of our rhythm of continuous improvement and continuous learning, and the DORA metrics started to be high-performing most of the time.

We stopped being an isolated team as the practices and methodologies got internalized by the team members, and we got exposed to the tribe's challenges.

The tribe had three products owned by three teams with the same name, let’s recall that we are naming them A, B, and C. We were the Team A, but the business bet wasn’t the product A.

The business bet was the product B owned by team B. They got all the business attention and pressure.

The Team B shared some difficulties in delivering the product, like:

• Lack of order and unpredictable results.

• Managing individuals.

• Heroics.

• Stress.

A lot of similarities to what I shared with you in the previous post. Why not apply the same thing in this context? Because context matters.

A key difference was that Product A was mainly company employees, while Product B was a mix of company employees and external consultancy.

As we learned, we had some people leaving the team and we had the opportunity to operate under the radar until we adopted the methodologies.

It would have been a mistake to apply the same approach.

We needed to think as a tribe how to address this high-stake business challenge.

Understanding the organization chart

The company was super big. It had a lot of products, user personas, and departments.

A tribe consisted of:

• Tribe leadership with Product, Business, Design, and Engineering.

• Different team types:

  • Product-Engineering

  • Operations

  • Customer-Success

• Teams were meant to deliver with as high autonomy as possible

• Reporting lines were quite strict. It created information flow too fast between types of teams due to most information being passed either in leadership 1:1s within the same department or slower between “departments”, as they required a meeting with all people involved which were usually led by business.

• Teams were able to gather all the context from different people and departments as the culture of seeking the context was super present and people were open to sharing the context as much as possible. This was the unofficial structure of how to gather information, by creating human relationships to support each other outside the formal communication channel. It worked quite well even though it created pressure within the organization of constant communication to have the whole context in a rapidly changing organization.

Understanding the flow of change

Tribe-level leadership had a key role in pushing progress further. As the products required more teams, the coordination of the initiatives was part of the tribe leadership responsibilities, creating a dynamic on how to distribute work between teams that are needed to deliver a new product feature or a business capability.

As the cost of coordinating and collaborating between multiple teams was higher, we added more importance to planning. It created pressure on tribe leadership to be more correct and unable to collaborate as much as they would like with the teams to have a shared understanding. Making the relationship between tribe and team leadership a management-heavy activity.

Here was the interesting behaviour, if an initiative doesn’t fit in a team backlog, it is pushed into another team’s backlog even though it’s not “completely” their domain.

As domains and responsibilities were blurry, it created a flexible backlog which helped business initiatives to be prioritized each quarter at the same time it started to create technical debt as a solution could be started by one team, and then it made a handover to another team. Ownership wasn’t there.

This worked when the domain complexity was reasonable, which helped with fast business iteration until the domain complexity wasn’t manageable anymore and slowed down the teams’ velocity until the point that it was hard to deliver value.

High-stake business challenge

The business potential of Product B is huge. The need to release the product as soon as possible the business opportunity is there and other players are moving ahead.

Vision

We saw the engineering practices of Team A performing, we bet that investing in Team B to adopt them will help Product B to be generally available sooner.

We considered multiple approaches:

• An enabling team to help Team B with the capability gaps that we identified.

• Split the domain into two teams.

• Merge Team A into Team B, and freeze the Product A development.

We decided to merge both teams and remove the handover from the tribe-level leadership so that Team B can be more autonomous E2E.

Why merge both teams and not the other options?

• Enabling Team

Team A was able to adopt new practices by being under the radar and not having a huge delivery pressure, giving the team the time and space to fail and learn. In a situation of high delivery pressure, it doesn’t have the best conditions for an Enabling Team and facilitation to succeed.

The tendency in this situation is to treat the enabling team as new team members to help with the delivery as more developers. Breaking the purpose of facilitation of an enabling team to fill the team capability gaps.

• Split the domain into two teams

Splitting the domain would have added a lot of pressure on the leadership team and Team B. You need to collaborate first to:

1. Understand the business and product context.

2. Understand the source code.

3. Decide which area to split into two for team responsibilities.

4. Coordinate the new collaboration with Team H (dependent on Team Product B) and the Operations Team. Increasing the pressure on leadership for coordination. (More managers here please!)

The possibility of just being one big team because of the constant required coordination but with splitter dailies was a plausible reality.

We discarded both options in favor of merging teams.

What about Product A development?

We were able to negotiate with the business the priorities, and after assessing product B's potential vs product A potential, it was fine to freeze the development until we overcome this situation.

Merging both teams

By being part of the same team, we could introduce some dynamics that would improve fast knowledge transfer such as pairing or mob programming.

• Team B shares the business, product, and code knowledge with new team members from Team A.

• Previous Team A members are able to share the practices and methodologies with Team B members.

• With pairing, we can create a higher trust relationship between people so that we have a strong relationship in a high-pressure context.

Win-win situation.

I also wanted to reduce the amount of coordination possible between teams at the cost of coordinating 12 team members.

Why?

Isn’t 12 people too much? We knew from past experience that some people left due to the new practices, we expected that this would happen and it would reduce the team size in the following weeks.

Indeed, it was what happened. At least 6 people left the company. It was key to introduce pairing to fast knowledge transfer, otherwise, the impact of that decision would have been huge within the organization. Impact on the delivery of the product, which was the key reason why we started this merge.

So, we improved internal knowledge transfer without the increase of cross-team coordination/collaboration. An essential point for a fast flow of change.

Removing the leadership handover

Changing organizational behavior that has been there for a long time is hard. It becomes part of the culture.

“We always did it this way”

At ThoughtWorks, I learned a way to make the decision process more collaborative and to help teams own the process E2E, using the Lean Inception.

Lean Inception has a lot of positive outcomes, the one I needed was aligning people to build the right product.

Instead of just receiving a backlog, Lean Inception focuses on building it collaboratively by having a conversation with all the important people in the room.

Even though the product was quite defined, I considered Lean Inception to be a great fit for:

1. Aligning people and knowledge transfer. Team A new joiners needed to ask people to the business experts.

2. Reduce scope. What’s done, what’s pending, and what’s essential to the first release.

   1. Moving from we need everything to iterate after releasing.

3. Create a shared ownership of what is being decided.

   1. If I receive a backlog, my buy-in is lower compared to when I understand where it comes from and my opinion is heard.

One key reason why waiting to have it all was that the delivery speed was slow. If I’m able to release every few days, we can plan ahead and know that we are able to handle new requirements at a good pace. On the other hand, if we take months to deliver something, we don’t have the flexibility to release it now and fix it as we go.

Delivery speed gives the option to release sooner rather than later.

That’s why we adopted many of the Team A dynamics to improve the DORA metrics to create the environment that allows us to release without a “feature-complete product”. It gives confidence not only to the engineering team but also to the whole people involved in the product.

The end flow of change based on that change was the next

The leadership didn’t behave as a platform or enabling team (yet), but they were part of the product definition within the team.

Fintech Engineering Strategy Post Series

• Post I. A legacy, a deadline, and no team.

• Post II. Building a product vision, and a team, and replacing Ktor with Spring Boot incrementally. [This post]

• Post III. Merging two teams for fast flow

• Post IV. Simplifying a Ports and Adapters architecture and remove anti-corruption layers made sense from a sociotechnical perspective.

• Post V. Going to production killed fast flow. Understanding how company culture materializes on unaligned team dynamics.

• Post VI. Reducing the chaos before addressing the complex socio-technical system

After the first deadline, we had time to understand the product vision, build the team, and start making steps forward to reduce the technical debt in the areas that could impact the coming deliveries.

The first thing that happened was that people who joined us for the delivery went back to their teams, so we started opening two positions, which we filled with internal rotation from another team within the tribe so that they would have already the context and be familiar with the source code as permanent moves.

The product that we owned as team A, let’s call it product A, is a legacy at least 1-year-old, which was in production for a small set of users before being generally available for all user base.

The tribe had three products, let’s call them A, B, and C. The order in which things happen is relevant.

1. The current tribe started as one team.

2. The team had the goal to build three products with an expectation of lower complexity by externalizing the high complexity to external partners.

3. At the beginning, the product A was the entry for all other products.

4. The products turned out to be more complex than expected.

5. The initial team left and three teams formed to drive those products forward.

Context

Team

Architecture

Macro

As you can guess, a change in any product will impact many shared services, creating the need for coordination between teams and adding processes to prevent a change in one product from breaking the other.

Micro

The service has a “ports and adapters architecture” using Ktor and (IIRC) Exposed for the persistence layer.

We can find a lot of duplicate code as DTOs between the presentation layer and the persistence layer, while the domain logic is almost anemic.

We need to follow the HTTP calls between services to understand the user journey as a whole.

There is a good enough test suite that covers the main use cases for each service.

Team dynamics and characteristics

The leadership and the team members that joined the team shared their pain points and I was able to see some areas of attention such as:

• Lack of order and unpredictable results.

• Stress.

• Managing individuals.

• Heroics.

• Onboardings require three months until people become confident about changing code.

Using the Wardley Mapping Doctrine, I could identify some behavior as Phase 1.

Using the Kanban Maturity Model, I could also identify the team as ML0 Oblivious.

The business challenge

We need to push the product forward to make it generally available. We need predictability and introduce change more easily.

Engineering Strategy

Diagnosis

We detect low-maturity engineering practices followed by several architectural decisions that are slowing down today given the team's knowledge. The team members are split as frontend and backend, creating the need for coordination and splitting the user stories as subtasks that require integration.

The business needs evolved and the product and engineering haven’t kept up due to the lack of people working on the product for a long time.

There are team behaviors based on the previous context that might not apply to today’s team needs.

Direction

We will start behaving as a team and moving away from individual heroics. We will work on creating processes and practices that would lead to the predictability of outputs.

The practices that we adopt are aligned with XP and aligned with creating end-to-end ownership of the feature to be delivered by up-skilling team members to be full-stack developers.

Coherent Actions

Isolate the team

We will keep the team isolated from the rest of the tribe deliveries so that we can focus on adopting the right practices and mature the team before we’re exposed to organizational goals.

Adopt Spring Boot over Ktor and Exposed

We need faster onboarding and lower maintenance for our services. We expect by adopting Spring Boot to be able to adopt off-the-shelf capabilities such as resilient mechanisms and integrations with commodity solutions like observability, queue systems, and testing frameworks.

Change the job descriptions and hiring process

Move the job descriptions from backend/frontend specializations into a product engineering job description, and adapt the hiring process for the new job expectations.

Start adopting the full-stack mindset and upskill team members

We need to train people inside the team for the technologies that they aren’t familiar with and help them own a user story end to end instead of waiting on the integration between components. We expect a lower lead for chances and cycle time.

The training needs to be followed with slack time to practice with the current code base. Pairing is encouraged between people of different skills to boost knowledge transfer.

Remove established processes until things break

We are a new team, and we can decide how can we work within the company principles and guidelines. When in doubt, challenge the status quo. It is better to find the system’s problems by making things fail fast versus moving slower and detecting system flaws later.

Those problems can be at different levels like application, continuous integration, monitoring systems, and how do we plan work. Long story short, the socio-technical system that produces the value.

Adopt DORA metrics

We will measure the impact of our decisions with DORA metrics. Those will be available as weekly metrics so that we can assess how fast we are improving. The goal is the speed of improvement, not the current state of the DORA metrics.

Outputs and Outcomes

Desired

We improved multiple parts of the doctrine as best practices. Yet, we had a lot of work to master Phase I of the Wardley Mapping Doctrine. The good part is that we created awareness and the speed of improvement was impressive.

We also started behaving as ML1 Team-Focused using the Kanban-Maturty Model. We still had delays, defects, and re-work. We started behaving as a team, but we still had some individual heroics in some cases.

The behaviors were relayed to me in order to create new behaviors and dynamics. We needed to keep practicing them so that they became part of the team culture.

We strategically wanted the team to have a different maturity level and practices different from other teams within the tribe in order to start the maturity in one team and then start propagating this engineering culture to other teams incrementally.

Incrementally migrated from Ktor to Spring Boot in Production

We discussed multiple ways to migrate from Ktor to Spring Boot.

1. All at once. Blocking new features for at least one month.

2. Start a new microservice and co-live both for some time. We will implement the new features in the new one, and then slowly migrate previous features to the new microservice.

3. Adopt Spring Boot in the existing Ktor service, and migrate the features to Spring Boot only when necessary.

We did some spikes and we finally chose the third option. Ktor and Spring Boot are in the same repo, the same service. This way, we could reuse the whole code and no need to create new infrastructure and route the features depending on what’s implemented and what’s not.

This way, we were able to reuse all the source code. We exposed Spring Boot, and using Feature Flags, we decided if one use case was implemented and handled by Spring Boot or we call the Ktor running in the same machine, in localhost, to perform the old behaviour.

We accomplished that we didn’t care if a use case wasn’t migrated because it wasn’t a problem as the technical debt and the cost of running both services in the same Docker was minimal.

Undesired

People leaving

The person with the most mobile experience left the company, along with a backend developer. Both were internal rotations, so, a lot of context was lost.

Shortly after their departure, a new product engineer joined the team.

Sharing how to evolve incrementally didn’t stop other teams from performing a full rewrite from Ktor to Spring Boot

We did learning and documented the process so that other teams could benefit from this incremental approach. Unlucky us, some teams chose to do a full migration to Spring Boot and freeze the product delivery until the migration was done.

Learnings

• Isolating a team and giving time for learning was key to adopting a full-stack mindset.

• When people are hired in one role/specialty, and then you adopt a new approach, they might either be on board or just start looking for a workspace that suits their needs.

• Sharing an incremental approach in a sync meeting and documenting seemed enough to help others follow the example, but it didn’t happen. Each team had its own approach to accomplishing a coherent action.

  • Here I have mixed feelings of what happened. On one side, we accomplished the tribe goal of adopting Spring Boot, Good! On the other side, those approaches needed a feature freeze until it was completed. The risk of a full migration up-front approach was way higher than an incremental approach and could cause a way longer delay in the delivery. What happens if the migration takes longer? What happens if we don’t deliver value for more than one month? The business was already concerned with the delivery speed. Missing the migration expected deadline would create a toll on the trust of the engineering team to accomplish deliverables, creating resistance to change in the future.
    IMHO, an unnecessary risk given an alternative such as an incremental approach.

• The new joiner onboarding time and complexity was already lower due to the adoption of Spring Boot.

  • The dynamics in place were already team-oriented instead of individual hero-oriented. This, for the person, was already the norm, and no resistance to those new methodologies compared to existing team members who decided to leave.

  • The hiring process took into account the new behaviors that we were looking for, which contributed to a faster adoption of the new behaviors and engineering culture that we were looking for.

• Changing the hiring process and job descriptions was way more energy-consuming than expected. I had to coordinate the new job expectations with higher leadership and it required a lot of back and forth since those expectations were outside my impact zone.

  • Having a highly collaborative People department was key for the success of implementing a new hiring process and job expectations within a time frame of one month. Kudos to the best People professional I worked with.

• Post I. A legacy, a deadline, and no team. [This post]

• Post II. Building a product vision, and a team, and replacing Ktor with Spring Boot incrementally.

• Post III. Merging two teams for fast flow

• Post IV. Simplifying a Ports and Adapters architecture and remove anti-corruption layers made sense from a sociotechnical perspective.

• Post V. Going to production killed fast flow. Understanding how company culture materializes on unaligned team dynamics.

• Post VI. Reducing the chaos before addressing the complex socio-technical system

You will find an Engineering Strategy that:

• Team level impact, suitable for people in a Tech Lead position.

• Limits Work in Progress to boost velocity.

• Tough deadline with high uncertainty and leadership attention to deliver on time.

• Build a team from 0 with internal rotation.

• Removing bureaucracy during a high risk delivery by implementing novel approaches to fast delivery.

This post assumes you are familiar with the Engineering Strategy structure. Please, read the Designing an Engineering Strategy post series if you find yourself unfamiliar with the structure Diagnosis → Direction → Coherent Actions to describe an Engineering Strategy.

March 2021, I join a FinTech as Engineer Manager in their offices at the lovely and sunny Valencia, Spain.

A FinTech based on São Paulo, Brazil with an office in Spain. It offers loans with the lowest rate on the market.

It had multiple things that made it attractive when I decided to join:

• Big and complex domain. FinTech, loans, and applied to multiple verticals such as loans based on payroll, cars, houses, and business services. I joined the latest business unit.

• Growing product-engineering team.

• Working for a company outside Spain which I could learn a new language.

• Good company values and employee benefits.

• Proved business model.

• Enough time on the market so that there’s an existing software in place that will need to be evolved and maintained (legacy).

• I join as a engineer manager with influence on:

  • Engineering.

  • People.

  • Organization.

• Space to make mistakes and learn from them.

With those points on the table, I considered the company to be a great company to push myself outside my comfort zone.

The context

I start on March 1st of 2021 and I start my onboarding. I recall having my calendar full already for the first three days with sessions like company values, ways of working, the first three months expectation, and so on.

My manager wasn’t in the office, she was in Brazil. Indeed, no one of my team was in the office, I’m the first for this team.

I recall one of the first conversations, the summary could be something like:

• You are hired for a product that hasn’t been released to the public but it needs maintenance and adjustments from time to time because it’s regulated by the Central Bank of Brazil.

• The team was disassembled almost a year ago.

• The main developers and product manager who developed the product from grown-up left the company almost a year ago.

• A couple of developers have the context of the application because we had to do some fixes the past quarter as an exceptional thing.

• We have an important delivery in two months due to new regulations.

  • We don’t know what needs to be delivered because we don’t have the full picture of what’s done and what’s missing.

  • We have Business Experts and a Product Manager that’s working to capture those requirements and will be delivered to you.

• You need to hire your team, and onboard them.

TLDR; You have a deadline. We don’t know what needs to be delivered yet. You don’t have a team, you need to hire people and onboard them, while being familiar with a code base that the people who wrote it is no longer here.

I loved the challenge. Let’s see what we done to overcome the situation.

Engineering Strategy

Context

My leader added me to several meetings to meet the business experts that could provide me as much as context as needed while supporting me on a different fronts like hiring, understanding the business requirements, and find the people that knew something about the software in place.

Organization and product structure

I’m part of the Team A, and I co-live in a tribe with three other teams. Each team is responsible for a product.

Architecture

The architecture diagram is messy and incomplete because the point is to show the coupling and services ownership.

DevEx

• Deployment on backend is fully automated and don’t require human intervention.

• Delivery experience on mobile require deeper expertise on how it works to deliver features in the next app publishing process to the Google Play and Apple Store.

• Both components have tests.

Diagnosis

• The product is heavy regulated and requires Brazilian regulation experience to understand and the new regulatory measures by the Central Bank of Brazil, what what is missing in our product to fulfill those regulatory requirements. Which means, all the product information is in Portuguese.

• The software is a mobile application based in React Native.

• The backend is a set of microservices in Kotlin and Ktor that talks with each other via HTTP APIs.

• The product is part of a set of initiatives that got more priority and it created technical debt on how the different products co-live in the mobile application and the logic is implemented cross microservices.

  • We can see the coupling of the services based on the past product prioritization.

  • We cannot fix the technical debt as part of the delivery.

Direction

We will focus on high visibility of the initiative state and communicate often with the main stakeholders. We will limit the work in progress and we will keep a small team with high cohesion on the delivery by doing pairing to limit the risk of integrating the mobile and backend later than sooner.

Actions

• Build a team with internal people, preference if they already worked with the source code before.

• Limit the work in progress to one, if not possible due to blocking external dependencies, two is OK.

• Limit people working in the initiative to limit the onboarding pressure.

• Pairing for fast knowledge transfer.

• Removing PR at least two approval requirements.

• Do not change technologies nor aim to reduce technical debt as part of the delivery.

• Start by mobile user journey and adapt the backend to support the new requirements.

Outputs and outcomes

Desired

• The delivery risk was more about the unknown state of the product than the new requirements that needed to be implemented.

  • A lot of the pressure went away when the PM collected the new requirements within the first two weeks and we were able to plan accordingly.

• An internal mobile developer joins the team to support the mobile part based on Brazil.

• A recent hire joins the team based on Spain.

• An internal experienced product manager joins the team.

• 8 of the 11 new features are delivered on time, the remaining with low priority low risk are delivered on the following week after the deadline.

• We become more familiar with the source code.

Undesired

• The mobile developer returns to their initial team and the mobile knowledge is lost in the team.

• Reaching the milestone shows that the product has lower priority, we lack a product vision, documentation, and understanding the past decisions.

Learnings

• Having a supporting leader is crucial.

• Having a supporting network with other teams and people is crucial.

• The higher the delivery pressure is, you will have more resources available to fix the problem ASAP but it can cause more harm than good.

  • Here, based on the experience I had, limiting the work in progress and not onbording new people creates better results than the opposite.

    • High cohesive team over a lot of individuals with the temporarily mission of delivering some features so they can come again to their usual teams.

• Remove delivery blockers as soon as you find them.

  • Being part of a high risk delivery, novel practices such as removing PR bureaucracy are good time to introduce due to pressure of “just deliver it ASAP, do what you need to do”.

    • The developer that did internal rotation wondered why that’s possible to change the status quo within one week and stay with the new approach.

      • They went from, this isn’t possible to we did it in a record time.

• Communicating often, async and sync, and showing progress early creates trust with leadership and team.

• If the new requirements would have been unachievable with the current team of four people, we could have known during the first two weeks of delivery. Leaving us with a month and a half to take more drastic decisions, but lucky us it wasn’t the case.

Today’s post is made in collaboration with Miguel Ángel Fernández, a Product Engineer at Qatium.

Miguel Ángel and I have been in touch after I published several posts on Engineering Strategy and wondered if someone would like to attend a Workshop about it.

Aleix's Learnings

Designing Engineering Strategy Workshop

Why At beginning of this year, when I was Designing the Engineering Strategy for the Tribe at Creditas, where I work, I realized that the approach I learned during trial and error could be helpful for others. That’s why I collected my learnings as a post series here 👇…

Read more

4 years ago · 3 comments · Aleix Morgadas

We chatted so l could understand better his workshop expectations. We were able to exchange some ideas and thoughts about our experiences in Engineering Strategy. I loved that conversation! We both faced similar situations and we were lost as everyone else in this field :D

Last week, we chatted about the current situation he is facing at Qatium, and how to drive an Engineering Strategy from there.

Miguel Ángel came with a deep analysis of the current situation, the company pains, the developer pains, where the product is, and how engineering is facing some challenges to adapt to the new product reality. I can tell that a meaningful reflection was made before the call, I played a rubber duck/facilitator role in that conversation.

What's Qatium?

Qatium is a water management platform that allows water utilities to improve their water network performance. Clearly speaking, we help water companies to reduce the water and energy that is lost on the way from the supply to the tap. If you are curious you can find more information at qatium.com

Which is the High-Stake Problem?

The product/business found the market fit and started moving to a prepare for scale phase while the engineering stayed approaching solutions with the market-fit mindset.

We identified the need to adapt the dev team mindset to the new 

Engineering Strategy

We decided to go for a classical strategy analysis as explained in the book good strategy/bad strategy, where the first step y stop and listen to what is happening in the organization, collect facts, and refine into insights

Analysis

After a few weeks of collecting facts, these are a few samples of what we saw. Nothing special or different from other organizations as you see.

• Integration tests are too slow

• An increasing number of bugs

• Increasing cognitive load

• A pile of architectural debt accumulated

• Large lead time in features without uncertainty

• Performance problems in different areas

Thinking about the whole range of facts is what led us to discover the high stake problem. The high stake is no more than a scientific hypothesis, so let’s go and test it

Direction

Our hypothesis was that the code was working for seeking a market fit, which means that the code is prepared for accepting features not related to each other and with a high degree of uncertainty. Once we started developing cohesive features and receiving more load is when we started with the problems. Inspired by the techniques described in Strategic Monolith and microservices we decided to plan a Context Remapping. That was new for us because seeking product-market fit leaves little space to actually sit down and think which are the best next features, the fact is you don’t know.

Now with a clear direction for the product we could actually anticipate and plan the system for specific extensions, anticipate performance problems, and even reduce the cognitive load by applying the Inverse Conway Manoeuvre.

This was one of the diagrams we used to remap the context. There is no secret to showing how to structure the code, that should match our roadmap, that by the way is open: https://qatium.com/roadmap/

Coherent Actions

With a direction present, it is time for describing actions. Through small daily efforts, we should be able to test if our hypothesis was correct or not in a short period of time.

Reflections

Can I participate in the Open Engineering Strategies?

Yes! I started a section about this at github.com/aleixmorgadas/awesome-engineering-strategy#open-engineering-strategies 🥳

If you want me to help you and co-author a post here, let me know at:

• twitter.com/aleixmorgadas, DMs are open or

• hello@aleixmorgadas.dev

Meme for you

Aleix's Learnings

Community Startup - Open Engineering Strategy

We all talk about strategy, but very few we can share them publicly. The main reason is that a Strategy contains business risks, opportunities, and other critical information that your competitors might use. It’s normal that those documents not to be available publicly…

Read more

4 years ago · 1 like · Aleix Morgadas

Explaining an Engineering Strategy is context-specific and hard to express in text. That’s why I recorded myself doing a video-based explanation of the previous post!

I will love to have your feedback if this is useful for you! 🧡

Please, share your thoughts in the post 😊

We all talk about strategy, but very few we can share them publicly.

The main reason is that a Strategy contains business risks, opportunities, and other critical information that your competitors might use. It’s normal that those documents not to be available publicly.

Another reason is that we might not know how to do strategy, and we don’t want to look stupid.

I questioned myself for a long time.

Am I doing strategy right? How others are doing this? why do they look so confident? I’m missing something.

The truth was, we all were making it up as we go 🤣

That’s why I did my post series about Designing an Engineering Strategy. I wanted to share the method I’m using after failing on Strategy for several years.

Today, I want to share a real application of the Designing an Engineering Strategy for a product I worked with.

Context

The product isn’t available due to we didn’t prove Product Market Fit with the MVP, and the decision was to discontinue it 😞

Yet, the Strategy we took it’s relevant to share with others in similar situations 🧡

⚠️ I’ll not use the specific case of the company and product because I want to make it a little more abstract to fit more use cases; otherwise, I would need to explain the whole company context, and we will be missing the point.

Business Hypothesis

As with any early-stage startup, you have a lot of assumptions and very few facts. You need to be comfortable with the uncertainty and take risks to find the information to help you know if you’re going in the right direction.

A well-known approach to do it is by using an MVP, and an MVP needs to provide:

• A business-hypothesis. What do we want to prove? What do we want to learn?

• A user-persona. To whom we are doing this?

• A set of features. What will we deliver to fulfill the user needs?

• Metrics to prove the business hypothesis. Which numbers do we need to have to prove that our business hypothesis is right?

Let’s consider the next generic business hypothesis that could suit several community/social-network startups:

We consider that exists the need for a place for the community to talk about and share their experiences for X topic (assumption).

By providing a web application that suits the user needs (MVP), we will measure (numbers made up):

• At least N accounts registered

• 5% Accounts Create Content

• 15% engagement by other’s users

• S% internal searches

(business hypothesis validation)

High-Stake Business Problem

From Business Hypothesis to Engineering Strategy

As I share in Designing an Engineering Strategy Part II, in creating an Engineering Strategy, you need to understand the business first.

To understand the business needs, we ran a Lean Inception Workshop with all the team. Lean Inception says:

Think Big,
Start Small,
Learn Fast.

Lean Inception is a good method to align people and build the right product. After the workshop, you have a good understanding of why you want to build this product, what needs to be done, and what we need to measure to prove the business hypothesis.

After that point, we were able to Design our Engineering Strategy!

Analysis

ProductTech Team Size

• 1 Product Owner

• 1 Product Designer

• 2 Full Stack Developers

There was no intention to scale up the team until Product Market Fit was accomplished or we had a good insight that we were going in the right direction to start a bigger founding round.

Need for fast experimentation

We identified several assumptions meanwhile the Lean Inception Workshop, which translates to we need to be flexible on the solution, we shouldn’t go deep into the business domain until we prove the need, and we need to adapt fast as we learn along with the users.

Technologies

The team was comfortable working with React, Django, PostgreSQL, and AWS. No need to change the technology stack.

🚧 We had a previous product built on that stack, thinking of reusing the same monolith for a new product. This could help us reduce the Time to Market by not redeploying a new service but reusing the previous.

We took into account maintainability. We don’t want to create as much as systems that later will lock us down to them and be slower due to premature early decisions.

Measuring

We had Google Analytics in place for the inbound marketing and user behavior. We considered using Mixpanel to have a more curated analysis of our users and be able to learn from them faster.

Understanding the Landscape with WardleyMapping

To understand our landscape, we thought about the user needs and how they are being solved in the industry.

We were able to place them in a Wardley Map, and we identified that most of the social part of the product was already solved by SaaS products. We wanted to identify what’s not solved and needs to be custom-built and what we can externalize.

Lucky for us, we found that the more tedious part, we could use GetStream for our solution. 🚀

DDDing with Bounded Contexts and Subdomains

Combining multiple tools to understand our product can drive better outcomes. That’s why we also used some patterns of DDD to understand our Business Domain.

Using DDD, we identified a similar scenario with Wardley Mapping. Probably you’re already biased toward a problem understanding, but it’s good to have another perspective and challenge assumptions.

We used the core-domain-charts technique to identify the higher business differentiator and the supportive models.

An MVP should aim to be as simplest as possible. That’s why we have only two subdomains 😄

At the end of the analysis, we concluded that we have the company bounded context with two subdomains:

• User Profile. Our core domain where we have more business differentiation

• SocialFeed. Our supportive domain that we need to offer the community/social network capabilities, but they don’t offer any business differentiator nowadays. We need to externalize this one as much as we can.

Direction

Our Engineering Strategy direction is:

Make heavy use of external providers and solutions to leverage our product without creating custom solutions where we cannot compete; neither by business diferenciation nor at cost.

Our MVP needs to be constant adaptation based on:

• Learning meanwhile delivering with our users.

• Deploy rapid adaptations as we learn from our users.

Coherent Actions

Heavy use GetStream

Building a social network-like solution might consume all the company resources. We must not take that path. We’re not innovating the social network ecosystem. We are filling the need for a community to have a space to talk about X topic.

• We will leverage our product features up to the getstream features.

• We will not customize their product.

Deploy aggressively, recover fast and externalize brutally

We need the shortest deployment cycles and infrastructure maintenance, so the developer team focuses on the product and not infra.

• Heavy use AWS products.

  • ECS Fargate to deploy our monolith

  • RDS as managed PostgreSQL service

  • DynamoDB and Lambda to record events. See code here

• Automate repetitive work as much as we can.

  • GitHub Actions with Continuous Delivery Pipeline

  • Terraform for IaC

• Heavy use of Django and Django Restframework build-in solutions

• In case of doubt, do a CRUD without logic until proven to be needed

Feature Flags for rapid experimentation

Use a Feature Flag system, like split.io, to leverage rapid experimentation safely.

Measure

Measure as much as we can about user behavior with Mixpanel. Each interaction that can help us validate our business hypothesis must be measured.

How did it go?

We couldn’t prove or dismiss our business hypothesis based on user engagement. BUT! We proved that running a custom solution to serve a niche community was more expensive than anticipated.

In terms of timings:

• September 2020. Lean Inception + Development Started

• October 2020. Functional beta

• November 2020. Product lunched 🚀

• February 2021. Product discontinued

IMHO, it was a success to know and learn that much in that short period of time. Imagine if you don’t:

• Externalize the Social Feed User Needs

• You build everything in-house, requiring more developers, time, and funding

• You don’t measure and challenge your business-hypothesis

That’s why Strategy is so important! Focus on the high-stake situation! Which, in our case, was building something that nobody needed.

Books and Resources

Excalidraw Version

Video Explained

Designing an Engineering Strategy

Aleix's Learnings

Designing an Engineering Strategy. Part I

I divided the topic into four posts/emails. Part I. What the heck is Strategy? [This post] Part II. Understanding the high-stake problem. Part III. Giving direction and coherent action. Part IV. Executing an Engineering Strategy…

Read more

4 years ago · 3 likes · Aleix Morgadas

Lean Inception

Meme for you