Success With Microservices

The Critical C's

Barry S. Stahl

Solution Architect & Developer

@bsstahl@cognitiveinheritance.com

https://CognitiveInheritance.com

Favorite Physicists

1. Harold "Hal" Stahl
2. Carl Sagan
3. Richard Feynman
4. Marie Curie
5. Nikola Tesla
6. Albert Einstein
7. Neil Degrasse Tyson
8. Niels Bohr
9. Galileo Galilei
10. Michael Faraday

Other notables: Stephen Hawking, Edwin Hubble, Leonard Susskind, Christiaan Huygens

Favorite Mathematicians

1. Ada Lovelace
2. Alan Turing
3. Johannes Kepler
4. Rene Descartes
5. Isaac Newton
6. Emmy Noether
7. George Boole
8. Blaise Pascal
9. Johann Gauss
10. Grace Hopper

Other notables: Daphne Koller, Grady Booch, Leonardo Fibonacci, Evelyn Berezin, Benoit Mandelbrot

• See my presentation from CodeMash 2025: How the Fediverse Could Save Democracy & Why It Probably Won't

• Hear my thoughts on why a thriving Fediverse is so critical on David Giard's Technology and Friends show.

• Learn more about the Fediverse at fediverse.party.

• Get your own Mastodon account at Join Mastodon.

  • Don't stress on what server to use, just pick one and go. You can always move later.

• Follow me on Mastodon at @bsstahl@CognitiveInheritance.com.

6 Conversations around building µservice architectures

• Consistency
• Context
• Contract
• Chaos
• Competencies
• Coalescence

Given: A telemetry message is sent by a bus in our network

When: The message requires maintenance action (details to be supplied)

Then: A WorkOrder is created for the bus

And: The bus manufacturer is notified

Data is consistent when it appears the same way when viewed from any perspective

Systems are consistent when all of the data in them is consistent

All consistency is simulated

The Universe is Eventually Consistent

Eliminates temporal coupling

• Enables Patterns & Practices that can improve

  • Resiliency & Fault Tolerance
    • Failures are isolated and self-healing
  • Maintainability & Extensibility
    • Easier to change individual components without disrupting others
  • Performance & Scalability
    • Components operate and scale independently

Get it safe, then get it done

• Do no work until the data is persisted
• Verify the message is well-formed
• Do not verify the message is valid if that requires work
• Once stored, return "Accepted"
  • i.e. HTTP 202

Embrace Eventual Consistency

• There is no such thing as "Full Consistency"
• Our systems RARELY need more than eventual consistency
• Don't force your users or other systems to wait for an answer
• Get it safe, then get it done

Development teams should have conversations around Consistency that are primarily focused around making certain that the system is assumed to be eventually consistency throughout.

The boundary within which a system or sub-system operates

• Actors
  • User
  • Other Sub-systems
  • Other Systems
• Entities
  • Owned (Read/Write)
  • Reference (Read only)

A process for modeling a business domain from the perspective of the business experts

A single business grammar

• Every term has exactly one meaning
• Example: Bus
  • Work Order Context: A vehicle needing diagostics or repair
    • Service Time
    • Repair History
  • Inventory Context: A depreciable asset
    • In-Service date
    • Manufacturer and Type
• Terms are captured in Ubiquitous Language

• The unit of work of all services

• The life-cycle of a single request

• The key to service reliability

Is is NOT possible to reliably make more than one change to system state in a single execution context

Two changes can be made to system state in a single execution context if:

• The 1st change is idempotent
• The 2nd change is unreliable

Should be rare, approved, documented & quickly remediated

• Logging & Telemetry
• Cache-aside

• Legacy systems integration
• Security or major incident

We should make our services idempotent when reasonable to do so

• Improves Reliability
  • Retry at will
• May Hurt Maintainability & Extensibility
  • Once published or utilized, it becomes part of the contract
• Can be abused
  • Unintentional DoS attacks

We must define our Bounded Contexts well and defend our Execution contexts

• Utilize Event Storming
• Avoid Dual-Writes
• Leverage Idempotence when possible

Development teams should have conversations around Context that are primarily focused around the tools and techniques that they intend to use to define their Bounded Contexts and to avoid the Dual-Writes Anti-Pattern.

An agreement between a service provider and the service's consumers

• Operations

• Data Types & Structures

• Quality of Service

  • i.e. SLAs

• Versioning Strategy

  • i.e. compatibility

• Behavior

  • i.e. idempotence

Both directions need to be carefully considered

• Upstream
  • How other systems pass info us
• Downstream
  • How we pass info to other systems

• Commands
  • Requests to modify data
• Queries
  • Requests for data

Synchronous interfaces considered harmful

• Synchronous
  • Combines downstream effort
  • Rapid feedback (see Consistency)
• Asynchronous
  • Improves Reliability
    • Avoids Temporal Coupling
    • Limits cascading failures
  • Maintains Team Agility
    • Single responsibility
  • Avoids Ineficiencies
    • Optimized Storage
    • Eliminates noisy-neighbors

Use the best-fit approach to publication

• Publish what you Know

  • Domain Events
  • Event Carried State Transfer

• Claim Check

  • Minimal messages with links
  • Requires an API

• Event Sourcing

  • When interpretation of events may change

• Request-Reply

  • REST over Reliable Messaging

Maintain a distinct internal model

• Preserve team agility
• Stability for Consumers
• Robustness to Change

• Only one service should write
• Only services in THIS SUBSYSTEM should read

Once a message is defined, all stakeholders have a compatibility expectation

• Protect your internal, isolated stream for rapid iteration
• Use upstream contracts to bring external data to local stores
• Avoid creating predicate queries for downstream use

Development teams should have conversations around Contract that are primarily focused around creating processes that define any integration contracts for both upstream and downstream services, and serve to defend their internal data representations and implementations against any external consumers.

Software will fail -- in the worst possible way and at the worst possible time

• Have low expectations of commodity infrastructure

  • Networks will segment
  • Pods, Servers and Drives will fail
  • Containers and OSs will become unstable

• Remember The Fallacies of Distributed Computing

Embrace the fact that failures will occur

• Analytical Testing

  • FMEA
  • Virtual Chaos Engineering

• Empirical Testing

  • Leverage Feature Flags
  • Simian Army (Chaos Monkey, etc)

• Start by Testing in Lower Environments

Development teams should have conversations around Chaos that are primarily focused around procedures for identifying and remediating possible failure points in the application.

The skills and capabilities that differentiate us from our competitors

• Focus on processes that
  • Are unique to our business
    • Provide competitive advantage
  • Change frequently
  • Are complex parts of the domain
    • Require frequent comms with domain experts

Avoid building in Supporting and Generic Domains

• Supporting Domains

  • Inventory Management
  • Employee Relations

• Generic Domains

  • Feature Flagging
  • Logging & Tracing

Development teams should have conversations around Competencies that are primarily focused around what systems, sub-systems, and components should be built, which should be installed off-the-shelf, and what libraries or infrastructure capabilities should be utilized.

Automate gathering operational data

• Deployment & Verfication Testing
  • Know if a deployment is successful
• Logging, Tracing, & Monitoring
  • Know if a system is unstable
  • Diagnose problems quickly
• Feature Flagging
  • Turn off a feature quickly
• Dependency Status
  • Know when required dependencies are unstable

Development teams should have conversations around Coalescence that are primarily focused around what data streams are available that provide insight into the system, and how to make them available easily to all who might need them.

• Start having them early

  • Later today will do fine

• Don't try to do them all at once

  • Never more than 90 min at a time
  • Usually 30-60 min

• Get help from local experts

• Leverage Domain Driven Design

6 Conversations around building µservice architectures

• Consistency
• Context
• Contract
• Chaos
• Competencies
• Coalescence

• Reference Website: Critical C's Website
• Blog Post: Cognitive Inheritance: Critical Questions to Ask Your Team about Microservices
• Microsoft Learn: Data Considerations for Microservices
• Book: Domain-Driven Design Distilled
• Coursera Course: Google Site Reliability Engineering
• Wikipedia: The Fallacies of Distributed Computing
• Video: Eliminating Hidden Dangers to your Applications: Patterns for Reliable Systems