Or: why a controller and a Service Bus consumer are two front doors onto one pipeline — and why the "onion" in this repo isn't the onion you think it is

TL;DR — the code lives at github.com/architect4hire/jobboard. JobBoard is an event-driven, multi-service .NET reference build, and this post is about four patterns inside it — the layered "onion" stack, a hand-rolled transactional outbox/inbox, an event-fed read projection, and the lightweight CQRS shape they add up to. The angle isn't "here are four patterns." Everybody's got four patterns. The angle is that these four aren't independent choices — they compose, and pulling the thread on one drags the other three into view. Everything below is me showing my work, with verbatim code from the repo at HEAD.

Let me open with the sentence I want you to hold onto for the whole post, because it's the thing most pattern write-ups get exactly backwards.

The outbox is what makes the projection reliable. The projection is what makes the CQRS read fast and decoupled. And the same five-layer stack is the one skeleton reused for both the write path and the projection-maintenance path.

Read a hundred blog posts and you'll get the outbox pattern in isolation, the read-model pattern in isolation, a CQRS diagram with a lightning bolt in the middle labeled "events (magic happens here)." What you rarely get is the load-bearing truth that these things only earn their complexity together. So I'm going to walk one feature front to back — a candidate viewing their own job applications — and show you all four patterns firing on a single screen, then take each one apart.

The one feature that demonstrates all four at once

GET /applications/mine — a candidate's applications, enriched with each job's title and the employer's company name, with zero synchronous cross-service calls on the read path.

Nobody writes to applicationsdb except Applications' own code. Jobs and Profiles publish facts to the bus and never learn who's listening. Here's the wiring that makes it real — and note the detail that makes it cheap (src/JobBoard.AppHost/AppHost.cs:60-88):

// Jobs already published JobPosted (Notifications and Audit consume it). Applications is a THIRD
// subscriber bolted onto a pre-existing topic — it mirrors Title/EmployerId into a local projection
// so "my applications" can show a job title without ever calling back into Jobs (ADR-0012 option B).
var jobPostedTopic = serviceBus.AddServiceBusTopic("JobPosted");
jobPostedTopic.AddServiceBusSubscription("notifications-jobposted");
jobPostedTopic.AddServiceBusSubscription("audit-jobposted");
jobPostedTopic.AddServiceBusSubscription("applications-jobposted");   // bolted onto an existing topic

// State-transfer twin of ProfileUpdated: this one carries the company name itself, so it is
// Applications-only — ProfileUpdated stays the PII-free fact Audit sees.
var employerProfileChangedTopic = serviceBus.AddServiceBusTopic("EmployerProfileChanged");   // genuinely new
employerProfileChangedTopic.AddServiceBusSubscription("applications-employerprofilechanged");

That comment is doing more than documenting a subscription — it's the whole thesis in eight lines. JobPosted costs one new subscription on a topic that already existed. Only EmployerProfileChanged is genuinely new, because Profiles had no event carrying a company name and I refused to leak PII into the audit fact to get one. Adding a reader was additive and boring. That's what "decoupled" is supposed to feel like, and almost never does.

Now let's take the four apart.

Pattern 1 — The layered stack (and the onion everyone copies)

Every service in JobBoard is a thin host (Controllers/Consumers + a Program.cs composition root) plus a .Core class library holding the whole pipeline: Controller/Consumer → Facade → Business → Data layer → Repository, with three model types at the boundary — ViewModel in, Domain internal, ServiceModel out. Only the Repository touches EF.

Here's one complete write, all five layers, top to bottom. The controller binds and delegates. Nothing else (ApplicationsController.cs:43-50):

[HttpPost]
public async Task<ActionResult<ApplicationDetailServiceModel>> Submit(
    [FromBody] SubmitApplicationViewModel viewModel,
    CancellationToken cancellationToken)
{
    var application = await _facade.SubmitAsync(viewModel, cancellationToken);
    return CreatedAtAction(nameof(Get), new { id = application.Id }, application);
}

The facade validates, then delegates (ApplicationFacade.cs:38-44):

public async Task<ApplicationDetailServiceModel> SubmitAsync(
    SubmitApplicationViewModel viewModel, CancellationToken cancellationToken = default)
{
    await _submitValidator.ValidateAndThrowAsync(viewModel, cancellationToken);
    return await _business.SubmitAsync(viewModel, cancellationToken);
}

The business layer does the shape translation and builds the integration event — no EF, no bus, no validation (ApplicationBusiness.cs:56-65):

public async Task<ApplicationDetailServiceModel> SubmitAsync(
    SubmitApplicationViewModel viewModel, CancellationToken cancellationToken = default)
{
    var application = viewModel.ToEntity();
    var @event = application.ToApplicationSubmitted(_requestContext.RootThread());

    var saved = await _dataLayer.SubmitAsync(application, @event, cancellationToken);
    return saved.ToDetailServiceModel();
}

The data layer is where the interesting part lives — it composes the repository write and the outbox enqueue into one atomic unit, and translates persistence failures into domain exceptions (ApplicationDataLayer.cs:39-63):

public async Task<Application> SubmitAsync(
    Application application, ApplicationSubmitted @event, CancellationToken cancellationToken = default)
{
    try
    {
        return await _repository.ExecuteInTransactionAsync(
            async token =>
            {
                var saved = await _repository.AddAsync(application, token);
                // Same DbContext, same transaction: the event ships iff this row commits.
                await _outbox.EnqueueAsync(@event, token);
                return saved;
            },
            cancellationToken);
    }
    catch (DbUpdateException ex) when (ApplicationRepository.IsDuplicateApplicationViolation(ex))
    {
        throw new DomainException(
            "application.duplicate",
            $"Candidate '{application.CandidateId}' has already applied to job '{application.JobId}'.",
            StatusCodes.Status409Conflict);
    }
}

And the repository is pure EF, nothing else — it doesn't even call SaveChanges (ApplicationRepository.cs:53-57):

public async Task<Application> AddAsync(Application application, CancellationToken cancellationToken = default)
{
    await Context.Applications.AddAsync(application, cancellationToken);
    return application;
}

The three boundary types are a structural defense, not decoration. Look at what the client can and can't set:

public sealed record SubmitApplicationViewModel   // in
{
    public Guid CandidateId { get; init; }
    public Guid JobId { get; init; }
    public string? ResumeReference { get; init; }
}

public class Application                          // internal / EF entity
{
    public Guid Id { get; set; }
    public ApplicationStatus Status { get; set; }
    public DateTime SubmittedOnUtc { get; set; }
    // ...
}

SubmitApplicationViewModel has no Id, no Status, no SubmittedOnUtc. A client can't set "status": "Offered" because the type it binds to doesn't expose the property. If the controller bound straight to the Application entity, the model binder would happily walk a request body right past ApplicationStatus.Submitted — the classic ASP.NET Core over-posting hole. The distinct ViewModel closes it, and it closes it structurally. There's no attribute to forget.

Here's the contrarian bit

I've been calling this the "onion." The repo's own docs call it "the facade → business → data layer → repository stack." And if you're picturing Jason Taylor's Clean Architecture template — concentric projects, Domain.csproj at the dead center depending on nothing, Dependency Inversion doing compiler-enforced work — you're picturing the wrong thing. Let me be precise, because "half of these diagrams get the arrows backwards" is a hill I'll die on.

The five-layer stack in JobBoard is one project, five folders, not five projects. Controller → Facade → Business → Data → Repository all live inside a single .Core library. Nothing stops ApplicationBusiness from using a Data/ type directly — and it does. The dependency direction is straight-line, not inverted: Business depends on IApplicationDataLayer, an interface declared in the same folder as its implementation, not in an inner layer that Data implements from the outside. There is no Dependency Inversion in the classic sense here. It's a top-down call chain with DI-for-testability at each hop. Architecturally, that's N-tier, not ports-and-adapters — and calling it "onion" without saying so is exactly the kind of copied-diagram imprecision that gets teams building a monolith with extra folders and congratulating themselves.

So where's the real onion? One level up, at the microservice boundary, and there it's compiler-enforced:

Contracts ← Shared ← <Service>.Core ← <Service> (host) ← AppHost

JobBoard.Contracts is a leaf that references nothing (the Domain-project role). JobBoard.Shared depends only on Contracts (the cross-cutting-mechanism role). Each service's .Core depends on Shared but never on another service's .Core — and that's not a convention, it's enforced by the absence of a project reference between, say, JobBoard.Applications.Core.csproj and JobBoard.Jobs.Core.csproj. Cross-service coupling isn't banned by a review rule you can forget. It's banned because there's no reference path for it to travel down. The only legal channel between services is an event flowing through Contracts and the bus.

So the onion in this repo isn't the five-layer pipeline — that's an N-tier pipeline inside one project. The onion is Contracts at the center, Shared around it, each service as an outer, non-overlapping petal. A shape you could fairly call microservices-of-layered-monoliths.

Why keep even the loose N-tier split, then, if it's not buying swappability? Because one deployable unit with one database wouldn't get much from a compiler wall between Business and Data — there's only ever one Repository implementation; nothing is swappable independently. The split earns its keep for testability (mock IApplicationDataLayer, unit-test ApplicationBusiness with no database) and for keeping each layer legible. It is not buying the thing Clean Architecture's stricter separation buys. Knowing which of those you're paying for is the difference between an architect and someone who saw a nice diagram.

Pattern 2 — The transactional outbox/inbox, hand-rolled

No MassTransit. No third-party outbox. The whole mechanism lives once, in JobBoard.Shared, and it's about ten small files you can read in a sitting. Let me show you the load-bearing parts and then tell you the honest truth about what it does and doesn't guarantee.

The outbox table's identity is the trick. The event's own Id is reused as the row key and the Service Bus MessageId:

public sealed class OutboxMessage
{
    /// The event's own identity, reused as the row key and the Service Bus MessageId.
    /// Because it is deterministic, a retried write cannot enqueue the same event twice.
    public Guid Id { get; set; }
    public string Type { get; set; } = default!;
    public string Destination { get; set; } = default!;   // the topic
    public string Payload { get; set; } = default!;        // JSON
    public DateTime OccurredOnUtc { get; set; }
    public DateTime? ProcessedOnUtc { get; set; }           // null until relayed
}

builder.Property(m => m.Id).ValueGeneratedNever();   // load-bearing: Id is ALWAYS the caller's event.Id
builder.HasIndex(m => new { m.ProcessedOnUtc, m.OccurredOnUtc });   // the dispatcher's poll query

Enqueue only stages — it never calls SaveChanges, and it guards against double-staging with a FindAsync:

public async Task EnqueueAsync(IIntegrationEvent @event, CancellationToken cancellationToken = default)
{
    // Deterministic id → replay-safe. FindAsync checks the change tracker first, so a retry that
    // re-runs the operation finds the row it already staged and does not add a second one.
    var existing = await _context.OutboxMessages.FindAsync([@event.Id], cancellationToken);
    if (existing is not null) return;

    var eventType = @event.GetType();
    await _context.OutboxMessages.AddAsync(new OutboxMessage
    {
        Id = @event.Id,
        Type = eventType.Name,
        Destination = eventType.Name,   // topic-per-event-type
        Payload = JsonSerializer.Serialize(@event, eventType, SerializerOptions),
        OccurredOnUtc = DateTime.UtcNow,
        ProcessedOnUtc = null,
    }, cancellationToken);
}

Why a FindAsync guard on an insert? Because of the shape of the transaction wrapper every write goes through:

public async Task<T> ExecuteInTransactionAsync<T>(
    Func<CancellationToken, Task<T>> operation, CancellationToken cancellationToken = default)
{
    var strategy = Context.Database.CreateExecutionStrategy();
    return await strategy.ExecuteAsync(async token =>
    {
        await using var transaction = await Context.Database.BeginTransactionAsync(token);
        var result = await operation(token);
        // Flush domain rows AND the outbox row inside the transaction, then commit together.
        // A throw on any leg skips the commit and rolls every staged change back.
        await Context.SaveChangesAsync(token);
        await transaction.CommitAsync(token);
        return result;
    }, cancellationToken);
}

That's a callback, not a using var tx = BeginTransaction(), for a specific reason: Aspire's Npgsql integration turns on retry-on-failure, and EF's execution strategy refuses to run inside a transaction it didn't open itself. Handing the whole unit of work in as a delegate lets the strategy own the boundary and replay the callback on a transient fault. The consequence that ripples through the entire codebase: the callback may run more than once. Which is exactly why the outbox enqueue — and the inbox mark, below — guard with a deterministic-id no-op instead of blindly inserting. The retry safety isn't sprinkled around defensively; it's a direct, traceable consequence of one framework decision.

Dispatch is split deliberately: a BackgroundService timer loop (OutboxDispatcher) and the actual query-send-stamp logic (OutboxRelay), separated so the send logic is unit-testable against a real DbContext and a fake ServiceBusClient without booting a hosted service. The relay's failure handling is the part worth reading closely:

foreach (var row in pending)   // pending = unprocessed, OrderBy(OccurredOnUtc), Take(BatchSize)
{
    try
    {
        var sender = _senders.GetOrAdd(row.Destination, _client.CreateSender);
        await sender.SendMessageAsync(new ServiceBusMessage(row.Payload)
        {
            MessageId = row.Id.ToString(),
            Subject = row.Type,
        }, cancellationToken);
        row.ProcessedOnUtc = DateTime.UtcNow;
    }
    catch (Exception ex) when (ex is not OperationCanceledException)
    {
        _logger.LogError(ex, "Failed to relay outbox message {MessageId}; will retry.", row.Id);
        break;   // preserves ordering — a later row never ships ahead of a stuck earlier one
    }
}
await context.SaveChangesAsync(cancellationToken);   // persists whatever stamped successfully

It sends, then stamps, then saves once per batch. Crash between send and stamp? The row resends next poll with the same MessageId, and Service Bus genuinely redelivers. Which brings us to the receive side, and the single most important line in the whole mechanism:

var processor = _client.CreateProcessor(topic, subscription, new ServiceBusProcessorOptions
{
    AutoCompleteMessages = false,   // ← the key line
});

processor.ProcessMessageAsync += async args =>
{
    await _processor.ProcessAsync(args.Message, args.CancellationToken);
    await args.CompleteMessageAsync(args.Message, args.CancellationToken);   // only on success
};

AutoCompleteMessages = false means any throw inside processing — bad payload, missing DI registration, the consumer's own bug — leaves the message unsettled. The lock expires, Service Bus redelivers. And that redelivery is absorbed by the inbox, a dedupe ledger so thin it's almost nothing:

public async Task MarkProcessedAsync(Guid messageId, CancellationToken cancellationToken = default)
{
    var existing = await _context.InboxMessages.FindAsync([messageId], cancellationToken);
    if (existing is not null) return;   // guards the same execution-strategy replay as the outbox
    await _context.InboxMessages.AddAsync(
        new InboxMessage { MessageId = messageId, ProcessedOnUtc = DateTime.UtcNow }, cancellationToken);
}

The honest version of what this guarantees

Here's the part I want to say plainly, because it's the part that separates a pattern you understand from one you copied.

The whole system is at-least-once, everywhere, never exactly-once. Three separate places dedupe on a deterministic id rather than trying to prevent a duplicate: the enqueue (the strategy can replay the whole business write), the send (a crash between send and stamp resends), and the receive (an unsettled message redelivers). Every one of those checks is plain Guid equality along a single unbroken identity chain: IIntegrationEvent.Id → OutboxMessage.Id → ServiceBusMessage.MessageId → InboxMessage.MessageId. No correlation logic, no payload hashing.

The dispatcher has zero idempotency logic of its own, and doesn't need any — "might send twice" is an accepted outcome, not a bug to prevent. All the correctness burden for "did we already do this?" sits on the consumer's inbox check. That's the sentence to build the whole pattern around: the outbox guarantees at-least-once send; the inbox is what turns that into effectively-once effect. Neither half carries the guarantee alone.

And the caveats, labeled as caveats: ordering is best-effort (the break avoids shipping a later row ahead of a stuck earlier one, but there's no session-partitioned ordering configured, so it's a caveat, not a promise). There's no backoff and no dead-lettering — a permanently-unsendable row retries every five seconds forever and nothing notices. Versus MassTransit, I gave up the built-in retry/redelivery/error-queue pipeline, scheduling, sagas, versioning conventions — a large configurable surface. What I bought is roughly ten single-purpose files, readable start to finish, with one fixed auditable policy instead of a config dialog. On a reference build that trade is correct. On your production system, "no dead-lettering" might be the line item that makes MassTransit worth it. Name the trade out loud; don't inherit it by accident.

Business translation, for the person signing off on this: at-least-once-plus-idempotency is what lets a service go down, come back, and catch up instead of losing writes or double-applying them. The outbox is why "Payments was down for ten minutes" is an availability blip and not a reconciliation nightmare.

Pattern 3 — The event-fed projection

The rule the projection obeys, verbatim from CLAUDE.md: "Duplicate the little reference data you need; don't couple." Applications needs a job's title and an employer's company name to render one screen. Its three options were: call Jobs and Profiles synchronously at read time (chatty, coupled, dead if either is down), share a database (banned outright), or keep its own local, denormalized copies, kept current by consuming events. It picked the third.

The projection tables are plain mirrors, keyed by the id they mirror, never a cross-service FK:

public class JobReference
{
    public Guid JobId { get; set; }
    public string Title { get; set; } = default!;
    public Guid EmployerId { get; set; }
}

And here's the part that ties Pattern 3 back to Pattern 1 — the consumer that maintains this projection re-enters the exact same five-layer stack a controller does. Consumer → Facade → Business → Data → Repository, top to bottom, except the "write" is a projection upsert gated by an inbox check:

// Consumer — one-line delegation, same as a controller
public Task ConsumeAsync(JobPosted @event, CancellationToken cancellationToken = default) =>
    _facade.HandleJobPostedAsync(@event, cancellationToken);

// Data layer — inbox check + repository upsert, one transaction
public Task UpsertJobReferenceAsync(Guid jobId, Guid messageId, string title, Guid employerId, CancellationToken ct) =>
    _repository.ExecuteInTransactionAsync(
        async token =>
        {
            if (await _inbox.HasProcessedAsync(messageId, token)) return;
            await _repository.UpsertJobReferenceAsync(jobId, title, employerId, token);
            await _inbox.MarkProcessedAsync(messageId, token);
        },
        ct);

This is the clearest evidence that "onion layering" and "event consumption" aren't two architectures bolted together. A consumer is just another entry point at the top of the same stack a controller enters at. The stack does not know or care whether it was invoked by an HTTP request or a Service Bus message. Same skeleton, two front doors. That's the compose story made concrete.

A real gotcha, worth the anecdote

Because the projection is entirely event-fed, anything that writes to Jobs or Profiles without going through their outbox never reaches it. This repo hit exactly that. The dev seed data originally wrote demo rows straight to the source DbContext — db.Jobs.AddRange(...) — bypassing the business layer and the outbox entirely. No JobPosted fired for seeded jobs. No EmployerProfileChanged for the seeded employer. So in a fresh environment, JobReference and EmployerReference were permanently empty, and every application against a seeded job rendered "Unknown job / Unknown employer."

The fix was to route the seeders through IOutbox.EnqueueAsync too, using the same mappers production code uses, so seed data exercises the real event path.

The lesson is bigger than seed data: a projection's correctness is only as good as every write path to its sources going through the same outbox — including the ones that don't look like "real" writes: seed data, admin scripts, one-off data fixes. There's no compiler and no migration that catches a write that skips the outbox. The projection just silently starves. If you've read my other stuff, you'll recognize the shape of this — it's the same move as debugging the specification instead of the output. The bug wasn't in the read code. It was in a write path nobody thought of as a write path.

The read itself is deliberately three separate queries stitched in memory, not one SQL JOIN, so a row whose reference data hasn't caught up yet still shows up instead of vanishing:

return applications.Select(a =>
{
    jobs.TryGetValue(a.JobId, out var job);
    var employerId = job?.EmployerId ?? Guid.Empty;
    employers.TryGetValue(employerId, out var employer);

    return new ApplicationHistoryServiceModel(
        a.Id, a.JobId, job?.Title ?? "Unknown job",
        employerId, employer?.CompanyName ?? "Unknown employer",
        a.Status, a.SubmittedOnUtc, a.StatusChangedOnUtc);
}).ToList();

Graceful degradation by construction: eventual consistency means the projection lags the source by however long the poll and processing take, and this read shows a sensible placeholder during that window instead of a 500 or a missing row. One honest wrinkle to flag precisely — the employer id set is derived from the jobs dictionary's values, so a missing JobReference cascades into a missing employer lookup even when the EmployerReference row exists. A narrow, real consequence of joining in application code instead of SQL. Worth knowing before it surprises you in a log.

Pattern 4 — Lightweight CQRS (and where the pattern actually ends)

Two things in this codebase get called "CQRS-ish," and only one of them is the real thing.

The minor sense: every write takes a ViewModel, every read returns a ServiceModel — command/query separation at the type level (that's the over-posting argument from Pattern 1). That's real and useful, but it's the same store, same schema, same code path, typed narrowly at each edge. One sentence, not a pattern.

The real sense is JobReference/EmployerReference themselves, and they qualify on four counts. They're a different shape than the write model (two aggregates in two databases, flattened into one row purpose-built for one query). They're physically separate storage — living in applicationsdb, a different database owned by a different service than the rows they mirror, which is more aggressive than the usual single-service two-tables-in-one-database CQRS split; the read model here crosses a boundary the write models can never cross back. They're asynchronously, eventually consistent — the read lags the write by the outbox poll plus consumer time, and the seed-data anecdote above is a direct consequence of that asynchrony. And they're independently query-optimized — the three-query stitch is shaped around one screen, with no obligation to serve any other access pattern.

Now the part most CQRS posts skip: where the pattern ends. There is no command/query bus here — no MediatR IRequest/IRequestHandler. There's no event sourcing — the write side is plain CRUD over Job/EmployerProfile, not an append-only log replayed to build state. There's no separate read-side database technology — still Postgres, not a search index. And the read model isn't rebuildable from history — it's upserted incrementally by consumers, so recovering from data loss means re-seeding from the source services' current state, not replaying old events, because the outbox doesn't retain processed rows forever.

So call it CQRS in spirit, not the full pattern: a materialized, service-owned read projection kept live by the outbox/inbox mechanism, without the heavier machinery — event sourcing, a command bus, snapshotting — that capital-C CQRS usually implies. Saying which parts you have and which you don't is the entire value of naming the pattern. "We do CQRS" with a hand-wave tells your team nothing about whether they can replay events (they can't) or whether the read lags (it does).

The read path even tightens security while it's at it. The candidate id comes from the JWT via the ambient request context the gateway projects — never a client-supplied parameter:

public Task<IReadOnlyList<ApplicationHistoryServiceModel>> ListMineAsync(CancellationToken ct = default)
{
    var candidateId = _requestContext.ActorId
        ?? throw new DomainException("application.unauthenticated",
            "No authenticated candidate on the request.", StatusCodes.Status401Unauthorized);
    return _dataLayer.ListMineAsync(candidateId, ct);
}

And the gateway needed zero changes for this whole feature — the catch-all route already proxied the path and already required the authenticated policy. That's the tell that this stayed a service-owned read model rather than edge-level composition. The frontend just reads enriched fields off one response, no client-side fan-out to a Jobs endpoint and a Profiles endpoint to assemble a row. The projection did that work asynchronously, ahead of time.

How the four compose — the whole story in six steps

Walk it front to back and every pattern shows up in order:

1. An employer posts a job. Pattern 1's five-layer write flow. Business builds JobPosted with Title/EmployerId already denormalized onto it. The data layer enqueues it into jobsdb.OutboxMessages in the same transaction as the Job row — durable before anything hits a broker.
2. Jobs' OutboxDispatcher notices within ~5s (Pattern 2), sends with MessageId = the row's own Id, stamps it processed.
3. Three subscribers wake up on the pre-existing JobPosted topic — Notifications, Audit, and now Applications.
4. JobPostedConsumer re-enters the same five-layer stack (Pattern 1 again), except this "write" maintains a projection (Pattern 3), gated by an inbox check so redelivery is a no-op (Pattern 2).
5. A candidate applies — another instance of the Pattern 1 write flow, publishing ApplicationSubmitted the same way.
6. The candidate opens "my applications." GET /applications/mine never touches Jobs or Profiles — it reads Application joined in application code against the projection built asynchronously in steps 1–4, scoped to the JWT-derived actor (Pattern 4).

The one sentence I'd tattoo on this build: the outbox is what makes the projection safe to build — durable, at-least-once delivery instead of "hope the HTTP call succeeds"; the projection is what makes the CQRS-style read fast and decoupled — no fan-out at read time; and the same five-layer stack is the one piece of scaffolding reused for both halves, because a controller and a Service Bus consumer are just two different front doors onto an identical pipeline underneath.

Four patterns. One skeleton. Zero of them worth the complexity alone.

The Practical Version

Stealing this for your own build:

Name your onion honestly. If it's five folders in one project with straight-line dependencies, it's N-tier — say so, and reserve "onion" for the boundary where the compiler actually enforces the arrows. Half the "clean architecture" diagrams in the wild are N-tier with better branding, and the imprecision costs you when someone tries to swap a layer that was never swappable.

Make the correctness live where the duplicates land, not where they're born. At-least-once send plus idempotent-on-a-deterministic-id receive beats chasing exactly-once you'll never actually get. One identity chain, Guid equality at every hop, no payload hashing.

Reuse one stack for both front doors. A consumer is an entry point, not a second architecture. If maintaining a projection looks nothing like handling a request, you've built two things where one would do.

Audit every write path to a projection's sources — including the ones that don't look like writes. Seed data, admin scripts, data fixes. Anything that skips the outbox silently starves the read model, and nothing in the toolchain will warn you.

Say where the pattern ends. "CQRS in spirit" that can't replay events and lags by five seconds is a fine, honest thing to build — but only if you tell your team those two facts out loud. The naming is worthless if it hides the boundaries.

The repo is at github.com/architect4hire/jobboard, and the SCRUB framework is how the specification for all of this stayed honest while it was generated. Read the four patterns as one system, not four — that's the part the diagrams never show you.

Robert Felkins is the principal architect at Architect4Hire and the author of the upcoming book Practical AI Assisted Development. He builds right-sized .NET and Azure systems, modernizes the ones you already have, and brings AI-assisted development into teams in a way that survives contact with production.

If your team is standing up an event-driven system that has to be correct under failure — outboxes, projections, the eventual-consistency gotchas that don't show up until production — that's the work I do. Fractional architecture engagements: senior architectural judgment a few days a month, without the full-time price tag. Reach out at robert@architect4hire.com.

A client needed full traceability — cradle to grave, every request — so support could replay a session and answer who did what, when, and where. Good actor or bad actor. My first plan was distributed tracing. It was wrong, and noticing it was wrong was the entire job. "Audit" and "trace" wear the same three words and point at opposite systems: one tells engineers how slow, the other tells support what happened and who.

This is how I built the second one into a five-service .NET stack with SCRUB and Claude Code — the reframe, the security hole it flushed out, the boring-tech call, and the code that folds it into an event-driven system I was already running. All grounded in real code and logic from github.com/architect4hire/jobboard.

There's a version of this post that's just an audit-trail implementation walkthrough. That's not the interesting part. The interesting part is that the request arrived wearing a disguise, and the single most valuable thing I did on the whole engagement was take the disguise off before writing a line of code. So let me start there, because the design lives or dies on it.

The three words that hide two systems

The ask, verbatim in spirit: "full cross-cutting audit, cradle to grave, for every request." My instinct — and my first plan — was distributed tracing. OpenTelemetry, a traceparent propagated across the bus, a Jaeger or Seq backend to look at it all. It's the reflex any distributed-systems person has when they hear "trace every request," and it was completely wrong.

Because two different use cases wear those exact three words, and they're opposites:

• Distributed tracing answers "how slow, and where's the latency?" It's for engineers. Spans, sampling, ephemeral by design, viewed in a trace UI, reset every run. You keep it long enough to debug a performance problem, then it ages out.
• Support audit answers "what happened to this request, this entity — and who did it?" It's for support, compliance, and security. Durable, queryable, attributed to a real actor, kept.

The tell was one phrase the client kept using: "so an agent can query it again." That is not a trace viewer. A trace viewer answers "give me this one trace." A support agent needs to answer "what did user Y do on Tuesday," and "what caused this account to end up in that state" — months later, from a stored record. The moment you hear query it again, you're not building observability for engineers. You're building a queryable system of record.

This is the whole reason I frame the work the way I do. The first bug wasn't in any code — it was in the word "audit." I spend a lot of time telling clients I practice debugging the specification, not the output, and this is the cleanest example I've got:

The hardest part of "add full traceability" was noticing it wasn't a tracing problem at all.

Get that wrong and no amount of good engineering saves you. You'd build a beautiful OTEL pipeline, demo it, and hand support a tool that can't answer a single one of their actual questions.

Why not OpenTelemetry, concretely

"Reframed away from the obvious tool" is easy to say and easy to hand-wave. Here's the concrete reasoning, because the client's engineers rightly pushed on it:

• OTEL spans are ephemeral and engineering-facing by default. To keep them you stand up a trace backend, and even then the query model is trace-centric — "fetch this trace id" — not "everything user Y did today." Wrong shape for the question.
• There's no free path from OTEL to a general queryable store. You'd hand-build an exporter or a collector-writer that lands spans somewhere SQL-shaped, and at that point you're reinventing an audit log — badly, and with someone else's data model.
• OTEL answers how slow. Support needs what happened, and who. Different question, different tool.

So I kept the one idea from the tracing world that actually mattered — a correlation id to tie a request's fan-out together — and dropped everything else. Which brings us to the spine.

The spine: put the thread in the fact, not the transport

The system already tells its own story. Every meaningful action publishes an integration event through a transactional outbox — JobPosted, JobClosed, ApplicationSubmitted, ApplicationStatusChanged. The events were there. They just didn't carry the thread that ties them together. So the spine of the whole design is two ids added to IIntegrationEvent:

• CorrelationId — constant across a request's entire fan-out. The thread.
• CausationId — the id of the event that directly caused this one. The parent pointer.

CausationId is the underrated one. Correlation alone gives you a flat pile: "everything that happened in this request." Causation gives you the causal tree — close-job → JobClosed → ApplicationStatusChanged → notification. For support, "what caused this?" is the actual question, and a flat list physically cannot answer it. The parent pointer is the difference between a bucket of events and a story.

Then a design fork worth naming, because it's the kind of choice that quietly decides whether the whole thing works: do you carry the thread in transport metadata — Service Bus ApplicationProperties — or in the event contract itself?

I chose the contract. Transport headers don't land in the durable record a support query reads, and they're awkward to query after the fact. Putting the ids in the fact makes them part of what's permanently recorded. The cost is real and I'm not going to pretend otherwise: it's a contract change that touches every event, every publisher, every consumer. That's the expensive step. It's also the right one.

Here's the event we'll trace for the rest of the post — a brand-new account registering. Notice what the contract carries, and what it refuses to:

namespace JobBoard.Contracts;

public sealed record AccountCreated(
    Guid Id,
    Guid AccountId,
    string Email,
    string Role,
    DateTime OccurredOnUtc) : IIntegrationEvent
{
    public Guid CorrelationId { get; init; }
    public Guid CausationId { get; init; }
    public Guid? ActorId { get; init; }
}

The event is the entire interface between the two services — no shared code, no shared table, just a fact on the bus. And notice what's deliberately absent: no password, no password hash. The event carries only what a support agent reading the trail needs — which account, what email, what role, when. That absence is a design decision I'll come back to, because privacy here is a property of the event's shape, not a filter bolted on downstream.

"Who did it" quietly forced a security fix

Here's where an observability feature turned into a security prerequisite, and it's my favorite part of the whole engagement.

You cannot record who did something if you don't actually trust your answer to "who." And this repo had a latent BOLA/IDOR seam: some services derived the acting user from body-supplied ids rather than from the validated token. On a normal day that's a garden-variety authorization bug. The instant you try to write a trustworthy audit trail, it becomes load-bearing — because an "actor" field populated from a spoofable body value isn't an audit trail, it's a liability that looks like one.

So the "who" column turned a feature into a prerequisite fix — one that already had a Proposed ADR waiting: the gateway must project validated identity inward, and services must stop trusting the body. Which is really one observation stated twice:

Observability and authorization are the same seam from two sides. "Record who did this" is unanswerable until identity is actually trustworthy.

Registration is the interesting edge of this, because it happens before any token exists — there's no ambient authenticated user to attribute the action to. The answer is a self-originated actor: the account is its own actor. One small extension method, added alongside the existing RootThread (a normal edge-authenticated request) and FollowOnThread (an event-to-event chain):

public static AuditThread SelfOriginatedThread(this IRequestContext context, Guid actorId) =>
    new(context.CorrelationId, context.CorrelationId, actorId);

Correlation and causation still come from the request, so the row lines up with everything else in that HTTP call. But the actor is the id the action itself just established — the newly created account's own id — never a client-supplied or ambient one. In the business layer, that's the whole registration path:

public async Task<AccountServiceModel> RegisterAsync(RegisterAccountViewModel viewModel, CancellationToken cancellationToken = default)
{
    var passwordHash = _passwordHasher.Hash(viewModel.Password);
    // Store email normalized so the unique index and login lookups are case/whitespace insensitive.
    var account = (viewModel with { Email = Normalize(viewModel.Email) }).ToEntity(passwordHash);

    // Registration runs before any token exists, so the edge projects no actor: the account is its own
    // actor (self-originated). The event ships iff the insert commits (same transaction, in the data layer).
    var created = account.ToAccountCreated(_requestContext.SelfOriginatedThread(account.Id));

    var saved = await _dataLayer.RegisterAsync(account, created, cancellationToken);
    return saved.ToServiceModel();
}

And because this is the kind of guarantee that regresses silently the first time someone "cleans up" the actor logic, it gets pinned by a test that asserts the property, not the plumbing:

[Fact]
public async Task RegisterAsync_BuildsAccountCreated_WithSelfOriginatedThread_AndNoSecrets()
{
    var dataLayer = new FakeAccountDataLayer();
    var business = CreateBusiness(dataLayer);

    await business.RegisterAsync(
        TestData.RegisterViewModel(email: "New@Example.com", password: "s3cret-password", role: AccountRole.Employer));

    var account = dataLayer.RegisteredAccount!;
    var created = dataLayer.CreatedEvent;
    Assert.NotNull(created);

    Assert.Equal(account.Id, created!.AccountId);
    Assert.Equal("new@example.com", created.Email);
    Assert.Equal("Employer", created.Role);
    Assert.DoesNotContain("s3cret-password", created.ToString());

    // Root of its request thread: correlation from the context, causation is the request's own id, and
    // the actor is the account itself — NOT the ambient context actor (registration is anonymous).
    Assert.Equal(CorrelationId, created.CorrelationId);
    Assert.Equal(CorrelationId, created.CausationId);
    Assert.Equal(account.Id, created.ActorId);
    Assert.NotEqual(ContextActorId, created.ActorId);
}

That same privacy-by-shape instinct runs one level up in the event model. AccountCreated and LoggedIn physically cannot carry a secret because the record has no field for one. LoginFailed goes further: it has no AccountId field at all and a hardcoded, uniform reason — so the trail itself can't be used to enumerate which emails have accounts, mirroring the endpoint's uniform 401. You don't filter the trail to be safe. You shape the facts so an unsafe row is unrepresentable.

Where the trail lives: a new service, on boring tech

A cross-service, one-place-to-query trail runs headfirst into database-per-service — the single most load-bearing rule in this whole codebase. Three options were on the table:

• Per-service audit tables. Most "correct" for the boundary, but a full-lifecycle query has to fan out across five databases and stitch the results. Wrong for "query it from one place."
• Fold it into the existing Notifications service (already a write-only log). Cheap, but it conflates notifications with audit and makes one database dual-purpose. A shortcut that becomes a mess.
• A dedicated Audit service fed by the bus — a consumer that subscribes to everything and appends.

I chose the dedicated service, and the reason matters more than the choice: this isn't a new idea. A cross-service read model is exactly what an audit trail is, and the architecture already had an ADR for the read-model pattern. Adding a service felt heavy right up until I realized I wasn't inventing a pattern — I was applying one I'd already decided on. Per the repo's own constitution, a new service is "propose, don't scaffold," so it went through an ADR and an explicit sign-off, not a silent addition.

On the datastore, Cosmos got floated — schemaless documents, easy query, seems tailor-made for heterogeneous event payloads. I talked us out of it, and this is the fork I'm most glad we surfaced instead of defaulted:

• The Linux Cosmos emulator is preview-grade; it would've been the only non-Postgres store in the stack, for one service.
• Postgres with a jsonb column already gives you document-shaped flexibility for heterogeneous payloads, plus plain SQL for the query surface.
• "An agent can query it" is strictly easier on ubiquitous SQL than on a bespoke dialect.

That's the boring-tech win in one line: reach for the new datastore only when the existing one genuinely can't do the job. jsonb meant it could.

None of this is hand-wired. The topic, the subscription, and every service reference are declared once in the Aspire orchestrator:

var accountCreatedTopic = serviceBus.AddServiceBusTopic("AccountCreated");
accountCreatedTopic.AddServiceBusSubscription("audit-accountcreated");

var identity = builder.AddProject<Projects.JobBoard_Identity>("identity")
    .WithReference(identityDb)
    .WithReference(serviceBus)
    .WaitFor(identityDb)
    .WaitFor(serviceBus)    // runs the outbox dispatcher, so it uses the bus
    .WithEnvironment("Jwt__SigningKey", jwtSigningKey);

Before this change, Identity was a pure request/response service with no bus reference at all. This diff is what turns it into an outbox publisher for the first time.

On the consume side, the thing I care most about is that the Audit service doesn't grow a bespoke class per event type. One generic consumer handles all of them:

public sealed class AuditConsumer<TEvent> : IIntegrationEventConsumer<TEvent>
    where TEvent : IIntegrationEvent
{
    private readonly IAuditFacade _facade;

    public AuditConsumer(IAuditFacade facade) => _facade = facade;

    public Task ConsumeAsync(TEvent @event, CancellationToken cancellationToken = default) =>
        _facade.RecordAsync(@event, cancellationToken);
}

Registered once per event type — a single line:

// Cradle-to-grave events (SCRUB A7): the actions that published nothing before now.
builder.Services.AddIntegrationEventConsumer<AccountCreated, AuditConsumer<AccountCreated>>("audit-accountcreated");

And one mapper turns any event into one row, with a switch that decides the subject and the occurred-at per event type:

public static AuditEntry ToAuditEntry(this IIntegrationEvent @event)
{
    var (subjectId, occurredOnUtc) = Describe(@event);

    return new AuditEntry
    {
        // The event id is the row key: a redelivery collides on the PK and lines up with the inbox key.
        Id = @event.Id,
        EventType = @event.GetType().Name,
        CorrelationId = @event.CorrelationId,
        CausationId = @event.CausationId,
        ActorId = @event.ActorId,
        SubjectId = subjectId,
        OccurredOnUtc = occurredOnUtc,
        Payload = JsonSerializer.Serialize(@event, @event.GetType(), PayloadOptions),
    };
}

private static (Guid? SubjectId, DateTime OccurredOnUtc) Describe(IIntegrationEvent @event) => @event switch
{
    JobPosted e => (e.JobId, e.PostedOnUtc),
    JobClosed e => (e.JobId, e.ClosedOnUtc),
    ApplicationSubmitted e => (e.ApplicationId, e.SubmittedOnUtc),
    ApplicationStatusChanged e => (e.ApplicationId, e.ChangedOnUtc),
    AccountCreated e => (e.AccountId, e.OccurredOnUtc),
    LoggedIn e => (e.AccountId, e.OccurredOnUtc),
    // A failed login has no account id (an unknown email has no account, and we don't disclose which):
    // no subject, just the attempt in the trail.
    LoginFailed e => ((Guid?)null, e.OccurredOnUtc),
    ProfileUpdated e => (e.ProfileId, e.OccurredOnUtc),
    _ => throw new NotSupportedException(
        $"No audit projection for event '{@event.GetType().Name}'. Add its subject id and occurred-at " +
        "to AuditEntryMapper.Describe when auditing a new event (SCRUB A7)."),
};

Adding the next audited action is a few-line diff — a Describe case, a consumer registration, a topic and subscription in the AppHost — not a new consumer class. Hold onto that NotSupportedException at the bottom; it's doing quiet work I'll come back to.

Append-only and idempotent — by reusing what was already there

Two properties make this an audit trail rather than a log that happens to store events, and neither one required new machinery.

Append-only. The trail is never updated or deleted. A correction is a new row. An audit log you can edit isn't one.

Idempotent. Service Bus — and the outbox relay in front of it — is at-least-once. That callback will fire twice eventually, and a handler that isn't safe to run twice is a latent double-effect bug. The audit consumer dedupes through the same inbox every other consumer in the system already uses.

The load-bearing piece on the publish side is that the account row and the outbox row commit in one transaction — so the event ships if and only if the account actually exists:

public async Task<Account> RegisterAsync(Account account, AccountCreated created, CancellationToken cancellationToken = default)
{
    try
    {
        // The insert and the outbox row are one transaction, so AccountCreated ships iff the account
        // commits. The repository only stages the insert; the transaction is what SaveChanges/commits it.
        return await _repository.ExecuteInTransactionAsync(
            async token =>
            {
                var saved = await _repository.AddAsync(account, token);
                await _outbox.EnqueueAsync(created, token);
                return saved;
            },
            cancellationToken);
    }
    catch (DbUpdateException ex) when (AccountRepository.IsDuplicateEmailViolation(ex))
    {
        throw new DomainException(
            "account.email_taken",
            "An account with this email already exists.",
            StatusCodes.Status409Conflict);
    }
}

There is no path where the account exists but the event doesn't, or vice versa. On the consume side, the inbox turns a redelivery into a no-op instead of a duplicate row:

public Task AppendAsync(AuditEntry entry, Guid messageId, CancellationToken cancellationToken = default) =>
    _repository.ExecuteInTransactionAsync(
        async token =>
        {
            // Idempotency: a redelivery of the same message finds its inbox row and no-ops.
            if (await _inbox.HasProcessedAsync(messageId, token))
            {
                return;
            }

            await _repository.AddAsync(entry, token);
            await _inbox.MarkProcessedAsync(messageId, token);
        },
        cancellationToken);

Here's the detail I'd frame on a wall: one id threads the entire pipeline. The event's own Guid, minted once at the publish site, becomes the outbox row's primary key, the Service Bus MessageId, the inbox dedupe key, and the audit row's own primary key. That single choice is what collapses "at-least-once delivery" into "recorded exactly once" with zero extra coordination. No distributed lock, no dedup table you have to reason about separately — the identity of the fact carries all the way through.

And because "atomic" is a claim, not a vibe, it gets a test that forces the failure:

[Fact]
public async Task RegisterAsync_LeavesNoAccountAndNoOutboxRow_WhenEnqueueThrows()
{
    using var harness = new IdentitySqliteHarness();
    var account = TestData.Account(email: "rollback@example.com");

    await using (var context = harness.CreateContext())
    {
        // The insert stages inside the transaction, then the AccountCreated outbox write throws — the
        // account must roll back with it, leaving nothing committed.
        var dataLayer = CreateDataLayer(context, new FakeOutbox { ThrowOnEnqueue = true });
        await Assert.ThrowsAsync<InvalidOperationException>(() => RegisterAsync(dataLayer, account));
    }

    await using var assert = harness.CreateContext();
    Assert.Empty(await assert.Accounts.ToListAsync());
    Assert.Empty(await assert.OutboxMessages.ToListAsync());
}

Coverage rots silently, so I made a robot enforce it

Here's the trap in "cradle to grave": it's only true if every mutating action is audited — including the ones that never emitted an event. The trail already covered the three job-and-application events. But Identity and Profiles were pure request/response services wired to zero bus references: registering an account, logging in, updating a résumé all left no trace support could query. That's a coverage gap, and coverage gaps don't announce themselves. They're discovered six months later by the one query that needed the row that was never written.

You can't fix that with willpower. You make the guardrail cheaper to follow than to skip, and you make something fail loudly when it's skipped:

• A rule (audit.md) that states the invariant — every mutating endpoint publishes an audited event — and loads when the agent touches the relevant code.
• A skill (add-audit-event) that turns "audit this action" into a one-shot playbook: contract → business → data layer → AppHost topic → consumer → mapper case → tests, in the right order.
• A read-only subagent (audit-coverage-checker) that walks the services and flags any mutating endpoint with no audited event.

And remember that NotSupportedException at the bottom of the mapper's switch? That's the loud failure in the code itself. Subscribe Audit to a new topic without teaching the mapper about it, and the pipeline throws instead of quietly writing a half-populated row. The guardrail isn't a document nobody reads — it's a build that stops, a subagent that reports, and an exception that refuses to paper over a gap.

The contrarian bit: the code wasn't the deliverable

Here's the part that surprises people, and it's the actual point of the post.

The valuable artifact of this engagement isn't the audit trail. It's the design, encoded — a handful of ADRs, a rule, a SCRUB prompt sequence, and two skills — such that the next session, human or agent, builds the next slice correctly without re-deriving any of it. The plan is the deliverable. The code is downstream.

That only worked because the system's own documents did the constraining. The CLAUDE.md is itself a SCRUB prompt; the ADRs record the decisions with their rejected alternatives; the area rules encode the boundaries. So when I drove this with Claude Code, the agent could check itself against "no shared database," "propose don't scaffold," "the outbox is the only sender" — and the plan landed inside those rails instead of inventing plausible nonsense outside them.

Three things made the difference, and none of them is "the tool is clever":

• Grounded in the real code first. Before a line was proposed, parallel exploration mapped the actual telemetry, the real outbox path, the existing conventions. You cannot reframe "audit vs tracing" correctly from a summary; you reframe it from what's actually wired up.
• Forks surfaced as decisions, not silently chosen. Audit vs tracing. New service vs fold-in. Postgres vs Cosmos. Each one got raised, reasoned, and recorded. The single best move of the session was the agent disagreeing with the Cosmos idea and explaining why — instead of cheerfully complying with a worse design.
• Small, verifiable steps. I ran this as a SCRUB prompt per layer, each independently reviewable — roughly 15 to 25 files at a time. Because here's the thing nobody selling AI velocity wants to say out loud: leveraging AI doesn't remove the code review from my chore list. It magnifies it. Lightweight, bespoke prompts keep each review small enough that I can actually do it properly.

That's the discipline. Not "generate faster." Generate in slices you can still verify, encode every decision that mattered, and let the repository carry the judgment forward.

What this buys a business

I work with two kinds of decision-makers, usually in the same conversation. So, plainly:

• Replayable answers to "who did what, when." Support and compliance can reconstruct a session from a durable, queryable record — including the causal chain, not just a flat pile of events. That's the capability the client actually asked for, delivered as a system of record rather than a trace viewer they'd have hated.
• A security hole closed as a side effect. Building trustworthy attribution forced a latent BOLA/IDOR seam into the open and onto the roadmap. Audit and authorization turned out to be the same seam; fixing one hardened the other.
• Privacy by construction. The events physically can't leak a password hash, and the failed-login record can't be used to enumerate accounts — because the facts are shaped to be safe, not filtered after the fact.
• Cheap to extend, and it stays honest. Auditing the next action is a few-line diff, and a subagent plus a fail-loud mapper make sure coverage can't quietly rot.
• You own all of it. SCRUB is a discipline, not a dependency, running on boring, ubiquitous tech — Postgres, SQL, the outbox you already had. No proprietary framework to keep paying for.

The honest part

Let me label the scaffold as a scaffold, because I always do.

This was a late-breaking, genuinely large change — a cross-cutting contract change touching every event, publisher, and consumer. On a reference build that stings; on a real enterprise system it's a project. And it's exactly the kind of thing that should have been a day-one non-functional requirement, reasoned out at inception rather than retrofitted. I'll own that I missed it the first time through on both JobBoard and OrderFlow. It's easy to miss and expensive to add late, which is precisely why it's worth writing about.

JobBoard is a reference build — a spike that runs on local Aspire-orchestrated containers talking to the real Service Bus SDK through an emulator. The cross-service read-model ADR that this work leans on was Proposed, not pre-solved, and I'm not going to pretend otherwise. The register→AccountCreated→audit-row path in this post is one action traced clean end to end; the other new events (LoggedIn, LoginFailed, ProfileUpdated) follow the identical shape, and the same coverage discipline is what keeps the rest honest. And the standing advice in the toolkit holds: Aspire, the Service Bus emulator, and Claude Code all ship fast — verify the exact API names against the live docs before you lean on them.

Where this starts

If your team has a system that publishes events but can't answer "who did what to this entity, and what caused it" — or if someone just asked you for "full traceability" and you're about to reach for a tracing stack — the first move isn't picking a tool. It's figuring out which of the two systems hiding inside that request they actually need. Get that wrong and the best engineering in the world builds the wrong thing beautifully.

That's the work I do: architect4hire.com/services/application-development. Read the repo first — github.com/architect4hire/jobboard — and the SCRUB framework in full at architect4hire.com/scrub. If it's the shape of what your team needs, reach out at robert@architect4hire.com. Fractional or contract, the first conversation is the same: what you're trying to build, and whether I'm the right person to help you build it. If I'm not, I'll tell you.

Robert Felkins is the principal architect at Architect4Hire — Chicago-based fractional and contract architecture for small and mid-sized teams. I design right-sized .NET and Azure systems, modernize the ones you already have, and bring AI-assisted development into teams in a way that survives contact with production: senior judgment without the full-time price tag. Reach me at robert@architect4hire.com.

Everyone's racing to write a CLAUDE.md and a folder of skills. But those files are containers — what goes in them is architectural judgment, and if the judgment is wrong, you've just automated a mistake five times faster. I built a five-service event-driven job board with Claude Code, and the .claude/ toolkit isn't the achievement. The architecture that authored it is. Hand the same tools to someone without that experience, and you get a plausible alpha. With it, you get something you'd actually push.

There's a lot of excitement right now about moving AI discipline into the repository — CLAUDE.md files, rules, skills, subagents, hooks. I'm all for it; I do it, and I'll show you exactly how below. But there's a quiet assumption underneath the excitement that's worth dragging into the light: that the toolkit is the hard part. It isn't. Writing a rule that says "no shared database" takes thirty seconds. Knowing that rule is load-bearing — knowing the eleven other boundaries that matter, in what order to prove them, and what breaks in production if you get one wrong — is the work of years. The file is easy. The judgment inside it is not.

That's the whole thesis of this post, and the reason I frame what I do as Architecture-Powered Claude Engineering rather than just Claude engineering. The public proof is at github.com/architect4hire/jobboard: a genuinely functional job board built as five bounded .NET services behind one gateway, an Angular front end, an event bus, all orchestrated locally by Aspire, driven agentically with Claude Code. The .claude/ toolkit that drives it is real, and it's good. But it's downstream. It's architectural experience compiled into a form an agent can execute — and it's only as good as the architecture it encodes.

The tool is neutral. The judgment is not.

Here's the uncomfortable part for anyone selling "just add a CLAUDE.md and go."

Claude Code will happily encode a bad architecture as faithfully as a good one. Ask it to build a job board with no guardrails, and it produces plausible code — code that compiles, demos, and looks right in the diff. Plausible is not correct. In a distributed system, plausible-but-wrong is where the money leaks out: the agent helpfully opens a second database connection to grab data it needs, and now two services share a table, and you have a distributed monolith wearing a microservices costume. It passes review if the reviewer doesn't know what they're looking at. I call this the Plausibility Trap, and no amount of tooling closes it — because the tool has no opinion about whether your boundaries are right. It just executes.

So the interesting question was never can Claude Code build a microservice system. It's: can it build one that behaves correctly when things go wrong, without the architecture drifting — and the answer to that lives entirely in the judgment you feed it. That's the experience I'm actually selling. Not "I know how to write a rules file." Anyone can. I know which rules are worth writing, because I've watched the alternatives fail in production. As my application-development service puts it: I don't just draw the architecture — I build in it, so the design and the code never drift apart. This is how I build the toolkit now. The architecture is why the toolkit is right.

Trace one decision, and you'll see where the value is

Take the single most important rule in the whole repo: no shared database, ever. A service reads and writes only its own database. Watch where that rule actually comes from, because the origin is the point.

It does not start in the .claude/ folder. It starts as a decision, reasoned out in the High-Level Design and recorded in an Architecture Decision Record — ADR-0001, database-per-service — with its context, its consequences, and the alternatives I rejected and why. That ADR exists because I've lived the failure it prevents: two services quietly coupled through a shared table, evolving independently right up until the day one migration takes them both down. The judgment is: pay the cost of duplicating a little reference data now, or pay for a rewrite later. That's not a preference. It's a scar.

Only then does it become tooling. The decision becomes a rule that the agent loads when it touches a service. It turns into a subagent that reviews for the violation. It turns into a compiler-enforced reference graph — the project references are one-way and acyclic, Contracts ← Shared ← Service.Core ← Service host ← AppHost, so a service reaching sideways into another is a build error, not a code-review conversation. I checked the repo: no service's .Core references another's. Not because everyone remembered to be careful — because the architecture made carelessness impossible.

That's the chain that matters: experience → decision → record → guardrail → enforcement. The .claude/ folder is the last two links. The first three are the architect. Remove the architect and the chain still runs — it just encodes whatever plausible thing the agent invented, and enforces that with the same mechanical rigor. You'd get a beautifully automated, ruthlessly consistent mistake.

The toolkit is architecture, encoded four ways

With that framing, here's what's in the .claude/ folder — and notice that every layer is a decision an experienced architect made, not a clever use of a feature.

Rules are the conventions the agent should know, and they're path-scoped so they load exactly when relevant — five of them (aspire.md, backend.md, frontend.md, gateway.md, messaging.md). Touch a service, the layered-architecture rules load; touch the bus, the outbox-and-idempotency rules load. But the layered architecture those rules describe — thin host plus a .Core library, facade → business → data → repository, ViewModel in and ServiceModel out — is ADR-0005's judgment. The rule is just its enforcement. A different architect's rules file would describe a different architecture, and the agent would obey it just as well. The rule is only as good as the layering it encodes.

Skills are procedures the agent should follow when a task matches — and the crown jewel, add-endpoint, encodes the single most error-prone operation in this architecture. Adding a REST endpoint here is a lot of correct steps in a specific order: pick the owning service, build the full controller → facade → business → data → repository slice, enqueue the integration event on the outbox atomically in the same transaction as the write, add the gateway route, wire the DI, write the tests. That ordering is architectural knowledge. The skill fires and the agent does it right every time — but only because someone who knew the right way wrote the skill. Hand the same skill mechanism to someone who thinks a fat controller is fine, and it'll reproduce a fat controller with perfect consistency.

Subagents are work to delegate, and I keep all three strictly read-only on purpose — code-reviewer, test-gap-analyzer, api-contract-checker. They report; they never touch code. And what they hunt for is not generic — it's the specific mistakes this architecture is prone to. The test-gap analyzer flags a consumer with no idempotency test, because at-least-once delivery means that callback will fire twice and a handler that isn't safe to run twice is a latent double-effect bug. Knowing that's the test that matters is the judgment. The subagent is just where I parked it.

Hooks are the non-negotiables — the things that happen whether the agent decides to or not — and there are two. secret-guard.sh runs before every edit and physically blocks anything credential-shaped from being written, no matter how I prompt or how tired I am. format.sh formats every touched file after the fact. Rules ask. Hooks enforce. Deciding which invariants deserve a hook instead of a rule — which things must be true versus merely encouraged — is, again, an architectural call.

Rules teach, skills guide, subagents check, hooks enforce. Four ways to encode a decision. But the encoding is the easy 20%. The decisions are the 80% you're hiring an architect for.

SCRUB: how I serialize judgment into something an agent can run

The connective tissue is a discipline I've used for months: SCRUB — Scope, Constraints, Restrictions, Usage, Behavior. It started as a way to write a good prompt. What it really is, I've come to understand, is a format for transferring architectural judgment into an executable form. Scope names what's in and out of bounds. Constraints pin the stack and conventions. Restrictions are the hard do-NOTs. Usage points at the right tool. Behavior demands a plan before any change.

The JobBoard's CLAUDE.md — the constitution loaded every session — is itself written as a SCRUB prompt, because the constitution is architecture. Its Scope declares that adding a service is an architectural decision, not a feature: "propose it, don't scaffold it." Its Restrictions are the load-bearing rulings — no shared database, the gateway is the only public door, publish through the outbox in the same transaction, consumers are idempotent. Every one of those is a decision with an ADR behind it. SCRUB is just the grammar that lets me hand those decisions to an agent without losing their meaning, and diagnose exactly which one failed when something goes wrong.

Alpha versus polished — which is the whole pitch

Here's the contrast I most want a client to hold onto, because it's the value stated plainly.

Give Claude Code to a competent developer with no distributed-systems architecture background, and you'll get an alpha: a working, demoable job board that looks finished. Post a job, apply, watch a status change — it all works on the happy path. And it's quietly wrong at every seam. A declined payment leaves stock reserved because there's no compensation path. A redelivered message is double-processed because the consumer isn't idempotent. Two services share a table because that was the plausible way to get the data. The design and the code drift apart the day after the build ships, because nothing was recording why anything was done. It'll get to about ninety percent and stall there forever, because the last ten percent is all the correctness-under-failure work that doesn't show up in a demo.

Give an architect the same tools, and the guardrails carry the experience the developer didn't have. The outbox and idempotent inbox make it correct under partial failure — no lost events, no double effects — by design, not by luck. The acyclic references keep the boundaries safe, enforced by the compiler. The ADRs and developer docs keep the design and the code locked together so nothing drifts. That's the difference between an alpha and something you'd push: not the toolkit, which is identical in both cases, but the judgment the toolkit is carrying. The architecture is what turns "it runs" into "it's ready."

And there's a document ecosystem that keeps it that way, each link feeding the next: the design (the high level design and its C4 diagrams) reasons out the decisions; the decisions become the .claude/ guardrails; the guardrails drive Claude Code to produce the src/; the output is recorded in twelve ADRs; and the ADRs are explained to the next human in developer docs — an add-an-endpoint-by-hand walkthrough, a seed-data guide, an ongoing architecture review with a ranked risk table. Design → guardrails → code → record → explanation, kept deliberately in sync, because a design that's drifted from its code is worse than no design at all. That closed loop is the deliverable. The job board is just the proof it works.

The contrarian bit: I barely write prompts anymore

Here's what surprises people. Once the architecture is encoded, I mostly stop writing prompts. The rules auto-load, the skills carry the procedures, the subagents catch the drift, the hooks enforce the invariants — so day-to-day work needs nothing more than "add a withdraw-application endpoint," because the repository already knows what "endpoint" means here and which service owns applications.

But read that carefully, because the reason I can stop prompting is not that the tooling is clever. It's that the architecture is doing the constraining. The leverage was never the automation. It was the judgment the automation carries.

Prompt engineering optimizes the prompt; architecture-powered Claude engineering makes the prompt unnecessary — by building years of hard-won structural decisions into the repo once, so they're paid forward on every task without me re-typing a single one.

What the architecture experience buys a business

I work with two kinds of decision-makers, often in the same room. So, plainly:

• Polished, not alpha. You get the ninety-to-a-hundred percent that the correctness-under-failure work represents — the part a toolkit alone never reaches.
• Correct when it counts. Compensation on every failure path, idempotent consumers, safe boundaries — stated as outcomes: partial-order incidents at zero, double-effects at zero, oversells at zero.
• No drift. The design, the code, and the reasoning stay locked together, enforced by the compiler and the hooks, so the system six months out still matches the diagram.
• Cheap onboarding. A new engineer reads the ADRs and developer docs and is productive in an afternoon, not a month.
• You own all of it. SCRUB is a discipline, not a dependency. No proprietary framework to keep paying for. Clone it, run it, keep it.

The honest part

Let me label the scaffold as a scaffold, because I always do. JobBoard is a reference build — a spike. It runs entirely on local Aspire-orchestrated containers at zero cloud spend, talking to the real Azure Service Bus SDK through an emulator; going live is a configuration and DevOps change, not a rewrite, but it is a change I haven't made here. Two of the twelve ADRs are still marked Proposed, not Accepted — the cross-service read-model strategy is named as a pending decision, not pre-built, because pretending to have solved it would be a dishonest move.

And the standing advice in the toolkit is verify before trusting: Aspire, the Service Bus emulator, YARP, and Claude Code all ship fast, so confirm the exact API names against the live docs before you lean on them. That caveat is in the repository, in writing, because a scaffold that lies about its completeness is worse than useless.

Where this starts

If your team has adopted an AI coding assistant and it keeps producing plausible-but-wrong output — code that demos and then drifts, an architecture that erodes a little with every generated file, a build that's perpetually ninety percent — the missing piece isn't a better tool. You already have the tool. The missing piece is the architectural judgment that tells the tool what "right" is, and encodes it so it holds.

That's the work I do: architect4hire.com/services/application-development. Read the repo first — github.com/architect4hire/jobboard — and the SCRUB framework in full at architect4hire.com/scrub. Then, if it's the shape of what your team needs, reach out at robert@architect4hire.com. Fractional or contract, the first conversation is the same: what you're trying to build, and whether I'm the right person to help you build it. If I'm not, I'll tell you.

Robert Felkins is the principal architect at Architect4Hire — Chicago-based fractional and contract architecture for small and mid-sized teams. I design right-sized .NET and Azure systems, modernize the ones you already have, and bring AI-assisted development into teams in a way that survives contact with production: senior judgment without the full-time price tag. Reach me at robert@architect4hire.com.

Or: why one architect could stand up five services and a compensating saga, and why that is emphatically not the headcount story you think it is

TL;DR — the code lives at https://github.com/architect4hire/orderflow2026. It's an event-driven order-fulfillment system I built for a fictional client, using the SCRUB framework end to end: an Angular front end, an order saga, four reacting services, and a compensation path on every failure. This post is less about the topology and more about what the build taught me — including the part where I keep reading that AI is going to thin out engineering teams, and the part where, having just delivered a system solo that used to need a squad, I'm here to tell you the developer never left the room.

Let me start with the fact that's going to be misread, so I can un-misread it before we go anywhere.

I built OrderFlow alone. One architect, no development squad. Five .NET services — an order saga plus four reactors — talking over a service bus, backed by four different data stores, fronted by an Angular operations view, with a compensating transaction on every failure path in the system. The kind of build that a few years ago would have been a team and a quarter. Solo, in about six weeks, most of it not spent typing.

And I know exactly how that reads on a nervous Monday: see, one senior person plus AI replaced the team. So let me be the guy who actually did it and tell you that's the wrong lesson.

Done right, AI-assisted development does not shrink your headcount. It relocates your work — out of the muddy middle where we used to hand-crank implementation, and to the two ends where it was always hardest.

One architect could stand up this system not because the AI did the engineering, but because the engineering moved: forward into specifying it precisely, and backward into verifying it ruthlessly. Both are still done by a human, and both are more senior, not less. The keystrokes went away. The judgment did not. That's the whole post — everything below is me showing my work.

Why OrderFlow Exists, and What It Actually Is

OrderFlow is a proof-of-concept I stand up in the open, for a client who doesn't exist, so that "I do disciplined AI-assisted development" stops being a slide and starts being a repository you can read. This one models order fulfillment — the genuinely hard kind, where money moves and inventory is finite.

Here's the landscape. An Angular 20 app with two views — customer order-status and an operations view — calls an Order service over REST. From there it's events all the way down: the Order service runs a saga that sends commands over a service bus, and four services react and reply — Inventory (reserve stock), Payment (charge), Fulfillment (dispatch), and Notification (tell the customer). Cosmos holds the event log, Redis the status read model the UI reads from, SQL the inventory and payment records, Blob the rest — all on Aspire-local emulators against the real Azure SDKs, so it boots on a laptop with no cloud spend.

Now — regular readers know my default template is the opposite of this. On PlannerPro I argued hard for one API, one database, and no microservices, because right-sizing usually means resisting the urge to distribute. So why the saga-and-reactors sprawl here?

Because right-sizing is bidirectional, and this domain sizes up. An order that reserves stock, charges a card, and dispatches a package is a multi-step transaction across systems that fail independently, on a bus that delivers at least once. You cannot right-size that into a clever monolith and a database transaction — the distribution isn't resume-driven; it's the shape the problem actually is. This deliberately departs from my house template, and inside the build, I worked to a fictional client's code conventions rather than my own standards (which matters later). The template is a starting bias, not a religion. When the domain moves money, you follow the domain.

SCRUB, For Those Just Joining Us

If you've read the PlannerPro post, skip ahead. Everyone else, ninety seconds.

Generative AI has a failure mode I call the Plausibility Trap: the output isn't wrong in a way that crashes. It's wrong in a way that compiles, passes its tests, looks idiomatic, and quietly encodes a domain decision you never made. The model knows how to write code. It does not know your business. Leave a gap in the prompt and it fills that gap with the most statistically average answer in its training data — which is, occasionally and silently, the wrong answer for you.

SCRUB is the checklist that closes those gaps before you submit. Five elements, every prompt: Scope (exactly what you're asking for), Constraints (your framework, your architecture, your naming), Restrictions (what you explicitly do not want, and why — the element everyone skips and the one that matters most), Usage (who calls this, in what world), and Behavior (what must not change). Hold onto Restrictions. On a system like this, it's the whole ballgame.

Four Places Where the Plausible Answer Is the Wrong Answer

Here's the part of OrderFlow that made me want to write this at all. A CRUD app has maybe one spot where the training-data idiom and your business reality point in opposite directions. An order saga has four — and every one is the difference between a demo that dazzles and a production system that double-charges a customer. Each is a Restriction I had to write by hand, because I understood the domain and the model could not.

Compensation on every failure path. (CRITICAL.) Ask an AI to build a saga step and it will hand you a beautiful happy path — reserve, charge, dispatch, done. What it won't do, unprompted, is compensate when step three fails after step two succeeded: a multi-step order failing mid-flow, leaving stock reserved or a charge captured with nothing shipped. It compiles, it demos, it passes the happy-path test — and it quietly rots your inventory and your customer trust in production. The restriction is CRITICAL: every failure path compensates — a declined payment releases the reservation, a failed dispatch refunds and releases, no order is ever left partial. That sentence isn't documentation. It's the product.

Idempotent payment callbacks. (CRITICAL.) At-least-once delivery means the payment callback will arrive twice — not sometimes, will. The naive handler the model writes processes both. Two identical callbacks, two charges. The restriction: a duplicate payment callback is a no-op; poison messages dead-letter, they do not loop. Idempotency is not a nice-to-have here; it's the line between "correct" and "class-action."

Concurrency-correct inventory reservation. (IMPORTANT.) Two buyers race for the last unit. The obvious reservation code — read stock, check it's positive, decrement — lets both of them win, because the check and the decrement aren't atomic. The AI writes the obvious code; the obvious code oversells. The restriction pins the reservation to be concurrency-correct so oversell incidents go to zero, and the status read model converges in a second so the UI doesn't lie about it.

A durable baseline. (IMPORTANT.) Without ADRs, C4 diagrams, and a committed design doc, the architecture and the code drift apart the moment the build ships — and six months on, nobody remembers why the payment handler is idempotent, so someone "simplifies" it. The restriction is the baseline itself: design doc, ADRs, and the SCRUB prompt library, committed as a durable source of truth.

Sit with those four, because they're the argument. Not one is a thing the AI can decide for you; every one is a senior human saying the plausible version is wrong here, and here's why. Take the human out of that loop and the AI ships four confident, compiling, demo-ready bugs that each cost real money. The developer isn't the bottleneck in that picture. The developer is the immune system.

Lessons From the Trenches

With the stakes clear, here's what the build actually cost me — in the order I hit it.

The first decision isn't code — it's documentation, and it's a coin-flip you can't afford to fumble. You can architect and scaffold the whole thing from a committed SCRUB prompt library — prompts as plan, spec, and record — or you can build it by hand and document it after. For the initial stand-up, those two roads cost about the same hours. All roads lead to Rome. But they diverge violently by next Tuesday: hand-built-then-documented gives you a system and a description that start drifting apart the moment you close the editor, while the prompt-library road gives you an executable, versioned, reviewable spec you can reuse on the next engagement. My recommendation, and what I did: build a default scaffolding prompt in your library and start from it every time. The juice is genuinely worth the squeeze — but I won't pretend the first pressing is free. It pays out on the second glass.

My toolchain moved. Since the GitHub Copilot price change I've shifted essentially all of this to Claude Code, with a cleaner division of labor: agentic Edit mode almost exclusively for the initial stand-up and the plan — the big multi-file scaffolding push where an autonomous build-and-verify loop earns its keep — and Chat mode for surgical edits, the one-file "change this and nothing else" work where I approve every character. The mode mapping evolved since PlannerPro, and that's the point: SCRUB is what survives a pricing email that reshuffles your whole toolchain overnight. The discipline is portable even when the buttons aren't.

Your prompt library is code, and code has bugs. I write a big slice of the library up front, and on OrderFlow that bit me exactly the way real code bites you. I'd named a project OrderFlow.Order.Api, and the singular Order collided with types and folders all over a system that already has an Orders saga. The fix wasn't in the generated C#. It was in the prompt: rename to OrderFlow.Orders.Api, regenerate, move on. Read that again — I debugged the specification, not the output. That is shift-left in one rename. The bug existed before the code did, which is exactly where I caught and killed it.

The AI finds the holes you left — and when it's right, you feed the fix back into the prompts. Throughout the build, Claude surfaced gaps in my prompts: places I'd underspecified, ambiguities it flagged, fixes it proposed. Often it was right. But here's the discipline that separates a tool that helps from a tool that quietly rewrites your intentions — when its fix is correct, you don't just accept the diff. You update the prompt so the library reflects the system you actually built. Otherwise, the record lies to you in six months. Keeping the spec true is your job. Still yours.

And sometimes you fix it by hand, on purpose. My favorite small one: Claude cheerfully generated multiple classes in a single file. Compiles, runs, ships — and violates a standard I've held for years, one class per file, and it grated every time I opened the offender. Fork in the road: fix the prompt so the generator never does it again, or fix the files by hand this once? For a PoC that isn't going to grow, I did it manually — pure annoyance-driven housekeeping, zero functional change, in service of a standard I refuse to drop. On a production system with a long road ahead, I'd have fixed the generator, because there you pay once and it never reoffends. Both calls are legitimate. The point is that a human weighed lifespan against effort, because no prompt on earth was going to make that judgment for me.

Every one of those is a developer doing developer work — naming things right, catching a collision, keeping the spec honest, enforcing a standard the machine doesn't care about. The tool didn't delete any of it. It relocated it to the two ends that were always the hard part: saying precisely what you want, and verifying precisely what you got.

So, About That Squad

Back to the fact I opened with. Yes, I delivered this solo. No, that is not the layoff memo.

Here's the honest reconciliation. One architect could ship this saga because the irreducibly human parts — the four restrictions above, the event contracts, the compensation semantics, the failure-injection validation — got done by a senior person, and the mechanical middle got generated against them. That doesn't scale to "fire the team." It scales the opposite way: the harder and more valuable the system, the more of that senior specification-and-verification work there is, because the Plausibility Trap grows with complexity, not with team size. A CRUD form has one silent-wrong bug waiting for you. A saga that moves money has four before breakfast, and everyone needs someone who understands the domain to pin it down.

The work shifted left, not away. On a reference build, you see that as "one architect, no squad." On a real product with a real backlog, you see it as a team whose job stopped being typing the implementation and became authoring the restrictions and verifying an ever-growing surface — a job that gets bigger as the system does. If your headcount drops after adopting this, you didn't automate the work. You stopped doing part of it, and OrderFlow's four restrictions are the part you stopped doing. Production will send you the invoice.

The Practical Version

Running OrderFlow's playbook on your own next build:

Decide the documentation fork first, out loud. SCRUB-library-first or hand-built-then-documented — pick before you generate. Equal effort on day one, wildly unequal on day thirty. Build a default scaffolding prompt and start every project from it.

Right-size in both directions. Sometimes right-sizing means one API; sometimes it means a saga and four reactors. Let the domain decide, and bake "match this architecture" into Constraints on every prompt.

Find all your no-project-filters. Every domain has the spots where the training-data idiom and your reality diverge. A saga has several: compensation on every failure path, idempotent callbacks, concurrency-correct reservation. Find them before the AI does and pin each with a tiered Restriction that carries its reason.

Treat the prompt library as code. It has bugs. You'll rename an Order to Orders to kill a collision. Fix the spec, not just the output — that is the debugging now.

Feed the AI's good catches back into the prompts. When the model finds a real hole, update the library so it matches what you shipped. A drifted record is worse than none.

Know when to fix by hand vs. fix the generator. Manual for one-off annoyances on a short-lived PoC. Generator for anything that will live and reoffend. A human weighs lifespan against effort — that call isn't going anywhere.

The repo is at github.com/architect4hire/orderflow2026. Read the prompt library alongside the code and watch how much of the real work happened before any C# existed and after it was generated — at the two ends, where the developer still very much lives.

Because that's the headline I'd tattoo on the industry if I could: nobody got removed from this build. The work moved to where it was always hardest. If you were good at the hard part, you're about to be worth more, not less.

Robert Felkins is the principal architect at Architect4Hire and the author of the upcoming book Practical AI Assisted Development, which covers the full SCRUB system — tiered restrictions, layered generation, Edit Mode patterns, prompt chaining, and applying the framework to codebases you inherited rather than started.

And if your team is somewhere in the middle of this transition — a distributed system that has to be correct under failure, tooling that changed under you, or a nagging worry about what all this means for the people on your team — that's the work I do. I take fractional architecture engagements: a few days a month of senior architectural judgment without the full-time price tag, covering .NET and Azure platform work, modernization, team enablement, and AI-assisted development that survives contact with production. Reach out at robert@architect4hire.com.

How a fractional or contract engagement with Architect4Hire runs — from first conversation to handoff — told through OrderFlow2026, a reference build I put in the open so you can see the work before you buy it.

Most consulting relationships ask you to buy on faith. You read the case studies, you like the person on the call, you sign — and then you find out what "senior architecture help" actually means in practice. Sometimes it means a diagram and an invoice. Sometimes it means a stack of slides that never turns into code. You don't know until the money's spent.

I'd rather you know first. So I put an entire engagement in a public repository — the design docs, the architecture decisions, the delivery plan, the prompt library, and a working system built from all of it — and left it open at github.com/architect4hire/orderflow2026. This post walks you through what hiring me actually looks like, using that repo as the worked example. Not a testimonial. The artifacts themselves.

First, the honest part

OrderFlow2026 is a reference build for a client who doesn't exist. It's an event-driven order-fulfillment system I designed and built solo, and as I write this it's a proof-of-concept sitting at roughly ninety percent — fully scaffolded, mostly wired, deliberately still carrying a few TODOs where real client requirements would land.

I build these on purpose. "I do disciplined, right-sized architecture" is a claim, and claims are cheap. Anyone can put clean architecture and AI-assisted development on a slide. A public repo you can clone, read, and run is a different kind of evidence — it shows the process, the decisions, the trade-offs, and yes, the seams. I'd rather show you a real build with an honest ninety-percent label than a polished lie at a hundred. If you're evaluating me, that repo is the most useful thing on this entire site, and it will tell you more in ten minutes of reading than any sales page.

Two shapes of engagement, one way of working

People hire me two ways, and the difference is mostly about time, not method.

A fractional engagement is ongoing: a few days a month of senior architectural judgment, embedded with your team, without the cost or commitment of a full-time principal architect. It's the right fit when you have developers who can build but need someone to set direction, make the hard calls, raise the team's floor, and stay accountable as the system evolves.

A contract engagement is bounded: a defined deliverable, a fixed scope, a clear start and end. It's the right fit when you know what you need built or modernized and you want a senior hand to design it, build it, and hand it over cleanly.

Underneath, the work runs the same way regardless of which one you pick — and OrderFlow2026 is the shape of it. Let me walk you through the phases, because the phases are what you're actually buying.

How the work runs

Every engagement moves through five phases, and I scope and sequence them out loud before any meaningful money is spent. Here's how they played out on OrderFlow.

Discovery. Before I write anything, I write down what I understand — the domain, the constraints, the failure modes that matter. On OrderFlow that produced a high-level design document, C4 diagrams, sequence diagrams, and a set of Architecture Decision Records. The business value here is unglamorous and enormous: misunderstandings get caught while they still cost a sentence to fix, not a sprint. The technical value is that the design exists as a durable artifact from day one, not as folklore in someone's head.

Vision. We settle scope, service boundaries, and contracts — the shape of the thing and the promises its parts make to each other. This is where an experienced architect earns their fee by saying no to the features that don't belong in version one, and yes, but not like that to the ones that do. Right-sizing is the whole game, and it cuts both ways: sometimes it means resisting a distributed system, and sometimes — as with OrderFlow — it means the domain genuinely demands one.

Breakdown. I turn the plan into a committed library of small, disciplined prompts using my SCRUB framework, and I prove the risky path first. On OrderFlow that meant standing up the order saga's happy path before anything else, because if the orchestration spine doesn't hold, nothing built on top of it matters. You're paying for the risky decisions to be made early, cheaply, and visibly.

Development. The build. All the services, the resilience, the frontend. This is the longest phase and, counterintuitively, the one where a disciplined approach spends the least of your attention — because the hard thinking already happened in the phases before it.

Delivery. Failure-injection validation, observability, and a clean handoff: the ADRs, the diagrams, and the prompt library all updated to match what actually got built. You don't get a system and a stale description of a different system. You get a system and its true source of truth.

Discovery through Delivery, OrderFlow ran roughly six weeks, architect-only, no development squad to assemble. That's the shape of a bounded engagement. A fractional one spreads the same discipline across a longer, lighter cadence.

What you actually receive

Here's the part most consulting engagements are quietly bad at: what's left in your hands when I'm gone.

Open the docs/ folder in the repo and you'll find the durable baseline — the high-level design, the C4 and sequence diagrams, the ADRs that record why each significant decision was made, and the SCRUB prompt library that generated the code. The src/ folder holds the scaffolded, near-complete system those documents describe. The two are kept in sync deliberately, because a design that has drifted from its code is worse than no design at all.

Why does that matter to a business, not just an engineer? Three reasons, and they're all about the day after I leave.

Onboarding gets cheap. A new developer reads the ADRs and understands the system in an afternoon instead of reverse-engineering it over a month. That's real money, every time you add a person.

The design stops drifting. Six months from now, when someone asks why the payment handler is built the way it is, the answer is in a committed file with a history — not lost, not guessed at, not "simplified" by someone who didn't know what it protected against.

You own all of it. No proprietary framework you have to keep paying for, no black box, no lock-in. The architecture, the code, and the reasoning are yours. If you never call me again, you lose nothing.

The technical spine — because leadership comes in two flavors

I work with two kinds of decision-makers, often in the same room: the business leader who needs to trust that the money buys an outcome, and the technical leader who needs to trust that the engineering is actually sound. I'd be doing you a disservice if I only spoke to one of you, so here's the engineering, stated plainly.

OrderFlow is an event-driven system: an Angular front end calls an Order service, which runs a saga that orchestrates four reacting services — Inventory, Payment, Fulfillment, and Notification — over a message bus, with a Cosmos event log, a Redis read model behind the status screens, and SQL for the records that have to be exact. It runs on local emulators against the real cloud SDKs, so it boots on a laptop with no cloud spend to evaluate it.

That architecture wasn't chosen to look impressive. It was chosen because order fulfillment is a place where money moves and inventory is finite, and that forces four correctness problems that a naive build gets quietly wrong:

• A multi-step order can fail halfway. Without a compensation path on every failure, a declined payment leaves stock reserved and a failed dispatch leaves a customer charged for nothing. OrderFlow compensates on every path — the measurable target is partial-order incidents at zero.
• Payment callbacks arrive more than once. Message delivery is at-least-once by nature, so the callback will fire twice. Without idempotency, that's a double charge. OrderFlow treats duplicates as no-ops — double-charge incidents at zero.
• Two buyers can race for the last unit. Naive inventory code lets both win. OrderFlow's reservation is concurrency-correct — oversell incidents at zero.
• Designs drift from code. Without a documented, versioned baseline, the two diverge the day the build ships. OrderFlow's ADRs and prompt library are that baseline.

Notice that every one of those technical properties is stated as a business outcome with a number attached. That's not an accident. Correctness under failure isn't an engineering vanity project — it's the difference between a system that quietly erodes customer trust and one that doesn't. Good architecture is a business decision that happens to be written in code.

What it costs you — in involvement, not just dollars

I keep specific figures out of blog posts because they change from engagement to engagement, and I'd rather quote you honestly than have you hold me to a stale number. But cost isn't only money, so here's the part people forget to ask about: your team's time.

A bounded engagement like OrderFlow needs a subject-matter expert available a few hours a week — someone who can answer domain questions and make business calls when a trade-off surfaces. That's it. There's no squad to stand up, no coordination tax, no six-people-in-a-standup overhead. One senior person, working with discipline and modern tooling, carries the build. That's deliberately how I've structured the practice: senior judgment without the full-time price tag, and without asking you to reorganize your world to receive it.

Why this de-risks the decision

Strip away the specifics and here's the actual pitch, stated as plainly as I can: you can inspect the work before you commit to it. The process is written down. The deliverables are real and public. The pricing is transparent and scoped before the spend. The outcomes are measurable, not vibes. And when it's over, you own everything and owe nothing.

That's not a claim I'm asking you to take on faith. It's a repository. Go read it.

How to start

If your team is staring at a problem that needs senior architectural judgment — a distributed system that has to be correct under failure, a modernization that keeps stalling, a codebase that's drifting away from its own design, or an AI-assisted practice that isn't paying for itself yet — that's the work I do.

Fractional or contract, the first step is the same: a conversation about what you're trying to build and whether I'm the right person to help you build it. If I'm not, I'll tell you. If I am, you'll already know exactly what to expect, because you'll have read it.

Reach out at robert@architect4hire.com, or start with the repo at github.com/architect4hire/orderflow2026.

Robert Felkins is the principal architect at Architect4Hire — Chicago-based fractional and contract architecture for small and mid-sized teams. I design right-sized .NET and Azure systems, modernize the ones you already have, and bring AI-assisted development into teams in a way that survives contact with production: senior judgment without the full-time price tag.

TL;DR — the code and the full prompt library live at https://github.com/architect4hire/plannerpro. The prompts document at docs/PLANNERPRO-SCRUB-PROMPTS.md is the interesting part. This post explains why.

I want to tell you about a standup.

Not a dramatic one. Nobody quit. Nothing shipped broken. The kind of standup that's much worse than that — the kind that happens every single morning, takes forty minutes, and produces less clarity than it started with.

The client had one delivery team. Six people. Good people. And at the moment I was engaged, that one team was carrying three completely independent projects: a customer portal rebuild, an internal reporting migration, and a compliance-driven integration with a hard regulatory date. Three backlogs. Three stakeholders. Three sets of priorities that had never once been in the same room together.

Their tooling handled this the way every project management tool handles it: three boards. Because every tool on the market is built on the same silent assumption — that a project has a team. One project, one board, one team, tidy.

This team had the assumption backward. The team had projects. Three of them. And the tooling had no answer for that at all.

So, the standup was tab-switching. Board one, board two, board three, back to board one because someone remembered something. The team lead kept a spreadsheet on the side — the real board, the one that actually reflected reality — updated by hand, wrong by Wednesday. And the thing that finally made someone call me: their most senior developer had nine items in progress. Nine. Three per board, and each board looked completely reasonable in isolation. No single view existed anywhere in the company that could show that number. The person carrying the most risk in the org was invisible to every tool they'd paid for, precisely because the risk was spread across three views that never met.

That's not a process problem. That's a data model problem. The unit of capacity is the person, and every board they owned modeled capacity as a property of the project.

I couldn't find a tool that modeled it correctly without an enterprise license and a consulting engagement to configure it. So I built one. It's called PlannerPro; it's small, and it does exactly one thing the big tools don't: one board, columns for workflow state, swimlanes for projects, and work-in-progress limits that belong to the human being — enforced across every project simultaneously.

This post is about what I built, but mostly it's about how I built it — because PlannerPro went from an empty repository to a working, tested product using a committed library of small, layered prompts, and the discipline behind those prompts is the part you can steal.

SCRUB, For Those Just Joining Us

If you read the BrandForge post, you know the framework. Short version for everyone else.

AI coding tools have a failure mode I call the Plausibility Trap: the output isn't wrong in ways that crash. It's wrong in ways that compile, pass tests, look idiomatic, and quietly encode a domain decision you never made. The model knows how to write code. It does not know your domain. When your prompt leaves a gap, the model fills it with the most statistically common answer from training data — and the most common answer is, occasionally and silently, the wrong one for you.

SCRUB is the checklist that closes those gaps before you hit submit. Five elements, every prompt:

• S — Scope. Exactly what you're asking for. Named methods, named files, nothing else.
• C — Constraints. The technical rules: framework versions, architecture, naming, DI style. Your codebase, not the training-data average.
• R — Restrictions. What you explicitly do not want, tiered CRITICAL / IMPORTANT / PREFERRED, each with a reason. The element everyone skips. The element that matters most.
• U — Usage. Who runs this, under what conditions, with what compliance and audit expectations?
• B — Behavior. What must not change. The star of every edit to existing code.

I've written a book about the full system — Pratical AI Assisted Development, out soon — which goes deep on tiered restrictions, Edit Mode patterns, prompt chaining, and applying the framework to codebases you inherited rather than started. But you don't need the book to follow this post. You need one idea from it:

A prompt is a specification, and specifications deserve engineering.

PlannerPro is what that looks like on a greenfield project, end to end.

The Architecture and the Choices Behind It

Let me get the stack out of the way, then talk about the decisions that actually mattered.

.NET 10 Web API on the backend. EF Core 10 over SQL Server. Angular 20 on the front, standalone components, signals for all component state, Angular CDK for the drag-and-drop. If you've read anything else I've written, none of that surprises you.

Here's the first real decision: PlannerPro is one API project. Not microservices. Not Aspire orchestration. One Web API, one database, one Angular app.

I want to dwell on that for a second, because I've spent a lot of this blog building distributed systems, and I can feel some of you raising an eyebrow. The honest architect's answer is that this system has one team of users, one write path that matters, and no independent scaling or deployment pressures anywhere. Microservices are a solution to organizational and operational problems that PlannerPro lacks. Reaching for them anyway would have been resume-driven development, and the whole point of this engagement was to solve a problem in days, not to build a platform. Right-sizing is an architectural decision too — arguably the one clients pay me for most often.

Inside that one project, the layering is strict, and it's the same layering I use everywhere: Controller → Facade → BusinessManager → DataManager → DbContext. Hand-rolled mapping extensions, no AutoMapper. Audit fields on every entity. Soft delete everywhere, enforced with a global query filter. Boring, deliberate, and — this matters for what comes later — predictable enough that an AI can be pointed at it and told to conform.

The domain model consists of three entities. Project. WorkItem. TeamMember. That's it — no sprints, no epics, no comments, no attachments. Version one is the board and the rule, and everything I declined to build was declined on purpose.

And the rule is the whole product, so let me state it precisely:

A team member's WIP limit applies to their in-progress items across all projects. The query that backs it — CountInProgressForMemberAsync — deliberately has no project filter. There is a CRITICAL restriction in the prompt library that says exactly that: do not filter this count by project. Because an AI generating a data manager for a project-management tool will absolutely scope that query to the project. Every example in its training data does. Project-scoped WIP is the idiom. Project-scoped WIP is also, for this client, the entire disease. Their senior dev with nine in-progress items looked fine on every board precisely because every count was project-scoped. One .Where() clause, present or absent, is the difference between a tool that solves the problem and a tool that re-implements it with nicer CSS.

That's the Plausibility Trap in one sentence, by the way. The wrong version compiles, passes tests, and demos beautifully.

Two more decisions worth naming:

The board is one payload. A single GET /api/board returns everything the screen needs, grouped and ready to render — the Angular component does no client-side regrouping. When five people are staring at a shared screen in standup, you want one fetch, one render, one source of truth.

The server is the transition authority. Status moves one step forward or one step back — Backlog → Ready → In Progress → Review → Done, no skipping — and the server enforces it. The UI does a drag with an optimistic update and rolls back on rejection. I explicitly restricted the AI from greying out "illegal" columns during a drag, which is the clever UX idea it will offer you unprompted. A greyed-out column that's wrong is a lie with good intentions; an honest 422 toast after the fact is merely an inconvenience. When client-side cleverness and server truth disagree, cleverness loses.

One drag-and-drop restriction earns special mention because it's my favorite in the whole library: drop lists are connected only within a swimlane. A card can move between columns, never between projects. Why CRITICAL and not IMPORTANT? Because cross-lane drag isn't a UX bug — it's a data corruption bug wearing a UX feature's clothes. Dragging a card one lane down would silently reassign a work item to a different project, and the CDK will happily wire all the lists together if you don't tell it otherwise, because connecting everything is what the examples do. One restriction. One line. An entire category of "why is this ticket in the wrong project" support conversations, never born.

The Prompts: Small on Purpose

Now the part I actually want to talk about.

There's a file in the repository — docs/PLANNERPRO-SCRUB-PROMPTS.md — that existed before any of the code did. Nineteen prompts. It's the plan I worked from, committed alongside the code it produced, and it follows the same chain the BrandForge component did: Context → Plan → Execute → Verify.

But the style of these prompts is different from what most people write, and the difference is the point. These are micro prompts. Every one of them fits in a paragraph. The Scope blocks name exact methods with exact signatures. The Restrictions come tiered, and every CRITICAL restriction carries its reason — not as documentation garnish, but because the reason is the thing that tells the model why the plausible default is wrong here.

Compare the mega-prompt approach — "build me a kanban tool with three projects and WIP limits, here are 40 requirements" — with what's in that file. The mega-prompt produces two thousand lines in one shot. Somewhere in those two thousand lines, the WIP query has a project filter, the drop lists are cross-connected, and the rollback path doesn't exist. You will find those problems the way my client found their nine-item developer: later, in production, by accident.

The micro prompts produce the same two thousand lines. But no single generation is ever more than a couple hundred of them, and every one of those chunks got a review that took two minutes instead of an afternoon.

Each prompt also names its mode, because the tools are not interchangeable. Claude Code Chat for the layers where thinking matters — contracts, business rules, orchestration methods. Copilot Agent Mode for the layers that are mechanical and multi-file — data access, template slices, CSS — where the autonomous build-and-verify loop earns its keep. Claude Code Edit Mode for surgery after the fact, with Behavior as the dominant element. Reasoning in Chat, generation in Agent, precision in Edit. Matching the mode to the cognitive work is half the discipline.

Why Layers Are the Whole Ballgame

If you take one thing from this post, take this section.

The single most important structural decision in the prompt library is not any individual restriction. It's that the work is layered, and each layer constrains the next.

The backend is four layers. Layer 1 is contracts — interfaces, enums, entity skeletons, view model shapes. Signatures only, no bodies, maybe eighty lines. Layer 2 is data and mapping: the DbContext, the data managers, the mapping extensions. Layer 3 is the business rules — the transition table and the WIP check, the two things that are the product. Layer 4 is the HTTP surface and the wiring.

The frontend gets the same treatment, six layers deep: a signal skeleton with every method stubbed, then the template built in vertical slices — header and WIP strip, then the swimlane grid, then cards and drag wiring — then the method bodies, then the styles. The methods layer comes second to last deliberately: by the time the AI writes onDrop(), the entire template it orchestrates already exists in front of it. It is not writing orchestration in a vacuum and hoping the UI that arrives later happens to match.

Why does layering matter this much? Four reasons, in ascending order of importance.

First, review becomes possible. Eighty lines of contracts is a code review. Two thousand lines of everything is a skim, and a skim is where the Plausibility Trap lives. Every ordering bug I've ever shipped from AI-generated code went out through a review I did too fast because the diff was too big.

Second, errors stay local. When Layer 2 has a mapping bug — and mapping bugs are the most common failure at that layer — I fix it with a Layer 2 prompt. Layers 3 and 4 never hear about it, because they reference Layer 2 through its contracts, not its internals. In a one-shot generation, a bug in one region sprays corrections across the entire output, and every correction risks disturbing something that was right.

Third, naming compounds. Layer 1 is where I spend disproportionate review time, because the names chosen there propagate through every subsequent layer. A bad facade method name in Layer 1 costs five minutes to fix in Layer 1 and a five-file threading exercise to fix after Layer 4. Cheap-to-fix and expensive-to-fix are the same mistake at different timestamps; layering lets you always pay the cheap price.

Fourth — and this is the one that changes how you work — each layer constrains the next. By Layer 3, the AI isn't inventing; it's conforming. The contracts exist. The data managers exist. The custom-instruction files pre-load the conventions. The generation space has been narrowed, layer by layer, until the plausible output and the correct output are nearly the same thing. That's the real trick. You're not hoping the model guesses right. You're removing its opportunities to guess.

The ordering echoes onion architecture, but the mapping isn't the point — the size is the point, and the direction of dependency is the point. Small enough to read in two minutes. Ordered so the shape exists before the behavior, and the behavior exists before the wiring.

And it doesn't feel slow. I expected it to, the first time. Nineteen prompts for a two-project solution? It felt fast, because I was never working on more than one small piece at a time, and I never once entered the mode every developer knows too well — spelunking through a giant generated diff looking for the line that's lying to you.

Fix the plan, not the code. If the plan is wrong, you fix a paragraph.

Verify: The Step That Catches You Moving Too Fast

The chain ends with Verify prompts — one for the backend, one for the board component. Each is a list of specific PASS/FAIL checks the AI runs against the finished code, with file-and-line evidence required for any FAIL.

Not "does it work." Specific. Does the WIP count query have a project filter? Is any drop list connected across lanes? Does the rollback restore captured state, or does it cheat and re-fetch the whole board? Does any orchestration method contain a .subscribe()? Every check on those lists maps back to a CRITICAL or IMPORTANT restriction from an earlier layer, because the Verify step exists for exactly one scenario: the day you review a diff too fast. On BrandForge, Verify caught a real bug my eyes had slid over. It stays in the chain forever.

The Verify lists are also the maintenance contract. When PlannerPro grows — per-column WIP limits are the obvious next ask, and a "blocked" flag is right behind it — the prompts document gets updated first: new restrictions, new Verify checks, then run the prompts. The document is versioned, committed, and reviewed like the code, because it is the reason the code looks the way it does. Six months from now, when somebody wonders why CountInProgressForMemberAsync pointedly ignores the project, the answer isn't in a comment or in my head. It's in a file with a commit history.

What the Client Got

A board their standup actually runs on. Swimlanes for the three projects, five columns, drag-and-drop with honest server-side rules, and a WIP strip across the top showing every team member's cross-project count — the number that used to live in a hand-maintained spreadsheet, wrong by Wednesday.

The nine-item developer, by the way, is now capped at three, and the standup conversation changed shape the first week: it stopped being three sequential status reports and started being an argument about which project gets the constrained person next. That's the right argument. The tool didn't resolve their priority conflict — no tool does — but it made the conflict visible on one screen, which is the precondition for resolving it like adults.

And I got a repository where every line of code traces back to a small, reviewed, committed prompt — which means the next feature starts from a plan, not from archaeology.

The Practical Version

If you want to run this pattern on your next greenfield build:

Write the domain brief first, and make the AI recite it back. One paragraph. The Context step costs five minutes and catches misunderstandings while they cost one sentence to fix.

Argue with the Plan. In prose, before any code exists. The plan step is the cheapest insurance in the entire chain, and it is the step everyone skips.

Layer the Execute. Contracts, then data, then rules, then wiring. Four to seven layers, each describable in three sentences. If a layer needs a fourth sentence, split it.

Write Restrictions before Scope, and tier them. Ask what could compile and still be wrong. CRITICAL means silent wrong data. IMPORTANT means debt. PREFERRED means style. Attach reasons to the CRITICAL ones — the reason is what teaches the model why the idiom is wrong here.

Find your no-project-filter. Every domain has one: the single line where the training-data idiom and your business reality point in opposite directions. Yours might be a tip default, a rounding rule, a scoped query. Find it before the AI does, and pin it with a CRITICAL restriction.

Match the mode to the work. Chat for contracts, rules, and orchestration. Agent for mechanical multi-file layers. Edit for surgery, with Behavior dominant.

End with Verify, and keep it. Specific PASS/FAIL checks derived from your restrictions. Re-run it after every change.

Commit the prompts. They're not notes. They're the architecture record.

The repo is at github.com/architect4hire/plannerpro, prompts and all. Read the prompts document before the code — it's the better half of the repository, and it will make the code make sense in a way the code alone can't.

Robert Felkins is the principal architect at Architect4Hire and the author of the upcoming book Pratical AI Assisted Development, which covers the full SCRUB system — tiered restrictions, layered generation, Edit Mode patterns, prompt chaining, and applying the framework to codebases you inherited rather than started.

And if your team is staring at a version of the three-boards problem — tooling, architecture, or an AI-assisted delivery practice that isn't paying for itself yet — that's the work I do. I take on fractional architecture engagements: a few days a month of senior architectural judgment without the full-time price tag, covering .NET and Azure platform work, team enablement, and AI-assisted development practices that survive contact with production. Reach out at robert@architect4hire.com.

TL;DR - wanna check out the framework and sample application I built, grab the code at https://github.com/architect4hire/brandforge

I want to tell you about a bug.

Not a dramatic one. No crash. No stack trace. The kind that's much worse than that — the kind where everything runs fine, the feature ships, and then three weeks later an operator notices that every single content card saved from the pipeline has the wrong social media copy attached to it.

Instagram post? Generic. TikTok hook? Missing. Pinterest description? Something the AI generated from the image name, not the actual image. The copy was there. It just belonged to the wrong image, from an earlier step in the workflow, because somewhere in twelve hundred lines of Angular TypeScript the social copy generation fired at the wrong moment. Before the operator had chosen an image. Before the system knew which image to describe.

The code compiled. The tests passed. The component ran. Nobody knew anything was wrong until they looked at the actual board cards.

That bug — or rather, the specific thing I did to prevent it from ever happening — is what this post is about.

The AI Does Something I Find Genuinely Terrifying

Before I get to the solution, I want to name the problem properly, because most people I talk to about AI-assisted development haven't quite articulated it yet.

The problem isn't that the AI gets things wrong. I can handle wrong. Wrong throws an error, wrong breaks a test, wrong crashes the app and puts a red line in the console. I can work with wrong.

The problem is that the AI gets things plausibly right.

Here's what that looks like in practice. You paste a task into Claude — build me a service, write me a component, add a method to this class — and the AI produces twelve hundred lines that are idiomatic. They reference the right services. The types are correct. The naming follows your conventions. The structure looks exactly like what a senior developer would write. You do a quick scan, it looks good, you run it, it runs. You ship it.

And then three weeks later you find out that the component has been defaulting the tip amount to twenty percent, or storing a PII field it shouldn't be touching, or firing an expensive AI call before the user has confirmed their intent, or — in my case — generating social copy off the wrong image because the ordering of two async calls was off by half a step.

I started calling this the Plausibility Trap. The output is so plausible-looking that you don't examine it carefully enough to catch the domain error buried inside it. A junior developer would produce code that's obviously wrong and you'd catch it in review. The AI produces code that's only subtly wrong and you catch it in production.

The reason this happens is structural, not a flaw in the AI. The model is trained to produce idiomatic, coherent, conventionally-structured code. It has no idea what your product is actually supposed to do. It does not know that social copy should fire after image selection and not before. It doesn't know that your food blogger operators are going to be looking at that copy on a desktop CMS and not a mobile device, and that the UX decisions that flow from that distinction are significant. It doesn't know that your company had a legal incident around default values in financial fields. It knows how to write code. It does not know your domain.

So when you hand it a vague prompt and it fills in the gaps, it fills them with plausible defaults from training data. And the defaults are usually fine. Until they're not.

I Got Mad About This and Built a Framework

I have been building software with AI tooling since before it was cool and after it got annoying. For a long time my standard approach was "write a really long prompt and hope for the best." Which worked often enough to feel like a strategy, and failed spectacularly often enough that I spent a lot of time in debug mode wondering which part of my fifty-line prompt the AI had decided to ignore.

The thing I eventually figured out — the thing that changed how I work — is that the prompt failures are not random. They fall into patterns. And once you can see the pattern, you can address it systematically before you hit submit.

That systematic approach turned into a framework I call SCRUB.

It's not a brand new category of idea. It's a checklist. Five elements, every prompt. It forces you to address the five categories of AI-generated code failure before the code gets generated. I wrote a whole book about it — Practical AI Assisted Development, which covers the full framework plus a lot of the tooling philosophy around it. But the heart of it is five letters, and you can use it today, right now, without reading anything else I've written.

Let me explain what each one means, and why the one most people skip is the most important.

What SCRUB Actually Is

S is Scope. What are you asking for, named precisely. Not "build me a service" but "build me exactly these four methods, with these exact signatures, in this file, and nothing else." The AI will expand your Scope if you let it. It will add helper methods you didn't ask for. It will create a base class when you just needed a concrete class. It will refactor adjacent code while it's in there because the adjacent code could be cleaner and the AI finds mess uncomfortable. Scope is the fence around what you actually want. Build it tight.

C is Constraints. The technical rules your output must respect. Framework version. Naming conventions. Architectural patterns. DI style. Which async pattern you use. The AI doesn't know you're on Angular 20 standalone with signals and @if/@for control flow unless you say so. If you don't say so, you'll get *ngFor in your template and a BehaviorSubject in your class because those are the idioms from the version of Angular that's most represented in training data. I've rebuilt the same component three times before I started writing constraints explicitly. I've been on Angular 20 with signals for months now. I still write the constraint every time. The AI does not have session memory that says "this person uses signal-based state management, use that."

R is Restrictions. This is the one everyone skips, and it is the single most important element in the entire framework. Pay attention here.

Restrictions are the things you explicitly do not want. Not what you want built, what you want avoided. This is where you prevent the Plausibility Trap, because the Plausibility Trap lives inside the gap between what you asked for and what you said was wrong.

Here's the example from the book that sticks with me. Imagine you're building a tip calculator UI. You write a great Scope block, solid Constraints, the AI produces a beautiful component. Clean code. Nice UI. Runs great. The default tip is twenty percent.

You didn't specify the default. The AI picked twenty percent because twenty percent is the most common default in tip calculator UX in training data. Your company had a conversation six months ago about defaulting to fifteen percent because the restaurant group you work with operates in a market where twenty percent feels presumptuous and you got customer complaints. That decision is not in your codebase. It's not in any document the AI can read. It's in a Slack thread from March.

The Restriction is: "Do NOT default the tip percentage to any value — the field should start empty and require explicit operator input." One line. Specific. With a reason.

That's what Restrictions do. They close the gaps that Scope leaves open. They're the part of the prompt that says "I know you'll make a reasonable-sounding decision here — here's what's actually required."

I tier my Restrictions:

• CRITICAL: "Do NOT default the tip" — missing this produces wrong data in production, silently
• IMPORTANT: "Do NOT add helper methods to this file" — missing this creates tech debt I'll pay later
• PREFERRED: "Do NOT abbreviate variable names" — style nit, not worth blocking on

The CRITICAL ones are the ones that prevent the Plausibility Trap. Write them first. Write them with reasons. The reason is not documentation fluff — it's what tells the AI why the seemingly-reasonable default is wrong for this domain.

U is Usage. Who runs this, under what conditions. A component built for a food blogger operator on a desktop CMS is different from the same component built for a tablet-based POS during a morning rush, even if the API contracts are identical. Usage is usually three to five lines. It is not usually interesting to write. It shapes dozens of downstream decisions and you won't notice most of them unless the AI gets them wrong.

B is Behavior. What must not change. This is the element that stars in Edit Mode prompts — "I want you to add these two validation rules and change nothing else, method order preserved, XML doc comments byte-for-byte identical." In greenfield work it's often N/A. In layered greenfield work — building a complex component in sequential steps — it becomes "all signals and computed values from Layer 1 are preserved unchanged," and it matters more than you think because the AI will cheerfully improve your earlier work if you don't explicitly tell it not to.

Five elements. The discipline is not the template — it's the habit of asking, before you hit submit, where is R? Because R is the one people skip. And R is what prevents the social copy from firing off the wrong image.

RecipeForge, and Why This Component Specifically Was a Problem

RecipeForge is a food blogger CMS. It runs on .NET 10 with Aspire 13.3 on the backend, Angular 20 standalone on the frontend. The idea is that a food blogger picks a channel — we have six of them: Cozy Kitchen, Baker's Bench, Quick Bites, Grill Master, Plant Forward, World Flavors — picks a posting day, and the system walks them through building a ready-to-post image and full platform-tailored social copy in one workflow. Image generation via Venice.ai. Prompt composition via OpenAI. Copy for seven platforms: Instagram, TikTok, Pinterest, Facebook, X, Threads, and a blog intro paragraph. One screen, start to finish, everything saved to a Digital Asset Manager and dropped onto a Kanban board.

The content pipeline component is the centerpiece of that workflow. When I sat down to build it, here's what I was looking at:

Nine discrete steps, represented as a TypeScript union: 'config' | 'seeding' | 'reviewSeed' | 'composing' | 'reviewPrompt' | 'rendering' | 'pickImage' | 'social' | 'done'. Thirty-one signals. Six API calls — two to OpenAI, two to Venice.ai, one to the DAM, one to the Kanban service. Two deliberate human-in-the-loop checkpoints where the operator can intervene and edit before anything expensive fires. A variant grid showing up to four generated images. A full-screen lightbox with keyboard navigation. Drag-and-drop reference image upload with a "reverse-engineer this image into a prompt" button. State that cleans itself up completely between runs.

That's not a complex component. That's a complex workflow expressed as a component. There's a difference. A complex component has a lot of moving parts. A complex workflow has ordering constraints — real business rules about what happens before what, enforced by the UI, with consequences if the ordering is wrong.

The consequence here, as I explained at the top, is social copy that doesn't match the image. Because there are three plausible places in this workflow where you could call runSocial(), and two of them are wrong. The AI, given a loose prompt, will pick the one that looks most natural from a "I'm building a step machine" perspective, which is "fire social copy generation after the render step completes." Completely reasonable. Wrong. Social needs to fire after image selection, not after image rendering, because social copy has to be written to describe a specific image — the one the operator actually kept — not a batch that might include images they're about to discard.

If I one-shotted this component, I would have gotten that wrong. And I probably wouldn't have caught it for days.

The Document I Wrote Before I Wrote a Single Line of TypeScript

There's a file in the RecipeForge repository at docs/SCRUB-PIPELINE-PROMPTS.md. It exists before the component exists. The prompts document is not documentation of what I did — it's the plan I worked from. The thing I reviewed and revised before I opened Claude Code Chat.

The document follows a four-step chain: Context → Plan → Execute → Verify.

Context gets the AI oriented to the codebase. Plan aligns on architecture before code exists. Execute generates the code in layers. Verify confirms the invariants held. The whole chain took about half a day to run. The component compiled cleanly at each layer.

Let me walk through what's in each step and why the decisions look the way they do.

Pre-Load Context: Making the AI Read Before It Writes

The very first thing in the document is a snippet I paste at the start of every Claude Code Chat session before doing anything else. It lists six files and tells the AI to read them:

• domain.models.ts — channel keys, pick lists, social platform definitions, all the seed and response types
• scene-presets.ts — nineteen food photography scenes across four scene groups
• workflow-api.service.ts — every API method the component will call, with their actual TypeScript signatures
• dam-api.service.ts — the saveAsset method
• kanban-api.service.ts — the createCard method
• image.component.ts — an existing working component that already does a version of the seed-and-render flow

That last one is pulling a lot of weight. The instruction doesn't say "here are the patterns I want you to follow, let me describe them." It says "here is a component that already does a version of this correctly, read it and match what you see." That is the most reliable Constraint you can write. Not a description of the convention — a pointer to an exemplar. The AI reads the actual code and extracts the pattern from it. Every time I've tried to describe a pattern in prose versus pointing at a live example, the example wins. Not by a little. By a lot.

The last line of the pre-load is: "Do not generate any code yet."

This matters because Claude will sometimes start generating the moment it finishes reading files. It's being helpful. I don't want helpful yet. I want understanding. "Do not generate any code yet" is four words that prevent a premature response and keep the session in the right state for the Context step.

Step 1 — Context: Ask for Understanding Before You Ask for Code

The Context prompt asks Claude to produce a structured summary. Not code — understanding. What signal and computed patterns does image.component.ts use? What are the exact shapes of ContentPromptSeedResult, ImageBatchResponse, and SocialWorkflowResponse? Which WorkflowApiService methods correspond to which pipeline stages?

There are two reasons I do this.

One: if the AI misunderstood something, I catch it here while it's cheap. A misunderstanding in the Context step costs me a one-line correction. A misunderstanding in Layer 4 costs me a template rewrite.

Two: asking the model to synthesize what it read before it produces new code improves the quality of the production step. I know this sounds like folk magic. It isn't. The model has to build an active representation of the relevant patterns before generating against them, and making that step explicit and separable — as a distinct prompt with a distinct deliverable — produces better output than jumping straight to execution. I noticed this pattern after running maybe thirty layered prompt chains. I tested it by removing the Context step and comparing results. The Context step wins.

I've caught two real errors in RecipeForge's Context step — both about how firstValueFrom() was being used in the existing service layer. Both would have produced subtle async bugs in the methods layer if they'd made it to Layer 6. One correction in prose, at the Context step, and neither showed up again.

Step 2 — Plan: The Architecture Review Before the Build

The Plan prompt asks for a prose proposal. The step union type. The thirty-one signals — which ones and what they represent. The two edit checkpoints and what state the operator needs editable at each. How the lightbox integrates with batch state. How social copy fits into the post-pick flow. All in prose, zero TypeScript.

This is the step everyone skips. I understand the impulse. You have the Context, you know what you want, just generate the code already. Skipping the Plan is how you get clean code that does the wrong thing.

A bad plan generates plausible code. A good plan generates code where the only surprises are in the formatting.

What I'm specifically probing in the Plan review: does the AI agree that social fires after image selection? Does it understand that reviewSeed and reviewPrompt are active edit windows and not loading spinners? Does it know that the variant discard happens inside selectImage(), not in a separate cleanup step? These are the decisions with ordering consequences. If the plan is wrong about any of them, the Execute prompts will generate wrong code from a correct-looking starting point.

I push back in prose when I disagree with the proposal. Sometimes I don't push back — the AI's proposal is better than my initial instinct, and I update the prompts document before running Execute. This has happened twice on RecipeForge. The AI suggested a derived computed signal for the visible-platforms list that I hadn't thought of, and it was cleaner than what I had planned. The Plan step is not "convince the AI to do what I decided" — it's alignment. It takes maybe fifteen minutes. On a component this size, it is the best-spent fifteen minutes in the whole build.

Execute: Seven Layers, Each One Verifiable

Execute is where the code gets built. Seven prompts, each one scoped to a piece of the component that can be checked independently, each one explicitly preserving what came before.

Layer 1: The Signal Skeleton

Layer 1 produces the class with all signals, computed derivations, constants, injected services, and read-only pick lists — and nothing else. The @Component decorator. An empty template. No method bodies. Every method a stub.

This is the most boring layer. It's also the most important one.

If the signal names are wrong in Layer 1, every subsequent layer references the wrong names and you get a cascade of TypeScript errors that trace back to a naming decision that would have taken two minutes to get right at the start. The goal is a clean compile from Layer 1 onward. Everything on top of a passing build, never in the dark.

The CRITICAL restrictions here are blunt: "Do NOT write template HTML beyond the bare decorator" and "Do NOT write any method bodies." The AI will try to fill these in because a component class with no template and no implementation is unusual-looking. "Helpful" means filling in what appears missing. Helpful is not what I want here. I want a typed signal inventory. I'll take helpful in Layer 2.

The IMPORTANT restriction — "no RxJS Subject or BehaviorSubject for component state" — is there because Angular 20 with signals means exactly that. Signals for state, firstValueFrom() to bridge HTTP observables into async methods. Every time I've run a pipeline-style prompt without this restriction, at least one BehaviorSubject appears. Not because the AI is doing something wrong — BehaviorSubject is a completely correct Angular state pattern, just from a slightly older era. Without the restriction, the training distribution wins.

Layers 2 Through 5: Building the Template in Slices

The template is four layers. Each one adds a vertical slice of the UI without touching what already exists.

Layer 2 is the page header and config card — channel selector, posting day, variant count, the Run Pipeline button. Unremarkable on its own. The important thing here is the Behavior block: "All signals and computed values from Layer 1 are preserved unchanged." That one line stops the AI from quietly improving my signal initializations while it adds the template. Without it, you get a template that's fine and a class that's been silently refactored. I have spent an afternoon tracing that particular class of diff-confusion. Not fun.

Layer 3 adds the mid-workflow steps: the seeding spinner, the reviewSeed edit block with scene selector and reference image upload zone, the composing spinner, the reviewPrompt block with the Venice prompt textarea and the character counter. These are the two human-in-the-loop checkpoints. This is where the operator can intervene.

One restriction from this layer I want to call out because it's a good illustration of how Restrictions work in practice: "Do NOT show the character counter outside the reviewPrompt step."

The first time I generated this template without that restriction, the counter appeared in the reviewSeed block, attached to the concept text field. Which makes a certain template-generation logic — it's a textarea, there's a counter component nearby, they belong together. What the AI doesn't know is that concept text has no character limit, and showing a counter on a field with no limit creates visual anxiety the operator doesn't need at that stage of the workflow. They're not trying to stay under a character budget there. They're editing a concept description. The counter is noise.

That's domain knowledge. The AI has no way to derive it from the code. The only way it ends up right is if I put it in the Restriction. I put it in the Restriction. Never saw the problem again.

Layer 4 is the variant grid and the lightbox overlay. This layer has what I consider the most important CRITICAL restriction in the entire prompts document:

"Do NOT allow the Keep button inside the lightbox to call saveAll() — it must call selectFromLightbox(), which chains to selectImage(), which chains to runSocial()."

Let me explain why this is CRITICAL and not just IMPORTANT.

The lightbox Keep button is the moment the operator says "this is the image I want." When they click Keep, three things need to happen in sequence: the selection gets recorded, the unchosen variants get discarded, and the social copy gets generated based on the selected image. That entire sequence lives inside selectImage().

If the Keep button calls saveAll() instead, the operator is saving a content card before the social copy has been generated. The DAM asset gets created. The Kanban card gets dropped. Both of them have empty or stale social copy fields because the social workflow hasn't run yet. No error. No warning. Just a board full of cards with wrong descriptions that an operator discovers weeks later.

Three words in a button binding. One CRITICAL restriction. The difference between a bug that's embarrassing to discover and a bug that's expensive to fix.

Layer 5 is the result grid — chosen image on the left, all seven platform copy blocks on the right, save bar below. The save bar has the platform selector, card title, scheduled date, and the Save button. The CRITICAL restriction: "Do NOT show the Save button as enabled while socialState() !== 'success'."

The operator cannot save until social copy generation has succeeded. That's not a UX detail — that's the product rule. Without the restriction, the AI produces a Save button that's enabled as soon as the step renders, because that's what most forms in training data look like. Forms have submit buttons and submit buttons are available. The contextually-gated submit button requires the explicit rule.

Layer 6: The Methods Layer

Layer 6 drops method bodies into all the stubs Layer 1 created. run(), continueFromSeed(), renderImage(), selectImage(), runSocial(), saveAll(), the reference image handlers, the lightbox helpers, the copy button, the restart flow. Every orchestration method.

I put this layer last deliberately. The entire template exists before the method bodies are written. When the AI is building selectImage(), it can see the full template context it's operating in — it knows what the lightbox looks like, it knows what the social grid expects, it knows what the save bar requires. It's not writing methods in a vacuum and hoping the template that comes later matches.

The sequencing restrictions in this layer are the ones I care most about:

"Do NOT call runSocial() before selectImage() resolves — social fires inside selectImage() only." CRITICAL because the violation produces silent wrong data. Social firing too early generates copy that describes the wrong image. No exception. No test failure. Just bad content on your board.

"Do NOT delete the chosen image variant — delete only the unchosen variants." This one seems obvious until you're reading auto-generated code and find .forEach(f => api.deleteImage(f).subscribe()) iterating over all the filenames, including the one the operator just kept. The AI is pattern-matching on "clean up after image selection" from training data. The correct behavior requires saying it explicitly, because "delete all but one" is a restriction, not a convention.

The Constraint for async style: "All orchestration methods use async/await with firstValueFrom() — no .subscribe() inside orchestration logic." The reason is error handling. Async/await with try/catch in an orchestration method gives you predictable recovery — you know which step failed, you set the right error signal, you return the UI to the right state. A .subscribe() in the middle of an orchestration chain makes error recovery harder to reason about and harder to test. The rule isn't theoretical. Every time I've let a .subscribe() sneak into orchestration code, I've regretted it.

Layer 7: Helpers and Styles

Layer 7 adds the private derivation helpers and the full component CSS. Smaller than what came before, but pickLocation() is worth a mention because it's a clean example of domain knowledge that has to live in the prompt.

pickLocation() maps cooking method keywords to scene presets — grilling and smoking go to outdoor scenes with a grill-adjacent preference, baking goes to kitchen scenes with a rustic preference, no-cook and raw-prep go to table setups. The function is maybe twenty lines. The mapping inside it represents product decisions we made about how food photography should relate to cooking context. Without those decisions in the Restriction block, the AI produces either a random mapping or a plausible-looking one based on whatever associations are strongest in training data.

Writing the mapping down in the prompt is also writing it down as product documentation. Anyone who reads the prompts file six months from now understands not just how the function works but why it maps the way it does.

The Verify Step: Thirteen Specific Checks

After all seven layers compile, there's a Verify prompt. It asks Claude to re-read the completed component and run thirteen specific checks, reporting each as PASS or FAIL.

The CRITICAL checks hit the exact failure modes I was most worried about:

Does any step transition skip a review checkpoint? Does runSocial() fire anywhere except inside selectImage()? Does selectImage() accidentally delete the chosen filename? Does saveAll() run while socialState() !== 'success'? Is there any localStorage, sessionStorage, or document.cookie anywhere in the file? Does any orchestration method use .subscribe() instead of firstValueFrom()? Are all seven social platforms rendered in the result grid? Is the Venice prompt character limit enforced at 1499 characters? Is there any *ngIf or *ngFor in the template? Does the lightbox Keep button call saveAll()?

That last one is on the list specifically because in an early iteration, before I had the CRITICAL restriction nailed down, the Keep button did call saveAll(). I caught it in the Verify step, not the diff review, because I was scanning the diff too fast. The Verify step is there precisely for that situation — the one where you're moving quickly and you miss the thing that matters.

Thirteen checks. Roughly half a day of build time. Zero production incidents related to ordering bugs on this component.

What I Would Do Differently

A few things.

I would write the Restrictions for each layer before writing the Scope. My current habit is still backward sometimes — I write what I want built, then I think about what could go wrong. The right habit is Restrictions-first. Sit down with a blank [R] block and ask "what are the five most plausible ways this layer could compile and still be wrong?" The list is almost always longer than five, but asking the question surfaces the critical ones before they bite you in production.

I'd be more explicit about CSS variable usage in Layer 7. I had a restriction that said "no hardcoded color values except platform dot colors," and the first pass had hex values showing up in animation keyframes and hover transitions. The restriction was right but not specific enough. Lesson filed: restrictions on styles should name specific properties, not just categories.

I would add a Known Limitations section to the prompts document earlier. I added it at the end. During the build I was tracking four edge cases in my head — the reference image .subscribe() pattern, the BRAND_ARCHETYPES dependency, the lightbox containing-block CSS caveat, the orphaned variants on navigation. Writing them down as I encountered them would have saved me the mental overhead of holding them until the build was done. The prompts document is not just a pre-build artifact. It's a living document you update as you learn things.

Why This Matters More Than It Looks Like It Does

I want to be clear about what the SCRUB prompts document actually is, because I've seen people mistake it for something else.

It is not documentation. It's not a write-up of what I did after the fact. It is the plan I worked from before I generated a single line of code for this component. And because it exists as a versioned, committed file in the repository, it becomes the first place I look when the component needs to change, not the last.

Buffer integration is on the roadmap — a "Schedule in Buffer" button that pushes copy and images directly to Buffer from the pipeline save step. When I add that feature, I will update SCRUB-PIPELINE-PROMPTS.md first. The new Layer 5 restrictions, the new Layer 6 method contracts, the additional Verify check for the Buffer call. Then I run the updated prompts. The component is built and maintained with the same discipline.

This is what the framework means when it says to treat prompts like code. Not a metaphor. A literal practice. The prompts directory is in source control. The prompts have authors and commit history. When something goes wrong in the component — and something always goes wrong eventually — the prompts tell me exactly which element of which layer broke.

Wrong default? Missing Restriction. Wrong state transition? Vague Scope or missing Restriction. Broke something I didn't touch? Weak Behavior block. The diagnostic card maps symptom to element. Element to fix. No hunting.

The other thing the prompts document gives me is a record of why the code is the way it is. Six months from now, when someone new is reading pipeline.component.ts and wondering why runSocial() only ever fires inside selectImage() and not in any of the three other plausible locations — the answer is in the prompts file. Not buried in a comment. Not in my head. In a file that's been reviewed, committed, and is available to anyone who runs git log.

That is worth the fifteen minutes it takes to write the plan before you write the code. Every time.

The Practical Version

If you want to try this on a component you're building:

Start with Pre-load Context. Name the files the AI needs to read. Tell it explicitly not to generate anything yet. Five minutes, not fifty.

Run a Context step first. Ask for a summary of what it read before asking for code. Read the summary. Correct misunderstandings in prose. This is the cheapest error-catching stage in the whole chain.

Write a Plan prompt and read the proposal. Push back on anything with wrong ordering or wrong behavior. Do not skip this step because it feels slow. It is the cheapest insurance in the whole process.

Split Execute into layers. Four to seven is usually right. If you can't describe what a layer produces in three sentences, the layer is too big. Split it.

Write Restrictions before Scope. Ask yourself: what are the five ways this layer could compile and still be wrong? Write those down first. The Scope block is usually easier.

Tier your Restrictions. CRITICAL means silent wrong data in production. IMPORTANT means technical debt you'll pay. PREFERRED means style. Know which ones you can compromise on and which ones you absolutely cannot.

Write a Verify prompt. Not "does it run" — specifically, does it have the behaviors you required? Thirteen specific PASS/FAIL checks on a twelve-hundred-line component caught a real bug I missed in review.

Commit the prompts document alongside the code. Update it when the component changes. It is not overhead. It is the architectural record of the component.

The docs/SCRUB-PIPELINE-PROMPTS.md file is in the RecipeForge repository if you want to read the actual prompts rather than my description of them. The component it produces — brandforge-ui/src/app/features/pipeline/pipeline.component.ts — is the result of running those prompts in sequence with the pre-loaded context. Clean compile at every layer. All thirteen Verify checks passed on first run.

That is what disciplined prompt engineering looks like. Not faster, necessarily. Not more impressive. Just fewer bugs that compile and run and don't get caught until production. And when something does go wrong, a clear map back to the decision that produced it.

That's a trade I'll take every time.

*Robert Felkins is the principal architect at Architect4Hire and the author of Practical AI Assisted Development. If you want to go deeper on SCRUB — the full tiered restriction system, the Edit Mode prompt patterns, applying the framework to existing codebases — the book covers all of it. Find it at the companion repository, or reach out at robert@architect4hire.com.

When I first mapped out AspireHire’s architecture, the backend came together fast. ASP.NET Core 9, .NET Aspire for orchestration, Azure SQL and Cosmos for persistence, and microservices for Profiles, Jobs, Recruiters, Contracts, Messaging. That’s my comfort zone. I could sketch it out in five minutes and feel good about it.

The UI, though? That was a different beast.

We needed something recruiters would trust and developers would enjoy using. Clean, modern, responsive. But here’s the thing—I’m not a front-end developer. I’m a backend architect. My brain runs on APIs, services, queues, and data models. Drop me into a blank Angular project and I’ll be muttering at flexbox for days.

So I had a decision to make.

Do I build the UI from scratch and spend weeks wrangling layout and styling? Or do I start with a polished Angular + Bootstrap template and get to value faster?

I chose the template. That one move saved us months and gave AspireHire a head start.

Why Angular Made Sense

Let’s talk about the framework choice first.

• Angular has a strong structure and built-in TypeScript support. That’s great for long-term maintainability.
• The ecosystem is deep. Charts, grids, auth helpers, translation tools—you name it, Angular has it.
• It scales with the team. As AspireHire grows, we’ll bring in front-end devs who already know Angular. It’s a safe bet.
• It works well with .NET. Angular and ASP.NET Core integrate smoothly when it comes to routing, authentication, and API calls.

So Angular was the right fit. The next question was how to get a usable UI up and running quickly.

Why Bootstrap Instead of Angular Material

Angular Material is solid, but it comes with a strong Google design identity. AspireHire needed something more flexible and easier to brand.

Bootstrap gave us that.

• It’s familiar. Buttons, grids, and navbars look exactly how users expect.
• It’s easy to customize. We could rebrand it with our own colors and typography without a fight.
• It’s mobile-first. The grid system is responsive and well-tested.
• The community is huge. Tons of examples, documentation, and support out there.

Bootstrap gave us a clean, professional starting point that didn’t feel like a cookie-cutter template. And we could make it our own with just a few tweaks.

Why I Didn’t Build It From Scratch

If I’d gone the hand-crafted route, here’s what I would’ve been dealing with:

• Setting up Angular project scaffolding
• Building layouts, navbars, and grid systems from scratch
• Writing custom components for buttons, forms, tables, and modals
• Debugging CSS across browsers
• Wrestling with flexbox to get mobile breakpoints right

That’s a lot of time before AspireHire even looks like an app. Meanwhile, stakeholders are asking when they’ll see something real.

Starting from scratch would’ve slowed us down and risked losing momentum.

What the Template Delivered

By choosing a solid Angular + Bootstrap template, we got:

• Clean layouts with dashboards, sidebars, and navbars already styled
• Reusable components like buttons, forms, modals, and tables
• Auth flows including login, signup, and password reset screens
• A responsive grid system that worked across devices
• Easy branding options to match AspireHire’s style

Instead of building a design system from the ground up, we could plug in our backend APIs and start delivering value.

How It Fit with .NET Aspire

This wasn’t just about saving time on the front end. It was about making sure everything lined up with our backend architecture.

Here’s how it worked:

1. A user logs in using a Bootstrap-styled form
2. Angular services call our .NET Identity microservice
3. JWT tokens come back and get stored in Angular
4. Angular interceptors attach those tokens to API calls
5. Microservices respond with JSON for profiles, jobs, recruiters, and contracts
6. Bootstrap components display the data in clean, responsive layouts

The UI felt polished and professional. The backend stayed scalable and observable. It was a smooth handoff between layers.

Branding with Bootstrap

Out of the box, Bootstrap is clean but generic. AspireHire needed a little more personality.

So we customized:

• Updated the color palette with warm oranges, deep reds, vibrant purples, and balanced blues
• Swapped in Google Fonts to match our brand voice—professional and approachable
• Dropped in AspireHire logos across headers, footers, and login screens
• Tweaked SCSS variables to give the UI a distinct look while keeping Bootstrap’s structure

In just a few days, the UI looked like AspireHire. Not just another Bootstrap site.

What I Learned Along the Way

Templates are helpful, but they’re not magic. Here are a few takeaways:

• Learn the template’s structure. Even if you’re not a front-end expert, understand how routing, services, and components are wired.
• Don’t over-customize too early. Focus on delivering business value first. Styling can come later.
• Use Bootstrap’s utilities. The spacing, grid, and responsive helpers are there for a reason. Don’t reinvent them.
• Keep your integration clean. Angular services should wrap backend APIs without adding business logic to the UI.
• Plan for growth. Bootstrap is a great starting point, but as AspireHire scales, we’ll likely build a design system on top of it.

Why This Was a Strategic Move

This wasn’t about cutting corners. It was about making smart choices.

• We got to demos faster. Stakeholders saw real screens within two weeks.
• We saved time and money. Less boilerplate meant more focus on features.
• We stayed focused. Developers worked on what mattered—like recruiter-matching algorithms and developer portfolios.
• We built a foundation. Bootstrap’s familiar patterns make it easy for future hires to jump in and contribute.

The template wasn’t a shortcut. It was leverage.

Wrapping It Up

AspireHire’s goal is to help developers showcase their skills. That meant we needed to get a working, branded UI in front of users quickly.

The Angular + Bootstrap template gave us that launchpad. It let me stay focused on backend architecture while still delivering a front end that looked and felt like AspireHire.

So if you’re a backend-first developer staring down a UI build, here’s my advice. Don’t get stuck trying to hand-roll every pixel. Use a template. Bootstrap it. Ship faster. And spend your energy where it counts.

Because at the end of the day, nobody’s impressed that you built a navbar from scratch. They care that your platform works, solves real problems, and feels trustworthy.

AspireHire is living proof that smart shortcuts lead to better outcomes.

Hey folks! So, I've been deep in the trenches building AspireHire lately, and I wanted to share something that might not seem sexy on the surface but is absolutely crucial for any serious application: our audit framework. Yeah, I know, "audit framework" sounds about as exciting as watching paint dry, but stick with me here—this stuff matters more than you think.

The "Oh Shit" Moment That Started It All

Picture this: You're running a job platform where candidates are uploading resumes, employers are posting positions, and money is changing hands. Then one day, someone calls and says, "Hey, my job posting disappeared, and I need to know exactly what happened and when."

Without an audit trail? You're basically playing detective with a blindfold on. With one? You can tell them exactly who did what, when they did it, and what changed. That's the difference between looking professional and looking like you're running your platform out of a garage (even if you literally are).

What We Actually Built

Instead of reinventing the wheel or bolting on some third-party solution that would cost us an arm and a leg, we built our own AuditDBContext that sits right on top of Entity Framework Core. It's elegant, it's automatic, and it just works.

Here's the beautiful part, every time someone saves changes to our database, our audit framework automatically kicks in and says, "Hold up, let me just record what's happening here." It captures:

• What entity changed (User profile? Job posting? Application?)
• What type of action (Created, updated, deleted)
• The exact fields that changed (Old value vs. new value)
• When it happened (Down to the millisecond)

The magic happens in our SaveChangesAsync override:

public override async Task<int> SaveChangesAsync(CancellationToken cancellationToken = default)
{
    var entityEntries = ChangeTracker.Entries()
        .Where(x => x.State != EntityState.Unchanged && x.State != EntityState.Detached);

    foreach (var entry in entityEntries)
    {
        var audit = new Audit
        {
            TableName = entry.Entity.GetType().Name,
            Action = entry.State.ToString(),
            KeyValues = GetKeyValues(entry),
            Changes = GetFieldChanges(entry) != null ? System.Text.Json.JsonSerializer.Serialize(GetFieldChanges(entry)) : null
        };
        Audits.Add(audit);
    }

    return await base.SaveChangesAsync(cancellationToken);
}

Zero extra code in our business logic. Zero chance of forgetting to log something. It just happens automatically, like a good DevOps pipeline.

Real Talk: Why This Matters for AspireHire

Trust is Everything in Hiring

When you're dealing with people's careers and livelihoods, trust isn't optional. Candidates need to know their data is safe. Employers need confidence that their job postings and candidate information are handled properly.

Having a complete audit trail means we can:

• Prove compliance when auditors come knocking
• Investigate issues quickly and thoroughly
• Build trust by being transparent about data handling
• Prevent disputes by having the receipts

The Technical Benefits Are Real Too

Beyond the obvious compliance stuff, our audit framework gives us superpowers:

Debugging Production Issues: Ever had a bug report that starts with "Something weird happened last Tuesday"? With full audit trails, we can reconstruct exactly what "weird" means.

Analytics and Insights: Want to know how users actually behave on the platform? Audit data is a goldmine for understanding user patterns and optimizing UX.

Data Recovery: When (not if) someone accidentally deletes something important, we can see exactly what was deleted and potentially restore it.

The Implementation That Actually Works

Here's what I love about our approach—it's dead simple but incredibly powerful. We have three main pieces:

1. The AuditDBContext - The orchestrator that captures everything
2. The Audit entity - Stores the high-level change information
3. The AuditFieldChange class - Captures field-level details

The framework is smart about what it tracks. It doesn't just dump everything into a log file like some solutions. Instead, it:

• Serializes complex objects properly (goodbye, [object Object] logs)
• Handles collections intelligently
• Only tracks actual changes (not just "something was saved")
• Maintains referential integrity with primary key tracking

Lessons Learned Building This

Start Simple, Stay Flexible

We started with basic audit logging and gradually added sophistication. The framework can handle complex scenarios now (like tracking changes to collections of related entities), but it started life tracking simple property changes.

Performance Matters

Adding audit trails to every database operation could theoretically slow things down. In practice? We've seen zero noticeable impact. Entity Framework's change tracking is already doing most of the heavy lifting, so we're just piggyback riding on that work.

JSON is Your Friend

Storing field changes as JSON gives us incredible flexibility. We can query it, we can deserialize it for reports, and we don't have to predict every possible audit scenario upfront.

What's Next

The audit framework is just the foundation. We're already thinking about:

• Real-time audit dashboards for monitoring platform health
• AI-powered anomaly detection to catch suspicious patterns
• Automated compliance reporting for when we inevitably need SOC 2 certification
• User-facing audit logs so people can see their own activity

The Bottom Line

Building AspireHire isn't just about connecting job seekers with employers—it's about doing it right. That means thinking about data integrity, compliance, and trust from day one, not retrofitting it later when some crisis forces our hand.

The audit framework might not be the flashiest feature we'll ever build, but it's the kind of foundational work that separates professional platforms from hobby projects. When candidates trust us with their career data and employers trust us with their hiring processes, we better have our act together.

And honestly? There's something deeply satisfying about knowing that every single change in our system is tracked, auditable, and recoverable. It's like having a really good backup strategy—you hope you never need it, but you sleep better knowing it's there.

Building AspireHire has been a masterclass in balancing feature velocity with technical excellence. Want to follow along with our journey? Check out our other posts about the platform architecture, or better yet, come try the platform when we launch. We're building something special here, one properly-audited database change at a time.

💡 Want to see how it all comes together in real code? The full implementation of the AspireHire audit framework complete with AuditDBContext, entity classes, and sample usage, is up on GitHub. Check it out, fork it, or even open an issue if you’ve got ideas for making it better.

👉 github.com/architect4hire/aspirehire

Setting the Stage

Let’s be real for a second. The world does not need another side project that fizzles out after a GitHub repo and a half-finished README. What it does need is projects that actually push developers forward, projects that teach as they grow, and projects that solve a real problem.

That’s why I’m launching AspireHire. At the surface, AspireHire looks like another freelance platform. The twist is that this one is built for developers, by a developer, with the entire journey shared in public. AspireHire is a proof of concept that doubles as a teaching tool. Over the next series of posts, I am going to break down every architectural choice, every piece of code, and every pipeline step so you can learn exactly how to build modern, cloud-native applications with ASP.NET Core 9, .NET Aspire, and Azure.

Think of this as a mix between a product launch and a masterclass in distributed system design. By the end, you will not only understand what AspireHire is but also how to apply the same patterns in your own work.

Why Build Another Freelance Platform?

If you’ve ever used Upwork or Fiverr, you know how they work. Upwork focuses on long-term projects and proposals. Fiverr is all about fast gigs at fixed prices. Both serve their markets well, but they share one big flaw: they don’t let developers show off what makes them special.

On Upwork you can write a nice cover letter and maybe share a portfolio. On Fiverr you can list a gig with a fixed price. Neither really helps a client figure out who is a serious technical professional and who just knows how to fill out a profile.

AspireHire flips the script. Instead of treating developers like interchangeable parts, it highlights what makes each one unique. Profiles include real portfolios, skill tags, certifications, GitHub integrations, and even coding test results. Clients can finally see more than a name, a rating, and a price tag.

This niche focus matters. By building only for software and IT work, AspireHire avoids the trap of being too broad. Developers get tools designed for them, and clients get confidence that they are hiring skilled professionals.

And for me, this is not just a product idea. It is a chance to show developers how to build a system like this from the ground up.

From Idea to Proof of Concept

Here is the important thing to understand. AspireHire is not just about launching another app. This project is a playground for learning. It is where I get to show you how a professional architect thinks about requirements, system design, data modeling, microservices, observability, and DevOps pipelines.

Instead of just handing you a GitHub repo at the end, I am going to walk you through the entire build. You will see the mistakes, the fixes, and the tradeoffs. You will learn not just what works but why it works.

If you have ever wanted to level up from hobby projects and tutorials into building production-ready systems, this series is going to give you that push.

What the Series Will Cover

Let me give you a roadmap of what’s coming. Each post will dive into a piece of AspireHire, with real examples and code along the way.

1. From Requirements to Architecture How to translate a vision into a technical blueprint. We will break down roles, workflows, and data models, then map them to services.
2. Setting Up .NET Aspire Why Aspire is a game-changer for cloud-native apps and how to use it for service discovery, observability, and local orchestration.
3. Microservice by Microservice Building Auth, Profile, Job, Proposal, Contract, Payment, and Messaging services with ASP.NET Core 9 and Clean Architecture.
4. Relational Data Models with EF Core Designing tables and relationships that support complex workflows like proposals, contracts, and reviews.
5. Event-Driven Messaging Using Azure Service Bus to decouple services and handle workflows like notifications and escrow payments.
6. CI/CD Pipelines with Azure DevOps Containerizing everything, pushing to Azure Container Registry, and deploying to AKS with zero downtime.
7. Observability that Actually Works Logs, metrics, and traces using .NET Aspire defaults, OpenTelemetry, and Application Insights.
8. Securing the Platform Authentication with Azure AD B2C, protecting secrets with Key Vault, and designing for PCI and GDPR compliance.
9. Building a Developer-First UI Creating a web experience that makes sense for technical users, from dark mode to GitHub integrations.
10. Scaling and Extending Planning for AI-powered recommendations, advanced search, and premium features once the core platform is solid.

By the time we are done, you will have the skills and the code samples to build your own distributed system on Azure.

The Starting Blueprint

Every project starts with a mental picture of what it will look like in production. Here’s the high-level blueprint for AspireHire:

• Platform: ASP.NET Core 9 with .NET Aspire
• Deployment: Azure Kubernetes Service
• Storage: Azure SQL for relational data, Blob Storage for files, Redis for caching
• Messaging: Azure Service Bus for events
• Observability: Application Insights with OpenTelemetry integration
• CI/CD: Azure DevOps Pipelines pushing Docker images to Azure Container Registry

The core microservices include:

• Auth Service for login and identity
• Profile Service for detailed freelancer and client profiles
• Job Service for creating and searching job postings
• Proposal Service for bids and negotiations
• Contract and Payment Service for escrow and transactions
• Messaging Service for real-time communication tied to jobs

Each service owns its own data store and can scale independently in AKS. All of them get logging, tracing, and health checks out of the box with Aspire.

Why Use .NET Aspire?

If you have ever built microservices the old way, you know the pain. You spend hours wiring up logging, tracing, configuration, and health checks before you even start writing business logic. Local orchestration is a mess.

.NET Aspire fixes that. With one method call, AddServiceDefaults, you get logging, tracing, metrics, and health endpoints already wired. Service discovery and orchestration happen in the AppHost, so running your entire stack locally feels like magic.

When you combine Aspire with Azure’s managed services, you get speed and reliability. Developers spend less time on plumbing and more time building features. That is why AspireHire is the perfect project to showcase what Aspire brings to the table.

Building with Azure in Mind

This project is built on Azure from the start. Not bolted on later. Not a side thought. From day one the services run on AKS and integrate with managed Azure services.

• AKS gives us container orchestration, scaling, and zero-downtime updates
• Azure SQL Database handles relational storage with built-in HA and backups
• Azure Blob Storage stores profile images, portfolio files, and message attachments
• Azure Service Bus manages asynchronous communication between services
• Application Insights collects telemetry and connects to Aspire’s OpenTelemetry pipeline
• Key Vault protects secrets and connects seamlessly to services through managed identities

By designing cloud-first, we get to learn not just how to build apps but how to build apps that run reliably in the environments companies actually use.

Teaching Through Real Problems

One promise I want to make is this: I am not going to hide the rough edges. You will see the broken deployments. You will see the EF Core migrations that fail. You will see the pipeline steps that don’t work the first time.

Because that is how real projects go. The value is not in seeing a perfectly polished example. The value is in seeing how to troubleshoot and fix problems like a professional. That is how you level up as a developer.

What You Will Get Out of This

If you stick with this series, here is what you will walk away with:

• Confidence to design and implement distributed systems on Azure
• Hands-on examples of microservice patterns in ASP.NET Core 9
• Real pipelines you can adapt for your own projects
• A clear mental model of how to debug, monitor, and scale apps in production
• A stronger resume, GitHub portfolio, or skill set for your next role or contract

This is not another “todo list in microservices.” This is the kind of project you can point to in an interview and say, “Yes, I know how to build that.”

Where We Go Next

This post is the kickoff. In the next one, we are going to dive into the architecture and requirements document in detail. I will show you how to take an idea like “freelance platform for developers” and turn it into concrete services, data models, and diagrams.

From there we will start building the services piece by piece. By the end, AspireHire will not only exist as a working app but also as a step-by-step guide to modern .NET cloud development.

Wrapping Up

AspireHire is both a product experiment and a learning journey. By focusing on developers as users, it solves a real gap in the freelance market. By building it out loud, it creates a chance for you to see exactly how modern cloud apps are designed and deployed.

If you have been waiting for a project that goes deeper than tutorials and takes you inside the head of an architect, this is it. The best way to learn is by building something real, and that is exactly what we are about to do together.

So grab your coffee, fire up your IDE, and get ready. This is going to be a fun ride.

Understanding the Problem: The Dual-Write Challenge

Before diving into the solution, let's understand the problem the Outbox Pattern solves. Consider this common scenario in a microservices architecture:

1. A service processes a business operation (e.g., creating an order)
2. The service needs to persist data to a database
3. The service must publish an event to notify other services

The challenge arises when trying to make both operations atomic. What happens if the database transaction succeeds but the message publishing fails? Or vice versa? This dual-write problem can lead to:

• Data inconsistency between services
• Lost events that other services never receive
• Duplicate processing when retries occur
• System-wide reliability issues

The Outbox Pattern Solution

The Outbox Pattern solves this by storing outbound messages in the same database transaction as the business data. A separate background process then reliably publishes these messages to the message broker. This approach ensures exactly-once delivery semantics and maintains consistency between local state changes and event publishing.

System Components

The solution consists of five main components:

1. OrderCoordinator - ASP.NET Core Web API (.NET 9)
2. InventoryApi - ASP.NET Core Web API (.NET 9)
3. PaymentsApi - ASP.NET Core Web API (.NET 9)
4. RollbackWorker - Azure Functions (.NET 8)
5. Contracts - Shared contract library (.NET 9)
6. Azure Service Bus - Message broker for async communication

Architectural Patterns

1. Microservices Architecture

• Each service owns its data (SQLite databases)
• Services communicate via HTTP APIs and async messaging
• Loose coupling through well-defined contracts

2. Outbox Pattern Implementation

The Outbox Pattern is implemented in the OrderCoordinator service to ensure reliable message publishing:

public sealed class OutboxMessage
{
    public Guid Id { get; init; } = Guid.NewGuid();
    public DateTime OccurredUtc { get; init; } = DateTime.UtcNow;
    public string Type { get; init; } = default!;
    public string PayloadJson { get; init; } = default!;
    public int AttemptCount { get; set; }
    public DateTime? LastAttemptUtc { get; set; }

    public static OutboxMessage From<T>(T payload) where T : class
        => new OutboxMessage
        {
            Type = typeof(T).FullName!,
            PayloadJson = JsonSerializer.Serialize(payload)
        };
}

Key Implementation Details:

1. Transactional Consistency: Outbox messages are stored in the same database transaction as business operations
2. Background Publisher: OutboxPublisher runs as a BackgroundService polling for pending messages
3. Retry Logic: Failed messages track attempt counts and timestamps
4. At-Least-Once Delivery: Messages are only removed after successful publishing

3. Orchestration-Based Saga Pattern

The OrderCoordinator implements an orchestration saga with compensating transactions:

// Try-Confirm-Cancel pattern
try
{
    // Try: Reserve inventory
    var resResp = await inventory.PostAsJsonAsync("inventory/reservations", ...);

    // Try: Authorize payment
    var authResp = await payments.PostAsJsonAsync("payments/authorize", ...);

    // Confirm both operations
    await inventory.PostAsJsonAsync("inventory/reservations/confirm", ...);
    await payments.PostAsJsonAsync("payments/capture", ...);
}
catch (Exception ex)
{
    // Compensate on failure
    db.Outbox.Add(OutboxMessage.From(new OrderRollbackRequested(...)));
}

4. Idempotency Pattern

Both the order coordinator and downstream services implement idempotency:

// Order coordinator uses idempotency keys
var idem = await db.Idempotency.FirstOrDefaultAsync(i => i.Key == idempotencyKey, ct);
if (idem is not null)
    return Results.Accepted($"/orders/{idem.ResultOrderId}", new { orderId = idem.ResultOrderId });

// Downstream services check for existing operations
var existing = await db.Authorizations.FirstOrDefaultAsync(a => a.OrderId == req.OrderId);
if (existing is not null)
    return Results.Ok(new AuthorizePaymentResponse(existing.AuthorizationId));

Step-by-Step Implementation Guide

Step 1: Design the Outbox Message Entity

The foundation of the Outbox Pattern is a database table that stores pending messages. Here's a well-designed outbox entity:

public sealed class OutboxMessage
{
    public Guid Id { get; init; } = Guid.NewGuid();
    public DateTime OccurredUtc { get; init; } = DateTime.UtcNow;
    public string Type { get; init; } = default!;
    public string PayloadJson { get; init; } = default!;
    public int AttemptCount { get; set; }
    public DateTime? LastAttemptUtc { get; set; }

    public static OutboxMessage From<T>(T payload) where T : class
        => new OutboxMessage
        {
            Type = typeof(T).FullName!,
            PayloadJson = JsonSerializer.Serialize(payload)
        };
}

Key Design Decisions:

• Immutable core properties: Id, OccurredUtc, Type, and PayloadJson are set once and never changed
• Tracking metadata: AttemptCount and LastAttemptUtc support retry logic and monitoring
• Generic serialization: The From method provides type-safe message creation
• Type information: Storing the full type name enables proper deserialization

Step 2: Integrate with Your Data Model

Add the outbox table to your existing DbContext:

public sealed class AppDb(DbContextOptions<AppDb> opt) : DbContext(opt)
{
    public DbSet<OrderRecord> Orders => Set<OrderRecord>();
    public DbSet<IdempotencyRecord> Idempotency => Set<IdempotencyRecord>();
    public DbSet<OutboxMessage> Outbox => Set<OutboxMessage>();

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        // Add indexes for efficient querying
        modelBuilder.Entity<OutboxMessage>()
            .HasIndex(x => x.OccurredUtc)
            .HasDatabaseName("IX_OutboxMessage_OccurredUtc");

        modelBuilder.Entity<OutboxMessage>()
            .HasIndex(x => new { x.AttemptCount, x.LastAttemptUtc })
            .HasDatabaseName("IX_OutboxMessage_Retry");
    }
}

Step 3: Implement Transactional Message Storage

The critical aspect is storing outbox messages within the same database transaction as your business operations:

public async Task<IResult> PlaceOrderAsync(CreateOrderDto dto, string idempotencyKey, CancellationToken ct)
{
    // Check idempotency first
    var idem = await db.Idempotency.FirstOrDefaultAsync(i => i.Key == idempotencyKey, ct);
    if (idem is not null)
        return Results.Accepted($"/orders/{idem.ResultOrderId}", new { orderId = idem.ResultOrderId });

    var order = new OrderRecord { Id = Guid.NewGuid(), Amount = dto.Amount, Status = "Pending" };
    db.Orders.Add(order);
    await db.SaveChangesAsync(ct);

    try
    {
        // Business logic with external services
        await ProcessInventoryReservation(order, dto.Items, ct);
        await ProcessPaymentAuthorization(order, ct);
        await ConfirmOrderComponents(order, ct);

        order.Status = "Confirmed";
    }
    catch (Exception ex)
    {
        order.Status = "Canceled";
        // Store rollback event in the same transaction
        db.Outbox.Add(OutboxMessage.From(new OrderRollbackRequested(
            order.Id, 
            order.InventoryReservationId, 
            order.PaymentAuthId, 
            ex.Message)));
    }

    // Store idempotency record and save everything atomically
    db.Idempotency.Add(new IdempotencyRecord { Key = idempotencyKey, ResultOrderId = order.Id });
    await db.SaveChangesAsync(ct);

    return Results.Accepted($"/orders/{order.Id}", new { orderId = order.Id });
}

Critical Points:

• Outbox messages are added to the same DbContext as business entities
• A single SaveChangesAsync() call ensures atomicity
• If the transaction fails, neither business data nor outbox messages are persisted

Step 4: Build the Background Publisher

The background service continuously polls for pending messages and publishes them:

public class OutboxPublisher(IServiceProvider sp, IConfiguration cfg, ILogger<OutboxPublisher> log) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var pollSeconds = cfg.GetValue<int?>("Outbox:PollSeconds") ?? 1;
        var batchSize = cfg.GetValue<int?>("Outbox:BatchSize") ?? 100;
        var topicName = cfg["ServiceBus:TopicName"] ?? "orders";

        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                await ProcessPendingMessages(batchSize, topicName, stoppingToken);
            }
            catch (Exception ex)
            {
                log.LogError(ex, "Outbox processing loop error");
            }

            await Task.Delay(TimeSpan.FromSeconds(pollSeconds), stoppingToken);
        }
    }

    private async Task ProcessPendingMessages(int batchSize, string topicName, CancellationToken stoppingToken)
    {
        using var scope = sp.CreateScope();
        var db = scope.ServiceProvider.GetRequiredService<AppDb>();
        var sbClient = scope.ServiceProvider.GetRequiredService<ServiceBusClient>();
        var sender = sbClient.CreateSender(topicName);

        var pending = await db.Outbox
            .OrderBy(x => x.OccurredUtc)
            .Take(batchSize)
            .ToListAsync(stoppingToken);

        foreach (var message in pending)
        {
            try
            {
                await PublishMessage(sender, message, stoppingToken);
                db.Outbox.Remove(message);
            }
            catch (Exception ex)
            {
                await HandlePublishFailure(message, ex);
            }
        }

        await db.SaveChangesAsync(stoppingToken);
    }

    private async Task PublishMessage(ServiceBusSender sender, OutboxMessage outboxMessage, CancellationToken ct)
    {
        var message = new ServiceBusMessage(outboxMessage.PayloadJson)
        {
            Subject = outboxMessage.Type,
            MessageId = outboxMessage.Id.ToString(),
            ContentType = "application/json"
        };

        await sender.SendMessageAsync(message, ct);
        log.LogInformation("Published message {MessageId} of type {Type}", 
            outboxMessage.Id, outboxMessage.Type);
    }

    private async Task HandlePublishFailure(OutboxMessage message, Exception ex)
    {
        message.AttemptCount++;
        message.LastAttemptUtc = DateTime.UtcNow;

        log.LogWarning(ex, "Failed to publish message {MessageId} (attempt {AttemptCount})", 
            message.Id, message.AttemptCount);

        // Implement exponential backoff or dead letter logic here
        if (message.AttemptCount >= 5)
        {
            log.LogError("Message {MessageId} exceeded max retry attempts", message.Id);
            // Consider moving to dead letter storage
        }
    }
}

Step 5: Configure Dependency Injection

Register all components in your DI container:

var builder = WebApplication.CreateBuilder(args);

// Database
builder.Services.AddDbContext<AppDb>(opt => 
    opt.UseSqlite(builder.Configuration.GetConnectionString("AppDb")));

// Message Broker
builder.Services.AddSingleton(sp =>
{
    var cfg = sp.GetRequiredService<IConfiguration>();
    var conn = cfg["ServiceBus:ConnectionString"]!;
    return new ServiceBusClient(conn);
});

// Background Service
builder.Services.AddHostedService<OutboxPublisher>();

// HTTP Clients for external services
builder.Services.AddHttpClient("Inventory", c =>
{
    c.BaseAddress = new Uri(builder.Configuration["Services:Inventory"]!);
});
builder.Services.AddHttpClient("Payments", c =>
{
    c.BaseAddress = new Uri(builder.Configuration["Services:Payments"]!);
});

var app = builder.Build();

// Ensure database is created
await using (var scope = app.Services.CreateAsyncScope())
{
    var db = scope.ServiceProvider.GetRequiredService<AppDb>();
    await db.Database.EnsureCreatedAsync();
}

Step 6: Add Monitoring and Observability

Create an endpoint to monitor outbox health:

app.MapGet("/outbox", async (AppDb db) =>
{
    var messages = await db.Outbox
        .AsNoTracking()
        .OrderBy(x => x.OccurredUtc)
        .ToListAsync();

    return Results.Ok(messages.Select(m => new 
    { 
        m.Id, 
        m.Type, 
        m.AttemptCount, 
        m.LastAttemptUtc, 
        m.OccurredUtc,
        Status = m.AttemptCount == 0 ? "Pending" : 
                m.AttemptCount < 5 ? "Retrying" : "Failed"
    }));
});

app.MapGet("/outbox/health", async (AppDb db) =>
{
    var stats = await db.Outbox
        .GroupBy(m => m.AttemptCount == 0 ? "Pending" : 
                     m.AttemptCount < 5 ? "Retrying" : "Failed")
        .Select(g => new { Status = g.Key, Count = g.Count() })
        .ToListAsync();

    var oldestPending = await db.Outbox
        .Where(m => m.AttemptCount == 0)
        .OrderBy(m => m.OccurredUtc)
        .Select(m => m.OccurredUtc)
        .FirstOrDefaultAsync();

    return Results.Ok(new 
    { 
        Statistics = stats,
        OldestPendingMessage = oldestPending,
        HealthStatus = oldestPending == default || 
                      DateTime.UtcNow - oldestPending < TimeSpan.FromMinutes(5) 
                      ? "Healthy" : "Warning"
    });
});

Requirements for Success

1. Transactional Database Support

The Outbox Pattern requires a database that supports ACID transactions. This includes:

• Relational databases: PostgreSQL, SQL Server, MySQL, SQLite
• Document databases with transactions: MongoDB (with replica sets), Azure Cosmos DB

Note: Simple key-value stores or eventually consistent databases won't work for this pattern.

2. Exactly-Once Processing Logic

Your business logic must be idempotent to handle potential duplicate processing:

// Always check for existing records
var existing = await db.Orders.FirstOrDefaultAsync(o => o.Id == orderId);
if (existing != null)
{
    return existing; // Idempotent response
}

// Proceed with creation only if not exists
var order = new OrderRecord { Id = orderId, /* ... */ };

3. Message Ordering Considerations

If message order matters, consider these strategies:

• Partition by entity: Group related messages using partition keys
• Sequence numbers: Add ordering metadata to messages
• Single-threaded processing: Process messages sequentially (impacts throughput)

4. Error Handling and Dead Letter Management

Implement robust retry logic with exponential backoff:

private static readonly TimeSpan[] RetryDelays = 
{
    TimeSpan.FromSeconds(1),
    TimeSpan.FromSeconds(5),
    TimeSpan.FromSeconds(30),
    TimeSpan.FromMinutes(5),
    TimeSpan.FromMinutes(30)
};

private bool ShouldRetry(OutboxMessage message)
{
    if (message.AttemptCount >= RetryDelays.Length)
        return false;

    var nextRetryTime = message.LastAttemptUtc?.Add(RetryDelays[message.AttemptCount - 1]);
    return nextRetryTime == null || DateTime.UtcNow >= nextRetryTime;
}

5. Performance and Scalability

For high-throughput scenarios:

• Batch processing: Process multiple messages in each iteration
• Parallel processing: Use multiple worker threads with proper coordination
• Database indexes: Ensure efficient querying of pending messages
• Message archival: Remove or archive successfully processed messages

Words of Caution

1. Storage Growth Management

Critical Warning: The outbox table can grow rapidly in high-volume systems. Without proper cleanup, it can:

• Consume excessive storage space
• Degrade query performance
• Impact application startup times

Solution: Implement message cleanup after successful publishing:

// Remove successful messages immediately
db.Outbox.Remove(message);

// Or archive for audit purposes
message.ProcessedAt = DateTime.UtcNow;
message.Status = "Completed";

2. Message Broker Dependency

The background publisher creates a dependency on your message broker. Consider:

• Circuit breaker patterns for broker outages
• Graceful degradation when messaging is unavailable
• Health checks to monitor broker connectivity

3. Delayed Message Processing

Messages aren't published immediately; there's a delay based on your polling interval. This means:

• Near real-time requirements may not be suitable for this pattern
• Dependent services must handle eventual consistency
• User expectations should account for processing delays

4. Database Transaction Size

Including outbox messages in business transactions can:

• Increase transaction size and duration
• Impact database performance under high load
• Create contention on the outbox table

Mitigation: Consider using separate transactions for outbox writes in extreme high-volume scenarios, accepting the risk of lost messages for performance gains.

5. Message Duplication Scenarios

Despite best efforts, duplicate messages can occur due to:

• Publisher retries after timeout (message was actually sent)
• Database connection issues during acknowledgment
• Process crashes between publishing and cleanup

Solution: Always implement idempotent message handlers:

public async Task HandleOrderCreated(OrderCreated orderEvent)
{
    // Check if already processed
    var existing = await db.ProcessedEvents
        .FirstOrDefaultAsync(e => e.EventId == orderEvent.Id);

    if (existing != null)
    {
        log.LogInformation("Event {EventId} already processed", orderEvent.Id);
        return; // Idempotent exit
    }

    // Process the event
    await ProcessOrder(orderEvent);

    // Record processing
    db.ProcessedEvents.Add(new ProcessedEvent 
    { 
        EventId = orderEvent.Id, 
        ProcessedAt = DateTime.UtcNow 
    });

    await db.SaveChangesAsync();
}

Conclusion

The Outbox Pattern is a powerful solution for ensuring reliable message delivery in distributed systems. When implemented correctly, it provides:

• Guaranteed message delivery with exactly-once semantics
• Data consistency between local and distributed state
• Resilience against partial failures
• Auditability of all system events

However, success requires careful attention to transactional boundaries, idempotency, error handling, and performance considerations. The pattern is particularly valuable in mission-critical systems where data consistency and reliability are paramount.

Remember that the Outbox Pattern is one tool in the distributed systems toolkit. Evaluate whether its benefits outweigh the added complexity for your specific use case. For simple scenarios with relaxed consistency requirements, direct message publishing might suffice. For complex, business-critical systems, the Outbox Pattern provides the reliability guarantees that enable robust distributed architectures.

By following this guide and heeding the cautions outlined, you'll be well-equipped to implement a production-ready Outbox Pattern that serves as a reliable foundation for your microservices communication strategy.

Hey fellow developers and architects! 👋

I wanted to share some insights from a recent project we completed for a major retail client who needed to modernize their monolithic e-commerce platform. They came to us with a classic story: "Our system is slow, hard to scale, and every deployment is a nail-biting experience." Sound familiar?

After six months of intense development, testing, and deployment, we successfully delivered a cloud-native microservices architecture using ASP.NET Core 9, .NET Aspire, and Kubernetes. Here are the key lessons we learned along the way – the good, the challenging, and the "wish we'd known this earlier" moments.

The Client Challenge: Why We Chose ASP.NET Core 9

Our client, a mid-size retail company, was struggling with their legacy .NET Framework application. Peak traffic during sales events would bring their system to its knees, and adding new features required coordinating deployments across multiple teams. They needed something modern, scalable, and maintainable.

After evaluating several options, we landed on ASP.NET Core 9 for several compelling reasons:

• Performance: The client's existing .NET expertise meant easier team adoption
• Container-First Approach: Perfect fit for their cloud-native aspirations
• Microsoft Ecosystem: They were already invested in Azure
• Long-term Support: .NET 9's roadmap aligned with their strategic plans

But here's the first lesson: choose your stack based on team capabilities, not just technical superiority. We could have gone with Go or Node.js, but the learning curve would have extended our timeline by months.

Lesson #1: .NET Aspire is a Game-Changer (But Mind the Learning Curve)

When we first heard about .NET Aspire, I'll be honest – I was skeptical. "Another framework to learn?" But after our initial spike, it became clear this wasn't just hype.

What Aspire Got Right

Here's how we structured our Aspire host for the client's catalog and inventory services:

var builder = DistributedApplication.CreateBuilder(args);

// Infrastructure components
var postgres = builder.AddPostgres("postgres", port: 5432)
    .WithDatabase("catalogdb")
    .WithDatabase("inventorydb");

var redis = builder.AddRedis("cache");

var serviceBus = builder.AddAzureServiceBus("messaging");

// Our microservices
var catalogApi = builder.AddProject<Projects.CatalogApi>("catalog-api")
    .WithReference(postgres.GetDatabase("catalogdb"))
    .WithReference(redis)
    .WithReference(serviceBus);

var inventoryApi = builder.AddProject<Projects.InventoryApi>("inventory-api")
    .WithReference(postgres.GetDatabase("inventorydb"))
    .WithReference(redis)
    .WithReference(serviceBus);

var webApp = builder.AddProject<Projects.WebApp>("webapp")
    .WithReference(catalogApi)
    .WithReference(inventoryApi)
    .WithExternalHttpEndpoints();

builder.Build().Run();

The Magic: With just this configuration, we got:

• Automatic service discovery
• Centralized configuration management
• Built-in observability with OpenTelemetry
• A gorgeous dashboard for local development

The Reality Check: Our junior developers initially struggled with the "magic." When things worked, they worked beautifully. When they didn't, debugging required understanding several abstraction layers.

Lesson Learned: Invest time upfront training your team on Aspire's internals. The productivity gains are massive once everyone understands what's happening under the hood.

Lesson #2: Container Optimization Matters More Than You Think

One of our early mistakes was treating containerization as an afterthought. "We'll just add a Dockerfile at the end," we thought. Wrong approach.

The Problem We Hit

Our initial container images were over 1GB, and startup times were exceeding 30 seconds in production. For a retail client expecting sub-second response times, this was unacceptable.

The Solution: Multi-Stage Builds and Alpine Images

Here's the optimized Dockerfile we ended up with:

# Build stage
FROM mcr.microsoft.com/dotnet/sdk:9.0-alpine AS build
WORKDIR /src

# Copy project files and restore dependencies
COPY ["CatalogApi/CatalogApi.csproj", "CatalogApi/"]
COPY ["CatalogApi.ServiceDefaults/CatalogApi.ServiceDefaults.csproj", "CatalogApi.ServiceDefaults/"]
RUN dotnet restore "CatalogApi/CatalogApi.csproj"

# Copy source and build
COPY . .
WORKDIR "/src/CatalogApi"
RUN dotnet build "CatalogApi.csproj" -c Release -o /app/build

# Publish stage
FROM build AS publish
RUN dotnet publish "CatalogApi.csproj" -c Release -o /app/publish /p:UseAppHost=false

# Runtime stage
FROM mcr.microsoft.com/dotnet/aspnet:9.0-alpine AS final
RUN apk add --no-cache icu-libs
ENV DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=false
WORKDIR /app
COPY --from=publish /app/publish .

# Create non-root user
RUN adduser -D -s /bin/sh appuser
USER appuser

EXPOSE 8080
ENTRYPOINT ["dotnet", "CatalogApi.dll"]

Results: Image size dropped to 180MB, startup time reduced to under 5 seconds.

Lesson Learned: Container optimization isn't just about size – it's about security, startup time, and resource utilization. Start optimizing from day one, not as an afterthought.

Lesson #3: Service Communication Strategy is Critical

Initially, we went HTTP-everywhere. Every service talked to every other service via HTTP APIs. This seemed clean and RESTful, but we quickly ran into issues during peak loads.

The HTTP-Only Approach (Our First Attempt)

// This seemed fine initially...
public class OrderService
{
    private readonly HttpClient _inventoryClient;
    private readonly HttpClient _catalogClient;
    private readonly HttpClient _paymentClient;

    public async Task<OrderResult> CreateOrderAsync(CreateOrderRequest request)
    {
        // Check inventory
        var inventory = await _inventoryClient.GetAsync($"api/inventory/{request.ProductId}");
        
        // Get product details
        var product = await _catalogClient.GetAsync($"api/products/{request.ProductId}");
        
        // Process payment
        var payment = await _paymentClient.PostAsync("api/payments", paymentData);
        
        // Create order...
    }
}

The Problem: During Black Friday testing, we discovered cascade failures. One slow service would cause timeouts across the entire chain.

The Hybrid Approach (Our Solution)

We implemented a hybrid communication strategy:

Synchronous (HTTP) for:

• User-facing operations requiring immediate response
• Operations needing strong consistency

Asynchronous (Message Bus) for:

• Business events and notifications
• Operations that can be eventually consistent

public class OrderService
{
    private readonly IInventoryService _inventoryService;
    private readonly IMessageBus _messageBus;

    public async Task<OrderResult> CreateOrderAsync(CreateOrderRequest request)
    {
        // Sync: Check inventory (needs immediate response)
        var inventoryStatus = await _inventoryService.CheckAvailabilityAsync(request.ProductId);
        
        if (!inventoryStatus.IsAvailable)
            return OrderResult.Failed("Product not available");

        // Create order first
        var order = await CreateOrderRecordAsync(request);

        // Async: Publish events for other services to handle
        await _messageBus.PublishAsync(new OrderCreatedEvent 
        { 
            OrderId = order.Id,
            ProductId = request.ProductId,
            Quantity = request.Quantity,
            CustomerId = request.CustomerId
        });

        return OrderResult.Success(order.Id);
    }
}

Lesson Learned: Don't default to all-sync or all-async. Choose the right communication pattern based on business requirements, not technical preferences.

Lesson #4: Observability Can't Be Bolted On Later

We initially focused on getting features working, planning to "add monitoring later." This was a mistake that cost us significant debugging time during UAT.

What We Should Have Done From Day One

Aspire gives you observability out of the box, but you need to instrument your business logic properly:

public class ProductService
{
    private static readonly ActivitySource ActivitySource = new("CatalogApi.Products");
    private static readonly Counter<int> ProductsCreated = 
        Meter.CreateCounter<int>("products_created_total");
    
    public async Task<Product> CreateProductAsync(CreateProductRequest request)
    {
        using var activity = ActivitySource.StartActivity("CreateProduct");
        activity?.SetTag("product.category", request.Category);
        activity?.SetTag("product.price", request.Price);

        try 
        {
            var product = new Product 
            {
                Name = request.Name,
                Category = request.Category,
                Price = request.Price
            };

            await _context.Products.AddAsync(product);
            await _context.SaveChangesAsync();

            ProductsCreated.Add(1, 
                new KeyValuePair<string, object>("category", request.Category));

            _logger.LogInformation("Product created: {ProductId} in category {Category}", 
                product.Id, product.Category);

            activity?.SetStatus(ActivityStatusCode.Ok);
            return product;
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            _logger.LogError(ex, "Failed to create product");
            throw;
        }
    }
}

The Aspire Dashboard Advantage: During development, we could see all our traces, metrics, and logs in one place. No more switching between multiple tools or guessing which service was causing issues.

Lesson Learned: Build observability into your code from the first commit. The Aspire dashboard makes local development debugging incredibly efficient, and the same instrumentation works in production.

Lesson #5: Kubernetes Deployment Complexity is Real

Moving from local Aspire development to Kubernetes production was our biggest challenge. The gap between "works on my machine with Aspire" and "works in Kubernetes" was larger than expected.

The Configuration Management Challenge

In Aspire, configuration "just works":

// This works magically in Aspire
builder.AddNpgsqlDbContext<CatalogDbContext>("catalogdb");

In Kubernetes, you need explicit configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: catalog-api-config
data:
  ConnectionStrings__CatalogDb: "Host=postgres-service;Port=5432;Database=catalogdb;Username=cataloguser"
  Redis__ConnectionString: "redis-service:6379"
  ServiceBus__ConnectionString: "Endpoint=sb://myservicebus.servicebus.windows.net/"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalog-api
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: catalog-api
        image: myregistry.azurecr.io/catalog-api:latest
        envFrom:
        - configMapRef:
            name: catalog-api-config
        - secretRef:
            name: catalog-api-secrets
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 30
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 10

Health Checks Saved Our Deployment

We learned the hard way that proper health checks are essential:

// In Program.cs
builder.Services.AddHealthChecks()
    .AddCheck("self", () => HealthCheckResult.Healthy())
    .AddNpgSql(connectionString, name: "database")
    .AddRedis(redisConnectionString, name: "redis")
    .AddCheck<InventoryServiceHealthCheck>("inventory-service");

app.MapHealthChecks("/health/live", new HealthCheckOptions
{
    Predicate = check => check.Name == "self"
});

app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("ready")
});

Lesson Learned: Aspire generates Kubernetes manifests, but you'll still need to understand and customize them. Don't assume the generated manifests are production-ready without review.

Lesson #6: Performance Testing Early Prevents Late Surprises

We thought our microservices would naturally be more performant than the monolith. We were wrong.

The Latency Discovery

During load testing, we discovered that our microservices architecture had introduced significant latency:

• Monolith: Average response time 200ms
• Our Microservices: Average response time 800ms

The culprit? Network calls and serialization overhead.

Our Performance Recovery Strategy

1. Caching Strategy: Implemented Redis caching at multiple levels

public class CachedProductService : IProductService
{
    private readonly IProductService _productService;
    private readonly IDistributedCache _cache;
    private readonly TimeSpan _cacheDuration = TimeSpan.FromMinutes(15);

    public async Task<Product> GetProductAsync(int productId)
    {
        var cacheKey = $"product:{productId}";
        var cachedProduct = await _cache.GetStringAsync(cacheKey);
        
        if (cachedProduct != null)
            return JsonSerializer.Deserialize<Product>(cachedProduct);

        var product = await _productService.GetProductAsync(productId);
        
        if (product != null)
        {
            var serializedProduct = JsonSerializer.Serialize(product);
            await _cache.SetStringAsync(cacheKey, serializedProduct, 
                new DistributedCacheEntryOptions 
                { 
                    AbsoluteExpirationRelativeToNow = _cacheDuration 
                });
        }

        return product;
    }
}

1. Connection Pooling: Properly configured database connections

builder.Services.AddNpgsqlDbContext<CatalogDbContext>(connectionString, options =>
{
    options.EnableSensitiveDataLogging(false);
    options.EnableServiceProviderCaching();
    options.EnableDetailedErrors(false);
});

// Configure connection pool
builder.Services.Configure<NpgsqlDbContextOptionsBuilder>(options =>
{
    options.EnableRetryOnFailure(maxRetryCount: 3, maxRetryDelay: TimeSpan.FromSeconds(5), null);
});

1. Async All the Way: Eliminated blocking calls

Final Results: Brought average response time down to 250ms – better than the original monolith.

Lesson Learned: Microservices don't automatically mean better performance. You need to be intentional about latency, caching, and resource utilization.

Lesson #7: The Deployment Pipeline is as Important as the Code

Setting up a reliable CI/CD pipeline took almost as much effort as writing the application code, but it was absolutely worth it.

Our Final Pipeline Strategy

# .github/workflows/deploy-production.yml
name: Production Deployment

on:
  push:
    branches: [main]
    tags: ['v*']

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Setup .NET
      uses: actions/setup-dotnet@v4
      with:
        dotnet-version: '9.0.x'
    
    - name: Run tests
      run: |
        dotnet restore
        dotnet test --configuration Release --logger trx --collect:"XPlat Code Coverage"
    
    - name: Upload test results
      uses: actions/upload-artifact@v3
      with:
        name: test-results
        path: '**/*.trx'

  security-scan:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Run security scan
      uses: securecodewarrior/github-action-add-sarif@v1
      with:
        sarif-file: 'security-scan-results.sarif'

  build-and-deploy:
    needs: [test, security-scan]
    runs-on: ubuntu-latest
    steps:
    - name: Build and push images
      run: |
        docker build -t $REGISTRY/catalog-api:$GITHUB_SHA .
        docker build -t $REGISTRY/inventory-api:$GITHUB_SHA .
        docker push $REGISTRY/catalog-api:$GITHUB_SHA
        docker push $REGISTRY/inventory-api:$GITHUB_SHA
    
    - name: Deploy to staging
      run: |
        kubectl set image deployment/catalog-api catalog-api=$REGISTRY/catalog-api:$GITHUB_SHA
        kubectl rollout status deployment/catalog-api
    
    - name: Run smoke tests
      run: |
        npm install
        npm run test:smoke
    
    - name: Deploy to production
      if: startsWith(github.ref, 'refs/tags/v')
      run: |
        kubectl set image deployment/catalog-api catalog-api=$REGISTRY/catalog-api:$GITHUB_SHA --namespace=production

Lesson Learned: Invest in your deployment pipeline early. Automated testing, security scanning, and staged deployments saved us from multiple production incidents.

What We'd Do Differently Next Time

1. Start with Aspire Service Defaults Earlier

We initially wrote a lot of boilerplate configuration code that Aspire's service defaults would have handled for us:

// Instead of writing all this manually...
builder.Services.AddOpenTelemetry()
    .WithMetrics(...)
    .WithTracing(...);
    
builder.Services.AddServiceDiscovery();
builder.Services.AddHealthChecks();

// Just use this from day one:
builder.AddServiceDefaults();

2. Plan Data Consistency Strategy Upfront

We underestimated the complexity of managing data consistency across microservices. Implementing the saga pattern later was painful.

3. Invest in Contract Testing

API contract changes between services caused integration issues. Tools like Pact would have caught these earlier.

4. Set Resource Limits from the Beginning

We experienced resource contention in Kubernetes because we didn't set proper CPU and memory limits initially.

The Bottom Line: Was It Worth It?

Absolutely. Six months later, our client is seeing:

• 99.9% uptime (vs. 95% with the monolith)
• 50% faster feature delivery (independent service deployments)
• 3x better performance during peak traffic
• Reduced operational costs (better resource utilization)

But more importantly, their development teams are happier and more productive.

Final Thoughts for Fellow Consultants

Building microservices with ASP.NET Core 9 and .NET Aspire isn't just about the technology – it's about the entire ecosystem:

1. Team Readiness: Ensure your team understands distributed systems concepts
2. Client Education: Help clients understand the complexity trade-offs
3. Incremental Approach: Don't try to rebuild everything at once
4. Observability First: Build monitoring and logging from day one
5. Performance Testing: Load test early and often

.NET Aspire genuinely changes the game for building cloud-native applications, but it's not a silver bullet. Success still requires solid architecture decisions, proper testing, and a team that understands distributed systems.

Would I recommend ASP.NET Core 9 and Aspire for similar projects? Absolutely. The developer experience improvements and production-ready defaults make it a compelling choice for teams building modern, cloud-native applications.

What's your experience been with ASP.NET Core 9 or .NET Aspire? I'd love to hear about your lessons learned in the comments!

Want to discuss this project or need help with your own microservices architecture? Feel free to reach out – always happy to chat about .NET, architecture, and lessons learned in the trenches.

Like many developers, I had fallen into some classic Entity Framework Core traps without realizing it—lazy loading, improper projections, and unoptimized query patterns. After a long night of debugging and tweaking, I finally got things running smoothly. And I figured: if I hit these issues, then others are likely to as well.

So I decided to write the guide I wish I had when I was pulling my hair out.

Entity Framework Core (EF Core) is Microsoft’s go-to object-relational mapper (ORM) for .NET developers, and for good reason. It makes data access clean, testable, and powerful. But with that power comes a whole set of performance landmines that can cripple your application if you’re not careful.

In this post, I’ll walk you through the most common EF Core performance pitfalls—and show you the wrong and right ways to handle them, with code examples and practical tips you can use immediately.

Understanding EF Core's Lazy Loading Trap

One of the most notorious performance killers in EF Core applications is the N+1 query problem, often caused by lazy loading. This occurs when you retrieve a list of entities and then access related data for each entity individually.

The Wrong Way: Triggering N+1 Queries

// ❌ BAD: This will execute N+1 queries
public async Task<List<BlogPostViewModel>> GetBlogPostsAsync()
{
    var posts = await _context.BlogPosts.ToListAsync();
    
    var result = new List<BlogPostViewModel>();
    foreach (var post in posts)
    {
        result.Add(new BlogPostViewModel
        {
            Title = post.Title,
            Content = post.Content,
            AuthorName = post.Author.Name, // Lazy loading triggers a query for each post
            CategoryName = post.Category.Name, // Another query for each post
            CommentCount = post.Comments.Count() // Yet another query for each post
        });
    }
    
    return result;
}

If you have 100 blog posts, this code will execute 301 queries: 1 to get the posts, 100 to get authors, 100 to get categories, and 100 to get comment counts.

The Right Way: Using Eager Loading with Include

// ✅ GOOD: Using Include to load related data in a single query
public async Task<List<BlogPostViewModel>> GetBlogPostsAsync()
{
    var posts = await _context.BlogPosts
        .Include(p => p.Author)
        .Include(p => p.Category)
        .Include(p => p.Comments)
        .ToListAsync();
    
    return posts.Select(post => new BlogPostViewModel
    {
        Title = post.Title,
        Content = post.Content,
        AuthorName = post.Author.Name,
        CategoryName = post.Category.Name,
        CommentCount = post.Comments.Count
    }).ToList();
}

Even Better: Using Projections to Select Only What You Need

// ✅ BETTER: Using projections to select only required data
public async Task<List<BlogPostViewModel>> GetBlogPostsAsync()
{
    return await _context.BlogPosts
        .Select(p => new BlogPostViewModel
        {
            Title = p.Title,
            Content = p.Content,
            AuthorName = p.Author.Name,
            CategoryName = p.Category.Name,
            CommentCount = p.Comments.Count()
        })
        .ToListAsync();
}

This approach loads only the data you actually need and executes a single, optimized query.

Avoiding the "Select N+1" Anti-Pattern with Batch Loading

When you need to load related data conditionally or in complex scenarios, consider using batch loading techniques.

The Wrong Way: Individual Queries in Loops

// ❌ BAD: Loading related data individually
public async Task<List<OrderSummary>> GetOrderSummariesAsync(List<int> orderIds)
{
    var summaries = new List<OrderSummary>();
    
    foreach (var orderId in orderIds)
    {
        var order = await _context.Orders
            .Include(o => o.OrderItems)
            .ThenInclude(oi => oi.Product)
            .FirstOrDefaultAsync(o => o.Id == orderId);
            
        if (order != null)
        {
            summaries.Add(new OrderSummary
            {
                OrderId = order.Id,
                TotalAmount = order.OrderItems.Sum(oi => oi.Quantity * oi.Product.Price),
                ItemCount = order.OrderItems.Count
            });
        }
    }
    
    return summaries;
}

The Right Way: Batch Loading

// ✅ GOOD: Loading all data in a single query
public async Task<List<OrderSummary>> GetOrderSummariesAsync(List<int> orderIds)
{
    return await _context.Orders
        .Where(o => orderIds.Contains(o.Id))
        .Select(o => new OrderSummary
        {
            OrderId = o.Id,
            TotalAmount = o.OrderItems.Sum(oi => oi.Quantity * oi.Product.Price),
            ItemCount = o.OrderItems.Count()
        })
        .ToListAsync();
}

Managing DbContext Lifecycle Properly

One critical aspect of EF Core that developers often get wrong is DbContext lifecycle management. The DbContext is designed to be short-lived and should not be shared across multiple operations.

The Wrong Way: Long-lived DbContext

// ❌ BAD: Keeping DbContext alive for too long
public class BadUserService
{
    private readonly ApplicationDbContext _context;
    
    public BadUserService(ApplicationDbContext context)
    {
        _context = context; // This context might be shared across requests
    }
    
    public async Task<User> GetUserAsync(int id)
    {
        return await _context.Users.FindAsync(id);
    }
    
    public async Task UpdateUserAsync(User user)
    {
        _context.Users.Update(user);
        await _context.SaveChangesAsync();
    }
}

The Right Way: Proper DbContext Lifecycle

// ✅ GOOD: Using scoped DbContext or explicit disposal
public class GoodUserService
{
    private readonly IDbContextFactory<ApplicationDbContext> _contextFactory;
    
    public GoodUserService(IDbContextFactory<ApplicationDbContext> contextFactory)
    {
        _contextFactory = contextFactory;
    }
    
    public async Task<User> GetUserAsync(int id)
    {
        using var context = _contextFactory.CreateDbContext();
        return await context.Users.FindAsync(id);
    }
    
    public async Task UpdateUserAsync(User user)
    {
        using var context = _contextFactory.CreateDbContext();
        context.Users.Update(user);
        await context.SaveChangesAsync();
    }
}

For dependency injection in ASP.NET Core, use the scoped lifetime:

// In Program.cs or Startup.cs
services.AddDbContext<ApplicationDbContext>(options =>
    options.UseSqlServer(connectionString), ServiceLifetime.Scoped);

Optimizing Queries with Proper Indexing and Query Structure

EF Core generates SQL queries based on your LINQ expressions, but not all LINQ expressions are created equal in terms of performance.

The Wrong Way: Inefficient Query Patterns

// ❌ BAD: Using client-side evaluation
public async Task<List<Product>> GetExpensiveProductsAsync()
{
    var products = await _context.Products.ToListAsync();
    
    // This calculation happens in memory, not in the database
    return products
        .Where(p => CalculateDiscountedPrice(p.Price, p.DiscountPercentage) > 100)
        .ToList();
}

private decimal CalculateDiscountedPrice(decimal price, decimal discountPercentage)
{
    return price * (1 - discountPercentage / 100);
}

The Right Way: Database-side Evaluation

// ✅ GOOD: Using database expressions
public async Task<List<Product>> GetExpensiveProductsAsync()
{
    return await _context.Products
        .Where(p => p.Price * (1 - p.DiscountPercentage / 100) > 100)
        .ToListAsync();
}

Using Raw SQL for Complex Queries

Sometimes, complex business logic is better expressed in raw SQL:

// ✅ GOOD: Using raw SQL for complex aggregations
public async Task<List<SalesReport>> GetMonthlySalesReportAsync(int year)
{
    return await _context.SalesReports
        .FromSqlRaw(@"
            SELECT 
                MONTH(o.OrderDate) as Month,
                COUNT(*) as OrderCount,
                SUM(oi.Quantity * oi.UnitPrice) as TotalRevenue
            FROM Orders o
            INNER JOIN OrderItems oi ON o.Id = oi.OrderId
            WHERE YEAR(o.OrderDate) = {0}
            GROUP BY MONTH(o.OrderDate)
            ORDER BY MONTH(o.OrderDate)", year)
        .ToListAsync();
}

Implementing Effective Change Tracking Strategies

EF Core's change tracking can become a performance bottleneck when working with large datasets. Understanding when to disable it can significantly improve performance.

The Wrong Way: Always Using Change Tracking

// ❌ BAD: Using change tracking for read-only operations
public async Task<List<ProductReport>> GenerateProductReportAsync()
{
    var products = await _context.Products
        .Include(p => p.Category)
        .Include(p => p.OrderItems)
        .ToListAsync(); // Change tracking is enabled by default
    
    return products.Select(p => new ProductReport
    {
        ProductName = p.Name,
        CategoryName = p.Category.Name,
        TotalSold = p.OrderItems.Sum(oi => oi.Quantity),
        Revenue = p.OrderItems.Sum(oi => oi.Quantity * oi.UnitPrice)
    }).ToList();
}

The Right Way: Selective Change Tracking

// ✅ GOOD: Disabling change tracking for read-only operations
public async Task<List<ProductReport>> GenerateProductReportAsync()
{
    return await _context.Products
        .AsNoTracking() // Disable change tracking
        .Select(p => new ProductReport
        {
            ProductName = p.Name,
            CategoryName = p.Category.Name,
            TotalSold = p.OrderItems.Sum(oi => oi.Quantity),
            Revenue = p.OrderItems.Sum(oi => oi.Quantity * oi.UnitPrice)
        })
        .ToListAsync();
}

// For updates, use change tracking selectively
public async Task UpdateProductPricesAsync(List<ProductPriceUpdate> updates)
{
    var productIds = updates.Select(u => u.ProductId).ToList();
    var products = await _context.Products
        .Where(p => productIds.Contains(p.Id))
        .ToListAsync(); // Change tracking enabled for updates
    
    foreach (var update in updates)
    {
        var product = products.First(p => p.Id == update.ProductId);
        product.Price = update.NewPrice;
        product.LastUpdated = DateTime.UtcNow;
    }
    
    await _context.SaveChangesAsync();
}

Implementing Efficient Bulk Operations

When dealing with large datasets, individual entity operations can be extremely slow.

The Wrong Way: Individual Operations

// ❌ BAD: Inserting records one by one
public async Task ImportUsersAsync(List<UserImportModel> users)
{
    foreach (var userModel in users)
    {
        var user = new User
        {
            Name = userModel.Name,
            Email = userModel.Email,
            CreatedDate = DateTime.UtcNow
        };
        
        _context.Users.Add(user);
        await _context.SaveChangesAsync(); // Saves after each insert
    }
}

The Right Way: Bulk Operations

// ✅ GOOD: Bulk insert with batching
public async Task ImportUsersAsync(List<UserImportModel> users)
{
    const int batchSize = 1000;
    
    for (int i = 0; i < users.Count; i += batchSize)
    {
        var batch = users.Skip(i).Take(batchSize);
        var userEntities = batch.Select(u => new User
        {
            Name = u.Name,
            Email = u.Email,
            CreatedDate = DateTime.UtcNow
        }).ToList();
        
        _context.Users.AddRange(userEntities);
        await _context.SaveChangesAsync();
        
        // Clear tracked entities to free memory
        _context.ChangeTracker.Clear();
    }
}

Best Practices for Configuration and Performance

Configure Your DbContext for Performance

// ✅ GOOD: Optimized DbContext configuration
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
    optionsBuilder
        .UseSqlServer(connectionString, options =>
        {
            options.CommandTimeout(30); // Set appropriate timeout
            options.EnableRetryOnFailure(); // Handle transient failures
        })
        .EnableSensitiveDataLogging(false) // Disable in production
        .EnableDetailedErrors(false) // Disable in production
        .ConfigureWarnings(warnings =>
            warnings.Throw(RelationalEventId.QueryPossibleUnintendedUseOfEqualsWarning));
}

Implement Connection Resiliency

// ✅ GOOD: Connection resiliency configuration
services.AddDbContext<ApplicationDbContext>(options =>
{
    options.UseSqlServer(connectionString, sqlOptions =>
    {
        sqlOptions.EnableRetryOnFailure(
            maxRetryCount: 3,
            maxRetryDelay: TimeSpan.FromSeconds(30),
            errorNumbersToAdd: null);
    });
});

Monitoring and Debugging Query Performance

Always monitor your EF Core queries in production:

// ✅ GOOD: Logging slow queries
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
    optionsBuilder
        .UseSqlServer(connectionString)
        .LogTo(Console.WriteLine, LogLevel.Information)
        .EnableSensitiveDataLogging() // Only in development
        .ConfigureWarnings(warnings =>
            warnings.Log(CoreEventId.FirstWithoutOrderByAndFilterWarning,
                        CoreEventId.RowLimitingOperationWithoutOrderByWarning));
}

Conclusion

Efficient use of Entity Framework Core requires understanding its internals and being mindful of the SQL queries it generates. The key principles to remember are:

1. Always think about the SQL being generated - Use tools like EF Core's logging to see what queries are executed
2. Avoid N+1 problems - Use Include, projections, or batch loading appropriately
3. Manage DbContext lifecycle properly - Keep contexts short-lived and dispose them correctly
4. Use AsNoTracking() for read-only operations - Don't track entities you won't modify
5. Batch operations when dealing with large datasets - Avoid individual saves in loops
6. Configure connection resiliency - Handle transient failures gracefully
7. Monitor and profile your queries - Use logging and profiling tools to identify bottlenecks

By following these practices and avoiding the common pitfalls outlined in this guide, you'll be able to build high-performance applications that scale effectively with Entity Framework Core. Remember that premature optimization can be counterproductive, so profile your application to identify actual bottlenecks before implementing complex optimizations.

The examples provided here represent real-world scenarios that many developers encounter. By understanding both the wrong and right approaches, you'll be better equipped to write efficient, maintainable code that performs well under load.

Testing is the backbone of reliable software development, yet it's often the most time-consuming aspect of building robust applications. Suppose you're working with an Onion Architecture (also known as Clean Architecture) service and need to test both your controller and business layers thoroughly. In that case, GitHub Copilot can be your secret weapon for generating comprehensive MSTest suites with DataRows that efficiently cover multiple test scenarios.

In this comprehensive guide, we'll explore how to leverage GitHub Copilot's AI-powered code generation to create sophisticated test suites that validate your onion architecture's controller and business layers. We'll cover practical techniques, provide real-world examples, and share expert tips to maximize your testing productivity.

Understanding Onion Architecture Testing Challenges

Before diving into Copilot techniques, let's briefly review the testing challenges in onion architecture:

1. Controller Layer: HTTP concerns, request/response mapping, validation, authorization
2. Business Layer: Core business logic, domain rules, orchestration between services
3. Data Layer: Repository patterns, data access, entity mapping

Each layer requires different testing approaches, and DataRows help us test multiple scenarios without code duplication.

Setting Up Your Testing Environment

First, ensure your test project has the necessary NuGet packages:

<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.6.0" />
<PackageReference Include="MSTest.TestAdapter" Version="3.0.4" />
<PackageReference Include="MSTest.TestFramework" Version="3.0.4" />
<PackageReference Include="Moq" Version="4.20.69" />
<PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="7.0.0" />

Effective Prompts for Controller Layer Testing

Basic Controller Test Generation

Prompt:

Generate MSTest with DataRows for testing a UserController PostUser action that accepts CreateUserRequest and returns UserResponse. Test cases should include valid user creation, invalid email format, missing required fields, and duplicate username scenarios. Use AAA pattern and include proper assertions.

Generated Result:

[TestClass]
public class UserControllerTests
{
    private UserController _controller;
    private Mock<IUserService> _mockUserService;

    [TestInitialize]
    public void Setup()
    {
        _mockUserService = new Mock<IUserService>();
        _controller = new UserController(_mockUserService.Object);
    }

    [TestMethod]
    [DataRow("john.doe@email.com", "John", "Doe", "johndoe", true, "User created successfully")]
    [DataRow("invalid-email", "John", "Doe", "johndoe", false, "Invalid email format")]
    [DataRow("", "John", "Doe", "johndoe", false, "Email is required")]
    [DataRow("john.doe@email.com", "", "Doe", "johndoe", false, "First name is required")]
    [DataRow("john.doe@email.com", "John", "Doe", "", false, "Username is required")]
    [DataRow("existing@email.com", "John", "Doe", "existinguser", false, "Username already exists")]
    public async Task PostUser_WithVariousInputs_ReturnsExpectedResults(
        string email, string firstName, string lastName, string username, 
        bool shouldSucceed, string expectedMessage)
    {
        // Arrange
        var request = new CreateUserRequest
        {
            Email = email,
            FirstName = firstName,
            LastName = lastName,
            Username = username
        };

        if (shouldSucceed)
        {
            _mockUserService.Setup(x => x.CreateUserAsync(It.IsAny<CreateUserRequest>()))
                .ReturnsAsync(new UserResponse { Id = 1, Message = expectedMessage });
        }
        else
        {
            _mockUserService.Setup(x => x.CreateUserAsync(It.IsAny<CreateUserRequest>()))
                .ThrowsAsync(new ValidationException(expectedMessage));
        }

        // Act
        var result = shouldSucceed 
            ? await _controller.PostUser(request) as OkObjectResult
            : await _controller.PostUser(request) as BadRequestObjectResult;

        // Assert
        Assert.IsNotNull(result);
        if (shouldSucceed)
        {
            var response = result.Value as UserResponse;
            Assert.AreEqual(expectedMessage, response.Message);
        }
        else
        {
            Assert.IsTrue(result.Value.ToString().Contains(expectedMessage));
        }
    }
}

Advanced Authorization Testing

Prompt:

Create MSTest DataRows for testing authorization in ProductController GetProducts action. Include test cases for admin role (full access), user role (own products only), guest (no access), expired token, and invalid token scenarios. Mock IAuthorizationService and include proper HTTP status code assertions.

Business Layer Testing Strategies

Domain Logic Validation

Prompt:

Generate MSTest with DataRows for OrderService CalculateTotal method in business layer. Test scenarios: normal order with multiple items, order with discounts, order with tax calculations, empty order, order with invalid quantities, and order with expired promotions. Include edge cases and boundary conditions.

Generated Result:

[TestClass]
public class OrderServiceTests
{
    private OrderService _orderService;
    private Mock<IDiscountRepository> _mockDiscountRepo;
    private Mock<ITaxCalculator> _mockTaxCalculator;

    [TestInitialize]
    public void Setup()
    {
        _mockDiscountRepo = new Mock<IDiscountRepository>();
        _mockTaxCalculator = new Mock<ITaxCalculator>();
        _orderService = new OrderService(_mockDiscountRepo.Object, _mockTaxCalculator.Object);
    }

    [TestMethod]
    [DataRow(new double[] { 10.00, 20.00, 15.00 }, new int[] { 2, 1, 3 }, 0.0, 0.10, 96.25)]
    [DataRow(new double[] { 100.00 }, new int[] { 1 }, 10.0, 0.08, 97.20)]
    [DataRow(new double[] { 50.00, 30.00 }, new int[] { 1, 2 }, 15.0, 0.12, 100.80)]
    [DataRow(new double[] { }, new int[] { }, 0.0, 0.0, 0.0)]
    [DataRow(new double[] { 25.00 }, new int[] { 0 }, 0.0, 0.0, 0.0)]
    public void CalculateTotal_WithVariousOrderConfigurations_ReturnsCorrectTotal(
        double[] itemPrices, int[] quantities, double discountPercent, 
        double taxRate, double expectedTotal)
    {
        // Arrange
        var orderItems = new List<OrderItem>();
        for (int i = 0; i < itemPrices.Length; i++)
        {
            if (i < quantities.Length && quantities[i] > 0)
            {
                orderItems.Add(new OrderItem 
                { 
                    Price = itemPrices[i], 
                    Quantity = quantities[i] 
                });
            }
        }

        var order = new Order { Items = orderItems };

        _mockDiscountRepo.Setup(x => x.GetApplicableDiscount(It.IsAny<Order>()))
            .Returns(discountPercent);
        _mockTaxCalculator.Setup(x => x.CalculateTax(It.IsAny<double>()))
            .Returns<double>(amount => amount * taxRate);

        // Act
        var result = _orderService.CalculateTotal(order);

        // Assert
        Assert.AreEqual(expectedTotal, result, 0.01);
    }
}

Complex Business Rules Testing

Prompt:

Create comprehensive MSTest DataRows for InventoryService ProcessRestockOrder method. Test scenarios include: sufficient stock levels, insufficient stock with backorder, priority customer handling, seasonal restrictions, vendor availability, quantity limits, and concurrent stock updates. Use realistic business scenarios and include proper exception handling tests.

Advanced Copilot Techniques

Using Context-Aware Prompts

Effective Strategy: Start your prompts with existing code context to get more accurate results.

Prompt:

Based on this existing UserService interface:
```csharp
public interface IUserService
{
    Task<UserResponse> CreateUserAsync(CreateUserRequest request);
    Task<UserResponse> GetUserByIdAsync(int id);
    Task<bool> DeleteUserAsync(int id);
}

Generate MSTest with DataRows for UserService implementation testing all methods. Include positive cases, negative cases, async exception handling, and edge cases. Mock dependencies appropriately.

### Chaining Prompts for Complex Scenarios

**Initial Prompt:**

Generate MSTest DataRows for PaymentService ProcessPayment method with basic scenarios: successful payment, insufficient funds, invalid card, expired card.

**Follow-up Prompt:**

Extend the previous PaymentService tests with additional DataRows for: fraud detection triggers, currency conversion scenarios, refund processing, partial payments, and payment gateway timeout handling.

### Integration Testing with Copilot

**Prompt:**

Create MSTest integration tests with DataRows for e-commerce checkout workflow spanning controller and business layers. Test complete user journey: add to cart, apply discount, calculate shipping, process payment, update inventory. Use TestServer and realistic data scenarios.

## Best Practices and Pro Tips

### 1. Structure Your Prompts Effectively

**Good Prompt Structure:**
- Context (what you're testing)
- Specific requirements (DataRows, scenarios)
- Technical constraints (frameworks, patterns)
- Expected outcomes (assertions, exceptions)

### 2. Leverage Copilot for Test Data Generation

**Prompt:**

Generate realistic test data objects for e-commerce testing including: valid/invalid credit cards, international addresses, product catalogs with various categories, user profiles with different roles, and order histories with edge cases.

### 3. Use Copilot for Mock Setup

**Prompt:**

Generate Moq setup code for testing OrderController that depends on IOrderService, IPaymentService, and IInventoryService. Include different return scenarios for each dependency and proper verify calls.

### 4. Exception Testing Patterns

**Prompt:**

Create MSTest DataRows for testing exception scenarios in UserRegistrationService. Include: ArgumentNullException for null inputs, ValidationException for invalid data, DuplicateUserException for existing users, and ServiceUnavailableException for external service failures.

## Complex Real-World Example

Let's create a comprehensive test suite for an order processing system:

**Prompt:**

Generate a complete MSTest suite with DataRows for OrderProcessingService in an e-commerce onion architecture. Include controller tests for OrderController endpoints and business layer tests for OrderService. Cover scenarios: new order creation, order status updates, inventory validation, payment processing, shipping calculations, order cancellation, refund processing, and concurrent order handling. Use realistic e-commerce data and include proper mocking of dependencies.

**Generated Result Structure:**
```csharp
[TestClass]
public class OrderProcessingIntegrationTests
{
    // Controller Layer Tests
    [TestMethod]
    [DataRow("valid-order.json", 201, "Order created successfully")]
    [DataRow("invalid-payment.json", 400, "Payment method invalid")]
    [DataRow("insufficient-inventory.json", 409, "Insufficient stock")]
    public async Task CreateOrder_WithVariousScenarios_ReturnsExpectedResults(
        string testDataFile, int expectedStatusCode, string expectedMessage)
    {
        // Implementation here
    }

    // Business Layer Tests
    [TestMethod]
    [DataRow(OrderStatus.Pending, OrderStatus.Processing, true)]
    [DataRow(OrderStatus.Processing, OrderStatus.Shipped, true)]
    [DataRow(OrderStatus.Shipped, OrderStatus.Delivered, true)]
    [DataRow(OrderStatus.Delivered, OrderStatus.Processing, false)]
    public void UpdateOrderStatus_WithStatusTransitions_ValidatesBusinessRules(
        OrderStatus currentStatus, OrderStatus newStatus, bool shouldSucceed)
    {
        // Implementation here
    }
}

Debugging and Refinement Tips

1. Iterative Improvement

Start with basic tests and refine:

Initial Prompt:

Generate basic MSTest for UserController Login method

Refinement Prompt:

Add DataRows to the previous UserController Login test for: valid credentials, invalid password, non-existent user, locked account, expired password, and rate limiting scenarios

2. Copilot-Assisted Test Maintenance

Prompt:

Update the existing UserService tests to include new DataRows for: two-factor authentication scenarios, OAuth login, password reset workflows, and account verification steps

Performance Considerations

1. Efficient Test Data

Prompt:

Generate MSTest DataRows using object arrays for complex scenarios in ProductCatalogService. Optimize for test execution speed while maintaining comprehensive coverage of product search, filtering, sorting, and pagination scenarios.

2. Parallel Test Execution

Prompt:

Create thread-safe MSTest DataRows for concurrent testing of OrderService methods. Include proper test isolation and avoid shared state issues.

Conclusion

GitHub Copilot transforms the traditionally tedious task of writing comprehensive test suites into an efficient, AI-assisted process. By crafting effective prompts and leveraging Copilot's understanding of testing patterns, you can generate robust MSTest suites with DataRows that thoroughly validate both your controller and business layers.

The key to success lies in:

1. Clear, contextual prompting that provides Copilot with sufficient information
2. Iterative refinement of generated tests to match your specific requirements
3. Strategic use of DataRows to maximize test coverage with minimal code duplication
4. Proper separation of concerns between controller and business layer tests

Remember that while Copilot excels at generating boilerplate test code and common patterns, you should always review and customize the generated tests to ensure they align with your specific business requirements and testing standards.

Start implementing these techniques in your next project, and you'll find that comprehensive testing becomes not just manageable, but genuinely productive and even enjoyable. Your onion architecture will be more robust, your code more reliable, and your development velocity significantly increased.

Happy testing with GitHub Copilot!

What is Refit and Why We Chose It

Refit is a REST library for .NET that transforms REST APIs into live interfaces. Instead of writing boilerplate HTTP client code, Refit generates the implementation for you at runtime using attributes to define the HTTP operations. This approach dramatically reduces code complexity while providing type safety and excellent IntelliSense support.

Our platform consists of multiple specialized services: ProfileService, UserService, AdminService, BuddyService, CheckedOutService, and several others. Each service needs to communicate with others to fulfill complex business requirements. Traditional HttpClient usage would result in hundreds of lines of repetitive code, but Refit transforms this into clean, declarative interfaces.

The Refit HTTP Client Portfolio

In our solution, we've implemented a comprehensive suite of HTTP client interfaces, each designed for specific inter-service communication patterns. Let's explore the key clients that power our ecosystem:

Core Service Clients

IProfileHttpClient serves as the central communication hub for profile-related operations. This client handles everything from profile creation and updates to user status management:

[Headers("Content-Type: application/json", "Another-Header: AnotherValue")]
public interface IProfileHttpClient
{
    [Post("/api/profile/v1/integration/profile/upsert/{userName}/{email}/{site}")]
    Task<Result<ProfileServiceModel>> UpsertDefaultProfile(string userName, string email, int site);

    [Post("/api/profile/v1/integration/profile/markloggedoff/{userGuid}/{site}")]
    Task<Result<NoResponseServiceModel>> MarkUserAsLoggedOff(Guid userGuid, int site);

    [Get("/api/profile/v1/integration/honeypot/{userGuid}/{site}")]
    Task<Result<HoneyPotServiceModel>> GetHoneypot(Guid userGuid, int site);
}

Notice how clean and readable this interface is. Each method clearly defines its HTTP verb, route template, and expected response type. The route parameters are automatically bound from method parameters, making the client both intuitive to use and resistant to errors.

ITokenHttpClient handles authentication token management across services:

public interface ITokenHttpClient
{
    [Post("/api/token/v1/token/")]
    Task<Result<TokenServiceModel>> FetchUserServiceToken([FromBody] TokenViewModel model);
}

This client is crucial for our authentication flow, providing service-to-service token exchange capabilities that maintain security boundaries while enabling necessary inter-service communication.

Specialized Domain Clients

IBuddyHttpClient manages relationship and social interaction features:

public interface IBuddyHttpClient
{
    [Get("/api/buddy/v1/integration/{userGuid}/{site}")]
    Task<Result<BuddyListServiceModel>> GetBuddies(Guid userGuid, int site);

    [Get("/api/buddy/v1/relationships/{userGuid}/{site}")]
    Task<Result<BuddyListServiceModel>> GetRelationships(Guid userGuid, int site);

    [Get("/api/buddy/v1/endorsements/{userGuid}/{site}")]
    Task<Result<EndorsementListServiceModel>> GetEndorsements(Guid userGuid, int site);
}

ICheckedOutHttpClient handles profile viewing analytics:

public interface ICheckedOutHttpClient
{
    [Post("/api/checkedout/v1/integration/{userGuid}/{site}")]
    Task<Result<ProfileViewCountServiceModel>> ProfileViewCountAsync(Guid userGuid, int site);

    [Post("/api/checkedout/v1/integration/{site}")]
    Task<Result<GuidListServiceModel>> GetMostPopularProfileViews(int site);
}

These specialized clients demonstrate how Refit enables us to create focused, single-responsibility interfaces that align perfectly with our domain boundaries.

Architectural Integration and Service Registration

The magic of our Refit implementation lies not just in the interface definitions, but in how we integrate them into our dependency injection container. Each service registers its required HTTP clients during startup through dedicated AddHttpClients methods:

void AddHttpClients(WebApplicationBuilder builder)
{
    builder.Services.AddRefitClient<ISocialAdHttpClient>()
           .ConfigureHttpClient(c => c.BaseAddress = new Uri(builder.Environment.EnvironmentName != "LocalDevelopment" ? builder.Configuration["socialadurl"] : "https+http://socialadservice"))
           .AddHttpMessageHandler(() => new AuthHeaderHandler(builder.Services.BuildServiceProvider()));

    builder.Services.AddRefitClient<IImagesHttpClient>()
        .ConfigureHttpClient(c => c.BaseAddress = new Uri(builder.Environment.EnvironmentName != "LocalDevelopment" ? builder.Configuration["imagesurl"] : "https+http://imagesservice"))
        .AddHttpMessageHandler(() => new AuthHeaderHandler(builder.Services.BuildServiceProvider()));
}

This configuration pattern provides several key benefits:

1. Environment-Aware Routing: Our services automatically switch between external URLs for production and local service discovery URLs for development
2. Automatic Authentication: Most clients include our custom AuthHeaderHandler for seamless token management
3. Centralized Configuration: All HTTP client settings are managed in one location per service

Authentication and Security Through HTTP Message Handlers

One of the most powerful features of our Refit implementation is the integration with custom HTTP message handlers. Our AuthHeaderHandler demonstrates how cross-cutting concerns like authentication can be elegantly handled:

public class AuthHeaderHandler : DelegatingHandler
{
    private readonly IServiceProvider _serviceProvider;

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var userHttpClient = _serviceProvider.GetRequiredService<ITokenHttpClient>();
        var configuration = _serviceProvider.GetRequiredService<IConfiguration>();

        var tokenViewModel = new TokenViewModel
        {
            Token = configuration["UserServiceToken"]
        };
        var tokenResult = await userHttpClient.FetchUserServiceToken(tokenViewModel);
        var token = tokenResult?.Data?.Token;

        if (!string.IsNullOrEmpty(token))
        {
            request.Headers.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", token);
        }

        return await base.SendAsync(request, cancellationToken);
    }
}

This handler automatically:

• Fetches authentication tokens when needed
• Adds Bearer token headers to outgoing requests
• Handles token refresh scenarios
• Provides a consistent authentication experience across all services

Multi-Environment Configuration Strategy

Our Refit configuration showcases a sophisticated approach to environment management. Notice how we handle different environments:

builder.Environment.EnvironmentName != "LocalDevelopment" ? 
    builder.Configuration["profileurl"] : 
    "https+http://profileservice"

In local development, we use Aspire's service discovery with the https+http:// scheme, which automatically resolves to the correct local service endpoints. In production and staging environments, we use configuration-driven URLs that point to the appropriate service instances.

This approach provides several advantages:

• Seamless local development experience
• Easy deployment across multiple environments
• No code changes required between environments
• Built-in support for service discovery and load balancing

Error Handling and Result Patterns

All our Refit clients return a consistent Result pattern that encapsulates both success and failure scenarios:

public class Result<T> where T : ServiceBaseModel 
{
    public bool IsSuccess { get; }
    public bool IsFailure { get; }
    public List<string> Messages { get; set; }
    public Error Error { get; }
    public T? Data { get; set; }
}

This pattern ensures that:

• All API responses follow a consistent structure
• Error handling is standardized across services
• Consumers can easily distinguish between success and failure cases
• Rich error information is available for debugging and user feedback

Performance and Scalability Considerations

Our Refit implementation includes several performance optimizations:

1. HTTP Client Factory Integration: All Refit clients are registered using the AddRefitClient extension, which leverages the built-in HTTP client factory for connection pooling and lifecycle management
2. Async/Await Throughout: Every client method returns Task, ensuring non-blocking I/O operations
3. Efficient Serialization: Refit uses System.Text.Json by default, providing excellent performance for our JSON-based APIs
4. Connection Reuse: The underlying HttpClientFactory ensures proper connection pooling and DNS refresh cycles

Integration with .NET Aspire

Our solution leverages .NET Aspire for cloud-native development, and Refit integrates seamlessly with this ecosystem. The builder.AddServiceDefaults() call in each service includes:

• Service Discovery: Automatic resolution of service endpoints
• Resilience Patterns: Built-in retry policies and circuit breakers
• Observability: Automatic OpenTelemetry instrumentation for HTTP calls
• Health Checks: Integrated health monitoring for all HTTP dependencies

Real-World Usage Patterns

Let's examine how these clients are used in practice. In our ProfileService, we inject multiple HTTP clients to gather comprehensive profile information:

public class ProfileBusiness : IProfileBusiness
{
    private readonly IBuddyHttpClient _buddyHttpClient;
    private readonly ICheckedOutHttpClient _checkedOutHttpClient;
    private readonly IImagesHttpClient _imagesHttpClient;

    public async Task<ProfileViewModel> GetCompleteProfile(Guid userGuid, Site site)
    {
        var profileTask = GetBaseProfile(userGuid, site);
        var buddiesTask = _buddyHttpClient.GetBuddies(userGuid, (int)site);
        var viewsTask = _checkedOutHttpClient.ProfileViewCountAsync(userGuid, (int)site);
        var imagesTask = _imagesHttpClient.GetImages(userGuid, (int)site);

        await Task.WhenAll(profileTask, buddiesTask, viewsTask, imagesTask);

        return BuildCompleteProfile(
            profileTask.Result,
            buddiesTask.Result,
            viewsTask.Result,
            imagesTask.Result
        );
    }
}

This pattern allows us to make parallel HTTP calls to multiple services, dramatically improving response times while maintaining clean, readable code.

Benefits and Lessons Learned

After implementing Refit across our entire microservices ecosystem, we've observed several key benefits:

Developer Productivity: Refit eliminated hundreds of lines of boilerplate HTTP client code. New developers can quickly understand and extend API integrations without learning complex HTTP client patterns.

Type Safety: Compile-time checking ensures that API contracts are maintained. When APIs change, the compiler immediately identifies affected clients.

Maintainability: Changes to API endpoints require updates only to interface definitions. The generated implementation handles all the underlying HTTP communication details.

Testing: Refit interfaces are easily mockable, enabling comprehensive unit testing of service interactions without complex HTTP mocking frameworks.

Consistency: All inter-service communication follows identical patterns, reducing cognitive load when working across different parts of the system.

Future Enhancements

Looking ahead, we're exploring several enhancements to our Refit implementation:

1. Automatic Retry Policies: Integration with Polly for sophisticated retry and circuit breaker patterns
2. Response Caching: Adding HTTP-level caching for appropriate GET operations
3. Request/Response Logging: Enhanced observability for debugging and performance monitoring
4. API Versioning: Support for multiple API versions within the same client interface

Conclusion

Refit has transformed how we handle inter-service communication in our platform. By providing a clean, type-safe abstraction over HTTP operations, it has enabled us to build a robust microservices architecture that is both maintainable and performant.

The combination of declarative interfaces, automatic serialization, integrated authentication, and seamless dependency injection makes Refit an invaluable tool for any .NET microservices architecture. Our implementation demonstrates how thoughtful architectural decisions can dramatically improve developer productivity while maintaining the flexibility and scalability that modern applications demand.

Whether you're building your first microservices system or looking to modernize an existing one, consider how Refit can simplify your HTTP client infrastructure and enable your team to focus on delivering business value rather than managing communication plumbing.

The Foundation: .NET Aspire and Service Defaults

Our journey begins with .NET Aspire, Microsoft's opinionated stack for building observable, production-ready cloud applications. At the heart of our implementation lies the ServiceDefaults project—a centralized configuration hub that standardizes observability across all our microservices.

The ServiceDefaults project serves as our single source of truth for common configurations. Every service in our solution references this project, ensuring consistent telemetry collection, health checks, and service discovery patterns. This approach eliminates configuration drift and makes it incredibly easy to add new services that automatically inherit our observability standards.

public static TBuilder AddServiceDefaults<TBuilder>(this TBuilder builder) 
    where TBuilder : IHostApplicationBuilder
{
    builder.ConfigureOpenTelemetry();
    builder.AddDefaultHealthChecks();
    builder.Services.AddServiceDiscovery();

    builder.Services.ConfigureHttpClientDefaults(http =>
    {
        http.AddStandardResilienceHandler();
        http.AddServiceDiscovery();
    });

    return builder;
}

This extension method is called in every service's Program.cs, immediately providing OpenTelemetry instrumentation, health checks, and service discovery capabilities. It's a perfect example of how .NET Aspire promotes best practices by making the right thing the easy thing.

OpenTelemetry: The Observability Backbone

OpenTelemetry forms the cornerstone of our observability strategy. Rather than relying on proprietary monitoring solutions that lock us into specific vendors, we've embraced the open standard that provides vendor-neutral instrumentation for metrics, logs, and traces.

Comprehensive Instrumentation Strategy

Our OpenTelemetry configuration captures three critical signals:

Tracing helps us understand request flows across our distributed system. We instrument ASP.NET Core applications to automatically capture incoming HTTP requests, and we track outgoing HTTP client calls to understand inter-service dependencies. Each trace creates a complete picture of how a user request travels through our system, making it invaluable for debugging performance bottlenecks and understanding system behavior.

Metrics provide the quantitative data we need for monitoring system health. We collect ASP.NET Core metrics for request rates and response times, HTTP client metrics for outbound call performance, and .NET runtime metrics for garbage collection and memory usage patterns. This gives us both application-level and infrastructure-level insights.

Logging integrates seamlessly with .NET's built-in logging framework. OpenTelemetry automatically correlates log entries with traces, so when we're debugging an issue, we can see exactly which log entries belong to a specific request trace.

builder.Services.AddOpenTelemetry()
    .WithMetrics(metrics =>
    {
        metrics.AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddRuntimeInstrumentation();
    })
    .WithTracing(tracing =>
    {
        tracing.AddSource(builder.Environment.ApplicationName)
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation();
    });

Flexible Export Strategy

One of OpenTelemetry's greatest strengths is its vendor-neutral approach to data export. Our configuration supports multiple export targets depending on the environment and requirements:

For development and testing environments, we can export to any OTLP-compatible endpoint by setting the OTEL_EXPORTER_OTLP_ENDPOINT environment variable. This might be a local Jaeger instance, a Grafana stack, or any other OpenTelemetry-compatible tool.

For production workloads, we integrate with Azure Monitor, leveraging Microsoft's OpenTelemetry distro to send telemetry data directly to Application Insights. This provides us with enterprise-grade monitoring capabilities while maintaining the flexibility to switch monitoring backends if needed.

private static TBuilder AddOpenTelemetryExporters<TBuilder>(this TBuilder builder) 
    where TBuilder : IHostApplicationBuilder
{
    var useOtlpExporter = !string.IsNullOrWhiteSpace(
        builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"]);

    if (useOtlpExporter)
    {
        builder.Services.AddOpenTelemetry().UseOtlpExporter();
    }

    if(builder.Environment.IsDevelopment() || builder.Environment.IsProduction())
    {
        if (!string.IsNullOrEmpty(
            builder.Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"]))
        {
            builder.Services.AddOpenTelemetry().UseAzureMonitor();
        }
    }

    return builder;
}

This dual-export capability means our development teams can use lightweight, local observability tools during development while production systems benefit from Azure's robust monitoring infrastructure.

Custom Instrumentation Through Action Filters

Beyond automatic instrumentation, we've implemented custom telemetry collection through our ActionControllerLogging filter. This filter captures detailed audit information for every API call, including user context, session information, request bodies, and routing data.

public class ActionControllerLogging : IAsyncActionFilter
{
    public async Task OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)
    {
        HttpAuditEntryMessageModel msg = new HttpAuditEntryMessageModel();

        msg.UserName = context.HttpContext.Request.Headers
            .Where(w => w.Key.ToLower() == "username").FirstOrDefault().Value;
        msg.Controller = ((ControllerBase)context.Controller)
            .ControllerContext.ActionDescriptor.ControllerName;
        msg.ActionMethod = ((ControllerBase)context.Controller)
            .ControllerContext.ActionDescriptor.ActionName;

        // Additional audit data collection...

        await next();
    }
}

This filter demonstrates how OpenTelemetry's extensible nature allows us to capture domain-specific telemetry that goes beyond standard web framework metrics. The audit data flows through Azure Service Bus to dedicated audit storage, creating a complete audit trail for compliance and security analysis.

Scalar: Modern API Documentation

While OpenTelemetry handles observability, Scalar addresses another critical need: comprehensive, developer-friendly API documentation. Traditional Swagger UI, while functional, often feels outdated and lacks the polish that modern development teams expect. Scalar provides a beautiful, responsive interface that makes API exploration intuitive and enjoyable.

Seamless Integration with OpenAPI

Our Scalar integration builds on .NET 9's native OpenAPI support. The Microsoft.AspNetCore.OpenApi package automatically generates OpenAPI specifications from our controller definitions, and Scalar consumes these specifications to create interactive documentation.

public static WebApplication ConfigureOpenApiAndScaler(this WebApplication app)
{
    app.MapOpenApi();
    app.MapScalarApiReference(options =>
    {
        options.WithPreferredScheme("Bearer")
            .WithHttpBearerAuthentication(bearer =>
            {
                bearer.Token = "your-bearer-token";
            });
        options.Title = $"{app.Environment.EnvironmentName} - {app.Environment.ApplicationName}";
    });
    return app;
}

This configuration provides several key benefits. The documentation automatically stays in sync with our code—when we add new endpoints or modify existing ones, the documentation updates automatically. Scalar's interface allows developers to test APIs directly from the documentation, eliminating the need to switch between documentation and testing tools.

Security-Aware Documentation

Our Scalar configuration includes built-in authentication support. The Bearer token authentication is pre-configured, so developers can easily test secured endpoints. In development environments, this dramatically reduces the friction involved in API testing and exploration.

The environment-aware titles ({app.Environment.EnvironmentName} - {app.Environment.ApplicationName}) help developers immediately understand which environment and service they're working with, reducing confusion when working across multiple environments.

Strategic Documentation Deployment

We only expose Scalar documentation in non-production environments, maintaining security best practices while maximizing developer productivity:

if (!app.Environment.IsProduction())
{
    app.ConfigureOpenApiAndScaler();
}

This approach ensures that our production endpoints remain secure while development and staging environments provide rich documentation experiences for both internal teams and external API consumers.

OpenAPI: The Contract-First Approach

OpenAPI serves as the foundation for both our documentation and our inter-service communication strategy. By leveraging .NET 9's built-in OpenAPI support, we ensure that our API contracts are automatically generated, accurate, and versioned.

API Versioning Strategy

Our solution implements comprehensive API versioning through the Asp.Versioning libraries. This allows us to evolve our APIs while maintaining backward compatibility for existing consumers.

builder.Services.AddApiVersioning(options =>
{
    options.DefaultApiVersion = new ApiVersion(primaryVersion, 0);
    options.ReportApiVersions = true;
    options.AssumeDefaultVersionWhenUnspecified = true;
    options.ApiVersionReader = ApiVersionReader.Combine(
        new UrlSegmentApiVersionReader(),
        new HeaderApiVersionReader("x-api-version"),
        new QueryStringApiVersionReader("api-version")
    );
});

This configuration supports multiple versioning strategies—URL segments, headers, and query strings—giving API consumers flexibility in how they specify version requirements. The ReportApiVersions setting ensures that API responses include version information, helping consumers understand what versions are available.

Code Generation and Type Safety

While not explicitly shown in our current codebase, our OpenAPI specifications enable powerful code generation scenarios. Teams can generate strongly-typed HTTP clients using tools like Refit (which we include in our shared project) or NSwag, ensuring that inter-service communication is type-safe and automatically stays in sync with API changes.

The presence of Refit in our dependencies suggests we're using this approach for HTTP client generation:

<PackageReference Include="Refit" Version="8.0.0" />
<PackageReference Include="Refit.HttpClientFactory" Version="8.0.0" />

This enables us to define HTTP APIs as C# interfaces and let Refit handle the HTTP implementation details, creating a more maintainable and less error-prone approach to service communication.

Architectural Integration: How It All Works Together

The true power of our observability and documentation strategy emerges from how these technologies work together to support our microservices architecture.

Service-to-Service Communication

When a request flows through our system, OpenTelemetry automatic instrumentation creates a distributed trace that spans multiple services. If our User Service calls the Profile Service, which then calls the Image Service, we get a complete trace showing the entire request flow, including timing information for each service call.

The OpenAPI specifications for each service define clear contracts, while Refit-generated clients provide type-safe communication. If the Profile Service needs to call the User Service, it uses a strongly-typed interface generated from the User Service's OpenAPI specification.

Development Workflow

During development, our setup provides an exceptional developer experience. Developers can:

1. Explore APIs using Scalar's beautiful interface, testing endpoints directly from the documentation
2. Monitor request flows using local OpenTelemetry exporters to understand how their changes affect the broader system
3. Debug issues using correlated logs and traces that show exactly what happened during a specific request
4. Ensure compatibility by testing against versioned APIs that clearly communicate breaking changes

Production Operations

In production, our observability stack provides operations teams with the insights they need to maintain system reliability:

• Azure Monitor integration provides enterprise-grade alerting, dashboards, and analysis capabilities
• Distributed tracing enables rapid root cause analysis when issues occur across service boundaries
• Comprehensive metrics support proactive monitoring and capacity planning
• Structured logging with trace correlation makes troubleshooting specific customer issues straightforward

Security and Compliance

Our ActionControllerLogging filter creates detailed audit trails for compliance requirements, while the audit data flows through Azure Service Bus to ensure reliable delivery and processing. The combination of OpenTelemetry traces and custom audit logs provides both technical observability and business-level audit capabilities.

Performance and Overhead Considerations

Implementing comprehensive observability always raises questions about performance impact. OpenTelemetry is designed with performance in mind, using efficient sampling strategies and asynchronous export mechanisms to minimize impact on application performance.

Our instrumentation focuses on high-value, low-overhead data collection. ASP.NET Core instrumentation adds minimal latency to request processing, while HTTP client instrumentation helps us identify slow downstream dependencies. Runtime instrumentation provides valuable insights into garbage collection and memory usage with negligible overhead.

The key is selective instrumentation—we capture the telemetry that provides actionable insights while avoiding over-instrumentation that could impact performance or create overwhelming amounts of data.

Future Enhancements and Roadmap

Looking ahead, our observability and documentation strategy continues to evolve. Some areas we're exploring include:

Enhanced Custom Metrics: Adding business-specific metrics that help us understand not just technical performance but business outcomes.

Improved Documentation: Exploring ways to include more context in our API documentation, such as usage examples and integration guides.

Advanced Tracing: Investigating custom span creation for critical business processes to provide even more detailed insights into application behavior.

Multi-Environment Correlation: Developing strategies to correlate telemetry across development, staging, and production environments to better understand how code changes affect system behavior.

Conclusion

The integration of OpenTelemetry, Scalar, and OpenAPI in our .NET 9 solution creates a powerful foundation for building and operating modern microservices. OpenTelemetry provides vendor-neutral observability that scales with our application, Scalar delivers beautiful, developer-friendly API documentation, and OpenAPI ensures our service contracts are clear, versioned, and automatically maintained.

This combination supports both development velocity and operational excellence. Developers get the tools they need to understand and test APIs, while operations teams get the observability data required to maintain reliable, performant systems.

Most importantly, this approach is future-proof. By embracing open standards like OpenTelemetry and OpenAPI, we avoid vendor lock-in while still benefiting from excellent tooling and integrations. As our architecture evolves and new requirements emerge, our observability and documentation strategy can adapt without requiring fundamental changes.

The investment in proper observability and documentation pays dividends throughout the application lifecycle—from initial development through production operations to long-term maintenance and evolution. In today's fast-paced development environment, these capabilities aren't just helpful; they're essential for building systems that teams can confidently develop, deploy, and operate at scale.

Understanding GitHub Copilot in 2025

GitHub Copilot has evolved significantly since its initial release, becoming an indispensable AI pair programmer that understands context, suggests complete functions, and helps developers navigate complex codebases. In 2025, Copilot offers a suite of powerful features specifically optimized for cloud-native development workflows.

Key Features Enhancing .NET Development

GitHub Copilot now offers specialized capabilities for .NET developers, including:

1. Intelligent Code Completion: Beyond simple autocompletion, Copilot now understands ASP.NET Core 9's architecture and can suggest entire controller methods, middleware configurations, and service registrations based on your project context.
2. Copilot in CLI: Get real-time assistance directly in your terminal for command suggestions and explanations when working with the dotnet CLI, Docker commands, or Kubernetes configurations.
3. Application Modernization: The dedicated Visual Studio extension helps upgrade legacy .NET projects to newer versions, update dependencies, and apply code fixes automatically, making migration to ASP.NET Core 9 significantly smoother.
4. .NET Aspire Integration: With the release of .NET Aspire 9.3, GitHub Copilot provides specialized assistance for cloud-native application development, including orchestrated container deployments and service discovery configurations.
5. Real-time Documentation: Copilot can generate and explain documentation for complex microservice architectures, saving valuable development time while ensuring knowledge transfer across teams.

ASP.NET Core 9: Cloud-Ready Foundation for Microservices

ASP.NET Core 9 brings substantial improvements specifically designed for cloud-native and microservices development:

Cloud-Native Features

1. Enhanced Performance: ASP.NET Core 9 offers improved request processing throughput, making it ideal for high-load microservice scenarios.
2. Built-in Security Enhancements: Advanced authorization policies, improved data protection APIs, and better integration with cloud identity providers streamline secure microservices deployment.
3. Minimal APIs Advancement: Further refinements to the minimal API approach make creating lightweight microservices even more straightforward and maintainable.
4. Cloud Integration: Native integration with major cloud providers simplifies deployment and management of containerized applications.
5. Cross-Platform Excellence: True platform independence ensures consistent behavior across Windows, Linux, and cloud environments.

Leveraging GitHub Copilot for Microservices Architecture

Microservices architecture continues to be the preferred approach for building scalable, maintainable cloud applications. Here's how GitHub Copilot can be leveraged effectively in this context:

DO: Use Copilot for Service Interface Design

When designing service interfaces, Copilot excels at suggesting consistent API patterns based on your existing codebase:

// Simply start defining your controller and let Copilot suggest the rest
[ApiController]
[Route("api/[controller]")]
public class OrderController : ControllerBase
{
    private readonly IOrderService _orderService;

    public OrderController(IOrderService orderService)
    {
        _orderService = orderService;
    }

    // Copilot will suggest CRUD operations with appropriate status codes,
    // validation, error handling, and documentation
}

Copilot will understand your domain model and suggest appropriate endpoints, status codes, and error handling approaches consistent with your existing services.

DON'T: Rely on Copilot for Critical Business Logic Without Review

While Copilot is excellent at generating boilerplate code and common patterns, critical business logic should always be reviewed carefully:

// DON'T blindly accept complex business logic suggestions
public async Task<decimal> CalculateOrderTotal(OrderRequest request)
{
    // Copilot might suggest a calculation that looks reasonable
    // but misses your specific business rules or edge cases
    // ALWAYS review these implementations carefully!
}

Always review business logic suggestions to ensure they align with your specific requirements, especially for financial calculations, security-critical operations, or complex domain rules.

DO: Use Copilot for Configuration and Middleware Setup

ASP.NET Core 9's middleware pipeline and service configuration can be complex. Copilot excels at suggesting appropriate middleware ordering and configuration:

// Start typing your Program.cs setup and Copilot will help
var builder = WebApplication.CreateBuilder(args);

builder.Services.AddControllers();
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

// Copilot can suggest appropriate service registrations
// for your microservices, including health checks, monitoring,
// and distributed tracing
builder.Services.AddHealthChecks()
    .AddCheck<DatabaseHealthCheck>("database")
    .AddCheck<ExternalApiHealthCheck>("external-api");

// Copilot will suggest appropriate middleware ordering
var app = builder.Build();

if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
}

app.UseHttpsRedirection();
app.UseAuthorization();
app.MapControllers();
app.MapHealthChecks("/health");

DON'T: Generate Complex Container Orchestration Without Understanding

Container orchestration with Docker Compose or Kubernetes manifests is an area where Copilot can help, but blindly accepting suggestions can lead to problems:

# Copilot can generate Docker Compose files, but ensure you understand
# the networking, volume mounts, and resource allocations
services:
  api:
    image: yourregistry/api:latest
    ports:
      - "8080:80"
    environment:
      - ConnectionStrings__DefaultConnection=Server=db;Database=master;User=sa;Password=${DB_PASSWORD}
    depends_on:
      - db
  # Review configurations thoroughly before deployment

Always understand the infrastructure implications of generated configurations, especially regarding security, networking, and resource allocation.

Best Practices for Microservices Development with Copilot

1. Service Boundary Definition

Use Copilot to help identify and maintain clear service boundaries:

// DO: Let Copilot help with defining clear service contracts
namespace OrderService.Contracts
{
    public interface IOrderService
    {
        // Copilot can suggest a well-defined service interface
        // with appropriate DTOs and operation contracts
    }
}

Copilot can analyze your domain model and suggest appropriate service boundaries, but always ensure these align with your business domains and team structures.

2. Communication Patterns

GitHub Copilot excels at implementing various communication patterns required for microservices:

// DO: Use Copilot to implement communication patterns
// Synchronous HTTP communication
public class OrderClient
{
    private readonly HttpClient _httpClient;

    // Copilot can generate resilient HTTP client implementations
    // with appropriate retry policies, circuit breakers, etc.
}

// Message-based communication
public class OrderCreatedEventHandler : IEventHandler<OrderCreatedEvent>
{
    // Copilot can suggest appropriate message handling patterns
    // based on your chosen messaging infrastructure
}

3. Data Consistency Strategies

Microservices often require sophisticated data consistency strategies. Copilot can help implement patterns like:

// Saga pattern implementation
public class OrderProcessingSaga : ISaga<OrderProcessingState>
{
    // Copilot can suggest appropriate saga steps, compensation actions,
    // and state management approaches
}

// Outbox pattern for reliable message publishing
public class OrderRepository : IOrderRepository
{
    public async Task CreateOrder(Order order)
    {
        using var transaction = await _dbContext.Database.BeginTransactionAsync();
        // Copilot can help implement the transactional outbox pattern
        // to ensure reliable message publishing with database transactions
    }
}

While Copilot provides good starting points, always review these implementations carefully as they're critical to system reliability.

Cloud Deployment and DevOps Integration

GitHub Copilot shines in streamlining cloud deployments and DevOps practices for ASP.NET Core 9 microservices:

DO: Generate Infrastructure as Code Templates

Copilot can help create comprehensive infrastructure definitions:

// DO: Use Copilot to generate Pulumi/Terraform/ARM templates
// Example Pulumi C# code for deploying a microservice to Azure
public class InfrastructureStack : Stack
{
    public InfrastructureStack()
    {
        var resourceGroup = new ResourceGroup("myResourceGroup", new ResourceGroupArgs
        {
            Location = "EastUS"
        });

        // Copilot can generate appropriate infrastructure definitions
        // for your specific cloud provider and requirements
    }
}

DON'T: Ignore Security Best Practices in Generated Configurations

Copilot might generate code that works but doesn't follow security best practices:

# DON'T: Accept security-sensitive configurations without review
apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
data:
  # Copilot might suggest hardcoded credentials or insecure practices
  # ALWAYS review security-sensitive configurations
  username: YWRtaW4=  # base64 encoded "admin"
  password: cGFzc3dvcmQxMjM=  # base64 encoded "password123"

Always ensure generated infrastructure code follows security best practices, especially regarding secrets management, network security, and access controls.

Testing Microservices with Copilot

GitHub Copilot can significantly accelerate test creation:

DO: Use Copilot for Test Case Generation

// DO: Let Copilot suggest test cases based on your implementation
[Fact]
public async Task GetOrder_WithValidId_ReturnsOrder()
{
    // Arrange
    var orderId = Guid.NewGuid();
    var expectedOrder = new Order { Id = orderId, Customer = "Test Customer" };
    _orderRepositoryMock.Setup(repo => repo.GetOrderByIdAsync(orderId))
        .ReturnsAsync(expectedOrder);

    // Act
    var result = await _controller.GetOrder(orderId);

    // Assert
    var okResult = Assert.IsType<OkObjectResult>(result.Result);
    var returnedOrder = Assert.IsType<OrderDto>(okResult.Value);
    Assert.Equal(orderId, returnedOrder.Id);
    Assert.Equal("Test Customer", returnedOrder.Customer);
}

Copilot can analyze your implementation and suggest comprehensive test cases covering various scenarios, including edge cases and error conditions.

DON'T: Skip Integration Testing of Microservice Communication

While unit tests are important, microservices require thorough integration testing:

// DO: Use Copilot to help write integration tests
[Fact]
public async Task OrderAPI_Integration_CreatesOrderAndNotifiesShipping()
{
    // Arrange
    await using var application = new WebApplicationFactory<Program>();
    var client = application.CreateClient();

    // Act & Assert
    // Copilot can help generate comprehensive integration test scenarios
    // that verify cross-service communication works correctly
}

Copilot can help generate test fixtures and assertions, but ensure your tests cover the critical integration points between services.

Performance Optimization and Monitoring

ASP.NET Core 9 offers advanced performance features that can be leveraged with Copilot:

DO: Use Copilot for Implementing Performance Monitoring

// DO: Let Copilot help implement monitoring and telemetry
public void ConfigureServices(IServiceCollection services)
{
    // Copilot can suggest appropriate monitoring configurations
    services.AddOpenTelemetry()
        .WithTracing(builder => builder
            .AddSource("OrderService")
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddSqlClientInstrumentation()
            .AddOtlpExporter())
        .WithMetrics(builder => builder
            .AddMeter("Microsoft.AspNetCore.Hosting")
            .AddMeter("Microsoft.AspNetCore.Server.Kestrel")
            .AddOtlpExporter());
}

DON'T: Over-Optimize Prematurely

Copilot might suggest aggressive optimization techniques that aren't necessary:

// DON'T: Implement complex optimizations without measuring first
public async Task<IEnumerable<OrderSummary>> GetRecentOrders()
{
    // Copilot might suggest complex caching strategies or query optimizations
    // that add unnecessary complexity before you've identified a performance issue

    // Start with simple, readable implementations and optimize based on actual metrics
    return await _orderRepository.GetRecentOrdersAsync();
}

Always measure performance before applying complex optimizations, even when Copilot suggests them.

Error Handling and Resilience

Modern cloud applications need robust error handling and resilience patterns:

DO: Use Copilot for Implementing Resilience Patterns

// DO: Let Copilot help implement resilience patterns
public class OrderService : IOrderService
{
    private readonly IHttpClientFactory _clientFactory;

    public async Task<OrderResult> CreateOrder(OrderRequest request)
    {
        // Copilot can suggest appropriate resilience patterns
        var client = _clientFactory.CreateClient("shipping-service");

        var policy = Policy
            .Handle<HttpRequestException>()
            .OrResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
            .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));

        var response = await policy.ExecuteAsync(() => client.PostAsJsonAsync("/api/shipments", request));
        // Handle response appropriately
    }
}

DON'T: Ignore Proper Error Propagation

Copilot might not always suggest the most appropriate error handling strategy:

// DON'T: Accept simplistic error handling suggestions
public async Task<IActionResult> ProcessPayment(PaymentRequest request)
{
    try
    {
        // Process payment
        return Ok();
    }
    catch (Exception ex)
    {
        // Copilot might suggest overly broad exception handling
        // that masks important errors or doesn't provide enough context
        return StatusCode(500, "An error occurred");
    }
}

// DO: Implement proper error handling with appropriate details and logging
public async Task<IActionResult> ProcessPayment(PaymentRequest request)
{
    try
    {
        // Process payment
        return Ok();
    }
    catch (PaymentProviderException ex)
    {
        _logger.LogWarning(ex, "Payment provider error for order {OrderId}", request.OrderId);
        return StatusCode(503, new ProblemDetails
        {
            Title = "Payment service temporarily unavailable",
            Detail = "The payment couldn't be processed due to a temporary issue with the payment provider",
            Status = 503
        });
    }
    catch (Exception ex)
    {
        _logger.LogError(ex, "Unexpected error processing payment for order {OrderId}", request.OrderId);
        return StatusCode(500, new ProblemDetails
        {
            Title = "Internal server error",
            Detail = "An unexpected error occurred processing your payment",
            Status = 500
        });
    }
}

Security Considerations

Security is paramount in cloud-based microservices. GitHub Copilot can both help and hinder:

DO: Use Copilot for Security Best Practices Implementation

// DO: Let Copilot help implement security best practices
public void ConfigureServices(IServiceCollection services)
{
    // Copilot can suggest appropriate security configurations
    services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
        .AddJwtBearer(options =>
        {
            options.Authority = Configuration["Authentication:Authority"];
            options.Audience = Configuration["Authentication:Audience"];
            options.TokenValidationParameters = new TokenValidationParameters
            {
                ValidateIssuer = true,
                ValidateAudience = true,
                ValidateLifetime = true,
                ValidateIssuerSigningKey = true
            };
        });

    services.AddAuthorization(options =>
    {
        options.AddPolicy("OrdersRead", policy =>
            policy.RequireClaim("scope", "orders:read"));
        options.AddPolicy("OrdersWrite", policy =>
            policy.RequireClaim("scope", "orders:write"));
    });
}

DON'T: Accept Generated Security-Critical Code Without Review

// DON'T: Blindly accept security-critical code
public class PasswordHasher : IPasswordHasher
{
    // Copilot might suggest insecure practices or outdated algorithms
    public string HashPassword(string password)
    {
        // NEVER accept security-critical code without careful review
        // and alignment with current best practices
    }
}

Always have security-critical code reviewed, regardless of whether it was generated by Copilot.

Conclusion

GitHub Copilot represents a transformative tool for modern cloud development with ASP.NET Core 9 and microservices. When used judiciously, it can dramatically accelerate development while maintaining code quality. The key to success lies in understanding where Copilot excels (boilerplate, patterns, configuration) and where human expertise remains essential (security, business logic, architecture).

By following the best practices outlined in this guide, developers can harness the power of AI-assisted coding while avoiding common pitfalls. As cloud-native development continues to evolve, the partnership between human developers and tools like GitHub Copilot will become increasingly symbiotic, enabling teams to deliver sophisticated microservice architectures with greater efficiency and quality than ever before.

The future of cloud development is collaborative—between humans and AI—and GitHub Copilot is at the forefront of this revolution, particularly for ASP.NET Core 9 and modern microservices architectures. Embrace this powerful tool, but do so thoughtfully, and you'll unlock new levels of productivity in your cloud development journey.

.NET has evolved dramatically over the past decade. With .NET 9 and ASP.NET Core 9, Microsoft’s flagship platform has doubled down on performance, cloud-native microservices, and developer productivity. But as the tools get more powerful, the architectures we choose to build with them matter more than ever.

If you’re a developer or architect trying to keep your codebases clean, maintainable, and adaptable in a world of microservices, buzzwords like Domain-Driven Design (DDD), Onion Architecture, and Clean Architecture are probably familiar. But how do these ideas actually fit together? How do you put them into practice in a real ASP.NET Core 9 solution? And how do you ensure your system stays robust as it scales?

In this post, we’ll break all that down. We’ll look at how Clean Architecture, Onion Architecture, and Domain-Driven Design complement each other — and how you can implement these principles in an ASP.NET Core 9 microservices solution that’s easy to reason about, test, and evolve.

By the end, you’ll have a practical roadmap to help you structure your next enterprise-ready .NET 9 application.

Why Architecture Still Matters

The .NET 9 ecosystem is rich with features: minimal APIs, powerful performance optimizations, improved cloud-native tooling, and first-class support for containerized deployments and orchestration.

But even the best framework can’t save a poorly structured application.

Modern business requirements change fast. You might need to pivot your domain logic, adapt to new integrations, split services apart, or rewrite pieces without taking the whole system down.

If your code is tangled up — if it’s just “controller spaghetti” directly tied to your database models — you’ll hit walls fast. That’s where proper architecture comes in.

Clean Architecture and Onion Architecture are design patterns that help you organize your solution in a way that:

• Puts your business logic at the center, protected from external concerns.
• Makes dependencies flow in one direction — inward.
• Keeps infrastructure details (like databases, web APIs, message queues) on the outer layers, easy to swap or replace.
• Encourages strong boundaries and separation of concerns.

When you pair this with Domain-Driven Design, you also ensure that your code reflects the real business domain — the real problems you’re solving — instead of just being a CRUD wrapper around tables.

The Microservices Landscape

The platform consists of multiple interdependent yet decoupled microservices, each responsible for a specific business domain:

// From AppHost/Program.cs - Our microservices orchestration
var token = builder.AddProject<Projects.Networks_Token>("tokenservice");

var buddy = builder.AddProject<Projects.Networks_BuddyService>("buddyservice")
    .WithReference(cache).WaitFor(cache)
    .WithReference(token).WaitFor(token)
    .WithReference(seq).WaitFor(seq);

var profile = builder.AddProject<Projects.Networks_ProfileService>("profileservice")
    .WithReference(cache).WaitFor(cache)
    .WithReference(token).WaitFor(token)
    .WithReference(seq).WaitFor(seq);

// Additional services...

This structure allows us to scale, deploy, and maintain each service independently. Let's explore how DDD and Onion Architecture principles guide the design of each service.

Domain-Driven Design: Aligning Code with Business Reality

Domain-Driven Design puts the focus on the core domain and domain logic, mapping business concepts to code structures. In the solution, each microservice represents a bounded context - a logical boundary around a specific domain model.

Ubiquitous Language

A cornerstone of DDD is establishing a common language shared between developers and domain experts. In our codebase, this is evident in how our domain models reflect business concepts:

// Domain models that reflect our business language
public class ProfileDomainModel
{
    public Guid Id { get; set; }
    public Guid Guid { get; set; }
    public string UserName { get; set; }
    public string NormalizedUserName { get; set; }
    public string Email { get; set; }
    public Site Site { get; set; }
    public string Location { get; set; }
    public int? LocationId { get; set; }
    public bool IsOnline { get; set; }
    public DateTime? LastOnline { get; set; }
    // Additional properties...
}

This domain model captures the essence of a user profile in our system, using terms that business stakeholders understand.

Bounded Contexts

Each microservice maintains its own bounded context. For example, the ProfileService manages everything related to user profiles, while the BuddyService handles relationships between users. These services communicate through well-defined interfaces, preserving context boundaries.

Onion Architecture: Dependencies Point Inward

Onion Architecture (also known as Clean Architecture or Hexagonal Architecture) organizes code in concentric layers, with domain models at the center. Dependencies always point inward, ensuring that the core business logic remains independent of infrastructure concerns.

The Domain Layer: At the Core

The domain layer represents the business concepts and rules:

// Domain interfaces define the contract without implementation details
public interface IIntegrationData
{
    Task<ProfileDomainModel> FetchProfileAsync(Guid userGuid, Site site);
    Task<List<ProfileDomainModel>> FetchProfilListAsync(List<Guid> userGuids, Site site);
    Task<List<HoneyPotDomainModel>> GetHoneyPotAsync(Guid userGuid, Site site);
    Task MarkUserAsLoggedOffAsync(Guid userGuid, Site site);
    Task<ProfileDomainModel> UpsertDefaultProfileAsync(string userName, string email, Site site);
}

These interfaces define what the system needs from a business perspective, without specifying how these operations are implemented.

The Data Layer: Infrastructure Concerns

The data layer implements the domain interfaces, connecting to databases and other external systems:

public class IntegrationData : IIntegrationData
{
    private readonly ProfileDbContext _context;

    public IntegrationData(ProfileDbContext context)
    {
        _context = context;
    }

    public Task<ProfileDomainModel> FetchProfileAsync(Guid userGuid, Site site)
    {
        return _context.Profiles
            .Where(up => up.Guid == userGuid && up.Site == site)
            .FirstOrDefaultAsync();
    }

    // Other implementation methods...
}

Here, the data layer depends on domain interfaces but implements them using Entity Framework Core. The domain layer remains completely unaware of these implementation details.

The Business Layer: Application Services

The business layer orchestrates operations and enforces business rules:

public class IntegrationBusiness : IIntegrationBusiness
{
    private readonly IIntegrationData _data;

    public IntegrationBusiness(IIntegrationData data)
    {
        _data = data;
    }

    public async Task<ProfileServiceModel> FetchProfileAsync(Guid userGuid, Site site)
    {
        var domain = await _data.FetchProfileAsync(userGuid, site);
        if (domain == null)
        {
            throw new ProfileNotFoundException($"Profile not found for userGuid: {userGuid} on site: {site}");
        }
        return domain.MapToServiceModel();
    }

    // Other business methods...
}

The business layer depends on abstractions from both the domain and data layers, but never on concrete implementations. This ensures flexibility and testability.

The Facade Pattern: Simplifying Interactions

One key architectural choice in solution is the extensive use of the Facade pattern. The Facade sits between the business layer and the API layer, providing a simplified interface for clients and handling cross-cutting concerns:

public class IntegrationFacade : IIntegrationFacade
{
    private readonly IIntegrationBusiness _business;
    private readonly CacheAside _cache;

    public IntegrationFacade(IIntegrationBusiness business, IDistributedCache cache)
    {
        _business = business;
        _cache = new CacheAside(cache);
    }

    public async Task<Result<ProfileServiceModel>> FetchProfileAsync(Guid userGuid, Site site)
    {
        try
        {
            if (userGuid == Guid.Empty || !Enum.IsDefined(typeof(Site), site))
                return Result<ProfileServiceModel>.Failure(Error.InvalidInput, 
                    new List<string> { "Invalid userGuid or site." });

            string cacheKey = $"profile_{userGuid}_{site}";
            var data = await _cache.FetchFromCache<ProfileServiceModel>(cacheKey, async () =>
            {
                return await _business.FetchProfileAsync(userGuid, site);
            }, new DistributedCacheEntryOptions() { AbsoluteExpiration = DateTime.UtcNow.AddHours(24) });

            return Result<ProfileServiceModel>.Success(data);
        }
        catch (ProfileNotFoundException)
        {
            return Result<ProfileServiceModel>.Failure(Error.NotFound, 
                new List<string> { "Profile not found." });
        }
    }

    // Other facade methods...
}

The Facade pattern offers several benefits:

1. Centralized Input Validation: The Facade layer validates inputs before passing them to the business layer.
2. Standardized Error Handling: Exceptions are caught and converted into consistent response formats.
3. Cross-Cutting Concerns: Caching, logging, and other aspects are managed here.
4. Simplified API: Controllers interact with a clean, consistent interface.

Dependency Injection: Wiring It All Together

.NET 9's built-in dependency injection container manages the instantiation and lifetime of our components:

void AddScopes(WebApplicationBuilder builder)
{
    builder.Services.AddScoped<IIntegrationFacade, IntegrationFacade>();
    builder.Services.AddScoped<IIntegrationBusiness, IntegrationBusiness>();
    builder.Services.AddScoped<IIntegrationData, IntegrationData>();
    builder.Services.AddScoped<IProfileFacade, ProfileFacade>();
    builder.Services.AddScoped<IProfileBusiness, ProfileBusiness>();
    builder.Services.AddScoped<IProfileData, ProfileData>();
    builder.Services.AddScoped<IHoneyPotFacade, HoneyPotFacade>();
    builder.Services.AddScoped<IHoneyPotBusiness, HoneyPotBusiness>();
    builder.Services.AddScoped<IHoneyPotData, HoneyPotData>();
    builder.Services.AddScoped<IServiceBusSender, ServiceBusSender>();
    // Additional registrations...
}

This configuration allows for flexible composition while maintaining the dependency inversion principle. Higher-level components depend on abstractions, not concrete implementations.

API Layer: Thin Controllers

With our layered architecture in place, controllers remain remarkably thin:

[ApiController]
[Route("api/buddy/v{v:apiVersion}/[controller]")]
[ApiVersion(1)]
public class IntegrationController : ControllerBase
{
    private readonly IIntegrationFacade _facade;

    public IntegrationController(IIntegrationFacade facade)
    {
        _facade = facade;
    }

    [HttpGet("{userGuid:guid}/{site:int}")]
    public async Task<IActionResult> GetBuddies(Guid userGuid, int site)
    {
        var result = await _facade.GetBuddies(userGuid, (Site)site);
        return result.Match();
    }

    // Other endpoint methods...
}

The controllers simply map HTTP requests to facade calls and convert the results to HTTP responses, without any business logic.

Inter-Service Communication

In a microservices architecture, services need to communicate. We implement two primary methods:

HTTP Clients with Refit

For synchronous communication, we use typed HTTP clients with Refit:

void AddHttpClients(WebApplicationBuilder builder)
{
    builder.Services.AddRefitClient<IProfileHttpClient>()
        .ConfigureHttpClient(c => c.BaseAddress = new Uri(
            builder.Environment.EnvironmentName != "LocalDevelopment" 
                ? builder.Configuration["profileurl"] 
                : "https+http://profileservice"))
        .AddHttpMessageHandler(() => new AuthHeaderHandler(builder.Services.BuildServiceProvider()));

    // Additional HTTP clients...
}

These strongly-typed clients ensure type safety when communicating between services.

Service Bus for Asynchronous Communication

For event-based communication, we leverage Azure Service Bus:

builder.Services.AddSingleton<ServiceBusClient>(sp =>
    new ServiceBusClient(builder.Configuration.GetConnectionString("servicebus")));

This facilitates loosely coupled, asynchronous interactions between services.

Infrastructure as Code with .NET Aspire

Our solution leverages .NET Aspire to define and orchestrate the entire microservices ecosystem:

var builder = DistributedApplication.CreateBuilder(args);

// Infrastructure components
var cache = builder.AddRedis("cache")
    .WithLifetime(ContainerLifetime.Persistent)
    .WithHostPort(6379)
    .WithRedisInsight()
    .WithRedisCommander();

// Service definitions with dependencies
var profile = builder.AddProject<Projects.Networks_ProfileService>("profileservice")
    .WithReference(cache).WaitFor(cache)
    .WithReference(token).WaitFor(token)
    .WithReference(seq).WaitFor(seq);

This declarative approach ensures consistent deployment and proper dependency management.

Cross-Cutting Concerns

Several aspects span multiple services and layers:

Caching

The Facade layer implements a cache-aside pattern to improve performance:

var data = await _cache.FetchFromCache<BuddyListServiceModel>(cacheKey, async () =>
{
    return await _business.GetBuddies(userGuid, site);
}, new DistributedCacheEntryOptions() { AbsoluteExpiration = DateTime.UtcNow.AddHours(24) });

Authentication and Authorization

Security is managed consistently across services:

void AddAuthentication(WebApplicationBuilder builder)
{
    builder.Services.AddAuthentication(options =>
    {
        options.DefaultAuthenticateScheme = JwtBearerDefaults.AuthenticationScheme;
        options.DefaultChallengeScheme = JwtBearerDefaults.AuthenticationScheme;
    })
    .AddJwtBearer(options =>
    {
        var key = Encoding.UTF8.GetBytes(
            builder.Configuration["Jwt:Key"];
            
        options.TokenValidationParameters = new TokenValidationParameters
        {
            ValidateIssuerSigningKey = true,
            IssuerSigningKey = new SymmetricSecurityKey(key),
            ValidateIssuer = true,
            ValidIssuer = builder.Configuration["Jwt:Issuer"],
            ValidateAudience = true,
            ValidAudience = builder.Configuration["Jwt:Audience"],
            ValidateLifetime = true,
            ClockSkew = TimeSpan.Zero
        };
    });
}

Logging and Monitoring

OpenTelemetry integration provides comprehensive observability:

public static TBuilder ConfigureOpenTelemetry<TBuilder>(this TBuilder builder) 
    where TBuilder : IHostApplicationBuilder
{
    builder.Logging.AddOpenTelemetry(logging =>
    {
        logging.IncludeFormattedMessage = true;
        logging.IncludeScopes = true;
    });

    builder.Services.AddOpenTelemetry()
        .WithMetrics(metrics =>
        {
            metrics.AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddRuntimeInstrumentation();
        })
        .WithTracing(tracing =>
        {
            tracing.AddSource(builder.Environment.ApplicationName)
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation();
        });

    // Configure exporters...
    return builder;
}

Benefits of Our Architectural Approach

This DDD + Onion Architecture approach delivers numerous advantages:

1. Testability

Since business logic depends only on abstractions, unit testing becomes straightforward:

[TestMethod]
public async Task AssociateUserAsync_Valid_CallsDataAndProfile()
{
    // Arrange
    var model = new AssociateViewModel { UserName = userName, Level = level };
    _userDataMock.Setup(d => d.AssociateUserAsync(userName, level, site))
        .ReturnsAsync(email);
    _profileMock.Setup(p => p.UpsertDefaultProfile(userName, email, (int)site))
        .ReturnsAsync(Result<ProfileServiceModel>.Success(
            new ProfileServiceModel { Guid = Guid.NewGuid() }));

    // Act
    await _business.AssociateUserAsync(model, site);

    // Assert
    _userDataMock.Verify(d => d.AssociateUserAsync(userName, level, site), Times.Once);
    _profileMock.Verify(p => p.UpsertDefaultProfile(userName, email, (int)site), Times.Once);
}

2. Maintainability

The clear separation of concerns makes the codebase more maintainable. Developers can understand and modify specific parts of the system without needing to comprehend the entire system.

3. Flexibility

Implementation details can change without affecting the core business logic:

• Switching from Cosmos DB to SQL Server would only affect the Data layer
• Changing from REST APIs to gRPC would only impact the API layer
• Updating caching strategies would be isolated to the Facade layer

4. Scalability

Each microservice can scale independently based on its specific load characteristics, optimizing resource usage.

Challenges and Solutions

While the benefits are significant, this architecture isn't without challenges:

1. Development Complexity

Challenge: More layers and abstractions mean more code to write initially.

Solution: We established clear templates and conventions to streamline development and maintain consistency.

2. Distributed Data Management

Challenge: With each service owning its data, ensuring consistency across services is complex.

Solution: We employ event-driven patterns for eventual consistency and carefully design service boundaries to minimize cross-service transactions.

3. Performance Overhead

Challenge: Additional layers and service boundaries can introduce performance overhead.

Solution: Strategic caching in the Facade layer, connection pooling, and efficient serialization mitigate these concerns.

Conclusion: Enabling Business Agility Through Technical Excellence

Our DDD + Onion Architecture approach has enabled our client to grow and evolve rapidly while maintaining code quality and developer productivity. By ensuring that our code structure reflects business realities and maintains clean separation of concerns, we've created a system that's both technically sound and business-aligned.

The combination of Domain-Driven Design, Onion Architecture, and the Facade pattern has proven particularly powerful. It allows domain experts to contribute meaningfully to the design, helps developers understand the business context, and maintains clean boundaries between different aspects of the system.

As the platform continues to grow, this architectural foundation will enable us to adapt to changing requirements, scale individual components as needed, and incorporate new technologies without destabilizing the entire system. The result is a microservices ecosystem that delivers business value consistently while remaining technically maintainable and evolvable.

In an era where software must continuously adapt to changing business needs, investing in solid architectural principles isn't just a technical concern—it's a strategic business advantage. Our experience with this project demonstrates that Domain-Driven Design, Onion Architecture, and clean separation of concerns provide a robust foundation for building complex, scalable, and maintainable microservices systems in .NET 9.

At Architect4Hire, we’ve been pushing the limits of what ASP.NET Core 9 and .NET Aspire can do, and our latest framework implementation for BuddyNetworks is a showcase of modern software architecture done right.

From microservice orchestration to clean separation of concerns, it’s the kind of stuff that makes devs sleep better at night and apps hum in production.

🧱 Building Blocks: Multi-Layer Architecture, the Real Way

At the heart of the Architect4Hire framework is a well-structured multi-layer architecture that enforces a strict separation of concerns. Each layer is responsible for a focused set of responsibilities. The Presentation Layer handles HTTP requests, routing, and authentication. The Domain Layer is subdivided into facade, business, and data sublayers—each encapsulating orchestration, core logic, and persistence, respectively. The Shared Layer houses reusable utilities and enums, and implements a Result pattern to standardize error handling. This clear architectural design improves maintainability, enhances testability, and keeps teams productive even as the codebase scales. This structure is the foundation for building reliable and adaptable systems.

Example controller:

[Authorize]
[ApiController]
[Route("api/v{v:apiVersion}/[controller]")]
[ApiVersion(1)]
public class UserController : ControllerBase
{
    private readonly IUserFacade _userFacade;

    public UserController(IUserFacade userFacade)
    {
        _userFacade = userFacade;
    }

    [HttpPost("register")]
    public async Task<IActionResult> RegisterUser([FromBody] RegisterViewModel model, [FromHeader] int site)
    {
        var result = await _userFacade.RegisterUserAsync(model, site);
        return result.Match();
    }
}

🔁 SOA + Aspire = Clean, Scalable Microservices

The framework is built around a service-oriented architecture (SOA) where each domain is represented by its own microservice. These services, such as UserService, AdminService, and ProfileService, communicate via REST and Azure Service Bus. The .NET Aspire project plays a critical role in service discovery, observability, and orchestration. Aspire allows each service to be discovered dynamically and monitored with health checks and distributed tracing. It makes the deployment and scaling of services in cloud-native environments seamless and secure. By leveraging Aspire, we're able to provide a modern, production-grade microservice ecosystem with very little friction and excellent visibility into system health.

AppHost and service discovery:

builder.Services.AddRefitClient<IAdminHttpClient>()
    .ConfigureHttpClient(c => c.BaseAddress = new Uri(
        builder.Environment.IsProduction()
            ? builder.Configuration["adminurl"]
            : "https+http://adminservice"));

builder.AddAzureServiceBusClient("servicebus");

🧪 Aspire Empowers Local Development Too

One of the lesser-sung advantages of .NET Aspire is how much it improves the local development experience. Aspire supports virtualized cloud resources, allowing developers to simulate complex distributed systems without requiring full Azure infrastructure locally. Developers can run all microservices in tandem, inspect traffic between them, and leverage local emulators for Azure services, such as Service Bus or Cosmos DB. The Aspire AppHost automatically wires up service dependencies, enabling consistent startup across environments. This means devs spend less time fiddling with Docker or local secrets and more time writing and debugging actual features. It’s local-first cloud-native development, done right.

Example Aspire app configuration:

var builder = DistributedApplication.CreateBuilder(args);

var cosmos = builder.AddAzureCosmosDB("cosmos")
    .WithDatabase("BuddyNetworks");

var userService = builder.AddProject<Projects.UserService>("userservice")
    .WithReference(cosmos);

builder.Build().Run();

✨ ASP.NET Core 9 in Action

ASP.NET Core 9 provides a modern, high-performance foundation for building cloud-native APIs. With support for minimal APIs, developers can write faster, leaner endpoints that are easier to test and maintain. Versioned routing, detailed response annotations, and OpenAPI integration make documentation and client development straightforward. In the Architect4Hire framework, we implement consistent response types using Result.Match(), which converts domain results into HTTP responses cleanly. We also integrate FluentValidation for input validation and use claims-based JWT authentication to secure endpoints. ASP.NET Core 9 gives us the flexibility, performance, and security features needed to meet enterprise-grade software requirements.

Using Match() for clean responses:

public static IActionResult Match<TModel>(this Result<TModel> result) where TModel : ServiceBaseModel
{
    if (result.IsSuccess)
    {
        return new OkObjectResult(result);
    }

    return result.Error switch
    {
        Error.NotFound => new NotFoundObjectResult(result),
        Error.InvalidInput => new BadRequestObjectResult(result),
        Error.Unauthorized => new UnauthorizedObjectResult(result),
        Error.Forbidden => new ForbidResult(),
        Error.InternalServerError => new ObjectResult(result) { StatusCode = 500 },
        Error.DuplicateEmail => new ObjectResult(result) { StatusCode = 432 },
        _ => new ObjectResult(result) { StatusCode = 500 }
    };
}

🧐 The Facade Pattern: Our Secret Weapon

The Facade Layer is one of the standout features of the Architect4Hire framework. It acts as an intermediary between the HTTP-facing Presentation Layer and the logic-rich Business Layer. This pattern abstracts away validation, exception translation, and cross-service orchestration, keeping the controllers thin and focused. Facades validate input using FluentValidation, convert domain exceptions into typed errors, and return standardized Result objects to maintain consistency across the API. The pattern ensures that changing business rules doesn't require controller refactoring, making the application more resilient to change. It also enhances testability and reusability by isolating orchestration from routing logic.

Example facade method:

public async Task<Result<NoResponseServiceModel>> RegisterUserAsync(RegisterViewModel model, int site)
{
    var validationResult = _registerValidator.Validate(model);
    if (!validationResult.IsValid)
    {
        var errorMessages = validationResult.Errors.Select(e => e.ErrorMessage).ToList();
        return Result<NoResponseServiceModel>.Failure(Error.InvalidInput, errorMessages);
    }

    try
    {
        await _userBusiness.RegisterUserAsync(model, site);
        return Result<NoResponseServiceModel>.Success();
    }
    catch (DuplicateEmailException)
    {
        return Result<NoResponseServiceModel>.Failure(Error.DuplicateEmail, new List<string>());
    }
    catch (Exception ex)
    {
        return Result<NoResponseServiceModel>.Failure(Error.InternalServerError, new List<string> { ex.Message });
    }
}

📃 Data Done Right

Data persistence in the Architect4Hire SOA Framework is handled through clean repository-style interfaces and EF Core-backed implementations. SQL Server is used for relational storage, especially for identity and transactional data, while Cosmos DB supports high-throughput document storage for services like profiles. The UserData class encapsulates all Identity-related operations, throwing custom domain exceptions for known failure conditions (e.g., duplicate username or email). This allows business logic to handle failures in a domain-aware fashion. Every data layer implementation is unit-testable, and our framework encourages clear boundaries, preventing business logic from leaking into persistence. It’s a clean, robust approach to modern data management.

Sample data operation:

public async Task<string> RegisterUserAsync(string email, string userName, string password, MemberLevel level, int site)
{
    var user = new ApplicationUser { Email = email, UserName = userName, EmailConfirmed = false };
    var result = await _userManager.CreateAsync(user, password);

    if (!result.Succeeded)
    {
        switch (result.Errors.FirstOrDefault()?.Code.ToLower())
        {
            case "duplicateusername": throw new DuplicateUserNameException("Username already exists.");
            case "duplicateemail": throw new DuplicateEmailException("Email already exists.");
            default: throw new Exception("User registration failed: " + string.Join(", ", result.Errors.Select(e => e.Description)));
        }
    }

    await _userManager.AddClaimAsync(user, new Claim("MemberBB4N", "true"));
    await _userManager.AddClaimAsync(user, new Claim("BB4NLevel", level.ToString()));

    return user.Id;
}

🧪 Testing, Because Production is Not QA

Robust testing practices are baked into every layer of our framework. We employ unit tests for business logic and validation, integration tests for verifying component interactions, and full-stack API tests for endpoint coverage. For example, RegisterUserAsync is tested from input validation all the way through to Cosmos DB writes and email orchestration. We use in-memory test doubles and mocking to simulate external services. Testing is integrated into our CI/CD pipeline using GitHub Actions and Azure DevOps, ensuring that no code reaches production without proper coverage. Testing isn't an afterthought—it's foundational to delivering stable, predictable software in real-world deployments.

Unit test strategy example:

[Fact]
public async Task RegisterUserAsync_ShouldReturnSuccess_WhenValidInput()
{
    var model = new RegisterViewModel { UserName = "testuser", Email = "test@example.com", Password = "Password123", Level = MemberLevel.Standard };
    var result = await _facade.RegisterUserAsync(model, site: 1);

    Assert.True(result.IsSuccess);
}

Want a Template Like This?

This architecture isn’t locked away behind corporate doors. The Architect4Hire SOA Framework is available as a reusable solution template that you can adopt or adapt to your own business needs. Whether you're building a dating app, a logistics platform, or a social media service, this architecture provides a head start with proven patterns, a working service mesh, and all the boilerplate wiring done correctly. Want a copy? Reach out to us directly.

📍 VisitArchitect4Hire.com and let’s talk about how we can help you build better software, faster.

Our Aspire Café project, built with .NET Aspire and ASP.NET Core, is like a hands-on blueprint for developers who want to learn by doing. It takes all those best practices we hear about—like clean architecture, microservices, and containerized deployments—and shows them in action. So if you want to level up your cloud-native game, Architect4Hire’s got your back.

Make sure to bookmark the git repo at https://github.com/architect4hire/aspirecafe

☕ Microservices: The Café's Backbone

The Aspire Café solution exemplifies a microservices architecture, where each service is dedicated to a specific business function:

• Product API: Manages the product catalog and inventory.
• Counter API: Handles order processing and payments.
• Kitchen API: Manages food preparation workflows.
• Barista API: Handles beverage preparation workflows.
• Order Summary API: Aggregates order information across services.

Each service adheres to the Single Responsibility Principle, allowing for independent development, deployment, and scaling. This modularity enhances fault isolation and facilitates targeted updates.

🔧 .NET Aspire Application Host: Orchestrating the Services

At the heart of the solution is the Aspire Application Host, which orchestrates the various services and infrastructure components. It simplifies the management of distributed applications by handling:

• Service registration and dependencies: Automatically wires up services and their dependencies.
• Infrastructure provisioning: Manages resources like databases, caches, and message brokers.
• Service discovery and configuration: Facilitates service communication through automatic discovery and configuration.

This orchestration enhances the local development experience by simplifying the management of the application's configuration and interconnections.

🏗️ Infrastructure Components: Building Blocks of the Café

The solution leverages several cloud-native services, managed through Aspire:

• Azure Cosmos DB: A NoSQL database for storing product and order information. The .NET Aspire integration enables seamless connection to existing Cosmos DB instances or creation of new ones.
• Redis Cache: Provides distributed caching for improved performance. Aspire's integration allows for easy setup and management of Redis instances.
• Azure Service Bus: Acts as a message broker for asynchronous communication between services. Aspire simplifies the integration with Azure Service Bus.
• Azure Key Vault: Securely stores credentials and secrets. The solution uses an emulator for development purposes.

Each component is containerized for local development, ensuring consistency across environments.

🧱 Design Patterns and Practices: Structuring the Codebase

The solution employs several design patterns to enhance maintainability and testability:

Layered (Onion Style) Architecture

Each microservice implements a clean layered architecture:

• API Layer: Controllers handling HTTP requests.
• Facade Layer: Orchestrates business logic and validation.
• Business Layer: Implements core business rules.
• Data Layer: Manages data access and persistence.

This separation of concerns improves maintainability and testability.

Facade Pattern

The facade pattern provides a simplified interface to complex subsystems, decoupling the client (controller) from the implementation details of the business logic. This pattern enhances code readability and maintainability.

Result Pattern

A custom Result type is used to handle operation outcomes consistently, avoiding exception-based flow control and providing clear error information.

Validation Pattern

FluentValidation is employed for model validation, separating validation rules from the models. This ensures robust input validation before business operations.

📚 API Documentation: Clear and Comprehensive

The controllers utilize XML documentation and ProducesResponseType attributes to generate comprehensive API documentation. This approach integrates well with OpenAPI and tools like Scalar.net, providing developers with explicit and interactive API references.

🚀 Modern C# and .NET Features: Leveraging the Latest

The solution takes advantage of the latest features in C# 13 and .NET 9:

• Async/await patterns: Facilitate non-blocking operations.
• Pattern matching: Enhances code clarity and conciseness.
• Extension methods: Promote cleaner code organization.
• Middleware: Handles cross-cutting concerns like exception handling.

These features contribute to a modern, efficient, and maintainable codebase.

☁️ Simply Put A Robust Foundation for Cloud-Native Applications

The Aspire Café solution demonstrates how modern .NET applications can be built using a microservices architecture with the .NET Aspire stack. The application achieves scalability and maintainability by leveraging clean design patterns, cloud-native infrastructure, and the latest language features.

This architecture provides a solid foundation for building complex, distributed systems that can evolve independently while working together seamlessly.