CxO.Observability 0.7.3

CxO.Observability

Shared observability primitives for CxO services. One NuGet package that wires up:

• OpenTelemetry metrics — ASP.NET Core, HttpClient, Runtime, Process instrumentation + Prometheus exporter on /metrics
• OpenTelemetry tracing — ASP.NET Core, HttpClient instrumentation + OTLP exporter, with well-known ActivitySources (CxO.Orders, CxO.Workflows, CxO.Dependencies)
• Standard health endpoints — /health, /ready, /live with tag-based filtering
• Dependency telemetry — HTTP (via DelegatingHandler) and WCF/SOAP (via three interception strategies) with a closed-vocabulary metric schema: cxo_dependency_request_total, cxo_dependency_request_duration_seconds, cxo_dependency_timeout_total, cxo_dependency_health_status
• Serilog logging, fully configured — console + OTLP sink to the OTel Collector with the service.name resource attribute set from the bootstrap argument; the deployment's Serilog config section extends/overrides (do NOT call UseSerilog yourself)
• Structured request logging — Serilog request middleware + a LogContext enricher that stamps ClientIp / UserId / OrderId / WorkflowId / CorrelationId on every request log and on the current Activity
• Conductor trace correlation — W3C trace-context propagation through Netflix Conductor workflows (ConductorSharp), registered by default; a no-op for services that never touch Conductor

Targets net8.0 and net10.0. Distributed via GitHub Packages (ghcr.io/codaxy) and nuget.org.

Install

dotnet add package CxO.Observability

Or as a PackageReference:

<PackageReference Include="CxO.Observability" Version="0.7.*" />

Quick start — three-line integration

In your service's Program.cs:

using CxO.Observability;

var builder = WebApplication.CreateBuilder(args);

// 1. Register the observability stack (metrics + tracing + LOGS + dependency
//    telemetry + health checks + Conductor trace correlation).
//    Do NOT also call builder.Host.UseSerilog(...) — the bootstrap owns Serilog.
builder.AddCxoObservability("cxo-your-service-name");

// ... your other service registrations ...

var app = builder.Build();

// 2. Activate request logging + LogContext middleware (AFTER auth)
app.UseAuthentication();
app.UseAuthorization();
app.UseCxoObservability();

// 3. Expose Prometheus + health endpoints
app.MapCxoObservabilityEndpoints();

app.MapControllers();
app.Run();

That's it. After this your service emits the platform-standard metric/log/trace fingerprint and can be picked up by the central Prometheus + Grafana + Tempo stack.

What you get, concretely

On GET /metrics

Prometheus-format metrics including:

On GET /health, /ready, /live

Aggregated health check responses. Tag a registration ready for readiness-probe inclusion, live for liveness, or leave untagged — it'll appear in /health but not /ready or /live:

builder.Services.AddHealthChecks()
    .AddNpgSql(pgConnectionString, tags: ["ready"])     // readiness: DB must be reachable
    .AddKafka(kafkaOpts, tags: ["ready"])
    .AddCheck("self", () => HealthCheckResult.Healthy(), tags: ["live"]);

OTLP traces

Sent to the collector endpoint resolved from OTEL_EXPORTER_OTLP_ENDPOINT env var (default http://cxo-otel-collector:4317). Lab environment already has the collector; override in other environments.

Structured request logs

Every request produces a Serilog log line enriched with ClientIp, UserId, OrderId, WorkflowId, CorrelationId, RequestPath, StatusCode. The same properties flow through LogContext to any log lines emitted within the request, and are also stamped as tags on Activity.Current so they appear on your traces too.

Probe paths (/metrics, /health, /ready, /live) log at Verbose to avoid drowning your logs in probe noise.

Oversized-property protection

String log properties are capped at 8192 chars at the source — scalar strings by a truncating enricher (suffix …[truncated]), @-captured values by destructuring caps (max depth 5, max 100 collection elements). One fat property (a serialized payload logged as {Payload}) would otherwise exceed Loki's 64KB structured-metadata limit and cost the OTel Collector a whole export batch — including other services' logs. Don't log whole objects; log IDs, types, and sizes. The cap protects every sink, Seq included.

Custom metrics + tracing

Add service-specific meters and activity sources via the optional hooks:

builder.AddCxoObservability(
    serviceName: "cxo-service-order-management",
    configureMeters: meters => meters.AddMeter("CxO.Orders"),
    configureTracing: tracing => tracing.AddSource("CxO.Orders"));

For the CxO order/workflow/dependency convention, the built-in ActivitySources are already registered:

using var activity = CxO.Observability.Tracing.CxoActivitySources.Orders
    .StartActivity("ProvisionOrder");
activity?.SetTag(CxoActivityTags.OrderId, orderId);

Structured business events — LogCxoEvent

For discrete business/operational facts someone will want on a dashboard — migration steps, order state changes, workflow tasks, background job runs — log a CxO event instead of a free-form line:

using CxO.Observability.Logging;

logger.LogCxoEvent(MigrationEvents.Step, "Migration", migrationId, CxoEventStatus.Completed,
    durationMs: 192400, data: new { Step = "ies", Operation = "Link", Linked = 14840 });

One standard property shape (EventType, EntityType, EntityId, Status, DurationMs, Reason, plus your flattened data members) means Grafana can count, rate, and measure events from Loki without parsing message text:

{service_name=~"cxo-.*"} | EventType="MigrationStep" | Status="Failed"
{service_name="cxo-service-ordering-engine"} | EventType="SweepRun" | unwrap DurationMs

Rules of the road:

• Event type constants are owned by the emitting service (PascalCase, noun + past-tense verb); the library fixes only the envelope names and the CxoEventStatus vocabulary (Started | Completed | Failed | Skipped).
• Events always log at Information — Status carries the outcome. Exceptions keep the existing LogError path.
• Null fields are emitted as null, so each event type has a stable shape.
• Use events for run/job-shaped work; where a Prometheus metric already aggregates (order rates), the event is the drill-down, not the source of truth.
• Ordinary Information/Debug logging is unaffected — don't wrap it in events.

Dependency telemetry — HTTP

Attach the shared DependencyTelemetryHandler to any IHttpClientFactory client. The minimal form (one bucket per dependency, cardinality-safe):

builder.Services
    .AddHttpClient<HansenClient>(c => c.BaseAddress = new Uri(cfg["Hansen:BaseUrl"]))
    .AddDependencyTelemetry("Hansen");
// → cxo_dependency_request_total{dependency="Hansen", operation="HansenClientApiCall", status="..."}

Pass a switch to break down by endpoint:

builder.Services
    .AddHttpClient<KeycloakClient>(c => c.BaseAddress = new Uri(cfg["Keycloak:BaseUrl"]))
    .AddDependencyTelemetry(
        dependency: "keycloak",
        operationNameResolver: request => request.RequestUri?.LocalPath switch
        {
            "/realms/cxo/protocol/openid-connect/token" => "GetToken",
            "/admin/realms/cxo/users"                   => "ListUsers",
            _                                           => "Other"
        });

Don't default the resolver to the URL path (r => r.RequestUri!.LocalPath). IDs in the path (/api/orders/12345) become a unique operation label each, exploding Prometheus cardinality. Constant per dependency is the only safe default; explicit switches are how you get finer granularity.

Every outbound call against KeycloakClient now emits cxo_dependency_request_total{dependency="keycloak", operation="GetToken|ListUsers|Other", status="ok|4xx|5xx|fault|timeout|error"} plus the latency histogram.

Dependency telemetry — WCF / SOAP

Three integration strategies depending on how your SOAP client is constructed. See docs/integration-guide.md in the repo for full examples; the short version:

// Option — DispatchProxy (recommended for ChannelFactory-based clients)
var channelFactory = new ChannelFactory<ILegacyService>(binding, endpoint);
var proxy = WcfTelemetryDispatchProxy.Create(
    channelFactory.CreateChannel(),
    metrics: dependencyMetrics,
    dependencyName: "legacy-billing-soap",
    resolveOperation: message => message.Headers.Action?.Split('/').Last() ?? "Unknown");

Dependency health checks

External-dependency probes — vendor billing systems, partner CRMs, IAM gateways like Keycloak, etc. — must be registered via AddDependencyCheck(...), not the standard AddCheck<> / AddNpgSql / AddKafka APIs. Only checks registered through this extension feed into cxo_dependency_health_status, which the Grafana External Dependencies dashboard reads.

var healthChecks = builder.Services.AddHealthChecks();

// ❌ Standard checks — visible in /health, NOT in External Dependencies dashboard
healthChecks.AddNpgSql(connectionString, name: "postgres", tags: ["ready"]);
healthChecks.AddKafka(producerConfig, name: "kafka", tags: ["ready"]);
healthChecks.AddCheck<ConductorHealthCheck>("conductor", tags: ["ready"]);

// ✅ External dependency — reaches the dashboard up/down panel
var billingApiUrl = builder.Configuration["BillingApi:HealthUrl"];
if (!string.IsNullOrWhiteSpace(billingApiUrl))
{
    healthChecks.AddDependencyCheck(
        name: "billing-api",
        probe: async (sp, ct) =>
        {
            var http = sp.GetRequiredService<IHttpClientFactory>().CreateClient();
            using var resp = await http.GetAsync(billingApiUrl, ct);
            resp.EnsureSuccessStatusCode();   // throw → Unhealthy
        },
        tags: ["ready"],                       // "dependency" tag added automatically
        timeout: TimeSpan.FromSeconds(5)
    );
}

Emitted metric:

cxo_dependency_health_status{name="billing-api"} 1   # 1=Healthy, 0.5=Degraded, 0=Unhealthy

The Grafana External Dependencies dashboard auto-discovers every dependency via label_values(cxo_dependency_health_status, name) — no dashboard changes needed per new dependency.

Update cadence

IHealthCheckPublisher ticks every 30 seconds by default. The gauge can lag the actual probe outcome by up to that interval. Configure HealthCheckPublisherOptions.Period if you need faster propagation (e.g., for tighter alert windows).

Conditional registration

Wrap calls in a config guard when the dependency is optional per environment. Otherwise the probe still runs (and flaps Unhealthy) in environments where the dependency isn't deployed — which adds noise to the dashboard and any down-alert rule.

Why filter out infra checks?

DependencyHealthCheckPublisher filters health-check reports through DependencyCheckRegistry, which is populated only by AddDependencyCheck. This intentionally separates infrastructure checks (postgres, kafka, redis — your own platform's plumbing) from external dependency checks (third-party APIs, vendor systems). The dashboard is meant to surface "is the outside world reachable from this service" — flapping postgres creates alert noise of the wrong shape and is already covered by Postgres-specific dashboards.

Metric names, labels, and the dependency status contract

The package intentionally uses a closed set of values for the status label:

Stick to these values. Adding a new one requires a coordinated update of Grafana dashboards and alert rules (DependencyHighErrorRate regex matches 5xx|fault|error|timeout).

Full reference in docs/dependency-naming-conventions.md.

Configuration surface

appsettings.json requirements

None. Logging works with no appsettings.json at all: the bootstrap configures Serilog with the platform defaults — Information minimum (Microsoft at Warning), FromLogContext + machine name / process id / thread id enrichment, a Console sink, and an OTLP sink shipping to the OTel Collector with service.name taken from the AddCxoObservability argument.

The Serilog configuration section is applied ON TOP of those defaults, so appsettings can add deployment sinks (Seq, Elasticsearch), override levels, or take over a default sink — a Console or OpenTelemetry entry declared in Serilog:WriteTo suppresses the matching built-in sink, so legacy configs never double-ship log lines.

{
  "Serilog": {
    "WriteTo": [
      { "Name": "Seq", "Args": { "serverUrl": "http://cxo-seq:5341" } }
    ]
  }
}

Deployment add-on sinks (Serilog.Sinks.Seq, Serilog.Sinks.Elasticsearch) must be referenced by the service; Serilog.Sinks.OpenTelemetry, Serilog.Expressions, and the standard enrichers ship with this package.

Configuration knobs (all optional; env var beats appsettings beats default):

Do not call builder.Host.UseSerilog(...) — the bootstrap registers the Serilog root itself; a second registration creates competing logger factories (symptoms: every log line twice, or logs under a stale service name).

Migrating from hand-rolled OTel wiring

If your Program.cs currently looks like:

builder.Services.AddOpenTelemetry()
    .WithMetrics(m => m
        .AddAspNetCoreInstrumentation()
        .AddRuntimeInstrumentation()
        .AddPrometheusExporter());

app.MapPrometheusScrapingEndpoint();
app.MapHealthChecks("/health");

Replace with:

builder.AddCxoObservability("cxo-your-service-name");
// ...
app.UseCxoObservability();
app.MapCxoObservabilityEndpoints();

You gain: AddProcessInstrumentation (CPU!), HttpClient instrumentation, OTLP tracing, dependency telemetry registration, Serilog LogContext middleware, and /ready / /live tag-filtered endpoints. Nothing you had breaks — any meter you previously registered via configureMeters hook still works.

Drop direct OpenTelemetry.Instrumentation.* / OpenTelemetry.Exporter.* PackageReference lines from your .csproj — they come transitively now.

Versioning

Semantic versioning. Breaking changes (metric names, ActivitySource names, public API shape) bump major. Additive features bump minor. Patch for bug fixes. While on 0.x, the minor version may contain breaking changes until 1.0 — watch the release notes.

Links

• Repo: github.com/codaxy/cxo-shared
• Design doc: docs/implementation-design.md
• Integration guide: docs/integration-guide.md
• Dependency naming conventions: docs/dependency-naming-conventions.md

License

Proprietary — © Codaxy. Consumption restricted to CxO platform services and Codaxy-authorized integrators.