Distributed tracing in .NET with OpenTelemetry

Distributed tracing is increasingly crucial for .NET applications in microservices. Solutions for distributed tracing, such as OpenTelemetry, simplify tracking and resolving issues in complex transactions across multiple services, especially intricate cloud-based systems.

Let’s explore how to use distributed tracing in a .NET application with OpenTelemetry. We’ll set it up, walk through how to perform distributed tracing, and set the stage for tackling more advanced topics in the second part of this two-part series. You’ll learn how to use distributed tracing effectively in your own .NET projects.

Understanding distributed tracing and OpenTelemetry

Tracing involves monitoring and recording how an application executes a user’s request. Distributed tracing tracks this information across multiple interconnected services. This aids in performance analysis and debugging.

OpenTelemetry is an open-source observability framework designed to collect, process, and export telemetry data (like these traces, plus metrics and logs) efficiently. The most important concepts are spans, traces, and instrumentations.

• Spans: Basic units of work in a trace, representing individual operations with metadata
• Traces: Collections of spans showing the entire journey of a request through a system
• Exporters: Components that send telemetry data to back-end systems for analysis
• Processors: Tools that modify or enhance telemetry data before it’s exported
• Instrumentations: Plugins that automatically capture telemetry data from frameworks and libraries
• Logs: Records of system events, including errors and informational messages
• Metrics: Numerical data measuring system performance, like response times and memory usage

.NET’s architecture, with its emphasis on high performance and scalability, aligns well with distributed tracing’s objectives.

OpenTelemetry provides .NET developers with a toolkit to implement distributed tracing without significantly altering their existing codebase. This framework makes it straightforward to trace requests across microservices while equipping you with the tools to comprehensively analyze and optimize your application.

Prerequisites

To follow this tutorial, you need:

• A fundamental grasp of Microsoft .NET development
• Free tier access to Microsoft Azure
• Visual Studio or Visual Studio Code for development. These instructions use Visual Studio Code.
• .NET 7 SDK
• A GitHub account

How to set up the environment

Now let’s set up a .NET development environment for distributed tracing.

First, install the .NET SDK. Open your terminal and run the command dotnet --version to check if .NET SDK is already installed. If not, install it from the official .NET website.

Next, set up Visual Studio Code (VS Code). Download and install VS Code from its official website. Then, install the C# extension for .NET development within VS Code.

Now, create a simple .NET web API. In your VS Code terminal, run the command dotnet new webapi -n DistributedTracingDemo.

While still in the terminal, type the command cd DistributedTracingDemo to go to the project directory. Run code . to open the project in the current VS code window.

Then, install OpenTelemetry Packages. In your VS Code terminal, run the following commands to install the basic OpenTelemetry SDK:

dotnet add package OpenTelemetry
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Exporter.Console
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.Http

This simple API is your base to implement distributed tracing. The API tracks air quality in cities, focusing on counting hazardous air quality days.

Add a new folder named Models to the existing project. In that folder, create a new file called AirQualityReport.cs. Paste the following code:

namespace Examples.AspNetCore;  
public class AirQualityReport  
{ 
    public DateTime Date { get; set; }  
    public string? City { get; set; } 
    public int AQI { get; set; }  // Air Quality Index 
    public string Category => GetCategory(AQI);  

    private static string GetCategory(int aqi)  
   {  
       if (aqi <= 50) return "Good"; 
        if (aqi <= 100) return "Moderate"; 
        if (aqi <= 150) return "Unhealthy for Sensitive Groups"; 
        if (aqi <= 200) return "Unhealthy";  
        if (aqi <= 300) return "Very Unhealthy"; 
        return "Hazardous"; 
    }  
}  

Next, add a new controller to the existing Controller folder. Call it AirQualityController.cs and paste the following code:

namespace Examples.AspNetCore.Controllers; 

using System; 
using System.Collections.Generic;  
using System.Linq;  
using Microsoft.AspNetCore.Mvc;  
[ApiController]  
[Route("[controller]")]  
public class AirQualityController : ControllerBase  
{  
    private static readonly string[] Cities = new[]  
    {  
        "New York", "Los Angeles", "Chicago", "Houston", "Phoenix" 
    };  

    [HttpGet]  
    public IEnumerable<AirQualityReport> Get()  
    {  
        var rng = new Random(); 
        return Enumerable.Range(1, 5).Select(index => new AirQualityReport 
        {  
            Date = DateTime.Now.AddDays(index),  
            City = Cities[rng.Next(Cities.Length)],  
            AQI = rng.Next(0, 500)  // Simulated AQI value 
        }).ToArray();  
    }  
}            

Integrating OpenTelemetry into the .NET application

Add the following code lines to Program.cs to integrate OpenTelemetry into a .NET application, like the demo .NET application. We’ll focus on the OpenTelemetry SDK’s tracing aspects.

Configure the resource (line 13)

Define the resource associated with all this service’s telemetry before setting the tracing. The configuration below sets the service name, version, and instance ID, making it straightforward to identify and correlate traces.

Action<ResourceBuilder> configureResource = r => r.AddService( 
    serviceName: appBuilder.Configuration.GetValue("ServiceName", defaultValue: "otel-test")!, 
    serviceVersion: typeof(Program).Assembly.GetName().Version?.ToString() ?? "unknown", 
    serviceInstanceId: Environment.MachineName);            

Initialize OpenTelemetry services (line 24)

To set up tracing, add OpenTelemetry services to the application’s service collection:

appBuilder.Services.AddOpenTelemetry(); 
    .ConfigureResource(configureResource)  
    .WithTracing(builder => 
    { …            

Configure tracing (line 26)

The WithTracing method below is crucial for tracing. It specifies custom ActivitySource names to set a sampler, adds instrumentation for HttpClient and ASP.NET Core, and configures tracing exporters.

.WithTracing(builder => 
{  
    builder  
        .AddSource(Instrumentation.ActivitySourceName)  
        .SetSampler(new AlwaysOnSampler())  
        .AddHttpClientInstrumentation() 
        .AddAspNetCoreInstrumentation();  
    builder.AddConsoleExporter();  
})            

Configure the tracing exporters (line 61)

Use the following code to configure the tracing exporter:

builder.AddConsoleExporter().             

This setup sends the traces to the console. You might want to use other exporters like Azure Monitor, OTLP, or Zipkin for production environments.

Add tracing processors and samplers (line 32)

The following SetSampler method with AlwaysOnSampler ensures that every trace is captured. You can adjust this setting based on your performance and data requirements.

.SetSampler(new AlwaysOnSampler())             

Use tracing instrumentation (line 33)

Use AddHttpClientInstrumentation() and AddAspNetCoreInstrumentation() to automatically prepare your application’s parts to generate traces for HTTP client calls and ASP.NET Core requests.

Add a custom activity source (line 31)

Use the following code to add the custom ActivitySource defined in the Instrumentation class to the tracing builder. This method enables manual tracing in parts of the application:

.AddSource(Instrumentation.ActivitySourceName)             

Implementing basic distributed tracing

Define and collect traces to implement basic distributed tracing in a .NET application, like the demo application. Use correct trace context to ensure that spans from different services are connected, making it straightforward to follow requests through the system and identify issues.

It’s crucial to name spans clearly, choose useful attributes, and use trace context correctly. Pick names that clearly describe what the operation does, but don’t use too many unique names so you don’t overwhelm the data.

Set up ActivitySource

ActivitySource is the core of custom distributed tracing in a .NET application, referring to the instrumentation class. Create an instrumentation.cs file and paste the following code to set up a custom instrumentation class:

public class Instrumentation: IDisposable 
{  
    internal const string ActivitySourceName = "Examples.AspNetCore"; 
    internal const string MeterName = "Examples.AspNetCore"; 
    private readonly Meter meter;  

    public Instrumentation() 
    {  
        string? version = typeof(Instrumentation).Assembly.GetName().Version?.ToString(); 
        this.ActivitySource = new ActivitySource(ActivitySourceName, version); 
        this.meter = new Meter(MeterName, version); 
        // Initialize HazardousDaysCounter for air quality tracking  
        this.HazardousDaysCounter = this.meter.CreateCounter<long>("airquality.days.hazardous",  
             description: "The number of days where the air quality is hazardous" 
        );  
    } 
    public ActivitySource ActivitySource { get; } 

    public Counter HazardousDaysCounter { get; } 

    public void Dispose() 
    {  
        this.ActivitySource.Dispose();  
        this.meter.Dispose();  
    } 
}           

Inject instrumentation into services

In line 20 of Program.cs, use the following code to register the Instrumentation class as a singleton in the application’s service container.

appBuilder.Services.AddSingleton<Instrumentation>();           

Use ActivitySource in Controllers

You can now create and start activities using the ActivitySource from Instrumentation in the controllers, such as AirQualityController. To do so, create a new Activity at the beginning of an operation (like an HTTP request handler) and stop it at the end, similar to the following example:

public IEnumerable<AirQualityReport> Get() 
{  
    using var activity = this.activitySource.StartActivity("GetAirQualityReport");  
    // The rest of the existing logic  
}            

Create spans

In .NET, an Activity represents a span. The OpenTelemetry client for .NET uses this existing Activity and its associated classes.

To create a Span in .NET, first, create a new Activity. It should look like the following code from line 15 in Instrumentation.cs:

this.ActivitySource = new ActivitySource(ActivitySourceName, version);             

To start recording a span in .NET, call StartActivity. The code will record everything that occurs within the using block to that span. It should look like the following code from line 35 in AirQualityController.cs.

using var activity = activitySource.StartActivity("GenerateAirQualityReports");    

Propagate the trace context

Propagate effective trace context for distributed tracing to be effective. To automatically handle this propagation for outgoing HTTP requests, use the following code to enable the HttpClient instrumentation in Program.cs (lines 33 and 51):

.AddHttpClientInstrumentation()   

View the traces

Since you’ve already configured the console exporter in line 61 above, you can now view the traces in your console output.

Testing and validating traces

Swagger UI is a convenient tool to test and validate trace data in your .NET application. After implementing tracing in Program.cs, you can use Swagger UI to simulate HTTP requests.

For validation, monitor the console while interacting with Swagger UI. With the console exporter configured, trace data should appear in the console for each request, displaying trace and span IDs.

If traces don’t appear, ensure you’ve correctly implemented ActivitySource in your controllers and accurately configured the OpenTelemetry SDK in Program.cs. Next, confirm the correct exporter settings in appsettings.json and Program.cs, and ensure that the application’s console output is accessible.

Preparing for trace analysis

Effective trace analysis depends on collecting comprehensive trace data. In this first part of our two-part series, you’ve set up your sample program to collect this essential data.

Conclusion

You’ve navigated through the essentials to implement fundamental distributed tracing via OpenTelemetry in a .NET application. You now know how to set up the OpenTelemetry SDK and define and collect traces with custom ActivitySource in the Instrumentation class. You also explored span creation and propagation.

These practices set the foundation to collect all the data you’ll need for greater visibility into your .NET application’s performance and behavior.

Continue to experiment with the demo application. This practical experience provides deeper insights into the second article, which focuses on trace analysis.

Explore Site24x7’s robust suite of distributed tracing solutions to expand your monitoring capabilities and enhance your application’s observability and performance management.