Developer Notes

HTTP Logging in ASP.NET Core

Posted 03 Feb 2023

Sometimes, you need to know all the details of the HTTP calls into and out of a .NET Web API.

I was recently developing an integration API to help feed data into a chat flow. The chat flow is a SaaS solution, and it is extensible enough to send requests to an API. The documentation is decent but lacks a bit of detail. 

There was, however, no way to make it send the requests directly to my development environment. In order to be able to verify the significant details, I decided to deploy an API and capture the HTTP calls going into that service

Capturing the calls enables quick verification of headers and content and reveals what headers the reverse proxy adds to the requests.

Know your toolbox

There's no need to reinvent the wheel and write custom middleware. ASP.NET Core has excellent middleware available to capture HTTP traffic: the HTTP Logging middleware.

Enabling it is as easy as adding it to the pipeline:

var builder = WebApplication.CreateBuilder(args);
// ... your config here
var app = builder.Build();

app.UseHttpLogging();

// .. pipeline setup here

app.Run();

Caveats

There is a price to using this middleware though:

Performance and cost

Logging HTTP traffic will generate large volumes of data when your service handles a lot of traffic. Larger log volumes generally cost more money and reduce your service's capacity.

Privacy

When you log too much, you risk sending personal information to your logs which could be a breach of policy or even GDPR regulation. 

To reduce this risk, I recommend using this module for debugging and troubleshooting only. Additionally you should always configure exactly what data and headers should be logged.

Specifying what data to capture

Configure HttpLoggingOptions to log the right information. The LoggingFields property controls what is logged. To prevent private data from accidentally being included in your logs, you need to explicitly enable this. 

If you need the contents of specific request headers, add them to the RequestHeaders property. Otherwise they will be logged with a redacted value.

builder.Services.Configure<HttpLoggingOptions>(opts =>
{
    // Information to log
    opts.LoggingFields = HttpLoggingFields.All;
    // Limit how much data to log
    opts.RequestBodyLogLimit = 8192;

    // Distributed tracing heades
    opts.RequestHeaders.Add("X-Request-ID");
    opts.RequestHeaders.Add("X-Trace");

    // Reverse proxy headers
    opts.RequestHeaders.Add("X-Real-IP");
    opts.RequestHeaders.Add("X-Forwarded-For");
    opts.RequestHeaders.Add("X-Forwarded-Host");
    opts.RequestHeaders.Add("X-Forwarded-Port");
    opts.RequestHeaders.Add("X-Forwarded-Proto");
    opts.RequestHeaders.Add("X-Forwarded-Scheme");
    opts.RequestHeaders.Add("X-Scheme");

    // Custom headers
    opts.RequestHeaders.Add("X-Vendor-Correlation-Id");
    opts.RequestHeaders.Add("X-Vendor-Client-IP");
});

Configuration

Filtering logs is a solid idea if you want to limit log volume and the associated cost. Make sure you enable the traces from the HttpLoggingMiddleware at Information level. For example in your appsettings.json:

{
  "Logging": {
    "LogLevel": {
      "Microsoft.AspNetCore.HttpLogging.HttpLoggingMiddleware": "Information"
    }
}

Further reading

• Docs: HTTP Logging in ASP.NET Core