Backend
The .NET HTTP Failure Nobody Sees Coming -Najeeb Ullah
Najeeb Ullah Dev.to (EN Zone)
1 views
Your CPU is fine. Your memory is fine. Your database is fine. So why did the application suddenly stop calling another service?
There is a class of production failure that is particularly dangerous because your application metrics can look completely healthy while the application is already running out of a critical resource.
The symptoms often look like this:
outbound API calls start timing out
SocketException begins appearing
latency suddenly increases
requests fail intermittently
restarting the application temporarily fixes everything
And when you look at the usual dashboards:
CPU: normal Memory: normal Database: normal
So what exactly is failing?
Sometimes, the answer is hiding underneath your C# code:
TCP connections.
And more specifically:
The lifecycle of those connections.
Start With the Production Problem
Imagine an ASP.NET Core application responsible for processing payments.
For every incoming request, it calls an external payment service.
The code looks harmless:
public async Task<PaymentResult> ProcessAsync(
PaymentRequest request)
{
using var client = new HttpClient();
return await client.PostAsJsonAsync(
"https://payments.example.com/process",
request);
}
Under light traffic, everything works.
Then production traffic increases.
Suddenly you start seeing:
System.Net.Sockets.SocketException
or connection failures and timeouts.
Restart the application.
Everything works again.
For a while.
Then the problem comes back.
That restart is an important clue.
It tells you something has accumulated inside the process or its operating-system networking resources.
The Mistake Is Not Really HttpClient
This is where many explanations stop too early.
They say:
“Don't create HttpClient repeatedly. Use IHttpClientFactory.”
That's useful advice.
But it doesn't explain why.
And if you understand the why, you can recognize the same failure pattern even when the technology changes.
The real question is:
What happens to the underlying network connection when I repeatedly create and dispose HTTP clients?
To answer that, we have to leave the application layer for a moment.
Your HTTP Request Is Sitting on Top of TCP
An HTTPS request isn't simply:
C# → API
There are several layers involved.
Conceptually:
Article content
Layers
This is where the interesting part begins.
An HttpClient instance owns or uses a connection pool through its underlying handler. Creating unnecessary clients/handlers can therefore create unnecessary connection pools and connection establishment. Microsoft explicitly recommends reusing HttpClient instances or using IHttpClientFactory rather than creating and disposing one for every request.
The Resource You Didn't Know You Were Consuming
Every outbound TCP connection needs a local source port.
For example:
Application Server
10.10.1.20:52341
↓
Payment API:443
That temporary source port comes from the operating system's ephemeral port range.
The exact range depends on the operating system and configuration.
The important concept is simpler:
The number of outbound connections your process can create is not infinite.
Now imagine an application repeatedly creating new connections instead of reusing existing ones.
The number of connections can grow surprisingly quickly.
Then TCP Gives You a Problem Called TIME_WAIT
Closing a connection does not necessarily mean:
“This port is immediately available for another connection.”
TCP has connection lifecycle rules.
After certain connection-close scenarios, the endpoint can remain in:
TIME_WAIT
for a period of time.
This exists for an important reason: TCP needs to protect against delayed packets from an earlier connection being mistaken for packets belonging to a newer connection.
So TIME_WAIT isn't a bug.
It is part of TCP's correctness model.
The problem occurs when your application creates connections much faster than the system can recycle the resources involved in those connections.
Microsoft's current .NET guidance explicitly calls out this interaction: TCP ports aren't released immediately after connection closure, and high request rates can exhaust available ports.
The Math Makes the Problem Obvious
Suppose an application establishes:
500 new outbound connections per second.
If a large number of those connections remain in a state that prevents immediate reuse for a significant period, the number of connection endpoints associated with that traffic can become very large.
The important equation is:
Connection creation rate
×
Connection lifetime / reuse characteristics
Resource pressure
That's why this problem is fundamentally different from:
“My C# method is slow.”
Your method may be perfectly fast.
The resource lifecycle underneath the method may be the problem.
Why using Doesnt Solve It
This is one of the most misunderstood parts.
You see:
using var client = new HttpClient();
and think:
“I'm disposing it correctly.”
You are disposing the object.
But the lifetime of the .NET object and the lifecycle of the network resources underneath it are not the same thing.
And more importantly, each HttpClient has its own connection pool. Creating clients repeatedly means you repeatedly create isolated pools instead of benefiting from connection reuse.
So the better question isn't:
“Did I dispose HttpClient?”
It's:
“Am I creating a new connection pool when I could have reused an existing one?”
That is the architectural question.
The First “Fix”: Create One HttpClient
A developer discovers the problem and changes the code to:
private static readonly HttpClient Client = new();
Much better.
Now the client can reuse its underlying connections.
This addresses a major cause of unnecessary connection creation.
But there's another concern.
DNS.
The DNS Problem
Suppose:
api.payment.com
currently resolves to:
10.20.1.15
Later, your infrastructure changes:
api.payment.com
↓
10.20.1.42
This can happen during:
deployment
failover
scaling
infrastructure migration
load-balancer changes
service discovery changes
HttpClient doesn't continuously re-resolve DNS for every request. DNS is resolved when a new connection is created. Long-lived connections can therefore outlive the DNS information that originally led to their creation. Microsoft recommends controlling connection lifetime with PooledConnectionLifetime when using a long-lived client.
So now we have two competing concerns:
Create clients too often
→ unnecessary connection creation → poor connection reuse → potential port/socket exhaustion
Keep connections forever
→ connection reuse is excellent → but stale network topology can become a concern
This is the actual engineering problem.
The Real Solution Is Connection Lifetime Management
This is the key idea I want developers to take away:
Don't optimize the lifetime of the HttpClient object. Optimize the lifetime of the underlying connections.
Once you understand that, the architecture becomes much clearer.
There are two strong approaches in modern .NET.
Option One: Long-Lived HttpClient + PooledConnectionLifetime
For applications where you don't need the features of IHttpClientFactory, a long-lived client with controlled connection lifetime is a valid approach.
For example:
var handler = new SocketsHttpHandler
{
PooledConnectionLifetime =
TimeSpan.FromMinutes(5)
};
var client = new HttpClient(handler);
Now the client can reuse connections while allowing them to be replaced periodically.
The value isn't:
“5 minutes is the magic number.”
There is no magic number.
The correct lifetime depends on:
DNS change frequency
infrastructure topology
traffic pattern
deployment model
connection characteristics
Microsoft's current guidance explicitly presents this as an alternative to IHttpClientFactory.
Option Two: IHttpClientFactory
For most ASP.NET Core applications with multiple outbound dependencies, this is where IHttpClientFactory becomes very useful.
builder.Services
.AddHttpClient<PaymentClient>(client =>
{
client.BaseAddress =
new Uri("https://payments.example.com");
client.Timeout =
TimeSpan.FromSeconds(15);
});
Then:
public sealed class PaymentClient
{
private readonly HttpClient _httpClient;
public PaymentClient(HttpClient httpClient)
{
_httpClient = httpClient;
}
public async Task<PaymentResponse?> ProcessAsync(
PaymentRequest request,
CancellationToken cancellationToken)
{
using var response =
await _httpClient.PostAsJsonAsync(
"payments",
request,
cancellationToken);
response.EnsureSuccessStatusCode();
return await response.Content
.ReadFromJsonAsync<PaymentResponse>(
cancellationToken);
}
}
The important thing is not merely that dependency injection created HttpClient.
The important thing is what happens underneath.
HttpClient Is Not the Whole Story
Article content
With IHttpClientFactory, the factory creates HttpClient instances while managing and pooling the underlying handlers. Those handlers typically own the connection pools. Reusing handlers avoids unnecessary connection creation, while handler lifetime management allows the system to respond to DNS changes.
The default handler lifetime is currently two minutes, but that is a configurable default, not a universal rule.
That distinction matters.
There's Another Problem: Too Much Concurrency
Here's a subtle point many HttpClientFactory articles miss.
Even if you're using IHttpClientFactory correctly, you can still create too many connections.
Imagine:
await Task.WhenAll(
requests.Select(x => client.GetAsync(x)));
with thousands of HTTP/1.1 requests starting simultaneously.
If the connection pool has no available connection and concurrency isn't bounded, the system can still attempt many connections.
Microsoft's troubleshooting guidance explicitly recommends considering MaxConnectionsPerServer and HTTP/2 multiplexing for high-concurrency scenarios.
For example:
builder.Services
.AddHttpClient<PaymentClient>()
.ConfigurePrimaryHttpMessageHandler(() =>
new SocketsHttpHandler
{
MaxConnectionsPerServer = 50
});
Now you've introduced another important concept:
Connection reuse and concurrency control are different problems.
You need to think about both.
HTTP/2 Changes the Picture Again
With HTTP/1.1, multiple concurrent requests may require multiple TCP connections depending on the connection behavior and limits.
HTTP/2 supports multiplexing.
Multiple requests can share a TCP connection.
Conceptually:
┌── Request A
│
TCP Connection├── Request B
│
├── Request C
│
└── Request D
That can dramatically change the connection profile of a high-concurrency application.
So when diagnosing “too many sockets,” the question shouldn't only be:
“How many HttpClient objects do we have?”
It should also be:
“What protocol are we using, how are connections pooled, and how much concurrency are we allowing?”
The Production Architecture I Prefer
For a typical enterprise .NET application, I want the outbound HTTP layer to have explicit boundaries.
Article content
Production Architecture
And around that I want:
Timeouts
Cancellation
Controlled concurrency
Observability
Resilience policies
Clear ownership of each external dependency
This turns HTTP communication from scattered infrastructure code into an intentional architectural component.
Dont Make IHttpClientFactory Your New Cargo Cult
This is important.
I wouldn't replace:
new HttpClient()
with:
IHttpClientFactory
and consider the problem solved.
That's just replacing one habit with another.
The actual questions should be:
How long should connections live?
How many concurrent connections do we allow?
Does DNS change?
Are we using HTTP/1.1 or HTTP/2?
Do we need connection pooling?
What happens when the downstream service becomes slow?
What happens when it becomes unavailable?
Can we observe the failure?
That's architecture.
A Small but Important Factory Trap
Even with IHttpClientFactory, you can accidentally defeat handler rotation.
For example, a typed client is normally short-lived.
If you capture that typed client inside a singleton and keep it alive indefinitely, the HttpClient can remain tied to an older handler longer than intended.
Microsoft specifically warns against capturing factory-created clients or typed clients in singleton services when that prevents timely handler rotation and DNS updates.
So:
Using IHttpClientFactory correctly is more important than merely registering it.
How I Would Investigate This in Production
I wouldn't start by changing HttpClient.
I'd start with evidence.
I'd want to know:
Application
outbound request rate
request latency
timeout rate
exception rate
concurrent requests
HTTP
HTTP/1.1 vs HTTP/2
connection reuse
connection pool behavior
MaxConnectionsPerServer
Operating System
active TCP connections
TIME_WAIT
ephemeral port usage
connection failures
Infrastructure
DNS changes
load balancer behavior
service deployment pattern
downstream health
The goal is to determine:
Are we creating too many connections, holding them too long, or failing to reuse them?
Only after answering that would I change the implementation.
The Bigger Lesson
This entire problem started with one innocent line:
new HttpClient()
But the failure had almost nothing to do with the syntax.
It involved:
C#
→ .NET HTTP abstractions
→ connection pooling
→ TCP
→ operating-system resources
→ DNS
→ network architecture
That's why some of the hardest production bugs are difficult to solve.
The code where the problem appears isn't necessarily the layer where the problem lives.
The Engineering Principle I Keep
When an application starts failing under load, I try not to ask:
“What line of code is broken?”
I ask:
“What resource lifecycle did our design misunderstand?”
For socket exhaustion, that resource is the network connection.
Once you understand its lifecycle, the solution stops being:
“Use IHttpClientFactory because everyone says so.”
And becomes:
Reuse connections. Control their lifetime. Bound concurrency. Respect DNS. Observe the system.
That's a much more useful engineering principle.
One Question for .NET Engineers
Have you ever had a production incident where CPU and memory looked completely healthy, but outbound HTTP calls were failing?
What turned out to be the real cause?
Thanks for reading.. Najeeb Ullah
Read original: https://dev.to/najeebullah/the-net-http-failure-nobody-sees-coming-najeeb-ullah-3g45
← Previous
My checker blamed the other tool, and the defect was in the one doing the blaming
Next →
Seven Patterns That Decide If Your AI App Survives 10,000 Users
Related
Droid ASC: A High-Performance Tool for Android Reverse Engineering and Vulnerability Discovery
Backend
0
DEV Community
zvec-grep (zg): A Local-First Hybrid Search Engine for Humans and AI Agents
Backend
0
Dev.to (EN Zone)
PicoCTF Easy1 Writeup — Recover an XOR Key with Crib Dragging
Backend
1
Dev.to (EN Zone)
Why we moved our Backstage platform from Yarn to pnpm
Backend
4
DEV Community
Comments0
No comments yet — be the first