Hot Stack Debugging with VS2022
🔥 What Is a “Hot Stack”?
A hot stack (or hot path) is a sequence of function calls consuming the most CPU during a profiling session. These are identified in Visual Studio using flame icons
🔥 in the Call Tree view.
This stack represents the most frequently sampled code path and typically surfaces the core logic or processing bottlenecks responsible for high resource consumption. Hot stacks often appear during:
- API endpoints under heavy load
- LINQ joins and materialization’s
- EF Core or database interactions
- JSON serialization or logging routines
- Middleware chains Understanding the hot stack allows engineers to trace the dominant execution path contributing to performance degradation and identify opportunities to refactor or optimize specific segments.
🔍 Visual Example:
Below is a simplified example of a hot stack path seen in Visual Studio:
WorkerThreadStart
→ ThreadPool.Dispatch
→ Controller.InvokeAsync
→ NotificationService.GetUnreadNotificationsForUser
→ LINQ.ToList()
→ JoinIterator
→ EFCore.MoveNext()
→ SqlDataReader.Read()
This stack shows how CPU time accumulates as execution flows from thread start to an EF Core database read via LINQ joins.
Key Metrics to Watch Understanding the Call Tree columns in Visual Studio 2022’s CPU Profiler is critical for pinpointing bottlenecks:
| Column | Description |
|---|---|
| Function Name | The name of the method or function being executed. Shows full namespace. |
| Total CPU | CPU time spent in the function plus all the functions it calls (inclusive). |
| Self CPU | CPU time spent only in this function (exclusive), not in any called methods. |
| Module | The DLL or assembly name where the function resides (e.g., efcore, system.linq). |
| Category | The nature of the operation — IO, Database, Json, Logging, ASP.NET , etc. |
| How to interpret these: |
- High Total CPU, Low Self CPU
- Means: This function delegates to expensive children — look deeper into nested calls.
This is where hot stacks hide! - High Self CPUMeans: The function itself is doing work, likely a tight loop, large memory copy, or inefficient algorithm.
Candidate for algorithm optimization or offloading to async/threaded processing. - Watch for functions in:
- System.Linq.Enumerable.* → In-memory LINQ
- Microsoft.EntityFrameworkCore.* → Slow DB materialization or tracking
- System.Text.Json.* → JSON serialization overhead
- Logger.Log* → Expensive logging during request pipeline
How to Read the Hot Stack The hot stack is the deepest and most expensive path your app takes when handling a request. In Visual Studio:
- It appears under the Call Tree tab with 🔥 icons
- It often starts with ThreadPool.Dispatch() or WorkerThreadStart()
- From there, it descends into your middleware, controllers, services, and LINQ/EF
Step-by-Step Analysis:
- Start at the topmost function with a high Total CPU
Usually w3wp.exe or WorkerThreadStart. - Expand all 🔥 icons down the tree
This shows the path that consumed the most samples (= most time on CPU). - Identify key framework layers
- Middleware pipeline (e.g., CorsMiddleware, SwaggerMiddleware)
- Controller methods (YourApp.Controllers.*)
- Business services (.Service.)
- LINQ and EF Core logic
- Focus on “hot” function chains
- If MoveNext() is present → EF query enumerating rows
- If JoinIterator → In-memory joins happening
- If ToList() happens before filtering → inefficiency
- Look for flat Self CPU spikes
These may indicate blocking calls, sync file IO, or busy loops.
Identify Bottlenecks
Here’s what to look for: ** High Total CPU but Low Self CPU**
Example:
MyController.GetData() → 30% Total CPU, 1% Self CPU
Means:
The function itself is fast, but something it calls is very expensive (probably a database call or heavy loop).
Expand it to find the real culprit.
High Self CPU
Example:
MyParser.Parse() → 10% Self CPU
Means:
This function itself is expensive, maybe a tight loop, recursion, or CPU-heavy operation. Consider optimizing it directly.
Example Walkthrough
Let’s say you see:
NotificationController.GetUnreadNotificationsForUser() → 30% Total CPU |– LINQ JoinIterator() → 29% Total CPU |– EF Query.MoveNext() → 15% Total CPU That means:
- Most of the time was spent doing a LINQ Join on a query result.
- Entity Framework is fetching and joining lots of data.
- You should consider:
- Using .AsNoTracking()
- Avoiding ToList() before the join
- Optimizing the SQL query
Common Patterns and Fixes Understanding these recurring patterns is critical when interpreting CPU-bound call stacks in Visual Studio Profiler. Below are the most common anti-patterns seen in real-world Azure Web Apps, their causes, and recommended remediations:
| Pattern | Cause | Fix |
|---|---|---|
| JoinIterator, .ToList() | LINQ joins evaluated in memory after fetching full result sets from EF Core. | Refactor LINQ to SQL-based joins using IQueryable. Avoid .ToList() until the end. |
| MoveNext, StateManager.Track() | EF Core is tracking every row for change detection, even on read-only queries. | Add .AsNoTracking() to skip tracking when querying read-only data. |
| JsonSerializer.Serialize() | Large or deeply nested objects are being synchronously serialized. | Use System.Text.Json (faster than Newtonsoft.Json) and pre-trim output DTOs. |
| Logger.LogInformation | Frequent or verbose logs inside high-throughput requests. | Lower the log level, batch log writes, or use async log sinks (e.g., ILogger.BeginScope). |
| Troubleshooting Table: | ||
| Symptom | Profiler Pattern | Kusto Signal |
| — | — | — |
| High CPU in LINQ | JoinIterator, ToList() | High duration |
| EF slowness | MoveNext, Track() | High result count trace |
| Logging appears in stack | Logger.LogInformation | Trace frequency |
| JSON slowness | JsonSerializer.Serialize() | N/A |
Visual Call Tree Example Use the flame path to trace down the slowest portion of execution.
Top-Level CPU Consumers
| Function | Total CPU [%] | Self CPU [%] | Module / Category |
|---|---|---|---|
| w3wp.exe (PID:4112) | 100.00 | 34.59 % | Multiple modules |
| └─ System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() | 53.01 | 0.05 | System.Private.CoreLib (IO) |
| └─ System.Threading.ThreadPoolWorkQueue.Dispatch() | 52.72 | 0.02 % | System.Private.CoreLib (IO) |
| └─ Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.Execute() | 45.72 | 0.00 % | AspNetCore.Server.IIS (ASP) |
| └─ … deep ASP .NET Core middleware & MVC pipeline … | 44.41–45.72 | 0–0.02 % | AspNetCore.Server / Swashbuckle (ASP) |
| Interpretation: |
- Roughly 53 % of all CPU time is spent just spinning up and dispatching thread‐pool work items.
- Once you’re on a thread‐pool thread, about 45 % of CPU is then consumed inside the ASP .NET Core request pipeline (authentication, static files, Swagger, MVC routing/controllers).
Framework vs. Your Code If you switch to the Top Functions view, you’ll see which specific methods (including EF Core, Reflection, Serialization) dominate:
| Function | Total CPU [%] | Self CPU [%] | Category |
|---|---|---|---|
| Microsoft.EntityFrameworkCore.Internal.DbSetInitializer.InitializeSets(DbContext) | 16.51 % | 6.82 % | Database |
| System.DefaultBinder.SelectMethod(…, BindingFlags, ParameterInfo[]) | 4.82 % | 3.69 % | Reflection |
| System.Collections.Generic.Dictionary`2.Resize(int,bool) | 2.85 % | 2.85 % | Other |
| Microsoft.AspNetCore.Server.IIS.NativeMethods.http_flush_response_bytes(…, bool, ref bool) | 2.45 % | 2.39 % | ASP .NET |
| EventPipeInterop–>Advapi32.EventWriteTransfer(…) | 2.11 % | 2.11 % | Logging/Etw |
| Key insights: |
- EF Core’s InitializeSets (model-initialization, change-tracker setup) alone is chewing up ~16 % of total CPU and ~7 % of self CPU.
- Reflection (method binding) is another 4.8 %, which likely comes from MVC action-selection or model-binding.
- Core framework (response flushing, event logging) accounts for another 4–5 %. Hot Path Drill-Down Looking at the expanded call tree (hot-path re-rooted at WorkerThreadStart):
- Middleware pipeline (AuthN, StaticFiles, HTTPS Redir, CookiePolicy, Swagger, MVC) accounts for ~44 % of the work.
- Controller activation & action execution itself (your code under dynamicClass.lambda_method…) shows up much lower—each action lambda only 2–4 %.
- Database model setup (the DbSet-initializer) is likely happening on every request if you’re not re-using the DbContext properly or if you haven’t precompiled your EF model.
Other Observations & Potential Issues
- ThreadPool Overhead: 50+ % of CPU just dispatching work items suggests very short-lived tasks or high request concurrency. Consider batching or reducing context-switch churn.
- EF Model Compilation: If you haven’t enabled EF Core’s pre-compiled models , each request will pay that ~16 % tax.
- Reflection/Binding: Heavy use of reflection in MVC (action selection, parameter binding) can be mitigated by endpoint routing or compiled expression trees.
- Swagger Middleware: Swashbuckle’s JSON generation runs on every request to the docs endpoint—ensure it’s not enabled in production or cache the JSON.
- Logging ETW: EventWriteTransfer at ~2 % could point to very chatty telemetry; consider batching or sampling.
EXAMPLE2… How to Read This Call Tree (Beginner Edition)
Top-Level Overview
- PID 5952 (w3wp.exe) is your IIS worker process handling the web request.
- The profiler recorded 2613 CPU samples. Each sample = 1 ms of CPU time, so: 2613 samples ≈ 2.613 seconds of total CPU time
Hot Path = Where CPU Time Is Spent Most
- Visual Studio highlighted the “hot path” — the most CPU-heavy sequence of calls.
- It begins from System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() and drills down through your app, middleware, controller, services, and LINQ.
Step-by-Step Interpretation
1. Controller Action: SMCT.WebAPI.Controllers.NotificationController.GetUnreadNotificationsForUser() → 785 samples (30.04% of total CPU) This means 30% of all CPU time was spent in this controller action — that’s a significant hotspot.
2. Service Logic: NotificationService.GetUnreadNotificationsForUser() → Also 785 samples (30.04%)
This means the controller is basically calling this one service — and the service is where all the work is happening. LINQ Operations: System.Linq.Enumerable+WhereSelectEnumerableIterator<T, T>.ToList() → 771 samples (29.51%)
Then:
1. System.Linq.Enumerable.JoinIterator() → 771 samples (29.51%) These LINQ methods are:
- Taking a large result set
- Running a join in memory
- Then converting it to a list
2. Entity Framework Core Querying Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable<T>.MoveNext() → 762 samples (29.16%) This is where EF is actually reading data from SQL and moving through the result set row-by-row.
3. SQL Execution (Database Time) Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReader() → ~15.5% (around 405 samples) This is the actual SQL command execution. Most of the heavy lifting is in:
- Reading from the database
- Materializing objects
- Then LINQ joins in memory
Where You’re Spending the Most Time
| Area | % of CPU | Time (if 1 ms/sample) | What it Means |
|---|---|---|---|
| Controller + Service | 30.04% | ~0.785 sec | High-level endpoint logic |
| LINQ Join + ToList | 29.51% | ~0.771 sec | In-memory join after DB read |
| EF Core MoveNext() | 29.16% | ~0.762 sec | Iterating through result set |
| SQL Command Execution | ~15.5% | ~0.405 sec | Database time |
Summary You’re doing everything right using the profiler — and here’s what it’s telling you:
| Insight | Recommendation |
|---|---|
| Most CPU is spent on LINQ and EF joins | Push logic to SQL, avoid in-memory joins |
| Controller code is thin, service is heavy | Focus optimization in the service method |
| Database call is expensive, but not the biggest cost | EF + LINQ materialization is more expensive than the SQL call itself |
| You’re CPU-bound, not I/O-bound | Optimize your C# and EF usage, not the SQL server |
Azure Advice 