Advanced .NET Analysis: SOSEX Extension Guide

SOSEX Extension Reference for Azure PaaS Engineers Advanced .NET Heap & Lock Diagnostics

“Debugging is like archaeology — except the artifacts are in memory, the civilization is your app, and the ancient curse is a memory leak.”

Welcome to your friendly neighborhood SOSEX guide — your one-stop shop for peeking under the hood of Azure App Services and poking around in managed heaps, locks, and threads like a digital detective. If you’ve ever stared at a memory dump wondering, “Why does this thing smell like a stale HttpClient?” , you’re in the right place.

This guide is tailored for Azure PaaS Engineers working with Windows-based App Services, .NET Framework/Core apps, and post-mortem memory dumps. Use it with WinDbg and the mighty !sosex extension to uncover deadlocks, memory leaks, and ghosts of objects past.

Heap Indexing

Command	Description
!bhi [filename]	BuildHeapIndex – Indexes heap objects to a file
!lhi [filename]	LoadHeapIndex – Loads the heap index
!chi	ClearHeapIndex – Frees and unloads heap index resources

GC & Finalization

Command	Description
!dumpgen <GenNum> [options]	Dumps a GC generation (options: -free, -stat, -type, -nostrings)
!gcgen <ObjectAddr>	Show GC generation for an object
!finq [GenNum] [-stat]	Show finalizable objects
!frq [-stat]	Show objects in Freachable queue
!gch [HandleType]	List GC handles (strong, pinned, etc.)
!mfrag [-stat] [-mt:<MT>]	Heap fragmentation stats & free block analysis

Threads, Waits, Deadlocks

Command	Description
!dlk [-d]	Deadlock detection (SyncBlocks, ReaderWriterLocks)
!mlocks [-d]	List managed locks & owners
!mwaits [-d]	List waiting threads and what they’re waiting on
`!rwlock [addr	-d]`

Managed Breakpoints (Advanced Debugging)

Command	Description
!mbp <SourceFile> <Line> [Col]	Set breakpoint at source location
!mbm <Type/Method> [ILOffset]	Set method breakpoint
`!mbc <ID	*>`
`!mbd <ID	*>`
`!mbe <ID	*>`
!mbl [ID]	List active breakpoints

Stack, Frames, and Variables

Command	Description
!mk [Count] [-l -p -a]	Mixed stack (managed + unmanaged)
!mframe [nFrame]	Set/view managed frame
!mdv [nFrame]	View method arguments and locals
`!mdt [Type	VarName
!mu [addr] [-s -il -n]	Disassembly with source/IL/asm
!muf [MD or CodeAddr] [-s -il -n]	Disassembly around method
!mt	Step into managed method
!mln [expression]	Identify managed data at address

Heap/Object Exploration

Command	Description
!mdso	Dump stack and register object refs
!mroot <ObjectAddr> [-all]	Show GC roots for object
`!refs <ObjectAddr> [-source	-target]`
!strings [ModuleAddr] [Options]	Search for strings in heap or module
!mx <filter>	Search for types, methods, fields

Help & Documentation

Command	Description
!help [Command]	General help
!sosexhelp [Command]	SOSEX-specific help

🗒️ Notes

!bpsc is deprecated; use !mbp instead.
Use !windex before running SOSEX commands after loading the extension.
Combine SOSEX with !mex.clrstack2 or !dumpheap -stat for deeper root cause analysis.

Heap Indexing

`!bhi [filename]`

BuildHeapIndex — Indexes all objects in the managed heap.

✅ Use When: You want to perform fast object searches (!mdt, !refs) across a large heap.
❌ Avoid When: Dump is small or already indexed. Building the index can be memory-intensive.
📘 Example: During high memory investigations, index the heap to quickly locate all instances of a MemoryStream that aren’t GC’d.

`!lhi [filename]`

LoadHeapIndex — Load an existing heap index.

✅ Use it after !bhi when reopening the same dump to skip re-indexing.

`!chi`

ClearHeapIndex — Unload the heap index from memory.

✅ Run before closing the dump if you indexed it.

🔎 GC & Finalization

`!dumpgen <GenNum>`

✅ Use When: Analyzing memory growth patterns.
📘 Example: !dumpgen 2 -stat shows retained objects in Gen 2 during a memory leak.
❌ Bad Sign: Lots of business objects (e.g., Invoice, HttpContext) stuck in Gen 2. Indicates poor disposal or GC pressure.

`!gcgen <ObjectAddr>`

✅ See how old an object is.
📘 Example: !gcgen 000001f02fd30080 shows if a StringBuilder is living in Gen 0 or has survived GC cycles.

`!finq` and `!frq`

✅ Use when analyzing high finalization delay.
❌ Red Flag: Finalizable objects (e.g., SqlConnection) piling up = IDisposable misuse.
📘 !finq -stat during post-mortem of sudden memory spike.

`!gch`

✅ Shows pinned handles and weak references.
📘 Helps when investigating OutOfMemoryException where pinning might prevent compaction.

`!mfrag`

✅ Useful when suspecting heap fragmentation in low memory scenarios.
❌ If large free blocks are scattered, it can lead to inefficient allocation patterns.

🧵 Locks & Concurrency

`!dlk [-d]`

✅ Use When: Hang or thread contention suspected.
❌ Bad Sign: Multiple threads holding/awaiting locks cyclically.
📘 Real case: App hangs due to deadlock between ReaderWriterLockSlim and async state machine.

`!mlocks`

✅ Show active lock owners.
📘 Diagnose thread contention in multi-threaded request handlers.

`!mwaits`

✅ Shows threads waiting for locks.
📘 Especially helpful when only a few worker threads are active, but app is unresponsive.

`!rwlock`

✅ Drill into ReaderWriterLock status.

🔖 Managed Breakpoints

`!mbp`, `!mbm`, `!mbc`, `!mbe`, `!mbd`, `!mbl`

✅ Use for live debugging managed code via source line or method.
📘 Example: !mbp MyController.cs 102 to break before DB call.
❌ Avoid in postmortem dumps (not applicable).

🏛 Stack & Variables

`!mk`

✅ Managed + native stack, crucial for mixed-mode .NET calls.
📘 Used in CPU spike cases to show System.Net.Sockets.Socket.Receive() native stalling.

`!mdt`, `!mdv`, `!mframe`

✅ Dump types, variables, locals.
📘 Used to track parameter values in a stuck async method.

`!mu`, `!muf`, `!mt`

✅ IL + source disassembly.
Great for .NET performance or ref call issues.

`!mln`

✅ Tells what object is at instruction pointer (or addr).
📘 Handy when crashing inside unknown JIT stub.

🔍 Object Reference Tracing

`!mdso`

✅ Stack + register object roots.
📘 Use during GC root analysis when !gcroot fails due to inlined locals.

`!mroot`

✅ Trace what is rooting a given object.
📘 Use when object is stuck in memory unexpectedly.

`!refs`

✅ Who references (or is referenced by) a given object.
📘 Combine with !dumpheap -type to isolate memory leaks.

`!strings`

✅ Dump large or suspect strings from heap.
📘 Ex: Regex errors or JWT payloads stuck in memory.

`!mx <filter>`

✅ Autocomplete field/method/type filters.
📘 Use when !mdt gives raw MT and you want type info.

❓ Help & Support

`!help`, `!sosexhelp`

✅ Help for new engineers and command discovery.

💡 Pro Tip

Run !windex before using SOSEX.
Use SOSEX in tandem with MEX: !mex.clrstack2, !mex.t, !mex.cn
Combine !dumpheap, !gcroot, and !refs for end-to-end leak triage.

Common Pitfalls

Forgetting !windex before SOSEX = most commands silently fail
Live-only breakpoints used in dump analysis
!bhi on small dumps – wastes time, gains nothing
Relying only on !gcroot – some roots are invisible due to JIT or inlining; use !mdso instead
Mistaking large Gen 2 for a leak – always pair !dumpgen with !refs to confirm

❓ Frequently Asked Questions (FAQ)

Q: Can I use SOSEX on Linux dumps?
A: Nope. SOSEX is Windows-only. Use dotnet-gcdump, dotnet-dump, or lldb/sos for Linux-based debugging.

Q: I see weird thread IDs in !dlk, how do I correlate them?
A: Match them using !Threads or !mex.t – look for managed vs native thread IDs.

Q: Why is !gcroot showing nothing for a stuck object?
A: Some roots aren’t visible if they’re in optimized methods or JIT inlined. Use !mdso as a backup.

Q: Is !sosex better than SOS or MEX?
A: It’s not a replacement — it complements SOS. Use them together for full coverage. Think of SOSEX as your lock and heap sidekick.

“Debugging is like archaeology — except the artifacts are in memory, the civilization is your app, and the ancient curse is a memory leak.”

Heap Indexing

!bhi [filename]

!lhi [filename]

!chi

🔎 GC & Finalization

!dumpgen <GenNum>

!gcgen <ObjectAddr>

!finq and !frq

!gch

!mfrag

🧵 Locks & Concurrency

!dlk [-d]

!mlocks

!mwaits

!rwlock

🔖 Managed Breakpoints

!mbp, !mbm, !mbc, !mbe, !mbd, !mbl