1. Overview

  • Unpatched Binary: cng_unpatched.sys
  • Patched Binary: cng_patched.sys
  • Overall Similarity Score: 0.9674
  • Diff Statistics:
  • Matched Functions: 1578
  • Changed Functions: 68
  • Identical Functions: 1510
  • Unmatched Functions (Unpatched/Patched): 0 / 0
  • Verdict: The patch adds a bounded iteration cap and error propagation to the ML-DSA (post-quantum, FIPS 204) rejection-sampling routine. The change is real and the direction is correct (the patched build is strictly more defensive), but the loop it hardens is driven by a SHAKE/Keccak sponge whose output an attacker cannot steer. The pathological "never terminates" condition is not attacker-reachable, so this is a defense-in-depth robustness improvement, not a fix for an exploitable denial-of-service.

2. Change Summary

  • Severity: Informational (defense-in-depth).
  • Change Class: Loop termination guarantee added to a rejection-sampling loop (code pattern: CWE-835, loop with unreachable exit condition; hardened, but not demonstrably reachable).
  • Affected Function: SymCryptMlDsaRejBoundedPoly (unpatched @ 0x18009767C, patched @ 0x18009A8C0).

What the function does: SymCryptMlDsaRejBoundedPoly implements the bounded rejection-sampling phase of ML-DSA (Module Lattice Digital Signature Algorithm / FIPS 204). It generates polynomial coefficients by pulling bytes from a SHAKE/Keccak sponge. The sponge is initialized in the function itself: SymCryptKeccakAppend absorbs the seed material, and each loop iteration calls SymCryptKeccakExtract to squeeze one output byte. Each byte yields two nibbles (low and high). Each nibble is compared against the parameter-set bound (mode 2: valid 0..14, reject nibble >= 0xF; mode 4: valid 0..8, reject nibble >= 9). Rejected nibbles are marked 0x80; accepted nibbles are mapped to a coefficient, written to the output buffer, and the output counter (edi in the unpatched build) is incremented.

What changed: In the unpatched build the loop terminates on a single condition: the output counter reaching 256 (cmp edi, 0x100; jb loop at 0x1800977DC). If a squeezed byte has both nibbles rejected, the output counter does not advance for that iteration. There is no separate cap on the raw number of iterations, so in principle a sponge stream that rejects both nibbles indefinitely would never let the loop exit.

The patched build adds a second counter (esi) that increments once per squeezed byte and forces an exit after 0x1E1 (481) iterations, returning error 0x8008 (see 0x18009A93E / 0x18009AA41). The output counter is tracked separately in ebx. The caller SymCryptMlDsaExpandS is updated to check and propagate this return value (it ignored the return in the unpatched build).

Why this is not an attacker-reachable denial of service: The bytes consumed by the loop are the output of a SHAKE/Keccak sponge, not raw attacker input and not a value an attacker can dictate. An attacker controls, at most, the seed absorbed before squeezing (and during key generation the seed is derived from system randomness, not chosen by the caller). To make the loop run more than 481 iterations, the sponge would have to emit a stream in which fewer than 256 coefficients are accepted across 481 squeezed bytes. For mode 2 the per-nibble rejection probability is 1/16, and for mode 4 it is 7/16; the expected number of accepted coefficients over 481 iterations is far above 256 in both cases. Producing a stream that stays below 256 acceptances would require choosing a seed whose SHAKE output has a specific pathological structure, a preimage-style manipulation of SHAKE that is computationally infeasible. The rejection bounds (eta) are static per-parameter-set constants, not runtime-tunable by an attacker. The added cap is therefore a termination guarantee for a condition that cannot be induced from outside, consistent with the upstream robustness guidance for FIPS 204 rejection sampling.

Reachable entry points (for context, key generation and signing both reach this function): BCryptGenerateKeyPair / BCryptFinalizeKeyPair and BCryptSignHash (Win32 / kernel CNG APIs) reach ML-DSA key and signature generation, which call SymCryptMlDsaExpandS, which calls SymCryptMlDsaRejBoundedPoly. These paths are real; they are listed only to show the function is live code, not to assert an exploitable trigger.

Call Chain (real code path to the changed function): 1. BCryptFinalizeKeyPair (0x1800557C0) or BCryptSignHash (0x180056140) 2. ML-DSA finalize / sign dispatch 3. ML-DSA key generation / signing core 4. SymCryptMlDsaExpandS (unpatched 0x180096AA8, patched 0x180099CBC) - polynomial vector generation loop 5. SymCryptMlDsaRejBoundedPoly (unpatched 0x18009767C, patched 0x18009A8C0) - the hardened loop

3. Pseudocode Diff

The rejection-sampling loop iterates until 256 coefficients are produced. The patch adds a raw iteration counter and an error exit.

// UNPATCHED SymCryptMlDsaRejBoundedPoly (0x18009767C)
int rdi = 0;                       // output coefficient counter

do {
    SymCryptKeccakExtract(&sponge, &byte, 1, 0);   // squeeze one sponge byte

    uint8_t low  = byte & 0xf;      // low nibble
    uint8_t high = byte >> 4;       // high nibble

    // nibble bounds check: rejected nibbles are set to 0x80

    if (low != 0x80) {
        output_buf[rdi] = map_coefficient(low);
        rdi++;                      // only increments when accepted
    }

    if (high != 0x80) {
        if (rdi >= 0x100) break;
        output_buf[rdi] = map_coefficient(high);
        rdi++;                      // only increments when accepted
    }
} while (rdi < 0x100);              // sole exit condition

// PATCHED SymCryptMlDsaRejBoundedPoly (0x18009A8C0)
int out  = 0;                      // output coefficient counter (ebx)
int iter = 0;                      // NEW raw iteration counter (esi)
int status = 0;                    // edi, returned to caller

do {
    if (iter >= 0x1e1) {           // NEW: cap at 481 squeezed bytes
        status = 0x8008;           // return error instead of continuing
        break;
    }

    SymCryptKeccakExtract(&sponge, &byte, 1, 0);
    // ... same nibble bounds check ...
    iter++;                        // NEW: count every iteration

    if (low != 0x80)  { output_buf[out++] = map_coefficient(low); }
    if (high != 0x80) { if (out >= 0x100) break; output_buf[out++] = map_coefficient(high); }
} while (out < 0x100);

return status;                     // NEW: caller (ExpandS) now checks this

4. Assembly Analysis

Unpatched loop (cng_unpatched.sys), showing the single exit condition on the output counter and the absence of any raw-iteration cap:

; SymCryptMlDsaRejBoundedPoly @ 0x18009767C - loop body
000000018009767C  mov     [rsp-8+arg_0], rbx
0000000180097696  mov     rax, cs:__security_cookie
00000001800976BA  call    memset
00000001800976D4  call    SymCryptWipeAsm
00000001800976D9  xor     edi, edi                 ; edi = 0 (output counter)
00000001800976EE  call    SymCryptKeccakAppend     ; absorb seed into sponge
; === LOOP START (0x1800976FB) - no iteration counter ===
00000001800976FB  xor     r9d, r9d
000000018009770C  call    SymCryptKeccakExtract    ; squeeze one sponge byte
0000000180097711  mov     cl, [rsp+140h+var_11F]   ; cl = squeezed byte
0000000180097715  mov     al, [r14+16h]            ; al = mode (2 or 4)
000000018009771B  and     dl, 0Fh                  ; dl = low nibble
0000000180097722  cmp     dl, 0Fh                  ; mode 2: reject if nibble >= 0xF
0000000180097725  jnb     short loc_18009775B
000000018009774B  cmp     dl, 9                    ; mode 4: reject if nibble >= 9
000000018009775E  mov     r8b, 80h                 ; low nibble REJECTED marker
0000000180097790  mov     dl, 80h                  ; high nibble REJECTED marker
0000000180097792  cmp     r8b, 80h                 ; low nibble rejected?
0000000180097796  jz      short loc_1800977B1      ; yes -> skip write/inc
00000001800977AC  mov     [rsi+rdi*4], ecx         ; write coefficient
00000001800977AF  inc     edi                      ; advance output counter
00000001800977B1  cmp     dl, 80h                  ; high nibble rejected?
00000001800977B4  jz      short loc_1800977D6
00000001800977B6  cmp     edi, 100h
00000001800977BC  jnb     short loc_1800977E2      ; output full -> exit
00000001800977D1  mov     [rsi+rdi*4], ecx
00000001800977D4  inc     edi
00000001800977D6  cmp     edi, 100h                ; SOLE LOOP CONDITION
00000001800977DC  jb      loc_1800976FB            ; loop back on output counter only
00000001800977E2  mov     edx, 0F0h                ; fall through -> wipe + return

Patched loop (cng_patched.sys), showing the added raw-iteration counter, the cap, and the error return:

; SymCryptMlDsaRejBoundedPoly @ 0x18009A8C0
000000018009A8FC  xor     edi, edi                 ; edi = 0 (return status)
000000018009A930  call    SymCryptKeccakAppend     ; absorb seed into sponge
000000018009A93A  mov     ebx, edi                 ; ebx = 0 (output counter)
000000018009A93C  mov     esi, edi                 ; esi = 0 (NEW iteration counter)
; === LOOP START (0x18009A93E) ===
000000018009A93E  cmp     esi, 1E1h                ; NEW: iteration cap (481)
000000018009A944  jnb     loc_18009AA41            ; NEW: exit to error path
000000018009A95B  call    SymCryptKeccakExtract    ; squeeze one sponge byte
000000018009A964  mov     al, [r15+16h]            ; al = mode (2 or 4)
000000018009A9E5  mov     dl, 80h                  ; high nibble REJECTED marker
000000018009A9E7  inc     esi                      ; NEW: count this iteration
000000018009A9E9  cmp     r8b, 80h
000000018009AA05  inc     ebx                      ; advance output counter
000000018009AA0B  cmp     dl, 80h
000000018009AA10  cmp     ebx, 100h
000000018009AA16  jnb     short loc_18009AA46
000000018009AA2D  inc     ebx
000000018009AA33  cmp     ebx, 100h                ; output-counter exit
000000018009AA39  jb      loc_18009A93E            ; loop back
000000018009AA3F  jmp     short loc_18009AA46
000000018009AA41  mov     edi, 8008h               ; NEW: error status on cap hit
000000018009AA46  mov     edx, 0F0h                ; wipe + return
000000018009AA55  mov     eax, edi                 ; return status to caller

Caller change: SymCryptMlDsaExpandS now inspects the return value. Unpatched (0x180096AA8) discards it:

0000000180096B29  call    SymCryptMlDsaRejBoundedPoly
0000000180096B2E  inc     di                       ; return value ignored

Patched (0x180099CBC) checks and propagates it:

0000000180099D47  call    SymCryptMlDsaRejBoundedPoly
0000000180099D4C  mov     edi, eax                 ; capture status
0000000180099D4E  test    eax, eax
0000000180099D50  jnz     short loc_180099D9A      ; bail out and propagate on error

5. Trigger / Reachability Analysis

  1. From user mode, opening an ML-DSA provider (BCryptOpenAlgorithmProvider, algorithm ML-DSA) and calling BCryptGenerateKeyPair / BCryptFinalizeKeyPair or BCryptSignHash reaches SymCryptMlDsaRejBoundedPoly through SymCryptMlDsaExpandS. The code path is real.
  2. The loop consumes SHAKE/Keccak sponge output squeezed by SymCryptKeccakExtract, one byte per iteration. The mode byte at [r14+0x16] (unpatched) / [r15+0x16] (patched) selects a static rejection bound: mode 2 rejects only nibble 0xF (≈ 1/16 per nibble), mode 4 rejects nibble >= 9 (≈ 7/16 per nibble).
  3. For the loop to exceed 481 iterations, the sponge would have to emit a stream in which fewer than 256 coefficients are accepted over 481 squeezed bytes. The expected acceptance count over 481 iterations is far above 256 for both modes, so this requires a large, sustained deviation from the sponge's output distribution.
  4. An attacker cannot force that deviation. The sponge output is not attacker-supplied bytes; the only external input is the absorbed seed, and steering SHAKE output by seed choice is a preimage-style problem that is computationally infeasible. During key generation the seed is not even caller-chosen. The rejection bound (eta) is a compile-time per-parameter-set constant.
  5. Net effect: In normal operation the loop terminates well before the cap. The cap changes behavior only under a sponge output that cannot be induced from outside the kernel, so there is no demonstrable attacker-reachable denial of service. The change is a robustness / termination guarantee.

6. Classification Notes

  • Primitive: None demonstrable. The change adds a bounded exit to a loop that is otherwise driven by cryptographic sponge output an attacker cannot control.
  • Memory safety: No memory corruption is involved. The output buffer is written only for indices < 0x100; both builds enforce this before every write.
  • CWE: The hardened code pattern corresponds to CWE-835 (loop with unreachable exit condition), but the exit is only unreachable under a sponge output that is not attacker-inducible, so no reachable weakness is established.

7. Inspection Notes

To observe the hardened behavior in a kernel debugger against the unpatched build:

  • Breakpoints: text bp cng_unpatched!SymCryptMlDsaRejBoundedPoly ; 0x18009767C bp 0x1800976FB ; loop entry bp 0x1800977DC ; loop-back branch
  • What to inspect:
  • At 0x1800976FB: edi is the output coefficient counter. Under normal sponge output it climbs steadily to 0x100 and the loop exits at 0x1800977E2.
  • byte [rsp+0x21]: the sponge byte squeezed this iteration by SymCryptKeccakExtract.
  • byte [r14+0x16]: the mode (02 or 04), which selects the static rejection bound.
  • r8b and dl at 0x180097792 / 0x1800977B1: the per-nibble accept/reject markers (0x80 = rejected).
  • Setup: cpp BCRYPT_ALG_HANDLE hAlg; BCryptOpenAlgorithmProvider(&hAlg, L"ML-DSA", NULL, 0); BCRYPT_KEY_HANDLE hKey; BCryptGenerateKeyPair(hAlg, &hKey, 0, 0); BCryptFinalizeKeyPair(hKey, 0);
  • Expected observation: The loop terminates normally on every real invocation. The patched build additionally exits with status 0x8008 if 481 iterations are reached; reaching that cap under real sponge output is not observed and is not inducible by manipulating the seed.

8. Changed Functions — Full Triage

  • SymCryptMlDsaRejBoundedPoly (unpatched 0x18009767C, patched 0x18009A8C0) (0.91 similarity): Defense-in-depth. Adds a raw iteration counter capped at 0x1E1 (481) that returns 0x8008, and the caller now propagates the status. Hardens the sponge-driven rejection-sampling loop; not attacker-reachable.
  • SymCryptMlDsaExpandS (unpatched 0x180096AA8, patched 0x180099CBC) (0.72 similarity): Behavioral. The two SymCryptMlDsaRejBoundedPoly call sites now check the return value (test eax, eax; jnz) and propagate the 0x8008 error instead of ignoring it.
  • SymCryptMlDsaSignEx (unpatched 0x18008A340, patched 0x18008C66C) (0.62 similarity): Defense-in-depth. The unpatched signing rejection loop is while (1) with a success break; the patched build guards loop entry with a computed 16-bit bound (814 * *(ctx+0x15), i.e. 0x32E * kappa) and adds error propagation from the sampling routine. Same sponge-driven rejection-sampling family; not a demonstrable attacker-reachable integer overflow.
  • SymCryptMlDsaVerifyEx (sub_18008A8BC) (0.92 similarity): Behavioral. ML-DSA verification path refactored with restructured decode/length validation.
  • SymCryptMlDsaKeyGenerateEx (sub_180089FD8) (0.83 similarity): Behavioral. Updates to sponge/seed parameter handling in key generation.
  • SymCryptMlDsaPkDecode (sub_180096E5C) (0.93 similarity): Behavioral. Handled zero-polynomial edge case in public-key export.
  • MSCryptMlDsaImportKeyPair (sub_1800744B0) (0.56 similarity): Behavioral. Dispatch refactoring for new PQ parameter sets.
  • MSCryptMlDsaSignHash (sub_180074900) (0.66 similarity): Behavioral. BCryptSignHash/VerifySignature dispatch refactored.
  • GetSignatureInterface (sub_180070110) (0.93 similarity): Cosmetic. Added 'Composite-ML-DSA' algorithm string matching.
  • SymCryptEckeyGetValue (sub_180084FC0) (0.95 similarity): Behavioral. Added FIPS validation check (0x46495053).
  • SymCryptRsakeyCalculatePrivateFields (sub_18002CB54) (0.74 similarity): Behavioral. New signature-verification flag (arg7 & 0x200).
  • MSCryptMlDsaCloseProvider (sub_180073E00) (0.16 similarity): Behavioral. Rewrite of key-object cleanup with registry TLS checks.
  • SymCryptEcurveInitialize (sub_180029370) (0.95 similarity): Cosmetic. Address relocations only.

9. Unmatched Functions

No functions were exclusively added or removed between the unpatched and patched binaries. All changes were implemented inline or as drop-in replacements for existing routines.

10. Confidence & Caveats

  • Confidence: High on the mechanics. The unpatched loop's sole exit condition (cmp edi, 0x100; jb) and the patched loop's added iteration cap (cmp esi, 0x1E1; jnb -> mov edi, 0x8008) and error propagation in SymCryptMlDsaExpandS are all present at the addresses cited above in both builds.
  • Severity rationale: The loop is fed by SHAKE/Keccak sponge output (SymCryptKeccakExtract), not attacker-supplied bytes. The rejection bound is a static per-parameter-set constant. Forcing non-termination would require steering SHAKE output through seed choice, which is computationally infeasible, so no attacker-reachable denial of service is demonstrated. The change is classified as defense-in-depth robustness.
  • Assumptions: Standard Windows CNG architecture where cng.sys services bcrypt.dll calls in kernel mode via ksecdd.sys. Much of the surrounding ML-DSA code is statically linked SymCrypt.

DONE