cng.sys — statically-linked SymCrypt ML-DSA rejection-sampling bounded-iteration hardening

1. Overview

Verdict: The diff is a rebuild of the statically-linked SymCrypt ML-DSA (FIPS 204 / CRYSTALS-Dilithium) code. The rejection-sampling primitives that draw coefficients from a SHAKE sponge — SymCryptMlDsaRejNttPoly, SymCryptMlDsaRejBoundedPoly and SymCryptMlDsaSampleInBall — gained a bounded-iteration cap and now return an error status (0x8008) if that cap is ever exceeded. That new status is propagated up through SymCryptMlDsaExpandA / SymCryptMlDsaExpandS, and the callers SymCryptMlDsaKeyGenerateEx and SymCryptMlDsaVerifyEx now test it. In the unpatched build these primitives have no failure path at all: they loop over the (effectively infinite) SHAKE output until the required number of coefficients is collected, then return with no meaningful status word. The condition the patched build newly checks for cannot arise in the unpatched build, and the sampling input is sponge output under static per-parameter-set tables, not an attacker-controlled primitive. There is no demonstrable, reachable security impact — this is robustness / bounded-runtime hardening in the crypto library, not a vulnerability fix.

2. Change Summary

Finding #1 — ML-DSA key generation: new bounded-iteration status plumbed into SymCryptMlDsaKeyGenerateEx

What changed. The unpatched SymCryptMlDsaKeyGenerateEx (@ 0x1800894B8) calls SymCryptMlDsaExpandA (@ 0x180095DD8) (expands the public matrix A from the rho seed) and SymCryptMlDsaExpandS (@ 0x180095F78) (expands the secret vectors s1, s2) and does not test a return value from either. The patched SymCryptMlDsaKeyGenerateEx (@ 0x18008C2F0) inserts test eax,eax / jnz cleanup after each of the two calls.

Why this is not a security fix. In the unpatched build, SymCryptMlDsaExpandA and SymCryptMlDsaExpandS — and the rejection-sampling helpers they call, SymCryptMlDsaRejNttPoly (@ 0x180096CEC) and SymCryptMlDsaRejBoundedPoly (@ 0x180096B4C) — have no failure path. They draw bytes from a SHAKE sponge and loop until a full poly (256 coefficients) is accepted, then return without ever setting a status in eax (the value the caller "ignores" is undefined, not an error code). There is no early return that would leave the intermediate vectors partially filled, so the downstream steps (SymCryptMlDsaVectorNTT (@ 0x180097AEC), SymCryptMlDsakeyComputeT (@ 0x180097D24), SymCryptMlDsaPkEncode (@ 0x180096584)) always run on fully-computed state. The patched build changes the sampling helpers to enforce a maximum number of draws and to return 0x8008 if that cap is exceeded, and threads that status up to the caller. The check the patch adds guards against a condition that the unpatched code cannot produce.

Reachability of the new failure. The cap fires only if the SHAKE stream yields far more rejections than expected. For SymCryptMlDsaRejNttPoly a 3-byte draw is rejected when its 23-bit value is >= 0x7FE001 (q = 8380417); that happens with probability ≈ 0.098% per draw, so collecting 256 coefficients needs ≈ 256 draws on average against a cap of 298. Producing 42+ rejections inside 298 draws is astronomically improbable, and the sampling input is derived from a SHAKE sponge over static per-parameter-set tables, so it is not a value an attacker can steer toward the reject region. Even if the cap were somehow reached, the only effect is a clean error return with no key produced — no corrupted-state use, no memory-safety fault, no key-integrity break.

Finding #2 — ML-DSA signature verification: same status plumbed into SymCryptMlDsaVerifyEx

What changed. The unpatched SymCryptMlDsaVerifyEx (@ 0x180089D9C) decodes the incoming signature inline (SymCryptMlDsaVectorDecode @ 0x180089EE4, SymCryptMlDsaHintBitUnpack @ 0x180089F0B) and, after the bounds check SymCryptMlDsaVectorInfinityNorm (@ 0x180089F1D), calls the challenge-sampling step SymCryptMlDsaSampleInBall (@ 0x180089F4F) and proceeds to SymCryptMlDsaVectorNTT (@ 0x180089F57) without testing a return value. The patched SymCryptMlDsaVerifyEx (@ 0x18008CBF8) consolidates the inline decode into a single SymCryptMlDsaSigDecode (@ 0x18008CD1F) call whose return is checked (jnz to cleanup at 0x18008CD28), keeps the same SymCryptMlDsaVectorInfinityNorm bounds check (@ 0x18008CD31), and adds a test eax,eax / jnz after SymCryptMlDsaSampleInBall (@ 0x18008CD62) (jnz to cleanup at 0x18008CD6B).

Why this is not a security fix. This is the verification-side face of the same library change. In the unpatched build SymCryptMlDsaSampleInBall (@ 0x180096DC8) loops over the SHAKE sponge until it places all challenge coefficients and returns with no status word, so it cannot fail; the added check therefore guards a condition the unpatched code cannot produce. The signature-decode consolidation is a refactor (SymCryptMlDsaSigDecode wraps the same decode steps), and the coefficient bounds check SymCryptMlDsaVectorInfinityNorm is present in both builds. The verification decision logic is unchanged: an invalid signature is rejected in both builds. Nothing here weakens or newly strengthens acceptance/rejection of signatures.

3. Pseudocode Diff

Finding #1 — SymCryptMlDsaKeyGenerateEx: unpatched @ 0x1800894B8 vs patched @ 0x18008C2F0

// ====== UNPATCHED: SymCryptMlDsaKeyGenerateEx @ 0x1800894B8 ======
// ExpandA / ExpandS have no failure return in this build; eax is not a status word.

SymCryptMlDsaExpandA(rsi + 0x56, r8 = *(rsi + 0xb8));     // no failure path in callee
rax = *(rsi + 0xd0);
SymCryptMlDsaExpandS(*(rsi + 0x8), rsp+0x40,
                     *(rsi + 0xc8), rax);                 // no failure path in callee

SymCryptMlDsaVectorNTT(*(rsi + 0xc8));                    // always reached
SymCryptMlDsaVectorNTT(*(rsi + 0xd0));
SymCryptMlDsakeyComputeT(/* args from rsi+0xb8..0xd8 */); // t = A*s1 + s2
SymCryptMlDsaVectorNTT(*(rsi + 0xd8));
SymCryptMlDsaVectorNTT(*(rsi + 0xc0));
SymCryptMlDsaPkEncode(rsi, *(rbp+0x128), *(r14+0x30));    // encode public key


// ====== PATCHED: SymCryptMlDsaKeyGenerateEx @ 0x18008C2F0 ======
// ExpandA / ExpandS now propagate the new bounded-iteration status; caller tests it.

ebx = SymCryptMlDsaExpandA(...);      // callee now returns 0x8008 if the draw cap is hit
if (ebx != 0) goto cleanup;           // *** ADDED: test eax,eax / jnz (0x18008C40F/0x18008C411) ***

ebx = SymCryptMlDsaExpandS(...);
if (ebx != 0) goto cleanup;           // *** ADDED: test eax,eax / jnz (0x18008C43A/0x18008C43C) ***

SymCryptMlDsaVectorNTT(...);
SymCryptMlDsakeyComputeT(...);
SymCryptMlDsaPkEncode(...);

The two added test eax,eax / jnz sequences are the entire behavioral delta in this function; the rest is call-target relocation from the recompilation. The guarded status only becomes non-zero because the callees changed (see Section 5), and that non-zero condition is not reachable in the unpatched build.

4. Assembly Evidence

SymCryptMlDsaKeyGenerateEx — unpatched, @ 0x1800894B8 (no return-value test around the expansions)

00000001800895C5  mov     r8, [rsi+0B8h]           ; key material pointer
00000001800895CC  lea     rcx, [rsi+56h]           ; arg for SymCryptMlDsaExpandA
00000001800895D0  call    SymCryptMlDsaExpandA     ; eax not tested (callee has no failure path)
00000001800895D5  mov     rax, [rsi+0D0h]
00000001800895DC  lea     rdx, [rsp+0C8h+var_88]
00000001800895E1  mov     r9, [rsi+0C8h]
00000001800895E8  mov     rcx, [rsi+8]
00000001800895EC  mov     [rsp+0C8h+var_A8], rax
00000001800895F1  call    SymCryptMlDsaExpandS     ; eax not tested (callee has no failure path)
00000001800895F6  mov     rcx, [rsi+0C8h]
00000001800895FD  call    SymCryptMlDsaVectorNTT
0000000180089602  mov     rcx, [rsi+0D0h]
0000000180089609  call    SymCryptMlDsaVectorNTT
...
0000000180089654  call    SymCryptMlDsakeyComputeT ; t = A*s1 + s2
...
000000018008967F  call    SymCryptMlDsaPkEncode    ; encode public key
0000000180089684  mov     ebx, eax                 ; PkEncode return IS captured/tested in both builds
0000000180089686  test    eax, eax
0000000180089688  jnz     short loc_1800896A4

SymCryptMlDsaKeyGenerateEx — patched, @ 0x18008C2F0 (return-value tests added around the expansions)

000000018008C3FD  mov     r8, [rsi+0B8h]
000000018008C404  lea     rcx, [rsi+56h]
000000018008C408  call    SymCryptMlDsaExpandA
000000018008C40D  mov     ebx, eax                 ; *** ADDED: capture status ***
000000018008C40F  test    eax, eax                 ; *** ADDED guard ***
000000018008C411  jnz     loc_18008C4F0            ;     skip keygen pipeline on non-zero
000000018008C417  mov     rax, [rsi+0D0h]
000000018008C41E  lea     rdx, [rsp+0C8h+var_88]
000000018008C423  mov     r9, [rsi+0C8h]
000000018008C42A  mov     rcx, [rsi+8]
000000018008C42E  mov     [rsp+0C8h+var_A8], rax
000000018008C433  call    SymCryptMlDsaExpandS
000000018008C438  mov     ebx, eax                 ; *** ADDED: capture status ***
000000018008C43A  test    eax, eax                 ; *** ADDED guard ***
000000018008C43C  jnz     loc_18008C4F0            ;     skip keygen pipeline on non-zero
000000018008C442  mov     rcx, [rsi+0C8h]
000000018008C449  call    SymCryptMlDsaVectorNTT

The two mov ebx,eax / test eax,eax / jnz sequences at 0x18008C40D–0x18008C411 and 0x18008C438–0x18008C43C are the only logical additions in the function.

SymCryptMlDsaVerifyEx — SampleInBall call site (Finding #2)

; ---- unpatched @ 0x180089D9C ----
0000000180089F4F  call    SymCryptMlDsaSampleInBall  ; eax not tested
0000000180089F54  mov     rcx, r14
0000000180089F57  call    SymCryptMlDsaVectorNTT     ; proceeds unconditionally

; ---- patched @ 0x18008CBF8 ----
000000018008CD62  call    SymCryptMlDsaSampleInBall
000000018008CD67  mov     ebx, eax                   ; *** ADDED: capture status ***
000000018008CD69  test    eax, eax                   ; *** ADDED guard ***
000000018008CD6B  jnz     loc_18008CFD0              ;     skip on non-zero
000000018008CD71  mov     rcx, rdi
000000018008CD74  call    SymCryptMlDsaVectorNTT

5. Root of the Change — the rejection-sampling helpers gained an iteration cap

The reason the callees now return a status is that the SymCrypt rejection-sampling loops were bounded in the patched build. This is the substantive part of the diff.

SymCryptMlDsaRejNttPoly — unpatched @ 0x180096CEC (unbounded loop, no return status)

0000000180096D6E  ...                                ; loop head
0000000180096D7F  call    SymCryptKeccakExtract      ; draw 3 bytes from the sponge
0000000180096D84  and     byte ptr [rsp+...+2], 7Fh
0000000180096D89  mov     eax, [rsp+130h+var_110]
0000000180096D8D  cmp     eax, 7FE001h               ; q = 8380417
0000000180096D92  jnb     short loc_180096D6E        ; reject -> draw again (no cap)
0000000180096D94  mov     [rdi], eax                 ; accept
0000000180096D96  add     rdi, 4
0000000180096D9A  sub     rbx, 1                     ; rbx counts 256 -> 0
0000000180096D9E  jnz     short loc_180096D6E
0000000180096DA0  ...                                ; epilogue: eax is never set to a status
0000000180096DC0  retn

SymCryptMlDsaRejNttPoly — patched @ 0x18009AA84 (bounded loop, returns 0x8008 on cap)

000000018009AB01  cmp     r14d, 12Ah                 ; *** ADDED cap: 298 draws ***
000000018009AB08  jnb     short loc_18009AB45        ;     exceeded -> error
...
000000018009AB1B  call    SymCryptKeccakExtract
000000018009AB25  inc     r14d                       ; *** ADDED: count draws ***
000000018009AB28  mov     eax, [rsp+140h+var_120]
000000018009AB2C  cmp     eax, 7FE001h
000000018009AB31  jnb     short loc_18009AB01
000000018009AB33  mov     [rsi], eax
000000018009AB35  inc     ebx
000000018009AB37  add     rsi, 4
000000018009AB3B  cmp     ebx, 100h
000000018009AB41  jb      short loc_18009AB01
000000018009AB43  jmp     short loc_18009AB4A
000000018009AB45  mov     edi, 8008h                 ; *** ADDED: error status ***
000000018009AB4A  mov     eax, edi                   ; return status in eax

The same edit appears in SymCryptMlDsaRejBoundedPoly (patched @ 0x18009A8C0: cap cmp esi, 1E1h = 481, returns 0x8008) and in SymCryptMlDsaSampleInBall (patched @ 0x18009AB78: cap cmp edi, 79h = 121). Their unpatched counterparts (@ 0x180096B4C, @ 0x180096DC8) loop without a cap and return no status word. SymCryptMlDsaExpandA (patched @ 0x180099B14) and SymCryptMlDsaExpandS (patched @ 0x180099CBC) were updated to capture and propagate that status (mov e?x, eax / test / jnz after the SymCryptMlDsaRejNttPoly / SymCryptMlDsaRejBoundedPoly calls), whereas the unpatched SymCryptMlDsaExpandA (@ 0x180095DD8) and SymCryptMlDsaExpandS (@ 0x180095F78) fall through to their epilogue without setting eax.

6. Impact Assessment

No exploit primitive. The report's guarding change does not correspond to a reachable fault:

• No corrupted-state use. In the unpatched build the expansions cannot return early, so the intermediate vectors at rsi+0xc8, rsi+0xd0, rsi+0xd8 are always fully populated before SymCryptMlDsakeyComputeT and SymCryptMlDsaPkEncode read them. There is no partial-fill or dangling-pointer condition to dereference.
• No key-integrity break. The unpatched sampling produces the same coefficients as the patched sampling; the only difference is that the patched code stops after a fixed number of draws. For any input that both builds accept, the generated key pair is identical.
• Not attacker-controllable. The new failure depends on an extreme rejection rate in a SHAKE-derived byte stream over static per-parameter-set tables. The rejection probability per draw for SymCryptMlDsaRejNttPoly is ≈ 0.098% (values >= 0x7FE001 out of 2^23); reaching the 298-draw cap requires 42+ rejections where ≈ 0.25 are expected, which is not achievable by steering a sponge whose output is pseudorandom.
• Failure is fail-closed anyway. Should the cap ever be hit in the patched build, the effect is a clean error return (0x8008) with no key or verification result produced — the safe direction.

Direction check. The direction is not reversed: the patched build is the stricter one (it adds the checks). But the stricter behavior only matters for a condition that the unpatched build never enters, so the delta carries no reachable security consequence.

7. How to Confirm Against the Binaries

The observations below can be reproduced statically from the two disassemblies; there is no runtime fault to trigger.

• Unpatched SymCryptMlDsaKeyGenerateEx (@ 0x1800894B8): no test eax,eax between the SymCryptMlDsaExpandA call at 0x1800895D0 and the SymCryptMlDsaExpandS call at 0x1800895F1, nor between that call and SymCryptMlDsakeyComputeT at 0x180089654. The SymCryptMlDsaPkEncode return at 0x180089684 is captured in both builds and is unrelated to the expansions.
• Patched SymCryptMlDsaKeyGenerateEx (@ 0x18008C2F0): mov ebx,eax / test eax,eax / jnz at 0x18008C40D–0x18008C411 (after SymCryptMlDsaExpandA) and 0x18008C438–0x18008C43C (after SymCryptMlDsaExpandS).
• Unpatched SymCryptMlDsaRejNttPoly (@ 0x180096CEC): the rejection loop at 0x180096D6E has no iteration counter and the epilogue at 0x180096DA0–0x180096DC0 never writes a status to eax.
• Patched SymCryptMlDsaRejNttPoly (@ 0x18009AA84): the cap cmp r14d, 12Ah / jnb at 0x18009AB01, the inc r14d at 0x18009AB25, and the error path mov edi, 8008h / mov eax, edi at 0x18009AB45–0x18009AB4A.
• SymCryptMlDsaVerifyEx: unpatched SymCryptMlDsaSampleInBall call at 0x180089F4F is followed directly by SymCryptMlDsaVectorNTT at 0x180089F57 with no test; patched adds mov ebx,eax / test eax,eax / jnz at 0x18008CD67–0x18008CD6B.

Key offsets in the ML-DSA key context (rsi) referenced by SymCryptMlDsaKeyGenerateEx

8. Changed Functions — Full Triage

The remaining changed functions fall into the same two buckets: (a) ML-DSA functions touched by the rejection-sampling-cap plumbing, and (b) call-target relocations and register-allocation shifts produced by recompilation. None contain a security-relevant logic delta.

9. Unmatched Functions

There are no added or removed functions (unmatched_unpatched: 0, unmatched_patched: 0). The patch is an in-place recompilation of existing routines — consistent with a crypto-library refresh rather than a targeted vulnerability fix.

10. Confidence & Caveats

Confidence: High that this is not a security-relevant change.

• The added test eax,eax / jnz guards in SymCryptMlDsaKeyGenerateEx and SymCryptMlDsaVerifyEx are real, but the status they test only becomes non-zero because SymCryptMlDsaRejNttPoly / SymCryptMlDsaRejBoundedPoly / SymCryptMlDsaSampleInBall were given an iteration cap. The unpatched versions of those helpers have no cap and no status return, so the guarded condition cannot occur in the unpatched build.
• The sampling input is SHAKE-sponge output over static per-parameter-set tables; the cap-exceeded condition is not an attacker-steerable primitive, and if it ever fired the result is a fail-closed error return.
• The verification decision logic (SymCryptMlDsaVectorInfinityNorm bounds check, hint-bit unpack, decode) is present and equivalent in both builds; signatures accepted/rejected identically.

Caveats:

1. The exact numeric caps (298 / 481 / 121 draws) and the 0x8008 status are read directly from the patched disassembly; they reflect a specific SymCrypt revision and are quoted only as evidence of the change, not as a tunable security boundary.
2. This assessment addresses the ML-DSA rejection-sampling delta and the recompilation churn in the remaining changed functions. No other function in the triage list shows a security-relevant logic change.