AI Infrastructure Security

TensorRT-LLM’s Restricted Unpickler Still Admitted Executable PyTorch Globals (CVE-2026-24233)

TensorRT-LLM's RLHF weight-update worker decoded a base64 pickle under a restricted unpickler, yet the call site admitted the entire torch module family through ^torch.* ; a dangerous global could therefore participate in object construction before the returned-list check, and release 1.3.0rc15 closes the confirmed path with a higher-priority exact deny list.

A warm hand-drawn checkpoint where a sealed pickle crate reaches a TensorRT-LLM worker, allowed globals queue at the workshop, and a shadowed callable is stopped by the repaired deny gate.
In this article

Research basisSOSEC AI Infrastructure Security Research · fix commit pinned to 634f86c3910f , with WorkerExtension.update_weights() , serialization.loads() , and Unpickler.find_class() reviewed against the parent revision

SourceNVIDIA security bulletin / TensorRT-LLM fix and parent commits / Python pickle semantics / SOSEC source review

1 A GPU tensor handle arrived at the worker as data, but the bytes still contained instructions for constructing Python objects

NVIDIA disclosed CVE-2026-24233 in July 2026 for TensorRT-LLM 1.3.0rc14 and earlier affected releases. The company assigns CVSS 3.1 score 8.4, High, with vector AV:L/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H and CWE-502. Version 1.3.0rc15 contains the vendor fix. Those labels are a useful starting point, but the source explains what authority enters the worker and why a seemingly restricted loader did not contain it.

The affected code supports a practical requirement in reinforcement-learning and continuously updated inference systems. A training process has fresh model weights already resident on a GPU. Copying every tensor through a neutral host representation would be expensive, so the producer exports CUDA inter-process communication handles. The inference worker receives parameter names, a reconstruction function, and its arguments, then opens the shared storage and reloads the model without a full conventional transfer.

At the receiving end, WorkerExtension.update_weights() selects the value associated with its current GPU UUID. When that value is a string, the worker treats it as a base64-encoded pickle, decodes it, and passes the bytes to TensorRT-LLM's own serialization.loads(). The call site approves several basic built-in classes and also supplies the module regular expression ^torch.*. That expression is the decisive fact in the vulnerability chain.

The custom unpickler defaults to denial and overrides find_class(), but an entire package-prefix rule allows far more than the small set of reconstruction helpers that the handle protocol needs. Pickle opcodes can resolve and invoke globals while loads() is still building the result. The later check that the result is a Python list occurs after those operations. It is useful protocol validation, but it cannot retroactively make object construction inert.

Fix commit 634f86c3910f81e6c5eae4e81dd27de3e823ab66 changes two source files. It threads a new disallowed_imports map through the unpickler and its convenience functions, then supplies three exact PyTorch globals at the RLHF call site. Most importantly, find_class() evaluates the deny map before either exact approval or the module regular expression. The patch preserves the broad compatibility rule while removing the vendor-confirmed dangerous capabilities from this path.

1.1 The local attack vector describes reachability, not a requirement for someone to sit at the server console

The bulletin's AV:L rating should prevent one inaccurate headline: this is not an unauthenticated Internet-facing HTTP remote-code-execution flaw by itself. The vulnerable operation lives in a worker-side weight-update channel. Whether an external tenant or compromised service can influence that channel depends on Ray topology, orchestration, job permissions, queue ownership, network policy, and the bridge between training and inference environments.

Likewise, PR:N has a precise CVSS meaning once the local vector has been satisfied. It does not say that an arbitrary Internet user can submit ipc_handles. An inventory should identify principals that can run code on a participating host, publish or alter a weight-update message, call the relevant collective control action, compromise a permitted training job, or write an artifact later consumed by the producer. Those are the useful operational questions.

UI:N agrees with the code path: a worker processes the update without an administrator opening a file or approving a prompt. Confidentiality, integrity, and availability are all rated High because the unpickler runs inside a process that can hold model weights, GPU mappings, service credentials, caches, and access to adjacent control services. The process identity and container permissions determine how far any behavior can travel.

The bulletin itself uses two platform labels that should be recorded rather than reconciled by assumption: its description names “TensorRT-LLM for Linux,” while the Security Updates row containing CVE-2026-24233 marks “Platform or OS” as “All.” Asset decisions should therefore follow the actual installed TensorRT-LLM package and vulnerable path together with the vendor's update table, not infer exclusion from an operating-system label alone. Version checks need to be made inside running images. A container tag may describe a product release while the installed prerelease wheel differs; an internal image may also carry a backport. Record tensorrt_llm.__version__, the wheel hash, image digest, SBOM, and hashes of rlhf_utils.py and serialization.py. For a backport, require equivalent ordering and tests, not merely a vendor commit string in a changelog.

Equivalent semantics mean that the unpickler accepts a deny map, checks it before all approvals, and receives the three exact entries from the affected RLHF call site. The load() and loads() wrappers must carry the map without dropping or transforming it. A distributor can use a different commit identifier, but it cannot change that execution order and still claim to have the same security property.

1.2 Fast RLHF weight replacement moved trust from a model file to a handle protocol

Large-model training and serving systems update billions of parameters. Materializing a complete neutral copy, moving it through host memory, and allocating it again on every serving GPU imposes a measurable pause. CUDA IPC lets one process export a representation of device memory so another process on a compatible host and device context can reconstruct access. That performance choice is sound when the producer, metadata, lifetime, and receiver are governed as one protocol.

TensorRT-LLM injects WorkerExtension through ray_worker_extension_cls and uses a collective control call for update_weights. The method is decorated as a control action and waits for active work under the surrounding lifecycle. This is not a normal inference payload. It is a privileged operation that changes the model executed for future requests, resets related state, and coordinates multiple workers.

The ipc_handles object is keyed by device UUID. For a device, the logical payload contains entries resembling (param_name, tensor_handle); a tensor handle contains a Python callable and an argument tuple. Pickle is attractive because it preserves container shapes and references to importable Python globals. That same feature makes the callable part of the input's authority, not a passive data field.

After deserialization, the receiver copies the arguments, replaces element six with its own device identifier, and executes func(*list_args). The function object is therefore an intentional protocol field. A future neutral format must represent the desired reconstruction operation in some other way, preferably a small versioned opcode whose implementation is selected from receiver-owned code. Merely replacing base64 with binary transport would leave the capability problem untouched.

The method later invokes model reload with partial-loading support, performs CUDA IPC cleanup, and may run finalization hooks, load-balancer work, prefix-cache changes, and synchronization. A malformed update can fail after the worker has begun a state transition. Security review must cover both code-execution capability and transaction behavior: which stages have run, which workers changed, and how a failed batch returns to a known model.

1.3 The code and response map: vulnerable function, pinned lines, release branch, repair decision, and temporary controls

2 update_weights() crosses model lifecycle, device selection, decoding, object construction, and explicit invocation in one function

The method first checks whether the model has completed its initial pre-reload preparation. On the first update it traverses modules, calls supported pre_reload_weights hooks, and records a marker. That sequencing matters during incident reconstruction: an exception in the later decode path may occur after preparatory state has changed, even though no new tensor was successfully loaded.

When handles are present, the code computes get_device_uuid(self.device_id) and requires a matching dictionary key. A missing key raises an error before decoding. This prevents an accidental handle set for another GPU from being used blindly, but it does not authenticate the dictionary, its producer, or the semantics of the selected value. Device selection is a consistency control, not input authorization.

The selected value is assigned to serialized_handles. A string enters base64 decoding and restricted pickle loading. A non-string value follows a compatibility branch and is used directly as all_handles. The latter branch avoids this particular pickle call but does not automatically create a safe path; investigators must find where the RPC framework or producer constructed the object and whether an untrusted caller can place a callable inside it.

Base64 performs an encoding conversion and nothing more. It can reject malformed text, and its length is useful for resource limits, but it does not authenticate the producer or constrain pickle opcodes. The code-execution decision begins when the decoded bytes reach the unpickler. Signing can authenticate an approved producer, but a signature does not reduce the behavior of an overpowered producer or a compromised signing service.

The function catches exceptions, logs an update error, and rethrows. A generic final message is insufficient security telemetry. A safe implementation should log a request or batch identifier, producer identity, device UUID, input hash, policy version, failure stage, and rejected module/name without printing the original pickle or an uncontrolled object representation. That evidence allows teams to correlate all workers that saw the same bytes.

2.1 The request path joins decoding, reconstruction, list timing, and the later callable step

The first stage is selection: device_uuid → ipc_handles[device_uuid]. The second is transport decoding: base64 text becomes pickle bytes. The third is interpretation: serialization.loads() turns those bytes into a Python object graph. The fourth unpacks application fields: parameter name, callable, and arguments. The fifth invokes the callable, creates tensors, and reloads model parameters. Treating the whole path as “loading weights” hides its distinct trust decisions.

CVE-2026-24233 is rooted in the third stage, but its consequences reach the fifth. During pickle interpretation, a global can be resolved and a reduction callable can run before the outer function receives all_handles. A callable that survives as an element can then run again when application code explicitly executes func(*args). Static review has to join pickle virtual-machine semantics to ordinary Python control flow.

The rc15 patch places exact barriers at the third stage for three confirmed globals. A stronger application schema would also protect the fourth and fifth stages by requiring each entry to name a receiver-defined operation and validating every argument before model preparation. These controls are complementary: restricting global construction does not fully validate the model protocol, while validating a returned list cannot undo earlier pickle activity.

The device rewrite deserves its own invariant. The code assumes the arguments have at least seven entries and changes index six to self.device_id. Tests should prove the expected tuple layout across supported PyTorch and CUDA combinations, show that the original field represents the device slot, and reject short or differently typed arguments before any reload hook. Otherwise a compatibility fault will appear later as an opaque CUDA failure.

Five-stage request path from device UUID selection through base64 decoding, restricted unpickling, handle unpacking, and callable execution into a tensor.
The returned-list check sits after pickle construction. It can validate the container that survived, not the instructions already interpreted to create it.

2.2 Pickle is a compact object-construction program, not a neutral mapping from bytes to fields

Python's documentation warns against unpickling untrusted data because the format preserves object graphs, shared references, importable classes and functions, and reduction recipes. The unpickler behaves like a stack machine. It reads opcodes, pushes values, resolves globals, calls reconstruction operations, mutates objects, and finally returns the top-level result. A valid pickle can therefore do meaningful work before any application-level type check runs.

GLOBAL and STACK_GLOBAL request a module and name. The unpickler calls find_class(module, name), despite the historic method name applying to functions as well as classes. REDUCE takes a callable and argument tuple from the stack and invokes them. Other opcodes build instances or state. A restriction is effective only if it removes dangerous capabilities before the corresponding operation receives them.

Overriding find_class() is a legitimate way to reduce pickle's authority, but “restricted” is an implementation objective rather than a result. The exact allowed module/name pairs, regular expressions, runtime dependency exports, and post-load uses determine the real authority. A single module-family expression can admit hundreds of symbols and can change meaning when a dependency updates without any edit to the application repository.

Checking whether the final result is a list addresses a different question: can later application code iterate the expected top-level container? If object construction already invoked a callable, rejecting the finished result is too late. A list can also contain a function and malformed argument graph that reaches the explicit invocation phase. The interpreter policy and the returned schema therefore require separate validation.

Encoding, compression, encryption, and signing do not alter these semantics. Encryption protects confidentiality in transit or at rest; a signature can bind bytes to an identity; base64 carries binary through a string field. Once the receiver has accepted and decoded the message, it still executes the pickle program. The producer's authorization must be narrow enough to justify that authority, or the format must be replaced.

At this point, the question is no longer whether pickle is dangerous in the abstract. It is which capability set the RLHF call actually hands to find_class(). The next chapter follows that policy rather than repeating the format warning.

3 Default denial still worked; the RLHF call site widened the capability set through a package-level exception

tensorrt_llm/serialization.py wraps standard pickle, maintains an exact BASE_EXAMPLE_CLASSES map, and exposes register_approved_class(). Before the fix, Unpickler accepted exact approvals and module patterns. find_class() called the parent resolver only when one matched and otherwise raised ValueError.

The problem was therefore not the absence of a restricted loader. Its effective policy came from the arguments supplied by each caller. The RLHF worker provided a small built-ins map and a module rule covering the torch namespace, restoring globals beyond the narrow handle grammar for the sake of compatibility.

load() and loads() are parallel entries: one handles a file object, while the other wraps bytes in io.BytesIO; both construct the custom Unpickler directly. Neither first converts input into a non-executing structure or adds object-graph limits beyond find_class(), so the repair parameter has to reach both wrappers independently.

The base map also serves other serialization contexts, while RLHF constructs a local policy. Release rc15 accordingly gives the common Unpickler a deny capability and has this business entry supply three denied pairs. It does not claim to redesign every pickle use in the repository.

3.1 The effective policy is an ordered allow-and-deny decision whose meaning moves with PyTorch

The code uses re.match, so the expression begins at the start of the module string. It matches torch, torch.storage, torch.hub, and every similarly prefixed importable module. Once the module matches, the pre-fix branch does not constrain the requested name. The parent unpickler imports and returns the actual global if Python can resolve it in that environment.

The engineering motivation is understandable. Tensor reconstruction helpers and internal storage names can vary by PyTorch version, tensor type, and execution path. Maintaining a precise list can break a supported training combination. A package prefix minimizes compatibility friction and enables legitimate CUDA IPC objects to survive updates. Security review must preserve that operational requirement while making the capability set explicit.

The cost is policy drift. Upgrading PyTorch can add or re-export a callable under the permitted namespace. TensorRT-LLM's source and configuration remain unchanged, yet the globals reachable through find_class() expand. An SBOM and pinned wheel hashes therefore belong to this control, not merely to general supply-chain hygiene. Dependency changes are authorization-policy changes for any module-pattern unpickler.

Release 1.3.0rc15 takes a narrow compatibility-preserving approach. It retains the regular expression and subtracts three exact module/name pairs. That closes the vendor-confirmed paths without forcing an immediate enumeration of every legitimate tensor reconstruction helper. The ordering of subtraction before approval is what makes the patch meaningful.

A measured path toward a smaller allow set starts with legitimate corpus capture. Run supported training-to-worker updates, record each module/name requested by find_class(), associate it with a tensor form and dependency matrix, and build an exact versioned policy. Any new global introduced by an upgrade then becomes a review event rather than an unobserved consequence of a package prefix.

3.2 Pickle reconstruction and later invocation occur at separate stages; outer type checks arrive between them

Phase A takes place inside serialization.loads(decoded_data). The pickle machine reads a global opcode, calls find_class(), and may invoke a resolved callable through a reduction before the outer function receives a result. Neither isinstance(all_handles, list) nor the subsequent loop can inspect or prevent activity that has already occurred at this point.

Phase B takes place in the application loop. The code unpacks func, args, copies the arguments, rewrites the device field, and deliberately calls func(*list_args). A global need not have a side effect during unpickling to be dangerous if it survives as the value of func. The handle grammar must therefore authorize the function identity, not merely the container type.

The rc15 deny map operates at Phase A. It prevents the three globals from being resolved into the object graph and consequently prevents them from reaching Phase B through this pickle path. Other admitted globals remain governed by the package expression. A later hardening patch could compare every callable against a receiver-owned exact map before executing it, giving Phase B a separate and visible check.

The returned-list check lies between these phases. It is useful because it stops an unexpected top-level object from flowing into the loop. It does not validate the number of entries, tuple arity, parameter name, argument count, storage size, shape, dtype, or callable identity. Those checks should occur before any model-state preparation and should produce distinct, searchable failure reasons.

Comparison of pickle-time global resolution and REDUCE execution with the later explicit func-args invocation in the weight-update loop.
Phase A finishes before loads() returns. Phase B is an intentional part of the weight-handle protocol. Both need an explicit capability set.

Once the two execution stages are separated, the two-file repair follows naturally: the serialization helper gains a deny-first decision, and the RLHF call site supplies the three exact capabilities that this business path must subtract.

4 The fix gives both load() and loads() a direct deny-map path into Unpickler, then supplies it at the RLHF call site

In serialization.py, the constructor gains a disallowed_imports parameter and stores an empty map when callers omit it. The change is backward compatible for existing users. In find_class(), the new first branch tests whether the requested name appears in the denied list for its module. A match raises ValueError before reaching any approval.

The exact-approved branch and regular-expression loop remain in place after the new check. This order creates a simple precedence rule: deny overrides exact allow, and deny overrides pattern allow. Without that order, the RLHF call site's broad module rule would return a blocked global before a later deny branch could see it. Merely adding a deny dictionary somewhere in the function would not be sufficient.

The module-level load() function accepts the new parameter and passes it directly to the Unpickler it constructs for a file object. The byte-oriented loads() function independently accepts the parameter, wraps the bytes in io.BytesIO, and passes the map directly to its own Unpickler. This parallel plumbing is security-critical. Either wrapper silently dropping the map would leave that entry point on the old policy. Tests should cover both public functions, not instantiate only the class.

In rlhf_utils.py, the worker builds the three-entry map next to the deserialization call and supplies it explicitly. The original built-in approvals and broad module rule remain. The public fix changes these two source files; downstream acceptance still needs policy and compatibility coverage.

The parent revision is essential evidence. It confirms the prior call used broad torch approval, performed the list check only after loads(), then iterated entries and called the deserialized function. Reviewing only the fixed file can show that a deny list exists; comparing the parent proves what capability was previously admitted and which data flow remains after the patch.

Three find_class decisions: an exact denial raises ValueError, either approval path returns through super().find_class, and a value matching no approval raises ValueError.
The complete built-ins list includes NoneType. A deny match raises immediately; an exact or module-pattern approval returns through super().find_class(); a value matching neither approval is rejected without reaching the parent resolver.

4.1 The patch closes the confirmed capability path without taking over identity, protocol, or model-transaction design

Rc15 narrows global resolution after input reaches the RLHF deserializer. It does not authenticate the principal invoking the Ray control action, approve a training job, or bind an update to a worker set. Nor does it replace the remaining package-level approval with a complete exact list. Upgrade is the immediate decision; control-plane identity and a corpus-derived operation map are separate hardening tasks.

The commit adds no encoded or decoded byte budgets, opcode, nesting, entry-count, or total-tensor limits. It also does not validate each parameter name, tuple shape, args layout, tensor shape and dtype, storage bounds, or callable identity. Correctly denying the three named capabilities can coexist with availability or protocol failures from other malformed data, so those limits belong to the actual handle schema.

The non-string compatibility branch uses an existing Python object and never enters the repaired loader; its safety depends on how RPC and the producer created that object. Both formats later share func(*args). Other standard-pickle, PyTorch-serialization, or differently configured callers elsewhere in the repository likewise do not inherit this local policy automatically. An application-level exact operation check can cover both formats, while out-of-scope findings need their own evidence.

The model transition is not made atomic by this patch: preparation may begin first, tensors are restored one by one, partial loading is allowed, and later control actions finish the lifecycle. Writing rc15 to disk also does not change an interpreter that already imported the old module. Rollout must restart or replace every worker, including elastic templates and cached nodes, and prove one model and policy identity across the fleet.

Finally, the bulletin and commit establish potential impact and a source-level capability change; they do not establish that any organization was exploited. Runtime conclusions require update, host, and model evidence. That distinction produces a clear sequence: upgrade or backport, pause or tightly mediate updates and reduce worker privilege while deploying, then complete identity, schema, resource, and transaction hardening.

4.2 A durable protocol removes callable choice while preserving bounded CUDA IPC reconstruction

The receiver appears to need a limited operation: rebuild a tensor view over CUDA IPC-backed storage using known metadata. Define that grammar explicitly. A message can contain protocol version, operation enum, parameter name, producer and target device identifiers, handle bytes, size, offset, shape, stride, dtype, storage metadata, batch ID, nonce, and integrity information. Each field receives a type and bound.

The worker maps the operation enum to a fixed internal function. Input never names a module or Python export. Unknown operations fail before model preparation. Adding an operation requires code review, tests, and a protocol version. This arrangement gives compatibility changes a visible place and removes dependency namespace growth from the authorization decision.

Authenticity belongs around the structured message. Bind the signature or message authentication code to the complete field set, intended cluster, device set, model version, expiry, and nonce. Enforce replay protection and producer authorization. A valid signature says which approved training service created the request; the receiver-owned opcode map limits what that service may ask the worker to do.

Validate the entire batch before calling any pre-reload hook. Check parameter names against an approved manifest, total and per-entry bytes, dimensions, dtypes, duplicate keys, device mapping, and whether partial loading is allowed for this release. Only after all workers report validation readiness should the controller commit the update. A transaction identifier lets the cluster distinguish prepared, committed, and rolled-back states.

Migration can run in observation mode. Decode the old legitimate path in a controlled producer, emit the new representation alongside it, compare reconstructed tensors in test and canary environments, and record unsupported cases. Do not keep indefinite silent fallback to pickle; fallback would preserve the old authority. Publish a cutoff date and measurable usage until no supported producer depends on it.

5 Practical exposure is the intersection of an affected worker, an update path, a controllable producer, and meaningful process privileges

Start with assets rather than CVE scanners. Locate every image, virtual environment, training node, inference worker, offline evaluator, and disaster-recovery template that includes TensorRT-LLM. Record version and source hash from a running instance. Then identify which of those deployments enable RLHF or another route to WorkerExtension.update_weights(). An affected package with no reachable call path remains an upgrade target but has different immediate urgency.

For each reachable deployment, draw the producer chain: who creates CUDA IPC handles, where serialization occurs, which queue or RPC transports them, what identity calls the control action, and which network principals can reach it. Include CI runners, notebooks, scheduling services, shared Ray jobs, and artifact stores. A nominally local vector can be reachable through a compromised tenant job or overly broad internal service.

Next inventory the worker's authority. Capture operating-system user, Linux capabilities, writable mounts, service-account or cloud role, secrets, metadata access, egress, Ray privileges, container runtime socket access, and access to model repositories. These facts bound confidentiality and integrity impact far better than a generic statement that “code runs locally.” GPU access and model-serving control are themselves valuable even in a heavily sandboxed container.

Finally, examine update frequency and observability. A system that changes weights hourly through several training pipelines offers more opportunities and more baseline data than an offline cluster updated manually once a month. Determine whether the organization can identify every historical caller and payload hash. Missing provenance raises incident-review priority because an accepted update cannot be traced back to an approved training output.

Use the intersection to schedule work. An exposed multitenant worker with a low-trust producer and a powerful cloud identity belongs in the first response window. A disabled feature in an immutable test image can follow after production, provided policy actually prevents the path from being enabled. Document the control, owner, and expiry rather than relying on an oral claim that the feature is unused.

5.1 Detection follows update intent from producer and payload identity to host and model behavior

At the control plane, record caller identity, training job, approved model version, request ID, target cluster, device set, protocol version, encoded and decoded sizes, and payload hash. An update without an approved change record, from a new principal, outside the expected training window, or targeting an unusual worker set is suspicious before any pickle-specific signal appears.

At the unpickler, record a bounded event for every rejected module/name, the policy version, and the same payload hash. A hit on any of the three rc15 entries deserves immediate triage. Also baseline legitimate globals. A previously unseen torch global after a dependency or producer change is not proof of exploitation, but it is a capability change that needs review before being normalized.

At the host, correlate update time with child processes, file writes, module loads, outbound connections, credential access, cloud API activity, runtime-socket use, and changes to application code or configuration. The worker may be a long-running Python process, so process creation alone is not a complete signal. Fileless network or cloud behavior and unexpected imports deserve equal attention.

At the model layer, record actual parameter names, missing and additional keys, total bytes, model checksum or attestation, reload result, canary output, and worker-wide consistency. A malicious but protocol-valid weight update may not trigger a deserialization alert. Conversely, a rejected pickle should leave the serving model at a known version. Both claims need evidence.

Detection rules should distinguish stages: base64 rejection, pickle-policy rejection, top-level type failure, entry-schema failure, reconstruction failure, reload failure, collective inconsistency, and post-load validation failure. Collapsing them into one update exception produces noisy alerts and hides the precise point at which an input gained or lost authority.

5.2 Containment stops low-trust production, narrows worker authority, and stages a controlled rollout

The strongest temporary control is to pause dynamic weight updates from untrusted or weakly governed producers. Continue serving a known model if operationally acceptable, or route updates through a tightly controlled and authenticated service. Do not attempt to scan base64 text for function names; pickle encodings and object graphs make substring filters an unreliable security control.

Restrict the Ray control channel and any message queue to named service identities. Remove shared tenant ability to invoke control actions, bind listeners to the intended network, require mutual authentication where supported, and rotate credentials that were broadly distributed. A feature flag is useful only if the server enforces it and produces evidence that the vulnerable path cannot execute.

Reduce worker impact by running non-root, dropping unnecessary capabilities, using read-only application and model mounts where feasible, separating writeable cache space, blocking container-runtime sockets, narrowing cloud roles, denying metadata access not required by the job, and restricting egress. These measures do not repair the loader, but they constrain what an unexpected callable can reach.

Preserve update records before changing the environment. Save control-plane logs, Ray events, image and package hashes, update payload hashes, worker identities, process telemetry, and model versions. Store suspicious bytes in an isolated evidence repository and never call ordinary pickle.loads() during triage. Static opcode inspection and sandboxed harmless policy replay come first.

Do not roll back from rc15 to a vulnerable image because of a compatibility failure. Hold the previous trusted model on a fixed, patched runtime while engineers reproduce the legitimate handle set. An operational rollback target must include both a known model and a known-safe deserialization policy; version labels alone are insufficient.

5.3 Verification joins deny precedence, harmless reproduction, legitimate IPC, and fleet evidence

Begin with provenance. Verify the running package reports 1.3.0rc15 or a reviewed backport, the image digest matches the release record, and the two source files contain the expected semantics. An AST or targeted unit check is stronger than grepping for blocked names because it can verify that the deny branch precedes the allow branches and that wrappers propagate the parameter.

Run the harmless truth table for find_class(). Then test each NVIDIA-listed module/name and assert rejection before parent resolution. Use local test doubles or monkeypatching to avoid calling actual storage, Hub, or save functions. Verify that logs contain the controlled identity and input hash but no full payload. Run the same tests for class, load(), and loads() entry points.

Compatibility validation must use a handle generated by the real supported training path. Exercise every required GPU architecture, PyTorch version, Python version, CUDA version, and TensorRT-LLM combination. Confirm device UUID selection, index-six replacement, tensor shape and dtype, parameter names, partial-loading policy, model reload, finalization, cache state, and identical canary inference across workers.

Test failure transactions. Reject a pickle before deserialization completes, reject a structurally invalid returned list, fail one handle reconstruction, fail reload on one worker, and interrupt a collective operation. For every case, document whether the old model continues serving, whether any worker changed parameters, how the controller detects divergence, and which automated recovery action is safe.

Finally, test containment. The worker should be unable to write application code, obtain unrelated cloud secrets, reach unrestricted networks, or manage the cluster even if a harmless test double attempts those actions. Application policy and operating-system policy are separate evidence layers. Passing only the three blocked-global tests leaves excessive process authority unexamined.

6 A restricted unpickler is only as narrow as its final capability set, not the adjective in its class name

CVE-2026-24233 is not a story about a project forgetting to use a custom loader. A package-level exception introduced for PyTorch handle compatibility became broad enough to dominate an otherwise clear default-deny design, and the outer list check arrived only after object construction.

Release rc15 adds deny precedence to the common Unpickler and has the RLHF entry subtract the three vendor-confirmed pairs, closing the known path while retaining CUDA IPC compatibility. Affected releases need the fixed version or a semantically equivalent backport, and rollout must prove that every serving worker loaded the new module.

Acceptance has two sides. Harmless policy tests show that denial occurs before the parent resolver; producer-generated handles show that legitimate multi-GPU updates, model output, and fleet consistency remain intact. Control-plane identity, least-privilege workers, and retained update evidence connect the source repair to an operable production control.

The durable design carries only versioned operations and bounded data, with receiver-owned code selecting the one permitted reconstruction function. New exports in a dependency can no longer enlarge protocol authority silently, and compatibility becomes an explicit version-and-test decision.

Research record

7Evidence, objects, and sources

The material below preserves the identifiers and references used in this report.

7.1Research objects

Products, actors, techniques, affected objects, and control points discussed in the report.

VulnerabilityCVE-2026-24233

TensorRT-LLM restricted-deserialization bypass

Affected throughTensorRT-LLM 1.3.0rc14 and earlier

NVIDIA bulletin affected range

Fixed inTensorRT-LLM 1.3.0rc15

Vendor fixed version

Fix commit634f86c3910f81e6c5eae4e81dd27de3e823ab66

Tighten allowed type

Broad allow ruleapproved_module_patterns=[r'^torch.*']

Permits the PyTorch module family

Denied globaltorch.storage._load_from_bytes

Denied before the torch pattern

Denied globaltorch.hub._load_local

Denied before the torch pattern

Denied globaltorch.save

Denied before the torch pattern

Function chainupdate_weights → b64decode → serialization.loads → Unpickler.find_class → func(*args) → model_loader.reload

Complete input and execution chain

7.2Event chronology

  1. Fix entered upstream

    Commit 634f86c added deny precedence and the RLHF-local map.

  2. NVIDIA bulletin published

    CVE-2026-24233 was disclosed with rc15 identified as fixed.

  3. SOSEC completed parent/fix comparison

    The handle, deserialization, and explicit-call path was reviewed line by line.

7.3Sources and material

  1. NVIDIA TensorRT-LLM July 2026 security bulletinhttps://nvidia.custhelp.com/app/answers/detail/a_id/5840
  2. CVE-2026-24233 recordhttps://www.cve.org/CVERecord?id=CVE-2026-24233
  3. TensorRT-LLM fix commit 634f86chttps://github.com/NVIDIA/TensorRT-LLM/commit/634f86c3910f81e6c5eae4e81dd27de3e823ab66
  4. Direct vulnerable parent commit bf61b67https://github.com/NVIDIA/TensorRT-LLM/commit/bf61b672597c21d480c0f3005218badb05df4737
  5. Fixed TensorRT-LLM serialization.py, commit-pinned lineshttps://github.com/NVIDIA/TensorRT-LLM/blob/634f86c3910f81e6c5eae4e81dd27de3e823ab66/tensorrt_llm/serialization.py#L128-L211
  6. Fixed TensorRT-LLM RLHF WorkerExtension, commit-pinned lineshttps://github.com/NVIDIA/TensorRT-LLM/blob/634f86c3910f81e6c5eae4e81dd27de3e823ab66/tensorrt_llm/llmapi/rlhf_utils.py#L37-L132
  7. TensorRT-LLM v1.3.0rc15 releasehttps://github.com/NVIDIA/TensorRT-LLM/releases/tag/v1.3.0rc15
  8. TensorRT-LLM upstream repositoryhttps://github.com/NVIDIA/TensorRT-LLM
  9. Python pickle documentation and security warninghttps://docs.python.org/3/library/pickle.html
  10. Python pickletools non-executing inspection documentationhttps://docs.python.org/3/library/pickletools.html
  11. CPython 3.13 pickle implementationhttps://github.com/python/cpython/blob/3.13/Lib/pickle.py
  12. PyTorch serialization semanticshttps://pytorch.org/docs/stable/notes/serialization.html
  13. PyTorch torch.save documentationhttps://pytorch.org/docs/stable/generated/torch.save.html
  14. PyTorch Hub documentationhttps://pytorch.org/docs/stable/hub.html
  15. Ray actors and workershttps://docs.ray.io/en/latest/ray-core/actors.html
  16. NVIDIA CUDA runtime IPC-related interfaceshttps://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__DEVICE.html
  17. CWE-502: Deserialization of Untrusted Datahttps://cwe.mitre.org/data/definitions/502.html