Reference Architecture for Multi-Tenant CXL Memory Disaggregation: Choosing Between Direct-Attach, Switched Pooling and Fabric Sharing

Hyperscale operators adopting CXL must decide whether tenant memory should stay on direct-attached Type 3 expanders (CXL 1.1/2.0), be pooled behind a single-level switch with Multi-Logical Devices (CXL 2.0), or be shared and dynamically reassigned across a fabric using Dynamic Capacity Devices (CXL 3.x). Each step adds capacity elasticity, but also more latency, a larger blast radius and more management dependencies. Interoperability and isolation should be anchored to the published specifications, not vendor claims: CXL 3.1/3.2 (IDE link encryption, TSP for confidential computing), PCIe 5.0/6.0 physical layers, DMTF SPDM 1.2+ for device attestation, DMTF Redfish plus the CXL Fabric Manager API for orchestration, and ACPI 6.5 CEDT/HMAT with CXL CDAT for exposing topology to the host OS.

Decision criteria (direct vs. pooled vs. fabric)

Keep memory direct-attached when per-host demand is predictable and tail latency is the primary SLO. Move to MLD pooling under a CXL 2.0 switch when fleet-wide stranded DRAM is material and workloads tolerate an extra switch hop. Consider CXL 3.x fabric/DCD sharing only when every host CPU, switch and device in the path claims 3.x feature support you have verified yourself (DCD, back-invalidate, multi-headed devices), and when your orchestrator can drive Fabric Manager API calls with transactional rollback.

Pre-implementation checklist

Collect, per SKU, the supported CXL spec revision and optional features; PCIe link width/speed and the retimer budget; the IDE mode (skid vs. containment) and key-management path; SPDM version and certificate chain; CDAT-reported latency/bandwidth; the BIOS CEDT/CFMWS window layout and interleave granularity; Linux CXL driver/cxl-cli version; and RAS reporting (poison, CPER, viral handling). Reject any component whose attestation or isolation feature you cannot test end to end before tenant onboarding.

Illustrative analytical scenario (not a measurement)

Rack of 16 hosts, one CXL 2.0 switch, 4 MLD expanders with 16 logical devices each. Tenants are bound 1:1 to LDs, and the link is IDE-protected with SPDM-attested devices. Failure handling: if a switch or expander fails, the FM marks the affected LDs offline, hosts receive poison or surprise-removal events, and the orchestrator evacuates or restarts the affected VMs on local-DRAM capacity reserved at roughly 1/N headroom. Roll out gradually: start with non-latency-critical tiers (caches, cold heaps), then expand only once error-rate and reassignment-time SLOs hold.

How do we verify that one tenant cannot read or retain another tenant's memory after a CXL capacity reassignment on cxlcloud-style infrastructure?

Treat isolation as a chain you verify link by link. (1) Attest each device with SPDM before binding, and pin the measured firmware. (2) Make sure the FM assigns the LD or DCD extent to exactly one host, then read the binding back through Redfish/FM API. (3) Require a sanitize or secure-erase command (CXL memory-device Sanitize/Secure Erase) or a verified zeroing pass before re-releasing capacity, and log its completion status. (4) Enable IDE on every link segment, and use TSP where confidential VMs are in scope. (5) In pre-production, run negative tests: attempt cross-host access, inject surprise removal, and confirm poison propagation. If any step fails, quarantine the extent and do not return it to the free pool.

Back to CXLCloud · Domains

← Back to CXLCloud
Engineering & Architecture Brief

Reference Architecture for Multi-Tenant CXL Memory Disaggregation: Choosing Between Direct-Attach, Switched Pooling and Fabric Sharing

Hyperscale operators adopting CXL must decide whether tenant memory should stay on direct-attached Type 3 expanders (CXL 1.1/2.0), be pooled behind a single-level switch with Multi-Logical Devices (CXL 2.0), or be shared and dynamically reassigned across a fabric using Dynamic Capacity Devices (CXL 3.x). Each step adds capacity elasticity, but also more latency, a larger blast radius and more management dependencies. Interoperability and isolation should be anchored to the published specifications, not vendor claims: CXL 3.1/3.2 (IDE link encryption, TSP for confidential computing), PCIe 5.0/6.0 physical layers, DMTF SPDM 1.2+ for device attestation, DMTF Redfish plus the CXL Fabric Manager API for orchestration, and ACPI 6.5 CEDT/HMAT with CXL CDAT for exposing topology to the host OS.

Decision criteria (direct vs. pooled vs. fabric)

Keep memory direct-attached when per-host demand is predictable and tail latency is the primary SLO. Move to MLD pooling under a CXL 2.0 switch when fleet-wide stranded DRAM is material and workloads tolerate an extra switch hop. Consider CXL 3.x fabric/DCD sharing only when every host CPU, switch and device in the path claims 3.x feature support you have verified yourself (DCD, back-invalidate, multi-headed devices), and when your orchestrator can drive Fabric Manager API calls with transactional rollback.

Pre-implementation checklist

Collect, per SKU, the supported CXL spec revision and optional features; PCIe link width/speed and the retimer budget; the IDE mode (skid vs. containment) and key-management path; SPDM version and certificate chain; CDAT-reported latency/bandwidth; the BIOS CEDT/CFMWS window layout and interleave granularity; Linux CXL driver/cxl-cli version; and RAS reporting (poison, CPER, viral handling). Reject any component whose attestation or isolation feature you cannot test end to end before tenant onboarding.

Illustrative analytical scenario (not a measurement)

Rack of 16 hosts, one CXL 2.0 switch, 4 MLD expanders with 16 logical devices each. Tenants are bound 1:1 to LDs, and the link is IDE-protected with SPDM-attested devices. Failure handling: if a switch or expander fails, the FM marks the affected LDs offline, hosts receive poison or surprise-removal events, and the orchestrator evacuates or restarts the affected VMs on local-DRAM capacity reserved at roughly 1/N headroom. Roll out gradually: start with non-latency-critical tiers (caches, cold heaps), then expand only once error-rate and reassignment-time SLOs hold.

Technical FAQ: How do we verify that one tenant cannot read or retain another tenant's memory after a CXL capacity reassignment on cxlcloud-style infrastructure?

Treat isolation as a chain you verify link by link. (1) Attest each device with SPDM before binding, and pin the measured firmware. (2) Make sure the FM assigns the LD or DCD extent to exactly one host, then read the binding back through Redfish/FM API. (3) Require a sanitize or secure-erase command (CXL memory-device Sanitize/Secure Erase) or a verified zeroing pass before re-releasing capacity, and log its completion status. (4) Enable IDE on every link segment, and use TSP where confidential VMs are in scope. (5) In pre-production, run negative tests: attempt cross-host access, inject surprise removal, and confirm poison propagation. If any step fails, quarantine the extent and do not return it to the free pool.

Related Architecture Specifications (CXLGATEWAY): cxl4ready.com · cxlgrid.com · cxlshield.com

More Technical Articles on cxlcloud.com

Eliminating Datacenter Memory Stranding: Empirical Evaluation of CXL 3.1 Dynamic Memory Pooling →End-to-End Latency Overhead and Throughput Scaling across Multi-Tier CXL Switch Topologies →