At the Pingtouge compute summit, I sat through two talks—one on CPUs, one on NICs.
On the way back I kept wondering why the stage was given to CPUs and network cards. After sorting it again, both speakers had circled the same point: the fight beyond the GPU.
How to put it? I’ll try.
Start with the CPU talk. Huang Wei, Pingtouge Semiconductor’s product director, set the frame immediately: in the agent era, the CPU’s role has fundamentally changed. Why? Look at whether the work itself changed.
Classic LLM inference was Q&A—mostly GPUs carrying the load.
Agents are different. To finish a task they loop: reason a step, call a tool, check the result, reason the next step, until done.
That tool-call step—running code, fetching a page, reading a file—is CPU work.
And the long context that must stay in hand—the whole thread of this job—is raw material for every reasoning round, mostly kept alive by CPU-side memory.
He threw out three numbers.
In agent scenarios, tool execution alone takes about 60% of end-to-end request time; for coding agents, container init plus tool execution eats 55%–60% on the CPU side. Overall, the CPU already carries more than half the load.
Those were on-stage figures; experimental conditions weren’t unpacked, but the conclusion was blunt: the CPU is back on AI’s critical path.
He split the work into two kinds—the framework that stuck with me most.
One kind sticks to the GPU: what he called the head node. The GPU does the heavy lifting; the head handles the chores—like a driver with a co-pilot. Requests hit it first for preprocessing.
KV cache is its job—the model’s scratch notes of what it already computed, so the next reply doesn’t start from zero. Where notes live and how they move are on the head.
When to wake accelerators, how to stitch results back—also its schedule and wrap-up. This sits on the critical path: if the head is a step slow, GPUs burning money by the hour idle a step.
So the head-node CPU is tuned everywhere for single-core speed.
Widen the frontend—broaden the single-lane intake so one beat swallows more instructions.
Enlarge the out-of-order window—like a cook who won’t wait for one dish to leave the pan before prepping the next; a bigger window looks dozens of steps ahead and runs what can run first.
KV cache shuttles between CPU and accelerator; address translation often misses—each miss is flipping a dictionary page by page. The fix is page-table prefetch: teach the CPU to turn the next page early.
Another subtle one: two unrelated ops get misjudged as hitting the same address and serialized when they could run in parallel. Memory renaming exposes that false sharing and clears the lane.
On the software side, even Python interpreter dispatch overhead gets squeezed line by line.
On cache, KV-cache metadata is tiered across L3 and DRAM—hot notes stay close so accelerators aren’t waiting for something that should be at hand.
The other kind is decoupled from inference: the agent rack—a row of agent rooms.
One cabinet packs hundreds of sandboxes; each runs an agent calling tools, executing code, planning steps, orchestrating chains, managing state.
None of that touches the GPU—all on CPU. This work doesn’t need fastest; it needs densest—cram as many agents as possible without them starving each other.
The toolkit is different.
Agent loops have lots of irregular branches. CPUs predict the next step; classic predictors misfire on agents, so neural branch prediction lets the CPU grow intuition from experience.
RAG retrieval and graph walks follow a vine to dig a string of potatoes—graph prefetch digs the next hop before it’s needed.
Hundreds of sandboxes sharing cache is hundreds of households fighting one storeroom; dynamic replacement rotates by activity and evicts idle junk.
MPAM gives each sandbox a meter for cycles and bandwidth so no one collapses the commons.
His line for the two jobs was vivid: for the head, memory means can you move it fast; for the rack, memory means can you hold a lot.
Two more numbers are easy to skim past—and they’re the foundation.
One is 41%. In a head-node iteration, he said, 41% of the work can’t be parallelized away—especially the control-plane schedule.
That’s Amdahl’s law: more people can’t speed a serial boil. System latency’s floor is that 41% serial slice. To go faster, only one lever remains: make the single core itself faster. The head’s obsession with single-thread performance roots here.
The other is 28GB.
Rack measurements: a single agent instance at peak can need memory on that scale. Pack hundreds per cabinet and you need memory stacked to the sky—or no door opens.
That’s why this CPU isn’t just about cores; capacity itself is competitiveness.
The analysis wasn’t idle. The next day at the Yunqi main stage, Pingtouge first disclosed the Yitian CPU roadmap: Yitian 720 and 730 in 2027, then 750, second-gen in-house cores, and in-house die-to-die links straight to Zhenwu AI chips.
Huang barely mentioned the roadmap at the end—he had been writing its instruction manual.
Talk two: NICs. Fu Binzhang, head of high-performance networking R&D at Alibaba Cloud Intelligent, opened with accounting.
Several earlier speakers had said that in the inference era, communication often decides more than half of performance. Yet NIC cost is under 20% of a cluster—sometimes one-tenth.
Meaning: the highest ROI piece in the machine room is the little board people rarely look at twice.
He laid out three headaches.
First, mixed workloads. Tenants run what they run—training, inference, RL share one fabric; training traffic is steady highway trucks, inference is cars forever changing lanes. Interrupt the trucks and the whole convoy slows.
Second, long distance.
Inference now loves PD separation: split a problem into “understand and draft” versus “emit tokens,” on different GPUs to save cards.
Once the two stages leave the same rack row, data must leave the room onto the WAN. A round trip in-room is ~100µs; across cities, 5–10ms—two orders slower.
Network transfer is like a convoy: scouts report the road before the army speeds up. If scouts report 100× slower, the army throttles and effective bandwidth falls off a cliff.
Third, elasticity.
Agent jobs appear on demand; containers spin up in seconds and die when done. RDMA’s fast lane (machines read each other’s memory without the CPU) wants the whole path’s resources and addresses laid out first—like chartered freight with a timetable before departure.
Containers living tens of seconds can’t wait, so many fall back to on-demand TCP and swallow the performance hit.
Against the three, he first unrolled a map—dense info, one line: pave roads by scenario.
Training rides HPN (high-performance network), each generation swallowing more attach. The prior gen’s 51.2T switch chips backed 7.0; mainline 8.0 uses 102.4T; 9.0 in R&D goes to 200T—“T” as terabit chip throughput.
The pitch: two fabric layers hang hundreds of thousands of 800G ports—like a two-level interchange that still clears tens of thousands of cars.
Inference gets a new architecture, TPN (token-performance network): in one line, stitch two nets that never talked.
Rooms used to have two fabrics—the head network for business I/O and the tail network for GPU-to-GPU high-speed data, sealed from each other.
Small cards without a tail doing PD separation had to jam the head path while the tail sat idle; TPN opens a door between the walls so pods in one cluster can feed each other, PD ends meet without leaving the room, and long-haul is skipped.
Across clusters, traffic rides an intelligent-compute dedicated link on the backbone Alibaba calls ECORE, an order of magnitude above ordinary backbone bandwidth, RDMA end to end.
Inside a supernode sits another fabric, UPN (ultra-node network)—optical, machines not physically glued, one layer spanning a thousand cards.
Then the star: Panmai 920. Specs he quoted: 400G—400 billion bits per second—aimed at intelligent compute and storage.
What excited him most was another design.
The NIC embeds a switch—its own power strip. CPUs disagree on PCIe temperament; without that board, swapping platforms can trip you.
With it, the same card on different platforms still delivers GPU-direct NIC I/O that skips the CPU.
It speaks two protocols: industry RoCEv2 plus Alibaba’s own SolarRDMA.
The in-house stack was first hardened in storage, then brought to compute—“we walked both pits,” in his words. The most valuable feature is per-connection auto path selection: new card to new card takes the fast proprietary lane; against old cards it falls back to the standard.
Old clusters needn’t be rebuilt—swap card by card with traffic live. He said that kind of frictionless incremental deploy is huge in engineering value.
Four scorecards, all on-stage measured numbers.
Before the first he dismantled a popular myth. Two years ago, saying pretraining didn’t need fancy protocols was fine. LLM training was one big job, ring algorithms passing around; each NIC only talked to neighbors—bandwidth was everything.
That line is expired. Clusters are train-and-serve together; inference traffic barges in and slows pretraining with it.
On thousand-card jobs, splitting big transfers into small packets on many paths that reroute around congestion or bad hardware cut data-parallel communication time 43.2% and lifted end-to-end performance 12%.
Second card: cross-cluster long haul.
The endpoint marks traffic; WAN switches pick better paths from the marks; lost packets are retransmitted selectively, not the whole stack. End-to-end throughput up 55%.
Third: elasticity.
New virtualization moves resource allocation onto the software control path; connection density is 64× traditional NIC virt; container bring-up saves 35% at least, 93.4% at most.
His hard line: this elasticity doesn’t trade performance—virtualized communication runs almost as fast as hardware passthrough; by his claim, only this card currently delivers both dedicated-line speed and ride-hail flexibility.
Fourth: small packets. HPC payloads are tiny; the envelope is thick. This card’s “Ethernet+” protocol trims preamble and skips link-layer overhead so useful content hits 85.4%—highest in the industry, he said.
Deployment was brief: Panmai 920 already ships with Lingjun Zhenwu M890 supernodes—those supernodes run this card.
Heard apart, the two talks own their lanes; together they’re interesting.
Huang Wei: in agent scenarios the CPU carries more than half the load. Fu Binzhang: in inference, communication decides more than half the performance. Two “halves,” different meters—one time, one performance.
Same direction. After a request lands, the GPU’s compute stretch is less of the whole road; the bulk sits on CPU, memory, and network.
How smart the model is, compare GPUs; how cleanly agents finish work, compare these foundations.
Huang closed hoping Yitian contributes an accessible compute base for the Agentic AI era. Fu skipped the platitudes and left only the landing—the supernode line above.
The host’s bridge line was the better summary.
AI’s arena has moved from parameter-scale contests into the deep water of engineering delivery; single-point breakthroughs can’t carry system-level agent loads.
Forty minutes on stage were testimony to that sentence.
What I brought back is a change in bookkeeping. We used to ask how many cards you have. Going forward we may ask: once a request arrives, where does time go, where does bandwidth jam, whose memory is enough.
More cards ≠ better jobs. The industry is only starting to count that ledger seriously.
Note:
[1]. Based on my on-site notes from the Pingtouge compute summit breakout on September 23, 2026; measured figures are as disclosed on stage.
相关文章
