AI pawg network tsim yog cov txheej txheem ntawm sizing GPU neeg rau zaub mov NICs, nplooj - qaum bandwidth, oversubscription ratio, RoCE nqis, optics thiab cabling thiaj li faib kev cob qhia tsheb tseem kwv yees raws li pawg teev. Tau txais ib qho ntawm cov tsis ncaj ncees lawm thiab lub network - tsis yog GPU - dhau los ua lub fwj.
Vim li cas AI Cluster Networking txawv
Hauv ib lub chaw lag luam cov ntaub ntawv ib txwm muaj, lub network tswj hwm kev sib xyaw ntawm sab qaum teb -sab qab teb cov neeg siv tsheb, kev nkag mus rau hauv chaw cia, virtualization thiab kev tswj hwm. Sab hnub tuaj - tsheb khiav sab hnub poob muaj tab sis tsis tshua muaj qhov tseem ceeb. Hauv pawg AI, qhov xwm txheej hloov pauv. GPU servers khiav cov kev cob qhia sib pauv hloov gradients thiab synchronize tsis nyob rau hauv txhua kauj ruam ntawm txoj hauj lwm. Qhov kev sib txuas lus no yog ib feem ntawm kev suav, tsis yog qhov tshwm sim ntawm nws.
Yog tias $ 30,000 GPU siv 30% ntawm nws lub sijhawm tos hauv lub network thaum tag nrho - txo cov haujlwm, pawg them nyiaj tau zoo rau 30% ntawm nws lub peev xwm suav los zaum tsis ua haujlwm. Qhov ntawd yog vim li cas kev lag luam AI networking tau txais kev saib xyuas ntau heev.
Peb tus yam ntxwv ua haujlwm ua rau tus qauv tsim:
- Bursty sab hnub tuaj -kev khiav tsheb sab hnub poob.Kev sib txuas lus sib txuas ua haujlwm zoo li txhua yam -txo, tag nrho- sib sau thiab txo- tawg ua kom tawg ua ke ua ke thoob plaws ntau lub pob ib txhij.
- Tail -latency rhiab heev.Ib qho qeeb qeeb qeeb tag nrho cov kauj ruam kev cob qhia. Kev kwv yees latency tseem ceeb tshaj qhov nruab nrab latency.
- Scale - tawm kev loj hlob.Cov pawg uas pib ntawm 32 GPUs feem ntau loj hlob mus rau 256 lossis 1,024 hauv 18 lub hlis. Cov ntaub yuav tsum scale yam tsis muaj redesign.
Vim li cas Kab laug sab-Nplooj Fits AI Pawg
Kab laug sab- nplooj yog cov qauv ntaub rau hyperscale cov chaw zov me nyuam vim nws muab txhua tus neeg rau zaub mov - rau - server txoj kev tib lub hop suav thiab tib lub theoretical bandwidth. Rau AI workloads, qhov sib xws no ncaj qha txhais mus rau ntau lub sij hawm kev cob qhia ua haujlwm.
Hauv kab laug sab- nplooj topology, GPU servers txuas rau nplooj ntawv hloov, thiab txhua nplooj txuas mus rau txhua tus txha nraub qaum. Ib qho GPU-rau-GPU kev sib txuas lus hla raws nraim ib nplooj, ib tus txha nraub qaum, thiab ib nplooj ntxiv. Tsis muaj cov khaubncaws sab nraud povtseg qhia txog qhov sib txawv latency lossis chokepoints.
Predictable Latency
Sib npaug -nqi ntau- txoj kev (ECMP) routing kis ntws hla tus txha nraub qaum. Thaum teeb tsa kom raug nrog kev hloov pauv hloov pauv lossis qhov sib npaug ntawm qhov sib npaug, qhov no tiv thaiv kev sib tsoo hash uas ua rau qee qhov ntws qeeb qeeb dua li lwm tus - ib qho teeb meem paub txog hauv static ECMP fabrics nqa ob peb tab sis loj ntws, uas yog raws nraim li AI kev cob qhia tsim.
High Bisection Bandwidth
Bisection bandwidth yog qhov kev sib tw muaj nyob nruab nrab ntawm ob qho tib si ib nrab ntawm pawg. Kev cob qhia AI tau txais txiaj ntsig los ntawm qhov tsis yog - thaiv lossis ze rau - tsis yog - thaiv cov qauv tsim qhov twg nplooj ntoo - mus rau - tus txha nraub qaum uplink muaj peev xwm sib npaug lossis ze li qhov sib npaug ntawm lub peev xwm downlink ntsib cov servers. IETF txhais thiab sib tham txog cov ntsiab lus no hauvwb 7938, uas npog BGP-routed Clos fabrics dav siv nyob rau hauv loj - teev cov ntaub ntawv chaw.
Easier Scale-Tawm
Ntxiv nplooj ntxiv kom ntxiv cov servers ntxiv. Ntxiv cov spines kom ntxiv ntau dua bisection bandwidth. Rau pawg tshaj li ob peb txhiab GPUs, super- qaum (5-theem Clos) lossis rail-optimized topology txuas rau tib txoj cai ib txheej ntxiv.
Cov Cheebtsam tseem ceeb ntawm AI Cluster Network
GPU Servers thiab NICs
NIC yog qhov chaw uas cov ntaub ntsib tus tswv tsev. Hauv AI pawg, NIC xaiv tsav txhua yam hauv qab - hloov chaw nres nkoj ceev, xaiv optics, thiab cabling ceev.
Cov txheej txheem xaiv rau AI workloads:
- Chaw nres nkoj ceev:200G, 400G lossis 800G ib qhov chaw nres nkoj. Sib phim rau GPU tiam thiab PCIe bandwidth.
- PCIe tiam:400G NIC xav tau PCIe Gen5 x16 kom tsis txhob muaj tus tswv tsev - sab throttling. PCIe Gen4 x16 caps ntawm ~ 256 Gbps siv tau.
- RDMA thiab RoCEv2 kev txhawb nqa:Yuav tsum tau rau kernel{0}}bypass GPU kev sib txuas lus cov tsev qiv ntawv zoo li NCCL.
- GPUDirect RDMA:Tso cai rau GPU ncaj qha-rau-NIC DMA, tshem tawm cov tswv cuab lub cim xeeb.
- Multi-rail muaj peev xwm:Ntau lub AI servers siv 4 lossis 8 NICs rau ntawm ib qho, ib qho ntawm GPU khub, rau kev tsheb ciav hlau -kev ua kom zoo tshaj plaws.
Ib tus neeg siv 8-GPU niaj hnub no siv 4 × 400G NICs (ib qho rau ob GPUs) lossis 8 × 400G NICs (ib qho rau GPU) nyob ntawm kev ua haujlwm thiab nyiaj txiag. Reference architectures los ntawmNVIDIA Networking cov ntaub ntawvnpog cov qauv kev lag luam hauv kev nthuav dav.
Nplooj thiab txha nraub qaum
Hloov xaiv cov qauv rau AI fabrics txawv ntawm kev xaiv lag luam. Buffer loj, congestion tswj tus cwj pwm thiab telemetry teeb meem ntau tshaj feature dav.
- Rau -port ceev thiab radix:A 51.2 Tbps hloov ASIC muab 64 × 800G chaw nres nkoj lossis 128 × 400G chaw nres nkoj. Radix txiav txim siab npaum li cas cov ntaub tuaj yeem ua tau.
- Buffer architecture:Sib sib zog nqus buffers nqus incast tawg tab sis ntxiv latency. Nthuav buffers txo latency tab sis yuav tsum muaj kev tswj xyuas kom meej.
- RoCE feature teeb:ECN kos npe, PFC, DCQCN lossis sib npaug sib npaug ntawm kev tswj hwm kev sib tw, thiab kev tswj xyuas cov txheej txheem tseem ceeb kawg - mus rau - xaus.
- Telemetry:Inband network telemetry (INT), ib -queue depth reporting, and microsecond-resolution counters for ECN marks and PFC pauses.
Optics, DAC thiab AOC Cable
Ntawm 400G thiab 800G, cov nroj tsuag cabling dhau los ua qhov teeb meem engineering tiag tiag. Cov qauv tsim, txuas pob nyiaj siv thiab kev teeb tsa kev teeb tsa txhua qhov xav tau kev npaj ua ntej.
- DAC (Direct Txuas tooj liab):Txog li ~ 3 meters rau 400G, tus nqi qis tshaj plaws thiab lub zog qis tshaj. Heavy thiab loj ntawm nplai.
- AOC (Active Optical Cable):Txog li ~ 30 meters, nyias dua li DAC, tab sis ruaj -ntev thiab siv lub zog optics ntawm ob qhov kawg.
- Pluggable optics:Yuav tsum tau tshaj AOC nrug. QSFP-DD thiab OSFP daim foos tseem ceeb ua tus thawj 400G/800G. MPO/MTP fiber ntau cov rooj sib txoos tswj cov kev sib txuas - fiber ntau txuas.
Rau kev sib tshuam - cov kab txuas thiab cov kab txuas txuas ntawm 400G / 800G, kev sib txuas optics dhau MPO kev txiav tawm tam sim no yog tus qauv. Qhov kev xaiv ntawm lub cev cables thiab breakout cov rooj sib txoos nyob ntawm koj qhov chaw nres nkoj hloov pauv - saib pebMPO breakout cable qhiarau cov tswv yim xaiv logic, thiab davMPO pob tw vs breakout sib pivthaum npaj nplooj - mus rau - qaum khiav.
RoCE thiab Lossless Ethernet hauv AI Fabrics
RoCEv2 (RDMA dhau Converged Ethernet v2) yog qhov tseem ceeb thauj Ethernet rau AI workloads. Nws tso cai rau NICs txav cov ntaub ntawv ncaj qha ntawm GPU nco thaj tsam yam tsis muaj kev cuam tshuam cov ntsiav ntawm ob qho kawg. NCCL, lub tsev qiv ntawv GPU kev sib txuas lus hauv qab yuav luag tag nrho cov kev qhia qhia tawm, siv RoCEv2 thaum InfiniBand tsis muaj.
RoCE ua haujlwm zoo thaum teeb tsa kom raug. Nws ua tsis tau dab tuag thaum configured incorrectly. CovInfiniBand Trade AssociationTshaj tawm RoCE cov lus qhia tshwj xeeb, thiab feem ntau NIC thiab cov neeg muag khoom hloov pauv tau tshaj tawm cov lus qhia ntxaws ntxaws uas yuav tsum tau ua raws qhov kawg - mus rau - kawg.
Vim Li Cas Lossless Cwj Pwm Tseem Ceeb
RDMA tau tsim los piv txwv li kev thauj mus los tsis zoo. Thaum cov pob ntawv poob, RDMA rov qab yog kim - mus-rov qab-N retransmission tuaj yeem nres ib kauj ruam kev cob qhia rau milliseconds, uas yog loj heev txheeb ze rau microsecond - nplai RDMA pob nyiaj siv.
Txhawm rau kwv yees tus cwj pwm tsis zoo ntawm Ethernet, cov ntaub siv ob lub tshuab ua haujlwm ua ke:
- PFC (Priority Flow Control, IEEE 802.1Qbb):Ib qho kev hloov pauv ncua kev nkag mus rau ntawm ib qho tseem ceeb ntawm cov kab ke thaum nws tsis nyob. Qhov no yog qhov kawg -resort mechanism.
- ECN (Daim Ntawv Ceeb Toom Qhia Txog Kev Cuam Tshuam, RFC 3168):Hloov cov cim pob ntawv thaum queues mus txog qhov pib. NIC txo nws qhov kev xa tawm ua ntej buffers tau sau, zoo kawg nkaus zam PFC nkaus.
Lub hom phiaj yog rau ECN ua yuav luag tag nrho cov kev tswj xyuas kev sib tsoo, nrog PFC ua qhov kev nyab xeeb. Yog tias koj pom PFC nquag nres nyob rau hauv tsis tu ncua - lub xeev kev khiav tsheb, koj qhov chaw pib ECN tsis raug lossis koj cov ntaub tsis zoo.
Common RoCE Deployment Failures
| Teeb meem | Cov tsos mob | Yuav kuaj li cas | Kho |
|---|---|---|---|
| MTU mismatch kawg -to-end | Fragmentation, RDMA rov ua dua, hla dhau qhov tawg | Sib piv NIC thiab hloov MTU; khiav ping nrog DF ntsis teeb ntawm MTU loj | Teeb jumbo MTU (feem ntau 9000 lossis 9216) tsis tu ncua hla NICs thiab txhua qhov hloov pauv |
| PFC qhov tseem ceeb misalignment | PFC thav duab tsim tab sis tsis quav ntsej; backpressure tsis propagated | Tshawb xyuas PFC qhov tseem ceeb tau teeb tsa ntawm NIC vs. hloov ingress queue mapping | Teem DSCP- rau- qhov tseem ceeb ntawm daim ntawv qhia ntawm txhua lub hops |
| Tsis ncaj ncees lawm ECN qhov pib | Txawm hais tias tsis muaj ECN cov cim (kev sib tsoo kom txog thaum PFC hluav taws kub) lossis cov cim tsis tu ncua (los ntawm kev tawm tsam) | Saib xyuas ib -queue ECN- cim cov ntawv ntim khoom raws li qhov tseeb | Tune Kmin/Kmax thresholds; default values tsis tshua haum AI tsheb profiles |
| Kev sib xyaw ua ke ntawm tib qhov tseem ceeb | Kev khaws cia lossis kev tswj xyuas tawg cuam tshuam kev cob qhia | Tshawb xyuas DSCP cov cim ntawm txhua chav tsheb khiav ntawm NIC thiab hloov | Muab cais qhov tseem ceeb rau kev suav, khaws cia thiab tswj |
| Kev tsis txaus siab los ntawm incast | random pob ntawv poob thaum tag nrho - txo | Raws li -queue buffer occupancy telemetry thaum lub sijhawm ua haujlwm ua ke | Txhim kho qhov kev faib tawm rau kev suav qhov tseem ceeb; kho kom haum adaptive routing |
Yuav Ua Li Cas Tsim Ib Pawg AI Pawg Network: Ib Lub Cev Ua Haujlwm
Qhov no yog ntu feem ntau "AI networking" cov lus hla. Xya cov kauj ruam hauv qab no muab koj cov tswv yim thiab cov txiaj ntsig ntawm txhua theem.
Kauj Ruam 1: Txhais qhov ua haujlwm thiab nplai
Inputs:Hom kev ua haujlwm (pretraining, fine{0}}tuning, inference, mix), phiaj GPU suav hnub no, phiaj GPU suav hauv 18 lub hlis, qauv ntau yam.
Cov zis:Ib daim ntawv ua haujlwm uas qhia txog NIC ceev thiab oversubscription kam rau ua. Kev qhia loj ntawm cov qauv pem hauv ntej xav kom tsis txhob- thaiv 400G + ntaub. Fine-tuning workloads tuaj yeem zam 2: 1 oversubscription. Inference pawg feem ntau xav tau qis bandwidth tab sis qis dua latency.
Kauj ruam 2: Xaiv NIC Ceev thiab suav rau ib tus neeg rau zaub mov
Kev txiav txim siab logic:
- Kev cob qhia ntawm cov qauv loj, 8-GPU servers → 4–8 × 400G NICs rau ib tus neeg rau zaub mov, lossis 4 × 800G
- Nruab nrab- teev kev cob qhia, 8-GPU servers → 2–4 × 400G NICs rau ib lub server
- Inference pab → 1–2 × 200G los yog 400G NICs rau ib tus neeg rau zaub mov, nyob ntawm tus qauv parallelism
Txheeb xyuas PCIe bandwidth ntawm tus tswv tsev. Ib qho chaw 400G nkaus xwb yuav tsum tau PCIe Gen5 x16 khiav ntawm kab nqi; ob npaug rau 800G yuav tsum tau Gen6 lossis faib hla ob qhov.
Kauj Ruam 3: Loj cov Nplooj Txheej
Ua haujlwm piv txwv - 32-node pawg, 8 GPUs rau ib ntawm, 4 × 400G NICs rau ib ntawm:
- Tag nrho cov neeg rau zaub mov-faus cov chaw nres nkoj xav tau: 32 × 4=128 chaw nres nkoj ntawm 400G
- Downlink bandwidth ib node: 4 × 400=1.6 Tbps
- Tag nrho pawg downlink bandwidth: 32 × 1.6=51.2 Tbps
Siv 64-chaw nres nkoj 400G nplooj hloov (25.6 Tbps tag nrho lub peev xwm), txhua nplooj tuaj yeem txuas tau 32 lub chaw nres nkoj neeg rau zaub mov thiab siv 32 cov chaw nres nkoj ntxiv los ua qhov txuas. Nrog 4 nplooj, koj npog tag nrho 128 server chaw nres nkoj. Txhua nplooj txhawb nqa 32 × 400G=12.8 Tbps ntawm uplink mus rau qaum.
Kauj Ruam 4: Loj qhov txha nraub qaum txheej
Rau qhov tsis yog - thaiv (1: 1) tsim, tag nrho cov peev xwm uplink yuav tsum sib npaug tag nrho cov peev xwm downlink. Los ntawm Kauj Ruam 3:
- Tag nrho cov nplooj uplink yuav tsum tau: 4 nplooj × 12.8 Tbps=51.2 Tbps
- Yog tias txhua tus txha nraub qaum muaj 32 × 400G chaw nres nkoj=12.8 Tbps, koj xav tau 4 tus txha nraub qaum
- Txhua nplooj txuas mus rau tag nrho 4 spines siv 8 uplinks ib leeg (8 × 400G × 4=12.8 Tbps ib nplooj - ntais ntawv)
Yog tias siv 64-chaw nres nkoj 400G txha nraub qaum keyboards, txhua tus txha nraub qaum muaj peev xwm loj hlob hauv pawg, pab tau rau 18-hli txoj kev npaj los ntawm Kauj Ruam 1.
Kauj ruam 5: Teem lub Oversubscription Ratio
| Ua haujlwm | Pom zoo Ratio | Kev xav |
|---|---|---|
| Loj - qauv pretraining | 1: 1 (tsis yog - thaiv) | Txhua yam -txo dominates; tej congestion compounds hla ntau txhiab tus kauj ruam |
| Fine-tuning / nruab nrab- teev kev cob qhia | 1.5:1 rau 2:1 | Kev sib sau me me; kev txuag nyiaj ntau dua qis qis qis |
| Inference / RAG pab | 2:1a4: 1 | Feem ntau yog kev thov ywj siab; bandwidth tawg yog me me thiab tsawg synchronized |
| Mixed research pawg | 1.5:1 | Kev sib haum xeeb ntawm tus nqi thiab kev ua haujlwm tsis sib haum xeeb |
Kauj Ruam 6: Sib cais xam, khaws cia thiab tswj tsheb
Peb txoj kev xaiv, txhawm rau nce kev sib cais:
- Sib koom ntaub nrog QoS chav kawm:Kev suav, khaws cia thiab kev tswj hwm ntawm DSCP qhov tseem ceeb. Tus nqi qis tshaj; yuav tsum ceev faj QoS configuration.
- Logically cais VLANs/VRFs:Tib yam khoom siv, cais cov dav hlau tswj. Pab tau rau ntau- pawg neeg xauj tsev.
- Lub cev cais cov ntaub:Dedicated NICs, keyboards thiab cabling rau xam thiab cia. Tus nqi siab tshaj; feem ntau nyob rau hauv pem hauv ntej- qauv pawg uas muaj kev sib cav tsis txaus ntseeg.
Kev thauj mus los rau AI yog nws tus kheej hnyav - checkpoint sau rau tus qauv loj tuaj yeem txav mus ntau pua gigabytes hauv luv luv tawg. Npaj kom meej meej. Ib lub siab - ntom ntom qauv cabling cog sivMPO / MTP lub cev cablessimplifies khiav parallel fabrics nyob rau hauv tib lub cev infrastructure.
Kauj Ruam 7: Siv tau ua ntej kev tsim khoom
Network-qib xeem ntsuas qee qhov teeb meem. Workload-qib kev xeem tau txais tus so.
- Bandwidth:iperf3 los yog ib_send_bw nruab nrab ntawm txhua tus node khub; yuav tsum ncav cuag 90% + ntawm NIC kab tus nqi.
- Latency:ib_read_lat or similar; check faib, tsis yog nruab nrab xwb. P99.9 tseem ceeb tshaj qhov txhais tau tias.
- Pob ntawv poob:Khiav 24-teev soj ntsuam hauv qab load; ib qho kev poob qis hauv RoCE tsheb yog qhov teeb meem.
- ECN cim kev coj cwj pwm:Txheeb xyuas cov cim tshwm ua ntej PFC hluav taws kub; Yog tias PFC nres ntau zaus hauv lub xeev khov kho, rov kho dua.
- Collective kev sib txuas lus:Khiav NCCL kev xeem (all_reduce_perf, all_gather_perf) ntawm tag nrho pawg loj. Muab piv rau tus neeg muag khoom siv tus lej.
- Txoj hauj lwm- qib xeem:Khiav ib tug neeg sawv cev cob qhia txoj hauj lwm rau 4-6 teev. Saib kev siv GPU - cov txiaj ntsig zoo hauv qab 50% ntawm tus qauv zoo- qhov loj me feem ntau qhia txog teeb meem hauv network.
Traditional Data Center Network vs AI Spine-Nplooj ntaub
| Thaj tsam | Ib txwm DC Network | AI qaum-Nplooj ntaub |
|---|---|---|
| Muaj kev khiav tsheb | Sib xyaw sab qaum teb-sab qab teb thiab sab hnub tuaj- sab hnub poob | Hnyav GPU-rau-GPU sab hnub tuaj-west, tawg |
| latency kam rau ua | Milliseconds tau txais | Microseconds tseem ceeb; Tail latency tseem ceeb |
| Oversubscription | 4: 1 txog 8: 1 ntau | 1: 1 txog 2: 1 rau kev cob qhia fabrics |
| Kev thauj mus los | TCP / IP tseem ceeb | RoCEv2 los yog InfiniBand |
| NCO role | Txuj kev sib txuas | Kev ua tau zoo- tseem ceeb, feem ntau ntau -rail |
| Tsis xav tau | Daim ntawv thov - nyob ntawm | Tuned rau incast burst absorption |
| Kev lees paub | Daim ntawv thov teb lub sijhawm | Raws li -flow telemetry + collective benchmarks |
Ethernet RoCE vs InfiniBand: Phau ntawv txiav txim ceev
Cov lus nug tshwm sim hauv yuav luag txhua qhov project AI pawg. Ob leeg ua haujlwm. Kev xaiv feem ntau yog los rau kev ua haujlwm haum, tsis yog kev ua haujlwm dawb huv.
- Xaiv InfiniBand yog tias:Koj pab neeg twb tau ua haujlwm InfiniBand fabrics, koj xav tau txoj hauv kev yooj yim tshaj plaws rau kev thauj mus los tsis zoo, lossis koj tab tom yuav tag nrho -cov neeg muag khoom siv kev sib koom ua ke.
- Xaiv Ethernet RoCE yog tias:Koj pab neeg ua haujlwm yog Ethernet- haiv neeg, koj xav tau ntau- tus neeg muag khoom hloov kev xaiv, koj yuav tsum tau muab cov ntaub ntawv AI nrog cov ntaub ntawv chaw nyob tam sim no, lossis koj cia siab tias yuav ntsuas dhau qhov tam sim no InfiniBand topologies txhawb kom huv.
Ultra Ethernet Consortium, tsim nyob rau hauv 2023, yog nquag ua hauj lwm ntawm standardizing Ethernet kev txhim kho tshwj xeeb rau AI workloads. Rau feem ntau pawg tshiab hauv 2026, Ethernet RoCE yog qhov kev tiv thaiv tsis zoo tshwj tsis yog tias muaj ib qho laj thawj tshwj xeeb los xaiv lwm yam.
Feem ntau yuam kev kom zam
Txhim kho cov hloov pauv yam tsis tau kuaj xyuas NICs
Ib qho 800G hloov ntaub tsis muaj dab tsi rau koj yog tias koj NICs khiav ntawm 400G lossis koj tus tswv PCIe khiav tawm ntawm bandwidth. Tsim tus tswv tsev ua ntej, tom qab ntawd hloov sab. PCIe Gen5 x16 txwv ib qho chaw nres nkoj txog li 504 Gbps tiag tiag- ntiaj teb dhau los - xis rau 400G, marginal rau 800G.
Optimizing Chaw nres nkoj ceev tab sis tsis quav ntsej Cabling ceev
Ntawm 64-chaw nres nkoj 400G tawm, cov cabling nyob rau hauv txhua qhov hloov tuaj yeem ua rau lub cev tswj tsis tau yam tsis tau npaj. Siv cov kab hluav taws xob sib cais qhov tsim nyog, txoj kev fibers los ntawm cov txheej txheem txheej txheem, thiab ua qauv ntawm cov khoom sib txuas. Connector zoo thiab txiav tawm teeb meem ntawm kev kub ceev - pebfiber optic connector hom qhianpog cov kev lag luam tawm ntawm LC, MPO thiab tshwm sim siab - daim ntawv ntau yam.
Kho RoCE li Plug-thiab-Play
Qhov tsim yuam kev loj tshaj plaws hauv pawg AI tiag tiag tsis yog xaiv qhov kev hloov pauv tsis raug - nws yog underestimating npaum li cas kawg - mus rau - xaus RoCE configuration ua hauj lwm yuav tsum tau. Lub sijhawm nyiaj txiag rau kev kho ECN qhov pib, PFC qhov tseem ceeb thiab MTU sib xws. Npaj ib theem validation ua ntej ib qho kev ua haujlwm ntau lawm.
Sib tov tag nrho cov tsheb khiav ntawm ib daim ntaub tsis muaj QoS
Kev khaws cia rov ua dua, saib xyuas cov neeg saib xyuas thiab kev tswj xyuas kev khiav tsheb tuaj yeem rhuav tshem cov kauj ruam kev cob qhia yog tias lawv sib koom nrog kev suav tsheb. Ob leeg cais lawv lub cev los yog tswj hwm QoS cov chav kawm nruj nrog kev ua haujlwm tshwj xeeb thiab ECN teeb tsa.
Lub tsev rau hnub no nkaus xwb
Feem ntau AI pawg loj hlob 4-8 × hauv ob xyoos ntawm kev xa tawm thawj zaug. Xaiv qhov hloov radix thiab txha nraub qaum uas tso cai rau tsis yog - cuam tshuam kev nthuav dav. Rub cov cables hauv qhov chaw nyob AI cov ntaub ntawv yog kim; npaj cov kav dej thiab thaj chaw muaj peev xwm ntawm lub sijhawm xa khoom yog pheej yig.
Thaum yuav nce qib ntawm 400G mus rau 800G
800G NICs thiab keyboards muaj tab sis kim dua ib qhov chaw nres nkoj. Xav txog kev nce qib thaum:
- Ib -GPU bandwidth xav tau tshaj li qhov 400G tuaj yeem muab tau - piv txwv li, H100 thiab GPUs tshiab nrog NVLink 5 cia siab tias yuav siab dua sab nraud bandwidth
- NCCL tag nrho -txo lub sij hawm teev tsis zoo nrog pawg loj, qhia txog kev sib txuas hauv network
- Cable ceev ntawm 400G tau dhau los ua lub cev tsis tuaj yeem tswj tau - tsawg dua 800G chaw nres nkoj tuaj yeem hloov 400G chaw nres nkoj ntau dua
- GPU tiam tom ntej hauv koj daim ntawv qhia kev yuav tsum xav tau nws nyob rau hauv pawg qhov kev poob nqi
- Koj tab tom tsim ib thaj tsam - qauv kev cob qhia pawg qhov twg txhua lub sijhawm suav tsis siv neeg raug nqi ntau dua li kev hloov kho kho qhov muag
Rau feem ntau cov pawg tsim khoom hauv xyoo 2026, 400G tseem yog qhov sib npaug ntawm tus nqi, kev loj hlob ntawm ecosystem, thiab muaj peev xwm. 800G ua rau kev txiav txim siab ntawm qhov kawg thiab raws li kev nqis peev rau pem hauv ntej rau pawg tau tsim hnub no thiab xav tias yuav khiav mus rau 4-5 xyoo.
FAQ
Q: Dab tsi yog qhov zoo tshaj plaws network architecture rau AI pawg?
A: Kab laug sab- nplooj Clos topology yog tus qauv xaiv. Rau pawg saum toj no ~ 1,000 GPUs, txuas mus rau 5-theem Clos (super- qaum) lossis rail-optimized topology. Lub architecture nws tus kheej yog to taub zoo; Cov teeb meem nyuaj yog bandwidth sizing, RoCE configuration thiab validation.
Q: Dab tsi oversubscription piv tau txais rau AI kev cob qhia?
A: Rau loj- qauv pretraining, tsom rau 1: 1 (tsis yog- thaiv). Rau kev nplua -tuning thiab nruab nrab- kev cob qhia nplai, 1.5: 1 txog 2: 1 yog ua haujlwm tau. Rau kev ua noj ua haus, 2: 1 txog 4: 1 tau txais. Qhov sib piv ntau dua txuag nyiaj tab sis txo qhov kev ua tau zoo, thiab qhov sib txawv ntawm qhov sib txawv nyob ntawm seb kev sib txuas lus li cas - khi koj cov khoom ua haujlwm.
Q: Puas yog RoCE xav tau rau AI pawg?
A: RoCEv2 lossis InfiniBand yog qhov yuav tsum tau ua rau txhua pawg ua haujlwm NCCL-raws li kev qhia faib tawm ntawm qhov ntsuas. Plain TCP / IP tsis tuaj yeem xa cov latency thiab CPU efficiency xav tau. Ntawm RoCEv2 thiab InfiniBand, xaiv raws li kev ua haujlwm haum thiab ecosystem es tsis yog kev ua haujlwm dawb huv.
Q: Muaj pes tsawg NICs xav tau GPU server?
A: Rau 8-GPU server, kev teeb tsa ntau yog 4 × 400G (ib NIC rau ob GPUs) lossis 8 × 400G (ib NIC rau GPU, rail-optimized). Inference servers yuav siv 1-2 NICs. Qhov kev txiav txim siab nyob ntawm kev ua haujlwm, GPU tiam, PCIe topology thiab pob nyiaj siv.
Q: Puas yog pawg AI xav tau kev khaws cia thiab suav cov ntaub?
A: Cov pawg me tuaj yeem faib cov ntaub nrog QoS chav sib cais. Nruab nrab- qhov loj thiab cov pawg loj feem ntau tau txais txiaj ntsig los ntawm cov khoom sib cais ntawm lub cev - suav ntawm RoCE Ethernet lossis InfiniBand, khaws cia rau ntawm daim ntaub Ethernet tshwj xeeb. Frontier- qauv pawg feem ntau sib cais ntawm lub cev vim tias ib qho kev hla - kev cuam tshuam kev tsheb yog qhov tsis tsim nyog.
Q: Puas yog Ethernet zoo dua InfiniBand rau AI workloads?
A: Tsis yog universally zoo dua. InfiniBand muaj cov ntaub ntawv teev tseg ntev dua hauv HPC thiab muaj cov cwj pwm tsis zoo uas paub tab. Ethernet RoCEv2 muaj ntau tus neeg muag khoom sib txawv, koom ua ke nrog cov ntaub ntawv chaw zov me nyuam uas twb muaj lawm, thiab tau txais txiaj ntsig los ntawm kev txhim kho nquag hauv Ultra Ethernet Consortium. Kev paub ntawm pab neeg ua haujlwm feem ntau yog qhov kev txiav txim siab.
Q: Qhov tsis yog - thaiv AI network txhais li cas?
A: Nws txhais tau tias tag nrho nplooj ntawv -rau- qaum uplink muaj peev xwm sib npaug tag nrho nplooj -rau- server downlink peev, yog li cov ntaub tuaj yeem txhawb nqa ib qho kev sib txuas lus ntawm ib khub ntawm kab ntawm tag nrho cov kab. Hauv kev xyaum, qhov tseeb tsis yog - thaiv yog kim; ntau ntau cov ntaub yog "ze tsis- thaiv" ntawm 1.1: 1 lossis 1.2: 1 thiab tseem ua tau zoo.
Q: Qhov kev sim no qhia txog cov teeb meem ntawm RoCE tiag tiag?
A: NCCL benchmark suites (all_reduce_perf, all_gather_perf) khiav ntawm tag nrho pawg scale yuav nthuav tawm cov teeb meem tiag tiag. Ib_send_bw kev xeem dawb huv ntawm ob lub nodes tuaj yeem dhau thaum 32-node tag nrho-txo ua haujlwm tsis zoo vim muaj teeb meem incast lossis PFC. Nco ntsoov siv tau ntawm qhov ntsuas koj npaj yuav khiav.
Xaus
Lub zog tshaj AI pawg network tsis yog ib qho uas muaj qhov hloov ceev tshaj plaws. Nws yog qhov uas NIC xaiv, nplooj / txha nraub qaum sizing, oversubscription, RoCE configuration, tsheb sib cais thiab lub cev cabling tag nrho txhawb ib leeg thiab cov hauj lwm uas lawv tau xaiv rau.
Pib los ntawm kev ua haujlwm thiab 18- lub hli txoj kev loj hlob. Xam cov bandwidth xav tau ntawm txhua txheej siv cov lej tiag, tsis yog cov cai ntawm tus ntiv tes xoo. Configure RoCE kawg-rau- xaus thiab siv tau nrog cov qauv sib txuas lus tiag tiag. Cov peev nyiaj rau cov nroj tsuag cab - ntawm 400G thiab 800G, lub cev txheej tsis tseem ceeb ntxiv lawm.
Cov pawg uas ua rau nws cov GPUs tsis khoom ntawm 95% + kev siv los ntawm txhua kauj ruam kev cob qhia yog ib qho uas tau them nyiaj rau tag nrho cov txheej txheem no. Cov pawg uas xa nrog kev hloov pauv sai dua thiab cov ntaub qeeb qeeb yuav siv sijhawm ntau xyoo piav qhia vim li cas GPUs tsis ua haujlwm.
