Data-center network design: traffic patterns, congestion problems, physical layouts, and modern fabrics.
A data center is a facility housing critical computing resources (servers, storage, networks, power, cooling). This course focuses on networking within the data center. The scatter-gather pattern (e.g. web search) distributes one request to many servers then aggregates responses — causing synchronized bursts.
مركز البيانات منشأة لاستضافة وإدارة موارد الحوسبة الحرجة (سيرفرات، تخزين، شبكات، تبريد، طاقة). التركيز هنا على الشبكة داخل مركز البيانات وليس شبكات المزود.
A teacher asking 30 students a question at once (scatter) and all answering simultaneously (gather) — a flood at the teacher's desk.
Every leaf connects to every spine → many equal-cost paths (great for East-West traffic)
Facility housing servers, storage, networks, power, and cooling.
Cloud provider's physical infrastructure.
Distribute a request to many nodes, then aggregate responses.
A search query fanned out to many index servers.
Data-center traffic is described by volume, locality (how local the destination is), number of concurrent flows, arrival rate, and flow size (mice vs elephant flows).
تتميز حركة مركز البيانات بعدة خصائص: الحجم، الموقعية (هل الوجهة قريبة؟)، عدد التدفقات المتزامنة، معدل الوصول، وحجم التدفق.
Like describing road traffic: how much, where it's going, how many cars at once, how fast they arrive, and trip length.
Total amount of traffic.
Terabytes per second across the fabric.
How close source and destination are (rack, cluster, DC, inter-DC).
Rack 12.9%, Cluster 57.5%, DC 11.9%, Inter-DC 17.7% (typical distribution).
Number of simultaneous connections.
Thousands of TCP flows at once.
How fast new flows/packets arrive.
Bursty arrivals during scatter-gather.
Amount of data per flow (mice vs elephant).
Small RPCs (mice) vs big transfers (elephants).
| Scope | Percentage |
|---|---|
| Rack | 12.9% |
| Cluster | 57.5% |
| Data Center | 11.9% |
| Inter-DC | 17.7% |
Key problems: congestion (oversubscribed links), TCP Incast (throughput collapse when many synchronized senders overflow a switch buffer to one receiver), and isolation (keeping tenants' traffic separated/fair).
أهم مشاكل الشبكة: الازدحام، وTCP Incast (انهيار الإنتاجية عند وصول ردود كثيرة متزامنة لمنفذ واحد)، ومشكلة العزل بين المستأجرين.
TCP Incast = everyone rushing through one door at once, jamming it so nobody gets through efficiently.
Demand exceeds link/buffer capacity, causing delay/drops.
Oversubscribed uplink during peak.
Throughput collapse from many synchronized senders to one receiver.
Scatter-gather overflowing a ToR switch buffer.
Separating/protecting tenant traffic for fairness & security.
One tenant's burst not harming another.
Cabling layouts: ToR (Top-of-Rack) places a switch in each rack (short cables, more switches); EoR (End-of-Row) centralizes switching at the row's end (longer cables, fewer switches). A fabric extender acts as a remote line card; an aggregation switch concentrates traffic from access switches.
تخطيطات الكابلات: ToR (سويتش أعلى كل رف) يقلل الكابلات الطويلة، وEoR (سويتش نهاية الصف) يركّز الإدارة. Fabric Extender يمدد السويتش، وAggregation Switch يجمع عدة سويتشات.
ToR = a mailbox on every floor; EoR = one mailroom per building wing.
| Aspect | ToR (Top-of-Rack) | EoR (End-of-Row) |
|---|---|---|
| Switch location | In each rack | At end of the row |
| Cabling | Short, within rack | Longer, across row |
| Switch count | More switches | Fewer switches |
| Management | More devices to manage | Centralized |
A Fabric Extender acts as a remote line card, extending an aggregation switch into the rack (copper stays in-rack) while keeping centralized management. It combines ToR benefits (short cabling, modular per-rack) with EoR benefits (fewer switches to manage, fewer aggregation ports, fewer STP instances).
Fabric Extender يعمل كـ line card بعيد — يمدد switch aggregation إلى داخل الرack (copper in-rack) مع إدارة مركزية. يجمع مزايا ToR (كابلات قصيرة) مع EoR (سويتشات أقل للإدارة).
Like a remote checkout counter (FEX in each rack) controlled by one central office (aggregation switch).
Remote I/O module extending a parent switch into racks.
Cisco FEX acting as remote line cards for Nexus.
Aggregation switch that manages FEX modules.
Core/aggregation switch controlling rack FEXs.
Data-center traffic is massive and ~76% is East-West (server-to-server within the cloud DC). Traffic grows ~25% annually. Server virtualization increases East-West traffic. Implications: tight network I/O deadlines, TCP Incast congestion, need for tenant/application isolation, and difficulty of centralized per-flow control.
حركة مركز البيانات كبيرة و76% منها East-West (بين السيرفرات). النمو ~25% سنوياً. الافتراضية تزيد East-West traffic. التحديات: deadlines ضيقة للـ I/O، TCP Incast، والحاجة لعزل بين التطبيقات.
Most traffic is coworkers talking to each other (East-West), not visitors coming in (North-South).
~76% of cloud DC traffic is server-to-server.
VM-to-VM, distributed databases, scatter-gather.
DC apps expect low-latency network responses.
Microservices RPCs with strict latency SLAs.
Tree networks connect ToR switches to aggregation switches but bottleneck at the top (expensive, hard to scale, congestion). Clos (spine-leaf) uses many small switches with equal-cost paths and high bisection bandwidth. East-West = server-to-server within DC; North-South = in/out of the data center.
شبكة Tree تستخدم Aggregation switches لكنها تعاني من اختناق وتكلفة N². Clos (Spine-Leaf) يحل ذلك بمسارات متعددة. East-West = بين السيرفرات داخل DC (~76% من حركة السحابة). North-South = داخل/خارج DC.
Tree = one narrow highway to downtown (jams); Clos = a grid of equal roads.
Every leaf connects to every spine → many equal-cost paths (great for East-West traffic)
| Aspect | Tree | Clos (Spine-Leaf) |
|---|---|---|
| Paths | Single/limited | Many equal-cost paths |
| Bottleneck | At aggregation/core | Avoided (scale-out) |
| Cost | Expensive large switches | Many cheap small switches |
| Scalability | Poor (N² problem) | High bisection bandwidth |
| Best for | Traditional North-South | East-West heavy workloads |
| Direction | Meaning | Example |
|---|---|---|
| East-West | Server-to-server inside the DC | VM-to-VM, distributed compute |
| North-South | In/out of the data center | Client request from the internet |