# How does an AI data center actually work? Date: 2026-08-11 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/how-ai-data-center-works/ **Answer:** An AI data center is a factory that turns electricity into calculations, built around racks of tightly linked chips that draw far more power than ordinary servers. Most of the design, from power lines and batteries to liquid cooling and fast networking, exists to keep those chips fed, cool and talking to each other. When you send a question to a chatbot, it travels to a building where a trained model runs on specialized chips, and the answer comes back in seconds. An AI data center is that building, scaled up until a single site can draw as much electricity as a mid-sized city. An ordinary data center hosts websites, email and databases, and each task can run on its own machine. AI is different. Training a large model, which means adjusting billions of internal numbers over weeks, needs many thousands of chips working on one job at once. That one requirement shapes the whole building. ## The rack: where the work happens The basic unit is the **rack**, a tall cabinet of computers. In AI, the chips that do most of the work are **GPUs**, graphics processors that perform many small calculations at the same time. A rack holds far more of them than it used to. A common current design is Nvidia's GB200 NVL72. Supermicro's product page lists 72 GPUs and 36 regular processors (CPUs) in one rack, cooled directly by liquid. The analysis firm SemiAnalysis puts its power need at about 120 kilowatts, and the comparison explains why older buildings struggle: | Type of rack | Approximate power | |---|---| | General-purpose CPU rack | 12 kW | | Air-cooled rack of H100 GPUs | 40 kW | | GB200 NVL72 | 120 kW | That is ten times the general-purpose rack, from one cabinet, and it is why a building designed for ordinary servers usually cannot take these machines without major changes. ## The network: making chips act as one Chips working on one model must constantly exchange results. If the links are slow, expensive chips sit idle while they wait. That is why the 72 GPUs in an NVL72 are joined by a dedicated high-speed connection, and why racks are packed close together, with more fast networking between them. A building spread out for convenience would waste the hardware it paid for. This also explains a difference between the two main jobs. **Training** benefits most from one huge, tightly linked cluster. **Inference**, the work of answering users, can be split across many smaller groups of chips in more places. ## Power: from the grid to the chip Electricity arrives at high voltage, is stepped down by transformers and is delivered to the racks. Two terms are worth knowing: - **IT power** is what the computing equipment itself uses. - **Facility power** adds everything else, mainly cooling. Epoch AI, a research group that tracks these sites with satellite images and filings, gives a Texas example: a site reported at 1.4 gigawatts of IT power is about 2 gigawatts in total, partly because of the hot climate and the type of cooling used. Batteries matter too. AI training can make power use jump up and down quickly, and the International Energy Agency says this makes storage important for reliable supply. Epoch's record of Colossus 2, the large xAI site in Memphis now owned by SpaceX, shows a battery facility connected to the building from an early stage. ## Cooling: moving the heat out Nearly all the electricity a chip uses ends up as heat. At 120 kilowatts a rack, moving air is no longer enough, so coolant runs through plates mounted directly on the chips. That heat is then passed outside through machines such as chillers. Colossus 2 shows the scale. Epoch counted 217 air-cooled chillers and 144 air-cooled condensers, large arrays of fans, in its most recent review. Whether a site evaporates water or uses a closed loop is a separate choice with real trade-offs, covered in [our article on water and cooling](/en/learn/ai-data-center-water-cooling/). ## A worked example: what 100 megawatts buys Here is a rough calculation for a site with 100 megawatts of IT power, using NVL72 racks and Epoch's estimates for Colossus 2. | Step | Calculation | Result | |---|---|---| | Racks | 100,000 kW ÷ 120 kW | about 830 | | GPUs | 830 × 72 | about 60,000 | | Cost per MW of IT power | $35.8 billion ÷ 946 MW | about $38 million | | Total cost | 100 × $38 million | about $3.8 billion | *This is illustrative arithmetic, not a price quote.* A check helps: citing SpaceX's stock market filing, Epoch reports a first cluster at Colossus 2 of about 110,000 GB200 processors on about 210 megawatts of compute power. That works out to roughly 1.9 kilowatts per processor, in the same range as our rack figure of about 1.7. Speed is part of the design as well. Epoch finds that going from the start of construction to 1 gigawatt of facility power typically takes one to four years in the sites it tracks. Colossus 2 reached its first cluster about eight months after land clearing began, partly by adapting an existing building and generating its own power at first. > **Try it:** Find a local news story about a data center project near you. Note the power figure, then divide it by 0.12 to estimate how many NVL72-style racks it could hold if every megawatt were IT power, which gives an upper limit. ## What should you check when you read about a new AI data center? Start with the unit and the date. A headline figure may be planned capacity years away, not power in use today, and it may mix IT power with facility power. Epoch warns that even its own careful estimates of IT power are approximate, missing the real value by more than a factor of 1.4 about one time in five. Then ask where the power comes from, how the heat leaves the building and who will use the computers. Those three answers tell you more than the total dollar figure, because they decide the local effects, the timeline and whether anyone is actually committed to paying for the capacity. Our explainer on [what a gigawatt means](/en/learn/gigawatt-ai-explained/) shows how to compare the numbers. ## Sources - Colossus 2 | AI data centers: https://epoch.ai/data/ai-data-centers/directory/colossus-2 - GB200 Hardware Architecture and Component Supply Chain: https://newsletter.semianalysis.com/p/gb200-hardware-architecture-and-component - SuperServer SRS-GB200-NVL72: https://www.supermicro.com/en/products/rack/srs-gb200-nvl72 --- From the Guide to Artificial Intelligence, by Robert Scales, published by Holistic Hybrid Inc. Share and adapt it under CC BY-NC-SA 4.0 (https://creativecommons.org/licenses/by-nc-sa/4.0/): credit the guide, link to https://guidetoartificialintelligence.com/en/learn/how-ai-data-center-works/, no commercial use without permission, and keep the same licence.