As the world waits for construction of the largest fusion reactor yet, called ITER, smaller reactors with similar designs are still running. These reactors, called tokamaks, help us test both hardware and software. The hardware testing helps us refine things like the materials used for container walls or the shape and location of control magnets.
But arguably, the software is the most important. To enable fusion, the control software of a tokamak has to monitor the state of the plasma it contains and respond to any changes by making real-time adjustments to the system’s magnets. Failure to do so can result in anything from a drop in energy (which leads to the failure of any fusion) to seeing the plasma spill out of containment (and scorch the walls of the container).
Getting that control software right requires a detailed understanding of both the control magnets and the plasma the magnets manipulate. Or it would be more accurate to say, getting that control software right has required. Because today, Google’s DeepMind AI team is announcing that its software has been successfully trained to control a tokamak.
Out of control
Developing the control software for a tokamak is a complicated process. Based on past experience with similar designs, engineers can extract some of the basic principles needed for the software to function, like what sensor inputs to read and how to respond to changes in them. But there are always quirks based on the design of the hardware and energies of the plasma being used. So, there tends to be an iterative process of measuring and modeling, followed by tweaks to the control process, all the while keeping the performance sufficient to make adjustments in near real time.
The resulting control software tends to be fairly specialized. If researchers want to experiment with a very different geometry for the plasma in the tokamak, a significant revision to the software may be required.
Researchers in the field had already identified artificial intelligence as a possible solution. Give the right AI sufficient examples, and it could figure out which control configurations produce the desired properties in the plasma. That would free people to focus on the desired end-state they wanted and then just letting the software produce it for them so that they could study it. An AI should also be more flexible; once it is trained on how to control the system, it should be able to produce very different plasma configurations for study without the need for reprogramming.
To move forward on this idea, all we needed were AI experts and a tokamak. For the new paper, the AI team came from Google’s DeepMind division, famed for developing software that could handle everything from protein folding to StarCraft. The tokamak comes courtesy of the Swiss Plasma Center at the EPFL in Lausanne.
Trained to fuse
Since setting the AI loose on actual hardware during the training process could be a disaster, the team started out with a tokamak simulator specific for the Swiss Plasma Center hardware. This was largely accurate, and they programmed limits into the AI that kept it from directing the plasma into a configuration where the simulator produced inaccurate results. DeepMind then trained a deep-reinforcement-learning program to reach a variety of plasma configurations by letting it control the simulator.
During training, an intervening layer of software provided a reward function that indicated how close the plasma’s properties were to the desired state. Another algorithm, termed a “critic,” learned the expected rewards for various changes to the tokamak’s control magnets. These were used by the actual control neural network to learn which actions it should take.
The critic was elaborate and computationally expensive, but it was only used during the training portion. When training was done, the control algorithm had learned which actions to take to reach a variety of states, and the critic could be discarded.
In order to allow real-time performance, the trained controller was bundled as an executable. The standard control software would be used to activate the tokamak and bring a plasma up to high energies. Once the plasma was stable, it handed off control to the AI.
It works!
The resulting software performed pretty much as you would want it to when set loose on actual hardware. The software could control experimental runs that targeted different conditions over time—in one test case, it ramped up the energy, held the plasma steady, then altered the plasma’s geometry, then relocated the plasma within the tokamak before ramping the energy back down. In another, it held two separate plasma structures in the same tokamak simultaneously.
The paper describing this work has a large list of the things that the authors needed. That list includes a tokamak simulator that was both detailed enough to be accurate but compact enough to provide feedback quickly enough to make reinforcement learning possible. The training set had to include both common conditions similar to where control was handed to it and unusual conditions that let it learn how to transition those to experimental configurations. Additionally, the researchers needed to develop software that was detailed enough to evaluate a huge range of potential control options but also able to train a fast-performing controller that could be compiled into an executable.
The people behind this work are also excited about what it might presage for future work. Rather than simply limiting things to modeling existing hardware, they suggest that it should be possible to give an iteration of this software a desired plasma configuration, and let it identify the hardware geometry that will allow it to create that. Alternately, it could optimize the performance of existing hardware.
Now we just need to wait for a fusion reactor worthy of the AI’s attention.
Nature, 2022. DOI: 10.1038/s41586-021-04301-9 (About DOIs).
