Dark Energy

Shortly after Einstein formulated the field equations for his general theory of relativity in 1915, he noticed something surprising: his equations implied that the universe should expand or contract over time. The idea of an expanding or contracting universe was too incredible even for Einstein, who initially refused to believe his own calculations. He assumed there was something wrong with his math. (This was before Hubble’s famous discovery that the universe is indeed expanding.)

Genius that he was, Einstein devised a way to make his theory compatible with the assumption that the universe is static, neither shrinking nor expanding. He realized that if the vacuum of space (so-called “empty” space) is not really empty, but has some intrinsic energy of its own, this energy would affect the curvature of spacetime in a way opposite to the gravitational curvature caused by massive objects. Over long distances, the intrinsic energy of the vacuum—vacuum energy—would act as a sort of anti-gravity, a repulsive force that could cancel out the gravitational pull between galaxies, allowing the universe to remain the same size. Einstein concluded that space must have just the right amount of vacuum energy to balance gravity and keep the universe from expanding or contracting. So, in 1917, he added an extra term to his equations, describing mathematically how vacuum energy would affect the curvature of spacetime. This extra term included a factor called the cosmological constant, symbolized by the Greek letter Λ (lambda), which represents the energy density of space itself.Einstein, “Cosmological Considerations in the General Theory of Relativity.” The paper was originally published in 1917 in German. An English translation is available here.

Twelve years later, Hubble made his astonishing discovery: space is in fact expanding. Einstein recognized, to his chagrin, that he could have predicted this discovery (as Lemaître did) if he had only trusted his original equations. He reportedly remarked that adding the cosmological constant to his equations was the “biggest blunder” of his life.The historicity of this famous quotation has been debated. See this article for more of that story. Einstein rescinded his idea that empty space contains intrinsic energy, and the cosmological constant fell out of favor. Until the end of the twentieth century, most physicists and cosmologists assumed that the value of Λ was zero, effectively deleting it from Einstein’s equations.

That’s not the end of the story, however. If there is no cosmological constant (in other words, if Λ = 0), then Einstein’s equations predict the expansion of the universe must gradually slow down, decelerated by the pull of gravity. That prediction can be tested. Although there was no way to test it in Einstein’s day, improvements in telescope technology eventually enabled astronomers to determine how the expansion of the universe has changed over time. In 1998, an international team of astronomers and cosmologists published the results of a study designed to chart the history of the universe’s expansion. By analyzing the redshift of light from distant supernovae, they were able to measure changes in the expansion rate with unprecedented precision. (See the fine print, below, for further explanation.) To their astonishment, they found that the expansion of the universe isn’t slowing down as expected. Instead, it is speeding up!Riess et al. (1998), “Observational Evidence from Supernovae for an Accelerating Universe and a Cosmological Constant,” Astronomical Journal, 116, 1009. A preprint draft is available online here. For further explanation of this remarkable discovery, I recommend this video by PBS Space Time.

To understand how the expansion of the universe can be measured, imagine a supernova exploding in some galaxy far, far away. As the flash of light travels toward us, the space between that galaxy and ours gradually expands, stretching like rubber. The expansion of space stretches the wavelength of the traveling light. By the time it reaches us, the light is “redder”—i.e., it has a longer wavelength—than it did when it was emitted. This effect, called cosmological redshift, tells us how much the universe has expanded during the time since the supernova occurred.

To determine how the rate of expansion has changed over time, an international team of scientists called the High-Z Supernova Search Team analyzed the light from dozens of type Ia supernovae. A type Ia supernova emits a distinctive spectrum of light: it lacks the hydrogen spectral lines found in other types of supernovae but displays the spectral lines of heavier elements like oxygen and silicon. By comparing the stretched wavelengths of its spectral lines to the known wavelengths emitted and absorbed by the corresponding chemical elements, the redshift of a type Ia supernova can be measured with a high degree of precision. Moreover, because a type Ia supernova is a standard candle, its distance can be calculated by comparing its luminosity with its apparent brightness. Thus, by calculating the distance to each supernova and also measuring its redshift, the High-Z scientists were able to determine how much the universe had expanded since the time of each supernova’s explosion. By doing this for many different supernovae (which occurred at varying distances and therefore at different times throughout the history of the universe), they were able to determine how the rate of expansion has changed over time.

Subsequent studies confirmed their surprising conclusion. The expansion of the universe began to accelerate about 5 billion years ago, and the rate of expansion has been gradually increasing since then. The cause of the acceleration has not yet been identified, but whatever it is, it seems to behave as an anti-gravitational force that is getting stronger with time. Physicists refer to this unknown force as dark energy.

Although the nature of dark energy is still unknown, the leading hypothesis is that Einstein’s notorious “blunder” was actually right after all: empty space has intrinsic energy that acts as an anti-gravitational force. According to this hypothesis, dark energy is just vacuum energy. In other words, the accelerating expansion of the universe is caused by the intrinsic energy of space itself. In order for vacuum energy to cause expansion, of course, it must be a little stronger than Einstein imagined. Rather than just counterbalancing the gravitational forces between galaxies, vacuum energy must be strong enough to overpower gravity, accelerating the expansion of the universe. Thus, the numerical value of the cosmological constant Λ is higher than Einstein supposed, but his basic idea was correct.

With an appropriately-chosen value for Λ, Einstein’s modified equations of general relativity (the 1917 version of his equations, which include the cosmological constant) fit the data remarkably well. On the other hand, even if the “vacuum energy” hypothesis is correct, this still doesn’t explain why or how the vacuum of space has its own energy. Where does the energy come from, and how does the total amount of vacuum energy keep increasing as space expands, apparently violating the first law of thermodynamics?

Interestingly, the Standard Model of particle physics—the currently accepted theory of subatomic particles—may explain how the vacuum of space contains energy. According to the Standard Model, space is never really empty. Quantum fields permeate all of spacetime, and these fields have some energy even where no particles are present. This seems to support the idea that vacuum energy is real, making it a plausible candidate for the role of dark energy.

There is a serious problem, however. The Standard Model says that the vacuum energy should be way too strong. According to the equations of quantum field theory, the value of Λ (i.e., the energy density of empty space) should be about 10120 times greater than the actual value measured by astronomers! In other words, the prediction is off by a factor of a trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion. (That’s a trillion multiplied by itself ten times.) Although the Standard Model has proven reliable in almost all of its other predictions, this absurd result has been called “the worst theoretical prediction in the history of physics.”Hobson, Efstathiou, and Lasenby describe the problem this way in their book General Relativity: An Introduction for Physicists (Cambridge: Cambridge University Press, 2014), page 187: “How can we calculate the energy density of the vacuum? This is one of the major unsolved problems in physics. The simplest calculation involves summing the zero-point energies of all the fields known in Nature. This gives an answer about 120 orders of magnitude higher than the upper limits set on Λ by cosmological observations. This is probably the worst theoretical prediction in the history of physics!”

The discrepancy between the Standard Model of particle physics and the ΛCDM model of cosmology is among the most pressing problems in the foundations of physics and cosmology today. No one knows how to reconcile the measured value of Λ with the predictions of quantum field theory. Even speculative extensions to the Standard Model (like the theory of supersymmetry, mentioned on the previous page) predict that the value of Λ should be either extremely large or exactly zero. In order for Λ to have a small, nonzero value, the energies of the various quantum fields—which can be positive or negative—would have to cancel each other out with exquisite precision. Such cancellation would be an extraordinarily improbable coincidence, if it happened just by chance.For a fuller explanation of the problem, see Luke A. Barnes and Geraint F. Lewis, A Fortunate Universe: Life in a Finely-Tuned Cosmos (Cambridge: Cambridge University Press, 2016), 159-164. See also this PBS Space Time episode on “The Vacuum Catastrophe,” which supplements the explanation with helpful animations and other visual aids.

What is especially remarkable about this mystery, though, is that the inexplicable value of the cosmological constant is just right for making the universe habitable. If the value of Λ were anywhere close to the number predicted by the Standard Model, the universe would have expanded so quickly that stars and galaxies could never have formed. In fact, it would have expanded too quickly for atoms to form! The only things that would exist would be elementary particles and lonely protons, separated by unfathomably vast reaches of empty space. Fortunately, the actual value of Λ is much, much lower—miraculously low, from the perspective of the Standard Model. It almost seems as though the value of Λ was deliberately chosen to allow the formation of stars, galaxies, and planets. This is just one of many ways in which the fundamental laws and constants of the universe appear to be “fine-tuned” for the existence of biological life, as we’ll see on the next page.