Every now and again cosmologists decide that the universe needs redecorating. Sometimes they declutter, as when Copernicus and Kepler shuffled the sun and the Earth to get rid of all those epicycles and make the planets move in straightforward orbits. Sometimes they embellish, as when Einstein decided that there's more to space than good old-fashioned nothingness, and introduced the concept of a deformable space-time.
They are at it again, but this time it's different. Like the decorator who strips away a layer of wallpaper to reveal a crumbling wall, cosmologists are realising that their findings point to serious problems with their models of the structure of the universe. This discovery is forcing them to contemplate bold changes to fix the damage.
When they are done, chances are we will hardly recognise the old place. "It will repaint not only our picture of the universe but perhaps particle physics, gravitational physics and string theory too," says Rocky Kolb, a cosmologist at Fermilab in Batavia, Illinois.
The problem giving cosmologists their big headache goes under the name of "dark energy". This enigmatic entity - which could be some kind of a substance, or a field, or maybe something else entirely - forced itself into cosmologists' consciousness in 1998, when astronomers discovered that something is speeding up the expansion of the universe. Almost a decade later, it is beginning to sink in that there is no easy way to understand what dark energy might be. The problem has become so intractable that many now see it as the greatest challenge facing physics.
The scale of the problem has galvanised astronomers into urgent action. Scanning the skies in ever greater detail, their observations could soon lead us to the origin and nature of what could, according to some theories, make up almost three-quarters of the cosmos, and which will ultimately dictate its fate. "Dark energy is more of a challenge for physicists than it is for astronomers," says Kolb. "Astronomers just measure the acceleration of the universe but physicists have to explain what dark energy actually is."
There is no way to detect dark energy directly, so we have to measure its effects. The most obvious of these is the one that gave it away in the first place: the way it forces the expansion of the universe to accelerate.
Cosmic cure-all
Its discovery came about like this. Two independent teams of astronomers were using the Hubble Space Telescope and a host of large ground-based telescopes to track down supernovae in the distant universe. By measuring the wavelength and intensity of the light from these exploding stars it is possible to look back through cosmic history and calculate how fast the universe has been expanding during the past few billion years. What everyone expected was that the expansion that started with the big bang would be slowing down, as the outward rush of individual galaxies gets pulled back by the gravitational attraction of the rest of the universe. To their surprise, both teams' calculations showed that the opposite was happening: the rate of expansion was actually increasing.
Though this went against everything we thought we knew about the universe, the results were beyond dispute. "The fact that two independent teams came to the same conclusion certainly boosted everyone's confidence," says Adam Reiss of the Space Telescope Science Institute in Baltimore, Maryland, who led one of the teams.
Even before these astonishing results, cosmologists had been getting uncomfortable hints that something was wrong with their models of how the universe works. One of these came from detailed observations of the radiation released by the big bang. The only way to fit the observations to existing cosmological models was to slightly warp the fabric of space-time. Such warping is impossible to explain unless there is something in addition to all the normal matter, neutrinos, dark matter and radiation that we know about. Dark energy now seems to fit the bill.
For a while, cosmologists could dream that dark energy would solve various other problems too. It explained why certain stars seemed to be older than the universe itself; it provided possible clues about the nature of the dark matter that seems to be holding individual galaxies together; and maybe it could explain "inflation", the sudden acceleration in the expansion of the universe that happened within a blink of the big bang.
Dark energy on trial
That honeymoon period is now well and truly over. Although dark energy is a ubiquitous term in cosmological conversations, no one actually knows what it is. As Kolb says: "Naming is not explaining."
Although there are plenty of tentative explanations, each one seems to suffer from some fatal flaw. The simplest of the solutions on offer is the so-called cosmological constant. This is an energy associated with space-time that was originally invoked by Einstein in his equations of general relativity. It represents a cosmic repulsion that Einstein fine-tuned to prevent the universe - which he did not at the time realise was expanding - from collapsing in on itself as a result of all the gravity generated by the various celestial objects.
When Einstein learned of Edwin Hubble's discovery that space is indeed expanding, he realised that the cosmological constant was superfluous and famously called it his "biggest blunder". Now the accelerating expansion of the universe is making astronomers wonder whether there might be a cosmological constant after all, driving the universe's acceleration (see "1: a new form of energy").
Unfortunately, physicists are having trouble finding a way to fit a cosmological constant into their best existing theories. "A small non-zero dark energy is more difficult to explain than zero," says Sean Carroll, a cosmologist from the California Institute of Technology in Pasadena. "So we are driven to wilder ideas."
One of those wild ideas is quintessence, which postulates the existence of a hitherto unsuspected quantum field permeating the universe (see "2: a new force of nature"). Because this implies that there would also be a new fundamental force of nature, the idea set some physicists thinking: instead of adding a new force, why not modify an old one? Perhaps there are unexpected properties of gravity that appear over gargantuan distances that Einstein's general relativity does not predict (see "3: modify an old force").
Defenders of general relativity point out that the problem is not with general relativity, but with an even more fundamental aspect of our universe. They point out that it has been assumed for almost a century that the universe is the same in every direction you look. Let go of that assumption and the more complicated solutions of general relativity that result could lead to acceleration without the need for dark energy (see "4: introduce complexity").
Faced with these disparate approaches, not to mention the several variations that exist within each one, it is no wonder that cosmologists are scratching their heads wondering what to do for the best. Last year, two independent committees of leading cosmologists were convened to answer this question. Kolb chaired the Dark Energy Task Force, which reported to the US Department of Energy, NASA and the National Science Foundation. Its recommendation is for an "aggressive program to explore dark energy as fully as possible, since it challenges our understanding of fundamental physical laws and the nature of the cosmos". In Europe, John Peacock of the University of Edinburgh, UK, convened a committee under the auspices of the European Space Agency (ESA) and the European Southern Observatory. It came to a similar conclusion. Of all the challenges in cosmology, the discovery of dark energy "poses the greatest challenge for physics" because there is no "plausible or natural" explanation for it, says Peacock's committee.
How do they propose to tackle this? It's simple: with the biggest ever survey of the universe, to see whether dark energy changes with time and, if it does, how fast it changes. If dark energy is a manifestation of the cosmological constant, it will be unchanging. By contrast, quintessence is variable and could change over time, or from place to place in the universe. Modified gravity has similar, though not identical, characteristics.
Astronomical surveys will show the distorting effects that dark energy has on the distribution of galaxies across the universe. The more galaxies astronomers examine, the more marked these effects will be; and the further the survey reaches into the universe, the easier it will become to see if dark energy has changed with time.
The most comprehensive study is due to start in 2012, when the Large Synoptic Survey Telescope begins operating from Cerro Pachón in Chile. With its whopping 8.4-metre mirror and wide-field camera, the LSST is a monster that will devour the sky. It will see 400 times the area of the full moon in a single glance, and take an image every 15 seconds. In just three days it will be able to record the entire visible night sky.
Eventually the search will move into space for even greater accuracy and sensitivity. NASA and the US Department of Energy are funding three design studies for the Joint Dark Energy Mission, which they hope will launch sometime between 2011 and 2017. Peacock recommends that ESA should also investigate a project.
Even before these mega-projects begin, we may start to get answers. Astronomers already have most of the equipment to hand to start their grand survey, as observatories around the world are littered with outmoded telescopes. About 15 years ago, 4-metre telescopes were at the cutting edge of research, but now they are floundering in the wake of a new generation of larger instruments. "The 4-metre telescopes have been eclipsed by 8-metre telescopes," says Peacock, who is now pushing for them to be used for surveys.
The most ambitious map of the sky to date is the Sloan Digital Sky Survey. Using a 2.5-metre telescope at Apache Point, New Mexico, it has over the past five years collected light from 675,000 galaxies. A 4-metre telescope could not only work faster than this, but also reach further back into the universe's history. All that is required to begin the survey is a wide-angle camera to take pictures of large areas of the sky simultaneously.
Ofer Lahav of University College London has a plan to do just this. He leads a consortium of astronomers who are planning to build the kind of wide-field camera necessary for survey work. "Our survey could see 500 million galaxies," says Lahav. These would be spread throughout three-quarters of the visible universe. To cope with the flood of data, Lahav's team has used existing images of the sky to train a neural network to recognise galaxies and estimate their distances. The team also has permission to use its camera in conjunction with the 4-metre Blanco telescope in Cerro Tololo, Chile, and is now looking for the $20 to $30 million that will be needed to build the highly sophisticated optics and run the telescope.
Peacock would lik e to see many more such efforts - and soon. "We have to start now," he says. It's a big sky and there are plenty of telescopes to do the job, he points out. The more of them that can be brought to bear, the bigger and better the eventual survey will be.
Gone are the days when astronomical surveys like this were viewed as mundane, speculative chores. By giving us detailed measurements of the acceleration of different parts of the universe, the next generation of surveys could reveal the nature of the dominant component of the universe. Whatever it turns out to be, it will be big news. "Dark energy could be the ether of the 21st century," says Carroll. Even if we explain it away, we will learn something profound about the universe.
It is a viewpoint shared by cosmologists everywhere. "We are definitely seeing something extra in the universe, we just do not know how to interpret it yet," says Lahav. And that has given cosmologists a new sense of purpose. A seismic shift in our understanding of the universe is coming. How soon it will arrive and from what direction it will come - that's still anyone's guess.
1 a new form of energy
Einstein himself flirted with a weird form of energy that might just fit the bill. He called it the cosmological constant. These days physicists prefer the name vacuum energy, and like to think of it as the "cost" of free space. By that they mean that every cubic metre of space, no matter how cold or empty, contains a certain amount of energy. According to the equations of general relativity, this energy drives the expansion of the universe.
"Had everyone been happy with the cosmological constant there would be no need to continue," says cosmologist Rocky Kolb of Fermilab in Illinois. The trouble is, no one really is happy with it. One reason for this is that quantum theory predicts a vacuum energy that is 120 orders of magnitude larger than what is needed to cause the observed acceleration in the universe's expansion. This colossal discrepancy is one reason why physicists formulated supersymmetry theory, which cancels out vacuum energy completely.
The trouble is, the universe has other ideas: if the dark energy pushing it apart really is vacuum energy, the small amount that exists is infuriatingly difficult to explain. It certainly defeats any existing model.
"If dark energy is the cosmological constant then we will just have to wait for the theorists to catch up," says Adam Reiss of the Space Telescope Science Institute in Baltimore.
2 a new force of nature
"When physicists don't understand something, they invent a new field to explain it," says cosmologist Rocky Kolb of Fermilab. "Now astronomers have also learned that trick."
In the case of the dark energy mystery, the result is a quantum field called quintessence. Like the cosmological constant, quintessence is said to pervade the universe, but one of its key differences from the cosmological constant is that it can vary depending on the time and the place. Various versions have sprung up depending on how fast they vary. One version, called phantom energy, builds with time, forcing the expansion faster and faster until eventually the universe rips itself to pieces.
In November 2006, a team led by astronomer Adam Reiss of the Space Telescope Science Institute in Baltimore, Maryland, announced that they had detected dark energy's influence on the universe as it existed 9 billion years ago (www.arxiv.org/abs/astro-ph/0611572). Reiss says his team's discovery rules out quintessence models that change rapidly. "It is narrowing our room to play a little," agrees cosmologist Sean Carroll of the California Institute of Technology in Pasadena. As more dark energy surveys get under way, he hopes they will narrow the field even more, eventually forcing everyone to converge on a single solution.
There are also some more fundamental problems that any solution involving quintessence will have to overcome. In the more familiar quantum fields, fluctuations in the field manifest themselves as particles. In the electromagnetic field, for instance, such fluctuations appear as photons.
Does this mean the same should happen for quintessence? Absolutely, says Carroll. Fluctuations in its field should lead to particles that can carry a quintessence force over large distances. This force would act between individual objects and be distinct from the general acceleration of the universe caused by the overall quintessence field.
The trouble is, no such quintessence force has shown itself. It should be apparent as a measurable deviation in the motion of celestial objects. "By all rights we should have detected it by now," Carroll says. This is forcing theorists to try to fine-tune their expectations to reduce the force of quintessence between individual objects while retaining its dominant character across the universe. Tricky.
3 modify an old force
Despite the slew of observations that make it look as though dark energy of one form or another is operating in the universe, astronomer Adam Reiss remains cautious. The common assumption, he points out, is that gravity operates the same way on large scales as it does on small scales. But what if it doesn't? If there were some unexpected gravitational effect that has remained undetected until now, dark energy might not be needed at all.
This idea that there might be some modification to gravity caught the attention of Caltech cosmologist Sean Carroll for a while, but he soon found it was not a short cut to a solution. "It turns out to be much harder than you imagine to find a modification that works," he says.
That's because modifying gravity to give large-scale acceleration also results in unwanted small-scale alterations, such as deviations to the way the planets orbit in the solar system. Carroll says he is now moving away from modified theories of gravity to explain away dark energy.
Not everyone is giving up. "No one promised it would be easy," says Gia Dvali, a theorist at New York University. He has developed a modified theory of gravity in which space-time is not as formless as we tend to think. According to the theory, which he developed with his colleagues Gregory Gabadadze and Massimo Porrati, space-time has a limited underlying shape that makes it look as if a weird form of energy is warping it.
The warping happens because gravitons - the as yet undiscovered particles that are presumed to carry gravity - have a small mass, and decay into other dimensions with half-lives of 15 billion years. This is strikingly similar to the age of the universe. "We don't know whether this is just a remarkable coincidence or the result of something more fundamental," says Dvali.
According to Dvali's calculations, such a modification of gravity would explain the acceleration of the universe's expansion. It would also alter the moon's orbit by about a millimetre away from the expectations of general relativity. A team of astronomers from Harvard University and the University of Washington in Seattle are planning to attempt this measurement using the mirrors left behind on the lunar surface by the Apollo astronauts.
4 introduce complexity
Perhaps the most outrageous - and yet paradoxically the most conservative - solution is to alter an assumption so ingrained in cosmology that most cosmologists have forgotten it is there. Called the cosmological principle it states, in essence, that viewed on sufficiently large scales the universe has no preferred directions or preferred places. "We have unquestioningly lived with this assumption for 85 years," says cosmologist Rocky Kolb.
It was introduced in the 1920s by Alexander Friedman to make the equations of general relativity tractable. It meant Friedman could think of the galaxies as particles in a uniform fluid that fills space. Cosmologists have stuck with Friedman's idea ever since, despite finding ever larger density variations across the universe. It might be time to ditch that assumption, suggests Kolb.
If the universe is no longer the same everywhere, effects of general relativity that are negligible in a uniform cosmos might become increasingly important. "It is just an idea at the moment, but sooner or later we are going to have to do the calculations and make a prediction," says Kolb.
That's where it gets tough, because to do that will require us finding a way to somehow meld general relativity with complexity theory. "We cannot do it yet, but one day a clever graduate student will see how to do the calculation," Kolb says. "I just hope he or she will be working for me.
