How Astronomers Revolutionized Our View of the Cosmos

In 1835 French philosopher Auguste Comte asserted that nobody would ever know what the stars were made of. “We understand the possibility of determining their shapes, their distances, their sizes and their movements,” he wrote, “whereas we would never know how to study by any means their chemical composition, or their mineralogical structure, and, even more so, the nature of any organized beings that might live on their surface.”

Comte would be stunned by the discoveries made since then. Today we know that the universe is far bigger and stranger than anyone suspected. Not only does it extend beyond the Milky Way to untold numbers of other galaxies–this would come as a surprise to astronomers of the 19th and early 20th century to whom our galaxy was “the universe”–but it is expanding faster every day. Now we can confidently trace cosmic history back 13.8 billion years to a moment only a billionth of a second after the big bang. Astronomers have pinned down our universe’s expansion rate, the mean density of its main constituents, and other key numbers to a precision of 1 or 2 percent. They have also worked out new laws of physics governing space–general relativity and quantum mechanics–that turn out to be much more outlandish than the classical laws people understood before. These laws in turn predicted cosmic oddities such as black holes, neutron stars and gravitational waves. The story of how we gained this knowledge is full of accidental discoveries, stunning surprises and dogged scientists pursuing goals others thought unreachable.

Our first hint of the true nature of stars came in 1860, when Gustav Kirchhoff recognized that the dark lines in the spectrum of light coming from the sun were caused by different elements absorbing specific wavelengths. Astronomers analyzed similar features in the light of other bright stars and discovered that they were made of the same materials found on Earth–not of some mysterious “fifth essence” as the ancients had believed.

But it took longer to understand what fuel made the stars shine. Lord Kelvin (William Thomson) calculated that if stars derived their power just from gravity, slowly deflating as their radiation leaked out, then the sun’s age was 20 million to 40 million years–far less time than Charles Darwin or the geologists of the time inferred had elapsed on Earth. In his last paper on the subject, in 1908, Kelvin inserted an escape clause stating that he would stick by his estimate “unless there were some other energy source laid up in the storehouse of creation.”

That source, it turned out, is nuclear fusion–the process by which atomic nuclei join to create a larger nucleus and release energy. In 1925 astrophysicist Cecilia Payne-Gaposchkin used the light spectra of stars to calculate their chemical abundances and found that, unlike Earth, they were made mainly of hydrogen and helium. She revealed her conclusions in what astronomer Otto Struve described as “the most brilliant Ph.D. thesis ever written in astronomy.” A decade later physicist Hans Bethe showed that the fusion of hydrogen nuclei into helium was the main power source in ordinary stars.

At the same time stars were becoming less mysterious, so, too, was the nature of fuzzy “nebulae” becoming clearer. In a “great debate” held before the National Academy of Sciences in Washington, D.C., on April 26, 1920, Harlow Shapley maintained that our Milky Way was preeminent and that all the nebulae were part of it. In contrast, Heber Curtis argued that some of the fuzzy objects in the sky were separate galaxies–“island universes”–fully the equal of our Milky Way. The conflict was settled not that night but just a few years later, in 1924, when Edwin Hubble measured the distances to many nebulae and proved they were beyond the reaches of the Milky Way. His evidence came from Cepheids, variable stars in the nebulae that reveal their true brightness, and thus their distance, by their pulsation period–a relation discovered by Henrietta Swan Leavitt.

Soon after Hubble realized that the universe was bigger than many had thought, he found that it was still growing. In 1929 he discovered that spectral features in the starlight from distant galaxies appeared redder–that is, they had longer wavelengths–than the same features in nearby stars. If this effect was interpreted as a Doppler shift–the natural spreading of waves as they recede–it would imply that other galaxies were moving away from one another and from us. Indeed, the farther away they were, the faster their recession seemed to be. This was the first clue that our cosmos was not static but was expanding all the time.

The universe also appeared to contain much that we could not see. In 1933 Fritz Zwicky estimated the mass of all the stars in the Coma cluster of galaxies and found that they make up only about 1 percent of the mass necessary to keep the cluster from flying apart. The discrepancy was dubbed “the missing mass problem,” but many scientists at the time doubted Zwicky’s suggestion that hidden matter might be to blame. The question remained divisive until the 1970s, when work by Vera Rubin and W. Kent Ford (observing stars) and by Morton Roberts and Robert Whitehurst (making radio observations) showed that the outer parts of galactic disks would also fly apart unless they were subject to a stronger gravitational pull than stars and gas alone could provide. Finally, most astronomers were compelled to accept that some kind of “dark matter” must be present. “We have peered into a new world,” Rubin wrote, “and have seen that it is more mysterious and more complex than we had imagined.” Scientists now believe that dark matter outnumbers visible matter by about a factor of five, yet we are hardly closer than we were in the 1930s to figuring out what it is.

Gravity, the force that revealed all that dark matter, has proved to be nearly as baffling. A pivotal moment came in 1915 when Albert Einstein published his general theory of relativity, which transcended Isaac Newton’s mechanics and revealed that gravity is actually the deformation of the fabric of space and time. This new theory was slow to take hold. Even after it was shown to be correct by observations of a 1919 solar eclipse, many dismissed the theory as an interesting quirk–after all, Newton’s laws were still good enough for calculating most things. “The discoveries, while very important, did not, however, affect anything on this earth,” astronomer W.J.S. Lockyer told the New York Times after the eclipse. For almost half a century after it was proposed, general relativity was sidelined from the mainstream of physics. Then, beginning in the 1960s, astronomers started discovering new and extreme phenomena that only Einstein’s ideas could explain.

One example lurks in the Crab Nebula, one of the best-known objects in the sky, which is composed of the expanding debris from a supernova witnessed by Chinese astronomers in a.d. 1054. Since it appeared, the nebula has kept on shining blue and bright–but how? Its light source was a longtime puzzle, but the answer came in 1968, when the dim star at its center was revealed to be anything but normal. It was actually an ultracompact neutron star, heavier than the sun but only a few miles in radius and spinning at 30 revolutions per second. “This was a totally unexpected, totally new kind of object behaving in a way that astronomers had never expected, never dreamt of,” said Jocelyn Bell Burnell, one of the discoverers of the phenomenon. The star’s excessive spin sends out a wind of fast electrons that generate the blue light. The gravitational force at the surface of such an incredibly dense object falls way outside of Newton’s purview–a rocket would need to be fired at half the speed of light to escape its pull. Here the relativistic effects predicted by Einstein must be taken into account. Thousands of such spinning neutron stars–called pulsars–have been discovered. All are believed to be remnants of the cores of stars that exploded as supernovae, offering an ideal laboratory for studying the laws of nature under extreme conditions.

The most exotic result of Einstein’s theory was the concept of black holes–objects that have collapsed so far that not even light can escape their gravitational pull. For decades these were only conjecture, and Einstein wrote in 1939 that they “do not exist in physical reality.” But in 1963 astronomers discovered quasars: mysterious, hyperluminous beacons in the centers of some galaxies. More than a decade passed before a consensus emerged that this intense brightness was generated by gas swirling into huge black holes lurking in the galaxies’ cores. It was the strongest evidence yet that these bizarre predictions of general relativity actually exist.

When did the universe begin? Did it even have a beginning? Astronomers had long debated these questions when, in the middle of the 20th century, two competing theories proposed very different answers. The “hot big bang” model said the cosmos began extremely small, hot and dense and then cooled and spread out over time. The “steady state” hypothesis held that the universe had essentially existed in the same form forever.

The contest was settled by a serendipitous discovery. In 1965 radio astronomers Arno Penzias and Robert Wilson were trying to calibrate a new antenna at Bell Labs in New Jersey. They had a problem: no matter what they did to reduce background interference, they measured a consistent level of noise in every direction. They even evicted a family of pigeons that had been nesting in the antenna in the hope that they were the source of the problem. But the signal persisted. They had discovered that intergalactic space is not completely cold. Instead it is warmed to nearly three kelvins (just above absolute zero) by weak microwaves. Penzias and Wilson had accidentally uncovered the “afterglow of creation”–the cooled and diluted relic of an era when everything in the universe was squeezed until it was hot and dense.

The finding tipped the balance firmly in favor of the big bang picture of cosmology. According to the model, during the earliest, hottest epochs of time, the universe was opaque, rather like the inside of a star, and light was repeatedly scattered by electrons. When the temperature fell to 3,000 kelvins, however, the electrons slowed down enough to be captured by protons and created neutral atoms. Thereafter light could travel freely. The Bell Labs signal was this ancient light, first released about 300,000 years after the birth of the universe and still pervading the cosmos–what we call the cosmic microwave background. It took a while for the magnitude of the discovery to sink in for the scientists who made it. “We were very pleased to have a possible explanation [for the antenna noise], but I don’t think either of us really took the cosmology very seriously at first,” Wilson says. “Walter Sullivan wrote a first-page article in the New York Times about it, and I began to think at that point that, you know, maybe I better start taking this cosmology seriously.”

Measurements of this radiation have since enabled scientists to understand how galaxies emerged. Precise observations of the microwaves reveal that they are not completely uniform over the sky. Some patches are slightly hotter, others slightly cooler. The amplitude of these fluctuations is only one part in 100,000, but they are the seeds of today’s cosmic structure. Any region of the expanding universe that started off slightly denser than average expanded less because it was subjected to extra gravity; its growth lagged further and further, the contrast between its density and that of its surroundings becoming greater and greater. Eventually these clumps were dense enough that gas was pulled in and compressed into stars, forming galaxies. The crucial point is this: Computer models that simulate this process are fed the initial fluctuations measured in the cosmic microwave background, which represent the universe when it was 300,000 years old. The output after 13.8 billion years of virtual time have elapsed is a cosmos where galaxies resemble those we see, clustered as they are in the actual universe. This is a real triumph: we understand, at least in outline, 99.998 percent of cosmic history.

It is not only the big cosmic picture that we have come to understand. A series of discoveries has also revealed the history of the elemental building blocks that make up stars, planets and even our own bodies.

Starting in the 1950s, progress in atomic physics led to accurate modeling of stars’ surface layers. Simultaneously, detailed knowledge of the nuclei not just of hydrogen and helium atoms but also of the rest of the elements allowed scientists to calculate which nuclear reactions dominate at different stages in a star’s life. Astronomers came to understand how nuclear fusion creates an onion-skin structure in massive stars as atoms successively fuse to build heavier and heavier elements, ending with iron in the innermost, hottest layer.

Astronomers also learned how stars die when they exhaust their hydrogen fuel and blow off their outer gaseous layers. Lighter stars then settle down to a quiet demise as dense, dim objects called white dwarfs, but heavier stars shed more of their mass, either in winds during their lives or in an explosive death via supernova. This expelled mass turns out to be crucial to our own existence: it mixes into the interstellar medium and recondenses into new stars orbited by planets such as Earth. The concept was conceived by Fred Hoyle, who developed it during the 1950s along with two other British astronomers, Margaret Burbidge and Geoffrey Burbidge, and American nuclear physicist William Fowler. In their classic 1957 paper in Reviews of Modern Physics (known by the initials of its authors as BBFH), they analyzed the networks of the nuclear reactions involved and discovered how most atoms in the periodic table came to exist. They calculated why oxygen and carbon, for instance, are common, whereas gold and uranium are rare. Our galaxy, it turns out, is a huge ecological system where gas is being recycled through successive generations of stars. Each of us contains atoms forged in dozens of different stars spread across the Milky Way that lived and died more than 4.5 billion years ago.

Scientists long assumed this process was seeding planets–and possibly even life–around stars other than our own sun. But we did not know for sure whether planets existed outside our solar system until the 1990s, when astronomers developed clever methods for identifying worlds that are too dim for us to see directly. One technique looks for tiny periodic changes in a star’s movement caused by the gravitational pull of a planet orbiting it. In 1995 Michel Mayor and Didier Queloz used this strategy to detect 51 Pegasi b, the first known exoplanet orbiting a sunlike star. The technique can reveal a planet’s mass, the length of its “year” and the shape of its orbit. So far more than 800 exoplanets have been found this way. A second technique works better for smaller planets. A star dims slightly when a planet transits in front of it. An Earth-like planet passing a sunlike star can cause a dimming of about one part in 10,000 once per orbit. The Kepler spacecraft launched in 2009 found more than 2,000 planets this way, many no bigger than Earth. A big surprise to come from astronomers’ success in planet hunting was the variety of different planets out there–many much larger and closer to their stars than the bodies in our solar system–suggesting that our cosmic neighborhood may be somewhat special.

By this point scientists understood where almost all the elements that form planets, stars and galaxies originated. The final piece in this puzzle, however, arrived very recently and from a seemingly unrelated inquiry.

General relativity had predicted a phenomenon called gravitational waves–ripples in spacetime produced by the movement of massive objects. Despite decades of searching for them, however, no waves were seen–until September 2015. That was when the Laser Interferometer Gravitational-wave Observatory (LIGO) detected the first evidence of gravitational waves in the form of a “chirp”–a minute shaking of spacetime that speeds up and then dies away. In this case, it was caused by two black holes in a binary system that had started out orbiting each other but gradually spiraled together and eventually converged into a single massive hole. The crash occurred more than a billion light-years away. LIGO’s detectors consist of mirrors four kilometers apart whose separation is measured by laser beams that reflect light back and forth between them. A passing gravitational wave causes the space between the two mirrors to jitter by an amount millions of times as small as the diameter of a single atom–LIGO is indeed an amazing feat of precision engineering and perseverance.

Since that first find, more than a dozen similar events have been detected, opening up a new field that probes the dynamics of space itself. One event was of special astrophysical interest because it signaled the merger of two pulsars. Unlike black hole mergers, this kind of collision, a splat between two ultradense stars, yields a pulse of optical light, x-rays and gamma rays. The discovery filled a gap in the classic work of BBFH: the authors had explained the genesis of many of the elements in space but were flummoxed by the forging of gold. In the 1970s David N. Schramm and his colleagues had speculated that the exotic nuclear processes involved in hypothetical mergers of pulsar stars might do the job–a theory that has since been validated.

Despite the incredible progress in astronomy over the past 175 years, we have perhaps more questions now than we did back then.

Take dark matter. I am on record as having said more than 20 years ago that we would know dark matter’s nature long before today. Although that prediction has proved wrong, I have not given up hope. Dark energy, however, is a different story. Dark energy entered the picture in 1998, when researchers measuring the distances and speeds of supernovae found that the expansion of the universe was actually accelerating. Gravitational attraction pulling galaxies toward one another seemed to be overwhelmed by a mysterious new force latent in empty space that pushes galaxies apart–a force that came to be known as dark energy. The mystery of dark energy has lingered–we still do not know what causes it or why it has the particular strength it does–and we probably will not understand it until we have a model for the graininess of space on a scale a billion billion times smaller than an atomic nucleus. Theorists working on string theory or loop quantum gravity are tackling this challenge, but the phenomenon seems so far from being accessible by any experiment that I am not expecting answers anytime soon. The upside, however, is that a theory that could account for the energy in the vacuum of space might also yield insights into the very beginning of our universe, when everything was so compressed and dense that quantum fluctuations could shake the entire cosmos.

Which brings us to another major question facing us now: How did it all begin? What exactly set off the big bang that started our universe? Did space undergo a period of extremely rapid early expansion called inflation, as many theorists believe? And there is something else: some models, such as eternal inflation, suggest that “our” big bang could be just one island of spacetime in a vast archipelago–one big bang among many. If this hypothesis is true, different big bangs may cool down differently, leading to unique laws of physics in each case–a “multiverse” rather than a universe. Some physicists hate the multiverse concept because it means that we will never have neat explanations for the fundamental numbers that govern our physical laws, which may in this grander perspective be just environmental accidents. But our preferences are irrelevant to nature.

About 10 years ago I was on a panel at Stanford University where we were asked by someone in the audience how much we would bet on the multiverse concept. I said that on a scale of betting my goldfish, my dog or my life, I was nearly at the dog level. Andrei Linde, who had spent 25 years promoting eternal inflation, said he would almost bet his life. Later, on being told this, physicist Steven Weinberg said he would happily bet my dog and Linde’s life. Linde, my dog and I will all be dead before the question is settled. But none of this should be dismissed as metaphysics. It is speculative science–exciting science. And it may be true.

And what will happen to this universe–or multiverse–of ours? Long-range forecasts are seldom reliable, but the best and most conservative bet is that we have almost an eternity ahead with an ever colder and ever emptier cosmos. Galaxies will accelerate away and disappear. All that will be left from our vantage point will be the remnants of the Milky Way, Andromeda and smaller neighbors. Protons may decay, dark matter particles may be annihilated, there may be occasional flashes when black holes evaporate–and then silence.

This possible future is based on the assumption that the dark energy stays constant. If it decays, however, there could be a “big crunch” with the universe contracting in on itself. Or if dark energy strengthens, there would be a “big rip” when galaxies, stars and even atoms are torn apart.

Other questions closer to home tantalize us. Could there be life on any of these new planets we are discovering? Here we are still in the realm of speculation. But unless the origin of life on Earth involved a rare fluke, I expect evidence of a biosphere on an exoplanet within 20 years. I will not hold my breath for the discovery of aliens, but I think the search for extraterrestrial intelligence is a worthwhile gamble. Success in the search would carry the momentous message that concepts of logic and physics are not limited to the hardware in human skulls.

Until now, progress in cosmology and astrophysics has owed 95 percent to advancing instruments and technology and less than 5 percent to armchair theory. I expect that balance to persist. What Hubble wrote in the 1930s remains a good maxim today: “Not until the empirical resources are exhausted, need we pass on to the dreamy realms of speculation.”

There have been many particularly exhilarating eras in the past 175 years–the 1920s and 1930s, when we realized the universe was not limited to the Milky Way, and the 1960s and 1970s, when we discovered objects that defy classical physics, such as neutron stars and quasars, and clues about the beginning of time from the cosmic microwave background. Since then, the pace of advancement has crescendoed rather than slackened.

When the history of science gets written, this amazing progress will be acclaimed as one of its greatest triumphs–up there with plate tectonics, the genome and the Standard Model of particle physics. And some major fields in astronomy are just getting going. Exoplanet research is only 25 years old, and serious work in astrobiology is really only starting. Some exoplanets may have life–they may even harbor aliens who know all the answers already. I find that encouraging.