Across the world’s great deserts, a mysterious sheen has been found on boulders and rock faces. These layers of manganese, arsenic and silica are known as desert varnish and they are found in the Atacama desert in Chile, the Mojave desert in California, and in many other arid places. They can make the desert glitter with surprising colour and, by scraping off pieces of varnish, native people have created intriguing symbols and images on rock walls and surfaces.

How desert varnish forms has yet to be resolved, despite intense research by geologists. Most theories suggest it is produced by chemical reactions that act over thousands of years or by ecological processes yet to be determined.

Professor Carol Cleland, of Colorado University, has a very different suggestion. She believes desert varnish could be the manifestation of an alternative, invisible biological world. Cleland, a philosopher based at the university’s astrobiology centre, calls this ethereal dimension the shadow biosphere. “The idea is straightforward,” she says. “On Earth we may be co-inhabiting with microbial lifeforms that have a completely different biochemistry from the one shared by life as we currently know it.”

It is a striking idea: We share our planet with another domain of life that exists “like the realm of fairies and elves just beyond the hedgerow”, as David Toomey puts it in his newly publishedWeird Life: The Search for Life that is Very, Very Different from Our Own. But an alternative biosphere to our own would be more than a mere scientific curiosity: it is of crucial importance, for its existence would greatly boost expectations of finding life elsewhere in the cosmos. As Paul Davies, of Arizona State University, has put it: “If life started more than once on Earth, we could be virtually certain that the universe is teeming with it.”

However, by the same token, if it turns out we have failed to realise that we have been sharing a planet with these shadowy lifeforms for eons, despite all the scientific advances of the 19th and 20th centuries, then we may need to think again about the way we hunt for life on other worlds. Robot spacecraft – such as the Mars rover Curiosity – are certainly sophisticated. But what chance do they have of detecting alien entities if the massed laboratories of modern science have not yet spotted them on our own planet? This point is stressed by the US biologist Craig Venter. As he has remarked: “We’re looking for life on Mars and we don’t even know what’s on Earth!”

Cleland – working with her Colorado colleague Shelley Copley – outlined her vision of the shadow biosphere in a paper in 2006 in theInternational Journal of Astrobiology. Other astrobiologists have also proposed ideas along these lines. They include Chris McKay, who is based at Nasa’s Ames Research Centre, California, and Paul Davies, who put forward his vision of this alternative living zone in a paper in Astrobiology in 2005.

These researchers believe life may exist in more than one form on Earth: standard life – like ours – and “weird life”, as they term the conjectured inhabitants of the shadow biosphere. “All the micro-organisms we have detected on Earth to date have had a biology like our own: proteins made up of a maximum of 20 amino acids and a DNA genetic code made out of only four chemical bases: adenine, cytosine, guanine and thymine,” says Cleland. “Yet there are up to 100 amino acids in nature and at least a dozen bases. These could easily have combined in the remote past to create lifeforms with a very different biochemistry to our own. More to the point, some may still exist in corners of the planet.”

Science’s failure to date to spot this weird life may seem puzzling. The natural history of our planet has been scrupulously studied and analysed by scientists, so how could a whole new type of life, albeit a microbial one, have been missed? Cleland has an answer. The methods we use to detect micro-organisms today are based entirely on our own biochemistry and are therefore incapable of spotting shadow microbes, she argues. A sample of weird microbial life would simply not trigger responses to biochemists’ probes and would end up being thrown out with the rubbish.

That is why unexplained phenomena like desert varnish are important, she says, because they might provide us with clues about the shadow biosphere. We may have failed to detect the source of desert varnish for the simple reason that it is the handiwork of weird microbes which generate energy by oxidising minerals, leaving deposits behind them.

The idea of the shadow biosphere is also controversial and is challenged by several other scientists. “I think it is very unlikely that after 300 years of microbiology we would not have detected such organisms despite the fact that they are supposed to have a different biochemistry from the kind we know about today,” says Professor Charles Cockell, of the UK Centre for Astrobiology at Edinburgh University. “It is really quite unlikely,” adds Cockell, whose centre will be officially opened this week at a ceremony in Edinburgh.

Ways need to be found to determine whether or not the shadow biosphere exists, says Dimitar Sasselov, professor of astronomy at Harvard University and director of the Harvard Origins of Life Initiative. “If you want a clue you can count up the amount of carbon that is emitted by living things – cows, sheep, grass, plants, forests and all the planet’s bacteria. When you do, you find there is a discrepancy of around 5% when you compare the amount given off from Earth’s standard biosphere and the amount you find in the atmosphere.”

In other words, there is slightly too much carbon dioxide in the atmosphere than can be explained by the emissions of standard lifeforms on Earth. There could be an error in these calculations, of course. Alternatively, the shadow biosphere could be responsible for this excess, says Sasselov. “There is plenty of room for a shadow biosphere. That is clear. Certainly, it is not true, as some allege, that we have strong evidence to show that it does not exist. In fact, the opposite is true: we do not have good enough evidence to dismiss it.”

A key point to note is that scientists – although describing the inhabitants of the shadow biosphere as weird – still assume they will be carbon-based entities. Complex chemistry based on other elements, such as silicon, is possible, they acknowledge but these alternatives cannot create the vast range of organic materials that carbon can generate. In other words, the shadow biosphere, if it exists, will almost certainly be inhabited by carbon life, albeit of an alien variety.

“Billions of years ago, life based on different types of carbon biochemistry could have arisen in several places on Earth,” says Cleland. “These varieties would have been based on different combinations of bases and amino acids. Eventually, one – based on DNA and on proteins made from 20 amino acids – formed multicellular entities and became the dominant form of life on Earth. That is why we find that life as we know it, from insects to humans and from plants to birds, has DNA as its genetic code. However, other lifeforms based on different bases and proteins could still have survived – in the shadow biosphere.”

A different prospect is highlighted by Sasselov, who points out that a complex organic chemical can come in two different shapes even though they have the same chemical formula. Each is a mirror-image of the other and are said to have a different chirality. “Amino acids are an example,” says Sasselov. “Each comes in a right-handed version and a left-handed version. Our bodies – in common with all other lifeforms – only use left-handed versions to create proteins. Right-handed amino acids are simply ignored by our bodies. However, there may be some organisms, somewhere on the planet, that use only right-handed amino acids. They could make up the weird life of the shadow biosphere.”

But how can scientists pinpoint this weird life? Microbes are usually detected in laboratories by feeding nutrients to suspected samples so they grow and expend. Then the resulting cultures can be analysed. A weird lifeform – such as one made only of proteins formed out of right-handed amino acids – will not respond to left-handed nutrients, however. It will fail to form cultures and register its existence.

One solution to this problem is being pursued by Sasselov and colleagues’ Harvard Origins of Life Initiative. They are building an artificial cell – or bionic system – made only of right-handed components including right-handed DNA and right-handed ribosomes. “If there are right-handed lifeforms out there, many of them will be viruses – which will attempt to hijack the DNA of our bionic cells,” adds Sasselov. “When they do that they will leave evidence of their existence. Essentially we are building honey traps to catch any right-handed viruses that might live in the shadow biosphere and so reveal their existence.”

Other scientists suggest a different approach – by looking at Earth’s most inhospitable ecological niches: hot vents on the seafloor, mountaintops, highly saline lakes, Antarctic ice sheets and deserts. Standard lifeforms, mainly bacteria, have been found in these places but only a few. Some niches, researchers speculate, may prove to be just too inhospitable for standard life but may just be tolerable enough to support weird life. Microscopic studies would reveal their existence while standard culture tests would show they had a different biochemistry from standard lifeforms.

Stripes of desert varnish line the canyon walls of Capitol Gorge in Utah. No laboratory has been able to re-create the phenomenon. Photograph: Larry Geddis/AlamyAnd a promising example is provided by the desert varnish proposed as a target by Cleland and backed by David Toomey in Weird Life. “No laboratory microbiologist has been able to coax bacteria or algae to make desert varnish,” he states. “It is also possible that the stuff is the end result of some very weird chemistry but no one has been able to reproduce that either.” So yes, these sites could provide proof of the shadow biosphere’s existence, he argues.

Not surprisingly, Cleland agrees. “The only trouble is that no one has yet got round to investigating desert varnish for weird life,” adds Cleland. “I confess I find that disappointing.”