Deep inside a handful of caves, scientists are pulling genetic secrets out of bones, teeth and even the surrounding dirt, and the results are upending what we thought we knew about our own species.

Scientists analyzing the genomes of thousands of people across Japan discovered evidence for a previously overlooked third ancestral group, challenging the long-accepted “dual origins” theory. The ...