It really doesn't, not to people or even animals. What really matters is habitat change, which may or may not be caused in part by climate change. Climate change may also have a negligible effect on habitats for some plants and animals. The main reason scientists don't talk about habitat change is because that would force scientists to assign a ranking to the factors that cause habitat change. If they were to be honest about the causes of habitat change, they would be forced to admit climate change usually ranks pretty low as a causal factor in most cases.
Examining the vast majority of cases of global habitat change and habitat depletion, the 2 main causes clearly are human overpopulation and human industrialization. If the climate were to remain static, habitats would still disappear. I'm not just talking about animal and plant habitats disappearing. Human habitats are also destroyed from resource depletion from runaway industrialization, and this depletion is then multiplied by runaway overpopulation. While climate change may be a concerning lesser factor toward habitat depletion, the hyperfocus on climate change does little to address the real major causes of habitat depletion.
Considering the world's general apathy toward climate change, it's alarming to note an even greater global apathy toward runaway population and runaway industrialization. So the question is thus: at what point will this apathy abate? How much habitat needs to be destroyed before scientists focus their studies on the effects of overpopulation and over industrialization? Most of it?
The one sure thing is that the planet has built in defenses from overpopulation and resource depletion with the mechanics of starvation and disease. If humans get to that point, which will happen if these 2 problems are not addressed, it will take thousands of years to reverse the damage if no actions are taken now. Population control will take thousands of years to address if it is to be done humanely through birth control policies. While most plants and animals may be able to migrate and adapt to different climates as they have done so throughout Earth's history, they are no match for human overpopulation, which has wiped out many many more habitats and caused the extinction of countless species long before anyone took a look at the effects of "climate change" on habitats.