What is the counterargument to the statement “theoretical physics is harder than pure mathematics because you need to know the math and the physics to be a physicist but only math to be a mathematician”?

My answer to a question in Quora: What is the counterargument to the statement “theoretical physics is harder than pure mathematics because you need to know the math and the physics to be a physicist but only math to be a mathematician”?

My world outlook is very skewed, I know mostly mathematicians.

While writing this answer, I went in my head through the list of mathematicians who I met personally, and who were known for contribution to development of other fields, first of all, physics, but also, say, genetics and molecular biology, or who were able to use physics as a source of ideas and problems for their mathematical research. They all are, first and foremost, excellent hardcore mathematicians.

It is my conjecture that work at a serious level in both serious pure mathematics and theoretical physics require capacity for a specific kind of abstract thinking: ability to keep in mind a mental image of the world hierarchically built from increasing levels of abstraction, and ability to live in this world and freely move from one level to another. This could be the Platonic world of mathematics or the world of relativistic quantum physics — but I dare to suggest that the two strands of thinking, mathematical and physical, start to converge at higher level of development.

I do not claim that these my observations have any depth; they are simply an output of five minutes of silent contemplation.

Speaking about mathematicians and physicists, what really astounds me is the description of the work of a theoretical physicist in Vasily Grossman’s novel Life and Fate(the name of the character is Victor Schtrum; it is not clear who was his real life prototype). How could Grossman know? Strongly recommend.