When used as a suffix in the context of Islamophobia, transphobia, or fatphobia, it is implied that the individual or group has a _fear_ of individuals and communities who identify in those ways. Rather than a direct translation, however, the implication and subtext of these terms is one of prejudice and discrimination. Using the term phobia falsely masks hate as fear.
Additionally, people with anxiety disorders and mental illness can also experience phobias, so conflating prejudice and discrimination (attitudes and behaviours that can be changed) with medical conditions that cannot be changed additionally harms people who experience phobias from their anxiety disorders.
As such, using the term phobia removes the responsibility from those who exhibit prejudice and discrimination as it implies it is outside of their control.
Hateful actions of prejudice and discrimination are unfairly conflated with mental illness. It can create a false parallel where one could imply that actual phobias are something that can be controlled, which harms people who experience actual phobias.