To me, one of the main issues is opportunity.
Trans actors struggle to get any roles, it's insulting to have the trans characters played by cis folk (especially since a lot of people see trans people as playing dress up, and this kind of casting can reinforce that).
Black actors are very rarely cast in roles that weren't specifically written as black. Same with disabled, native, etc.
And openly gay actors still struggle to get cast in ostensibly straight roles, too.