It does not work in only one direction: X owns X’s culture. It also works the other way: X belongs to X’s culture.
We are, from even before birth since we can hear starting in the 24th week of pregnancy, processed, shaped and molded by the languages we can hear, by the moods of the mother who reacts to the world outside, and as soon as we are born we are thrown into a maelstrom of life in which the only emergency we have is to adapt and to survive and to cry for help and to learn in several years how to be independent.
How could we not be shaped by that outside world and that world changes so much from place X to place Y, from family Z to family N. We are nothing but differences, each one of us, and yet there is a vast and solid base that makes us part of groups, cultures and so many other things like religions and philosophies that make us be the members of communities.
If we follow your approach we are going to tell me that women do not have the right to claim they are women and men are not and have no right to pretend to be: transvestites are pirates for sure but at the same time we all of us individually and collectively define ourselves as men and not women, as women and not men, as other and neither men nor women, etc.
My point here is what I said at the beginning.
X owns X’s culture AND X belongs to X’s culture.
The two directions are true and it is that dialectic that explains why we can “borrow” and not “appropriate.” And borrowing requires two sides who are equally willing to accept that borrowing. If there are not two sides to accept that borrowing it no longer is borrowing but it becomes stealing and that is also wrong.
Intellectual property is property like any other possession, as we know since the Donaldson vs Beckett case ruled upon by the House of Lords in 1774.