if the shoe fits, wear it

(idiomatic) If a description fits, it is probably the case; if an insult accurately describes you, you must bear it; if you take offence at an insult that was only implicitly directed at you, you admit the insult is true enough to be recognizable.