Swinton’s brilliant, harrowing performance deserved an Oscar nod, but she is upstaged by the film’s lurid style