Duffy hasn’t been a recluse this summer nor unwelcome in Cavendish, where politics is a second religion and his trial looms large