It was a place where the PM’s family could be spotted dining and where politicians and lobbyists went when work ran late