Difference operators to refund nearly $40 million to retail investors after exposing widespread misconduct in one of the market’s riskiest corners.