Put off by traditional philanthropy, Wall Streeters and entrepreneurs are setting up businesslike charities that pursue social returns as aggressively as their firms do profits.