Mutual fund companies may dominate the 401(k) business, but brokerages are trouncing them in the battle for hundreds of billions in rollover IRAs, especially among the better-off.