TA Associates did well in financial and technology niches that other private equity firms neglected. But now the field is crowded, returns are pinched, and TA is pressed to keep its edge.