In letters to investors, firm founder Stephen Mandel Jr. and colleagues say the equity market looks “extended” and that private companies are reaching frothy valuations.