The bigger the buyout, the harder it may be to convince large institutional investors that private equity firms are doing the right thing by participating in so-called “club deals.”