Investors face a “game theory problem” in determining the optimal size of their allocations to private markets and other harder-to-sell assets, according to new research.