Manage nearly $3 billion, watch your flagship small-cap portfolio fall 40 percent in one year, and an island fortress might sound good to you, too.