Hedge funds’ search for alpha comes with a hefty price tag, as prime brokerage spending has zoomed to $10 billion a year, while investment in market data and technology needed for the search hit more than $1 billion.