The New York Stock Exchange's new payment schedule for specialist firms could cut their net profits from market making by more than half after the first six months, traders said.