WITH REVENUES DECLINGING AND THE REGULATORY COST of doing business rising sharply, investment banks around the world are scaling back and seeking to find the right size and scope for their operations. Few institutions are doing so in the face of as much political scrutiny — and interference — as Royal Bank of Scotland Group.

RBS is the U.K.’s poster child for banking excess. The bank’s breakneck expansion in the past decade — culminating in its ill-fated £48 billion ($76 billion) purchase of ABN Amro in 2007, just before the U.S. subprime mortgage market imploded — left it hopelessly overextended when the financial crisis erupted in full fury after the collapse of Lehman Brothers Holdings the following year. With losses mounting and wholesale funding markets shutting down, RBS needed a massive public capital injection of £45 billion, the largest bank bailout in the world. Not for nothing did CEO Stephen Hester last year describe RBS as “an emblem of the financial crisis.”

Today, Hester is struggling to restore the bank to health while managing the public pressures that come with being 82 percent government-­owned. The former is proving easier than the latter.

Through asset sales and the disposal of noncore businesses, Hester and his team have slashed RBS’s bloated balance sheet by £712 billion, or 44 percent, from its 2008 peak and reduced its reliance on short-term wholesale funding by nearly two thirds, to £102 billion. “We have to defuse the biggest balance-sheet time bomb in history, and we’re making very good progress in doing that,” Hester said last month in announcing the group’s 2011 results.

Although the bank remains unprofitable — sovereign debt write-downs caused the net loss to widen by 78 percent last year, to £2 billion — RBS’s core retail and investment banking activities are in reasonable shape. Operating profits on those core businesses were £6.1 billion in 2011, down 18 percent from a year earlier. Global banking and markets, or GBM, as the investment banking division is known, has generated £10.7 billion of operating profits over the past three years — 44 percent of the group’s total.

Yet RBS and Hester remain convenient public whipping boys. The board’s decision in January to award Hester a bonus of £963,000 for 2011 provoked a public outcry that led the CEO to waive the payment. Ed Miliband, leader of the opposition Labour Party, railed against the proposed bonus and called a parliamentary debate on the issue, prompting Iain Duncan Smith, the government’s minister for Work and Pensions, to urge Hester to give up the award. The incident undermined the government’s contention that it can effectively manage its RBS stake at arm’s length through U.K. Financial Investments (UKFI), a Treasury agency created expressly for the task. The political uproar will also make it harder for RBS to recruit and retain the talented professionals needed to sustain the bank’s turnaround, especially in investment banking. Hester acknowledged the problem when he presented the bank’s 2011 results last month, saying that the “noise around RBS is damaging to the prospect of achieving the goals everyone needs of it.”

The recent controversy will almost surely delay any attempt by the government to sell off some or all of its stake, keeping RBS firmly in the political crosshairs. The bank’s shares were trading at 28 pence late last month, well below the government’s purchase price of 51 pence. Just over a year ago, when the share price was approaching 50 pence, UKFI started preparations for a possible privatization within 12 months, but regulatory and market pressures have put those plans on hold, and Treasury sources say a sale is unlikely before the next parliamentary election, which is due to be held by 2015. Hester told the BBC last month that the government would retain its stake for “quite a few years.”

Given this climate, the prospects for GBM, led by the division’s chief executive, John Hourican, have dimmed. It is not simply that fees are evaporating across the industry and that the new Basel III regulatory framework is forcing all banks to raise their capital and liquidity buffers at a time when investors have little appetite for bank equity or debt. The U.K. government is imposing an additional layer of restraints on its banks because of the costly bailouts of RBS and HBOS, which was later merged with Lloyds Banking Group. The changes, recommended by the Independent Commission on Banking, will require banks to ring fence their retail and commercial banking operations from their investment banking activities, a move that will substantially raise the costs of investment banking.

RBS’s sorry legacy means that Hourican and his colleagues have to look over their shoulders at British politicians. That fact was driven home in December when Chancellor of the Exchequer George Osborne called on RBS to curb its riskier activities and concentrate on the U.K.

As a result, RBS is now pressing ahead with a new round of cuts in GBM and effectively abandoning its experiment with full-service investment banking.

Under a plan announced in January, RBS is cutting 3,500 jobs at the division, reducing its head count to 13,500. The group sold Hoare Govett, its corporate brokerage subsidiary, to U.S. securities firm Jefferies & Co. for an undisclosed amount last month. RBS is preparing to sell its cash equities and M&A advisory businesses. The significance of the moves is as much symbolic as financial. The three businesses generated just £220 million of revenue, a mere 3.3 percent of GBM’s total, in the first nine months of 2011, and they were second-tier businesses at best. RBS ranked tenth by revenue in European equity trading in 2011 and was outside of the top 15 in the other categories, according to Coalition Development, a London-based financial research firm.

The bank is also reducing the capital allocated to GBM’s core fixed-income, currency and derivatives business. RBS plans to cut the unit’s balance sheet, already slashed by nearly 60 percent since 2008, by a further 25 percent, to £300 billion, over the next three years. Executives say that the bank is merely responding to market and regulatory realities like most of its rivals and that it won’t be eliminating any products or territories in its main debt and currency businesses.

“Most banks have been exiting noncore activities and prioritizing client facilitation,” says Suneel Kamlani, the Stamford, Connecticut–­based president of GBM. “Our core proposition is that we are a top player in the fixed-income markets.”

Hester insists that he remains fully committed to the debt-focused heart of the investment bank, and for good reason: RBS is a top five player in global fixed-income trading and foreign exchange trading. “These are the cornerstone products and services for the global economy in any economic environment,” Hester said in an internal memo in January.

Yet some observers believe the scale of the cutbacks at RBS and the political scrutiny the bank faces threaten to cause a long-term decline at the investment bank and make it unlikely that it can achieve management’s target annual return on equity of 12 percent. Christopher Wheeler, banking analyst at Mediobanca in London, says RBS, like many other banks, is cutting back much more sharply than the big fixed-income powerhouses: Barclays Capital, Deutsche Bank and JPMorgan Chase & Co. “These banks are the global flow monsters, with the capital, with the global critical mass to make money at a difficult time in the markets,” he says. “RBS is certainly strong in certain areas, but it’s not in the same league as these three. The senior managements of these banks are also stronger.”

Other analysts believe further cutbacks are likely as RBS implements the sharper separation between retail and investment banking mandated by U.K. regulators. “I’d be very surprised if head count is as high as 13,500 in three years’ time,” says Vivek Raja, a banking analyst at Oriel Securities in London.