His sudden exit follows a damming letter sent to the Treasury Select Committee by the former head of group risk at HBOS criticising the sales-driven culture at the group and reluctance to heed warnings that the risks being taken in the mortgage market was creating problems.
Crosby, chief executive of HBOS between 2001 and 2006, presided over a massive push into the UK mortgage market. He said there was "no substance" to the allegations but "I nonetheless feel that the right course for the FSA is for me to resign from the FSA board, which I do with immediate effect".
His dramatic resignation will be an acute embarrassment, not just to HBOS and the FSA but also to Prime Minister Gordon Brown who commissioned Crosby to undertake a study of the UK mortgage market.
And it begs many questions about the wisdom of drafting in ex bankers as advisors to the Treasury and the FSA.
Speaking in Prime Minister's Questions, Gordon Brown said it was "right" that Sir James stepped down from the FSA. Referring to the allegations made by Mr Moore, Mr Brown said they had been independently investigated and were found "not to be substantiated".
The latest explosive twist to the inquest on the collapse of HBOS and RBS began yesterday with a letter sent by Paul Moore, ex Head of Group Regulatory Risk at HBOS, who was fired in 2005 by Crosby, then chief executive of the bank.
The evidence that Moore put before the committee laid bare the corporate culture within HBOS when it was going full tilt towards the fateful growth in mortgage market share.
Moore had formal responsibility for the bank's policy and oversight of executive management's compliance with Financial Services Authority regulation. In his statement he claimed how he had raised concerns on numerous occasions over the group's mortgage lending policies but was thwarted at every turn.
He was dismissed by Crosby ("my decision, and my decision alone", he said Crosby told him at the time) and appealed on the grounds of unfair dismissal. The case was then settled out of court for a "substantial" sum.
"When I was head of Group Regulatory Risk at HBOS", he wrote, "I certainly knew that the bank was going too fast (and told them) , had a cultural indisposition to challenge (and told them) and was a serious risk to financial stability (what the FSA call 'maintaining Market Confidence') and consumer protection (and told them)".
"I told the board they ought to slow down but was prevented from having this properly minuted by the Chief Financial Officer. I told them that their sales culture was significantly out of balance with their systems and controls."
"What my personal experience of being on the inside as a risk and compliance manager has shown me is that, whatever the very specific, final and direct causes of the financial crisis, I strongly believe that the real underlying cause of all the problems was simply this - a total failure of all key aspects of governance.
"In my view and from my personal experience at HBOS, all the other specific failures stem from this one primary cause."
What is damning about Moore's account is the unflattering light it threw on the rampant sales and marketing culture of HBOS at the time, and how difficult it was for internal compliance officers to apply the brake on practices that they thought could endanger the group.
He was replaced by a new group risk director who, said Moore, had never carried out a role as a risk manager of any type before. The individual involved had primarily been a sales manager.
Sir James said in his statement that HBOS had "extensively investigated" Mr Moore's allegations, concluding that they "had no merit".
The Treasury told the BBC it was Sir James' personal decision to resign from the FSA.
Speaking in Prime Minister's Questions, Gordon Brown said it was "right" that Sir James stepped down from the FSA. Referring to the allegations made by Mr Moore, Mr Brown said they had been independently investigated and were found "not to be substantiated".
There is one particularly colourful passage in Moore's letter:
"To mix a few well known metaphors, the current financial crisis is a bit like the story of the Emperor's new clothes. Anyone whose eyes were not blinded by money, power and pride (hubris) who really looked carefully knew there was something wrong and that economic growth based almost solely on excessive consumer spending based on excessive consumer credit based on massively increasing property prices which were caused by the very same excessively easy credit could only ultimately lead to disaster. But sadly, no-one wanted or felt able to speak up, for fear of stepping out of line with the rest of the lemmings who were busy organising themselves to run over the edge of the cliff behind the pied piper CEOs and executive teams that were being paid so much to play that tune and take them in that direction. I am quite sure that many, many more people in internal control functions, non-executive positions, auditors, regulators who did realise that the Emperor was naked, but knew if they spoke up they would be labelled "trouble makers" and "spoil sports" and would put themselves at personal risk."
It is this letter - rather than the apologies to the Treasury Select Committee yesterday - that has clearly had the greater resonance.
