By Angus McDowall and William Maclean
RIYADH/DUBAI, Aug 28 (Reuters) - With militant Islamists gaining the upper hand in Syria's rebel movement and grabbing big tracts of Western Iraq, Saudi Arabia's ruling family faces an increasingly uncomfortable dilemma.
The Al Saud have long seen the conflicts in Iraq and Syria as a pivotal battle for the future of the Middle East, pitting Sunni Muslims against a radical, revolutionary, Shi'ite Iran.
But in both Syria and Iraq the kingdom's preferred Sunni allies have lost out to more militant groups, and Riyadh faces its nightmare scenario of watching two key Arab states become proxies for its rival Tehran or, worse, perpetual failed states.
What the Al Saud dynasty most wants in both countries is a stable government with strong Sunni representation that could act as a bulwark both against what they see as Iranian expansionism and a Sunni militant ideology that threatens their own rule.
In Syria, where the Saudis are a leading backer of rebel groups including the secular Free Syrian Army and the Islamic Front, which includes less militant Sunni fighters, Riyadh still has some options to influence the outcome of the war.
But in Iraq, its most populous neighbour, with which it shares an 850 kilometre (530 mile) frontier, Saudi Arabia has few tested friends or established links with Sunni groups, and knows that the majority Shi'ites will continue to dominate power.
"In terms of strategic games, the Saudis are waiting to see what will happen," said Mustafa Alani, an Iraqi security analyst with close ties to Riyadh's Interior Ministry.
"They don't have any group they can rely on among the Sunni Arabs. They've been absent since 2003, and it cost them a lot."
For the Saudis, the militant advance this summer might have given a welcome bloody nose to Tehran and Iraqi Prime Minister Nouri al-Maliki, whom it has accused of being "Iranian 100 percent", but it did so at the partial expense of their own security.
While Islamic State's territory does not yet extend to the Saudi border, and appears unlikely to pose a military threat, many of the kingdom's citizens have joined the group, raising fears they will turn against their own government.
For the Al Saud, most Islamist factions represent a dangerous ideological challenge to their system of dynastic rule, leading to their campaign against the Muslim Brotherhood in Egypt and cooperation with Washington to tackle al Qaeda.
The ruling family has grown so worried, spurred by memories of attacks by Saudi veterans of Iraqi fighting last decade, that King Abdullah in February decreed tough new laws and has mobilised the powerful clergy to preach against radicalism.
"We have done and will do everything we can to stop the spread of this corrosive poison in our country and region and encourage all other governments to do the same," Riyadh's ambassador to London, Prince Mohammed bin Nawaf, wrote in a British newspaper this month.
GIFT TO IRAN
Saudi authorities say they are constantly in touch with Iraqis.
And the kingdom may have some ability to use traditional connections to influence Sunni tribes with extensive membership on both sides of the border.
"There is a long tradition of tribal elders from Iraq and other Arab countries visiting Saudi princes and other important personalities and petitioning for financial support to help them advance their broad social and political interests," said Neil Partrick, an associate fellow at the Royal United Services Institute.
Nevertheless, Riyadh has had no ambassador based in Iraq since 1990, when Saddam Hussein invaded Kuwait and threatened Saudi soil, and relations have soured even further since 2003, when the U.S.-led invasion gave the Shi'ite majority more power.
Saudi Arabia is the biggest humanitarian donor to Iraq, following a $500 million gift via the United Nations, said one of several diplomats in the Gulf interviewed for this article.
But its attempts to back Sunni political leaders north of the border have been sporadic and unsuccessful.
"People think the Saudis have more influence than they do. There are some contacts, but not much," said a diplomat.
Diplomats said both Riyadh and Qatar were in touch with a number of "moderate Sunni leaders" in Iraq, whom they supported to defuse growing support for Islamic State radicals.
However, such ties are not strong, said Alani.
While Riyadh did provide some funding to Iraq's Sunni tribes after Baghdad stopped financing the Awakening movement there, a diplomat said, the connections, mainly through the Shammar tribe of King Abdullah's wife, were limited.
Even Saudi religious leaders have little influence over fellow Sunnis in Iraq, most of whom follow different schools of Muslim thought to the Wahhabi school dominant in the kingdom.
WAIT AND SEE
Immediately after Haidar al-Abadi was tapped to be prime minister, Saudi Foreign Minister Prince Saud al-Faisal described it as the "only good news I heard lately", a sign of how far Riyadh distrusted Maliki.
Abadi is from the same political bloc as Maliki, however, and was described by a diplomat as "the lowest common denominator in terms of what was acceptable to everybody".
"Abadi is from the same extremist Shi'ite party as Maliki, so we will have to wait to see what his actions are like, not just his words," said Abdullah al-Askar, head of the foreign affairs committee of the appointed Shoura Council, which advises the government on policy.
An early attempt to reach out had indifferent results, said one senior Gulf source who declined to be named, though his account could not be confirmed.
He said national guard head Prince Miteb bin Abdullah met Iraqi officials after Abadi was nominated to offer advice on tackling the Sunni insurgency, but that they were "less welcoming than expected" and he was rebuffed.
"The prince communicated his message with officials working with Abadi and his message was firm and clear that issue of insurgencies must be dealt with firmly," said the source.
While the departure of Maliki removes a poisonous personal enmity from Saudi-Iraqi ties, Riyadh also accepts the reality of political and sectarian constraints in Baghdad, said the source.
"There's an understanding that for the time being, the prime minister in Iraq needs to be a Shi'ite," he said.
Some Iraqis hope the changes in Baghdad could lead to a wider accommodation between Tehran and Riyadh, helping to cool tensions in their country and across the Middle East.
"I think there will be positive developments between Iran and Saudi Arabia because of the advance of the Islamic State ... They will both play a role now in Iraq because of Islamic State," Iraqi Foreign Minister Hoshiyar Zebari told Reuters.
But after Saudi and Iranian officials met for their first bilateral talks since moderate President Hassan Rouhani was elected last year, official media in both countries kept reporting of the exchange to a minimum. It was a sign, say analysts, of how far their mutual suspicions persist, and how difficult it will be to work together to tackle Islamic State.
(Additional reporting by Michael Georgy in Baghdad and Amena Bakr in Doha; Editing by Will Waterman)