* Residents report 185 dead; security says an exaggeration
* Fighting in northern fishing town near Chad border
* Boko Haram Islamists are country's worst security threat (Adds defence spokesman, details, U.N. Secretary General, writes through)
By Tim Cocks and Isaac Abrak
ABUJA, April 22 (Reuters) - Nigerian authorities said on Monday there had been heavy fighting between security forces and Islamist militants in a remote part of the northeast, but there was no confirmation of reports from a local official that 185 people had been killed.
Fighting erupted on Thursday in Baga, a fishing town in Borno state on the shores of Lake Chad, by the Chadian border -- an area officials say is a stronghold for Islamist fighters and a smuggling point for weapons from across the Sahara.
Defence spokesman Brigadier General Chris Olukolade said Nigerian forces had exchanged fire with militants, killing 25 of them, while only one soldier was killed.
Many hundreds have died in the rebellion by Boko Haram, a movement loosely modeled on the Afghan Taliban that is seen as the number one security threat to Africa's top energy producer.
A local community representative had put the death toll at 185, Umar Gusau, spokesman for Borno state which administers the area, said after visiting the town with a delegation on Sunday.
"For now, we don't have a very good basis for the figure," Gusau said. "These people say ... they have buried them. From my experience, most times residents exaggerate figures."
He said since they'd buried the victims quickly, in line with the Muslim custom, authorities had been unable to count the bodies independently, although an investigation was ongoing.
Sagir Musa, a spokesman for the mixed military and police Joint Task Force (JTF) in Borno state, he said by phone that death toll had been "terribly inflated" by residents.
The military, which rarely ever admits killing civilians, is often accused by locals of understating civilian casualties.
A spokesman for U.N. Secretary General Ban ki-Moon said he was "shocked and saddened at reports of high numbers of civilians killed", adding that he urged "all concerned to fully respect human rights and safeguard the lives of civilians".
The violence came as President Goodluck Jonathan awaits a report from a panel he set up to offer an amnesty to the insurgents if they give up their struggle for an Islamic state.
Boko Haram has so far shown no interest in talks and two mediators have already pulled out, including Islamic cleric Ahmed Datti, the only one whom Boko Haram have said they trust.
Jonathan, a Christian, has failed to quell the violence through military means and traditional leaders in the mostly Muslim north have put pressure on him to cut a deal.
If 185 people did die in the Baga fighting, it would be the greatest loss of life in the conflict since 186 people were killed in coordinated strikes by Boko Haram fighters in January 2012 in the north's main city of Kano.
Gusau said parts of Baga were badly damaged when he visited it on Sunday, with several houses burned. He said Nigerian soldiers sometimes over-react when attacked by Boko Haram gunmen, killing many in retaliation. (Reporting by Tim Cocks and Isaac Abrak; Editing by Michael Roddy)