Fighting erupted when Marines on patrol encountered about 200 members of the Abu Sayyaf Islamist group near the borders of Patikul and Panamao towns, Sulu Army chief Col. Alexander Aleo said.
Three soldiers were killed and at least six others were wounded, he said.
The Marines were on a mission to find and rescue three Indonesian sailors and four Filipina Christian preachers being held by Abu Sayyaf rebels in Sulu.
None of the hostages were sighted during the clash, the military said.
The Abu Sayyaf is a small group of self-styled Islamic freedom fighters that has been tagged a terrorist organization by Manila and Washington.
US-Philippine counter-terrorism operations last year led to the rescue of American hostage Gracia Burnham, but her husband, Martin, and Filipina nurse Edibora Yap were killed in the same clash. A third American hostage, Guillermo Sobero, was beheaded in 2001.
The military meanwhile freed several villages occupied by the main Muslim separatist group Moro Islamic Liberation Front (MILF) in Sultan Kudarat last week.
Five days of fighting that began last Friday had left 28 MILF rebels dead and 17 wounded, with one rebel captured, Defense Secretary Angelo Reyes said, citing reports by soldiers from the field.
Reyes said fighting began when members of the Pentagon gang, a kidnap-for-ransom group made up of former MILF rebels, sought shelter in MILF territory.
The MILF and the Pentagon joined forces and took over several villages near the town of Colombio, forcing hundreds of civilians to flee.
The military however later overran the rebel position, Reyes said.
The 12,000-strong MILF has been fighting for the establishment of an independent Islamic state in the southern island of Mindanao since 1978. It signed a ceasefire with Manila in 2001 ahead of formal peace talks, but sporadic clashes persist. – With Roel Pareño, Jaime Laude