The military said one Moro Islamic Liberation Front (MILF) member was killed but the MILF denied this.
MILF gunmen harassed an Army camp in Shariff Aguak town with gunfire over the weekend and fired rocket-propelled grenades into Army positions on Monday, prompting the soldiers to fire back, said Col. Julieto Ando, regional military spokesman.
The Army said it reported the firefight to a joint ceasefire monitoring panel staffed by both sides.
MILF spokesman Eid Kabalu denied the rebels started the shooting and said it was the military and pro-government militiamen who initiated the hostilities.
He would not confirm any casualties on either side.
The violence comes despite a ceasefire signed in 2003 between the government and the 12,000-strong MILF, paving the way for peace talks.
Malaysian Brig. Gen. Ismael Ahmad Khan, head of an international ceasefire monitoring team, said he met with a local military commander, Lt. Col. Roseller Murillo, on Sunday to discuss the renewed fighting.
But this resulted in a "heated argument" with Murillo raising his voice and telling the Malaysian to leave his camp, said Khan.
Ando said Murillo was being investigated over the alleged argument.
The MILF has been waging a rebellion for an independent Islamic state in the South since 1978.
Although it has opened peace talks with Manila, the talks are now deadlocked over disagreements over the MILF’s demand for economic control over areas in the South that they claim as the Muslims’ traditional domain.
The MILF has warned of more bloodshed and a return to "all-out war" if the government failed to come up with an offer acceptable to them.