Like any other place about which not much written is known, Pandan’s early beginnings is a matter of conjecture. However, the frequent mention of the raids of sea pirates in the tradition of the people suggests that there must have been a thriving settlement already at least during the second half of the seventeenth century. And since there is no trace of an ethnic tribal minority, the original inhabitants of the place must have been migrants themselves, coming either from mainland Bicol or even from Deep South as part of the great Malay migrations.

The first settlement is believed to have been a fishing village along the banks of the river Takason. Conveniently located near both the sea and the land they tilled, the people lived happily in peace and prosperity. But their prosperity also proved a lure for the raids of sea pirates. It is told that some people fleeing inland were actually trapped and massacred by the raiders in a place now called Binonoan, some two kilometers south of the present town site.

By necessity, the people realized that they had to find a better site and somehow stick together in order to defend themselves against the raids. So they moved to the delta left by the river where the thick growth of prickly plants extending to the shore could hide their houses from sight of the sea, and where a small hill nearby could hide their houses from sight of the sea, and where a small hill nearby could provide a lookout. Those who had farms on the mountainside overlooking the bay also took turns in watching for the coming if passing the hostile boats in order to warn the people of the pirates.