Pontevedra is a much bigger town with a very large pedestrianized historic center, beautiful architecture, and a great food and restaurant scene, including outstanding ice cream shops! You can easily spend a day or two there.
Padron is smaller, but has a good sized historic center, including a beautiful square by the river Sar, a few very good restaurants (Pulperia Rial in the Center; O Pazo, up the hill toward Santiago, one Michelin star), and a wonderful small museum of modern Galician culture and letters, the Casa Museo Rosália de Castro. (A few blocks from the center. She revived Gallego as a literary language. The 20th century author Camilo José Cela also has museum in town, as well as an imposing statute in the square by the river.
Caldas has a pretty setting, Roman bridge, and a number of handsome historic buildings connected to its hot springs. It has a hot spring water lavadero (laundry washing table/pool) if you want to dangle your feet in very warm water. (Lavadero Termal, Rua Don Carlos García Bayón) To do the same thing in cool water, go to the the central park just across the bridge, the Carballeira de Caldas de Reis. Less to see and do than the other towns, fewer restaurants and more empty shopfronts like many small towns in Galicia.