Chamber d'hotes tend to cater to non-pilgrim travelers. So while dinner may be at a set time, breakfast may be more flexible. It's only coffee and bread with jam, after all. So breakfast is usually set out at, say, 7:30 and it will be left out for maybe an hour or an hour-and-a-half, or until all the guests have made an appearance.
Once you ascertain the time for dinner, you will want to be a few minutes early. Hosts will attempt to seat the English-speakers together (that is to say, all the non-French/Quebecois/French Swiss).
Getting an early breakfast to support an early start can be a real challenge in France. If you are the only pilgrims that night, let your host know you would like to get underway early. If you want to start by 6:30, you will have to arrange your own breakfast. Breakfast at 7 or 7:30 is generally the earliest available; so a 7:30 or 8:00 start time is generally the earliest you'll be able to manage regularly.
On the other hand, if staying at a gite, catering to pilgrims, both breakfast and dinner will tend to be a bit earlier. Bon Chemin!