I would not wear a bra to the doctor's office unless you are out to him/her as trans and are actively in transition, or you have bad gynecomastia and need a sports bra for support or to hide the obvious. The doctor will appreciate it. My wife is a doctor, and I know it would bother her but she would be too professional to bring it up other than perhaps adding "is there anything else you'd like to talk about today" at the end of the exam, in case your bra was a signal you wanted help with gender dysphoria. If you don't take the bait then she'd let it slide.
If there was the *slightest* indication that wearing a bra to the doctor's office gives you the jollies, you'd be fired as a patient faster than you can do your bra back up after the exam. Where I live getting a family doctor is very difficult so why take that risk? There's no place, IMHO, for even the remotest thing that can be taken as a sign of impropriety in the doctor's office.
Remember doctors are human too. While they are trained to keep their feelings to themselves, they still have feelings and can be upset! They have a difficult job as it is, why make it harder?
