More than half of men have ignored potentially worrying physical or mental health issues, according to a report.