A key objective of health promotion capacity building and professional development programmes is to enhance competencies in the workforce or training participants. The goal of this study was to investigate and validate the psychometric properties of a recently developed self-assessment tool, based on ‘The CompHP core competencies framework for health promotion’, to establish whether it is suitable for evaluating training. The tool was employed in 2018 to evaluate two different programmes, with 584 participants submitting their assessment and evaluation of 77 different health promotion training courses in a paper and pencil questionnaire. Longitudinal data were available for 148 participants in the programme for workplace health promotion because they completed an online questionnaire beforehand. The reliability and validity of the assessment tool were tested in several confirmatory factor analyses and structural equation models. Structurally, there was a significant increase in health promotion competencies when comparing measurements before and after training. Furthermore, participants were shown to have different levels of competencies due to their prior knowledge, the format of the course or the professional development programme attended but not based on gender, age or sector. The results justify the creation of an overall scale to assess core health promotion competencies that are suitable for evaluating training courses. However, various methodological issues need to be considered in future applications including decisions about the design of the study (e.g. pre–post test, control groups), type of measurements (e.g. general and specific instrument, background information) and depth of analysis (e.g. confirmatory model and invariance testing).