‘Getting better all the time’: Using professional human coach competencies to evaluate the quality of AI coaching agent performance

This paperexplores the evaluation of Artificial Intelligence (AI)AI coaching agent (coachbot) as a tool for workplace performance and wellbeingconversations. The paper starts by considering methods for the evaluation ofcoach performance, including client evaluation, qualitative and quantitativemethods and coach competencies. Using a coach competency as a framework, thisstudy uses a qualitative approach to assess a real-world AI coaching agent’s‘behaviours’ in the form of an assessment by trained ICF assessors of a genuinesession with a human client. The study involved a sample of 43 managers whovolunteered to be coached by an AI Coaching agent. ICF assessors used the ICFcoach competency framework and supporting behavioural anchored rating (BARS)framework developed by the ICF. The results indicated the AI coaching agent wasable to demonstrate many elements of both ICF ACC level and PCC level coachcompetencies. The assessment also revealed gaps in AI coach competence. Weconclude AI coaching agents are highly competent at some aspects of the coachprocess, while less competent at others. Further, AI development is needed toaddress these gaps.

‍