Sign Up
ENعربي
All articles

Project Management

Competency Framework Mapping with Large Language Models (LLMs) in project management,

  • Introduction

Competency frameworks are indispensable tools for organizations seeking to align their workforce with strategic objectives, as well as for individuals aiming to assess and enhance their professional capabilities. These frameworks act as a vital conduit between the academic and professional realms, facilitating the synchronization of university curricula with the ever-evolving demands of industry, and aiding industries in refining their recruitment strategies. They also empower employees in their skill development and career progression. However, the absence of a universal or standardized competency framework poses a significant challenge. Each framework is often tailored to specific professions or domains, adheres to distinct guidelines and standards, and undergoes continuous updates to reflect changes in industry practices. These ongoing modifications can lead to equivalent competencies being expressed in diverse ways, which impedes interoperability and complicates the aggregation of data from multiple frameworks. This paper addresses this critical issue by introducing an innovative approach that leverages large language models (LLMs) for mapping competency frameworks. The purpose of this literature review is to thoroughly summarize, synthesize, and critically evaluate existing research on competency framework mapping, with a particular focus on the transformative role of LLMs within the project management domain.

  • Background on Competency Frameworks

Competency frameworks are structured systems that define the essential skills, knowledge, behaviors, and attributes necessary for effective performance in specific roles or within an organization. These frameworks provide a structured approach to identifying and developing the competencies required for various positions, thereby aligning individual capabilities with organizational goals. The significance of competency frameworks is evident across a broad spectrum of domains, including but not limited to educational resources, university-industry partnerships, aircrew proficiency, and pharmaceutical professions. In the specialized field of project management, several influential organizations such as the International Project Management Association (IPMA), the Project Management Institute (PMI), and the Australian Institute of Project Management (AIPM) have developed their own distinct competency frameworks. The IPMA framework adopts a comprehensive perspective that incorporates technical competence, contextual intelligence, and behavioral elements. In contrast, the PMI framework, as detailed in the Project Management Body of Knowledge (PMBOK), emphasizes proficiency in specific knowledge areas such as scope, time, and risk management. The AIPM framework, on the other hand, focuses on addressing competencies across different stages of a project manager's career, highlighting the importance of adaptability and ongoing development. Despite the availability of these multiple frameworks, they often exhibit shared competencies and significant overlap, yet they are inherently tailored to the unique requirements of specific professions or domains. This complexity is further amplified by the diverse and specialized demands of various types of projects, such as those in construction, industrial product development, and IT, each potentially requiring tailored frameworks.

III. Challenges in Competency Framework Mapping

The central challenge in working with diverse competency frameworks lies in achieving interoperability through effective mapping. Traditional methods of competency framework mapping have largely relied on manual processes. These manual approaches are not only time-consuming and resource-intensive, but also prone to human error. Furthermore, the continuous updates and modifications to standards and framework editions introduce additional layers of complexity to the mapping process. The absence of robust automated solutions in this domain has significantly hindered the efficient integration of data from multiple frameworks, underscoring the critical need for more sophisticated and scalable approaches. The goal is to identify relationships between competencies across frameworks, including exact matches, equivalencies, hierarchical relationships, and part-whole relationships.

  • Existing Approaches to Competency Framework Mapping

Historically, competency framework mapping has primarily depended on the manual efforts of experts. Several studies have employed quantitative assessments to evaluate the alignment of competency frameworks, often relying on manual mapping processes. For instance, one study quantitatively assessed the alignment of eight competency frameworks with an international standard for public health training, highlighting the diversity inherent in competency frameworks through manual comparison. In recent years, research efforts have begun to explore automated techniques. Early attempts at automation included rule-based approaches designed to systematically extract features from skill descriptions within a given competency standard, and using those features to train machine learning classifiers to map the features to another standard. Other methods have employed concept mapping tools to create visual representations of knowledge and to measure the similarity between skills through graph matching. For instance, the C-map tool, a visual representation technique, has been used to map the concepts connecting interviewees’ experiences with the project management competencies required for the development of the framework. While these methods have provided valuable insights, they often depend on traditional natural language processing (NLP) models and still require manual revisions, limiting their scalability and overall efficiency. Some studies have used text embedding and cosine similarity to map skills across digital learning platforms, but these have relied on traditional NLP models, without exploring the potential of LLMs.

  • The Role of Large Language Models in Competency Mapping

This paper introduces a novel approach that addresses the limitations of previous methods by leveraging large language models (LLMs). This study marks the first application of LLMs specifically to the field of competency management, with a focus on mapping diverse competency frameworks. The use of pre-trained LLMs, such as BERT, DistilBERT, RoBERTa, and MPNet, offers considerable advantages in capturing semantic nuances and contextual meanings. These LLMs can generate high-quality vector embeddings that accurately represent the intent and meaning of competency descriptions. By using cosine similarity, the semantic similarity between competency vectors can be measured, facilitating the identification of equivalent or closely related competencies across different frameworks. This approach is particularly effective at automatically detecting similar or related competencies from multiple frameworks, enabling the effective management of multiple frameworks, as well as the analysis of historical data from various editions or different frameworks. The approach facilitates the identification of relationships between competencies across distinct frameworks, including exact matches, equivalencies, hierarchical relationships, and part-whole relationships.

  • Evaluation of the Proposed LLM-Based Approach

The proposed LLM-based approach was rigorously evaluated using three project management competency frameworks developed by the PMGS company. These frameworks were each based on different editions of the PMBOK standard (5th, 6th, and 7th editions), allowing for a comprehensive analysis of the approach’s effectiveness. The mapping process involved several key steps: first, the textual data was transformed into vector embeddings using the pre-trained LLMs; then, cosine similarity scores were calculated to rank the most similar competencies; finally, the relationships between competencies were categorized. The study utilized information retrieval evaluation metrics, specifically Recall@k and Mean Reciprocal Rank (MRR), to assess the accuracy of the mapping. The experimental results for mapping competencies between Framework 1 (based on the 5th edition of PMBOK) and Framework 2 (based on the 6th edition of PMBOK) demonstrated high performance across all tested LLMs. Notably, the DistilBERT model achieved the highest MRR score of 0.97, indicating its superior performance in retrieving relevant competencies. For the mapping between Framework 2 and Framework 3 (based on the 7th edition of PMBOK), the MPNet model showed the strongest Recall rates, while the BERT-based model performed the best in terms of MRR. The study also observed that mapping between Frameworks 1 and 2 was more successful than the mapping between Frameworks 2 and 3. This was attributed to the significant differences introduced in the 7th edition of the PMBOK, which shifted the focus from processes to principles and performance domains.

VII. Discussion and Synthesis

The central theme of this research is the use of LLMs to improve the interoperability of competency frameworks by automating the mapping process. The study underscores the capability of LLMs in accurately capturing semantic nuances and contextual meanings, which is essential for identifying equivalent competencies across disparate frameworks. The field shows a general consensus on the need for automated competency mapping solutions due to the limitations of manual mapping methods. The study illustrates a clear transition from traditional rule-based methods and machine learning classifiers to the more recent and effective application of LLMs. However, there are several notable gaps in the existing research. These include the limited scope of the study, which focused on only three project management frameworks all aligned with the PMBOK standard, potentially impacting the generalizability of the findings. Additionally, the lack of publicly available data also limits the broad application of these results to other domains. The models used, including BERT, RoBERTa, DistilBERT, and MPNet, all have a transformer architecture, and therefore performed with high consistency in the study. This consistency is attributed to the attention mechanisms they employ to capture contextual relationships in language data. The research shows that DistilBERT, a more compact version of BERT, is more resource efficient while maintaining over 95% of BERT’s performance. MPNet is shown to perform well on a variety of language understanding tasks.

VIII. Critical Insights and Conclusion

This body of work shows a significant alignment in demonstrating the successful application of LLMs for competency mapping, particularly within the context of project management. The study effectively illustrates how LLMs can improve interoperability among competency frameworks, which remains a critical challenge in the field. However, a significant divergence lies in the context of the study itself, as it utilizes frameworks all related to the PMBOK standard, potentially limiting the generalizability of the results. This research contributes to the understanding of competency framework mapping by introducing an automated, LLM-based approach that surpasses traditional methods in terms of both accuracy and efficiency. Key findings include the high performance of DistilBERT in mapping similar frameworks, as well as the effectiveness of MPNet and BERT when mapping more distinct frameworks. This research highlights the significant value of LLMs in facilitating competency mapping and provides a pathway for future research to explore the application of this approach across broader domains using public data. Further research should investigate techniques to improve the contextual understanding of LLMs through fine-tuning them for specific contexts. Future studies could also explore the application of this approach across a broader array of competency frameworks and leverage public data sources, such as taxonomies like Lightcast and ESCO.

(Jemal et al., 2025)

References

Jemal, I., Armand, N. S. W., & Chikhaoui, B. (2025). A new approach for competency frameworks mapping using large language models. Expert Systems with Applications, 263. https://doi.org/10.1016/j.eswa.2024.125648

Frequently Asked Questions About Competency Framework Mapping Using Large Language Models

  • What are competency frameworks and why are they important? Competency frameworks are structured tools that define the skills, knowledge, behaviors, and attributes necessary for effective performance in specific roles or within an organization. They serve as a bridge between industry and academia by ensuring that educational curricula align with industry demands. These frameworks are crucial for optimizing recruitment processes, identifying essential qualifications, and supporting employee development and career advancement. They are valuable across various domains, including education, university-industry cooperation, and specific professions like project management.
  • What challenges exist in using and comparing different competency frameworks? Several challenges hinder the effective use and comparison of different competency frameworks. The absence of a universal framework means that each one is often tailored to specific professions or domains, even when they share common competencies. Continuous updates and changes to standards also lead to different iterations of the same framework, complicating interoperability and the aggregation of data from multiple frameworks. Furthermore, previous approaches to mapping these frameworks have often relied on manual processing, which is time-consuming and not scalable.
  • How does the proposed approach using Large Language Models (LLMs) address these challenges?The proposed approach uses LLMs to automatically map competencies across different frameworks. It leverages pre-trained models like BERT, DistilBERT, RoBERTa, and MPNet to convert competency descriptions into vector embeddings. These embeddings capture the semantic meaning of the text, which allows for the calculation of similarity scores using cosine similarity. This method enables the identification of equivalent or closely related competencies across different frameworks, thus enhancing interoperability and automating the mapping process.
  • What are the key steps in the proposed competency mapping approach?The mapping process consists of three main steps: (1) Text Embedding: Pre-trained LLMs are used to convert text descriptions of competencies into vector embeddings. (2) Similarity Calculation: Cosine similarity is used to measure the semantic similarity between these embeddings. (3) Competency Mapping: Relationships between competencies, such as exact matches, equivalencies, and hierarchical relationships, are identified and categorized based on the similarity scores.
  • Which pre-trained LLMs were used, and why were they chosen? The study utilized four pre-trained transformer models: BERT, DistilBERT, RoBERTa, and MPNet. These models were chosen because they are known for their effectiveness in capturing semantic nuances and contextual meanings in language. They were trained on vast datasets, allowing them to leverage acquired knowledge for text embedding without resource-intensive computations. The models represent different approaches and optimizations within the transformer architecture, providing a comparative analysis of their performance in competency mapping.
  • How is the performance of the competency mapping approach evaluated? The performance of the proposed approach is evaluated using two information retrieval metrics: Recall@k and Mean Reciprocal Rank (MRR). Recall@k measures the model’s ability to retrieve relevant items within the top k results, while MRR emphasizes the importance of the position of the first relevant item in the retrieved list. These metrics provide insight into both the retrieval of relevant matches and the ranking accuracy of the similar competencies identified by the model.
  • What were the results of mapping competencies between frameworks based on different editions of the PMBOK Guide? The study mapped competencies across three frameworks based on the 5th, 6th, and 7th editions of the Project Management Body of Knowledge (PMBOK) Guide. Results showed that mapping between the 5th and 6th edition-based frameworks was highly effective, due to the substantial overlap in competencies and their similar foundations. However, mapping between the 6th and 7th edition frameworks was more challenging due to the significant changes in the 7th edition, although the models performed reasonably well, highlighting the effectiveness of the approach even with variations. The DistilBERT model generally outperformed the others in mapping between the 5th and 6th edition frameworks while MPNet and BERT performed best in the 6th to 7th edition mapping.
  • What are the limitations of the study and what future work is suggested? The study's main limitations include a focus solely on project management frameworks aligned with the PMBOK standard and the use of non-public data. The study also acknowledges that, while effective, LLMs may not capture all nuances and contextual factors. Future research should address these limitations by applying the approach to a broader range of frameworks, leveraging publicly available datasets, enhancing the contextual understanding of LLMs, and exploring fine-tuning for specific domains to improve mapping accuracy and relevance