Exercise 1: Credit Model Development Prompt Engineering
Objective:
Learn to craft effective prompts for developing robust credit risk models that meet business and regulatory requirements.
Background:
Credit Modellers are responsible for developing models that assess borrower creditworthiness and predict default risk. A key challenge is designing models that balance predictive power, interpretability, and regulatory compliance.
Exercise:
1. Scenario:
You need to develop a new probability of default (PD) model for small business loans that balances predictive accuracy with interpretability and regulatory compliance.
2. Basic Prompt Example:
How do I build a small business PD model?3. Prompt Improvement Activity:
- Identify the limitations of the basic prompt
- Add specific details about the small business portfolio
- Include context about regulatory requirements
- Request a structured model development approach
- Ask for feature selection and model evaluation criteria
4. Advanced Prompt Template:
I am a Credit Modeller at a [size] financial institution developing a new probability of default (PD) model for our small business loan portfolio with these characteristics:
Portfolio details:
- Loan size range: $[X] to $[Y]
- Business types: [industry segments, e.g., retail, services, manufacturing]
- Current portfolio size: approximately [Z] loans, $[A] total exposure
- Default rate: [B]% over the past 3 years
- Available data: [financial statements, payment history, credit bureau, etc.]
- Data history: [years of historical data available]
- Current model: [brief description of existing approach if any]
Development constraints:
- Regulatory framework: [SR 11-7, IFRS 9, CECL, etc.]
- Interpretability requirements: [need to explain decisions to customers/regulators]
- Implementation timeline: [target completion date]
- Available resources: [team size, technical environment]
- Validation approach: [independent validation requirements]
Please help me develop a comprehensive credit model development plan by:
1. Recommending a structured development approach that includes:
- Data preparation and exploration methodology
- Feature engineering techniques
- Model selection framework
- Training and testing approach
- Calibration methodology
- Documentation requirements
- Implementation considerations
2. For data preparation and feature engineering:
- Key financial ratios to consider
- Non-financial indicators to incorporate
- Macroeconomic factors to include
- Data transformation techniques
- Missing data handling approaches
- Outlier treatment methods
- Feature selection criteria
3. For model methodology selection:
- Comparison of appropriate techniques (logistic regression, tree-based methods, etc.)
- Pros and cons of each approach for this specific use case
- Regulatory considerations for each methodology
- Interpretability trade-offs
- Implementation complexity
- Maintenance requirements
- Recommended approach with rationale
4. For model development and validation:
- Training/testing/validation split approach
- Cross-validation methodology
- Performance metrics to optimize
- Calibration techniques
- Stability testing approach
- Sensitivity analysis methodology
- Benchmarking recommendations
5. For model implementation and governance:
- Documentation requirements
- Model monitoring framework
- Override process design
- Annual review considerations
- Version control approach
- Model risk management integration
Format your response as a comprehensive credit model development plan that balances predictive power, interpretability, and regulatory compliance for our small business loan portfolio.5. Evaluation Criteria:
- Does the prompt clearly describe the small business portfolio and available data?
- Does it provide context about regulatory and business constraints?
- Does it request a structured development approach with specific methodologies?
- Does it ask for feature selection and model evaluation criteria?
- Does it consider implementation and governance requirements?
6. Practice Activity:
Create your own advanced prompt for credit model development related to:
- A retail mortgage loss given default (LGD) model
- A commercial real estate risk rating model
- A credit card application scoring model
Exercise 2: Model Calibration and Validation Prompt Engineering
Objective:
Develop skills to craft prompts that help calibrate credit models and prepare for validation.
Background:
Credit Modellers must ensure their models are properly calibrated and can withstand rigorous validation. A key challenge is developing effective calibration approaches and preparing comprehensive documentation for validation.
Exercise:
1. Scenario:
You have developed a corporate probability of default (PD) model and need to calibrate it to through-the-cycle (TTC) estimates and prepare for independent validation.
2. Basic Prompt Example:
How do I calibrate a PD model?3. Prompt Improvement Activity:
- Identify the limitations of the basic prompt
- Add specific details about the corporate PD model
- Include context about calibration objectives
- Request structured calibration methodologies
- Ask for validation preparation guidance
4. Advanced Prompt Template:
I am a Credit Modeller at a [size] financial institution who has developed a corporate probability of default (PD) model that now needs calibration and validation preparation.
Model details:
- Methodology: [logistic regression, machine learning, etc.]
- Risk drivers: [financial ratios, market indicators, behavioral factors, etc.]
- Segmentation: [industry sectors, size categories, etc.]
- Development sample: [time period, number of observations, default rate]
- Current performance: [discrimination metrics like AUC, Gini, etc.]
- Regulatory framework: [Basel, IFRS 9, CECL, etc.]
Calibration objectives:
- Through-the-cycle (TTC) PD estimates
- Alignment with historical default experience
- Regulatory conservatism requirements
- Economic capital calculation needs
- Stress testing application requirements
- Consistency with external ratings (if applicable)
Please help me develop a comprehensive calibration approach and validation preparation plan by:
1. Recommending calibration methodologies that address:
- Point-in-time (PIT) to through-the-cycle (TTC) conversion
- Central tendency estimation
- Low default portfolio challenges
- Long-run average default rate calculation
- Margin of conservatism determination
- Rating philosophy alignment
- Calibration sample selection
2. For each recommended methodology:
- Detailed step-by-step process
- Data requirements and preparation
- Statistical techniques to employ
- Common pitfalls and how to avoid them
- Regulatory considerations
- Documentation requirements
3. Suggesting approaches for:
- Calibration testing and verification
- Stability assessment over time
- Sensitivity analysis of calibration parameters
- Benchmarking against external sources
- Back-testing framework
- Monitoring triggers and thresholds
4. Providing validation preparation guidance for:
- Required documentation structure and content
- Developmental evidence organization
- Testing results presentation
- Limitations and assumptions documentation
- Implementation plan documentation
- Monitoring plan documentation
- Common validation challenges and how to address them
Format your response as a comprehensive calibration and validation preparation plan that ensures our corporate PD model meets regulatory requirements and business objectives.5. Evaluation Criteria:
- Does the prompt clearly describe the corporate PD model and its characteristics?
- Does it provide context about calibration objectives?
- Does it request specific calibration methodologies with detailed steps?
- Does it ask for calibration testing and verification approaches?
- Does it request validation preparation guidance?
6. Practice Activity:
Create your own advanced prompt for model calibration and validation preparation related to:
- A behavioral scoring model for existing customers
- An expected loss model for CECL/IFRS 9 compliance
- A prepayment model for mortgage portfolio management
Exercise 3: Model Documentation Prompt Engineering
Objective:
Learn to craft prompts that help develop comprehensive model documentation that meets regulatory requirements.
Background:
Credit Modellers must document their models thoroughly to meet regulatory requirements and support model governance. A key challenge is creating clear, comprehensive documentation that explains complex methodologies to different stakeholders.
Exercise:
1. Scenario:
You need to create comprehensive documentation for a newly developed IFRS 9 / CECL expected credit loss model for regulatory review and model governance.
2. Basic Prompt Example:
What should I include in model documentation?3. Prompt Improvement Activity:
- Identify the limitations of the basic prompt
- Add specific details about the IFRS 9 / CECL model
- Include context about regulatory requirements
- Request structured documentation framework
- Ask for approaches to document complex methodologies
4. Advanced Prompt Template:
I am a Credit Modeller at a [size] financial institution creating comprehensive documentation for a newly developed IFRS 9 / CECL expected credit loss model.
Model details:
- Purpose: Expected credit loss calculation for [loan portfolio type]
- Components: PD, LGD, EAD models and economic scenario generation
- Methodology: [statistical techniques, machine learning, etc.]
- Data sources: [internal data, external data, economic forecasts]
- Implementation status: [development complete, pre-implementation]
- Model risk rating: High (due to financial statement impact)
Documentation requirements:
- Regulatory framework: [IFRS 9, CECL, SR 11-7, etc.]
- Internal model governance standards
- Intended audience: [model validators, auditors, regulators, business users]
- Technical complexity: [highly technical, mixed technical/business]
- Documentation deadline: [target completion date]
Please help me develop a comprehensive model documentation framework by:
1. Outlining a complete documentation structure that includes:
- Executive summary for non-technical stakeholders
- Detailed technical sections for validators and auditors
- Required appendices and supporting materials
- Cross-referencing approach
- Version control methodology
- Approval and governance documentation
2. For the model development section, recommend:
- Business context and model purpose documentation
- Data sources and preparation methodology
- Feature selection and engineering approach
- Model selection process and alternatives considered
- Training and testing methodology
- Performance metrics and results
- Limitations and assumptions
3. For the model methodology section, suggest approaches for:
- Explaining complex statistical concepts to non-technical audiences
- Documenting model equations and algorithms
- Presenting model coefficients and parameters
- Explaining variable transformations and interactions
- Documenting segmentation approaches
- Describing calibration methodology
- Explaining economic scenario integration
4. For implementation and governance, provide guidance on documenting:
- Implementation plan and controls
- User acceptance testing approach
- Model monitoring framework
- Override process and governance
- Periodic review requirements
- Model interdependencies
- Change management procedures
5. For regulatory compliance, recommend approaches for:
- Mapping documentation to specific regulatory requirements
- Addressing common regulatory questions
- Documenting model risk assessment
- Explaining methodology choices and justifications
- Documenting expert judgment and management adjustments
- Addressing model limitations and compensating controls
Format your response as a comprehensive model documentation framework that meets regulatory requirements while effectively communicating complex methodologies to different stakeholders.5. Evaluation Criteria:
- Does the prompt clearly describe the IFRS 9 / CECL model and its characteristics?
- Does it provide context about documentation requirements and audience?
- Does it request a structured documentation framework with specific sections?
- Does it ask for approaches to document complex methodologies?
- Does it consider regulatory compliance documentation needs?
6. Practice Activity:
Create your own advanced prompt for model documentation related to:
- A stress testing model for capital planning
- A credit line management model for portfolio optimization
- A machine learning model for early warning indicators