Prerequisites
Create Azure OpenAI
Sign in to Azure Portal
- Go to https://portal.azure.com
- Sign in with your Azure account
Create Resource
- Click "Create a resource" (top left)
- Search for "Azure OpenAI"
- Click "Azure OpenAI" → "Create"
Create a Deployment Model
Access Azure AI Foundry Portal
- Go to https://ai.azure.com
- OR from Azure Portal → Your OpenAI resource → "Explore Azure AI Foundry portal"
Create/Select Project
- Create new project: AI-PR-Review
- Select your Azure OpenAI resource
Navigate to Deployments
- Left sidebar → "Deployments" (under Shared resources)
- Click "+ Deploy model" or "+ Create deployment"
Select Model
- Choose "Deploy base model"
- Pick your model:
- gpt-4o - Latest optimized (recommended)
- gpt-4-turbo - Fast, cost-effective
- gpt-4 - Best quality
- gpt-35-turbo - Lowest cost
Generate Personal Access Token
- Go to Azure DevOps → Click your profile (top right) → Personal access tokens
- Create new token:
- Name: AI-PR-Review-Token
- Organization: Select your organization
- Expiration: Set appropriate expiration (e.g., 1 year)
- Scopes: Custom defined → Code: Read & Write
- Click Create
Step 1: Add Pipeline Files
Add azure-pipelines-pr-review.yml File
Create a file named azure-pipelines-pr-review.yml in your repository root. This YAML file will define the pipeline that runs on every pull request to the develop or main branch.
The pipeline will:
- Trigger automatically on pull requests to develop and main branches
- Use Ubuntu latest as the build agent
- Install Python dependencies (openai, requests)
- Execute the AI PR review script
- Pass environment variables for Azure OpenAI and Azure DevOps authentication
Add ai-pr-review.py Script
Create a scripts folder in your repository root, then add the ai-pr-review.py file inside it.
This Python script will:
- Fetch changes from the pull request using Azure DevOps REST API
- Analyze each modified file using Azure OpenAI
- Identify security vulnerabilities, performance issues, and code quality problems
- Post detailed comments back to the pull request
Add Configuration Variables
Configure the required variables in Azure DevOps:
Add these variables to your pipeline or create a variable group:
- AZURE_OPENAI_ENDPOINT - Your Azure OpenAI endpoint URL
- AZURE_OPENAI_API_KEY - Your Azure OpenAI API key (mark as secret)
- AZURE_OPENAI_DEPLOYMENT - Your deployment model name
- AZURE_DEVOPS_PAT - Your Personal Access Token (mark as secret)
Step 2: Configure Pipeline
Now you need to create and configure the pipeline in Azure DevOps:
- Go to Pipelines → Pipelines in Azure DevOps
- Click "New pipeline" or "Create pipeline"
- Select "Azure Repos Git"
- Choose your repository
- Select "Existing Azure Pipelines YAML file"
- Choose your develop or feature branch
- Select the path to your azure-pipelines-pr-review.yml file
- Click "Continue"
- Review the pipeline configuration
- Click "Save" (don't run it yet)
- Optionally, rename the pipeline to something descriptive like "AI PR Review"
Step 3: Branch Policy and Testing
Edit Branch Policies
Configure your branch to automatically trigger the AI review on pull requests:
- Go to Repos → Branches
- Find your develop branch
- Click the ⋮ menu → Branch policies
- Scroll down to "Build Validation" section
- Click "+ Add build policy"
- Select your "AI PR Review" pipeline
- Set Trigger to: Automatic
- Click Save
Add Code with Issues
To test the system, add or modify code that contains security vulnerabilities or performance issues. For example, code with:
- Hardcoded credentials or API keys
- SQL injection vulnerabilities
- Inefficient loops or string concatenation
- Blocking async calls
- Missing input validation
Create Pull Request
Now create a pull request to test the AI review system:
- Create a feature branch with your changes
- Commit and push your code
- Go to Repos → Pull requests
- Click "New pull request"
- Select your feature branch as source
- Select develop as target
- Add a title and description
- Click "Create"
View AI Review Results
After you create the pull request:
- The AI PR Review pipeline will automatically trigger
- You'll see it running in the PR checks section
- Wait 1-2 minutes for the pipeline to complete
- Once finished, refresh the PR page
- You'll see AI-generated comments highlighting issues found in your code
The AI will provide detailed feedback including:
- Security Issues: Hardcoded credentials, SQL injection, command injection, weak cryptography, etc.
- Performance Issues: N+1 queries, blocking calls, inefficient loops, memory leaks, etc.
- Code Quality: Best practices violations, potential bugs, maintainability concerns
- Suggestions: Specific recommendations on how to fix each issue
Benefits of AI PR Reviews
- Automated: Every PR gets reviewed automatically without manual intervention
- Consistent: Same quality standards applied to every pull request
- Fast: Reviews complete within 1-2 minutes
- Comprehensive: Checks for security, performance, and quality issues
- Educational: Developers learn from AI feedback
- Early Detection: Catches issues before code review or production
- 24/7 Availability: Works around the clock, no waiting for reviewers
What the AI Detects
Security Vulnerabilities
- Hardcoded credentials (passwords, API keys, tokens)
- SQL injection risks
- Command injection vulnerabilities
- Path traversal issues
- Cross-site scripting (XSS)
- Weak password validation
- Insecure cryptography
- Information disclosure in error messages
- Missing HTTPS enforcement
- Lack of input validation
Performance Issues
- N+1 query problems
- Inefficient string concatenation in loops
- Blocking async calls (.Result, .Wait)
- Creating HttpClient instances repeatedly
- Reading entire files into memory
- Regex compilation in loops
- Exceptions used for control flow
- Multiple LINQ enumerations
- Synchronous database calls
Code Quality Issues
- Missing error handling
- Resource disposal problems
- Code duplication
- Magic numbers without explanation
- Poor naming conventions
- Missing null checks
Troubleshooting
Pipeline Fails with Authentication Error
- Verify AZURE_OPENAI_API_KEY is correct
- Check that AZURE_DEVOPS_PAT has Code Read & Write permissions
- Ensure PAT hasn't expired
- Confirm variable group is linked to pipeline
No AI Comments Appear
- Check pipeline logs for errors
- Verify SYSTEM_ACCESSTOKEN has permissions
- Ensure you're modifying supported file types (.cs, .py, .js, etc.)
- Confirm the script has access to read PR changes
AI Review is Too Strict or Too Lenient
- Adjust the temperature parameter in the script (0.1-0.5 for stricter)
- Modify the AI prompt to focus on specific areas
- Change the severity thresholds
Customization Tips
Add More File Types:
Edit the file extension check in ai-pr-review.py to include additional languages like .go, .rb, .php, .cpp, etc.
Focus on Specific Issues:
Modify the AI prompt to prioritize certain types of issues (e.g., security over style).
Adjust Review Depth:
Change max_tokens parameter to get more or less detailed feedback.
Integration with Other Tools:
Combine with SonarQube, CodeQL, or other static analysis tools for comprehensive coverage.
Conclusion
You've successfully set up an AI-powered Pull Request review system that automatically analyzes code for security vulnerabilities, performance issues, and code quality problems. This automation helps your team maintain high standards, catch issues early, and ship better code with confidence.
The system runs automatically on every pull request, providing fast, consistent, and actionable feedback without requiring manual intervention. Developers get immediate insights into potential problems and learn from the AI's suggestions over time.
- Customize the AI prompts for your team's specific coding standards
- Expand file type support as needed
- Create metrics dashboards to track code quality trends
- Integrate with team notifications (Slack, Teams, etc.)
- Fine-tune the AI model for your specific codebase
Comments
Post a Comment