Join Our Team

About the Role

We're looking for experienced AI Developers in Test who want to help shape the future of AI system validation and quality assurance. You'll work with cutting-edge LLM-based applications, multi-agent systems, and RAG pipelines to ensure AI outputs are safe, reliable, and beneficial for end users.

AI System Validation & Testing

• Design and implement testing frameworks for LLM-based applications, multi-agent systems and RAG pipelines
• Evaluate AI outputs for conciseness, completeness, semantic accuracy, factual correctness, contextual relevance, tone consistency, bias, safety and compliance
• Implement governance workflows, risk scoring methods and compliance reporting
• Set up continuous monitoring for production AI systems, with automated alerts for anomalies

Testing Infrastructure & Automation

• Validate LLM APIs across multiple providers using tools like Postman, REST Assured and pytest
• Integrate AI testing suites into CI/CD pipelines for regression testing, benchmarking and deployment gates
• Manage version control for prompts and test cases using Git and leverage MLOps tools such as MLflow or Weights & Biases

Data Engineering & Evaluation

• Manage diverse data formats (JSON, CSV, Parquet, JSONL) and build evaluation datasets from real-world scenarios
• Design automated metrics and human-in-the-loop workflows, including inter-annotator agreement processes
• Create and maintain domain-specific evaluation benchmarks and ground truth datasets

Leadership & Client Management

• Build a high-performing team and serve as the subject matter expert
• Cultivate and strengthen long-term client relationships, focusing on retention and business growth
• Contribute with your knowledge and experience within the bsure-digital test consulting community

Requirements

• Proven hands-on experience in GenAI application development (prompt engineering, API integrations, output workflows)
• Familiarity with RAG concepts, vector databases, embedding strategies and multi-agent systems
• Strong understanding of GenAI challenges such as hallucinations, prompt sensitivity, consistency issues and output variability
• Solid software testing background in enterprise environments, with the ability to communicate technical concepts to varied stakeholders
• Proficiency in API testing frameworks, CI/CD integration and Git-based collaboration
• Data handling expertise and knowledge of evaluation methodologies for AI performance and risk assessment
• Self-starter mindset, comfortable working in ambiguous environments to define new testing methodologies and quality standards
• Passionate about ensuring AI systems are safe, reliable and beneficial for end users
• Demonstrate outstanding communication and presentation skills beyond pure technical expertise

Nice to Have

• Experience with LLMs (ChatGPT, Claude, etc.) for test generation
• Knowledge of computer vision for visual testing
• Experience with cloud platforms (AWS, Azure, GCP)
• Certifications in testing (ISTQB, etc.)

What We Offer

• Competitive salary based on experience
• Flexible remote/hybrid work arrangements
• Professional development budget for training and conferences
• Access to latest AI tools and technologies
• Work with diverse international clients
• Collaborative and innovative work environment