Software Development Engineer in Test - AI

About the role

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Software Development Engineer in Test – AI based in India.

We are looking for a forward-thinking Software Development Engineer in Test specializing in AI to drive quality, reliability, and innovation across AI-powered applications and intelligent systems. In this role, you will go beyond traditional software testing by designing advanced evaluation frameworks for LLMs, AI agents, copilots, and RAG-based applications. You will combine strong automation engineering skills with hands-on AI expertise to validate complex workflows, improve system performance, and ensure safe and reliable AI outputs. Working closely with Engineering, AI/ML, Platform, Product, and DevOps teams, you will help integrate intelligent testing solutions throughout the software development lifecycle. You will also explore emerging AI frameworks, agentic architectures, and automation technologies to accelerate engineering productivity. This is an exciting opportunity to shape next-generation quality engineering practices in a collaborative, innovation-driven, and globally distributed environment.

Accountabilities: - AI/LLM Quality Engineering: Design, implement, and optimize testing strategies for AI-powered applications, chatbots, copilots, Retrieval-Augmented Generation (RAG) pipelines, and autonomous AI agents.

- AI Evaluation Frameworks: Develop and maintain AI evaluation (Evals) frameworks to assess hallucinations, groundedness, toxicity, bias, response quality, contextual relevance, prompt regressions, output consistency, and safety guardrails.

- Automated AI Validation: Build automated validation pipelines for LLM-generated outputs using structured assertions, semantic scoring, embeddings, and evaluation metrics. Create synthetic and benchmark datasets to validate different AI scenarios.

- Agentic Systems Testing: Validate multi-agent orchestration workflows, agent planning and execution, memory retention, context management, tool calling, function calling, and structured outputs. Test MCP (Model Context Protocol) integrations and interoperability.

- Prompt and Context Engineering: Contribute to prompt engineering, context engineering, and prompt optimization strategies to improve AI response quality, reliability, and consistency.

- Performance and Cost Optimization: Evaluate AI application latency, token consumption, throughput, scalability, and model usage costs to ensure efficient system performance.

- Automation Engineering: Develop and maintain scalable UI and API automation frameworks using Python, with tools such as Robot Framework, Playwright, Selenium, and Cypress. Build reusable testing libraries, utilities, and AI-assisted testing workflows.

- API and Application Testing: Automate testing across REST APIs, GraphQL APIs, event-driven systems, streaming services, webhooks, microservices, and micro-frontend architectures.

- RAG and AI Platform Validation: Validate RAG architectures, including document chunking strategies, embedding quality, retrieval accuracy, reranking, and vector search relevance. Work with vector databases such as Pinecone, Weaviate, ChromaDB, Qdrant, and FAISS.