Lead QA Engineer

About the role

ABOUT TITAN

Titan builds AI software for banks: purpose-built small language models, a banking ontology, and AI bankers that financial institutions can trust. Our models outperform general-purpose LLMs by 30 to 80 percent on banking tasks. Customers include community banks, credit unions, and large regional and super-regional institutions. We are backed by leading fintech investors and operate under the compliance, audit, and model-risk standards that banking requires.

WHY THIS ROLE EXISTS

Titan is scaling from a handful of live banking customers to thirty, then to hundreds. Right now, there is no formal QA function. There is no evaluation framework, no regression baseline, no quality gate in CI/CD. A QA failure at a bank is not a user experience problem. It is an operational and regulatory risk. This role exists because that gap has to close before the customer count grows.

This is a hands-on, individual-contributor role first. You are coming in to do the work: write the test cases, build the evaluation framework, set up CI/CD gates, and triage bugs alongside engineering. The function gets built because you build it yourself. Once the practice is stable and documented, you bring in QE engineers to scale it.

WHAT YOU OWN

AI Evaluation. You personally design and execute the evaluation framework for LLM and agentic AI outputs across Foundry, Agent Builder, and client-deployed instances. You write the assertions, define the behavioral contracts, and own regression baselines for model behavior. Standard QA methods break down here: you cannot write a deterministic assertion for whether an AI accurately summarized a 200-page loan agreement. You need to think in distributions and confidence intervals, and you need to build tooling that does too.

Test Coverage. You write and maintain the automated test suite: end-to-end, integration, and regression coverage for backend APIs, document ingestion pipelines, AI inference workflows, and frontend surfaces. You own performance and load testing for latency-sensitive inference paths. You set up and enforce quality gates in CI/CD pipelines. When a bug surfaces in production, you are in the triage, you write the reproduction case, and you own the regression test that prevents it from coming back.

Compliance and Client Quality. You produce the test artifacts, audit logs, and process documentation that meet SOC 2 Type II standards. You work directly with Forward Deployed Engineering on client-side validation and production issue reproduction. Bank examiners will scrutinize this work. It needs to be defensible on its own.

WHO YOU ARE

Seven or more years in software QA engineering, with at least two years personally testing AI or ML systems. You have written test cases against LLM outputs, built evaluation pipelines from scratch, and know the difference between a flaky test and a genuinely non-deterministic system. You are fluent in Python and have built automated suites using pytest, Playwright, or Selenium. You have hands-on experience with RAGAS, DeepEval, LangSmith, or comparable evaluation tooling—not just familiarity with the names.