Senior Quality Engineer - AI/LLM Test Automation
Tek Inspirations LLC
- Job
- 28894
- Posted
- Location
- Birmingham, AL
- Work type
- Full Time
- Tax terms
- W2, Yearly
- Experience
- Experience open
- Rate
- $120,000 (Annum)
- Openings
- 1 opening
Opens your email app with a message to the employer, its subject naming this job. Attach your resume and send it from your own email.
Skills
- Python
- TypeScript
- CI/CD
- Playwright
- Agile
About the job
Position : Senior Quality Engineer - AI/LLM Test Automation
Location: Birmingham, AL
Type: Full-Time
Must Have
- 5+ years of QA / SDET / Test Automation experience.
- 1-2+ years of hands-on experience testing AI/LLM applications.
- Hands-on experience with LLM evaluation / AI evaluation.
- Experience creating or working with evaluation datasets and ground truth.
- Understanding of LLM judges / AI-as-a-judge, evaluators, scoring, and quality thresholds.
- Experience testing nondeterministic AI-generated outputs.
- Hands-on LangSmith experience.
- Hands-on LangGraph or agent workflow experience.
- Strong Playwright experience, preferably with TypeScript.
- Strong Python and/or TypeScript skills.
- API, integration, regression, and end-to-end testing experience.
- Experience integrating automated tests into CI/CD.
- Strong understanding of Git and Agile development.
Ideal Background
Someone who has worked as an SDET / Senior QA Automation Engineer / AI Test Engineer / AI Quality Engineer and can explain, with real examples:
Production/User Behavior ? Evaluation Dataset ? Ground Truth ? Evaluator/LLM Judge ? Experiment ? Score/Threshold ? Regression ? CI/CD
The candidate should be able to explain how they tested AI outputs, not simply how they used AI tools to write traditional automation tests.