How AI Agents, GenAI and Autonomous Testing Are Changing Software Quality.

A forward-looking workshop on the evolution of software testing in the age of Generative AI and AI agents. Participants will explore how AI already supports requirements analysis, test-case and test-data generation, API testing, automation, log analysis, and defect analysis and what this shift means for the tester’s role. The workshop moves from AI as a QA co-worker to agentic testing, testing of LLM-based systems, AI red teaming, and the emerging role of the AI Quality Engineer and Quality Architect.

Objectives

• Explain how the software tester’s role is evolving as AI takes over more traditional QA activities.
• Distinguish simple prompting from professional Prompt Engineering for software testing.
• Apply appropriate approaches for testing non-deterministic LLM-based systems.
• Use AI as a QA co-worker for requirements ambiguity detection, risk analysis, decision tables, test cases, API tests, automation skeletons, and bug reports.
• Explain the transition from AI Copilot assistance to Agentic Testing that can plan, execute, evaluate, adapt, and collect evidence.
• Identify AI-specific quality and security risks including hallucination, prompt injection, jailbreaks, data leakage, bias, and unsafe agent permissions.

Target Audience

Software testers, QA engineers, automation engineers, QA leads, test managers, and quality professionals who want to understand how AI is changing modern testing.

Prerequisites & Experience

Participants should be familiar with core software-testing concepts. Familiarity with API testing and test automation is helpful for practical examples.

Environment Setup

• Google Chrome or Microsoft Edge: Required for AI tool utilization, DevTools analysis, and web demonstrations.
• ChatGPT Account: A Plus, Business, or Enterprise subscription is preferred, though a large portion of exercises can be executed using a free tier account. Ensure organizational network policies permit access.
• Visual Studio Code: Required for viewing, editing, and executing automation code.
• Node.js (LTS version)
• Playwright: Recommended to be pre-installed prior to the workshop.
• Postman: The desktop application is recommended for enhanced stability and convenience during the session, though the web version is also supported.

Recommended Installations

• Git: Required for cloning the practice repository and version control.
• Playwright Test for VS Code Extension
• ZIP Archive Utility: Required for handling compressed project files.

Agenda

The Tester Is Not Dead – The Job Is Changing

What AI can already perform: requirements analysis, test-case generation, test data, API tests, automation, log analysis, and defect analysis. The key question: if AI can do all of this, what remains for the tester?

AI as Your QA Co-Worker

Live flow: Requirement → ambiguity detection → Risk Analysis → Decision Table → Test Cases → API Tests → Playwright skeleton → Bug Report. Professional Prompt Engineering for QA, including practical do’s and don’ts.

From Copilot to Agentic Testing

Use an agent to inspect a business flow, plan actions, execute tests, evaluate results, change direction when needed, and produce evidence.

Testing AI Is Different from Testing Software

Why Input → Expected Result is not enough for LLMs. Probabilistic testing, test oracles, statistical evaluation, metamorphic testing, LLM-as-a-Judge, non-determinism, and the AI testing oracle problem.

How Do You Test an LLM?

Example: an AI Customer Support Box. Test hallucination, relevance, factual correctness, consistency, refusal behavior, prompt injection, jailbreaks, toxic output, information leakage, bias, and context handling.

AI Red Teaming for Testers

Prompt injection, jailbreaks, indirect injection, data leakage, instruction hierarchy, and agent permissions. Applying exploratory testing and red-teaming thinking to GenAI systems

Testing AI-Generated Code

As developers and AI generate code faster, QA must strengthen automated verification, regression, security, observability, and quality gates, with human governance above increasingly autonomous testing systems.

The Quality Architect – The Tester’s Next Role

Career evolution: Tester → Automation Engineer → AI-Assisted Tester → AI Quality Engineer → Quality Architect. Focus on defining what must be tested, which risks matter, whether results can be trusted, and how to build verification mechanisms for AI itself.

Key Topics & Practical Coverage

• AI-assisted requirements analysis and ambiguity detection
• Test data, API testing, and Playwright automation skeletons
• Probabilistic testing and test-oracle strategies for LLMs
• Risk analysis, decision tables, and AI-generated test cases
• Agentic test planning, execution, adaptation, and evidence collection
• Statistical evaluation, metamorphic testing, and LLM-as-a-Judge

Tools & Technologies Covered

AI & Quality

Generative AI / LLMs, AI Agents, Prompt Engineering, Agentic Testing, LLM-as-a-Judge, AI Red Teaming

Testing & Automation

Requirements analysis, risk analysis, decision tables, test cases, test data, API testing, Playwright, regression, security, observability, quality gates