About The Training

This advanced hands-on workshop is designed to immerse participants in the fast-evolving world of AI security. The training will cover adversarial AI testing, prompt injection, jailbreak attacks, red teaming strategies, and defensive techniques for modern AI systems. Through a combination of theory and practical labs, attendees will learn to simulate real-world attack scenarios and build robust defenses for LLMs and AI agents. By the end of the course, participants will have practical experience in both offensive and defensive AI security tactics, preparing them for challenges in real operational environments.
 

Intermediate

Day 1 – Foundations: Model Risk Evaluation and Automated Red Teaming

We begin with a deep dive into the real-world risks and historical incidents that have shaped AI security. Moving beyond basic setup, the focus is on understanding the taxonomy of AI model risks and evaluating models using automated tools against established benchmarks like AIRBENCH. The day progresses into LLM fundamentals, covering tokenization, embeddings, and transformer architectures. This foundational knowledge prepares participants to dive into jailbreaking techniques, from fundamentals to advanced automated attacks using tools like dtx and Pyrit. Participants will gain practical experience with Garak for red teaming and take a leap into advanced scenarios by extending Garak with custom modules, empowering them to automate complex AI attack simulations.

Topics Covered

• History of AI Security and notable incidents
• Introduction to AI SDLC (Secure Development Lifecycle)
• Understanding taxonomy of model risks (European priority risks included)
• Setting up testing environments (Local, Cloud, Kaggle, Colab)
• Benchmarking AI models with AIRBENCH
• LLM Fundamentals: Tokenizatio, Embeddings, and overview of transformer architecture
• Jailbreaking attacks: Fundamentals to automated techniques with AutoDAN, FLIPAttack

Automated Model Evaluation:

• Using dtx to generate jailbreak datasets
• Automated model red teaming with Garak and Pyrit
• Understanding and analyzing Garak reports
• Extending Garak: Writing custom plugins, integrating FLIPAttack
  
   

Day 2 – OWASP Top 10 LLM Applications Testing
We will focuse on applying offensive security techniques aligned with the OWASP Top 10 risks for LLM applications. Participants will uncover vulnerabilities in chatbots, RAG apps, and agents, with a focus on data and model poisoning, prompt injection attacks, and multi-modal threat vectors. Emphasis is placed on understanding embedding vulnerabilities (OWASP LLM08), automating application security evaluations, and escalating attacks using tools like dtx, promptfoo, and Pyrit.

Topics Covered

• Testing AI applications: Chatbot, RAG, Agent apps
• Exploiting security risks in AI applications:
• Bypassing prompt structures
• Generating malicious outputs
   

Data and model poisoning attacks:

• PoisonGPT demonstration
• ConfusedPilot and Kaggle-based poisoning
• Manipulating model weights
   

Embedding attacks:

• Understanding their role and vulnerabilities
• OWASP LLM08 risks exploration
   

Advanced prompt injections:

• Direct, indirect, and multi-modal attacks
• Crafting prompts to exploit system vulnerabilities
• Automating prompt injection testing
• Automated AI app security evaluations with dtx, promptfoo, Pyrit
  
   

Day 3 – AI Agents Red Teaming and Defending AI
The final day advances into the world of AI agents and their unique attack surface. Participants will construct, model, and execute attacks against agents, focusing on real-world scenarios and the OWASP Top 10 vulnerabilities. The training covers practical defense mechanisms using advanced tools like PromptGuard and LlamaGuard, and delves into adversarial attacks and state-of-the-art techniques including poisoning, backdooring models, and hijacking AI assistants. With a view to the future, participants will explore upcoming AI threats in 2025, concluding with an exciting, hands-on hackathon to apply their new offensive and defensive skills in real time.

Topics Covered

• Understand AI Agents:
• Understanding architectures and attack vectors
• Real-world agent architecture demo
• Building OSINT discovery agents with AutoGen
• Threat modeling with HAIstings and AurgusGPT
   

Security controls for AI agents:

• Input/output/context filtering strategies
• Security architecture and tools landscape
   

Detection and defense strategies:

• Detecting jailbreaks and prompt injections
• Mitigating toxicity and harmful outputs with LlamaGuard
• Adversarial bypass testing
   

Adversarial testing techniques:

• Using ART, TextAttack for model evasion
• Poisoning models with ART DLBD attack
   

Advanced AI attack scenarios:

• Implanting backdoors/trojans
• Phishing via hijacked AI assistants
• Insights from 100+ AI red teamings
• Future threats from AI Threats, Landscape and Frameworks

Participants should bring a laptop running a Linux-based OS or a virtual machine. To support the hands-on labs, ensure Docker and Docker Compose are pre-installed. Recommended specifications include a minimum of 16 GB RAM (32 GB preferred), a stable internet connection of at least 50 Mbps, and over 250 GB of free disk space to comfortably run containerized labs.
 

Before attending, participants should have basic proficiency in Python programming and a foundational understanding of cybersecurity concepts. Familiarity with Linux command-line basics is important, as well as having Docker and Docker Compose installed for the labs. Comfort with AI fundamentals, especially LLMs, will help maximize learning during the training.
 

This training is best suited for cybersecurity professionals, penetration testers, and red teamers who want to specialize in AI security. Software engineers and AI researchers working on LLMs will also benefit. AI security analysts focusing on adversarial AI testing and data scientists or ML engineers interested in the security of generative AI applications are highly encouraged to join.
 

Attendees will gain a basic understanding of LLM fundamentals, AI applications, and AI agent structures. The training is hands-on, with practical labs simulating real-world AI attacks and defensive strategies. Participants will explore prompt injection, jailbreaks, adversarial testing, and the application of red teaming methodologies against AI systems.

Key Outcomes from the Training:

1. Advanced AI Red-Teaming Proficiency
Gain practical experience in prompt injection, jailbreak exploits, and adversarial prompt manipulation.
Identify and weaponize vulnerabilities in AI systems to enhance offensive security capabilities.

2. Execution of Real-World Red Team Operations
Design and deploy red-teaming engagements targeting operational AI environments.
Simulate real-world attack scenarios to validate AI system security postures.

3. Development of Robust Defensive Architectures
Engineer mitigation strategies against prompt injection and adversarial exploitation.
Architect resilient AI/ML pipelines incorporating proactive security controls.

Participants will have full access to the lab environment throughout the training, allowing them to practice offensive and defensive techniques in a safe setup. In addition, they will receive workshop slides, detailed notebooks, and offline exercises. Tools and automation scripts for AI security testing will also be provided for further learning beyond the course.
 

This training does not focus on building AI models or in-depth AI engineering. Attendees should not expect to develop custom tools and models from scratch or receive extensive coverage of non-security aspects of AI models and agents. The emphasis remains firmly on offensive and defensive AI security techniques.