Securing AI/ML Systems: Protecting the Intelligent Attack Surface

A practitioner course on securing AI and machine learning systems as systems — the data, the model, the pipeline and the application — rather than as a policy topic. It separates traditional cybersecurity risk from AI-specific risk, then runs the AI lifecycle from design through retirement with security gates at each stage. Adversarial machine learning is the core: poisoning, evasion, privacy attacks, and model extraction, paired with the defenses that raise robustness, grounded in the NIST adversarial ML taxonomy (AI 100-2 E2025). Generative AI gets its own module: direct and indirect prompt injection, sensitive information disclosure, insecure output handling, and the risks retrieval-augmented generation, plugins and external data sources introduce.

Two further modules cover ground usually missing entirely: treating trained models, weights and checkpoints as protected assets requiring provenance verification, and the AI supply chain — third-party models, datasets, libraries and frameworks as dependencies that can be malicious or vulnerable, since teams routinely pull weights off public repositories with far less scrutiny than a software dependency gets. A closing workshop has participants threat-model a fictional AI system and defend a prioritized hardening plan. This is the engineering course, not the policy course; for the awareness tier, see AI Security Fundamentals for the Cyber Workforce.

What You Will Learn

• Identify the components of an AI/ML system architecture and distinguish models, datasets, pipelines, applications and infrastructure
• Distinguish traditional cybersecurity risk from AI-specific security risk
• Map security considerations across the full AI lifecycle from design through retirement
• Protect the confidentiality, integrity, availability and provenance of training and operational data
• Identify data poisoning and manipulation risks across collection, labeling and preprocessing
• Explain poisoning, evasion, privacy and model-extraction attacks, and apply defenses that increase robustness
• Treat models, weights and checkpoints as protected assets, and verify provenance and integrity before deployment
• Evaluate third-party models, datasets, libraries and frameworks as supply-chain dependencies
• Identify direct and indirect prompt injection, insecure output handling, and retrieval-augmented generation risk
• Secure ML pipelines, orchestration platforms, APIs and deployment processes
• Detect abnormal inputs, outputs and model behavior, and distinguish performance degradation from compromise
• Develop AI-specific incident response including containment, model rollback and post-incident validation

Course Outline

Module 1

• Know What You're Defending: AI/ML Security Foundations
• AI, machine learning, deep learning, generative AI and foundation-model concepts
• System architecture components
• Assets, trust boundaries, interfaces and dependencies
• Traditional versus AI-specific risk
• Security, resilience, robustness, privacy and trustworthiness

Module 2

• Follow the Model: The AI/ML Security Lifecycle
• Security across design, development, training, testing, deployment, operation and retirement
• Responsibilities across developers, operators, users and service providers
• Risks in experimentation and model development environments
• Protecting development, testing, staging and production
• Security gates through the lifecycle

Module 3

• Data Is the Fuel: Securing Training and Operational Data
• Training, validation, testing, inference and operational datasets
• Confidentiality, integrity, availability and provenance
• Access control for sensitive datasets
• Data poisoning and manipulation
• Protecting collection, labeling, transformation and preprocessing
• Monitoring quality, integrity and lineage

Module 4

• Attack the Learning Process: Adversarial Machine Learning
• Attacker objectives
• Poisoning attacks against training data and learning processes
• Evasion attacks against deployed models
• Privacy attacks targeting models and training information
• Model extraction and information disclosure
• Defensive concepts for robustness and resilience

Module 5

• Guard the Model: Model Integrity & Intellectual Property
• Models, weights, parameters and configurations as protected assets
• Securing repositories and storage
• Access control for model files, checkpoints and artifacts
• Provenance and integrity verification before deployment
• Theft, substitution and tampering
• Secure versioning, approval and change management

Module 6

• Secure the AI Supply Chain: Models, Libraries & Dependencies
• Third-party models, datasets, libraries, frameworks and services as dependencies
• Evaluating externally sourced components
• Verifying provenance and integrity of acquired artifacts
• Malicious or vulnerable dependencies in development environments
• Controlling imports, updates, plugins and integrations
• Ongoing supplier monitoring

Module 7

• When AI Starts Talking: Generative AI & LLM Security
• Security implications of large language models
• Direct and indirect prompt injection
• Sensitive information disclosure and unintended exposure
• Insecure output handling and downstream risk
• Retrieval-augmented generation, plugins, tools and external data sources
• Isolation, access control, validation and least privilege for generative AI applications

Module 8

• Secure the Pipeline: MLOps, APIs & Deployment
• Protecting pipelines, orchestration platforms and automation workflows
• Securing repositories, build environments and deployment
• Protecting secrets, keys, tokens and credentials
• Authentication and authorization for AI services and APIs
• Separating development, training, testing and production privileges
• Detecting unauthorized change

Module 9

• Watch the Machine: Monitoring, Detection & Incident Response
• Security logging and telemetry for AI environments
• Monitoring model access, administrative actions, API activity and configuration change
• Detecting abnormal inputs, outputs, behavior and usage
• Distinguishing performance degradation from compromise
• AI-specific incident response
• Containment, model rollback, recovery and post-incident validation

Module 10

• Govern, Test, Defend: AI Security Risk Workshop
• Identify assets, trust boundaries and dependencies in a fictional AI system, develop AI-specific threat scenarios and attack paths, find data, model, pipeline, infrastructure and generative AI vulnerabilities, evaluate likelihood and impact, then select, prioritize and defend a risk-based hardening plan