AI Engineering & Automation

Full Curriculum

Click any level to see what you'll learn.

The foundation everything else builds on. You'll learn to navigate a Linux system, create and manage files, control who can access what, and chain commands together to automate tasks. By the end, you'll be able to set up users, connect to servers over SSH, install software, and run your first Docker container.

T01 Files & Navigation (+Quiz)

Learn to move around a Linux filesystem, create and organize directories, copy, move, and remove files, and search for anything on the system.

ls, cd, pwd, mkdir, rm, cp, mv, find

T02 Viewing, Editing & Permissions (+Quiz)

Read files from the terminal, edit them with nano, and control exactly who can read, write, or execute every file on the system.

cat, head, tail, less, nano, chmod, chown

T03 Pipes & Redirection (+Quiz)

Chain commands together to filter, sort, count, and search through data. Redirect output to files, combine streams, and build one-liners that do real work.

|, >, >>, 2>&1, tee, sort, uniq, wc, grep

T04 Processes & System (+Quiz)

See what's running on your system, monitor resource usage in real time, stop runaway processes, and manage background jobs.

ps, top, kill, bg, fg, jobs

Lab Disk Space Incident

A disk space alert fired overnight. Investigate the cause, find what's eating the space, and preserve the evidence.

Covers: df, ls -lh, tail, grep -c, output redirection

T05 Users & SSH (+Quiz)

Create and manage user accounts, configure sudo access, generate SSH keys, and connect to remote servers securely.

useradd, passwd, sudo, ssh, ssh-keygen, authorized_keys

T06 Networking (+Quiz)

Check your network interfaces, test connectivity, make HTTP requests from the terminal, understand how DNS resolves names, and identify what's listening on which ports.

ip, ss, ping, curl, DNS resolution, ports

T07 Package Management (+Quiz)

Install, update, and remove software. Manage repositories, resolve dependency issues, and keep your system current.

apt, dpkg, repositories, dependencies

T08 Docker Basics (+Quiz)

Pull images, run containers, map ports, and manage the container lifecycle. Your first step into the tool that changed how software gets deployed.

images, containers, run, ps, stop, rm, port mapping

Lab Shift Handoff

Your colleague left two issues to check before their shift ended. Read the handoff, investigate both, and close them out.

Covers: cat, ls, find, wc, free

Exam Level 1 Master Exam

Prove you've mastered the fundamentals. 10 questions covering everything from Level 1. Score 80% or higher to pass.

Covers: files, permissions, pipes, processes, users, SSH, networking, packages, docker

Cards Level 1 Flashcards

Drill every command and concept from Level 1. Cards pull from this level's quizzes and exam, and the ones you miss come back more often.

Python-first, no filler. Variables, data structures, file I/O, JSON, CSV, regex, error handling, HTTP calls, and CLI tool construction — every skill you need to start calling AI APIs in Level 3. Every practice runs against the actual sandbox files.

T09 Variables, Types & Functions (+Quiz)

The Python you actually need: variables, types, f-strings, the string methods that replace awk, conditionals, loops, and functions. Every example pulls a real value out of a real-looking log line.

python3, type, f-strings, split, strip, replace, startswith, join, if/else, for, def, lambda, return

T10 Data Structures (+Quiz)

Lists, dicts, sets, tuples, comprehensions, and the list-of-dicts shape every API response uses. Each practice uses server-and-status data so the patterns transfer directly to Level 3 API work.

list, dict, set, tuple, len, in, .get, .keys, .values, .items, list comprehension

T11 File I/O & JSON (+Quiz)

Read and write files the right way (with-statement), parse JSON into native Python objects, pretty-print results, and count patterns in a log file. Every output below was computed against the actual sandbox files.

open, with, read, readlines, write, json.load, json.loads, json.dump, json.dumps

T12 CSV, Data Parsing & Regex (+Quiz)

csv.reader vs DictReader, computing stats from real inventory data, and the regex toolkit (search, findall, match, sub) you'll use to extract IPs, timestamps, and status codes from logs.

csv.reader, csv.DictReader, csv.writer, csv.DictWriter, re.search, re.findall, re.match, re.sub

Lab Fleet Status Report

Build a Python pipeline that reads JSON server inventory, cross-checks against a CSV status export, and outputs a clean status summary your team can paste into Slack.

Covers: file discovery, JSON parsing, CSV parsing, data cross-checking, dict counting, formatted output

T13 Error Handling & HTTP (+Quiz)

Catch the right exceptions, call HTTP APIs with the requests library, and handle every failure mode an LLM call can throw at you in Level 3. Both pure-pattern practices and offline-runnable file-based practices are included.

try, except, finally, ZeroDivisionError, ValueError, KeyError, FileNotFoundError, json.JSONDecodeError, requests.get, status_code, json, raise_for_status, ConnectionError, Timeout

T14 Environment & Dependencies (+Quiz)

Set up an AI project the right way: env vars for secrets, .env files, pip-managed dependencies pinned in requirements.txt, and isolated virtual environments per project. All commands are simulated with realistic outputs — the patterns transfer directly to your real shell.

export, env, echo, os.environ, .env, .gitignore, pip install, pip freeze, requirements.txt, python3 -m venv, source activate

T15 Building CLI Tools (+Quiz)

Turn Python scripts into real command-line tools — shebang + chmod + argparse for arguments, flags, subcommands, and auto-help; sys.exit for proper exit codes; clean separation of stdout (results) and stderr (errors).

#!/usr/bin/env python3, chmod +x, sys.argv, argparse.ArgumentParser, add_argument, add_subparsers, parse_args, sys.exit

T16 Python Capstone — Log Parser (+Quiz)

A full log analyzer — argparse, error handling, regex extraction, Counter aggregation, two output formats, and proper exit codes. The 'before AI' baseline that Level 4 will rewrite with five lines and a system prompt.

argparse, try/except, FileNotFoundError, re.compile, Counter, sys.exit, sys.stderr, json.dumps

Lab Log File Analyzer — No AI

Recreate scripts/log_parse.py from first principles — read auth.log, filter failures, extract attacker IPs, build a histogram, and write a structured JSON report. No AI yet; just the Python plumbing the L4 tools build on.

Covers: log parsing, regex extraction, Counter histograms, JSON serialization, report writing

Exam Level 2 Master Exam

Prove you've mastered Python for AI engineering. 10 questions covering variables and types, string methods, data structures, file I/O, JSON and CSV, regex, error handling, HTTP calls, and CLI tool construction. Score 8 of 10 to pass.

Covers: variables, types, string methods, lists, dicts, file I/O, JSON, CSV, regex, error handling, requests, argparse, sys.exit

Cards Level 2 Flashcards

Drill every command and concept from Level 2. Cards pull from this level's quizzes and exam, and the ones you miss come back more often.

First contact with AI APIs. How tokens, context windows, and temperature actually work; how to make a real call to Claude Haiku and read every field of the response; how system prompts shape voice and format; the three prompting techniques you'll reach for daily — zero-shot, few-shot, chain of thought; structured JSON output with recovery and validation; deterministic iteration with temperature=0.0; prompt injection and defense in depth; per-call and monthly cost math, prompt compression, and hash-based caching. All offline-runnable with the L3 sandbox; with a real ANTHROPIC_API_KEY in env, the same commands run live.

T17 How LLMs Work (+Quiz)

What a token is, how the context window works, what temperature actually controls, and how to put a dollar figure on every API call. All offline — no real API needed. The token_counter.py from Level 2 powers every estimate.

tokens, context window, temperature, top_p, stop_sequences, token_counter.py, cost estimation

T18 First API Call (+Quiz)

Make your first real call to Claude. Read the response object: text, usage, stop_reason. Stream tokens as they arrive. Every example uses Claude Haiku 4.5 with the L3 sandbox tools — when you have a key in the env, the same commands run live.

anthropic.Anthropic, messages.create, messages.stream, content[0].text, usage.input_tokens, usage.output_tokens, stop_reason

T19 System Prompts (+Quiz)

Set the model's persona and format with the system prompt. The same user question yields wildly different answers under sysadmin, analyst, and generic system prompts — same model, different voice. Constrain length and format to control cost and parseability.

system=, prompts/system_sysadmin.txt, prompts/system_analyst.txt, JSON output, length caps, prompt_tester.py

T20 Prompt Techniques — Zero-shot, Few-shot, Chain of Thought (+Quiz)

Three prompting techniques on the ladder of accuracy: zero-shot for simple cases, few-shot for classification with custom categories, chain of thought for problems that need reasoning. Combine them when the task earns it.

zero-shot, few-shot, chain of thought, system prompt, classification, diagnosis

Lab The Broken Classifier

Diagnose and fix a broken ticket classifier — identify why zero-shot fails, apply few-shot examples to improve accuracy, and see how chain-of-thought reasoning handles ambiguous inputs that few-shot alone can't fix.

Covers: prompt diagnosis, zero-shot vs few-shot, classification accuracy, chain of thought, prompt iteration

T21 Structured Output (+Quiz)

Get the model to return JSON your code can actually use. Schema in the system prompt, json.loads to parse, try/except + recovery for malformed responses, explicit validation of keys and enums. The shape every Level 4 tool will wear.

system prompt schema, json.loads, json.JSONDecodeError, code-fence recovery, regex extraction, key validation, enum validation

T22 Temperature & Iteration (+Quiz)

Why temperature=0.0 is the right default for anything you measure, why temperature=1.0 belongs to brainstorming, and how three iterations of the same prompt evolve from generic prose to ship-ready triage.

temperature=0.0, temperature=1.0, prompt iteration, evaluation, prompt_tester.py

T23 Prompt Safety (+Quiz)

Prompt injection: what it is, why it works, and the layered defenses (role isolation, delimiters, input sanitization, output validation, token caps, logging) you stack on top of the model's own refusal to harden the tool against adversarial inputs.

prompt injection, input sanitization, delimiter isolation, output validation, defense in depth, injection_tests.txt

T24 Cost Management (+Quiz)

Compute per-call and projected monthly costs from message.usage, pick the right model tier, compress prompts, cap max_tokens, and add hash-based response caching with measurable savings.

cost = tokens × rate, model selection, prompt compression, max_tokens, hash-based caching, hashlib.sha256, usage logging

Lab API Wrapper Library — Reusable Functions

Build a small reusable library (scripts/ai_lib.py) that wraps the Anthropic API with env-var key handling, JSON-recovery try/except, response caching, and structured usage logging — the safe defaults you'll reach for in every tool from here on.

Covers: API wrapper design, env-var secrets, JSON parsing with recovery, response caching, usage logging, CLI integration

Exam Level 3 Master Exam

Prove you've internalized AI APIs and prompt engineering. 10 questions covering tokens and cost, the messages.create surface, system prompts, prompt techniques (zero-shot/few-shot/CoT), structured output, prompt safety, and cost management. Score 8 of 10 to pass.

Covers: tokens, cost, messages.create, content[0].text, system prompts, zero-shot, few-shot, chain of thought, JSON output, json.loads, prompt injection, defense in depth, model selection, caching

Cards Level 3 Flashcards

Drill every command and concept from Level 3. Cards pull from this level's quizzes and exam, and the ones you miss come back more often.

Three real tools deep, not eight shallow. Log analyzer (T25-T27): build, multi-source, hardened with caching/logging/chunking. Report generator (T28-T29): build, plus exec briefing and tech appendix templates from the same input. Ticket classifier (T30-T31): build, batched, then accuracy iteration from 70% to 94% with confidence routing. T32 names the pattern: every L4 tool is read → process → output, with prompts in files and AI logic in lib/ai_utils.py.

T25 Log Analyzer — Build (+Quiz)

Replace the L2 hand-written analysis with a 5-line API call. tools/log_analyzer.py wraps that core in argparse, error handling, and a --output text|json switch. Multi-paragraph analyst verdicts for humans; structured JSON for pipelines. Cost is roughly a tenth of a penny per call.

tools/log_analyzer.py, --output text, --output json, jq, anthropic.Anthropic, system_analyst.txt

T26 Log Analyzer — Multi-Source (+Quiz)

Three log files, one analysis. Concatenate auth.log + syslog + access.log with source labels and let Claude correlate IPs, reconstruct timelines, and distinguish independent actors. The cross-source view surfaces the gap between events that single-source analysis would miss.

multi-source concatenation, source labels, IP correlation, timeline reconstruction, distinguishing actors, JSON output

T27 Log Analyzer — Hardened (+Quiz)

Five hardening features turn the T25 prototype into something you'd schedule. Hash-based caching avoids paying twice. Per-call JSONL logging feeds Level 5's monitoring. Four error modes (API down, rate limit, empty input, malformed response) are handled with clear messages. --output text/json/markdown serves three audiences. Chunking + deduplication handles logs that don't fit in one prompt — auth_large.log's 2000 lines split into 4 chunks of 500.

lib/ai_utils, @with_retry, @log_call, @cached, cache_key, --output text|json|markdown, --chunk-size, deduplication

T28 Report Generator — Build (+Quiz)

Same skeleton as log_analyzer.py, different persona — sysadmin instead of analyst — and different output template — markdown documentation instead of a triage verdict. Reads a config (nginx, sshd, docker-compose, .env, crontab), produces an Overview / Directives table / Security Concerns section. Catches the lazy mistakes (defaults left in place) consistently for every file.

tools/report_generator.py, prompts/system_sysadmin.txt, --output, markdown documentation, security concerns, output validation

Lab The Misclassified Alerts

The triage tool classified 20 alerts overnight — but the on-call engineer flagged one as wrong. Diagnose why the model misclassified a privilege escalation, identify the prompt's blind spot, and fix it with authorization context.

Covers: classification review, false positive analysis, prompt diagnosis, authorization context, prompt iteration

T29 Report Generator — Multiple Formats (+Quiz)

Same configuration, two reports. The executive briefing serves the CISO in 500 words of business language; the technical appendix serves the on-call engineer with line-by-line citations and exact remediation commands. Templates live as files in prompts/, the --format flag picks one, and adding a new audience is a new file rather than a code change.

tools/report_generator.py, --format exec, --format tech, prompts/exec_briefing.txt, prompts/tech_appendix.txt, template-based prompts

T30 Ticket Classifier — Build (+Quiz)

Build a 50-ticket classifier. Single-ticket baseline first, then batched (10 per call) for the cost/latency win. Measure accuracy against the labeled true_category column; read the confusion matrix to see where the classifier confuses categories. 92% out of the box; T31 takes it to 95%+.

tools/ticket_classifier.py, csv.DictReader, batch processing, BATCH_SIZE, classify_batch, accuracy measurement, confusion matrix

T31 Ticket Classifier — Accuracy & Edge Cases (+Quiz)

Climb classifier accuracy on tickets_test.csv from 70% to 94% (on confident predictions) in three iterations. v1: basic prompt (70%). v2: four few-shot examples that pin the cross-category boundaries (85%). v3: confidence threshold routes the ambiguous 15% to a human (94% accuracy on the rest, 100% safe routing on the flagged ones). Tone and language don't change category — the prompt handles angry customers and non-English tickets natively.

tickets_test.csv, predictions.csv, accuracy measurement, confusion matrix, few-shot examples, confidence routing, edge cases

T32 Tool Architecture Patterns (+Quiz)

Name the pattern. Every L4 tool is read → process → output, where 'process' is one Claude call. Prompts live in .txt files; knobs in api_config.json; AI logic in lib/ai_utils.py; tools compose them. Each tool is also an importable module. Apply these five rules and the next AI tool you write takes a day, not a week.

read → process → output, prompts as files, configs/api_config.json, lib/ai_utils.py, importable modules, if __name__

Lab Infrastructure Documentation Suite

Read five config files (nginx, sshd, docker-compose, .env, crontab), generate per-file documentation with Claude, and merge them into a single wiki-ready markdown reference with a table of contents and a consolidated security-concerns section.

Covers: multi-file processing, AI documentation generation, markdown synthesis, table of contents, consolidated security report

Exam Level 4 Master Exam

Prove you've internalized AI tool building. 10 questions covering log analyzer (build, multi-source, hardened), report generator (single and multi-format), ticket classifier (build and accuracy), and tool architecture patterns. Score 8 of 10 to pass.

Covers: log_analyzer.py, multi-source correlation, hash-based caching, JSONL logging, chunking, deduplication, report generator, exec briefing vs tech appendix, ticket classifier, batching, accuracy iteration, confidence routing, tool architecture, prompts as files, importable modules

Cards Level 4 Flashcards

Drill every command and concept from Level 4. Cards pull from this level's quizzes and exam, and the ones you miss come back more often.

Ship the L4 tools as production systems. Tests (T33), packaging + systemd (T34), cron + flock + timers (T35), pipelines (T36), concurrent batch processing (T37), structured logging + alerts (T38), cost optimization (T39), and the full production capstone (T40).

T33 Testing AI Tools (+Quiz)

Four kinds of tests for AI tools. Unit tests mock the SDK and verify your logic — fast, free, run on every commit. Integration tests make real calls and check key fields are present. Hallucination tests assert outputs are subsets of inputs. Regression tests compare against golden snapshots. Run all four; ship with confidence.

pytest, unittest.mock, @patch, MagicMock, pytest -v, pytest -m integration, golden snapshots, GitHub Actions

T34 Deployment (+Quiz)

Turn the L4 tools into installable CLI commands and run them as a systemd service. setup.py + pip install -e . in dev; systemd unit + ProtectSystem hardening in production. Three commands (status, journalctl, restart) cover 90% of operations.

setup.py, find_packages, entry_points, console_scripts, pip install -e ., pip wheel, systemctl, systemd unit file, ProtectSystem, journalctl, systemd-analyze security

T35 Scheduling & Automation (+Quiz)

Cron entries with flock locking, timestamped output directories, and systemd timers as the modern alternative. Three monitoring signals catch silent failures: lock-file mtime, today's output exists, error log size. Schedule once, observe forever.

crontab -l, crontab -e, cron syntax, flock, timestamped output, find -mtime, systemd timers, OnUnitActiveSec, Persistent=true, systemctl list-timers, journalctl

T36 Chaining & Pipelines (+Quiz)

Three steps, one config file, declarative orchestration. analyze → classify → report. Each step's output becomes the next step's input on disk; each step is skipped if its output already exists. Conditional routing (when predicates) branches on the previous step's data. Error propagation halts the pipeline by default; opt in to continue per step.

pipeline/config.json, pipeline.py, --force, conditional routing, when predicate, error propagation, exit codes, resumable pipeline, numeric output ordering

Lab Production Readiness Audit

Audit a junior engineer's sloppy_analyzer.py — hardcoded keys, raw tracebacks, no caching, no logging, no tests — then fix each problem one at a time with the L5 hardening checklist.

Covers: code audit, secret hygiene, error handling, response caching, structured logging, pytest with mocks

T37 Batch Processing (+Quiz)

Scale from 50 tickets to 1000+. ThreadPoolExecutor with 10 workers cuts a 51-second sequential run to 6 seconds — 10× faster, same correctness. Rate limits become the bottleneck; @with_retry absorbs 429s. Progress tracking on stderr shows '450/1000 (45%)'. Crash-safe resume reads existing output to skip completed items. Three properties combine into a production batch processor.

concurrent.futures, ThreadPoolExecutor, as_completed, max_workers, RateLimitError, @with_retry, progress tracking, [X/N (Y%)], stderr progress, crash-safe resume, already_processed

T38 Monitoring & Observability (+Quiz)

Structured JSON logging on every API call (one JSONL line per request) plus jq aggregation gives you 80% of a monitoring stack with no infrastructure. Three alert classes — failures, cost spikes, accuracy degradation — each is one jq query plus a threshold. The weekly roll-up (monitoring/usage_weekly.json) is what the on-call reads Monday morning.

JSONL, jq -s, jq aggregation, error_rate, cost_spike, accuracy_degradation, regression suite, monitoring/usage_weekly.json, by_tool, by_day, alerts[]

T39 Cost Optimization (+Quiz)

Profile, then optimize. Hash-based caching with per-content TTLs reclaims 30-60% of cost for stale-tolerant content. Prompt compression cuts input tokens ~40% with no quality loss. Model routing (Haiku for classify, Sonnet for analyze, Opus for forensic) saves 60-70% on classification-heavy workloads. Stacked, the three techniques cut a $50/day pipeline to $9-10/day — without dropping accuracy.

by_tool cost breakdown, cache TTL, prompt compression, model routing, Haiku vs Sonnet vs Opus, baseline measurement, before/after reporting

T40 Production Capstone (+Quiz)

Combine T33–T39 into one production system. The log analyzer now ships with tests, CLI packaging, systemd timer, cron-driven daily reports, hash-based caching, JSONL monitoring, and three-way model routing. 552 lines across 11 files. A 12-step deployment runbook turns 'works on my laptop' into 'runs unattended on production'. The 24-hour ops report is what you read every morning to know the system is still doing the job.

deployment runbook, systemctl status, journalctl, deployment health check, 24-hour ops report, cost-vs-value, line count summary, production journey

Lab Cost Optimization Challenge

Cut a $50/day AI pipeline to under $10/day without dropping accuracy below 85% — profile first, then add response caching, compress prompts, route to cheaper model tiers, and verify accuracy holds on a held-out test set.

Covers: cost profiling, response caching with TTL, prompt compression, model tier routing, regression testing, accuracy verification

Exam Level 5 Master Exam

Prove you've internalized production AI engineering. 10 questions covering testing (unit/integration/hallucination/regression), deployment (setup.py + systemd), scheduling (cron + flock), pipelines (error propagation), batch processing (ThreadPoolExecutor), monitoring (JSONL + jq), and cost optimization (caching + compression + routing). Score 8 of 10 to pass.

Covers: pytest, unittest.mock, integration tests, hallucination tests, regression tests, setup.py, entry_points, pip install -e, systemd unit, cron, flock, timestamped output, pipeline.py, conditional routing, error propagation, ThreadPoolExecutor, JSONL monitoring, jq, alerts, cost spike detection, hash-based caching with TTL, prompt compression, model routing (Haiku/Sonnet/Opus)

Cards Level 5 Flashcards

Drill every command and concept from Level 5. Cards pull from this level's quizzes and exam, and the ones you miss come back more often.

Build agent.py from a 20-line loop to a 150-line autonomous agent. Observe-decide-act loop (T41), tool dispatch + multi-tool chains (T42), SQLite memory with store and recall (T43), multi-step planning + replanning on unexpected results (T44), guardrails — rate limits, approval gates, audit logging, graceful shutdown, domain boundaries (T45), and a 24-hour agent simulation capstone (T46).

T41 Your First Agent (+Quiz)

Build a complete agent in three functions and one loop. observe() reads the latest log line, decide() sends it to an LLM for triage, act() executes the decision. Run it against 8 events and watch the model ignore routine entries, note warnings, and escalate errors.

observe, decide, act, agent.py, events.json, requests.post, json.loads, Ollama API

T42 Tool Use (+Quiz)

Give the agent tools it can request by name. Define read_log, check_process, and list_files as Python functions, add a dispatch dict, and build a tool-call loop that executes what the LLM asks for and feeds results back until it reaches a decision.

TOOLS, DISPATCH, read_log, check_process, list_files, tool-dispatch loop, multi-tool chaining, JSON tool requests

T43 Memory (+Quiz)

Add memory so the agent stops re-investigating problems it already solved. SQLite stores resolved incidents across runs; session history keeps conversation context within a run. trim_history summarizes old messages before the context window overflows.

sqlite3, Memory class, store, recall, LIKE, session history, trim_history, MAX_HISTORY, query_database tool

T44 Multi-Step Reasoning (+Quiz)

Move from single-step reactions to multi-step investigation plans. The agent asks the LLM for numbered steps, executes each as a tool call, and replans when a step returns unexpected information. Memory-informed planning skips known incidents entirely.

plan_and_execute, numbered steps, replan, revised_plan, memory.recall, memory-informed planning

Lab Autonomous Log Analyzer

Run your agent against unseen events and verify: crash-free completion, incident detection, memory recall, rate limit compliance, and accurate daily report.

Covers: autonomous agent operation, output verification, audit log inspection, memory recall, rate limit compliance

T45 Guardrails (+Quiz)

Make the agent safe. Build a Guardrails class with rate limiting on tool calls, token budget tracking, human approval gates for dangerous actions like send_alert, and timestamped audit logging. Wrap the main loop in try/finally for graceful shutdown.

Guardrails, check_rate, track_cost, require_approval, log_action, DANGEROUS_TOOLS, audit.log, TOKEN_BUDGET, MAX_TOOL_CALLS, try/finally, KeyboardInterrupt

T46 Capstone — Full Integration (+Quiz)

Run every L6 concept against a 24-hour simulation. 24 events, one per hour — routine logs, investigations, approval gates, memory recall, and a daily operations report. One command, clean slate, full integration.

24-hour simulation, daily_report.txt, memory recall, approval gate, audit log verification, incident summary, tool call budget

Lab Agent Handoff

Run agent against events including a security event it can't handle, verify the handoff protocol, run the security agent, then wire your agent to read resolutions.

Covers: agent boundaries, structured handoff, domain recognition, cross-agent resolution, memory persistence across runs

Exam Level 6 Master Exam

10 questions covering agentic AI patterns: tool dispatch, memory, planning, safety, and lifecycle. Score 8 of 10 to pass.

Covers: observe-decide-act loop, tool dispatch, multi-tool chains, JSON parsing, SQLite memory, session trimming, multi-step planning, replanning, rate limiting, token budget, approval gates, audit logging, graceful shutdown, domain boundaries

Cards Level 6 Flashcards

Drill every command and concept from Level 6. Cards pull from this level's quizzes and exam, and the ones you miss come back more often.

Build ml_pipeline.py from a 15-line data loader to a 120-line training and evaluation pipeline. pandas exploration + class imbalance analysis (T47), LabelEncoder + train_test_split (T48), DecisionTreeClassifier + confusion matrix + classification_report (T49), RandomForestClassifier + feature importance + overfitting checks (T50), regression vs classification + target leakage detection (T51), full pipeline comparison — sklearn vs Claude zero-shot on the same ticket dataset (T52).

T47 Data and First Look (+Quiz)

Load the ticket dataset with pandas, inspect shape, column types, and priority distribution. Spot the class imbalance — 198 low tickets vs 53 critical — before training anything.

pandas, pd.read_csv, DataFrame, .shape, .dtypes, .value_counts, .describe, class imbalance, data/tickets.csv

T48 Train/Test Split and Your First Model (+Quiz)

Engineer features from raw columns — LabelEncoder for categories, word counts, urgent-keyword flags — then split 67.5/32.5 train/test and fit a DecisionTreeClassifier. First accuracy number: 0.78.

LabelEncoder, fit_transform, feature_cols, train_test_split, test_size, random_state, DecisionTreeClassifier, .fit, .predict, .score

T49 Evaluation Metrics (Accuracy Is a Lie) (+Quiz)

Prove that 78% accuracy is a lie. The confusion matrix shows the model misses 12 of 18 critical tickets — 0.33 recall. Learn to read precision, recall, and F1 per class and know which metric matters for your problem.

confusion_matrix, classification_report, precision, recall, f1-score, support, macro avg, weighted avg

T50 Better Models (+Quiz)

Replace the single DecisionTree with a RandomForestClassifier. 100 trees voting together cut the overfitting gap and lifted critical recall from 0.33 to 0.72. Compare train vs test accuracy to detect memorization, and read feature importances to see what drives predictions.

RandomForestClassifier, n_estimators, overfitting, train vs test accuracy, feature_importances_, show_feature_importance

Lab Incident Classifier

Apply the L7 ML pipeline to a new incident dataset. Train a RandomForest, read the classification report, and reason about what the numbers mean for production.

Covers: class imbalance detection, RandomForest training, classification report interpretation, overfitting diagnosis, recall-based risk assessment

T51 Regression (+Quiz)

Switch from classification to regression — predict resolution time in hours instead of priority label. Drop the target from the feature set to avoid leakage, fit a LinearRegression, and evaluate with MSE, RMSE, and R².

LinearRegression, mean_squared_error, r2_score, RMSE, target leakage, prepare_regression_features, classification vs regression

T52 Full Pipeline and the Claude Comparison (+Quiz)

Wire every stage into run_pipeline() and compare sklearn to Claude's zero-shot classification from L3. RandomForest wins on accuracy (0.86 vs 0.72) but Claude catches more critical tickets (0.78 vs 0.72 recall). Neither is the whole answer — that's L8.

run_pipeline, compare_to_api, save_results, results/ml_results.json, sklearn vs Claude, cost/1K comparison

Lab Model Autopsy

Diagnose three pre-built models by reading their evaluation reports. Identify overfitting, majority-class collapse, and proper evaluation.

Covers: evaluation methodology, test set size, majority-class prediction, overfitting detection, deployment risk assessment

Exam Level 7 Master Exam

10 questions covering ML foundations: pandas exploration, class imbalance, encoding, train/test splitting, decision trees, confusion matrices, random forests, and model evaluation. Score 8 of 10 to pass.

Covers: pandas data exploration, class imbalance, LabelEncoder, train_test_split, DecisionTreeClassifier, confusion matrix, classification_report, precision vs recall, accuracy limitations, RandomForestClassifier, overfitting, feature importance, regression vs classification

Cards Level 7 Flashcards

Drill every command and concept from Level 7. Cards pull from this level's quizzes and exam, and the ones you miss come back more often.

Build neural_net.py from a 15-line tensor loader to a 100-line training and evaluation pipeline. Tensor conversion + torch.Size (T53), TicketClassifier with nn.Linear + ReLU (T54), training loop + learning rate tuning + NN-vs-RF comparison (T55), pre-trained sentence transformer embeddings (T56), fine-tuning via API with JSONL training data (T57), and the decision framework — when to use each approach (T58).

T53 Neural Networks and Tensors (+Quiz)

Convert L7's features into PyTorch tensors — same 5 features, same 338/162 split, new format. FloatTensor for features, LongTensor for labels. Tensor arithmetic is the foundation for everything the network does.

torch, torch.FloatTensor, torch.LongTensor, tensor indexing, tensor arithmetic, .shape, neural_net.py

T54 Building a Network (+Quiz)

Build a TicketClassifier with nn.Module — three Linear layers, ReLU activation, 5 inputs to 16 hidden neurons to 4 output classes. 436 parameters, all random. Run untrained predictions and confirm the network is guessing.

nn.Module, nn.Linear, nn.ReLU, __init__, forward, TicketClassifier, total parameters, untrained predictions

T55 The Training Loop (+Quiz)

Train the network with the five-line loop: zero_grad, forward, loss, backward, step. CrossEntropyLoss measures error, Adam adjusts weights. Compare learning rates (0.001 too slow, 0.1 explodes, 0.01 converges). Result: 82% — beats DecisionTree, loses to RandomForest.

nn.CrossEntropyLoss, torch.optim.Adam, zero_grad, backward, optimizer.step, learning rate, epochs, torch.no_grad, argmax, classification_report

T56 Pre-trained Models (+Quiz)

Use a pre-trained embedding model to turn ticket text into 384-dimensional vectors that capture meaning, not just word counts. Train a LogisticRegression on those vectors. 88% accuracy, 0.78 critical recall — first model to beat RandomForest.

text embeddings, embed_tickets, cosine similarity, 384 dimensions, LogisticRegression, pre-trained model, feature extraction

Lab Training Diagnostics

Diagnose three neural network training runs by reading their logs and configs. Identify exploding gradients, overfitting, and healthy training.

Covers: learning rate tuning, exploding gradients, overfitting diagnosis, parameter counting, train-test gap analysis

T57 Fine-Tuning via API (+Quiz)

Fine-tune a pre-trained model on your labeled tickets via the three-step API: submit JSONL training data, poll for completion, evaluate predictions. 91% accuracy, 0.83 critical recall — best result in the course.

JSONL, prepare_training_data, submit_finetune, check_finetune_status, evaluate_finetuned, fine-tuning API, model_id

T58 The Decision Framework (+Quiz)

No new code — just judgment. Compare all six approaches side by side (accuracy, critical recall, cost, latency, text understanding) and learn the four-question decision framework for picking the right one given your constraints.

compare_all_approaches, decision framework, model selection, cost/accuracy tradeoff, data drift, results/comparison.json

Lab Choose the Right Model

Apply the L8 decision framework to real deployment scenarios. Match models to constraints: budget, latency, new categories, and drift.

Covers: model selection, cost-accuracy trade-offs, critical recall, zero-shot flexibility, data drift awareness, combined deployment

Exam Level 8 Master Exam

10 questions covering neural networks, training loops, embeddings, fine-tuning, and the decision framework for choosing ML approaches. Score 8 of 10 to pass.

Covers: tensor conversion, torch.Size, nn.Module architecture, parameter counting, untrained predictions, training loop, CrossEntropyLoss, Adam optimizer, learning rate effects, NN vs RF on structured data, pre-trained embeddings, fine-tuning via API, decision framework, cost-latency tradeoffs

Cards Level 8 Flashcards

Drill every command and concept from Level 8. Cards pull from this level's quizzes and exam, and the ones you miss come back more often.

Build model_server.py from a 15-line joblib script to a 100-line deployment pipeline. Model persistence + version metadata (T59), Flask POST /predict + CATEGORY_MAP + prepare_features_single (T60), batch /batch endpoint (T61), JSONL prediction logging + input hashing for privacy (T62), drift detection against training baseline (T63), retrain-compare-swap pipeline + keep-current guardrail (T64), and full lifecycle integration with systemd deployment (T65).

T59 Save and Load Models (+Quiz)

Save a trained model to disk with joblib and load it back without retraining. Version each retrain (model_v1.pkl, model_v2.pkl) and track accuracy, data source, and timestamp in metadata.json so you can roll back.

joblib.dump, joblib.load, models/metadata.json, model versioning, save_model, load_model, model_server.py

T60 Flask Predict Endpoint (+Quiz)

Wrap the saved model in a Flask server. Load once at startup, serve predictions on POST /predict with input validation, confidence scores from predict_proba, and sub-5ms latency.

Flask, create_app, /predict, /health, prepare_features_single, predict_proba, confidence, request validation, jsonify

T61 Batch Prediction (+Quiz)

Add a /batch endpoint that accepts a CSV and returns predictions for every row in one vectorized model.predict call. 50 tickets in 45ms instead of 150ms from individual calls.

/batch, prepare_features_batch, pd.read_csv, model.predict, vectorized prediction, summary counts, batch latency

T62 Prediction Logging (+Quiz)

Log every prediction to JSONL — timestamp, input hash, prediction, confidence, model version, latency. MD5 hashing identifies and deduplicates tickets without storing full text. These logs feed drift detection.

log_prediction, logs/predictions.jsonl, hashlib.md5, input_hash, JSONL append, prediction aggregation

Lab Log Analysis

Analyze 14 days of JSONL prediction logs to find total volume, model version distribution, the swap point, latency impact, and post-swap stability.

Covers: JSONL log parsing, model version tracking, drift detection, latency comparison, post-deployment monitoring

T63 Drift Detection (+Quiz)

Detect when the model goes stale. Compare the training-time priority distribution to recent predictions — if the divergence exceeds 0.15, flag drift. Month 2 data shifts the distribution and drops v1 accuracy from 0.86 to 0.71.

get_training_distribution, get_recent_distribution, check_drift, divergence, threshold, tickets_month2.csv, distribution comparison

T64 Retraining Pipeline (+Quiz)

Retrain on new data, compare the new model to the current one on the same test set, and swap only if accuracy improves. The guardrail prevents deploying a worse model when training data is corrupted.

retrain_model, compare_models, swap_if_better, accuracy_score, model swap guardrail, v1 vs v2, rollback

T65 Full Integration (+Quiz)

Wire serve, log, detect, and retrain into one lifecycle. A four-phase demo runs normal operation, detects drift, retrains automatically, and stabilizes — the model self-heals from 0.71 back to 0.84. Deploy with systemd.

lifecycle demo, /status, four-phase simulation, self-healing model, systemd deployment, model_server.service, serve → log → drift → retrain

Lab Production Incident

Investigate a production accuracy drop by examining drift reports, weekend ticket data, prediction logs, and the feature preparation code.

Covers: production debugging, data drift investigation, category_map limitations, code vs model fixes, zero-shot as fallback

Exam Level 9 Master Exam

10 questions covering MLOps: model serialization, Flask serving, prediction logging, drift detection, retrain-compare-swap pipeline, and deployment lifecycle. Score 8 of 10 to pass.

Covers: joblib save/load, model versioning, metadata.json, Flask /predict, CATEGORY_MAP, prepare_features_single, predict_proba confidence, batch endpoint, JSONL logging, input hashing, drift detection, distribution shift, retrain-compare-swap, guardrails, systemd deployment, lifecycle integration

Cards Level 9 Flashcards

Drill every command and concept from Level 9. Cards pull from this level's quizzes and exam, and the ones you miss come back more often.

Capstone projects combining everything from Levels 1-5. Non-counting toward the course completion percentage; bonus material for graduates who want to prove they can integrate the whole stack. Lab A: cross-host security analysis with hallucination validation. Lab B: a self-healing infrastructure agent that knows when to escalate. Lab C: an incident correlation engine that distinguishes coincidence from correlation. Lab D: take your best tool and ship it production-grade with all the L5 hardening — then run it for 24 simulated hours and prove the operational metrics hit target.

Lab Master Lab A: Automated Security Report Generator

Pull 24-hour auth logs from three production servers, run AI analysis on each, correlate findings across hosts (the same attacker IP appearing on multiple servers), and produce both an executive briefing and a hallucination-validated technical appendix.

Covers: multi-host log analysis, cross-host correlation, hallucination validation, executive briefing, technical appendix, ground-truth checking

Lab Master Lab B: Self-Healing Infrastructure Agent

Build the self-healing agent loop — watch system state, ask Claude what to do, execute whitelisted tools, escalate when uncertain, and audit-log every action. Tested against five known scenarios with pass/fail criteria.

Covers: agent loops, whitelisted tools, structured reasoning, audit logging, escalation patterns, scenario-based testing, destructive-action gating

Lab Master Lab C: Incident Correlation Engine

Three monitoring tools fired alerts in the same hour. Build a correlation engine that ingests Prometheus, OSSEC, and nginx events, groups them into incidents, distinguishes coincidence from genuine causal chains, and proposes a defensible root-cause hypothesis per incident.

Covers: multi-source ingestion, event normalization, AI correlation, root cause hypothesis, false-correlation detection, ground-truth validation, incident timelines

Lab Master Lab D: Full-Stack Tool Deployment

Take your best tool from the course and ship it production-grade — caching with TTL, full test suite, structured logging, cost tracking, CLI packaging, systemd unit. Run it for 24 simulated hours and report on operational metrics (cache-hit rate, error rate, p95/p99 latency, total cost).

Covers: full-stack production deployment, caching with TTL, retry + fallback, structured logging, cost tracking, CLI packaging, systemd, 24-hour operations report, p95/p99 latency

AI Engineering & Automation

Full Curriculum

Where these skills take you

Start building AI skills