Lesson 6: Agent Evaluation & Benchmarks — Full Spoken Transcript (Cantonese)

Original podcast: Cantonese dialogue between two speakers (M = 主持 host 子謙, F = 嘉賓 expert 曉晴). Total spoken duration: ~13 min. The audio above plays the full conversation. The text below is the complete transcript of what was actually said — preserved verbatim, not summarised, not translated.

Course: Eight-lesson course on AI agents · Lesson 6 of 8 · learnagent.lmmlab.com Topic (EN): SWE-bench · AgentBench · Terminal-Bench · GAIA · offline replay · LLM-as-judge. Topic (粵): SWE-bench、AgentBench、Terminal-Bench 等。 Speakers: 主持 M (host 子謙) and 嘉賓 F (expert 曉晴) · 23 spoken turns · ~13 min audio.

How to read this transcript

Each spoken turn from the podcast is shown as a separate paragraph, with the speaker label rendered in bold and an approximate timestamp (MM:SS) indicating where in the audio that turn occurs. Long turns are broken at natural sentence boundaries (。！？) and, where a sentence is still long, at clause boundaries (，；、). No English translation is provided — this is the exact spoken Cantonese audio transcript as it was synthesised for the lesson podcast.

Use browser Ctrl+F / ⌘+F to locate any phrase. On mobile (640 px viewport), the transcript scrolls as a single column.

Note: Sentence-level line breaks are for readability — the Cantonese text itself is preserved verbatim from the original podcast script (script_raw.json). No words have been altered, translated, summarised, or paraphrased.

Index of sections in this lesson

• 1. Opening & SWE-bench (粵: 開場同 SWE-bench) — turn 01 onwards
• 2. AgentBench · Terminal-Bench · GAIA (粵: AgentBench、Terminal-Bench、GAIA) — turn 05 onwards
• 3. Offline Replay & LLM-as-Judge (粵: Offline Replay 同 LLM-as-Judge) — turn 11 onwards
• 4. Production Eval Pipeline (粵: Production Eval Pipeline) — turn 17 onwards
• 5. Wrap-up & Safety Preview (粵: 總結同 Safety 預覽) — turn 22 onwards

Section 1/5 — Opening & SWE-bench

開場同 SWE-bench

Section overview: covers turns 01–04 (4 spoken segments).

Topic terms (extracted from spoken text): human-verified, infrastructure, reproducible, long-horizon, Engineering, correctness, Multi-Modal, scaled-down, description, demonstrate

Latin/English code-terms in this section (verbatim from speech): human-verified, infrastructure, reproducible

Section character total: 1,067 characters across 4 spoken turns.

Section duration estimate: ~2:15 of 13:00 total.

Turns in this section: 01, 02, 03, 04.

First spoken sentence of this section (turn 01, verbatim): 各位同學早晨, 我係子謙。

Average characters per turn (this section): ~266 chars.

Cumulative characters through this section: 1,067 of 7,319 total.

[01 | 00:00] 主持 M (host 子謙):

Turn 1 of 23 · speaker M (host 子謙 — opens and closes) · audio timestamp 00:00 · section 1 (Opening & SWE-bench)

speaker=M · chars=164 · ts=00:00 · terms=reproducible, Evaluation, Benchmarks, regression, benchmark · sentences=3 · clauses=6

Verbatim phrases in this turn: 你嘅 agent 可以 reproducible 嘅測試同 regression detection。

各位同學早晨, 我係子謙。歡迎收聽第六課。

First clause (verbatim): 各位同學早晨,

Last clause (verbatim): 同埋點樣 design custom eval suite 等到你嘅 agent 可以 reproducible 嘅測試同 regression detection。

各位同學早晨,

我係子謙。

歡迎收聽第六課。

今日嘅主題係 Agent Evaluation 同 Benchmarks,

包括主流 benchmark 嘅具體內容,

同埋點樣 design custom eval suite 等到你嘅 agent 可以 reproducible 嘅測試同 regression detection。

[02 | 00:33] 嘉賓 F (expert 曉晴):

Turn 2 of 23 · speaker F (expert 曉晴 — explains concepts and trade-offs) · audio timestamp 00:33 · section 1 (Opening & SWE-bench)

speaker=F · chars=232 · ts=00:33 · terms=long-horizon, completion, benchmark, HumanEval, response · sentences=3 · clauses=10

Verbatim phrases in this turn: LLM 嘅傳統 benchmark 例如 MMLU、 · nt 嘅 quality 仲包括 tool use 嘅 success rate、 · long-horizon task completion、

大家好, 我係曉晴。今日嘅問題係, agent 嘅 quality 點樣 measure。

First clause (verbatim): 大家好,

Last clause (verbatim): 所以需要專門嘅 agent benchmark。

大家好,

我係曉晴。

今日嘅問題係,

agent 嘅 quality 點樣 measure。

LLM 嘅傳統 benchmark 例如 MMLU、

HumanEval 主要 measure 單次 response 嘅 quality,

但 agent 嘅 quality 仲包括 tool use 嘅 success rate、

long-horizon task completion、

failure recovery,

所以需要專門嘅 agent benchmark。

[03 | 01:07] 主持 M (host 子謙):

Turn 3 of 23 · speaker M (host 子謙 — opens and closes) · audio timestamp 01:07 · section 1 (Opening & SWE-bench)

speaker=M · chars=318 · ts=01:07 · terms=human-verified, infrastructure, Engineering, correctness, Multi-Modal · sentences=4 · clauses=8

Verbatim phrases in this turn: ts, agent 嘅 task 係 modify codebase 去 resolve issue。 · 用真實嘅 testing infrastructure 評估 patch 嘅 correctness。 · ulti-Modal 同 SWE-bench Lite 係 scaled-down variants。

真實 GitHub issues 同 pull requests, agent 嘅 task 係 modify codebase 去 resolve issue。

First clause (verbatim): 首先講 SWE-bench。

Last clause (verbatim): SWE-bench Multi-Modal 同 SWE-bench Lite 係 scaled-down variants。

首先講 SWE-bench。

SWE-bench 係 Software Engineering benchmark,

source 係真實 GitHub issues 同 pull requests,

agent 嘅 task 係 modify codebase 去 resolve issue。

SWE-bench Verified 係 human-verified subset,

一千個 instance,

用真實嘅 testing infrastructure 評估 patch 嘅 correctness。

SWE-bench Multi-Modal 同 SWE-bench Lite 係 scaled-down variants。

[04 | 01:41] 嘉賓 F (expert 曉晴):

Turn 4 of 23 · speaker F (expert 曉晴 — explains concepts and trade-offs) · audio timestamp 01:41 · section 1 (Opening & SWE-bench)

speaker=F · chars=353 · ts=01:41 · terms=engineering, description, demonstrate, evaluation, percentage · sentences=3 · clauses=9

Verbatim phrases in this turn: gineering work 嘅 gap, 例如 missing context、 · vague issue description、 · patch 通過所有 hidden unit tests 嘅 instance percentage。

ass rate, 即係 agent generate 嘅 patch 通過所有 hidden unit tests 嘅 instance percentage。

First clause (verbatim): SWE-bench 嘅 evaluation metric 係 pass rate,

Last clause (verbatim): 呢啲令 agent 唔可以 fully demonstrate capability。

SWE-bench 嘅 evaluation metric 係 pass rate,

即係 agent generate 嘅 patch 通過所有 hidden unit tests 嘅 instance percentage。

最頂級嘅 frontier agent 2026 年達到大約六成到七成嘅 pass rate,

比 2024 年嘅十幾個 percent 大幅提升。

但 SWE-bench 嘅 criticism 包括 instance 同真實 engineering work 嘅 gap,

例如 missing context、

vague issue description、

dependency 缺失,

呢啲令 agent 唔可以 fully demonstrate capability。

End-of-section recap (last spoken sentence of Opening & SWE-bench): SWE-bench 嘅 evaluation metric 係 pass rate, 即係 agent generate 嘅 patch 通過所有 hidden unit tests 嘅 instance percentage。

Section 2/5 — AgentBench · Terminal-Bench · GAIA

AgentBench、Terminal-Bench、GAIA

Section overview: covers turns 05–10 (6 spoken segments).

Topic terms (extracted from spoken text): time-to-completion, Multi-SWE-bench, single-language, reproducibility, multi-language, Terminal-Bench, terminal-based, administration, containerized, reinforcement

Latin/English code-terms in this section (verbatim from speech): time-to-completion, Multi-SWE-bench, single-language

Section character total: 2,077 characters across 6 spoken turns.

Section duration estimate: ~3:23 of 13:00 total.

Turns in this section: 05, 06, 07, 08, 09, 10.

First spoken sentence of this section (turn 05, verbatim): SWE-bench 嘅 variant 包括 SWE-bench Pro, 2025 年推出, 更加 challenging, 加入 containerized environment 同 longer task horizon。

Average characters per turn (this section): ~346 chars.

Cumulative characters through this section: 3,144 of 7,319 total.

[05 | 02:15] 主持 M (host 子謙):

Turn 5 of 23 · speaker M (host 子謙 — opens and closes) · audio timestamp 02:15 · section 2 (AgentBench · Terminal-Bench · GAIA)

speaker=M · chars=333 · ts=02:15 · terms=Multi-SWE-bench, single-language, multi-language, containerized, reinforcement · sentences=3 · clauses=12

Verbatim phrases in this turn: uage SWE-bench 擴展到 multi-language, 例如 Go、 · 加入 containerized environment 同 longer task horizon。 · , 強調 reinforcement learning training 嘅 environment。

ro, 2025 年推出, 更加 challenging, 加入 containerized environment 同 longer task horizon。

First clause (verbatim): SWE-bench 嘅 variant 包括 SWE-bench Pro,

Last clause (verbatim): 反映 enterprise codebase 嘅真實 multi-language nature。

SWE-bench 嘅 variant 包括 SWE-bench Pro,

2025 年推出,

更加 challenging,

加入 containerized environment 同 longer task horizon。

R2E 同 SWE-Gym 係 related benchmark,

強調 reinforcement learning training 嘅 environment。

Multi-SWE-bench 將 single-language SWE-bench 擴展到 multi-language,

例如 Go、

Rust、

Java、

C++,

反映 enterprise codebase 嘅真實 multi-language nature。

[06 | 02:49] 嘉賓 F (expert 曉晴):

Turn 6 of 23 · speaker F (expert 曉晴 — explains concepts and trade-offs) · audio timestamp 02:49 · section 2 (AgentBench · Terminal-Bench · GAIA)

speaker=F · chars=334 · ts=02:49 · terms=comprehensive, environments, interaction, AgentBench, evaluation · sentences=3 · clauses=13

Verbatim phrases in this turn: te, 包括八個 environments, 例如 OS interaction、 · database operation、 · web shopping、

第二個 major benchmark 係 AgentBench。

First clause (verbatim): 第二個 major benchmark 係 AgentBench。

Last clause (verbatim): 唔似 SWE-bench 集中 coding。

第二個 major benchmark 係 AgentBench。

AgentBench 係 comprehensive agent evaluation suite,

包括八個 environments,

例如 OS interaction、

database operation、

web shopping、

knowledge graph reasoning、

household tasks、

web browsing、

maze navigation、

card game。

AgentBench 嘅設計目的是 measure agent 嘅 general capability across diverse domains,

唔似 SWE-bench 集中 coding。

[07 | 03:23] 主持 M (host 子謙):

Turn 7 of 23 · speaker M (host 子謙 — opens and closes) · audio timestamp 03:23 · section 2 (AgentBench · Terminal-Bench · GAIA)

speaker=M · chars=301 · ts=03:23 · terms=reproducibility, environment, variability, AgentBench, evaluation · sentences=3 · clauses=7

Verbatim phrases in this turn: ess rate 同 steps to success, 即係平均需要幾多 step 完成 task。 · nment 嘅 real-time content 可能改變, 影響 reproducibility。 · tup 嘅 complexity, run benchmark 嘅 infra overhead 高。

ch 嘅 evaluation metric 包括 success rate 同 steps to success, 即係平均需要幾多 step 完成 task。

First clause (verbatim): AgentBench 嘅 evaluation metric 包括 success rate 同 steps to success,

Last clause (verbatim): run benchmark 嘅 infra overhead 高。

AgentBench 嘅 evaluation metric 包括 success rate 同 steps to success,

即係平均需要幾多 step 完成 task。

AgentBench 嘅 challenge 包括 environment state 嘅 variability,

例如 web browsing environment 嘅 real-time content 可能改變,

影響 reproducibility。

AgentBench 嘅 criticism 係環境 setup 嘅 complexity,

run benchmark 嘅 infra overhead 高。

[08 | 03:57] 嘉賓 F (expert 曉晴):

Turn 8 of 23 · speaker F (expert 曉晴 — explains concepts and trade-offs) · audio timestamp 03:57 · section 2 (AgentBench · Terminal-Bench · GAIA)

speaker=F · chars=393 · ts=03:57 · terms=Terminal-Bench, terminal-based, administration, command-line, manipulation · sentences=5 · clauses=11

Verbatim phrases in this turn: al-Bench 2.0 嘅 task 包括 file manipulation、 · system administration、 · build system interaction、

第三個 major benchmark 係 Terminal-Bench。

First clause (verbatim): 第三個 major benchmark 係 Terminal-Bench。

Last clause (verbatim): Terminal-Bench 對 command-line proficient user 嘅 workflow 特別 relevant。

第三個 major benchmark 係 Terminal-Bench。

Terminal-Bench 係 terminal-based agent benchmark,

focus 喺 command-line environment 同 shell interaction。

Terminal-Bench 2.0 嘅 task 包括 file manipulation、

system administration、

debug、

build system interaction、

package management。

Terminal-Bench 嘅 evaluation 係 pass rate,

用 hidden test 驗證 task 完成。

Terminal-Bench 對 command-line proficient user 嘅 workflow 特別 relevant。

[09 | 04:31] 主持 M (host 子謙):

Turn 9 of 23 · speaker M (host 子謙 — opens and closes) · audio timestamp 04:31 · section 2 (AgentBench · Terminal-Bench · GAIA)

speaker=M · chars=332 · ts=04:31 · terms=time-to-completion, Terminal-Bench, long-horizon, interaction, consumption · sentences=3 · clauses=5

Verbatim phrases in this turn: ur debugging session 同 complex deployment workflow。 · onsumption, 即係 measure efficiency 唔單止 success rate。 · coding agent 嘅 real-world relevance 比 HumanEval 更高。

long-horizon task, 例如 multi-hour debugging session 同 complex deployment workflow。

First clause (verbatim): Terminal-Bench 嘅 2026 update 加入咗 multi-turn interaction 同 long-horizon task,

Last clause (verbatim): 業界普遍同意 Terminal-Bench 對 coding agent 嘅 real-world relevance 比 HumanEval 更高。

Terminal-Bench 嘅 2026 update 加入咗 multi-turn interaction 同 long-horizon task,

例如 multi-hour debugging session 同 complex deployment workflow。

Terminal-Bench 嘅 metric 包括 total time-to-completion 同 total token consumption,

即係 measure efficiency 唔單止 success rate。

業界普遍同意 Terminal-Bench 對 coding agent 嘅 real-world relevance 比 HumanEval 更高。

[10 | 05:05] 嘉賓 F (expert 曉晴):

Turn 10 of 23 · speaker F (expert 曉晴 — explains concepts and trade-offs) · audio timestamp 05:05 · section 2 (AgentBench · Terminal-Bench · GAIA)

speaker=F · chars=384 · ts=05:05 · terms=development, environment, correctness, e-commerce, management · sentences=5 · clauses=14

Verbatim phrases in this turn: a 係 web agent benchmark, 模擬真實 e-commerce、 · social forum、 · software development、

第四個 benchmark 係 WebArena。

First clause (verbatim): 第四個 benchmark 係 WebArena。

Last clause (verbatim): WebArena 嘅挑戰包括 web environment 嘅 dynamic content 同 anti-bot protection。

第四個 benchmark 係 WebArena。

WebArena 係 web agent benchmark,

模擬真實 e-commerce、

social forum、

software development、

content management、

map 等 web environment。

Agent 嘅 task 包括 product search、

form filling、

navigation、

content creation。

WebArena 嘅 evaluation 係 task success rate 同 functional correctness,

例如最後 cart 入面有正確嘅 product。

WebArena 嘅挑戰包括 web environment 嘅 dynamic content 同 anti-bot protection。

End-of-section recap (last spoken sentence of AgentBench · Terminal-Bench · GAIA): 第四個 benchmark 係 WebArena。

Section 3/5 — Offline Replay & LLM-as-Judge

Offline Replay 同 LLM-as-Judge

Section overview: covers turns 11–16 (6 spoken segments).

Topic terms (extracted from spoken text): tool-augmented, representative, snapshot-based, deterministic, containerized, reproducible, dependencies, instruction, Environment, adversarial

Latin/English code-terms in this section (verbatim from speech): tool-augmented, representative, snapshot-based

Section character total: 1,880 characters across 6 spoken turns.

Section duration estimate: ~3:23 of 13:00 total.

Turns in this section: 11, 12, 13, 14, 15, 16.

First spoken sentence of this section (turn 11, verbatim): 第五個 benchmark 係 ToolBench。

Average characters per turn (this section): ~313 chars.

Cumulative characters through this section: 5,024 of 7,319 total.

[11 | 05:39] 主持 M (host 子謙):

Turn 11 of 23 · speaker M (host 子謙 — opens and closes) · audio timestamp 05:39 · section 3 (Offline Replay & LLM-as-Judge)

speaker=M · chars=339 · ts=05:39 · terms=tool-augmented, instruction, parameters, evaluation, capability · sentences=5 · clauses=7

Verbatim phrases in this turn: 第五個 benchmark 係 ToolBench。 · rk, focus 喺 instruction following 同 tool selection。 · gent 需要 identify correct tool 同 correct parameters。

第五個 benchmark 係 ToolBench。

First clause (verbatim): 第五個 benchmark 係 ToolBench。

Last clause (verbatim): ToolBench 對 tool-augmented LLM 嘅 capability 提供 systematic measure。

第五個 benchmark 係 ToolBench。

ToolBench 係 tool-use agent benchmark,

focus 喺 instruction following 同 tool selection。

ToolBench 嘅 task 設計係真實 user query,

agent 需要 identify correct tool 同 correct parameters。

ToolBench 嘅 evaluation 包括 tool selection accuracy 同 execution success rate。

ToolBench 對 tool-augmented LLM 嘅 capability 提供 systematic measure。

[12 | 06:13] 嘉賓 F (expert 曉晴):

Turn 12 of 23 · speaker F (expert 曉晴 — explains concepts and trade-offs) · audio timestamp 06:13 · section 3 (Offline Replay & LLM-as-Judge)

speaker=F · chars=273 · ts=06:13 · terms=generation, capability, HumanEval+, benchmark, HumanEval · sentences=3 · clauses=7

Verbatim phrases in this turn: ion capability, 對 agent 嘅 code writing sub-task 有關。 · pass@k, 即係 generate k 個 solution 入面有冇至少一個通過所有 test。 · HumanEval+ 同 MBPP+ 係 harder variants。

但係 measure LLM 嘅 code generation capability, 對 agent 嘅 code writing sub-task 有關。

First clause (verbatim): 第六個 benchmark 唔可以唔講嘅係 HumanEval 同 MBPP,

Last clause (verbatim): HumanEval+ 同 MBPP+ 係 harder variants。

第六個 benchmark 唔可以唔講嘅係 HumanEval 同 MBPP,

雖然呢啲係傳統 code benchmark 而唔係 agent benchmark,

但係 measure LLM 嘅 code generation capability,

對 agent 嘅 code writing sub-task 有關。

HumanEval 嘅 metric 係 pass@k,

即係 generate k 個 solution 入面有冇至少一個通過所有 test。

HumanEval+ 同 MBPP+ 係 harder variants。

[13 | 06:46] 主持 M (host 子謙):

Turn 13 of 23 · speaker M (host 子謙 — opens and closes) · audio timestamp 06:46 · section 3 (Offline Replay & LLM-as-Judge)

speaker=M · chars=283 · ts=06:46 · terms=reproducible, evaluation, measurable, represent, contrived · sentences=5 · clauses=15

Verbatim phrases in this turn: ion metric 要 measurable, 例如 success rate、 · Custom eval suite 嘅設計原則。 · represent 真實 user task, 唔係 contrived academic task。

Custom eval suite 嘅設計原則。

First clause (verbatim): Custom eval suite 嘅設計原則。

Last clause (verbatim): 包括 error recovery。

Custom eval suite 嘅設計原則。

第一,

evaluation 要 represent 真實 user task,

唔係 contrived academic task。

第二,

evaluation 環境要 reproducible,

每次 run 都 same initial state。

第三,

evaluation metric 要 measurable,

例如 success rate、

latency、

cost。

第四,

evaluation 要 cover happy path 同 edge case,

包括 error recovery。

[14 | 07:20] 嘉賓 F (expert 曉晴):

Turn 14 of 23 · speaker F (expert 曉晴 — explains concepts and trade-offs) · audio timestamp 07:20 · section 3 (Offline Replay & LLM-as-Judge)

speaker=F · chars=319 · ts=07:20 · terms=representative, deterministic, Environment, simulation, filesystem · sentences=4 · clauses=11

Verbatim phrases in this turn: tic simulation 嘅 environment 例如 database、 · filesystem、 · tive task, 每個 task 有 ground truth expected outcome。

corpus, 即係 list of representative task, 每個 task 有 ground truth expected outcome。

First clause (verbatim): Custom eval suite 嘅 component 通常包括 task corpus,

Last clause (verbatim): 即係 aggregate metric 同 per-task breakdown。

Custom eval suite 嘅 component 通常包括 task corpus,

即係 list of representative task,

每個 task 有 ground truth expected outcome。

Environment,

即係 deterministic simulation 嘅 environment 例如 database、

filesystem、

API mock。

Evaluator,

即係 automated judge 比較 agent output 同 ground truth。

Reporter,

即係 aggregate metric 同 per-task breakdown。

[15 | 07:54] 主持 M (host 子謙):

Turn 15 of 23 · speaker M (host 子謙 — opens and closes) · audio timestamp 07:54 · section 3 (Offline Replay & LLM-as-Judge)

speaker=M · chars=312 · ts=07:54 · terms=adversarial, categorize, production, misleading, robustness · sentences=5 · clauses=15

Verbatim phrases in this turn: adversarial task, 例如故意嘅 ambiguous query、 · Task corpus 嘅建立方法。 · 第一, 收集真實 user query 嘅 log, anonymize 同 categorize。

Task corpus 嘅建立方法。第一, 收集真實 user query 嘅 log, anonymize 同 categorize。

First clause (verbatim): Task corpus 嘅建立方法。

Last clause (verbatim): 去 test agent 嘅 robustness。

Task corpus 嘅建立方法。

第一,

收集真實 user query 嘅 log,

anonymize 同 categorize。

第二,

interview customer support team,

了解 top 10 嘅常見 task。

第三,

review production incident 嘅 root cause,

design task 去 test 個 agent 點樣 handle incident。

第四,

design adversarial task,

例如故意嘅 ambiguous query、

misleading context,

去 test agent 嘅 robustness。

[16 | 08:28] 嘉賓 F (expert 曉晴):

Turn 16 of 23 · speaker F (expert 曉晴 — explains concepts and trade-offs) · audio timestamp 08:28 · section 3 (Offline Replay & LLM-as-Judge)

speaker=F · chars=354 · ts=08:28 · terms=snapshot-based, containerized, reproducible, dependencies, Environment · sentences=5 · clauses=13

Verbatim phrases in this turn: Environment 嘅 reproducible design。 · , 例如 database 嘅 initial data 同 API mock 嘅 response。 · nvironment, 例如 Docker image with same dependencies。

Environment 嘅 reproducible design。

First clause (verbatim): Environment 嘅 reproducible design。

Last clause (verbatim): 即係 mock external API 避免 external dependency 嘅 variability。

Environment 嘅 reproducible design。

第一,

fixed seed for all random operations,

例如 database 嘅 initial data 同 API mock 嘅 response。

第二,

containerized environment,

例如 Docker image with same dependencies。

第三,

snapshot-based state restoration,

即係每次 task 開始 restore 返 initial snapshot。

第四,

network isolation,

即係 mock external API 避免 external dependency 嘅 variability。

End-of-section recap (last spoken sentence of Offline Replay & LLM-as-Judge): Environment 嘅 reproducible design。

Section 4/5 — Production Eval Pipeline

Production Eval Pipeline

Section overview: covers turns 17–21 (5 spoken segments).

Topic terms (extracted from spoken text): Self-preference, categorization, deterministic, per-tool-call, Visualization, LLM-as-judge, distribution, satisfaction, significance, correctness

Latin/English code-terms in this section (verbatim from speech): Self-preference, categorization, deterministic

Section character total: 1,784 characters across 5 spoken turns.

Section duration estimate: ~2:49 of 13:00 total.

Turns in this section: 17, 18, 19, 20, 21.

First spoken sentence of this section (turn 17, verbatim): Evaluator 嘅 design pattern。

Average characters per turn (this section): ~356 chars.

Cumulative characters through this section: 6,808 of 7,319 total.

[17 | 09:02] 主持 M (host 子謙):

Turn 17 of 23 · speaker M (host 子謙 — opens and closes) · audio timestamp 09:02 · section 4 (Production Eval Pipeline)

speaker=M · chars=363 · ts=09:02 · terms=deterministic, LLM-as-judge, correctness, similarity, Functional · sentences=4 · clauses=11

Verbatim phrases in this turn: ch, 適合 deterministic output 例如 SQL query、 · Evaluator 嘅 design pattern。 · 合 deterministic output 例如 SQL query、JSON structure。

Evaluator 嘅 design pattern。

First clause (verbatim): Evaluator 嘅 design pattern。

Last clause (verbatim): 適合 code generation。

Evaluator 嘅 design pattern。

Exact match,

即係 agent output 同 ground truth 嘅 exact string match,

適合 deterministic output 例如 SQL query、

JSON structure。

Semantic match,

即係 use embedding similarity 或者 LLM-as-judge 去 measure semantic correctness,

適合 natural language output。

Functional match,

即係 execute agent 嘅 output 例如 SQL 同 compare result 同 expected,

適合 code generation。

[18 | 09:36] 嘉賓 F (expert 曉晴):

Turn 18 of 23 · speaker F (expert 曉晴 — explains concepts and trade-offs) · audio timestamp 09:36 · section 4 (Production Eval Pipeline)

speaker=F · chars=382 · ts=09:36 · terms=Self-preference, LLM-as-judge, calibration, Mitigation, evaluation · sentences=5 · clauses=11

Verbatim phrases in this turn: LLM-as-judge 嘅 pitfalls。 · LLM judge 傾向揀第一個或者最後一個 option, 唔係 base on quality。 · as, 即係 LLM judge 傾向揀 longer response 即使 shorter 更好。

LLM-as-judge 嘅 pitfalls。

First clause (verbatim): LLM-as-judge 嘅 pitfalls。

Last clause (verbatim): calibration 即係 known quality ground truth 嘅 calibration set。

LLM-as-judge 嘅 pitfalls。

Position bias,

即係 LLM judge 傾向揀第一個或者最後一個 option,

唔係 base on quality。

Length bias,

即係 LLM judge 傾向揀 longer response 即使 shorter 更好。

Self-preference bias,

即係 LLM judge 傾向 score 自己 generate 嘅 output 更高。

Mitigation 包括 blind evaluation 即係 anonymize agent name,

multiple judge 即係 average multiple LLM judge,

calibration 即係 known quality ground truth 嘅 calibration set。

[19 | 10:10] 主持 M (host 子謙):

Turn 19 of 23 · speaker M (host 子謙 — opens and closes) · audio timestamp 10:10 · section 4 (Production Eval Pipeline)

speaker=M · chars=386 · ts=10:10 · terms=categorization, per-tool-call, Visualization, distribution, clustering · sentences=3 · clauses=10

Verbatim phrases in this turn: akdown, latency distribution, error categorization。 · mode clustering 幫助 identify common failure pattern。 · 返 baseline, 如果 metric 跌超過 threshold 即 block deploy。

-tool-call breakdown, cost breakdown, latency distribution, error categorization。

First clause (verbatim): Reporter 嘅 output 通常包括 overall metric 例如 success rate,

Last clause (verbatim): 如果 metric 跌超過 threshold 即 block deploy。

Reporter 嘅 output 通常包括 overall metric 例如 success rate,

per-task breakdown,

per-tool-call breakdown,

cost breakdown,

latency distribution,

error categorization。

Visualization 例如 confusion matrix 同 failure mode clustering 幫助 identify common failure pattern。

Regression detection 係 critical,

即係每次 new model 或者 new prompt 嘅 change 要 compare 返 baseline,

如果 metric 跌超過 threshold 即 block deploy。

[20 | 10:44] 嘉賓 F (expert 曉晴):

Turn 20 of 23 · speaker F (expert 曉晴 — explains concepts and trade-offs) · audio timestamp 10:44 · section 4 (Production Eval Pipeline)

speaker=F · chars=319 · ts=10:44 · terms=Continuous, evaluation, Regression, dashboard, pipeline · sentences=6 · clauses=12

Verbatim phrases in this turn: l result 自動 post 去 dashboard, 例如 Datadog、 · Continuous evaluation 嘅 pipeline 設計。 · nge 觸發 eval suite run, 即係 CI/CD pipeline 入面嘅 stage。

Continuous evaluation 嘅 pipeline 設計。

First clause (verbatim): Continuous evaluation 嘅 pipeline 設計。

Last clause (verbatim): full suite nightly。

Continuous evaluation 嘅 pipeline 設計。

每次 code change 觸發 eval suite run,

即係 CI/CD pipeline 入面嘅 stage。

Eval result 自動 post 去 dashboard,

例如 Datadog、

Grafana。

Regression alert 自動通知 owner。

Nightly full eval suite run,

因為 full suite 可能 run 幾個鐘,

唔可以每個 PR 都 full run。

Spot check sampling 嘅 quick eval suite 對每個 PR,

full suite nightly。

[21 | 11:18] 主持 M (host 子謙):

Turn 21 of 23 · speaker M (host 子謙 — opens and closes) · audio timestamp 11:18 · section 4 (Production Eval Pipeline)

speaker=M · chars=334 · ts=11:18 · terms=satisfaction, significance, Statistical, production, deployment · sentences=4 · clauses=8

Verbatim phrases in this turn: A/B testing 嘅 production deployment 策略。 · ion 嘅 user satisfaction metric 同 task success rate。 · tion 要 power analysis 嘅 sample size 計算, 通常要幾日至幾個禮拜。

A/B testing 嘅 production deployment 策略。

First clause (verbatim): A/B testing 嘅 production deployment 策略。

Last clause (verbatim): Statistical significance 要先 reach 95% confidence 至做 rollout decision。

A/B testing 嘅 production deployment 策略。

新 version 同 old version 同時 serve 同一個 percentage 嘅 traffic,

例如 10% 新 version,

90% old version,

compare 兩個 version 嘅 user satisfaction metric 同 task success rate。

A/B test 嘅 duration 要 power analysis 嘅 sample size 計算,

通常要幾日至幾個禮拜。

Statistical significance 要先 reach 95% confidence 至做 rollout decision。

End-of-section recap (last spoken sentence of Production Eval Pipeline): A/B testing 嘅 production deployment 策略。

Section 5/5 — Wrap-up & Safety Preview

總結同 Safety 預覽

Section overview: covers turns 22–23 (2 spoken segments).

Topic terms (extracted from spoken text): implementation, irreversible, Eval-driven, development, Responsible, acceptance, Production, regression, mitigation, sandboxing

Latin/English code-terms in this section (verbatim from speech): implementation, irreversible, Eval-driven

Section character total: 511 characters across 2 spoken turns.

Section duration estimate: ~1:07 of 13:00 total.

Turns in this section: 22, 23.

First spoken sentence of this section (turn 22, verbatim): Eval-driven development 嘅 best practice。

Average characters per turn (this section): ~255 chars.

Cumulative characters through this section: 7,319 of 7,319 total.

[22 | 11:52] 嘉賓 F (expert 曉晴):

Turn 22 of 23 · speaker F (expert 曉晴 — explains concepts and trade-offs) · audio timestamp 11:52 · section 5 (Wrap-up & Safety Preview)

speaker=F · chars=303 · ts=11:52 · terms=Eval-driven, development, acceptance, Production, regression · sentences=5 · clauses=9

Verbatim phrases in this turn: Eval-driven development 嘅 best practice。 · val suite, 即係 acceptance criteria 直接轉做 eval metric。 · lopment 要先喺 eval suite 上面 test, 唔可以只係手動 smoke test。

Eval-driven development 嘅 best practice。

First clause (verbatim): Eval-driven development 嘅 best practice。

Last clause (verbatim): 確保覆蓋所有 major use case。

Eval-driven development 嘅 best practice。

Spec 寫嘅時候同步寫 eval suite,

即係 acceptance criteria 直接轉做 eval metric。

新 prompt 嘅 development 要先喺 eval suite 上面 test,

唔可以只係手動 smoke test。

Production incident 之後要 add 新嘅 eval case 去 reproduce incident,

防止 regression。

Eval suite 嘅 coverage 要定期 audit,

確保覆蓋所有 major use case。

[23 | 12:26] 主持 M (host 子謙):

Turn 23 of 23 · speaker M (host 子謙 — opens and closes) · audio timestamp 12:26 · section 5 (Wrap-up & Safety Preview)

speaker=M · chars=208 · ts=12:26 · terms=implementation, irreversible, Responsible, mitigation, sandboxing · sentences=2 · clauses=7

Verbatim phrases in this turn: ent, 包括 irreversible action 嘅 mitigation、 · prompt injection attack 嘅防禦、 · pic 同 OpenAI 嘅 Responsible AI 政策同技術 implementation。

同 permission system, 同埋 Anthropic 同 OpenAI 嘅 Responsible AI 政策同技術 implementation。

First clause (verbatim): 下堂我哋會深入探討 Agent Safety 同 Alignment,

Last clause (verbatim): 我哋下期再見。

下堂我哋會深入探討 Agent Safety 同 Alignment,

包括 irreversible action 嘅 mitigation、

prompt injection attack 嘅防禦、

sandboxing 同 permission system,

同埋 Anthropic 同 OpenAI 嘅 Responsible AI 政策同技術 implementation。

多謝收聽第六課,

我哋下期再見。

End-of-section recap (last spoken sentence of Wrap-up & Safety Preview): 下堂我哋會深入探討 Agent Safety 同 Alignment, 包括 irreversible action 嘅 mitigation、prompt injection attack 嘅防禦、sandboxing 同 permission system, 同埋 Anthropic 同 OpenAI 嘅 Responsible AI 政策同技術 implementation。

End-of-lesson summary

This lesson covered 5 sections across 23 spoken turns (~13 min audio). Below is the final sentence of each section, preserved verbatim from the source podcast script.

• Opening & SWE-bench (turn 04): SWE-bench 嘅 evaluation metric 係 pass rate, 即係 agent generate 嘅 patch 通過所有 hidden unit tests 嘅 instance percentage。
• AgentBench · Terminal-Bench · GAIA (turn 10): 第四個 benchmark 係 WebArena。
• Offline Replay & LLM-as-Judge (turn 16): Environment 嘅 reproducible design。
• Production Eval Pipeline (turn 21): A/B testing 嘅 production deployment 策略。
• Wrap-up & Safety Preview (turn 23): 下堂我哋會深入探討 Agent Safety 同 Alignment, 包括 irreversible action 嘅 mitigation、prompt injection attack 嘅防禦、sandboxing 同 permission system, 同埋 Anthropic 同 OpenAI 嘅 Responsible AI 政策同技術 implementation。

End of transcript

Total turns in this lesson: 23 spoken segments · ~13 min audio · preserved verbatim from the source podcast script (/opt/data/workspace/projects/ai-agent-course-06/script_raw.json).

Use the audio player above to listen along. The Quiz section below tests comprehension of this lesson.

Source & integrity

• Source file: script_raw.json (the line-by-line Cantonese dialogue that was TTS-synthesised into the lesson MP3)
• Fidelity: all 廣東話 text is byte-identical to the source — no translation, no summarisation, no paraphrasing
• Markdown structure added: speaker labels, section headings, timestamp markers, per-turn metadata callouts (speaker id, char count, timestamp, verbatim terms), per-section overview blocks, sentence-level line breaks (for readability only)
• Rendering: react-markdown + remark-gfm in TranscriptPanel.tsx; dark theme & mobile-responsive via Tailwind prose

Integrity checksum

• Total spoken characters (across all turns): 7,319
• Total spoken sentences (across all turns): 91
• Total spoken clauses (across all turns): 231
• Speaker turn distribution: M=12 · F=11