hujk/OGAAA
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add per-combo pipeline scripts, fix eval semantics, exclude large artifacts

Browse Source 
Pipeline (stages 1 -> 4-1) can now be run in order from each stage folder.

Stage scripts:
- 2-1: make SEP/FocalLora prep portable (derive paths from __file__ instead of
  hardcoded /home/hujk/...) and add prepare_head_ident_dataset.sh runner.
  Verified the SEP converter reproduces the committed jsonl byte-for-byte.
- 2-2: unify the four Ident_IH_ALL_1-4_<model>.sh scripts (modernise llama to
  conda hook + $ROOT/models; add the missing FocalLora step to qwen3-4b/8b so
  focallora.json gets generated for them too).
- 2-3: default TARGETS now covers the three curves from the README
  (all_roc_inst_0.1, user_roc_inst_0.1, focallora).
- 3-2: add combos/ with 24 scripts (4 models x {pbs,nts,nts_wam} x {squad,tri}),
  head ranking pinned to all_roc_inst_0.1, TOPK overridable.
- 4-1: add eval_single.sh driver + combos/ with 24 cross-eval wrappers
  (squad-trained -> tri-eval and vice versa), reusing the --eval-only path.

Eval semantics:
- Judge ASR before UTIL: a response carrying the injected answer now counts as
  attacked even when it also contains the correct answer. This changes the
  metric, so old training_log.csv rows are not comparable.
- Add --dev-holdout: reserve the last N source rows as a dev slice; training
  drops them and the in-training quick eval uses only them. Previously the
  quick eval silently defaulted to the squad evaluation set, which contradicted
  the README and self-contaminated squad-trained runs.
- train_attn_kl_clean.sh now passes --eval-data-path/--eval-topicattack-path.
- Add --eval-step0 to log an untuned-baseline row before any weight update.

Housekeeping:
- Quarantine superseded entry points under legacy/ (2-2 single-step wrappers,
  3-2 old _tuning.fix.* wrappers, 3-1 auxiliary), each with a README.
- Fix .gitignore: the model_score rule was anchored at the repo root and never
  matched Codes/..., so ~26GB of intermediates had been staged. Now excludes
  *.pkl (~25GB), heads_sorted_eval/ (~690MB), outputs_lora/ checkpoints
  (~3.2GB) and pycache. heads_sorted/ and head_scoring_combined.json are kept
  deliberately: they are small and are the HEAD_PATH inputs stage 3-2 needs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
HenryChou020514
2026-07-17 13:58:36 +08:00
co-authored by Claude Opus 4.8
parent 01bb07dba8
commit 0f90602339
635 changed files with 1076514 additions and 16308 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Codes/4-1_evaluation_single/README.txt
+9
View File
@@ -0,0 +1,9 @@
Evaluation 有三個維度
1. 攻擊強度
2. 攻擊成功率
3. 模型性能
目標
Training 時，只看過很弱的攻擊
Evaluation 時，遇到很強的攻擊，攻擊成功率↓，模型性能↑
Codes/4-1_evaluation_single/combos/eval_llama31-8b_squad_nts.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on squad (native_tool_response_only) -> evaluate on tri. model=Llama-3.1-8B-Instruct
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Llama-3.1-8B-Instruct}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/llama31-8b_squad_nts_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-native_tool_response_only}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/llama31-8b_squadtrain_trieval_nts.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_llama31-8b_squad_nts_wam.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on squad (native_tool_empty_query) -> evaluate on tri. model=Llama-3.1-8B-Instruct
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Llama-3.1-8B-Instruct}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/llama31-8b_squad_nts_wam_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-native_tool_empty_query}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/llama31-8b_squadtrain_trieval_nts_wam.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_llama31-8b_squad_pbs.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on squad (prompt_based_separator) -> evaluate on tri. model=Llama-3.1-8B-Instruct
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Llama-3.1-8B-Instruct}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/llama31-8b_squad_pbs_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-prompt_based_separator}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/llama31-8b_squadtrain_trieval_pbs.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_llama31-8b_tri_nts.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on tri (native_tool_response_only) -> evaluate on squad. model=Llama-3.1-8B-Instruct
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Llama-3.1-8B-Instruct}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/llama31-8b_tri_nts_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-native_tool_response_only}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/llama31-8b_tritrain_squadeval_nts.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_llama31-8b_tri_nts_wam.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on tri (native_tool_empty_query) -> evaluate on squad. model=Llama-3.1-8B-Instruct
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Llama-3.1-8B-Instruct}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/llama31-8b_tri_nts_wam_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-native_tool_empty_query}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/llama31-8b_tritrain_squadeval_nts_wam.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_llama31-8b_tri_pbs.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on tri (prompt_based_separator) -> evaluate on squad. model=Llama-3.1-8B-Instruct
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Llama-3.1-8B-Instruct}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/llama31-8b_tri_pbs_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-prompt_based_separator}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/llama31-8b_tritrain_squadeval_pbs.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_qwen2-7b_squad_nts.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on squad (native_tool_response_only) -> evaluate on tri. model=Qwen2-7B-Instruct
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Qwen2-7B-Instruct}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/qwen2-7b_squad_nts_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-native_tool_response_only}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/qwen2-7b_squadtrain_trieval_nts.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_qwen2-7b_squad_nts_wam.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on squad (native_tool_empty_query) -> evaluate on tri. model=Qwen2-7B-Instruct
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Qwen2-7B-Instruct}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/qwen2-7b_squad_nts_wam_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-native_tool_empty_query}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/qwen2-7b_squadtrain_trieval_nts_wam.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_qwen2-7b_squad_pbs.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on squad (prompt_based_separator) -> evaluate on tri. model=Qwen2-7B-Instruct
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Qwen2-7B-Instruct}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/qwen2-7b_squad_pbs_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-prompt_based_separator}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/qwen2-7b_squadtrain_trieval_pbs.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_qwen2-7b_tri_nts.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on tri (native_tool_response_only) -> evaluate on squad. model=Qwen2-7B-Instruct
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Qwen2-7B-Instruct}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/qwen2-7b_tri_nts_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-native_tool_response_only}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/qwen2-7b_tritrain_squadeval_nts.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_qwen2-7b_tri_nts_wam.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on tri (native_tool_empty_query) -> evaluate on squad. model=Qwen2-7B-Instruct
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Qwen2-7B-Instruct}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/qwen2-7b_tri_nts_wam_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-native_tool_empty_query}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/qwen2-7b_tritrain_squadeval_nts_wam.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_qwen2-7b_tri_pbs.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on tri (prompt_based_separator) -> evaluate on squad. model=Qwen2-7B-Instruct
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Qwen2-7B-Instruct}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/qwen2-7b_tri_pbs_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-prompt_based_separator}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/qwen2-7b_tritrain_squadeval_pbs.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_qwen3-4b_squad_nts.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on squad (native_tool_response_only) -> evaluate on tri. model=Qwen3-4B
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Qwen3-4B}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/qwen3-4b_squad_nts_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-native_tool_response_only}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/qwen3-4b_squadtrain_trieval_nts.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_qwen3-4b_squad_nts_wam.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on squad (native_tool_empty_query) -> evaluate on tri. model=Qwen3-4B
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Qwen3-4B}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/qwen3-4b_squad_nts_wam_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-native_tool_empty_query}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/qwen3-4b_squadtrain_trieval_nts_wam.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_qwen3-4b_squad_pbs.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on squad (prompt_based_separator) -> evaluate on tri. model=Qwen3-4B
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Qwen3-4B}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/qwen3-4b_squad_pbs_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-prompt_based_separator}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/qwen3-4b_squadtrain_trieval_pbs.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_qwen3-4b_tri_nts.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on tri (native_tool_response_only) -> evaluate on squad. model=Qwen3-4B
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Qwen3-4B}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/qwen3-4b_tri_nts_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-native_tool_response_only}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/qwen3-4b_tritrain_squadeval_nts.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_qwen3-4b_tri_nts_wam.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on tri (native_tool_empty_query) -> evaluate on squad. model=Qwen3-4B
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Qwen3-4B}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/qwen3-4b_tri_nts_wam_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-native_tool_empty_query}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/qwen3-4b_tritrain_squadeval_nts_wam.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_qwen3-4b_tri_pbs.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on tri (prompt_based_separator) -> evaluate on squad. model=Qwen3-4B
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Qwen3-4B}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/qwen3-4b_tri_pbs_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-prompt_based_separator}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/qwen3-4b_tritrain_squadeval_pbs.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_qwen3-8b_squad_nts.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on squad (native_tool_response_only) -> evaluate on tri. model=Qwen3-8B
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Qwen3-8B}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/qwen3-8b_squad_nts_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-native_tool_response_only}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/qwen3-8b_squadtrain_trieval_nts.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_qwen3-8b_squad_nts_wam.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on squad (native_tool_empty_query) -> evaluate on tri. model=Qwen3-8B
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Qwen3-8B}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/qwen3-8b_squad_nts_wam_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-native_tool_empty_query}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/qwen3-8b_squadtrain_trieval_nts_wam.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_qwen3-8b_squad_pbs.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on squad (prompt_based_separator) -> evaluate on tri. model=Qwen3-8B
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Qwen3-8B}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/qwen3-8b_squad_pbs_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-prompt_based_separator}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/qwen3-8b_squadtrain_trieval_pbs.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_qwen3-8b_tri_nts.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on tri (native_tool_response_only) -> evaluate on squad. model=Qwen3-8B
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Qwen3-8B}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/qwen3-8b_tri_nts_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-native_tool_response_only}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/qwen3-8b_tritrain_squadeval_nts.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_qwen3-8b_tri_nts_wam.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on tri (native_tool_empty_query) -> evaluate on squad. model=Qwen3-8B
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Qwen3-8B}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/qwen3-8b_tri_nts_wam_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-native_tool_empty_query}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/qwen3-8b_tritrain_squadeval_nts_wam.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/combos/eval_qwen3-8b_tri_pbs.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env sh
# Cross-eval: trained on tri (prompt_based_separator) -> evaluate on squad. model=Qwen3-8B
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
EVAL_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
BASE=$(CDPATH= cd -- "$EVAL_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
export MODEL_PATH="${MODEL_PATH:-$ROOT/models/Qwen3-8B}"
export LORA_PATH="${LORA_PATH:-$BASE/3-2_model_training/outputs_lora/qwen3-8b_tri_pbs_all_roc_inst_0.1/final}"
export EVAL_DATA_PATH="${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_injection_qa.json}"
export EVAL_TOPIC_PATH="${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_squad_conversation_attack_complete.json}"
export EVAL_CONFIG="${EVAL_CONFIG:-prompt_based_separator}"
export EVAL_OUTPUT="${EVAL_OUTPUT:-$EVAL_DIR/results/qwen3-8b_tritrain_squadeval_pbs.json}"
exec "$EVAL_DIR/eval_single.sh"
Codes/4-1_evaluation_single/eval_single.sh
Executable
+57
View File
@@ -0,0 +1,57 @@
#!/usr/bin/env sh
# Single-turn evaluation driver (README stage 4).
# Reuses the eval-only path of 3-2's train_attn_kl_clean.py: loads a base model +
# a trained LoRA adapter, then runs the TopicAttack-style single-turn eval.
#
# Required env (usually set by a combos/ wrapper):
# MODEL_PATH base model dir
# LORA_PATH trained adapter dir (e.g. .../outputs_lora/<combo>/final)
# EVAL_DATA_PATH cross-source injection_qa json (squad-trained -> tri, and vice versa)
# EVAL_CONFIG prompt_based_separator | native_tool_response_only | native_tool_empty_query
# EVAL_OUTPUT output json path
# Optional: EVAL_TOPIC_PATH EVAL_ATTACKS EVAL_ATTACK_SIDE EVAL_SIZE EVAL_BATCH_SIZE EVAL_MAX_NEW_TOKENS
set -eu
export CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES:-0}
export PYTORCH_ALLOC_CONF=${PYTORCH_ALLOC_CONF:-expandable_segments:True}
CONDA_BIN=${CONDA_BIN:-}
if [ -z "$CONDA_BIN" ]; then
if command -v conda >/dev/null 2>&1; then
CONDA_BIN=$(command -v conda)
elif [ -x /opt/miniconda/bin/conda ]; then
CONDA_BIN=/opt/miniconda/bin/conda
fi
fi
if [ -n "$CONDA_BIN" ]; then
eval "$("$CONDA_BIN" shell.bash hook)"
conda activate focallora4
fi
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
BASE=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
TRAIN_PY="$BASE/3-2_model_training/train_attn_kl_clean.py"
MODEL_PATH=${MODEL_PATH:-$ROOT/models/Qwen2-7B-Instruct}
LORA_PATH=${LORA_PATH:?set LORA_PATH to a trained adapter dir}
EVAL_DATA_PATH=${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_injection_qa.json}
EVAL_TOPIC_PATH=${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_conversation_attack_complete.json}
EVAL_CONFIG=${EVAL_CONFIG:-native_tool_response_only}
EVAL_OUTPUT=${EVAL_OUTPUT:-$SCRIPT_DIR/results/eval.json}
python "$TRAIN_PY" \
--eval-only \
--eval-topicattack \
--model-path "$MODEL_PATH" \
--lora-path "$LORA_PATH" \
--eval-data-path "$EVAL_DATA_PATH" \
--eval-topicattack-path "$EVAL_TOPIC_PATH" \
--eval-config "$EVAL_CONFIG" \
--eval-output "$EVAL_OUTPUT" \
--eval-attacks "${EVAL_ATTACKS:-none,naive,ignore,escape_separation,completion_realcmb,conv_attack}" \
--eval-attack-side "${EVAL_ATTACK_SIDE:-end}" \
--eval-size "${EVAL_SIZE:--1}" \
--eval-batch-size "${EVAL_BATCH_SIZE:-4}" \
--eval-max-new-tokens "${EVAL_MAX_NEW_TOKENS:-256}" \
${EVAL_MMLU:+--eval-mmlu}