Add per-combo pipeline scripts, fix eval semantics, exclude large artifacts

Pipeline (stages 1 -> 4-1) can now be run in order from each stage folder.

Stage scripts:
- 2-1: make SEP/FocalLora prep portable (derive paths from __file__ instead of
  hardcoded /home/hujk/...) and add prepare_head_ident_dataset.sh runner.
  Verified the SEP converter reproduces the committed jsonl byte-for-byte.
- 2-2: unify the four Ident_IH_ALL_1-4_<model>.sh scripts (modernise llama to
  conda hook + $ROOT/models; add the missing FocalLora step to qwen3-4b/8b so
  focallora.json gets generated for them too).
- 2-3: default TARGETS now covers the three curves from the README
  (all_roc_inst_0.1, user_roc_inst_0.1, focallora).
- 3-2: add combos/ with 24 scripts (4 models x {pbs,nts,nts_wam} x {squad,tri}),
  head ranking pinned to all_roc_inst_0.1, TOPK overridable.
- 4-1: add eval_single.sh driver + combos/ with 24 cross-eval wrappers
  (squad-trained -> tri-eval and vice versa), reusing the --eval-only path.

Eval semantics:
- Judge ASR before UTIL: a response carrying the injected answer now counts as
  attacked even when it also contains the correct answer. This changes the
  metric, so old training_log.csv rows are not comparable.
- Add --dev-holdout: reserve the last N source rows as a dev slice; training
  drops them and the in-training quick eval uses only them. Previously the
  quick eval silently defaulted to the squad evaluation set, which contradicted
  the README and self-contaminated squad-trained runs.
- train_attn_kl_clean.sh now passes --eval-data-path/--eval-topicattack-path.
- Add --eval-step0 to log an untuned-baseline row before any weight update.

Housekeeping:
- Quarantine superseded entry points under legacy/ (2-2 single-step wrappers,
  3-2 old _tuning.fix.* wrappers, 3-1 auxiliary), each with a README.
- Fix .gitignore: the model_score rule was anchored at the repo root and never
  matched Codes/..., so ~26GB of intermediates had been staged. Now excludes
  *.pkl (~25GB), heads_sorted_eval/ (~690MB), outputs_lora/ checkpoints
  (~3.2GB) and pycache. heads_sorted/ and head_scoring_combined.json are kept
  deliberately: they are small and are the HEAD_PATH inputs stage 3-2 needs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Codes/3-2_model_training/legacy/README.md

@@ -0,0 +1,8 @@
+# Legacy (superseded)
+
+- `train_single_turn.sh`, `train_single_turn_chunked.sh` use the old
+  `_tuning.fix.modified.py` / `_tuning.fix.chunked.py` trainers.
+- `test_chunked_attention_llama.sh` is a chunked-attention micro-benchmark.
+
+The canonical trainer is `train_attn_kl_clean.py` / `train_attn_kl_clean.sh`.
+Per-combo entry points live in `../combos/train_<model>_<source>_<config>.sh`.

Codes/3-2_model_training/legacy/test_chunked_attention_llama.sh

@@ -0,0 +1,31 @@
+#!/usr/bin/env sh
+set -eu
+
+CONDA_BIN=${CONDA_BIN:-}
+if [ -z "$CONDA_BIN" ]; then
+  if command -v conda >/dev/null 2>&1; then
+    CONDA_BIN=$(command -v conda)
+  elif [ -x /opt/miniconda/bin/conda ]; then
+    CONDA_BIN=/opt/miniconda/bin/conda
+  fi
+fi
+
+if [ -n "$CONDA_BIN" ]; then
+  eval "$("$CONDA_BIN" shell.bash hook)"
+  conda activate focallora4
+fi
+
+SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
+MODEL_PATH=${MODEL_PATH:-/data/local/hujk/models/Llama-3.1-8B-Instruct}
+OUT_DIR=${OUT_DIR:-$SCRIPT_DIR/test_outputs}
+GPUS=${GPUS:-"0 1"}
+
+mkdir -p "$OUT_DIR"
+
+for gpu in $GPUS; do
+  echo "[test_chunked_attention_llama] running on GPU $gpu"
+  CUDA_VISIBLE_DEVICES="$gpu" python "$SCRIPT_DIR/test_chunked_attention.py" \
+    --model-path "$MODEL_PATH" \
+    --output "$OUT_DIR/chunked_attention_gpu${gpu}.json" \
+    "$@"
+done

Codes/3-2_model_training/legacy/train_single_turn.sh

@@ -0,0 +1,38 @@
+#!/usr/bin/env sh
+set -eu
+
+export CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES:-0}
+
+CONDA_BIN=${CONDA_BIN:-}
+if [ -z "$CONDA_BIN" ]; then
+  if command -v conda >/dev/null 2>&1; then
+    CONDA_BIN=$(command -v conda)
+  elif [ -x /opt/miniconda/bin/conda ]; then
+    CONDA_BIN=/opt/miniconda/bin/conda
+  fi
+fi
+
+if [ -n "$CONDA_BIN" ]; then
+  eval "$("$CONDA_BIN" shell.bash hook)"
+  conda activate focallora4
+fi
+
+SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
+BASE=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
+
+MODEL_PATH=${MODEL_PATH:-/data/local/hujk/models/Llama-3.1-8B-Instruct}
+DATA_PATH=${DATA_PATH:-$BASE/3-1_model_training_data_gen/single_turn/tri_native_tool_response_only.json}
+HEAD_PATH=${HEAD_PATH:-$BASE/2-2_head_identification_scoring/model_score/sep_Llama-3.1-8B-Instruct/heads_sorted/all_roc_inst_0.1.json}
+OUTPUT_DIR=${OUTPUT_DIR:-$SCRIPT_DIR/outputs_lora/llama31-8b_tri_native_tool_response_only}
+
+python "$SCRIPT_DIR/_tuning.fix.modified.py" \
+  --model_path "$MODEL_PATH" \
+  --data_path "$DATA_PATH" \
+  --head_path "$HEAD_PATH" \
+  --output_dir "$OUTPUT_DIR" \
+  --topk "${TOPK:-100p}" \
+  --epochs "${EPOCHS:-30}" \
+  --batch_size "${BATCH_SIZE:-6}" \
+  --batch-save-interval "${BATCH_SAVE_INTERVAL:-100}" \
+  --lr "${LR:-5e-4}" \
+  --lambda_preserve "${LAMBDA_PRESERVE:-1.0}"

Codes/3-2_model_training/legacy/train_single_turn_chunked.sh

@@ -0,0 +1,46 @@
+#!/usr/bin/env sh
+set -eu
+
+export CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES:-0}
+
+CONDA_BIN=${CONDA_BIN:-}
+if [ -z "$CONDA_BIN" ]; then
+  if command -v conda >/dev/null 2>&1; then
+    CONDA_BIN=$(command -v conda)
+  elif [ -x /opt/miniconda/bin/conda ]; then
+    CONDA_BIN=/opt/miniconda/bin/conda
+  fi
+fi
+
+if [ -n "$CONDA_BIN" ]; then
+  eval "$("$CONDA_BIN" shell.bash hook)"
+  conda activate focallora4
+fi
+
+SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
+BASE=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
+
+MODEL_PATH=${MODEL_PATH:-/data/local/hujk/models/Llama-3.1-8B-Instruct}
+DATA_PATH=${DATA_PATH:-$BASE/3-1_model_training_data_gen/single_turn/tri_native_tool_response_only.json}
+HEAD_PATH=${HEAD_PATH:-$BASE/2-2_head_identification_scoring/model_score/sep_Llama-3.1-8B-Instruct/heads_sorted/all_roc_inst_0.1.json}
+OUTPUT_DIR=${OUTPUT_DIR:-$SCRIPT_DIR/outputs_lora/llama31-8b_tri_native_tool_response_only_chunked}
+
+python "$SCRIPT_DIR/_tuning.fix.chunked.py" \
+  --model_path "$MODEL_PATH" \
+  --data_path "$DATA_PATH" \
+  --head_path "$HEAD_PATH" \
+  --output_dir "$OUTPUT_DIR" \
+  --topk "${TOPK:-100p}" \
+  --epochs "${EPOCHS:-30}" \
+  --batch_size "${BATCH_SIZE:-6}" \
+  --batch-save-interval "${BATCH_SAVE_INTERVAL:-100}" \
+  --lr "${LR:-5e-4}" \
+  --lambda_preserve "${LAMBDA_PRESERVE:-1.0}" \
+  --attn-chunk-size "${ATTN_CHUNK_SIZE:-4096}" \
+  --max-train-steps "${MAX_TRAIN_STEPS:--1}" \
+  --repeat-single-sample "${REPEAT_SINGLE_SAMPLE:-1}" \
+  --max-len "${MAX_LEN:-50000}" \
+  --logits-to-keep "${LOGITS_TO_KEEP:-1}" \
+  ${SKIP_SAVE_EVAL:+--skip-save-eval} \
+  ${NO_SAVE:+--no-save} \
+  ${GRADIENT_CHECKPOINTING:+--gradient-checkpointing}