0f90602339
Pipeline (stages 1 -> 4-1) can now be run in order from each stage folder.
Stage scripts:
- 2-1: make SEP/FocalLora prep portable (derive paths from __file__ instead of
hardcoded /home/hujk/...) and add prepare_head_ident_dataset.sh runner.
Verified the SEP converter reproduces the committed jsonl byte-for-byte.
- 2-2: unify the four Ident_IH_ALL_1-4_<model>.sh scripts (modernise llama to
conda hook + $ROOT/models; add the missing FocalLora step to qwen3-4b/8b so
focallora.json gets generated for them too).
- 2-3: default TARGETS now covers the three curves from the README
(all_roc_inst_0.1, user_roc_inst_0.1, focallora).
- 3-2: add combos/ with 24 scripts (4 models x {pbs,nts,nts_wam} x {squad,tri}),
head ranking pinned to all_roc_inst_0.1, TOPK overridable.
- 4-1: add eval_single.sh driver + combos/ with 24 cross-eval wrappers
(squad-trained -> tri-eval and vice versa), reusing the --eval-only path.
Eval semantics:
- Judge ASR before UTIL: a response carrying the injected answer now counts as
attacked even when it also contains the correct answer. This changes the
metric, so old training_log.csv rows are not comparable.
- Add --dev-holdout: reserve the last N source rows as a dev slice; training
drops them and the in-training quick eval uses only them. Previously the
quick eval silently defaulted to the squad evaluation set, which contradicted
the README and self-contaminated squad-trained runs.
- train_attn_kl_clean.sh now passes --eval-data-path/--eval-topicattack-path.
- Add --eval-step0 to log an untuned-baseline row before any weight update.
Housekeeping:
- Quarantine superseded entry points under legacy/ (2-2 single-step wrappers,
3-2 old _tuning.fix.* wrappers, 3-1 auxiliary), each with a README.
- Fix .gitignore: the model_score rule was anchored at the repo root and never
matched Codes/..., so ~26GB of intermediates had been staged. Now excludes
*.pkl (~25GB), heads_sorted_eval/ (~690MB), outputs_lora/ checkpoints
(~3.2GB) and pycache. heads_sorted/ and head_scoring_combined.json are kept
deliberately: they are small and are the HEAD_PATH inputs stage 3-2 needs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
58 lines
2.4 KiB
Bash
Executable File
58 lines
2.4 KiB
Bash
Executable File
#!/usr/bin/env sh
|
|
# Single-turn evaluation driver (README stage 4).
|
|
# Reuses the eval-only path of 3-2's train_attn_kl_clean.py: loads a base model +
|
|
# a trained LoRA adapter, then runs the TopicAttack-style single-turn eval.
|
|
#
|
|
# Required env (usually set by a combos/ wrapper):
|
|
# MODEL_PATH base model dir
|
|
# LORA_PATH trained adapter dir (e.g. .../outputs_lora/<combo>/final)
|
|
# EVAL_DATA_PATH cross-source injection_qa json (squad-trained -> tri, and vice versa)
|
|
# EVAL_CONFIG prompt_based_separator | native_tool_response_only | native_tool_empty_query
|
|
# EVAL_OUTPUT output json path
|
|
# Optional: EVAL_TOPIC_PATH EVAL_ATTACKS EVAL_ATTACK_SIDE EVAL_SIZE EVAL_BATCH_SIZE EVAL_MAX_NEW_TOKENS
|
|
set -eu
|
|
|
|
export CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES:-0}
|
|
export PYTORCH_ALLOC_CONF=${PYTORCH_ALLOC_CONF:-expandable_segments:True}
|
|
|
|
CONDA_BIN=${CONDA_BIN:-}
|
|
if [ -z "$CONDA_BIN" ]; then
|
|
if command -v conda >/dev/null 2>&1; then
|
|
CONDA_BIN=$(command -v conda)
|
|
elif [ -x /opt/miniconda/bin/conda ]; then
|
|
CONDA_BIN=/opt/miniconda/bin/conda
|
|
fi
|
|
fi
|
|
if [ -n "$CONDA_BIN" ]; then
|
|
eval "$("$CONDA_BIN" shell.bash hook)"
|
|
conda activate focallora4
|
|
fi
|
|
|
|
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
|
|
BASE=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
|
|
ROOT=$(CDPATH= cd -- "$BASE/.." && pwd)
|
|
TRAIN_PY="$BASE/3-2_model_training/train_attn_kl_clean.py"
|
|
|
|
MODEL_PATH=${MODEL_PATH:-$ROOT/models/Qwen2-7B-Instruct}
|
|
LORA_PATH=${LORA_PATH:?set LORA_PATH to a trained adapter dir}
|
|
EVAL_DATA_PATH=${EVAL_DATA_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_injection_qa.json}
|
|
EVAL_TOPIC_PATH=${EVAL_TOPIC_PATH:-$BASE/1_raw_dataset/topicattack/data/crafted_instruction_data_tri_conversation_attack_complete.json}
|
|
EVAL_CONFIG=${EVAL_CONFIG:-native_tool_response_only}
|
|
EVAL_OUTPUT=${EVAL_OUTPUT:-$SCRIPT_DIR/results/eval.json}
|
|
|
|
python "$TRAIN_PY" \
|
|
--eval-only \
|
|
--eval-topicattack \
|
|
--model-path "$MODEL_PATH" \
|
|
--lora-path "$LORA_PATH" \
|
|
--eval-data-path "$EVAL_DATA_PATH" \
|
|
--eval-topicattack-path "$EVAL_TOPIC_PATH" \
|
|
--eval-config "$EVAL_CONFIG" \
|
|
--eval-output "$EVAL_OUTPUT" \
|
|
--eval-attacks "${EVAL_ATTACKS:-none,naive,ignore,escape_separation,completion_realcmb,conv_attack}" \
|
|
--eval-attack-side "${EVAL_ATTACK_SIDE:-end}" \
|
|
--eval-size "${EVAL_SIZE:--1}" \
|
|
--eval-batch-size "${EVAL_BATCH_SIZE:-4}" \
|
|
--eval-max-new-tokens "${EVAL_MAX_NEW_TOKENS:-256}" \
|
|
${EVAL_MMLU:+--eval-mmlu}
|