0f90602339
Pipeline (stages 1 -> 4-1) can now be run in order from each stage folder.
Stage scripts:
- 2-1: make SEP/FocalLora prep portable (derive paths from __file__ instead of
hardcoded /home/hujk/...) and add prepare_head_ident_dataset.sh runner.
Verified the SEP converter reproduces the committed jsonl byte-for-byte.
- 2-2: unify the four Ident_IH_ALL_1-4_<model>.sh scripts (modernise llama to
conda hook + $ROOT/models; add the missing FocalLora step to qwen3-4b/8b so
focallora.json gets generated for them too).
- 2-3: default TARGETS now covers the three curves from the README
(all_roc_inst_0.1, user_roc_inst_0.1, focallora).
- 3-2: add combos/ with 24 scripts (4 models x {pbs,nts,nts_wam} x {squad,tri}),
head ranking pinned to all_roc_inst_0.1, TOPK overridable.
- 4-1: add eval_single.sh driver + combos/ with 24 cross-eval wrappers
(squad-trained -> tri-eval and vice versa), reusing the --eval-only path.
Eval semantics:
- Judge ASR before UTIL: a response carrying the injected answer now counts as
attacked even when it also contains the correct answer. This changes the
metric, so old training_log.csv rows are not comparable.
- Add --dev-holdout: reserve the last N source rows as a dev slice; training
drops them and the in-training quick eval uses only them. Previously the
quick eval silently defaulted to the squad evaluation set, which contradicted
the README and self-contaminated squad-trained runs.
- train_attn_kl_clean.sh now passes --eval-data-path/--eval-topicattack-path.
- Add --eval-step0 to log an untuned-baseline row before any weight update.
Housekeeping:
- Quarantine superseded entry points under legacy/ (2-2 single-step wrappers,
3-2 old _tuning.fix.* wrappers, 3-1 auxiliary), each with a README.
- Fix .gitignore: the model_score rule was anchored at the repo root and never
matched Codes/..., so ~26GB of intermediates had been staged. Now excludes
*.pkl (~25GB), heads_sorted_eval/ (~690MB), outputs_lora/ checkpoints
(~3.2GB) and pycache. heads_sorted/ and head_scoring_combined.json are kept
deliberately: they are small and are the HEAD_PATH inputs stage 3-2 needs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
73 lines
1.8 KiB
Python
73 lines
1.8 KiB
Python
class Colors:
|
|
RED = '\033[91m'
|
|
GREEN = '\033[92m'
|
|
YELLOW = '\033[93m'
|
|
BLUE = '\033[94m'
|
|
ENDC = '\033[0m' # Reset color
|
|
|
|
|
|
def print_tok_color(tok, ids, colormask):
|
|
if ids.shape != colormask.shape:
|
|
raise ValueError(f"{ids.shape} != {colormask.shape}")
|
|
|
|
# ids and colormask are 2D: (batch, seq_len)
|
|
batch_size, seq_len = ids.shape
|
|
|
|
for b in range(batch_size):
|
|
out = []
|
|
for i in range(seq_len):
|
|
token_id = int(ids[b, i])
|
|
masked = bool(colormask[b, i])
|
|
|
|
# Convert id → token (use decode if you prefer)
|
|
token = tok.decode(token_id)
|
|
|
|
if masked:
|
|
out.append(f"{Colors.RED}{token}{Colors.ENDC}")
|
|
else:
|
|
out.append(token)
|
|
|
|
print(" ".join(out))
|
|
|
|
def print_data_color_in_batch(index, tok, input_ids, data_mask):
|
|
"""
|
|
Print decoded tokens for sample `index`, with data-masked tokens in red.
|
|
|
|
Args:
|
|
index: batch index
|
|
tok: tokenizer
|
|
input_ids: (batch, seq_len) tensor
|
|
data_mask: (batch, seq_len) bool/int tensor — True/1 = data token (will be red)
|
|
"""
|
|
RED = "\033[91m"
|
|
RESET = "\033[0m"
|
|
|
|
ids = input_ids[index].tolist()
|
|
mask = data_mask[index].bool().tolist()
|
|
|
|
parts = []
|
|
cur_text = ""
|
|
cur_is_data = mask[0] if ids else False
|
|
|
|
for tid, m in zip(ids, mask):
|
|
decoded = tok.decode([tid])
|
|
if m == cur_is_data:
|
|
cur_text += decoded
|
|
else:
|
|
if cur_text:
|
|
parts.append((cur_is_data, cur_text))
|
|
cur_text = decoded
|
|
cur_is_data = m
|
|
|
|
if cur_text:
|
|
parts.append((cur_is_data, cur_text))
|
|
|
|
out = ""
|
|
for is_data, text in parts:
|
|
if is_data:
|
|
out += f"{RED}{text}{RESET}"
|
|
else:
|
|
out += text
|
|
|
|
print(out)
|