0f90602339
Pipeline (stages 1 -> 4-1) can now be run in order from each stage folder.
Stage scripts:
- 2-1: make SEP/FocalLora prep portable (derive paths from __file__ instead of
hardcoded /home/hujk/...) and add prepare_head_ident_dataset.sh runner.
Verified the SEP converter reproduces the committed jsonl byte-for-byte.
- 2-2: unify the four Ident_IH_ALL_1-4_<model>.sh scripts (modernise llama to
conda hook + $ROOT/models; add the missing FocalLora step to qwen3-4b/8b so
focallora.json gets generated for them too).
- 2-3: default TARGETS now covers the three curves from the README
(all_roc_inst_0.1, user_roc_inst_0.1, focallora).
- 3-2: add combos/ with 24 scripts (4 models x {pbs,nts,nts_wam} x {squad,tri}),
head ranking pinned to all_roc_inst_0.1, TOPK overridable.
- 4-1: add eval_single.sh driver + combos/ with 24 cross-eval wrappers
(squad-trained -> tri-eval and vice versa), reusing the --eval-only path.
Eval semantics:
- Judge ASR before UTIL: a response carrying the injected answer now counts as
attacked even when it also contains the correct answer. This changes the
metric, so old training_log.csv rows are not comparable.
- Add --dev-holdout: reserve the last N source rows as a dev slice; training
drops them and the in-training quick eval uses only them. Previously the
quick eval silently defaulted to the squad evaluation set, which contradicted
the README and self-contaminated squad-trained runs.
- train_attn_kl_clean.sh now passes --eval-data-path/--eval-topicattack-path.
- Add --eval-step0 to log an untuned-baseline row before any weight update.
Housekeeping:
- Quarantine superseded entry points under legacy/ (2-2 single-step wrappers,
3-2 old _tuning.fix.* wrappers, 3-1 auxiliary), each with a README.
- Fix .gitignore: the model_score rule was anchored at the repo root and never
matched Codes/..., so ~26GB of intermediates had been staged. Now excludes
*.pkl (~25GB), heads_sorted_eval/ (~690MB), outputs_lora/ checkpoints
(~3.2GB) and pycache. heads_sorted/ and head_scoring_combined.json are kept
deliberately: they are small and are the HEAD_PATH inputs stage 3-2 needs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
117 lines
3.2 KiB
JSON
117 lines
3.2 KiB
JSON
{
|
|
"base_tokenizer": "llama",
|
|
"tokenizer_paths": {
|
|
"llama": "/data/local/hujk/models/Llama-3.1-8B-Instruct",
|
|
"qwen3": "/data/local/hujk/models/Qwen3-8B"
|
|
},
|
|
"files": [
|
|
{
|
|
"path": "/home/hujk/gitrs/Paper2026/SortedCode2/3-1_model_training_data_gen/len_test/len_00512.json",
|
|
"target_total_len": 512,
|
|
"filler_repeat": 18,
|
|
"token_lengths": {
|
|
"llama": 500,
|
|
"qwen3": 482
|
|
}
|
|
},
|
|
{
|
|
"path": "/home/hujk/gitrs/Paper2026/SortedCode2/3-1_model_training_data_gen/len_test/len_01024.json",
|
|
"target_total_len": 1024,
|
|
"filler_repeat": 41,
|
|
"token_lengths": {
|
|
"llama": 1006,
|
|
"qwen3": 988
|
|
}
|
|
},
|
|
{
|
|
"path": "/home/hujk/gitrs/Paper2026/SortedCode2/3-1_model_training_data_gen/len_test/len_02048.json",
|
|
"target_total_len": 2048,
|
|
"filler_repeat": 88,
|
|
"token_lengths": {
|
|
"llama": 2040,
|
|
"qwen3": 2022
|
|
}
|
|
},
|
|
{
|
|
"path": "/home/hujk/gitrs/Paper2026/SortedCode2/3-1_model_training_data_gen/len_test/len_04096.json",
|
|
"target_total_len": 4096,
|
|
"filler_repeat": 181,
|
|
"token_lengths": {
|
|
"llama": 4086,
|
|
"qwen3": 4068
|
|
}
|
|
},
|
|
{
|
|
"path": "/home/hujk/gitrs/Paper2026/SortedCode2/3-1_model_training_data_gen/len_test/len_08192.json",
|
|
"target_total_len": 8192,
|
|
"filler_repeat": 367,
|
|
"token_lengths": {
|
|
"llama": 8178,
|
|
"qwen3": 8160
|
|
}
|
|
},
|
|
{
|
|
"path": "/home/hujk/gitrs/Paper2026/SortedCode2/3-1_model_training_data_gen/len_test/len_12288.json",
|
|
"target_total_len": 12288,
|
|
"filler_repeat": 553,
|
|
"token_lengths": {
|
|
"llama": 12270,
|
|
"qwen3": 12252
|
|
}
|
|
},
|
|
{
|
|
"path": "/home/hujk/gitrs/Paper2026/SortedCode2/3-1_model_training_data_gen/len_test/len_16384.json",
|
|
"target_total_len": 16384,
|
|
"filler_repeat": 740,
|
|
"token_lengths": {
|
|
"llama": 16384,
|
|
"qwen3": 16366
|
|
}
|
|
},
|
|
{
|
|
"path": "/home/hujk/gitrs/Paper2026/SortedCode2/3-1_model_training_data_gen/len_test/len_24576.json",
|
|
"target_total_len": 24576,
|
|
"filler_repeat": 1112,
|
|
"token_lengths": {
|
|
"llama": 24568,
|
|
"qwen3": 24550
|
|
}
|
|
},
|
|
{
|
|
"path": "/home/hujk/gitrs/Paper2026/SortedCode2/3-1_model_training_data_gen/len_test/len_32768.json",
|
|
"target_total_len": 32768,
|
|
"filler_repeat": 1484,
|
|
"token_lengths": {
|
|
"llama": 32752,
|
|
"qwen3": 32734
|
|
}
|
|
},
|
|
{
|
|
"path": "/home/hujk/gitrs/Paper2026/SortedCode2/3-1_model_training_data_gen/len_test/len_40960.json",
|
|
"target_total_len": 40960,
|
|
"filler_repeat": 1857,
|
|
"token_lengths": {
|
|
"llama": 40958,
|
|
"qwen3": 40940
|
|
}
|
|
},
|
|
{
|
|
"path": "/home/hujk/gitrs/Paper2026/SortedCode2/3-1_model_training_data_gen/len_test/len_49152.json",
|
|
"target_total_len": 49152,
|
|
"filler_repeat": 2229,
|
|
"token_lengths": {
|
|
"llama": 49142,
|
|
"qwen3": 49124
|
|
}
|
|
},
|
|
{
|
|
"path": "/home/hujk/gitrs/Paper2026/SortedCode2/3-1_model_training_data_gen/len_test/len_65536.json",
|
|
"target_total_len": 65536,
|
|
"filler_repeat": 2974,
|
|
"token_lengths": {
|
|
"llama": 65532,
|
|
"qwen3": 65514
|
|
}
|
|
}
|
|
]
|
|
} |