feat: reproducibility when saving & uploading a heretic model (#191)

* feat: implement reproducibility features with safetensors

* feat: prompt user before creating reproducibility folder

* fix: use prompt_confirm wrapper

* style comment

* style comment

* fix: ignore None values in Settings dump for TOML compatibility

* fix: imports

* feat: auto-generate seed if none provided for full reproducibility

* style: fix ruff formatting issues

* style: ruff

* style: fix ty check errors with ty:ignore

* Update src/heretic/main.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update src/heretic/utils.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* add period at end.

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Improve: Add README, checkpoint.jsonl, to Reproduce

* fix: use centralize device info, remove random states file

* feat: Add CUDA driver version

* ruff

* ruff...

* ty fix

* LGTM: Rich native strip, use nvidia-smi

* ruff fix

* ruff

* revert kaggle hack)

* normalize names for deduplication of packages/versions

* docstring

* rufff

* cleanup, add suffix for torch CUDA version, distinguish ROCm

* add PyTorch index URL detection

* revert index URL to be simple

* flip priority of index..

* add Important note

* add exact suffix for WHL in instruction

* add warning for heterogeneous GPU env

* extend driver version info (more accelerators)

* fix: style

* sync

* no abbreviation

* use multi-line string

* fix: prompt_confirm

* feat: CPU info

* strip 'slow' warning from environment.txt

* feat: Add virtual env info to environment.txt

* ruffff

* feat: AMD (Radeon) GPU driver version

* Refactor: system.py

* feat: LGTM capturing specifc installation origin of heretic

* feat: Include chosen trial into reproduce/README

* style: run ruff format on utils.py

* feat: reproduce.json

* fix: seperate values in different keys

* restore comment

* style, clean, seperate commit key

* no abbreviation, cleanup

* remove labels, store only dependencies

* missed import, ruff

* sort import

* feat: More CPU Info

* only store direct dependencies of heretic

* complete comment

* refactor: use cpuinfo package instead

* ruff import sort

* distinguish cores & threads

* move function amd-driver

* rename

* moving heretic package info,

* rufff

* Move: cleanup memory cache

* fix: model.py import

* no unknowns

* generalize all accelerator info stuff

* ruff f

* move package info

* type change

* feat: no reproducibility suite for local saving/model used

* import fix

* fix: type check

* style change

* style ruff

* feat: no env.txt, SHA256SUMS file, cleanup

* feat: ADD tip to readme

* remove trial index, two-keys only

* fix: No time-zone

* feat: No suite for local datasets allowed

* simplify

* featt: capture both direct and transitive dependencies

* style: sort readme of reproducibility suite

* feat: Store commit hash for datasets too

* add total refusal prompts for evaluation display

* remove try/except from cpu

* extend SHA256 support

* remove .txt

* only have safetensors for SHA256

* style comment

* use HF api to get commit hash

* fix: requirements containing irrelevant dependencies

* only store heretic-llm if from PyPI..

* add SELECTED tag to the trial that was pushed

* AttributeError fix

* simplify trial preservation

* add direction_index in trial info

* remove unwanted CPU info

* style: rename

---------

Co-authored-by: Vinayyyy7 <vinayumrethe99@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

config.default.toml

@@ -91,6 +91,10 @@ n_trials = 200
 # Number of trials that use random sampling for the purpose of exploration.
 n_startup_trials = 60
 
+# Random seed for reproducible optimization. Set to an integer to enable.
+# Applies to Python's random module, NumPy, PyTorch, and Optuna.
+# seed = 75
+
 # Directory to save and load study progress to/from.
 study_checkpoint_dir = "checkpoints"
 
@@ -140,6 +144,7 @@ split = "train[:400]"
 column = "text"
 residual_plot_label = '"Harmless" prompts'
 residual_plot_color = "royalblue"
+commit = ""
 
 # Dataset of prompts that tend to result in refusals (used for calculating refusal directions).
 [bad_prompts]
@@ -148,15 +153,18 @@ split = "train[:400]"
 column = "text"
 residual_plot_label = '"Harmful" prompts'
 residual_plot_color = "darkorange"
+commit = ""
 
 # Dataset of prompts that tend to not result in refusals (used for evaluating model performance).
 [good_evaluation_prompts]
 dataset = "mlabonne/harmless_alpaca"
 split = "test[:100]"
 column = "text"
+commit = ""
 
 # Dataset of prompts that tend to result in refusals (used for evaluating model performance).
 [bad_evaluation_prompts]
 dataset = "mlabonne/harmful_behaviors"
 split = "test[:100]"
 column = "text"
+commit = ""