92d0c0d551
feat: enumerate all available GPUs on startup ( #86 )
...
* feat: enumerate all available GPUs on startup
* feat: extend device enumeration to all accelerator types
2025-12-16 17:42:15 +05:30
243f821d93
feat: Add 4-bit loading + LoRA support for low VRAM optimization ( #60 )
...
* Add files via upload
* perf: optimize abliteration matrix op (#46 )
* perf: optimize abliteration matrix op
* refactor: comments and var names correspond with arditi
* refactor: fix comments and improve var notation
* fix: accidental line change and improve comments
---------
Co-authored-by: mad-cat-lon <113548315+mad-cat-lon@users.noreply.github.com >
* Fix line endings to LF
* Add hybrid approach for GPT-OSS compatibility
- Check for LoRA adapters before attempting LoRA abliteration
- Fall back to direct weight modification for nn.Parameter (GPT-OSS)
- Ensures compatibility across all model architectures
* Fix projector bug, update print statement, revert README
* Revert README changes to match upstream
* Fix import sorting for ruff
* Fix reload_model for evaluate_model, add type hints and validation
* Apply ruff formatting
* Replace load_in_4bit with quantization enum
* Fix precision loss: use FP32 refusal direction directly
* Move r assignment into non-LoRA path
* Fix linting: apply ruff formatting
* Add auto-merge for LoRA adapters on save/upload
* Fix linting: apply ruff formatting
* Implement CPU-based merge for 4-bit models with OOM fallback
* Remove use_lora flag (LoRA always on), add user prompt for 4-bit export
* Fix: PEFT target_modules expects module names without path prefix
* Fix linting: apply ruff formatting
* Add LoRA fallback and fix quantization_config handling
- Add try/except around LoRA initialization with fallback to direct weight modification
- Only pass quantization_config when not None (fixes gpt-oss loading)
- Use simple forward pass instead of generate() for model test (avoids chat template issues)
- Reset non-LoRA models by reloading in reload_model()
- Check self.use_lora before accessing LoRA adapters in abliterate()
* Add 8-bit quantization support via bitsandbytes
- Add BNB_8BIT option to QuantizationMethod enum
- Add --load-in-8bit CLI support (auto via pydantic-settings)
- Update documentation in config.py and config.default.toml
- Useful for mid-range VRAM (12-16 GB) as balance between memory and numeric stability
* Improve LoRA merge warning and fix linting
* Apply final ruff formatting
* Fix CI: apply ruff import sorting
* Use tiny model for CI efficiency
* Fix import sorting in test_lora.py
* Fix formatting in test_lora.py
* feat: Show merge warning for all models (requires high RAM)
* style: Apply ruff fixes
* Fix undefined Style import in main.py
* Fix(model): Support MoE/3D tensors and enforce dtype safety in abliterate
* Fix(ci): Format model.py with ruff
* Fix(main): Remove invalid style argument from prompt_select and unused import
* Fix logic errors, memory leak, and redundant merges in main.py
* Fix linting and formatting issues (isort, ruff)
* chore: Simplify .gitattributes as requested
* refactor: Remove defensive try-except around LoRA initialization
* chore: Update uv.lock with peft and bitsandbytes
* chore: Regenerate uv.lock to include missing peft dependency
* style: Fix import sorting (isort) for CI compliance
* style: Simplify .gitattributes to single line as requested
* Address PR #60 feedback: Remove caching, fix LoRA reload, global LoRA usage, style fixes
* Address PR review comments: clarify code, fix quantization, rename method
- Add explanatory comments for warning suppression and gc behavior
- Remove redundant gc.collect() calls (empty_cache handles it)
- Fix output message order (ask merge strategy before 'Uploading...')
- Add comment explaining 8-bit quantization doesn't need compute_dtype
- Remove extra newline after dtype comment
- Add future-proofing note for hybrid layer support (#43 )
- Remove leftover comment in get_merged_model
- Delete test_lora.py (debug script, not a real test)
- Add comment explaining needs_reload flag purpose
- Extract quantization config into _get_quantization_config() helper
- Rename reload_model() to reset_model_for_trial() for clarity
- Fix reload_model to respect quantization config (fixes evaluate_model bug)
- Remove unused gc import
* Restore gc.collect() before empty_cache() for large models
* refactor: Remove LoRA fallback remnants, simplify code
- Remove use_lora flag (always true since LoRA is always applied)
- Remove isinstance(PeftModel) check in get_merged_model() (always true)
- Simplify reset_model_for_trial() by removing defensive try/except
- Remove redundant gc.collect() calls (empty_cache handles GC)
- Remove unused gc import from main.py
* Address p-e-w review feedback: rename reset_model, remove loaded_model_name, fix type hints, remove GPT-OSS MoE, update assertion
* Restore skip logic for non-LoRA modules and fix 4-bit base_layer.weight access
* Remove defensive lora_A check per review - get_layer_modules already filters
* Fix try_add: nest component init inside Module check, add assert for unexpected types
* Add note about module.weight assumption for type checking
* Change 'Reloading model' to 'Resetting model' in logging
---------
Co-authored-by: accemlcc <accemlcc@users.noreply.github.com >
Co-authored-by: mad-cat-lon <113548315+mad-cat-lon@users.noreply.github.com >
Co-authored-by: Hager <Michael.Hager@bruker.com >
2025-12-14 20:19:09 +05:30
9d1734855d
feat: avoid excessive low divergence iteration ( #73 )
...
* feat: adjust scoring to avoid useless iteration
Adjusts the scoring function to avoid targeting meaninglessly low KL divergences.
Below a threshold value, the KL divergence score switches to the refusal count.
Adds config option kl_divergence_target (defaulting to 0.01).
* fix: Clean up parameter selection in objective
Create variables for num_layers and last_layer_index
* Improves readability and makes choices explicit
* feat: Print the parameters of the selected model
2025-12-14 14:26:48 +05:30
740aab61ba
feat: add max_memory parameter to limit memory usage ( #83 )
...
* add max_memory parameter to limit memory usage
* Added to reload_model also
* forgot to add self
* Process max_memory once in __init__ and store it as an instance variable, then reuse it in both locations
2025-12-11 20:57:40 +05:30
d9f2b0407a
build: bump version to 1.1.0
2025-12-10 16:54:03 +05:30
ca783db6c9
docs: update README
2025-12-10 16:30:35 +05:30
6acccac994
feat: add progress bars for plotting operations
2025-12-10 13:07:34 +05:30
ac154a55a0
fix: suppress CoT output for thinking models
...
Ref #75
2025-12-09 11:54:08 +05:30
15781a8a0c
fix: skip common response prefix for thinking models
...
Ref #75
2025-12-09 08:25:10 +05:30
24c3aeb442
feat: turn boolean settings into CLI flags
2025-12-07 11:37:07 +05:30
ffbde3ac2a
fix: follow up after recent PRs
2025-12-07 10:26:16 +05:30
932d737edf
feat: add silhouette coefficient to residual geometry output
2025-12-07 08:48:38 +05:30
1f5e977f4f
Revert "perf: optimize abliteration matrix op ( #46 )" ( #74 )
...
This reverts commit 60bd531fde
2025-12-07 06:30:37 +05:30
da27ba8054
fix: always left-pad inputs, and avoid optimizing for empty responses
...
Fixes #70
Co-authored-by: arnomatic <acc@eml.cc >
2025-12-06 06:31:09 +05:30
baf5b0b0d1
feat: add geometric median to residual geometry output
2025-12-05 20:15:50 +05:30
eeb28b28c1
feat: add option to plot residual vectors
2025-12-04 14:22:29 +05:30
d836fb2da9
ci: add PR title lint ( #66 )
...
* ci: add PR title lint
* style: ending newline
---------
Co-authored-by: mad-cat-lon <113548315+mad-cat-lon@users.noreply.github.com >
2025-12-03 09:25:48 +05:30
60bd531fde
perf: optimize abliteration matrix op ( #46 )
...
* perf: optimize abliteration matrix op
* refactor: comments and var names correspond with arditi
* refactor: fix comments and improve var notation
* fix: accidental line change and improve comments
---------
Co-authored-by: mad-cat-lon <113548315+mad-cat-lon@users.noreply.github.com >
2025-12-02 08:13:43 +05:30
1f74ac2888
Guard against refusals in broken English ( #45 )
...
* Guard against refusals in broken English
* Normalize whitespace between words
2025-11-26 11:29:08 +05:30
63fc0e7d5a
feat: Add bfloat16 to default dtypes list ( #44 )
...
Co-authored-by: Vinay Umrethe <vinayumrethe99@gmail.com >
2025-11-25 12:22:52 +05:30
1efc4ee9e1
Featuring Notebook (Colab/Kaggle) Compatibility ( #42 )
...
* feat: Add hybrid UI for notebook compatibility
* Restore notebook detection logic
* fix: Improve notebook detection with env vars
* chore: cleanup
* chore: cleanup
* correct ruff format
* refactor: Address code review feedback
- Move password handling to prompt_password
- Use only_directories=True for save path prompt
- Simplify prompt_text arguments
---------
Co-authored-by: Vinay Umrethe <vinayumrethe99@gmail.com >
2025-11-24 19:46:39 +05:30
452b35e7b7
Add trust_remote_code configuration option ( #31 )
...
* Add `trust_remote_code` configuration option and apply it when loading models and tokenizers
* Default `trust_remote_code` to `None` and set it to `True` if previously `None` so the user wouldn't be asked multiple times
* Consistently access `trust_remote_code` from `self.settings` instead of the global `settings` object.
* Introduce `trusted_models` dictionary to manage and confirm `trust_remote_code` settings during model loading
* Assign `trust_remote_code` to `evaluate_model` in `trusted_models` instead of `model`
2025-11-24 06:27:44 +05:30
b79b8b1475
Improve support for loading local datasets ( #33 )
...
* Handle loading local datasets
* Reorder branches to avoid chain of negatives
2025-11-23 11:15:34 +05:30
83cbf0612a
Add option to print refusal geometry
2025-11-22 13:18:54 +05:30
c35f3031f8
Allow stopping the optimization process early with Ctrl+C
2025-11-21 10:11:00 +05:30
2e1bb4b655
Use PYTORCH_ALLOC_CONF instead of deprecated PYTORCH_CUDA_ALLOC_CONF ( #32 )
...
* Use `PYTORCH_ALLOC_CONF` instead of deprecated `PYTORCH_CUDA_ALLOC_CONF`
* style: reformat environment variable check
2025-11-21 07:27:28 +05:30
af02bc6ece
Fix support for MXFP4 quantized models with Triton tensors ( #28 )
...
When loading models with MXFP4 quantization (e.g., openai/gpt-oss-20b),
the transformers library uses Triton tensors to wrap the quantized weights.
These Triton tensors have a .data attribute containing the underlying
PyTorch tensor, but torch.is_tensor() returns False for them.
This caused a KeyError: 'mlp.down_proj' when trying to load such models,
as the try_add() function would fail the assertion check before adding
the down projection matrices.
The fix extracts the underlying PyTorch tensor via the .data attribute
when encountering Triton tensors, allowing heretic to work with MXFP4
quantized models while maintaining full compatibility with standard models.
Tested with openai/gpt-oss-20b on PyTorch 2.9.1+cu130, transformers 4.57.1,
triton 3.5.1, and kernels 0.11.0.
2025-11-20 13:43:06 +05:30
22a4a5b5b5
Add citation information to README
2025-11-19 12:14:17 +05:30
694edf18d3
Follow up after recent PRs
2025-11-19 11:19:47 +05:30
c9c022a143
Fix linting issues
2025-11-19 10:16:58 +05:30
9905d9517f
Fix formatting issues
2025-11-19 10:04:43 +05:30
f06e939791
Add Ruff as a dev dependency
2025-11-19 09:59:18 +05:30
f3b9826ca4
Add CI workflow
2025-11-19 09:45:54 +05:30
13bb7b24d6
Fix KeyError when HuggingFace user profile fields are missing ( #20 )
...
Handle optional fullname and email fields in user profile gracefully
using .get() method with fallback values to prevent KeyError when
uploading models to HuggingFace.
This fixes an issue where users without a public email or fullname
set in their HuggingFace profile would encounter an error during
the upload process.
Co-authored-by: ricyoung <riyoung@gmail.com >
2025-11-19 05:32:50 +05:30
c8b6663b93
Fix multi-GPU support and memory management ( #17 )
...
* Ensure projector is on the same device as the matrix for multi-GPU support
* Optimize memory management for loaded model weights
* Refactor memory management by removing unnecessary gc.collect() calls
* Optimize memory usage (#1 )
* Improve memory management by explicitly deleting model layers and optimizing projector usage
* Optimize memory management by explicitly deleting the model and forcing garbage collection
* Add back deleted `empty_cache` call
* Fix broken file
* Remove unnecessary deletions
* Remove unnecessary empty_cache() calls
* Remove unused import of gc
* Duplicate `gc.collect` call in `empty_cache()`
* Move additional `gc.collect` call in front of `torch.x.empty_cache`
2025-11-19 05:09:12 +05:30
61fdf72b42
Add support for Granite MoE Hybrid in model.py by including down projections for shared MLP and MoE experts ( #14 )
2025-11-18 08:32:58 +05:30
7bad84b4f1
perf: clear residuals after computing direction ( #15 )
...
Co-authored-by: mad-cat-lon <113548315+mad-cat-lon@users.noreply.github.com >
2025-11-17 22:18:22 +05:30
09730bad70
MPS support ( #5 )
...
* MPS support
* oops, added issue tracker.
* Delete .beads/issues.jsonl
2025-11-17 18:42:01 +05:30
b3545e4b1e
Fix retrieving package version
2025-11-16 17:35:13 +05:30
3f346b6150
Change package name
2025-11-16 17:01:50 +05:30
1a59d226c1
Fix spacing after images in README
2025-11-16 16:06:08 +05:30
12ecf50033
Add README
2025-11-16 15:19:27 +05:30
ea699dce46
Improve appearance of selection menus
2025-11-16 11:32:58 +05:30
8a1aceff11
Switch to multi-objective optimization
2025-11-14 18:04:23 +05:30
0bae27f359
Fix some of the problems with Falcon-E-3B
2025-11-13 11:39:08 +05:30
e24080db64
Add metadata to pyproject.toml
2025-11-02 10:06:15 +05:30
fae39ffb89
Move default configuration to Python
2025-11-02 09:29:55 +05:30
850c21b534
Make multivariate TPE work properly
2025-11-01 16:57:12 +05:30
a24e6eba96
Improve optimization
2025-10-31 16:04:28 +05:30
a9655c8d31
Perform calculations involving residual vectors in float32
...
Credit to Jim Lai for pointing out potential numerical problems in https://huggingface.co/blog/grimjim/projected-abliteration
2025-10-31 13:47:24 +05:30
michaelh