…for deduplication- Added `score_columns` parameter to `SemHash` for scoring records based on specified columns.- Implemented `_score` method to scale and compute scores for records.- Updated `deduplicate` and `self_deduplicate` methods to incorporate scoring and budget constraints.- Refactored initialization and documentation to reflect new parameters and functionality.

- Removed unnecessary score cleanup from the records after sorting.- Moved the scoring logic to ensure records are sorted by score before deduplication.- Improved code clarity by consolidating score handling within the deduplication process.

- Renamed `_score` method to `_sort_and_scale_scores` to better reflect its functionality.- Updated references to the scoring method in the deduplication process for improved readability and accuracy.

- Added a new method `compute_nearest_neighbor_alignment_scores` to calculate embedding similarity based on nearest neighbor alignment.- Updated the SemHash initialization to optionally compute alignment scores when creating an instance.- Removed the unused `score_columns` parameter from the SemHash constructor for clarity.- Enhanced the deduplication process by integrating alignment scoring, improving overall functionality and performance.

…lt dataclass- Added a new `FilterResult` dataclass to encapsulate the results of filtering operations, including selected and filtered records along with their scores.- Implemented `filter_by_score` and `self_filter_by_score` methods in the `SemHash` class to filter records based on their scores, allowing for budget constraints and sorting options.- Updated the `Index` class to replace the deprecated `compute_nearest_neighbor_alignment_scores` method with a more streamlined `query_top_k` method for querying top-k records.- Removed unused parameters and methods to enhance code clarity and maintainability.

- Replaced `filter_by_score` with `filter_by_entropy` to compute record diversity using scipy's entropy function- Modified `query_top_k` method to handle vector queries more robustly- Added scipy>=1.13.1 as a dependency in pyproject.toml- Updated README with new entropy filtering examples and documentation- Renamed methods to use `entropy` instead of `score` for clarity- Improved filtering functionality with more intuitive parameters like `descending` and `k`

- Implemented tests for `self_filter_by_entropy` method with various scenarios- Added test cases for absolute and percentage-based budget filtering- Verified sorting order with ascending and descending entropy options- Included validation tests for invalid budget inputs- Tested string and dictionary input compatibility for entropy filtering

- Replaced `filter_by_score` with `filter_by_entropy` in README documentation- Clarified description of entropy-based filtering method- Maintained existing explanation of filtering functionality

…ation- Removed scipy>=1.13.1 from project dependencies- Created a custom `entropy_from_distances` function in utils.py to replace scipy's entropy function- Updated `semhash.py` to use the new entropy calculation method- Reshaped vector inputs to ensure compatibility with `query_top_k` method- Simplified entropy calculation with a more explicit implementation

- Renamed `deduplicated` to `selected` and `duplicates` to `filtered`- Added deprecation warnings for old field names- Updated methods to use new field names while maintaining backward compatibility- Implemented `__post_init__` to handle field migrations and warnings

- Enhanced deprecation warnings for `deduplicated` and `duplicates` fields to indicate their removal in the upcoming `semhash==0.3.0` release.- Added a new test to verify the deprecation warnings and ensure backward compatibility with existing functionality.- Cleaned up imports in the test file for better organization.

- Replaced instances of `deduplicated` with `selected` and `duplicates` with `filtered` in the README documentation to align with recent refactoring.- Ensured that examples and explanations are consistent with the updated field names for clarity and accuracy.

- Replaced instances of `deduplicated` with `selected` and `duplicates` with `filtered` across various files to align with recent refactoring.- Updated tests to ensure they reflect the new field names and maintain functionality.- Adjusted README examples to provide accurate documentation for the updated field names.