Arnab/swiper #899
Arnab/swiper #899
Conversation
Adds the optional 'super-swiper' feature to fastcrypto-tbls, adding dependencies on a local solver and num-rational. Implements Nodes::new_super_swiper_reduced for advanced weight reduction using the super_swiper algorithm, and provides a dedicated test for this functionality with realistic validator weights.
Adds a copy of the 'weight-reduction' repo under fastcrypto-tbls, including solver algorithms, utilities, tests, and datasets. Includes Cargo.toml, license, documentation, and data files for reproducibility and usage as described in the referenced research paper.
Added scripts to fetch all Sui validators and generate CSVs of stake weights and ticket assignments under src/weight-reduction/scripts. Integrated super_swiper_test.rs as a conditional test module and updated its path and imports. Added a new data file (sui_stakes_and_tickets.csv) for validator stake/ticket analysis.
Added sui_real_all.dat and sui_real_all_details.txt containing Sui validator stake amounts and corresponding validator names for use in weight reduction analysis.
Removed unused parameters 't' and 'total_weight_lower_bound' from Nodes::new_super_swiper_reduced and updated threshold calculation to use beta ratio directly. Updated tests to match new API and added a new test for additional weight scenarios.
Deleted several solver implementations and their associated tests from the weight-reduction module, including bruteforce_sorted, faster_swiper, gcd_wr, and swiper. Updated mod.rs to only include super_swiper. This streamlines the codebase by removing unused or deprecated algorithms and their test files.
The 'super-swiper' feature flag and related optional dependencies were removed, making the super_swiper functionality always available.
Introduces the weight_reduction_checks module for validating weight reduction properties, and updates Nodes::new_super_swiper_reduced to iteratively adjust beta to meet a total weight upper bound and validate reductions. Test cases are expanded to cover new validation logic, CSV output, and upper bound scenarios.
Updated Nodes::new_super_swiper_reduced to use a slack-based approach for weight reduction validation, replacing the previous total weight upper bound check. The function now iterates beta to satisfy a configurable slack constraint, and returns additional beta parameters. Tests and weight_reduction_checks were updated to support and verify the new slack-based logic.
Changed the Nodes::new_super_swiper_reduced function to accept an absolute threshold 't' instead of an alpha ratio, computing alpha internally. Updated related test to pass 't' and clarified output for threshold and alpha calculation.
Replaces slack-based validation with delta-based checks in super swiper weight reduction logic. Aligns interface with new_reduced.
Introduces a test for the new_reduced function using an 8% delta constraint on the original total weight. The test verifies weight reduction, threshold adjustment, node ID preservation, and writes results to a CSV file for further analysis.
Added a script to fetch all Sui validators' voting power and save it to a new data file. Integrated the new voting power data into super_swiper_test.rs, updating tests to use the new dataset and output results to new CSV files.
Introduces voting power data files for epochs 100, 200, 400, 800, and renames the previous file to epoch_974. Updates test helpers and adds a comprehensive test to compare weight reduction algorithms across all epochs. The fetch script is enhanced to support epoch selection via command line and outputs files with epoch-specific names.
| i: usize, | ||
| } | ||
|
|
||
| impl Ord for QueueElement { |
There was a problem hiding this comment.
You can just derive Ord and PartialOrd here since that automatically does lexicographical ordering from the first field and down.
| } | ||
|
|
||
| fn get(&self, index: usize) -> u64 { | ||
| match self.tickets.get(index) { |
There was a problem hiding this comment.
This is the same as unwrap_or
| } | ||
|
|
||
| fn update(&mut self, index: usize) { | ||
| while index >= self.tickets.len() { |
There was a problem hiding this comment.
Here, you can use self.tickets.resize instead.
| 0 | ||
| } | ||
| fn adv_tickets_target(&self) -> u64 { | ||
| self.dp.capacity() as u64 |
There was a problem hiding this comment.
Do you mean length here? I'm not a 100% sure, but I don't think you can guarantee anything about the capacity except that it's larger than what you've requested.
| // Returns the maximum achievable adversarial number of tickets. | ||
| #[cfg(test)] | ||
| fn adversarial_tickets(&self) -> u64 { | ||
| for (t, &w) in self.dp.iter().enumerate().rev() { |
There was a problem hiding this comment.
You can use find_position here.
| }) | ||
| } | ||
|
|
||
| // Apply an element with weight w and t tickets. Returns None iff |
There was a problem hiding this comment.
Function docs should start with a triple dash, ///
|
|
||
| impl<'a> Generator<'a> { | ||
| fn new(weights: &'a [u64], c: Ratio) -> Self { | ||
| assert!(!weights.is_empty()); |
There was a problem hiding this comment.
For at least some of these, we'd probably want to return an error instead of asserting
|
|
||
| let mut queue = BinaryHeap::new(); | ||
| queue.push(QueueElement { | ||
| s: (Ratio::from_integer(1) - c) / Ratio::from_integer(*weights.first().unwrap()), |
There was a problem hiding this comment.
nit: It looks like there's an Ratio::one function
| indices_tail.sort_unstable_by(|a, b| b.cmp(a)); | ||
| indices_tail.dedup(); | ||
|
|
||
| let mut indices_head: Vec<_> = (0..tickets_len).collect(); |
There was a problem hiding this comment.
Using swap_remove here may modify the order of the remaining elements, so is this safe? Also, isn't there a better way to do this instead of creating a vector and then remove elements?
| let mut g = generate_deltas(weights, alpha); | ||
|
|
||
| let mut batch_size: usize = 1; | ||
| loop { |
There was a problem hiding this comment.
Can you write a note about why this doesn't run forever?
| &self.tickets | ||
| } | ||
|
|
||
| fn extract_data(self) -> Vec<u64> { |
There was a problem hiding this comment.
It's common to use the name to_vec for something like this and as_slice for the one above.
| if tt >= adv_tickets_target { | ||
| return None; | ||
| } | ||
| while self.dp.len() <= tt { |
| } | ||
|
|
||
| let mut dp = Vec::with_capacity(adv_tickets_target); | ||
| for &x in self.dp.iter().take(adv_tickets_target) { |
There was a problem hiding this comment.
You can just collect the iterator as dp here
| // Copyright (c) 2022, Mysten Labs, Inc. | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| // The implementation of the algorithms presented in the paper |
There was a problem hiding this comment.
Module level docs should start with //!
|
|
||
| for i in (1..self.dp.len()).rev() { | ||
| if self.dp[i] != 0 { | ||
| let ww = self.dp[i] + w; |
There was a problem hiding this comment.
Are these names, ww and tt, from the paper? I find them a bit confusing
| } | ||
|
|
||
| let t = t as usize; | ||
| while self.dp.len() <= t { |
| .unwrap(); | ||
|
|
||
| let indices_head = calc_indices_head(tickets.data().len(), deltas); | ||
| for index in indices_head { |
There was a problem hiding this comment.
nit: This can be done with an iterator and the reduce function, but it's perhaps a matter of taste if it's cleaner
| let mut dp = dp_head.make_copy(adv_tickets_target)?; | ||
|
|
||
| let exclude_indices = exclude_indices.iter().copied().collect::<BTreeSet<_>>(); | ||
| let add_indices = add_indices.iter().copied().collect::<BTreeSet<_>>(); |
There was a problem hiding this comment.
Are these already sorted? In that case you don't need to copy into a sorted set but can just use the binary_search function
|
|
||
| for index in add_indices { | ||
| if !exclude_indices.contains(&index) { | ||
| dp = dp.apply(weights[index], tickets.get(index))?; |
There was a problem hiding this comment.
This can be done with an iterator and the filter and reduce functions
| Some(dp) | ||
| } | ||
|
|
||
| fn update_tickets(deltas: &[usize], tickets: &mut Tickets) { |
There was a problem hiding this comment.
Why is this (and perhaps also the functions below) not a function defined on Tickets?
2 participants
This PR adds a "super_swiper" algorithm for weight reduction in the
fastcrypto-tblscrate, adds supporting infrastructure for weight reduction experiments, and enhances the codebase with tools and data for working with real validator sets.The super_swiper code is adapted from: https://github.com/tolikzinovyev/weight-reduction
The implementation of the algorithms presented in the paper:
Weight reduction in distributed protocols: new algorithms and analysis, by Anatoliy Zinovyev
(https://eprint.iacr.org/2025/1076).