Commit Graph

11821 Commits

Author SHA1 Message Date
Yong Kang
1568e7ff8a perf: reuse proof workers across block 2025-10-14 09:36:10 +00:00
Yong Kang
1619408dfc fmt 2025-10-13 10:05:19 +00:00
Yong Kang
1e9874b0ad remove active handle 2025-10-13 10:01:47 +00:00
Yong Kang
d8c855932a remove wrapper 2025-10-13 09:57:46 +00:00
YK
45765ac50a Merge branch 'yk/worker_pool_acc' into yk/pool_clean 2025-10-13 17:31:59 +08:00
Yong Kang
bc4ecf5077 addressed sync overhead
- Removed lock-free atomic counters from ProofTaskMetrics and replaced them with direct method calls for recording blinded node counts.
- Updated storage and account worker loops to utilize the new metrics recording methods, enhancing clarity and maintainability.
- Simplified the ProofTaskManagerHandle by removing unnecessary metrics fields, streamlining the overall structure.
2025-10-13 09:30:35 +00:00
Yong Kang
f9e167ec39 refactor: replace spawn_proof_workers with ProofTaskManagerHandle
- Updated the code to utilize ProofTaskManagerHandle for spawning proof workers instead of the deprecated spawn_proof_workers function.
- This change enhances code clarity and maintainability by consolidating the worker management logic within the ProofTaskManagerHandle struct.
2025-10-13 09:19:28 +00:00
Yong Kang
f692d75be0 fmt 2025-10-13 08:40:20 +00:00
Yong Kang
18ff58d146 fix clippy 2025-10-13 07:43:25 +00:00
YK
3e957a5f9e refactor(tree): remove unused Factory generic from multiproof system (#18933) 2025-10-13 13:13:34 +08:00
Yong Kang
6a4766ce47 fix clippy 2025-10-13 05:07:38 +00:00
Yong Kang
c68eb7a9fd refactor: simplify worker count handling in TreeConfig
- Removed the `clamp_worker_count` function and replaced its logic with direct usage of `max` in the setter methods for storage and account worker counts.
- This change enhances code clarity and reduces unnecessary function overhead while ensuring the minimum worker count is enforced.
2025-10-13 02:11:29 +00:00
Yong Kang
2698cc8007 refactor: enhance error handling in trie_node method
- Removed redundant helper functions for error conversion in ProofTaskTrieNodeProvider.
- Simplified error handling by directly mapping errors to SparseTrieError, improving code clarity and maintainability.
2025-10-13 02:11:12 +00:00
Yong Kang
c02a68dc78 refactor: improve error handling in trie_node method
- Updated the error conversion helper function in ProofTaskTrieNodeProvider to directly wrap the ProviderError, enhancing clarity and maintainability.
- This change simplifies the error handling logic within the trie_node method.
2025-10-10 12:43:21 +00:00
Yong Kang
e49791e9ae refactor: simplify ProofTaskMetrics default implementation
- Updated the ProofTaskMetrics struct to derive Default, removing the manual implementation of the default method.
- This change enhances code clarity and reduces boilerplate, while maintaining the same functionality.
2025-10-10 12:42:50 +00:00
Yong Kang
40bb506162 refactor: consolidate proof task handles in MultiproofManager
- Merged separate storage and account proof task handles into a single proof task handle for improved code clarity and maintainability.
- Updated related methods to utilize the consolidated handle, streamlining the management of proof tasks.
2025-10-10 12:42:32 +00:00
Yong Kang
833b031025 refactor: streamline worker count validation
- Introduced a `clamp_worker_count` function to centralize the logic for enforcing the minimum worker count.
- Updated setter methods in `TreeConfig` to utilize the new clamping function, improving code readability and maintainability.
2025-10-10 11:04:50 +00:00
Yong Kang
4e009444d1 merge 2025-10-10 11:01:58 +00:00
YK
d1eb0ecd5c Update crates/trie/parallel/src/proof_task.rs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-10 18:58:31 +08:00
Yong Kang
f302447480 fix count 2025-10-10 10:55:40 +00:00
Yong Kang
3f6400eb8c refactor: improve error handling in ProofTaskTrieNodeProvider
- Introduced helper functions to streamline error conversion from ProviderError and channel receive errors to SparseTrieError.
- Enhanced readability and maintainability of the trie_node method by reducing repetitive error handling code.
2025-10-10 10:50:54 +00:00
Yong Kang
2b90133ca0 fmt 2025-10-10 10:39:25 +00:00
Yong Kang
8e00a4a5b9 clippy 2025-10-10 10:39:12 +00:00
Yong Kang
42ceeddef7 fix: prevent active_handles underflow in ProofTaskManagerHandle
- Added a debug assertion to ensure active_handles does not underflow when dropping a ProofTaskManagerHandle.
- Implemented metrics recording to flush before exit when the last handle is dropped, enhancing monitoring capabilities.
2025-10-10 10:34:14 +00:00
Yong Kang
0b18f6488a feat: introduce minimum worker count configuration
- Added a constant `MIN_WORKER_COUNT` to enforce a minimum number of workers for storage and account proof tasks.
- Updated `default_storage_worker_count` and `default_account_worker_count` functions to utilize the new minimum constraint.
- Enhanced setter methods in `TreeConfig` to ensure worker counts do not fall below the minimum.
- Modified command-line argument parsing to validate worker counts against the minimum requirement.
2025-10-10 10:26:56 +00:00
Yong Kang
d44180dc1b fix comment 2025-10-10 10:25:49 +00:00
Yong Kang
ed45ebda8a refactor: yeet proof task manager 2025-10-10 10:18:58 +00:00
Yong Kang
3516451740 refactor: update proof task management to use spawn_proof_workers
- Replaced the ProofTaskManager with a new spawn_proof_workers function for better clarity and maintainability.
- Updated related code to utilize the new function, simplifying the worker spawning process.
- Enhanced metrics tracking for storage and account proof requests, ensuring thread-safe operations.
- Improved error handling and code structure across proof task implementations.
2025-10-10 09:49:05 +00:00
Yong Kang
44bca43515 made count same as storage worker 2025-10-10 09:17:07 +00:00
Yong Kang
14def0761e fmt 2025-10-10 09:11:30 +00:00
YK
870938f7f3 Merge branch 'main' into yk/worker_pool_acc 2025-10-10 17:00:55 +08:00
Yong Kang
0a88a18203 fmt 2025-10-10 08:16:45 +00:00
Yong Kang
47da382d01 clippy 2025-10-10 08:12:09 +00:00
Yong Kang
11130c7fce refactor: simplify MultiProofConfig and related structures
- Removed the generic Factory type from MultiProofConfig and related structs, streamlining their definitions and improving code clarity.
- Updated methods to reflect the removal of the Factory type, enhancing maintainability.
- Adjusted the implementation of PendingMultiproofTask and its associated methods to eliminate unnecessary type parameters, simplifying the codebase.
2025-10-10 08:01:38 +00:00
YK
397a30defb perf(tree): worker pooling for storage in multiproof generation (#18887)
Co-authored-by: Brian Picciano <me@mediocregopher.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Alexey Shekhirin <5773434+shekhirin@users.noreply.github.com>
2025-10-10 07:58:15 +00:00
Yong Kang
444b24c7e9 refactor: remove unused Factory type from proof tests
- Eliminated the Factory type definition from the proof tests, simplifying the code structure.
- This change contributes to improved clarity and maintainability of the test implementation.
2025-10-10 05:59:30 +00:00
Yong Kang
d5fedd80ce removing from test 2025-10-10 05:59:03 +00:00
Yong Kang
47398ce49a refactor: simplify ParallelProof struct by removing generic Factory type
- Removed the Factory type parameter from the ParallelProof struct, streamlining its definition and implementation.
- Updated the constructor and related methods to reflect this change, enhancing code clarity and maintainability.
- Eliminated unused PhantomData field, reducing complexity in the struct's design.
2025-10-10 05:57:19 +00:00
Yong Kang
f23cfab54f refactor: addressed dan's comment on making structs that encapsulate loops:
- an account worker / loop
- a storage worker / loop

- streamline worker spawning functions in ProofTaskManager

- Consolidated worker spawning logic by renaming and simplifying the `spawn_worker_pool` function to `spawn_storage_workers` and `spawn_account_workers`, enhancing clarity and maintainability.
- Removed the `WorkerType` enum and associated logic, as the worker type is now inferred from the context, reducing complexity.
- Updated comments to better describe the purpose and parameters of the new functions, improving code documentation.
2025-10-10 04:56:15 +00:00
YK
5a69925ce0 Merge branch 'yk/worker_pool_storage_2' into yk/worker_pool_acc 2025-10-10 12:41:58 +08:00
Yong Kang
1bc2862d17 move the AccountMultiproofParams struct to just below AccountMultiproofInput so all account multiproof types live together. 2025-10-10 04:35:16 +00:00
Yong Kang
aae5c7a5ac refactor: simplify account multiproof function parameters
- Introduced `AccountMultiproofParams` struct to encapsulate parameters for building account multiproofs, reducing the number of arguments in the `build_account_multiproof_with_storage_roots` function.
- Updated the `account_worker_loop` to use the new struct, improving code readability and maintainability.
2025-10-10 04:32:58 +00:00
Yong Kang
05f177adf9 added a todo 2025-10-10 04:24:41 +00:00
Yong Kang
5d109344c0 feat: add storage and account worker count configuration options
- Introduced `storage_worker_count` and `account_worker_count` fields in `EngineArgs` to allow configuration of the number of workers in the Tokio blocking pool.
- Updated the `tree_config` method to incorporate these new fields, enhancing flexibility in managing proof worker resources.
- Updated CLI documentation to reflect the new configuration options for improved user guidance.
2025-10-10 04:24:25 +00:00
Yong Kang
8e64738b03 docs: clarify storage-worker-count uses Tokio blocking pool 2025-10-10 00:21:52 +00:00
Yong Kang
4aba3de076 docs: update CLI reference for storage-worker-count flag 2025-10-10 00:19:27 +00:00
Léa Narzis
d2070f4de3 feat: wait for new blocks when build is in progress (#18831)
Co-authored-by: Roman Hodulák <roman.hodulak@polyglot-software.com>
2025-10-09 16:42:59 +00:00
Yong Kang
d66ed61397 cli flag for storage 2025-10-09 14:57:53 +00:00
Yong Kang
4ca404e517 bump up workers 2025-10-09 14:35:13 +00:00
emmmm
6f96a32812 chore: align node_config threshold constant (#18914) 2025-10-09 11:18:49 +00:00