github/AutoGPT
1
1
Fork 0
mirror of https://github.com/Significant-Gravitas/AutoGPT.git synced 2026-02-16 17:55:55 -05:00
Code Issues Packages Projects Releases Wiki Activity

feat(backend/blocks): enhance list concatenation with advanced operations (#12105)

Browse Source 
## Summary

Enhances the existing `ConcatenateListsBlock` and adds five new
companion blocks for comprehensive list manipulation, addressing issue
#11139 ("Implement block to concatenate lists").

### Changes

- **Enhanced `ConcatenateListsBlock`** with optional deduplication
(`deduplicate`) and None-value filtering (`remove_none`), plus an output
`length` field
- **New `FlattenListBlock`**: Recursively flattens nested list
structures with configurable `max_depth`
- **New `InterleaveListsBlock`**: Round-robin interleaving of elements
from multiple lists
- **New `ZipListsBlock`**: Zips corresponding elements from multiple
lists with support for padding to longest or truncating to shortest
- **New `ListDifferenceBlock`**: Computes set difference between two
lists (regular or symmetric)
- **New `ListIntersectionBlock`**: Finds common elements between two
lists, preserving order

### Helper Utilities

Extracted reusable helper functions for validation, flattening,
deduplication, interleaving, chunking, and statistics computation to
support the blocks and enable future reuse.

### Test Coverage

Comprehensive test suite with 188 test functions across 29 test classes
covering:
- Built-in block test harness validation for all 6 blocks
- Manual edge-case tests for each block (empty inputs, large lists,
mixed types, nested structures)
- Internal method tests for all block classes
- Unit tests for all helper utility functions

Closes #11139

## Test plan

- [x] All files pass Python syntax validation (`ast.parse`)
- [x] Built-in `test_input`/`test_output` tests defined for all blocks
- [x] Manual tests cover edge cases: empty lists, large lists, mixed
types, nested structures, deduplication, None removal
- [x] Helper function tests validate all utility functions independently
- [x] All block IDs are valid UUID4
- [x] Block categories set to `BlockCategory.BASIC` for consistency with
existing list blocks


<!-- greptile_comment -->

<h2>Greptile Overview</h2>

<details><summary><h3>Greptile Summary</h3></summary>

Enhanced `ConcatenateListsBlock` with deduplication and None-filtering
options, and added five new list manipulation blocks
(`FlattenListBlock`, `InterleaveListsBlock`, `ZipListsBlock`,
`ListDifferenceBlock`, `ListIntersectionBlock`) with comprehensive
helper functions and test coverage.

**Key Changes:**
- Enhanced `ConcatenateListsBlock` with `deduplicate` and `remove_none`
options, plus `length` output field
- Added `FlattenListBlock` for recursively flattening nested lists with
configurable `max_depth`
- Added `InterleaveListsBlock` for round-robin element interleaving
- Added `ZipListsBlock` with support for padding/truncation
- Added `ListDifferenceBlock` and `ListIntersectionBlock` for set
operations
- Extracted 12 reusable helper functions for validation, flattening,
deduplication, etc.
- Comprehensive test suite with 188 test functions covering edge cases

**Minor Issues:**
- Helper function `_deduplicate_list` has redundant logic in the `else`
branch that duplicates the `if` branch
- Three helper functions (`_filter_empty_collections`,
`_compute_list_statistics`, `_chunk_list`) are defined but unused -
consider removing unless planned for future use
- The `_make_hashable` function uses `hash(repr(item))` for unhashable
types, which correctly treats structurally identical dicts/lists as
duplicates
</details>


<details><summary><h3>Confidence Score: 4/5</h3></summary>

- Safe to merge with minor style improvements recommended
- The implementation is well-structured with comprehensive test coverage
(188 tests), proper error handling, and follows existing block patterns.
All blocks use valid UUID4 IDs and correct categories. The helper
functions provide good code reuse. The minor issues are purely stylistic
(redundant code, unused helpers) and don't affect functionality or
safety.
- No files require special attention - both files are well-tested and
follow project conventions
</details>


<details><summary><h3>Sequence Diagram</h3></summary>

```mermaid
sequenceDiagram
    participant User
    participant Block as List Block
    participant Helper as Helper Functions
    participant Output
    
    User->>Block: Input (lists/parameters)
    Block->>Helper: _validate_all_lists()
    Helper-->>Block: validation result
    
    alt validation fails
        Block->>Output: error message
    else validation succeeds
        Block->>Helper: _concatenate_lists_simple() / _flatten_nested_list() / etc.
        Helper-->>Block: processed result
        
        opt deduplicate enabled
            Block->>Helper: _deduplicate_list()
            Helper-->>Block: deduplicated result
        end
        
        opt remove_none enabled
            Block->>Helper: _filter_none_values()
            Helper-->>Block: filtered result
        end
        
        Block->>Output: result + length
    end
    
    Output-->>User: Block outputs
```
</details>


<sub>Last reviewed commit: a6d5445</sub>

<!-- greptile_other_comments_section -->

<sub>(2/5) Greptile learns from your feedback when you react with thumbs
up/down!</sub>

<!-- /greptile_comment -->

---------

Co-authored-by: Otto <otto@agpt.co>
This commit is contained in:
Eve
2026-02-16 00:39:53 -05:00
committed by GitHub
parent 27d94e395c
commit 647c8ed8d4
4 changed files with 2164 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
docs/integrations/README.md
View File

@@ -56,12 +56,16 @@ Below is a comprehensive list of all available blocks, categorized by their prim
| [File Store](block-integrations/basic.md#file-store) | Downloads and stores a file from a URL, data URI, or local path |
| [Find In Dictionary](block-integrations/basic.md#find-in-dictionary) | A block that looks up a value in a dictionary, list, or object by key or index and returns the corresponding value |
| [Find In List](block-integrations/basic.md#find-in-list) | Finds the index of the value in the list |
| [Flatten List](block-integrations/basic.md#flatten-list) | Flattens a nested list structure into a single flat list |
| [Get All Memories](block-integrations/basic.md#get-all-memories) | Retrieve all memories from Mem0 with optional conversation filtering |
| [Get Latest Memory](block-integrations/basic.md#get-latest-memory) | Retrieve the latest memory from Mem0 with optional key filtering |
| [Get List Item](block-integrations/basic.md#get-list-item) | Returns the element at the given index |
| [Get Store Agent Details](block-integrations/system/store_operations.md#get-store-agent-details) | Get detailed information about an agent from the store |
| [Get Weather Information](block-integrations/basic.md#get-weather-information) | Retrieves weather information for a specified location using OpenWeatherMap API |
| [Human In The Loop](block-integrations/basic.md#human-in-the-loop) | Pause execution for human review |
| [Interleave Lists](block-integrations/basic.md#interleave-lists) | Interleaves elements from multiple lists in round-robin fashion, alternating between sources |
| [List Difference](block-integrations/basic.md#list-difference) | Computes the difference between two lists |
| [List Intersection](block-integrations/basic.md#list-intersection) | Computes the intersection of two lists, returning only elements present in both |
| [List Is Empty](block-integrations/basic.md#list-is-empty) | Checks if a list is empty |
| [List Library Agents](block-integrations/system/library_operations.md#list-library-agents) | List all agents in your personal library |
| [Note](block-integrations/basic.md#note) | A visual annotation block that displays a sticky note in the workflow editor for documentation and organization purposes |
@@ -84,6 +88,7 @@ Below is a comprehensive list of all available blocks, categorized by their prim
| [Store Value](block-integrations/basic.md#store-value) | A basic block that stores and forwards a value throughout workflows, allowing it to be reused without changes across multiple blocks |
| [Universal Type Converter](block-integrations/basic.md#universal-type-converter) | This block is used to convert a value to a universal type |
| [XML Parser](block-integrations/basic.md#xml-parser) | Parses XML using gravitasml to tokenize and coverts it to dict |
| [Zip Lists](block-integrations/basic.md#zip-lists) | Zips multiple lists together into a list of grouped elements |
## Data Processing

197
docs/integrations/block-integrations/basic.md
View File

@@ -637,7 +637,7 @@ This enables extensibility by allowing custom blocks to be added without modifyi
## Concatenate Lists
### What it is
Concatenates multiple lists into a single list. All elements from all input lists are combined in order.
Concatenates multiple lists into a single list. All elements from all input lists are combined in order. Supports optional deduplication and None removal.
### How it works
<!-- MANUAL: how_it_works -->
@@ -651,6 +651,8 @@ The block includes validation to ensure each item is actually a list. If a non-l
| Input | Description | Type | Required |
|-------|-------------|------|----------|
| lists | A list of lists to concatenate together. All lists will be combined in order into a single list. | List[List[Any]] | Yes |
| deduplicate | If True, remove duplicate elements from the concatenated result while preserving order. | bool | No |
| remove_none | If True, remove None values from the concatenated result. | bool | No |
### Outputs
@@ -658,6 +660,7 @@ The block includes validation to ensure each item is actually a list. If a non-l
|--------|-------------|------|
| error | Error message if concatenation failed due to invalid input types. | str |
| concatenated_list | The concatenated list containing all elements from all input lists in order. | List[Any] |
| length | The total number of elements in the concatenated list. | int |
### Possible use case
<!-- MANUAL: use_case -->
@@ -820,6 +823,45 @@ This enables conditional logic based on list membership and helps locate items f
---
## Flatten List
### What it is
Flattens a nested list structure into a single flat list. Supports configurable maximum flattening depth.
### How it works
<!-- MANUAL: how_it_works -->
This block recursively traverses a nested list and extracts all leaf elements into a single flat list. You can control how deep the flattening goes with the max_depth parameter: set it to -1 to flatten completely, or to a positive integer to flatten only that many levels.
The block also reports the original nesting depth of the input, which is useful for understanding the structure of data coming from sources with varying levels of nesting.
<!-- END MANUAL -->
### Inputs
| Input | Description | Type | Required |
|-------|-------------|------|----------|
| nested_list | A potentially nested list to flatten into a single-level list. | List[Any] | Yes |
| max_depth | Maximum depth to flatten. -1 means flatten completely. 1 means flatten only one level. | int | No |
### Outputs
| Output | Description | Type |
|--------|-------------|------|
| error | Error message if flattening failed. | str |
| flattened_list | The flattened list with all nested elements extracted. | List[Any] |
| length | The number of elements in the flattened list. | int |
| original_depth | The maximum nesting depth of the original input list. | int |
### Possible use case
<!-- MANUAL: use_case -->
**Normalizing API Responses**: Flatten nested JSON arrays from different API endpoints into a uniform single-level list for consistent processing.
**Aggregating Nested Results**: Combine results from recursive file searches or nested category trees into a flat list of items for display or export.
**Data Pipeline Cleanup**: Simplify deeply nested data structures from multiple transformation steps into a clean flat list before final output.
<!-- END MANUAL -->
---
## Get All Memories
### What it is
@@ -1012,6 +1054,120 @@ This enables human oversight at critical points in automated workflows, ensuring
---
## Interleave Lists
### What it is
Interleaves elements from multiple lists in round-robin fashion, alternating between sources.
### How it works
<!-- MANUAL: how_it_works -->
This block takes elements from each input list in round-robin order, picking one element from each list in turn. For example, given `[[1, 2, 3], ['a', 'b', 'c']]`, it produces `[1, 'a', 2, 'b', 3, 'c']`.
When lists have different lengths, shorter lists stop contributing once exhausted, and remaining elements from longer lists continue to be added in order.
<!-- END MANUAL -->
### Inputs
| Input | Description | Type | Required |
|-------|-------------|------|----------|
| lists | A list of lists to interleave. Elements will be taken in round-robin order. | List[List[Any]] | Yes |
### Outputs
| Output | Description | Type |
|--------|-------------|------|
| error | Error message if interleaving failed. | str |
| interleaved_list | The interleaved list with elements alternating from each input list. | List[Any] |
| length | The total number of elements in the interleaved list. | int |
### Possible use case
<!-- MANUAL: use_case -->
**Balanced Content Mixing**: Alternate between content from different sources (e.g., mixing promotional and organic posts) for a balanced feed.
**Round-Robin Scheduling**: Distribute tasks evenly across workers or queues by interleaving items from separate task lists.
**Multi-Language Output**: Weave together translated text segments with their original counterparts for side-by-side comparison.
<!-- END MANUAL -->
---
## List Difference
### What it is
Computes the difference between two lists. Returns elements in the first list not found in the second, or symmetric difference.
### How it works
<!-- MANUAL: how_it_works -->
This block compares two lists and returns elements from list_a that do not appear in list_b. It uses hash-based lookup for efficient comparison. When symmetric mode is enabled, it returns elements that are in either list but not in both.
The order of elements from list_a is preserved in the output, and elements from list_b are appended when using symmetric difference.
<!-- END MANUAL -->
### Inputs
| Input | Description | Type | Required |
|-------|-------------|------|----------|
| list_a | The primary list to check elements from. | List[Any] | Yes |
| list_b | The list to subtract. Elements found here will be removed from list_a. | List[Any] | Yes |
| symmetric | If True, compute symmetric difference (elements in either list but not both). | bool | No |
### Outputs
| Output | Description | Type |
|--------|-------------|------|
| error | Error message if the operation failed. | str |
| difference | Elements from list_a not found in list_b (or symmetric difference if enabled). | List[Any] |
| length | The number of elements in the difference result. | int |
### Possible use case
<!-- MANUAL: use_case -->
**Change Detection**: Compare a current list of records against a previous snapshot to find newly added or removed items.
**Exclusion Filtering**: Remove items from a list that appear in a blocklist or already-processed list to avoid duplicates.
**Data Sync**: Identify which items exist in one system but not another to determine what needs to be synced.
<!-- END MANUAL -->
---
## List Intersection
### What it is
Computes the intersection of two lists, returning only elements present in both.
### How it works
<!-- MANUAL: how_it_works -->
This block finds elements that appear in both input lists by hashing elements from list_b for efficient lookup, then checking each element of list_a against that set. The output preserves the order from list_a and removes duplicates.
This is useful for finding common items between two datasets without needing to manually iterate or compare.
<!-- END MANUAL -->
### Inputs
| Input | Description | Type | Required |
|-------|-------------|------|----------|
| list_a | The first list to intersect. | List[Any] | Yes |
| list_b | The second list to intersect. | List[Any] | Yes |
### Outputs
| Output | Description | Type |
|--------|-------------|------|
| error | Error message if the operation failed. | str |
| intersection | Elements present in both list_a and list_b. | List[Any] |
| length | The number of elements in the intersection. | int |
### Possible use case
<!-- MANUAL: use_case -->
**Finding Common Tags**: Identify shared tags or categories between two items for recommendation or grouping purposes.
**Mutual Connections**: Find users or contacts that appear in both of two different lists, such as shared friends or overlapping team members.
**Feature Comparison**: Determine which features or capabilities are supported by both of two systems or products.
<!-- END MANUAL -->
---
## List Is Empty
### What it is
@@ -1452,3 +1608,42 @@ This makes XML data accessible using standard dictionary operations, allowing yo
<!-- END MANUAL -->
---
## Zip Lists
### What it is
Zips multiple lists together into a list of grouped elements. Supports padding to longest or truncating to shortest.
### How it works
<!-- MANUAL: how_it_works -->
This block pairs up corresponding elements from multiple input lists into sub-lists. For example, zipping `[[1, 2, 3], ['a', 'b', 'c']]` produces `[[1, 'a'], [2, 'b'], [3, 'c']]`.
By default, the result is truncated to the length of the shortest input list. Enable pad_to_longest to instead pad shorter lists with a fill_value so no elements from longer lists are lost.
<!-- END MANUAL -->
### Inputs
| Input | Description | Type | Required |
|-------|-------------|------|----------|
| lists | A list of lists to zip together. Corresponding elements will be grouped. | List[List[Any]] | Yes |
| pad_to_longest | If True, pad shorter lists with fill_value to match the longest list. If False, truncate to shortest. | bool | No |
| fill_value | Value to use for padding when pad_to_longest is True. | Fill Value | No |
### Outputs
| Output | Description | Type |
|--------|-------------|------|
| error | Error message if zipping failed. | str |
| zipped_list | The zipped list of grouped elements. | List[List[Any]] |
| length | The number of groups in the zipped result. | int |
### Possible use case
<!-- MANUAL: use_case -->
**Creating Key-Value Pairs**: Combine a list of field names with a list of values to build structured records or dictionaries.
**Parallel Data Alignment**: Pair up corresponding items from separate data sources (e.g., names and email addresses) for processing together.
**Table Row Construction**: Group column data into rows by zipping each column's values together for CSV export or display.
<!-- END MANUAL -->
---