tochunkedlistasync()
I recently came across this fun little snippet I wrote a while back. I don’t remember
the exact case where I used it, but I thought it was worth sharing. I think I needed
to process a huge IEnumerable<T> and wanted to distribute it as simply as possible.
This is a simple extension method on IEnumerable<T> that returns it as
multiple List<T> objects from a IAsyncEnumerable<List<T>>.
The chunk size is configurable, and the method is cancellable.
I present to you the ToChunkedListAsync extension method:
public static async IAsyncEnumerable<List<T>> ToChunkedListAsync<T>(
this IEnumerable<T> enumerable,
int chunkSize = 100,
[EnumeratorCancellation] CancellationToken token = default)
{
var itemList = new List<T>(chunkSize);
var enumerator = enumerable.GetEnumerator();
for (var count = 1; enumerator.MoveNext(); count++)
{
token.ThrowIfCancellationRequested();
var item = enumerator.Current;
itemList.Add(item);
if (count % chunkSize == 0)
{
yield return itemList;
itemList.Clear();
await Task.Yield();
}
}
if (itemList.Count > 0)
{
yield return itemList;
}
}
I think my original idea for the IEnumerable<T> distribution would have spent more
time on the distribution than the actual processing. This seemed to be a quick and easy
way to break it into chunks get the job done. It’s also lowers the memory pressure as
opposed to a single huge ToList. Here’s a simple example of how to use it:
await foreach (var chunk in Enumerable.Range(1, 42).ToChunkedListAsync(10))
{
Console.WriteLine($"Chunk: {string.Join(", ", chunk)}");
}
// Output:
// Chunk: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
// Chunk: 11, 12, 13, 14, 15, 16, 17, 18, 19, 20
// Chunk: 21, 22, 23, 24, 25, 26, 27, 28, 29, 30
// Chunk: 31, 32, 33, 34, 35, 36, 37, 38, 39, 40
// Chunk: 41, 42
You may never need this, but I thought it was a fun little snippet to share. It may give you some ideas for something else.