uploadSplit

fun S3AsyncClient.uploadSplit(bucket: String, upstream: Flow<Byte>, splitStrategy: GroupStrategy = GroupStrategy.Count(ONE_MB), concurrency: Int = 1, key: (Int) -> String): Flow<S3Response>

This function uploads a file in chunks to an Amazon S3 bucket using the S3AsyncClient.

Particularly useful for handling streams of unknown size, since it automatically splits the flow into separate files, allowing for seamless processing and storage.

When the split size exceeds 5MB, the function automatically utilizes S3's multipart file upload to prevent retaining large chunks of data in-memory. If it is smaller, the function uses the put object operation for a quicker and more efficient processing.

Return

A flow of S3Response objects for each uploaded chunk.

Example usage:

val s3Client: S3AsyncClient = ...
val bucket = "my-bucket"
val byteArrayFlow = flowOf<ByteArray> { ... } // A Flow<ByteArray> containing the byte arrays to upload.
val oneMB = 1024 * 1024

s3Client.uploadSplit(bucket = bucket, upstream = byteArrayFlow, splitEach = oneMB) { part ->
"folder/file-part-$part"
}
.collect { response ->
println("Uploaded part: ${response.key}")
}

Parameters

bucket

The S3 bucket to upload the file to.

upstream

A flow of bytes representing the file to be uploaded.

splitStrategy

The strategy to describe the split to be uploaded, in bytes. Default is 1 MB.

concurrency

The number of concurrent uploads to use. Default is 1.

key

A function that takes an integer (part number) and returns the key of the object in the S3 bucket.