Java – S3 multipart upload fails after partial completion

I’m trying to upload dynamically generated data from memory to S3.

This is the code that actually uploads the part:

UploadPartRequest partReq = new UploadPartRequest()
    .withBucketName( this.getBucket() )
    .withKey( this.getKey() )
    .withUploadId( uploadId )
    .withPartNumber( partNumber )
    .withPartSize( partSize )
    .withInputStream( is )
    .withGeneralProgressListener( (ProgressEvent e) -> {
            log.info( "{} {} {} {} {} {} bytes transferred",
            this.getBucket(), this.getKey(), uploadId, partNumber,
            e.getEventType(), e.getBytesTransferred() );
    });
UploadPartResult partRes = this.getClient().uploadPart( partReq );

After all the sections are uploaded, the following line appears in my log, which I think means that all uploaded parts were successfully uploaded.

1 TRANSFER_PART_COMPLETED_EVENT 
2 TRANSFER_PART_COMPLETED_EVENT 
3 TRANSFER_PART_COMPLETED_EVENT 
4 TRANSFER_PART_COMPLETED_EVENT 
5 TRANSFER_PART_COMPLETED_EVENT 
6 TRANSFER_PART_COMPLETED_EVENT 
7 TRANSFER_PART_COMPLETED_EVENT 
8 TRANSFER_PART_COMPLETED_EVENT 
9 TRANSFER_PART_COMPLETED_EVENT 
10 TRANSFER_PART_COMPLETED_EVENT
11 TRANSFER_PART_COMPLETED_EVENT
12 TRANSFER_PART_COMPLETED_EVENT
13 TRANSFER_PART_COMPLETED_EVENT
14 TRANSFER_PART_COMPLETED_EVENT
15 TRANSFER_PART_COMPLETED_EVENT
17 TRANSFER_PART_COMPLETED_EVENT
16 TRANSFER_PART_COMPLETED_EVENT

For each upload section, the following line also appears many times while they are still uploading, which tells me that it is indeed sending data for those uploads.

REQUEST_BYTE_TRANSFER_EVENT 8192 bytes transferred

This is the code I used to finalize the entire upload process.

CompleteMultipartUploadRequest cmpur = new CompleteMultipartUploadRequest()
    .withBucketName( this.getBucket() )
    .withKey( this.getKey() )
    .withUploadId( result.getUploadId() )
    .withPartETags( new ArrayList<>( this.getUploadPartResults() ) )
    .withGeneralProgressListener((e) -> {
        log.info( "{} {} {} {} bytes transferred",
        this.getBucket(), this.getKey(),
        e.getEventType(), e.getBytesTransferred() );
    });
this.getClient().completeMultipartUpload( cmpur );

After sending a request to complete the entire multipart upload, I get the following information in the logs:

REQUEST_CONTENT_LENGTH_EVENT
CLIENT_REQUEST_STARTED_EVENT
HTTP_REQUEST_STARTED_EVENT
HTTP_REQUEST_COMPLETED_EVENT
CLIENT_REQUEST_FAILED_EVENT

Then the thread gets stuck there.

I’m assuming CLIENT_REQUEST_FAILED_EVENT means the whole upload failed. What am I missing?

I tried increasing the part size so that the entire multipart upload had only one upload part and it worked. If the total number of parts is greater than 1, it fails.

This is only one part when the log is uploaded successfully.

REQUEST_CONTENT_LENGTH_EVENT 0 bytes transferred
CLIENT_REQUEST_STARTED_EVENT 0 bytes transferred
HTTP_REQUEST_STARTED_EVENT 0 bytes transferred
HTTP_REQUEST_COMPLETED_EVENT 0 bytes transferred
RESPONSE_CONTENT_LENGTH_EVENT 0 bytes transferred
HTTP_RESPONSE_STARTED_EVENT 0 bytes transferred
RESPONSE_BYTE_TRANSFER_EVENT 309 bytes transferred
HTTP_RESPONSE_COMPLETED_EVENT 0 bytes transferred
CLIENT_REQUEST_SUCCESS_EVENT 0 bytes transferred
REQUEST_BYTE_TRANSFER_EVENT 135 bytes transferred

Perhaps, what else can I add to the logs to get more information to see what the problem is?

I found out this is because my part is actually smaller than the minimum size required of 5MB.

https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html

I’ve visited the page several times, but I’ve ignored it. It’s really easy to overlook. The logs didn’t help either.