[Errno 32] Broken pipe in s3cmd

Submit to reddit

In order to backup a bunch of servers I’m using s3cmd to backup files to a Amazon S3 account. s3cmd is a really nice tool as it works from the command line, does encryption on the fly and can use the https protocol for upload.

I used s3cmd --configure and set everything up. The first error I got when I tried to test it was:

s3cmd put file.tar.gz s3://my_bucket_name/file.tar.gz

[Errno 32] Broken pipe

and it tried throttling down and uploading again. What I didn’t know was that:

  • if your file is over 5Gb it will give this error
  • if you just created the bucket it may take a few minutes to start working
  • you get this error if the bucket doesn’t exist (you mistyped it)
  • Interestingly we started getting this error from a bucket (which is new-ish, but about an hour old now) which we pushed a few thousand small files (< 100MB, mostly ~4KB) into in two batches, and now the third batch is just dying on the first file.

    Leaving it for 4-5 minutes (in case it's an indexing thing) does not seem to have helped.

    So there might be a fourth bullet point for your list – I'm just not sure what it is yet.

    • ¬†AFAIK S3 buckets impose no limits on the number of files in a bucket, so that can’t be it. I’m really curious what causes this if you manage to track it down.

  • aqui_c

    I’m using DreamObjects and get this error constantly, when deleting files from the bucket, when uploading (didn’t try downloading yet…) If I’m not mistaken I’ve also seen this error (or similar) with boto-rsync…