Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

copyObject silently failing #1320

Open
dustinwilson opened this issue Oct 22, 2022 · 0 comments
Open

copyObject silently failing #1320

dustinwilson opened this issue Oct 22, 2022 · 0 comments

Comments

@dustinwilson
Copy link

I am trying to copy a list of ~50 files from one bucket on one region to a bucket on the region I'm using, and I'm running into issues where random large-ish files (~270MB) are failing to copy. Sometimes they will copy, but most times they won't. I get no errors.

// ...

// $config is data imported from a yaml file
$cred = $config['credentials'];
$bucket = $cred['bucket'];
$destination = 'my-bucket';

$s3 = new S3Client([
    'accessKeyId' => $cred['access_key_id'],
    'accessKeySecret' => $cred['access_key_secret'],
    'region' => $config['local_region']
]);
$extS3 = new S3Client([
    'accessKeyId' => $cred['access_key_id'],
    'accessKeySecret' => $cred['access_key_secret'],
    'region' => $cred['region'] ?? $config['local_region']
]);

$result = $extS3->listObjectsV2([
    'Bucket' => $bucket
]);
$generator = $result->getIterator();

foreach ($generator as $g) {
    $key = $g->getKey();
    
    $s3->copyObject([
        'Key' => $key,
        'Bucket' => $destination,
        'CopySource' => "$bucket/$key",
        'region' => $config['local_region']
    ]);
}

For most files this works just fine, but occasionally it just silently fails to transfer files. If I use Amazon's official API it works just fine, but I'd rather use this one if I can. I've tried using ->resolve() with a high timeout, but it doesn't seem to make any difference. Is there something else I should be doing?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant