小编RNH*_*TTR的帖子

代码如下所示：

import multiprocessing as mp
from functools import partial

import boto3
import numpy as np


s3 = boto3.client('s3')

def _something(**kwargs):
    # Some mixed integer programming stuff related to the variable archive
    return np.array(some_variable_related_to_archive)


def do(s3):
    archive = np.load(s3.get_object('some_key')) # Simplified -- details not relevant
    pool = mp.pool()
    sub_process = partial(_something, slack=0.1)
    parts = np.array_split(archive, some_int)
    target_parts = np.array(things)

    out = pool.starmap(sub_process, [x for x in zip(parts, target_parts)] # Error occurs at this line

    pool.close()
    pool.join()

do(s3)

错误：

_pickle.PicklingError: Can't pickle …

我正在尝试将Python日志文件直接写入S3，而不先将它们保存到stdout。我希望在程序运行完毕后将日志文件自动写入S3。我想使用boto3 put_object方法：

import atexit
import logging

import boto3  


def write_logs(body, bucket, key):
    s3 = boto3.client("s3")
    s3.put_object(Body=body, Bucket=bucket, Key=key)

log = logging.getLogger("some_log_name")
log.info("Hello S3")

atexit.register(write_logs, body=log, bucket="bucket_name", key="key_name")

quit()

上载到S3时会引发错误。如果我没记错的话，它要求上传到S3的对象必须是字节状的。我有时间重新创建问题后，将使用确切的错误来更新问题。