客观此代码的是读取从指定S3桶现有CSV文件到一个数据帧,过滤数据帧为所需的列,然后编写过滤数据帧使用StringIO的一个CSV对象,我可以上传到不同S3桶。
除了函数“prepare_file_for_upload”的代码块之外,现在一切正常。下面是完整的代码块:
from io import StringIO
import io #unsued at the moment
import logging
import pandas as pd
import boto3
from botocore.exceptions import ClientError
FORMAT = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
logging.basicConfig(level=logging.INFO, format=FORMAT)
logger = logging.getLogger(__name__)
#S3 parameters
source_bucket = 'REPLACE'
source_folder = 'REPLACE/'
dest_bucket = 'REPLACE'
dest_folder = 'REPLACE'
output_name = 'REPLACE'
def get_file_name():
try:
s3 = boto3.client("s3")
logging.info(f'Determining filename from: {source_bucket}/{source_folder}')
bucket_path = s3.list_objects(Bucket=source_bucket, Prefix=source_folder)
file_name =[key['Key'] for key in bucket_path['Contents']][1] …Run Code Online (Sandbox Code Playgroud)