["python","csv","amazon-s3","dataframe","boto3"]
Save Dataframe to csv directly to s3 Python

I have a pandas DataFrame that I want to upload to a new CSV file. The problem is that I don't want to save the file locally before transferring it to s3. Is there any method like to_csv for writing the dataframe to s3 directly? I am using boto3. Here is what I have so far: import boto3 s3 = boto3.client('s3', aws_access_key_id='key', aws_secret_access_key='secret_key') read_file = s3.get_object(Bucket, Key) df = pd.read_csv(read_file['Body']) # Make alterations to DataFrame # Then export DataFrame to CSV through direct transfer to s3
You can use: from io import StringIO # python3; python2: BytesIO import boto3 bucket = 'my_bucket_name' # already created on S3 csv_buffer = StringIO() df.to_csv(csv_buffer) s3_resource = boto3.resource('s3') s3_resource.Object(bucket, 'df.csv').put(Body=csv_buffer.getvalue()) In addition, pandas now also handles reading and writing remote files via fspec, if installed. In other words, in this case df.to_csv('s3://bucket/folder/path/file.csv) also works.