AWS - Read dataframe from S3

Give Feedback | Bug report

Tags: #aws #cloud #storage #S3bucket #operations #snippet #dataframe

Author: Maxime Jublou

Last update: 2023-11-20 (Created: 2022-04-28)

Description: This notebook demonstrates how to read a dataframe from an Amazon Web Services (AWS) Simple Storage Service (S3) bucket.

Input

Import libraries

In [ ]:

import naas
try:
    import awswrangler as wr
except:
    !pip install awswrangler --user
    import awswrangler as wr
from os import environ

Setup variables

Mandatory

aws_access_key_id: This variable is used to store the AWS access key ID.
aws_secret_access_key: This variable is used to store the AWS secret access key.
bucket_path: The name of the S3 bucket from which you want to list the files.

In [ ]:

# Mandatory
aws_access_key_id = naas.secret.get("AWS_ACCESS_KEY_ID") or "YOUR_AWS_ACCESS_KEY_ID"
aws_secret_access_key = naas.secret.get("AWS_SECRET_ACCESS_KEY") or "YOUR_AWS_SECRET_ACCESS_KEY"
bucket_path = f"s3://naas-data-lake/example/"

Model

Set environ

In [ ]:

environ["AWS_ACCESS_KEY_ID"] = aws_access_key_id
environ["AWS_SECRET_ACCESS_KEY"] = aws_secret_access_key

Get dataframe

In [ ]:

df = wr.s3.read_parquet(bucket_path, dataset=True)
print("Rows:", len(df))

Output

Display result

In [ ]:

df