Category : amazon-s3

I have below function which will create S3 bucket with cdk code: def __create_s3_components_bucket(self, this_dir: str, props): """Create S3 components bucket""" s3_bucket = s3.Bucket( self, "BucketForImageBuilder", bucket_name="some_bucket_name_1234", block_public_access=s3.BlockPublicAccess( block_public_acls=True, block_public_policy=True, ignore_public_acls=True, restrict_public_buckets=True, ), public_read_access=False, encryption=s3.BucketEncryption.S3_MANAGED, removal_policy=cdk.RemovalPolicy.DESTROY, auto_delete_objects=True, lifecycle_rules=[ s3.LifecycleRule( abort_incomplete_multipart_upload_after=cdk.Duration.days(amount=2), enabled=True, expiration=cdk.Duration.days(amount=180), transitions=[ s3.Transition( transition_after=cdk.Duration.days(amount=30), storage_class=s3.StorageClass.ONE_ZONE_INFREQUENT_ACCESS, ) ], ) ], ) I would like to ..

Read more

I have a question about performance with pushdown filters. I have data stored in parquet S3 under s3://mybucket/data it has ‘lat’, ‘lon’ and ‘qk’ columns. There are about 750 files. Sorted by qk7 column (min value = 0230001 max value = 0233333). When I do a simple query with spark: path = "s3://mybucket/data" prod_df = ..

Read more

I am doing a Django project. I have hosted my static files on AWS S3.It has been successfully uploaded to it. But,the images are not loading when I run the server. When I inspect the image field it shows: https://django-ecommerce-files.s3.amazonaws.com/images/logo.png%22%20id=%22image%22%20style=%22width:%2040px;%20height:40px%22%3E When I double clicked it. It shows this error: <Error> <Code>AccessDenied</Code> <Message>Access Denied</Message> <RequestId>07PX6KHYASHT3008</RequestId> <HostId>pJCxChq1JHlw/GL0Zy/W+PvX1TevOf/C60Huyidi8+0GMAs8geYlXSrEgo6m9vllL0PouTn6NAA= ..

Read more

I am trying to completely move file from one folder to another folder completly(similar action we do ‘cut and past to another folder’) using boto3 method s3_client = boto3.resource(‘s3′) bucket=’mybucket’ source = { ‘Bucket’: bucket, ‘Key’: ‘some key’ } s3_client.meta.client.copy(source,bucket, ‘my/destination_file_path’) The above code perfectly copy the file from source to destination path, but it ..

Read more

I am using SageMaker to train a model with multiple GBs of data. My data is loaded using huggingface’s datasets.load_dataset method. Since data is huge and I want to re-use it, I want to store it in S3 bucket. I tried below: from datasets import load_dataset dataset = load_dataset(‘s3://bucket_name/some_dir/data’, ‘oscar’, ‘unshuffled_deduplicated_en’) but this results in: ..

Read more

So I am trying to write a python script that checks if all the values in public_access_block on s3 buckets are set to TRUE and if one of the bucket is not, then print out which bucket it is. s3 = boto3.client(‘s3’) buckets = s3.get_public_access_block(Bucket = ‘my-test-bucket) #this is for one bucket print(buckets) My question ..

Read more

Trying to restore an s3 object from GLACIER with the code below. import boto3 s3 = boto3.resource(‘s3’, verify=False) bucket_name = r"my-source-bucket" bucket = s3.Bucket(bucket_name) key ="glacier_file2.txt" try: bucket.meta.client.restore_object(Bucket=bucket_name, Key=key, RestoreRequest={‘Days’: 1, ‘GlacierJobParameters’: {‘Tier’: ‘Expedited’}}) except Exception as e: print({"Problem Restoring": str(e)}) The code submits successfully, however the object still shows as in GLACIER in the ..

Read more

While trying to list objects with a prefix, the return is only fetching only 1 object in my Lambda. Not sure what is missing. import boto3 s3 = boto3.resource(‘s3′) def lambda_handler(event, context): try: ## Bucket to use bucket = s3.Bucket(mybucket) ## List objects within a given prefix for obj in bucket.objects.filter(Prefix=’output/group1’): print(obj.key) Source: Python..

Read more

I use this function that AWS give as a tutorial: How can I read the files from Workmail that I stored in S3 bucket raw_msg = workmail_message_flow.get_raw_message_content(messageId=message_id) parsed_msg: Message = email.message_from_bytes(raw_msg[‘messageContent’].read()) # Updating subject. For more examples, see https://github.com/aws-samples/amazon-workmail-lambda-templates. parsed_msg.replace_header(‘Subject’, f"[Hello World!] {subject}") # Try to get the email bucket. updated_email_bucket_name = os.getenv(‘UPDATED_EMAIL_S3_BUCKET’) if not ..

Read more