Category : bq

I try to update 2K rows in BQ def update_bq_ads_status_failed(self, update_ads): affected_rows = 0 for update_ads_chunk in split(update_ads, _UPDATE_CHUNK_SIZE): ad_ids = [item["ad_id"] for item in update_ads_chunk] removal_errors = [item["removal_error"] for item in update_ads_chunk] update_removal_error = "" for ad_id, removal_error in zip(ad_ids, removal_errors): update_removal_error = update_removal_error + f”’ WHEN ad_id = ‘{ad_id}’ Then ‘{removal_error}’ ”’ affected_rows ..

Read more

I’m trying to update ~5K rows in bq using python client. Here is my current try: update_bq(table_id=_ADS_TO_REMOVE_TABLE_NAME, set_clasue="status =’removed’", where_in_clause=f”'[{[item[‘ad_id’] for item in current_ads_removed_json_chunk]}]”’) and def update_bq(self, table_id, ad_ids, set_clasue, where_in_clause): table_full_name = self.get_table_full_name(table_id) query_text = f""" UPDATE `{table_full_name}` SET {set_clasue} WHERE account_id IN {where_in_clause} """ query_job = self.client.query(query_text) How can I map the account ..

Read more

I’m trying to create a bq client python. I have created the GCP json key and exported it toGOOGLE_APPLICATION_CREDENTIALS. I have tried to run the following in the PyCharm IDE: export GOOGLE_APPLICATION_CREDENTIALS="/home/…..7e7a385.json" echo $GOOGLE_APPLICATION_CREDENTIALS /home/…7e7a385.json GOOGLE_APPLICATION_CREDENTIALS="/home/….7e7a385.json" (following this post https://stackoverflow.com/a/67224391/311130) I still get this error afterwards: google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS ..

Read more

I’m trying to create a BQ table schema, as seen on this page But I get a compilation error for all the mode="REQUIRED" I didn’t see anything special to import but the bq module. Expected type ‘_SpecialForm[str]’, got ‘str’ instead The code: bqServiceWrapper.create_table(_ADS_TO_REMOVE_TABLE_NAME, [ bigquery.SchemaField("add_id", "STRING", mode="REQUIRED"), mode="REQUIRED"), bigquery.SchemaField("timestamp_str", "TIMESTAMP", mode="REQUIRED"), bigquery.SchemaField("timestamp", "TIMESTAMP", mode="REQUIRED") ]) ..

Read more

I’m trying to create a bq client python. But I get this error: google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started Source: Python..

Read more

i am trying to load multiple csv files into table by below code but it is failing: Can anyone let me know where i am wrong: ################## def csv_loader(data, context): client = bigquery.Client() dataset_id = os.environ[‘DATASET’] dataset_ref = client.dataset(dataset_id) job_config = bigquery.LoadJobConfig( source_format=bigquery.SourceFormat.CSV, field_delimiter="|", write_disposition="WRITE_TRUNCATE", skip_leading_rows=1, ) # get the URI for uploaded CSV in ..

Read more