Load pipe delimited using bq load and python subprocess on Windows

  bq, gsutil, python-3.x, subprocess, windows

I’m trying to load a table with the BQ LOAD command from Google Cloud Storage to Bigquery.
For this, I’m using a python script that uses the subprocess library.
It works fine, except for a table that has a pipe (|) as delimiter.

I’ve tryied many ways for scape this, without any good result.

If I try this in the CMD (Windows) it works:

bq load --source_format=CSV --field_delimiter="|" Dataset.TableName gs://sourcefile.CSV C:schemafile.json

but if I try to pass the same arguments in python, I get the following error:

BigQuery error in load operation: Field delimiter must be a single character,
found:""|"".

This is the code I am using (in Windows):

params.append(str("bq"))
params.append(str("load"))
params.append(str("--source_format=CSV"))
params.append(str("--field_delimiter="+field_delimiter))
params.append(str(dest_dataset_table))
params.append(str(source_path))
params.append(str(schema_path))
process = subprocess.run(params, shell=True, capture_output=True)

How can I scape or load this?
Thanks!

Source: Python-3x Questions

LEAVE A COMMENT