I’m running a python script which submits a pyspark job on a cluster. However, the job fails with GLIBC issues.
Log Contents: dev-env/bin/python: /lib64/libc.so.6: version 'GLIBC_2.14' not found (required by dev-env/bin/python) dev-env/bin/python: /lib64/libc.so.6: version 'GLIBC_2.17' not found (required by dev-env/bin/python)
The problem, I think, is that the GLIBC version on my machine is 2.17. GLIBC version on pyspark cluster is 2.10. I obtained this by opening python and running the command
>>> import platform >>> platform.libc_ver() ('glibc', '2.17') # This is my machine
One way to resolve this problem IMO is to make sure python on my machine uses GLIBC 2.10. However, I don’t know how to do it. I’m using anaconda to create python virtual env. How should I approach this?
Source: Python Questions