import os Causing Counterintuitive Errors in Sagemaker

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/function/awslambdaric/__main__.py", line 25, in <module>
    main(sys.argv)
  File "/function/awslambdaric/__main__.py", line 19, in main
    lambda_runtime_api_addr = os.environ["AWS_LAMBDA_RUNTIME_API"]
  File "/usr/lib/python3.8/os.py", line 675, in __getitem__
    raise KeyError(key) from None
KeyError: 'AWS_LAMBDA_RUNTIME_API'

My suspicion is that the our internal package requires os as a dependency to open a file, and that causes some kind of clash with AWS.

But it gets weirder! The dependency’s Constants module has an import os that works just fine! BUT it uses this to get the assets path:

os.path.join(os.path.dirname(os.path.abspath(__file__)), "assets")

When I look at the logs, guess what this resolves to? The Lambda code directory (/functions) not the Sagemaker one (/opt/ml/code), even though my Dockerfile specifies the Sagemaker CODE_DIR.

Update 2023-05-24#

This came up again while testing some new changes that didn’t change imports. I deployed the full pipeline with a push_all alias I made that calls all the aliases for each individual stage in the dockerfile which represents each part of the pipeline. I redeployed the Sagemaker stage individually (with a whitespace code change to force a rebuild) and the problem went away. I don’t know why this is happening, or why LambdaRIC is being referenced at all in Sagemaker - it’s not used or referenced at all in the TRAIN stage.