Ubuntu 上安装的 AWS EFS 文件系统中的 Python AWS Lambda 错误“libgomp.so.1：无法打开共享对象文件：没有此类文件或目录”

Question

我有一个 AWS lambda 函数连接到安装在 Ubunutu 18 的 EC2 实例上的 AWS EFS。我相信在导入 LightGBM 模型时出现以下错误。

\n

    {\n  "errorMessage": "libgomp.so.1: cannot open shared object file: No such file or directory",\n  "errorType": "OSError",\n  "stackTrace": [\n    "  File \\"/var/lang/lib/python3.8/imp.py\\", line 234, in load_module\\n    return load_source(name, filename, file)\\n",\n    "  File \\"/var/lang/lib/python3.8/imp.py\\", line 171, in load_source\\n    module = _load(spec)\\n",\n    "  File \\"<frozen importlib._bootstrap>\\", line 702, in _load\\n",\n    "  File \\"<frozen importlib._bootstrap>\\", line 671, in _load_unlocked\\n",\n    "  File \\"<frozen importlib._bootstrap_external>\\", line 848, in exec_module\\n",\n    "  File \\"<frozen importlib._bootstrap>\\", line 219, in _call_with_frames_removed\\n",\n    "  File \\"/var/task/lambda_function.py\\", line 9, in <module>\\n    from supervised.automl import AutoML\\n",\n    "  File \\"/mnt/access/supervised/__init__.py\\", line 3, in <module>\\n    from supervised.automl import AutoML\\n",\n    "  File \\"/mnt/access/supervised/automl.py\\", line 3, in <module>\\n    from supervised.base_automl import BaseAutoML\\n",\n    "  File \\"/mnt/access/supervised/base_automl.py\\", line 21, in <module>\\n    from supervised.algorithms.registry import AlgorithmsRegistry\\n",\n    "  File \\"/mnt/access/supervised/algorithms/registry.py\\", line 74, in <module>\\n    import supervised.algorithms.lightgbm\\n",\n    "  File \\"/mnt/access/supervised/algorithms/lightgbm.py\\", line 8, in <module>\\n    import lightgbm as lgb\\n",\n    "  File \\"/mnt/access/lightgbm/__init__.py\\", line 8, in <module>\\n    from .basic import Booster, Dataset\\n",\n    "  File \\"/mnt/access/lightgbm/basic.py\\", line 43, in <module>\\n    _LIB = _load_lib()\\n",\n    "  File \\"/mnt/access/lightgbm/basic.py\\", line 34, in _load_lib\\n    lib = ctypes.cdll.LoadLibrary(lib_path[0])\\n",\n    "  File \\"/var/lang/lib/python3.8/ctypes/__init__.py\\", line 451, in LoadLibrary\\n    return self._dlltype(name)\\n",\n    "  File \\"/var/lang/lib/python3.8/ctypes/__init__.py\\", line 373, in __init__\\n    self._handle = _dlopen(self._name, mode)\\n"\n  ]\n}\n

\n

我安装的包列表

\n

ubuntu@ip-***-**-**-***:~/mounted/access$ ls\nConfigSpace                    category_encoders-2.2.2.dist-info  examples                                   matplotlib                         plotlywidget                     scikit_plot-0.3.7.dist-info       tabulate.py\nConfigSpace-0.4.19.dist-info   click                              fsspec                                     matplotlib-3.4.2-py3.8-nspkg.pth   pluggy                           scikitplot                        tblib\nCython                         click-8.0.1.dist-info              fsspec-2021.5.0.dist-info                  matplotlib-3.4.2.dist-info         pluggy-0.13.1.dist-info          scipy                             tblib-1.7.0.dist-info\nCython-0.29.23.dist-info       cliff                              graphviz                                   mljar_supervised-0.10.4.dist-info  prettytable                      scipy-1.6.1.dist-info             testing\nHeapDict-1.0.1.dist-info       cliff-3.8.0.dist-info              graphviz-0.16.dist-info                    mpl_toolkits                       prettytable-2.1.0.dist-info      scipy-1.6.3.dist-info             threadpoolctl-2.1.0.dist-info\nMako-1.1.4.dist-info           cloudpickle                        greenlet                                   msgpack                            psutil                           scipy.libs                        threadpoolctl.py\nMarkdown-3.3.4.dist-info       cloudpickle-1.3.0.dist-info        greenlet-1.1.0.dist-info                   msgpack-1.0.2.dist-info            psutil-5.8.0.dist-info           seaborn                           tlz\nMarkupSafe-2.0.1.dist-info     cloudpickle-1.6.0.dist-info        heapdict.py                                numba                              py                               seaborn-0.11.1.dist-info          toml\nPIL                            cmaes                              include                                    numba-0.48.0.dist-info             py-1.10.0.dist-info              setuptools                        toml-0.10.2.dist-info\nPillow-8.2.0.dist-info         cmaes-0.8.2.dist-info              iniconfig                                  numba-0.53.1.dist-info             pyarrow                          setuptools-57.0.0.dist-info       toolz\nPillow.libs                    cmd2                               iniconfig-1.1.1.dist-info                  numpy                              pyarrow-4.0.1.dist-info          shap                              toolz-0.11.1.dist-info\nPyYAML-5.4.1.dist-info         cmd2-1.5.0.dist-info               intel_openmp-2021.2.0.dist-info            numpy-1.20.3.dist-info             pylab.py                         shap-0.36.0.dist-info             tornado\nSQLAlchemy-1.4.17.dist-info    colorama                           joblib                                     numpy.libs                         pynisher                         share                             tornado-6.1.dist-info\n__pycache__                    colorama-0.4.4.dist-info           joblib-1.0.1.dist-info                     optuna                             pynisher-0.6.4.dist-info         six-1.16.0.dist-info              tqdm\n_distutils_hack                colorlog                           kiwisolver-1.3.1.dist-info                 optuna-2.7.0.dist-info             pyparsing-2.4.7.dist-info        six.py                            tqdm-4.61.0.dist-info\n_plotly_future_                colorlog-5.0.1.dist-info           kiwisolver.cpython-38-x86_64-linux-gnu.so  packaging                          pyparsing.py                     sklearn                           wcwidth\n_plotly_utils                  colour-0.1.5.dist-info             lazy_import                                packaging-20.9.dist-info           pyperclip                        sklearn-0.0.dist-info             wcwidth-0.2.5.dist-info\n_pytest                        colour.py                          lazy_import-0.2.2.dist-info                pandas                             pyperclip-1.8.2.dist-info        slicer                            wheel\n_yaml                          cycler-0.10.0.dist-info            liac_arff-2.5.0.dist-info                  pandas-1.2.0.dist-info             pyrfr                            slicer-0.0.7.dist-info            wheel-0.36.2.dist-info\nalembic                        cycler.py                          licensing                                  pandas-1.2.4.dist-info             pyrfr-0.8.2.dist-info            smac                              wordcloud\nalembic-1.6.5.dist-info        cython.py                          lightgbm                                   partd                              pytest                           smac-0.13.1.dist-info             wordcloud-1.8.1.dist-info\napt.txt                        dask                               lightgbm-3.0.0.dist-info                   partd-1.2.0.dist-info              pytest-6.2.4.dist-info           somefile.txt                      xgboost\narff.py                        dask-2021.6.0.dist-info            llvmlite                                   patsy                              python_dateutil-2.8.1.dist-info  sortedcontainers                  xgboost-1.3.3.dist-info\nattr                           dateutil                           llvmlite-0.31.0.dist-info                  patsy-0.5.1.dist-info              python_editor-1.0.4.dist-info    sortedcontainers-2.4.0.dist-info  xgboost-1.4.2.dist-info\nattrs-21.2.0.dist-info         distributed                        llvmlite-0.36.0.dist-info                  pbr                                pytz                             sqlalchemy                        xgboost.libs\nauto_sklearn-0.12.6.dist-info  distributed-2021.6.0.dist-info     local                                      pbr-5.6.0.dist-info                pytz-2021.1.dist-info            statsmodels                       yaml\nautosklearn                    distutils-precedence.pth           locket                                     pip                                pyximport                        statsmodels-0.12.2.dist-info      zict\nbin                            dtreeviz                           locket-0.2.1.dist-info                     pip-21.1.2.dist-info               retrying-1.3.3.dist-info         stevedore                         zict-2.0.0.dist-info\ncatboost                       dtreeviz-1.3.dist-info             mako                                       pkg_resources                      retrying.py                      stevedore-3.3.0.dist-info\ncatboost-0.24.4.dist-info      editor.py                          markdown                                   plotly                             scikit_learn-0.24.2.dist-info    supervised\ncategory_encoders              etc                                markupsafe                                 plotly-4.14.3.dist-info            scikit_learn.libs                tabulate-0.8.7.dist-info\n

\n

如果有人知道如何解决此错误或需要更多信息，请帮助！

\n

Function Logs\nSTART RequestId: 576d0bf7-1cdb-4f98-bd6d-ba807d0af0bb Version: $LATEST\nOpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k\n/mnt/access/joblib/_multiprocessing_helpers.py:45: UserWarning: [Errno 38] Function not implemented.  joblib will operate in serial mode\n  warnings.warn('%s.  joblib will operate in serial mode' % (e,))\nOpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k\n[WARNING]   2021-06-05T00:57:04.711Z        Matplotlib created a temporary config/cache directory at /tmp/matplotlib-bw2u754h because the default path (/home/sbx_user1051/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.\n[ERROR] OSError: libgomp.so.1: cannot open shared object file: No such file or directory\nTraceback (most recent call last):\n\xc2\xa0\xc2\xa0File "/var/lang/lib/python3.8/imp.py", line 234, in load_module\n\xc2\xa0\xc2\xa0\xc2\xa0\xc2\xa0return load_source(name, filename, file)\n\xc2\xa0\xc2\xa0File "/var/lang/lib/python3.8/imp.py", line 171, in load_source\n\xc2\xa0\xc2\xa0\xc2\xa0\xc2\xa0module = _load(spec)\n\xc2\xa0\xc2\xa0File "<frozen importlib._bootstrap>", line 702, in _load\n\xc2\xa0\xc2\xa0File "<frozen importlib._bootstrap>", line 671, in _load_unlocked\n\xc2\xa0\xc2\xa0File "<frozen importlib._bootstrap_external>", line 848, in exec_module\n\xc2\xa0\xc2\xa0File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed\n\xc2\xa0\xc2\xa0File "/var/task/lambda_function.py", line 9, in <module>\n\xc2\xa0\xc2\xa0\xc2\xa0\xc2\xa0import lightgbm\n\xc2\xa0\xc2\xa0File "/mnt/access/lightgbm/__init__.py", line 8, in <module>\n\xc2\xa0\xc2\xa0\xc2\xa0\xc2\xa0from .basic import Booster, Dataset, register_logger\n\xc2\xa0\xc2\xa0File "/mnt/access/lightgbm/basic.py", line 95, in <module>\n\xc2\xa0\xc2\xa0\xc2\xa0\xc2\xa0_LIB = _load_lib()\n\xc2\xa0\xc2\xa0File "/mnt/access/lightgbm/basic.py", line 86, in _load_lib\n\xc2\xa0\xc2\xa0\xc2\xa0\xc2\xa0lib = ctypes.cdll.LoadLibrary(lib_path[0])\n\xc2\xa0\xc2\xa0File "/var/lang/lib/python3.8/ctypes/__init__.py", line 451, in LoadLibrary\n\xc2\xa0\xc2\xa0\xc2\xa0\xc2\xa0return self._dlltype(name)\n\xc2\xa0\xc2\xa0File "/var/lang/lib/python3.8/ctypes/__init__.py", line 373, in __init__\n\xc2\xa0\xc2\xa0\xc2\xa0\xc2\xa0self._handle = _dlopen(self._name, mode)END RequestId: 576d0bf7-1cdb-4f98-bd6d-ba807d0af0bb\nREPORT RequestId: 576d0bf7-1cdb-4f98-bd6d-ba807d0af0bb  Duration: 22908.85 ms   Billed Duration: 22909 ms   Memory Size: 5000 MB    Max Memory Used: 243 MB \nUnknown application error occurred\n

\n

这是来自 AWS lambda 函数的日志，我在我的 ec2 Ubuntu 环境上安装了 libgomp1，但是我不确定问题是在 lambda 还是 ubuntu 中。我可以很好地导入 lightgbm

\n

Answer 1

值得注意的是，libgomp.so 和 glibc 的版本可能会因 Amazon Lambda 运行时版本而异。有关更多信息，请参阅https://repost.aws/questions/QUrXOioL46RcCnFGyELJWKLw/glibc-2-27-on-amazon-linux-2。 为了解决这个问题，您可以通过在 EC2 上配置 Amazon Linux 2 实例、下载所需的 libgomp.so 和 glibc 包并将它们打包到 Lambda 层中来创建自定义 AWS Lambda 层。确保 AWS Lambda 层对象位于 /lib 文件夹下，如文档中指定的： https: //docs.aws.amazon.com/lambda/latest/dg/packaging-layers.html。

添加此层后，Lightgbm 库应该可以导入到您的 Lambda 环境中。

这是我的 python 3.8 层！ AWS-Lambda-Layers-v0.zip