Bereitstellen von MMCV/MMDet auf Databricks – GLIBC_2.32 nicht gefunden
Posted: 07 Jan 2025, 13:26
Ich versuche, das MMDetection-Modell auf Databricks Serving (auf Azure) zu hosten. Das Modell ist auf 15,4 LTS ML trainiert. Während der Aktualisierung des Bereitstellungsendpunkts wird jedoch GLIBC_2.32:
gemeldet
Normalerweise würde ich einfach apt-get install glibc oder etwas Ähnliches in einem Startskript ausführen. Allerdings heißt es in der Databricks-Dokumentation, dass Init-Skripte nicht unterstützt werden.
Frage
Wie kann man also diesen Fehler umgehen und ein MMDet bereitstellen? Modell auf Databricks?
Zuvor habe ich viele PyTorch Lightning-Modelle bereitgestellt (also kein OpenMMLab) und glibc war kein Problem für Databricks Serving.
Lösung Skizzen
Ich sehe zwei mögliche Wege zu einer möglichen Lösung.
Auszug der verwendeten Schlüsselpakete:
Vollständiger Stack-Trace:
Update 7. Januar
Hier ist ein Update meiner bisherigen Forschung.
gemeldet
Code: Select all
An error occurred while loading the model: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmcv/_ext.cpython-311-x86_64-linux-gnu.so)
Frage
Wie kann man also diesen Fehler umgehen und ein MMDet bereitstellen? Modell auf Databricks?
Zuvor habe ich viele PyTorch Lightning-Modelle bereitgestellt (also kein OpenMMLab) und glibc war kein Problem für Databricks Serving.
Lösung Skizzen
Ich sehe zwei mögliche Wege zu einer möglichen Lösung.
- Vielleicht gibt es einen Zaubertrick, der es ermöglicht, fehlende Bibliotheken auf dem zu installieren Endpunkt bedienen?
- Alternativ können wir vielleicht die Grundursache des Problems beseitigen, nämlich mmcv, und dieser Fehler klingt ein bisschen wie ein opencv-Problem mir? Das Modell, das ich zu verwenden versuche, ist CoDETR mit einer darauf basierenden Konfiguration.
Auszug der verwendeten Schlüsselpakete:
Code: Select all
pip install --upgrade pip
pip install uv
uv pip install torch==2.1.0 torchvision==0.16.0 numpy==1.26.4 openmim "mmengine==0.10.5"
uv pip install "mmcv==2.1.0" -f https://download.openmmlab.com/mmcv/dist/cu121/torch2.1.0/index.html
uv pip install albumentations==1.4.18 pycocotools==2.0.7 mlflow python-snappy==0.7.3
uv pip install lightning==2.2.2 mmdet==3.3.0
Code: Select all
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 130, in _load_model_closure
[mlrpj] [2024-12-30 09:50:06 +0000] model = load_model_fn(path)
[mlrpj] [2024-12-30 09:50:06 +0000] ^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracing/provider.py", line 309, in wrapper
[mlrpj] [2024-12-30 09:50:06 +0000] is_func_called, result = True, f(*args, **kwargs)
[mlrpj] [2024-12-30 09:50:06 +0000] ^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/__init__.py", line 1067, in load_model
[mlrpj] [2024-12-30 09:50:06 +0000] model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)
[mlrpj] [2024-12-30 09:50:06 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 561, in _load_pyfunc
[mlrpj] [2024-12-30 09:50:06 +0000] context, python_model, signature = _load_context_model_and_signature(model_path, model_config)
[mlrpj] [2024-12-30 09:50:06 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 555, in _load_context_model_and_signature
[mlrpj] [2024-12-30 09:50:06 +0000] python_model.load_context(context=context)
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/computer_vision/engine/mmdet/utils.py", line 44, in load_context
[mlrpj] [2024-12-30 09:50:06 +0000] from mmdet.apis import DetInferencer
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/apis/__init__.py", line 2, in
[mlrpj] [2024-12-30 09:50:06 +0000] from .det_inferencer import DetInferencer
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/apis/det_inferencer.py", line 22, in
[mlrpj] [2024-12-30 09:50:06 +0000] from mmdet.evaluation import INSTANCE_OFFSET
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/evaluation/__init__.py", line 4, in
[mlrpj] [2024-12-30 09:50:06 +0000] from .metrics import * # noqa: F401,F403
[mlrpj] [2024-12-30 09:50:06 +0000] ^^^^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/evaluation/metrics/__init__.py", line 5, in
[mlrpj] [2024-12-30 09:50:06 +0000] from .coco_metric import CocoMetric
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/evaluation/metrics/coco_metric.py", line 16, in
[mlrpj] [2024-12-30 09:50:06 +0000] from mmdet.datasets.api_wrappers import COCO, COCOeval, COCOevalMP
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/datasets/__init__.py", line 31, in
[mlrpj] [2024-12-30 09:50:06 +0000] from .utils import get_loading_pipeline
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/datasets/utils.py", line 5, in
[mlrpj] [2024-12-30 09:50:06 +0000] from mmdet.datasets.transforms import LoadAnnotations, LoadPanopticAnnotations
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/datasets/transforms/__init__.py", line 6, in
[mlrpj] [2024-12-30 09:50:06 +0000] from .formatting import (ImageToTensor, PackDetInputs, PackReIDInputs,
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/datasets/transforms/formatting.py", line 11, in
[mlrpj] [2024-12-30 09:50:06 +0000] from mmdet.structures.bbox import BaseBoxes
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/structures/bbox/__init__.py", line 2, in
[mlrpj] [2024-12-30 09:50:06 +0000] from .base_boxes import BaseBoxes
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/structures/bbox/base_boxes.py", line 9, in
[mlrpj] [2024-12-30 09:50:06 +0000] from mmdet.structures.mask.structures import BitmapMasks, PolygonMasks
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/structures/mask/__init__.py", line 3, in
[mlrpj] [2024-12-30 09:50:06 +0000] from .structures import (BaseInstanceMasks, BitmapMasks, PolygonMasks,
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/structures/mask/structures.py", line 12, in
[mlrpj] [2024-12-30 09:50:06 +0000] from mmcv.ops.roi_align import roi_align
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmcv/ops/__init__.py", line 3, in
[mlrpj] [2024-12-30 09:50:06 +0000] from .active_rotated_filter import active_rotated_filter
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmcv/ops/active_rotated_filter.py", line 10, in
[mlrpj] [2024-12-30 09:50:06 +0000] ext_module = ext_loader.load_ext(
[mlrpj] [2024-12-30 09:50:06 +0000] ^^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmcv/utils/ext_loader.py", line 13, in load_ext
[mlrpj] [2024-12-30 09:50:06 +0000] ext = importlib.import_module('mmcv.' + name)
[mlrpj] [2024-12-30 09:50:06 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/importlib/__init__.py", line 126, in import_module
[mlrpj] [2024-12-30 09:50:06 +0000] return _bootstrap._gcd_import(name[level:], package, level)
[mlrpj] [2024-12-30 09:50:06 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:06 +0000] ImportError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmcv/_ext.cpython-311-x86_64-linux-gnu.so)
[mlrpj] [2024-12-30 09:50:06 +0000] [595] [INFO] Worker exiting (pid: 595)
[mlrpj] [2024-12-30 09:50:07 +0000] An error occurred while loading the model: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmcv/_ext.cpython-311-x86_64-linux-gnu.so)
[mlrpj] [2024-12-30 09:50:07 +0000] Traceback (most recent call last):
[mlrpj] [2024-12-30 09:50:07 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 212, in get_model_option_or_exit
[mlrpj] [2024-12-30 09:50:07 +0000] self.model = self.model_future.result()
[mlrpj] [2024-12-30 09:50:07 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:07 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result
[mlrpj] [2024-12-30 09:50:07 +0000] return self.__get_result()
[mlrpj] [2024-12-30 09:50:07 +0000] ^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:07 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
[mlrpj] [2024-12-30 09:50:07 +0000] raise self._exception
[mlrpj] [2024-12-30 09:50:07 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/thread.py", line 58, in run
[mlrpj] [2024-12-30 09:50:07 +0000] result = self.fn(*self.args, **self.kwargs)
[mlrpj] [2024-12-30 09:50:07 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:07 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 130, in _load_model_closure
[mlrpj] [2024-12-30 09:50:07 +0000] model = load_model_fn(path)
[mlrpj] [2024-12-30 09:50:07 +0000] ^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:07 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracing/provider.py", line 309, in wrapper
[mlrpj] [2024-12-30 09:50:07 +0000] is_func_called, result = True, f(*args, **kwargs)
[mlrpj] [2024-12-30 09:50:07 +0000] ^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:07 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/__init__.py", line 1067, in load_model
[mlrpj] [2024-12-30 09:50:07 +0000] model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)
[mlrpj] [2024-12-30 09:50:07 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:07 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 561, in _load_pyfunc
[mlrpj] [2024-12-30 09:50:07 +0000] context, python_model, signature = _load_context_model_and_signature(model_path, model_config)
[mlrpj] [2024-12-30 09:50:07 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:07 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 555, in _load_context_model_and_signature
[mlrpj] [2024-12-30 09:50:07 +0000] python_model.load_context(context=context)
[mlrpj] [2024-12-30 09:50:07 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/computer_vision/engine/mmdet/utils.p
Hier ist ein Update meiner bisherigen Forschung.
- < li>Ein möglicher und steiniger Weg zur Lösung wäre die Konvertierung des Modells in ONNX mit mmdeploy, aber CoDETR wird davon derzeit noch nicht unterstützt.
- Während Databricks auf Azure Es ist nicht gestattet, benutzerdefinierte Bilder beim Servieren zu verwenden Interessanterweise ist es möglich, sie auf AWS anzupassen.