Dash/Plotly - long_callback 在 celery/redis 后端失败

Question

概括

\n

我\xe2\x80\x99一直在开发一个使用 的 dash 应用程序long_callback，并且为了开发，我\xe2\x80\x99一直在diskcache为我的 使用后端long_callback_manager，正如我在这里找到的指南所建议的：https://dash.plotly。 com/长回调

\n

当我尝试使用 Gunicorn 运行我的应用程序时，它无法启动，因为diskcache. 因此，我决定切换到 celery/redis 后端，因为 \xe2\x80\x99s 无论如何都建议用于生产。

\n

我运行了一个 redis 服务器（正确响应 with redis-cli ping）PONG，然后再次启动该应用程序。这次它启动得很好，所有正常的回调都起作用，但不起作用long_callback。

\n

细节：

\n

\n
• 页面或多或少挂起，页面标题在正常标题和标题之间闪烁Updating...，表明应用程序认为\xe2\x80\x99s\xe2\x80\x9c正在等待\xe2\x80\x9d的响应/更新这long_callback。
\n
• 由运行参数设置的值long_callback被设置为它们的起始值，表明应用程序识别出long_callback应该运行。
\n
• 通过将打印语句放置在函数内的第一行long_callback并看到它没有打印，我确定该函数永远不会启动。
\n
• 使用gunicorn 和不使用gunicorn 都会发生故障。
\n

\n

这些细节都表明问题出在 celery/redis 后端。无论是在客户端/浏览器上还是在服务器\xe2\x80\x99s stdout/sterr 上都没有显示错误。

\n

如何让 celery/redis 后端工作？

\n

更新：在意识到该变量正在被使用并且其值根据引用它的文件而变化之后，我还尝试将创建和 的__name__代码移至，但无济于事。完全相同的事情发生了。celery_appLONG_CALLBACK_MANAGERapp.py

\n

代码

\n

应用程序.py

\n

import dash\nimport dash_bootstrap_components as dbc\n\nfrom website.layout_main import define_callbacks, layout\nfrom website.long_callback_manager import LONG_CALLBACK_MANAGER\n\n\napp = dash.Dash(\n    __name__,\n    update_title="Loading...",\n    external_stylesheets=[\n        dbc.themes.BOOTSTRAP,\n        "https://codepen.io/chriddyp/pen/bWLwgP.css"\n    ],\n    long_callback_manager=LONG_CALLBACK_MANAGER\n)\n\napp.title = "CS 236 | Project Submissions"\napp.layout = layout\ndefine_callbacks(app)\nserver = app.server  # expose for gunicorn\n\nif __name__ == "__main__":\n    app.run_server(debug=True, host="0.0.0.0")\n

\n

website/long_callback_manager.py 与磁盘缓存（功能）

\n

import os\nimport shutil\n\nimport diskcache\nfrom dash.long_callback import DiskcacheLongCallbackManager\n\nfrom util import RUN_DIR\n\n\ncache_dir = os.path.join(RUN_DIR, "callback_cache")\nshutil.rmtree(cache_dir, ignore_errors=True)  # ok if it didn\'t exist\n\ncache = diskcache.Cache(cache_dir)\n\nLONG_CALLBACK_MANAGER = DiskcacheLongCallbackManager(cache)\n

\n

website/long_callback_manager.py 与 celery/redis （不起作用）

\n

from dash.long_callback import CeleryLongCallbackManager\nfrom celery import Celery\n\ncelery_app = Celery(\n    __name__,\n    broker="redis://localhost:6379/0",\n    backend="redis://localhost:6379/1"\n)\n\nLONG_CALLBACK_MANAGER = CeleryLongCallbackManager(celery_app)\n

\n

网站/layout_main.py

\n

from typing import Union\n\nimport dash\nimport dash_bootstrap_components as dbc\nfrom dash import dcc, html\nfrom dash.dependencies import Input, Output, State\n\nfrom util.authenticator import authenticate\nfrom website import ID_LOGIN_STORE, NET_ID, PASSWORD\nfrom website.tabs.config import define_config_callbacks, layout as config_layout\nfrom website.tabs.log import define_log_callbacks, layout as log_layout\nfrom website.tabs.submit import define_submit_callbacks, layout as submit_layout\nfrom website.util import AUTH_FAILED_MESSAGE, STYLE_RED\n\n\n# cache\nLOGIN_INFO_EMPTY = {NET_ID: None, PASSWORD: None}\n# button display modes\nVISIBLE = "inline-block"\nHIDDEN = "none"\n\n# header\nID_LOGIN_BUTTON = "login-button"\nID_LOGGED_IN_AS = "logged-in-as"\nID_LOGOUT_BUTTON = "logout-button"\n# tabs\nID_TAB_SELECTOR = "tab-selector"\nID_SUBMIT_TAB = "submit-tab"\nID_LOG_TAB = "log-tab"\nID_CONFIG_TAB = "config-tab"\n# login modal\nID_LOGIN_MODAL = "login-modal"\nID_LOGIN_MODAL_NET_ID = "login-modal-net-id"\nID_LOGIN_MODAL_PASSWORD = "login-modal-password"\nID_LOGIN_MODAL_MESSAGE = "login-modal-message"\nID_LOGIN_MODAL_CANCEL = "login-modal-cancel"\nID_LOGIN_MODAL_ACCEPT = "login-modal-accept"\n# logout modal\nID_LOGOUT_MODAL = "logout-modal"\nID_LOGOUT_MODAL_CANCEL = "logout-modal-cancel"\nID_LOGOUT_MODAL_ACCEPT = "logout-modal-accept"\n\n\nlayout = html.Div([\n    dcc.Store(id=ID_LOGIN_STORE, storage_type="session", data=LOGIN_INFO_EMPTY),\n    html.Div(\n        [\n            html.H2("BYU CS 236 - Project Submission Website", style={"marginLeft": "10px"}),\n            html.Div(\n                [\n                    html.Div(id=ID_LOGGED_IN_AS, style={"display": HIDDEN, "marginRight": "10px"}),\n                    html.Button("Log in", id=ID_LOGIN_BUTTON, style={"display": VISIBLE}),\n                    html.Button("Log out", id=ID_LOGOUT_BUTTON, style={"display": HIDDEN})\n                ],\n                style={\n                    "marginRight": "25px",\n                    "display": "flex",\n                    "alignItems": "center"\n                }\n            )\n        ],\n        style={\n            "height": "100px",\n            "marginLeft": "10px",\n            "marginRight": "10px",\n            "display": "flex",\n            "alignItems": "center",\n            "justifyContent": "space-between"\n        }\n    ),\n    dcc.Tabs(id=ID_TAB_SELECTOR, value=ID_SUBMIT_TAB, children=[\n        dcc.Tab(submit_layout, label="New Submission", value=ID_SUBMIT_TAB),\n        dcc.Tab(log_layout, label="Submission Logs", value=ID_LOG_TAB),\n        dcc.Tab(config_layout, label="View Configuration", value=ID_CONFIG_TAB)\n    ]),\n    dbc.Modal(\n        [\n            dbc.ModalHeader("Log In"),\n            dbc.ModalBody([\n                html.Div(\n                    [\n                        html.Label("BYU Net ID:", style={"marginRight": "10px"}),\n                        dcc.Input(\n                            id=ID_LOGIN_MODAL_NET_ID,\n                            type="text",\n                            autoComplete="username",\n                            value="",\n                            style={"marginRight": "30px"}\n                        )\n                    ],\n                    style={\n                        "marginBottom": "5px",\n                        "display": "flex",\n                        "alignItems": "center",\n                        "justifyContent": "flex-end"\n                    }\n                ),\n                html.Div(\n                    [\n                        html.Label("Submission Password:", style={"marginRight": "10px"}),\n                        dcc.Input(\n                            id=ID_LOGIN_MODAL_PASSWORD,\n                            type="password",\n                            autoComplete="current-password",\n                            value="",\n                            style={"marginRight": "30px"}\n                        )\n                    ],\n                    style={\n                        "display": "flex",\n                        "alignItems": "center",\n                        "justifyContent": "flex-end"\n                    }\n                ),\n                html.Div(id=ID_LOGIN_MODAL_MESSAGE, style={"textAlign": "center", "marginTop": "10px"})\n            ]),\n            dbc.ModalFooter([\n                html.Button("Cancel", id=ID_LOGIN_MODAL_CANCEL),\n                html.Button("Log In", id=ID_LOGIN_MODAL_ACCEPT)\n            ])\n        ],\n        id=ID_LOGIN_MODAL,\n        is_open=False\n    ),\n    dbc.Modal(\n        [\n            dbc.ModalHeader("Log Out"),\n            dbc.ModalBody("Are you sure you want to log out?"),\n            dbc.ModalFooter([\n                html.Button("Stay Logged In", id=ID_LOGOUT_MODAL_CANCEL),\n                html.Button("Log Out", id=ID_LOGOUT_MODAL_ACCEPT)\n            ])\n        ],\n        id=ID_LOGOUT_MODAL,\n        is_open=False\n    )\n])\n\n\ndef on_click_login_modal_accept(net_id: Union[str, None], password: Union[str, None]) -> Union[str, None]:\n    # validate\n    if net_id is None or net_id == "":\n        return "BYU Net ID is required."\n    if password is None or password == "":\n        return "Submission Password is required."\n    # authenticate\n    auth_success = authenticate(net_id, password)\n    if auth_success:\n        return None\n    else:\n        return AUTH_FAILED_MESSAGE\n\n\ndef define_callbacks(app: dash.Dash):\n    @app.callback(Output(ID_LOGIN_MODAL, "is_open"),\n                  Output(ID_LOGIN_MODAL_MESSAGE, "children"),\n                  Output(ID_LOGOUT_MODAL, "is_open"),\n                  Output(ID_LOGIN_STORE, "data"),\n                  Input(ID_LOGIN_BUTTON, "n_clicks"),\n                  Input(ID_LOGIN_MODAL_CANCEL, "n_clicks"),\n                  Input(ID_LOGIN_MODAL_ACCEPT, "n_clicks"),\n                  Input(ID_LOGOUT_BUTTON, "n_clicks"),\n                  Input(ID_LOGOUT_MODAL_CANCEL, "n_clicks"),\n                  Input(ID_LOGOUT_MODAL_ACCEPT, "n_clicks"),\n                  State(ID_LOGIN_MODAL_NET_ID, "value"),\n                  State(ID_LOGIN_MODAL_PASSWORD, "value"),\n                  prevent_initial_call=True)\n    def on_login_logout_clicked(\n            n_login_clicks: int,\n            n_login_cancel_clicks: int,\n            n_login_accept_clicks: int,\n            n_logout_clicks: int,\n            n_logout_cancel_clicks: int,\n            n_logout_accept_clicks: int,\n            net_id: str,\n            password: str):\n        ctx = dash.callback_context\n        btn_id = ctx.triggered[0]["prop_id"].split(".")[0]\n        if btn_id == ID_LOGIN_BUTTON:\n            # show the login modal (with no message)\n            return True, None, dash.no_update, dash.no_update\n        elif btn_id == ID_LOGIN_MODAL_CANCEL:\n            # hide the login modal\n            return False, dash.no_update, dash.no_update, dash.no_update\n        elif btn_id == ID_LOGIN_MODAL_ACCEPT:\n            # try to actually log in\n            error_message = on_click_login_modal_accept(net_id, password)\n            if error_message is None:  # login success!\n                # hide the modal, update the login store\n                return False, dash.no_update, dash.no_update, {NET_ID: net_id, PASSWORD: password}\n            else:  # login failed\n                # show the message and keep the modal open\n                return dash.no_update, html.Span(error_message, style=STYLE_RED), dash.no_update, dash.no_update\n        elif btn_id == ID_LOGOUT_BUTTON:\n            # show the logout modal\n            return dash.no_update, dash.no_update, True, dash.no_update\n        elif btn_id == ID_LOGOUT_MODAL_CANCEL:\n            # hide the logout modal\n            return dash.no_update, dash.no_update, False, dash.no_update\n        elif btn_id == ID_LOGOUT_MODAL_ACCEPT:\n            # hide the logout modal and clear the login store\n            return dash.no_update, dash.no_update, False, LOGIN_INFO_EMPTY\n        else:  # error\n            print(f"unknown button id: {btn_id}")  # TODO: better logging\n            return [dash.no_update] * 4  # one for each Output\n\n    @app.callback(Output(ID_LOGIN_BUTTON, "style"),\n                  Output(ID_LOGGED_IN_AS, "children"),\n                  Output(ID_LOGGED_IN_AS, "style"),\n                  Output(ID_LOGOUT_BUTTON, "style"),\n                  Input(ID_LOGIN_STORE, "data"),\n                  State(ID_LOGIN_BUTTON, "style"),\n                  State(ID_LOGGED_IN_AS, "style"),\n                  State(ID_LOGOUT_BUTTON, "style"))\n    def on_login_data_changed(login_store, login_style, logged_in_as_style, logout_style):\n        # just in case no style is provided\n        if login_style is None:\n            login_style = dict()\n        if logged_in_as_style is None:\n            logged_in_as_style = dict()\n        if logout_style is None:\n            logout_style = dict()\n        # are they logged in or not?\n        if login_store[NET_ID] is None or login_store[PASSWORD] is None:\n            # not logged in\n            login_style["display"] = VISIBLE\n            logged_in_as_style["display"] = HIDDEN\n            logout_style["display"] = HIDDEN\n            return login_style, None, logged_in_as_style, logout_style\n        else:  # yes logged in\n            login_style["display"] = HIDDEN\n            logged_in_as_style["display"] = VISIBLE\n            logout_style["display"] = VISIBLE\n            return login_style, f"Logged in as \'{login_store[NET_ID]}\'", logged_in_as_style, logout_style\n\n    # define callbacks for all of the tabs\n    define_submit_callbacks(app)\n    define_log_callbacks(app)\n    define_config_callbacks(app)\n

\n

网站/选项卡/submit.py

\n

import os\nimport time\nfrom io import StringIO\nfrom typing import Callable, Dict, Union\n\nimport dash\nimport dash_bootstrap_components as dbc\nimport dash_gif_component as gif\nfrom dash import dcc, html\nfrom dash.dependencies import Input, Output, State\nfrom dash.exceptions import PreventUpdate\n\nfrom config.loaded_config import CONFIG\nfrom driver.passoff_driver import PassoffDriver\nfrom util.authenticator import authenticate\nfrom website import ID_LOGIN_STORE, NET_ID, PASSWORD\nfrom website.util import AUTH_FAILED_MESSAGE, save_to_submit, STYLE_DIV_VISIBLE, STYLE_DIV_VISIBLE_TOP_MARGIN, STYLE_HIDDEN, text_html_colorizer\n\n\n# submit tab IDs\nID_SUBMISSION_ROOT_DIV = "submission-root-div"\nID_SUBMIT_PROJECT_NUMBER_RADIO = "submit-project-number-radio"\nID_UPLOAD_BUTTON = "upload-button"\nID_UPLOAD_CONTENTS = "upload-contents"\nID_FILE_NAME_DISPLAY = "file-name-display"\nID_SUBMISSION_SUBMIT_BUTTON = "submission-submit-button"\nID_SUBMISSION_OUTPUT = "submission-output"\nID_SUBMISSION_LOADING = "submission-loading"\n# clear/refresh to submit again\nID_SUBMISSION_REFRESH_BUTTON = "submission-refresh-button"\nID_SUBMISSION_REFRESH_DIV = "submission-refresh-div"\nID_SUBMISSION_RESETTING_STORE = "submission-resetting-store"\n# info modal\nID_SUBMISSION_INFO_MODAL = "submission-info-modal"\nID_SUBMISSION_INFO_MODAL_MESSAGE = "submission-info-modal-message"\nID_SUBMISSION_INFO_MODAL_ACCEPT = "submission-info-modal-accept"\n# submission confirmation modal\nID_SUBMISSION_CONFIRMATION_MODAL = "submission-confirmation-modal"\nID_SUBMISSION_CONFIRMATION_MODAL_CANCEL = "submission-confirmation-modal-cancel"\nID_SUBMISSION_CONFIRMATION_MODAL_ACCEPT = "submission-confirmation-modal-accept"\n# store to trigger submission\nID_SUBMISSION_TRIGGER_STORE = "submission-trigger-store"\n\n\nLAYOUT_DEFAULT_CONTENTS = [\n    html.H3("Upload New Submission"),\n    html.P("Which project are you submitting?"),\n    dcc.RadioItems(\n        id=ID_SUBMIT_PROJECT_NUMBER_RADIO,\n        options=[{\n            "label": f" Project {proj_num}",\n            "value": proj_num\n        } for proj_num in range(1, CONFIG.n_projects + 1)]\n    ),\n    html.Br(),\n    html.P("Upload your .zip file here:"),\n    html.Div(\n        [\n            dcc.Upload(\n                html.Button("Select File", id=ID_UPLOAD_BUTTON),\n                id=ID_UPLOAD_CONTENTS,\n                multiple=False\n            ),\n            html.Pre("No File Selected", id=ID_FILE_NAME_DISPLAY, style={"marginLeft": "10px"})\n        ],\n        style={\n            "display": "flex",\n            "justifyContent": "flex-start",\n            "alignItems": "center"\n        }\n    ),\n    html.Button("Submit", id=ID_SUBMISSION_SUBMIT_BUTTON, style={"marginTop": "20px"}),\n    html.Div(id=ID_SUBMISSION_OUTPUT, style=STYLE_HIDDEN),\n    html.Div(\n        html.Div(\n            gif.GifPlayer(\n                gif=os.path.join("assets", "loading.gif"),\n                still=os.path.join("assets", "loading.png"),\n                alt="loading symbol",\n                autoplay=True\n            ),\n            style={"zoom": "0.2"}\n        ),\n        id=ID_SUBMISSION_LOADING,\n        style=STYLE_HIDDEN\n    ),\n    html.Div(\n        [\n            html.P("Reset the page to submit again:"),\n            html.Button("Reset", id=ID_SUBMISSION_REFRESH_BUTTON),\n        ],\n        id=ID_SUBMISSION_REFRESH_DIV,\n        style=STYLE_HIDDEN\n    ),\n    dbc.Modal(\n        [\n            dbc.ModalHeader("Try Again"),\n            dbc.ModalBody(id=ID_SUBMISSION_INFO_MODAL_MESSAGE),\n            dbc.ModalFooter([\n                html.Button("OK", id=ID_SUBMISSION_INFO_MODAL_ACCEPT)\n            ])\n        ],\n        id=ID_SUBMISSION_INFO_MODAL,\n        is_open=False\n    ),\n    dbc.Modal(\n        [\n            dbc.ModalHeader("Confirm Submission"),\n            dbc.ModalBody("Are you sure you want to officially submit?"),\n            dbc.ModalFooter([\n                html.Button("Cancel", id=ID_SUBMISSION_CONFIRMATION_MODAL_CANCEL),\n                html.Button("Submit", id=ID_SUBMISSION_CONFIRMATION_MODAL_ACCEPT)\n            ])\n        ],\n        id=ID_SUBMISSION_CONFIRMATION_MODAL,\n        is_open=False\n    )\n]\n\nlayout = html.Div(\n    [\n        html.Div(LAYOUT_DEFAULT_CONTENTS, id=ID_SUBMISSION_ROOT_DIV),\n        # having this store outside of the layout that gets reset means the long callback is not triggered\n        dcc.Store(id=ID_SUBMISSION_TRIGGER_STORE, storage_type="memory", data=False)  # data value just flips to trigger the long callback\n    ],\n    style={\n        "margin": "10px",\n        "padding": "10px",\n        "borderStyle": "double"\n    }\n)\n\n\ndef on_submit_button_clicked(\n        proj_number: Union[int, None],\n        file_name: Union[str, None],\n        file_contents: Union[str, None],\n        login_store: Union[Dict[str, str], None]) -> Union[str, None]:\n    # validate\n    if login_store is None or NET_ID not in login_store or PASSWORD not in login_store:\n        return "There was a problem with the login store!"\n    net_id = login_store[NET_ID]\n    password = login_store[PASSWORD]\n    if net_id is None or net_id == "" or password is None or password == "":\n        return "You must log in before submitting."\n    if proj_number is None:\n        return "The project number must be selected."\n    if not (1 <= proj_number <= CONFIG.n_projects):\n        return "Invalid project selected."\n    if file_name is None or file_name == "" or file_contents is None or file_contents == "":\n        return "A zip file must be uploaded to submit."\n    if not file_name.endswith(".zip"):\n        return "The uploaded file must be a .zip file."\n    # all good, it seems; return no error message\n    return None\n\n\ndef run_submission(proj_number: int, file_contents: str, login_store: Dict[str, str], set_progress: Callable):\n    # authenticate\n    print("authenticate")\n    net_id = login_store[NET_ID]\n    password = login_store[PASSWORD]\n    auth_success = authenticate(net_id, password)\n    if not auth_success:\n        set_progress([AUTH_FAILED_MESSAGE])\n        return\n    # write their zip file to the submit directory\n    print("save_to_submit")\n    save_to_submit(proj_number, net_id, file_contents)\n    # actually submit\n    print("actually submit")\n    this_stdout = StringIO()\n    this_stderr = StringIO()\n    driver = PassoffDriver(net_id, proj_number, use_user_input=False, stdout=this_stdout, stderr=this_stderr)\n    driver.start()  # runs in a thread of this same process\n    while True:  # make sure we print the output at least once, even it it finishes super fast\n        time.sleep(1)  # check output regularly\n        # TODO: change saved final_result in PassoffDriver to diff/error info?\n        # show results to the user\n        output = list()\n        output.append(html.P(f"submission for Net ID \'{net_id}\', project {proj_number}"))\n        stdout_val = this_stdout.getvalue()\n        output.append(html.Pre(text_html_colorizer(stdout_val)))\n        # output.append(html.Br())\n        stderr_val = this_stderr.getvalue

    

Answer 1

从plotly社区论坛重新发布解决方案：
https://community.plotly.com/t/long-callback-with-celery-redis-how-to-get-the-example-app-work/57663

概括

为了让长回调起作用，我需要启动 3 个协同工作的独立进程：

1. Redis 服务器：redis-server
2. 芹菜应用程序：celery -A app.celery worker --loglevel=INFO
3. 达世币应用程序：python app.py

上面列出的命令是最简单的版本。下面给出了使用的完整命令并进行了适当的修改。

细节

我将 celery 应用程序的声明从 移至 ，src/website/long_callback_manager.py以便src/app.py于外部访问：

import dash
import dash_bootstrap_components as dbc
from celery import Celery
from dash.long_callback import CeleryLongCallbackManager

from website.layout_main import define_callbacks, layout


celery_app = Celery(
    __name__,
    broker="redis://localhost:6379/0",
    backend="redis://localhost:6379/1"
)
LONG_CALLBACK_MANAGER = CeleryLongCallbackManager(celery_app)

app = dash.Dash(
    __name__,
    update_title="Loading...",
    external_stylesheets=[
        dbc.themes.BOOTSTRAP,
        "https://codepen.io/chriddyp/pen/bWLwgP.css"
    ],
    long_callback_manager=LONG_CALLBACK_MANAGER
)

app.title = "CS 236 | Project Submissions"
app.layout = layout
define_callbacks(app)
server = app.server  # expose for gunicorn

if __name__ == "__main__":
    app.run_server(debug=True, host="0.0.0.0")

然后我使用以下 bash 脚本来简化启动一切的过程：

#!/bin/bash
set -e  # quit on any error

# make sure the redis server is running
if ! redis-cli ping > /dev/null 2>&1; then
  redis-server --daemonize yes --bind 127.0.0.1
  redis-cli ping > /dev/null 2>&1  # the script halts if redis is not now running (failed to start)
fi

# activate the venv that has our things installed with pip
. venv/bin/activate

# make sure it can find the python modules, but still run from this directory
export PYTHONPATH=src

# make sure we have a log directory
mkdir -p Log

# start the celery thing
celery -A app.celery_app worker --loglevel=INFO >> Log/celery_info.log 2>&1 &

# start the server
gunicorn --workers=4 --name=passoff_website_server --bind=127.0.0.1:8050 app:server >> Log/gunicorn.log 2>&1

该脚本的进程是 celery 和 Gunicorn 子进程的父进程，并且所有进程都可以通过终止父进程作为一个捆绑包来终止。

编辑/更新：弃用long_callback

正如 @punit-vara 所指出的，Dash 2.6 现在允许@dash.callback使用background=True，这是推荐的 using 替代品long_callback。

请参阅Dash 的官方页面了解该主题