运行Flask服务器(Apache)几天后的MySQL OperationalError

Question

我在Apache下有一个Flask服务器,我用它作为应用程序的Rest API,当服务器运行2-3天时它突然停止工作并加注OperationalError: MySQL Connection not available.

错误总是发生在login方法中,因为它是应用程序打开时首先调用的(但所有方法都遵循相同的模式).

这是login方法:

@app.route(LOGIN_API_URL, methods=['POST'])
def login():
    if (request.method == 'POST'):
        cursor = connection.cursor(buffered=True, dictionary=True)
        cursor.execute('select * from users where username = %s', (request.form['username'],))
        user = cursor.fetchone()
        if user is None or user['password'] != str(request.form['password']):
            abort(403)
        else:
            cursor.execute('update users set last_login = (%s) where user_id = %s', str(int(round(time.time() * 1000))), user['user_id'],)
            utils.safe_commit(connection, cursor)
            return utils.sanitize_response({'status':200, 'message':'Logged in'})

无论safe_commit与sanitize_response如下:

def sanitize_response(response, is_array=False):
    if response is None:
        return '[]' if is_array else '{}'
    else:
        return jsonify(response)

def safe_commit(connection, cursor):
    try:
        connection.commit()
    except:
        connection.rollback()
    finally:
        cursor.close()

起初我以为问题发生了,因为我没有buffered=True在调用fetchone方法的游标中使用.我又补充阅读之后参数此.

这是我的wsgi档案:

#!/usr/bin/python
import sys
sys.path.append("/var/www/protestr/")
from protestr import app as application

这是我的sites-available conf文件(我想说我已经尝试了很多组合的两个threads和processes参数,这个组合是让服务器运行最多的时间,通常是2到3天):

<VirtualHost *:80>
    ServerName protestr.tk
    DocumentRoot /var/www/protestr/

    WSGIDaemonProcess protestr user=www-data group=www-data processes=2 threads=25
    WSGIScriptAlias / /var/www/protestr/protestr.wsgi

    <Directory /var/www/protestr>
        WSGIProcessGroup protestr
        WSGIApplicationGroup %{GLOBAL}
        Require all granted
    </Directory>
</VirtualHost>

这些是error.log文件的内容:

[Fri May 12 03:34:14.967624 2017] [wsgi:error] [pid 18673:tid 2849002544] [remote 192.168.1.139:25727] [2017-05-12 03:34:14,963] ERROR in app: Exception on /api/v1/users/login [POST]
[Fri May 12 03:34:14.967812 2017] [wsgi:error] [pid 18673:tid 2849002544] [remote 192.168.1.139:25727] Traceback (most recent call last):
[Fri May 12 03:34:14.967861 2017] [wsgi:error] [pid 18673:tid 2849002544] [remote 192.168.1.139:25727]   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1982, in wsgi_app
[Fri May 12 03:34:14.967900 2017] [wsgi:error] [pid 18673:tid 2849002544] [remote 192.168.1.139:25727]     response = self.full_dispatch_request()
[Fri May 12 03:34:14.967937 2017] [wsgi:error] [pid 18673:tid 2849002544] [remote 192.168.1.139:25727]   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1614, in full_dispatch_request
[Fri May 12 03:34:14.967973 2017] [wsgi:error] [pid 18673:tid 2849002544] [remote 192.168.1.139:25727]     rv = self.handle_user_exception(e)
[Fri May 12 03:34:14.968007 2017] [wsgi:error] [pid 18673:tid 2849002544] [remote 192.168.1.139:25727]   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1517, in handle_user_exception
[Fri May 12 03:34:14.968043 2017] [wsgi:error] [pid 18673:tid 2849002544] [remote 192.168.1.139:25727]     reraise(exc_type, exc_value, tb)
[Fri May 12 03:34:14.968076 2017] [wsgi:error] [pid 18673:tid 2849002544] [remote 192.168.1.139:25727]   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1612, in full_dispatch_request
[Fri May 12 03:34:14.968111 2017] [wsgi:error] [pid 18673:tid 2849002544] [remote 192.168.1.139:25727]     rv = self.dispatch_request()
[Fri May 12 03:34:14.968144 2017] [wsgi:error] [pid 18673:tid 2849002544] [remote 192.168.1.139:25727]   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1598, in dispatch_request
[Fri May 12 03:34:14.968179 2017] [wsgi:error] [pid 18673:tid 2849002544] [remote 192.168.1.139:25727]     return self.view_functions[rule.endpoint](**req.view_args)
[Fri May 12 03:34:14.968251 2017] [wsgi:error] [pid 18673:tid 2849002544] [remote 192.168.1.139:25727]   File "/var/www/protestr/protestr.py", line 89, in login
[Fri May 12 03:34:14.968290 2017] [wsgi:error] [pid 18673:tid 2849002544] [remote 192.168.1.139:25727]     cursor = connection.cursor(buffered=True, dictionary=True)
[Fri May 12 03:34:14.968326 2017] [wsgi:error] [pid 18673:tid 2849002544] [remote 192.168.1.139:25727]   File "/usr/local/lib/python2.7/dist-packages/mysql/connector/connection.py", line 809, in cursor
[Fri May 12 03:34:14.968363 2017] [wsgi:error] [pid 18673:tid 2849002544] [remote 192.168.1.139:25727]     raise errors.OperationalError("MySQL Connection not available.")
[Fri May 12 03:34:14.968399 2017] [wsgi:error] [pid 18673:tid 2849002544] [remote 192.168.1.139:25727] OperationalError: MySQL Connection not available.

---

附加信息:

我在Banana Pi的armbian(Debian)下运行Apache/2.4.10 .

我真的不知道为什么服务器在运行一段时间后停止,我想我已经尝试了几乎所有的东西.

---

编辑:我已经在方法中cursor.close()抛出403错误之前添加了login.但这并不重要,因为我是唯一登录应用程序的人,而且我总是输入正确的凭据.

编辑2:正如@stamaimer告诉我的那样,如果我connection.ping()在获取任何游标之前添加它,它运行良好,但这种方法对我来说似乎是一种hacky方式,我不知道它是一个很好的解决方案,甚至是MySQL服务器为什么会掉线连接.

Answer 1

这个例子有点粗糙，但希望展示有关处理断开连接的逻辑。具体细节取决于在特定框架中获得连接的方式。

下面的代码假设使用retry; 如果需要，您可以使用不同的重试逻辑。

它还假设您在应用程序中使用框架提供的连接池。通常，您可以要求它重新连接，或者至少关闭空闲连接，以便另一个连接请求可以创建一个新连接。

def reconnect_on_failure(func):
    @retry(OperationalError, delay=0.25, backoff=1.5, max_delay=5)
    @wraps(func)
    def reconnecting_func(*args, **kwargs):
        try:
            return func(*args, **kwargs)
        except OperationalError as e:
            if 'connect' in e.msg.lower():
                force_reconnection_somehow()  # Look at your connection pool API.
                raise  # We want to retry on it
            raise Exception('Unhandled MySQL error', e)  # Will not retry.
    return reconnecting_func


@reconnect_on_failure
def something(...):
    connection = get_connecton_somehow()  # Look at the framework API.
    # A transaction implicitly begins with the first statement executed.
    cursor = connection.cursor()
    result = cursor.execute(...)  # do stuff
    connection.commit()

Exception您可以使用更窄的类，例如特定于您的应用程序的类，而不是过于宽泛的类；这个想法是，提出任何问题但 OperationalError不会触发重试，并且会Exception立即提出报告问题。

Grender 的编辑：我添加了@wraps装饰器以避免出现此处AssertionError所示的情况。