无法使用Elastic Beanstalk中的Flask读取JSON消息

Question

我有一个使用Elastic Beanstalk的Python'worker',它从SQS读取消息.虽然扩展很笨重,因为它基于cpu,但这种方法很好.因此,我正在尝试将其转换为使用AWS的新"工人层级环境".

在Flask的深处,但我在EB工作层上运行了Flask.目前它被设置为只记录它收到的消息信息 - 这是为了确保我可以在移动其他所有内容之前读取信息.不幸的是我看不到任何消息的迹象？

这是我的Flask测试代码:

import logging
import logging.handlers

from flask import Flask, request

logfile = "/opt/python/log/scan.log"
mylog = logging.getLogger('helloworld')
# (for brevity, log format/config code removed)

application = Flask(__name__)
app = application
app.Debug=True

@app.route('/', methods=['POST'])
def hello():
    global mylog
    err = "Unrecognized method"
    mylog.warning("Hello called")

    request_detail = """
# Before Request #
request.endpoint: {request.endpoint}
request.method: {request.method}
request.view_args: {request.view_args}
request.args: {request.args}
request.form: {request.form}
request.user_agent: {request.user_agent}
request.files: {request.files}
request.is_xhr: {request.is_xhr}
## request.headers ##
{request.headers}
    """.format(request=request).strip()
    mylog.warning(request_detail)

    mylog.warning("Moreinfo:")

    mylog.warning("Args:")
    for k in request.args.keys():
        mylog.warning(k + ": "+request.args[k])
    mylog.warning("Form:")
    for k in request.form.keys():
        mylog.warning(k + ": "+request.form[k])
    mylog.warning("Files:"+len(request.files))
    for k in request.files.keys():
        mylog.warning(k + ": "+request.files[k])

    try:
        myJSON = request.get_json(force=True)
        if myJSON is None:
            mylog.warning("JSON could not be forced")
        else:
            mylog.warning("MyJSON size: " + len(myJSON))
            mylog.warning( "MyJSON: {myJSON}".format(myJSON=myJSON))
        if request.json is None:
            mylog.warning("NO JSON")
    except Exception as e:
        mylog.warning("Exception: " + e)

    # the code below is executed if the request method
    # was GET or the credentials were invalid
    mylog.warning("failure 404")
    return 'Failure: '+err , 404, {'Content-Type': 'text/plain'}


if __name__ == '__main__':
    app.run(host='0.0.0.0', debug=True)

是的,大型长格式声明是从一本书借来的:-)

以下是消息的典型日志输出:

WARNING:2014-02-20 15:34:37,418: Hello called

WARNING:2014-02-20 15:34:37,419: 
# Before Request # 
request.endpoint: hello 
request.method: POST 
request.view_args: {} 
request.args: ImmutableMultiDict([])
request.form: ImmutableMultiDict([])
request.user_agent: aws-sqsd 
request.files: ImmutableMultiDict([])
request.is_xhr: False
## request.headers ## 
X-Aws-Sqsd-Msgid: 232eea42-5485-478c-a57f-4afddbf77ba9 
X-Aws-Sqsd-Receive-Count: 199 
X-Aws-Sqsd-Queue: #<AWS::SQS::Queue:0xb9255e90> 
Content-Length: 59 
User-Agent: aws-sqsd 
X-Aws-Sqsd-First-Received-At: 2014-02-20T13:55:34Z 
Host: localhost 
Content-Type: application/json

WARNING:2014-02-20 15:34:37,419: Moreinfo:

WARNING:2014-02-20 15:34:37,419: Args:

WARNING:2014-02-20 15:34:37,420: Form:

请注意,ImmutableMultiDict结构似乎都没有任何键.此外,没有任何JSON方法/属性返回任何内容.

Content-Length字段确实在日志条目之间变化,因此它看起来就像信息一样.但是我怎么看呢？

我的JSON消息使用BOTO写入SQS,例如:

  my_queue = conn.get_queue('my_queue_name')
  m = Message()
  m.set_body( json.dumps( my_structure ) )
  my_queue.write(m)

我还尝试使用SQS Web界面手动输入原始JSON消息.这也不起作用 - 我猜测我们可能有字符编码问题/

Answer 1

目前尚不清楚为什么你的消息记录被切断了; 也许你有一个缩进错误,你的部分日志代码不被视为view方法的一部分.

但是,如果我理解正确,Boto SQS消息不仅编码为JSON,JSON消息本身也是base64编码,这意味着该flask.get_json()方法不会为您解码.

相反,使用该request.get_data()方法访问原始POST数据并自行进行解码; 首先验证该request.content_length值是否在可容忍的大小内,以防止攻击者向您发送超大邮件:

from flask import json
import base64

if request.mime_type == 'application/json' and request.content_length <= 1024**2:
    # message is JSON and smaller than 1 megabyte
    try:
        decoded = base64.b64decode(request.get_data())
        data = json.loads(decoded)
    except (ValueError, TypeError):
        mylog.exception('Failed to decode JSON')
    else:
        # do something with the decoded data