Python中的大型Web数据集 - 如何处理非常大的数组？

Question

我们目前正在使用Cassandra(http://cassandra.apache.org/)获取时间序列数据.Cassandra读取速度非常快,但是在我们提供数据之前,我们必须对数据执行一系列计算(实际上我们正在模仿SQL的SUM和GROUP BY功能 - Something Cassandra不支持开箱即用)

我们熟悉Python(在某种程度上)并决定构建一个脚本来查询我们的Cassandra集群以及执行数学并以JSON格式呈现结果:

query = (
    "SELECT query here...")

startTimeQuery = time.time()

# Executes cassandra query
rslt = cassession.execute(query)

print("--- %s seconds to query ---" % (time.time() - startTimeQuery))

tally = {}

startTimeCalcs = time.time()
for row in rslt:
    userid = row.site_user_id

    revenue = (int(row.revenue) - int(row.reversals_revenue or 0))
    accepted = int(row.accepted or 0)
    reversals_revenue = int(row.reversals_revenue or 0)
    error = int(row.error or 0)
    impressions_negative = int(row.impressions_negative or 0)
    impressions_positive = int(row.impressions_positive or 0)
    rejected = int(row.rejected or 0)
    reversals_rejected = int(row.reversals_rejected or 0)

    if tally.has_key(userid):
        tally[userid]["revenue"] += revenue
        tally[userid]["accepted"] += accepted
        tally[userid]["reversals_revenue"] += reversals_revenue
        tally[userid]["error"] += error
        tally[userid]["impressions_negative"] += impressions_negative
        tally[userid]["impressions_positive"] += impressions_positive
        tally[userid]["rejected"] += rejected
        tally[userid]["reversals_rejected"] += reversals_rejected
    else:
        tally[userid] = {
            "accepted": accepted,
            "error": error,
            "impressions_negative": impressions_negative,
            "impressions_positive": impressions_positive,
            "rejected": rejected,
            "revenue": revenue,
            "reversals_rejected": reversals_rejected,
            "reversals_revenue": reversals_revenue
        }


print("--- %s seconds to calculate results ---" % (time.time() - startTimeCalcs))

startTimeJson = time.time()
jsonOutput =json.dumps(tally)
print("--- %s seconds for json dump ---" % (time.time() - startTimeJson))

print("--- %s seconds total ---" % (time.time() - startTimeQuery))

print "Array Size: " + str(len(tally)) 

这是我们得到的那种输出:

--- 0.493520975113 seconds to query ---
--- 23.1472680569 seconds to calculate results ---
--- 0.546246051788 seconds for json dump ---
--- 24.1871240139 seconds total ---
Array Size: 198124

我们在计算上花费了大量时间,我们知道问题不在于总和和分组本身:这只是阵列的绝对大小.

我们已经听到了关于numpy的一些好消息,但是我们数据的性质使得矩阵大小未知.

我们正在寻找有关如何处理此问题的任何提示.包括完全不同的编程方法.

Answer 1

Cassandra 2.2 及更高版本允许用户定义聚合函数。您可以使用它在 cassandra 端执行列汇总。请参阅DataStax 文章，了解有关用户定义聚合的数据