小编Jef*_*all的帖子

假设我有多行电话记录格式:

[CallingUser, ReceivingUser, Duration]

如果我想知道给定用户在电话上的总时间(用户是CallingUser或ReceivingUser的持续时间总和).

实际上,对于给定的记录,我想创建2对(CallingUser, Duration)和(ReceivingUser, Duration).

最有效的方法是什么？我可以加2 RDDs,但我不清楚这是一个好方法:

#Sample Data:
callData = sc.parallelize([["User1", "User2", 2], ["User1", "User3", 4], ["User2", "User1", 8]  ])


calls = callData.map(lambda record: (record[0], record[2]))

#The potentially inefficient map in question:
calls += callData.map(lambda record: (record[1], record[2]))


reduce = calls.reduceByKey(lambda a, b: a + b)