我正在使用PySpark和MLlib使用Spark 1.3.0,我需要保存并加载我的模型.我使用这样的代码(取自官方文档)
from pyspark.mllib.recommendation import ALS, MatrixFactorizationModel, Rating
data = sc.textFile("data/mllib/als/test.data")
ratings = data.map(lambda l: l.split(',')).map(lambda l: Rating(int(l[0]), int(l[1]), float(l[2])))
rank = 10
numIterations = 20
model = ALS.train(ratings, rank, numIterations)
testdata = ratings.map(lambda p: (p[0], p[1]))
predictions = model.predictAll(testdata).map(lambda r: ((r[0], r[1]), r[2]))
predictions.collect() # shows me some predictions
model.save(sc, "model0")
# Trying to load saved model and work with it
model0 = MatrixFactorizationModel.load(sc, "model0")
predictions0 = model0.predictAll(testdata).map(lambda r: ((r[0], r[1]), r[2]))
在我尝试使用model0之后,我得到一个很长的回溯,结束于此:
Py4JError: An error occurred …我正在为一个项目评估Mongo Spark连接器,我得到了不一致的结果.我在我的笔记本电脑上使用MongoDB服务器版本3.4.5,Spark(通过PySpark)版本2.2.0,Mongo Spark Connector版本2.11; 2.2.0本地.对于我的测试数据库,我使用安然数据集http://mongodb-enron-email.s3-website-us-east-1.amazonaws.com/我对Spark SQL查询感兴趣,当我开始运行简单的测试查询时对于计数我每次运行都收到不同的计数.这是我的mongo shell的输出:
> db.messages.count({'headers.To': 'eric.bass@enron.com'})
203
以下是我的PySpark shell的一些输出:
In [1]: df = spark.read.format("com.mongodb.spark.sql.DefaultSource").option("uri", "mongodb://127.0.0.1/enron_mail.messages").load()
In [2]: df.registerTempTable("messages")
In [3]: res = spark.sql("select count(*) from messages where headers.To='eric.bass@enron.com'")
In [4]: res.show()
+--------+
|count(1)|
+--------+
| 162|
+--------+
In [5]: res.show()
+--------+
|count(1)|
+--------+
| 160|
+--------+
In [6]: res = spark.sql("select count(_id) from messages where headers.To='eric.bass@enron.com'")
In [7]: res.show()
+----------+
|count(_id)|
+----------+
| 161|
+----------+
In [8]: res.show()
+----------+
|count(_id)|
+----------+
| 162|
+----------+ …