我想将一个大的 spark 数据框转换为超过 1000000 行的 Pandas。我尝试使用以下代码将 spark 数据帧转换为 Pandas 数据帧：
spark.conf.set("spark.sql.execution.arrow.enabled", "true")
result.toPandas()
但是，我得到了错误：
TypeError Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/pyspark/sql/dataframe.py in toPandas(self)
1949 import pyarrow
-> 1950 to_arrow_schema(self.schema)
1951 tables = self._collectAsArrow()
/usr/local/lib/python3.6/dist-packages/pyspark/sql/types.py in to_arrow_schema(schema)
1650 fields = [pa.field(field.name, to_arrow_type(field.dataType), nullable=field.nullable)
-> 1651 for field in schema]
1652 return pa.schema(fields)
/usr/local/lib/python3.6/dist-packages/pyspark/sql/types.py in <listcomp>(.0)
1650 fields = [pa.field(field.name, to_arrow_type(field.dataType), nullable=field.nullable)
-> 1651 for field in schema]
1652 return pa.schema(fields)
/usr/local/lib/python3.6/dist-packages/pyspark/sql/types.py in to_arrow_type(dt)
1641 else:
-> 1642 raise TypeError("Unsupported type in …我在机器（Ubuntu）上安装了apache-spark和pyspark，在Pycharm中，我还更新了环境变量（例如spark_home，pyspark_python）。我正在尝试做：
import os, sys
os.environ['SPARK_HOME'] = ".../spark-2.3.0-bin-hadoop2.7"
sys.path.append(".../spark-2.3.0-bin-hadoop2.7/bin/pyspark/")
sys.path.append(".../spark-2.3.0-bin-hadoop2.7/python/lib/py4j-0.10.6-src.zip")
from pyspark import SparkContext
from pyspark import SparkConf
sc = SparkContext('local[2]')
words = sc.parallelize(["scala", "java", "hadoop", "spark", "akka"])
print(words.count())
但是，我收到一些奇怪的警告：
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: java.lang.IllegalArgumentException
at org.apache.xbean.asm5.ClassReader.<init>(Unknown Source)
at org.apache.xbean.asm5.ClassReader.<init>(Unknown Source)
at org.apache.xbean.asm5.ClassReader.<init>(Unknown Source)
at org.apache.spark.util.ClosureCleaner$.getClassReader(ClosureCleaner.scala:46)
at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:449)
at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:432)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:103)
at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:103)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap$$anon$1.foreach(HashMap.scala:103)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.util.FieldAccessFinder$$anon$3.visitMethodInsn(ClosureCleaner.scala:432)
at org.apache.xbean.asm5.ClassReader.a(Unknown Source)
at org.apache.xbean.asm5.ClassReader.b(Unknown Source)
at org.apache.xbean.asm5.ClassReader.accept(Unknown Source)
at org.apache.xbean.asm5.ClassReader.accept(Unknown …我想使用 Python 中的 ElasticSearch 从给定 URL（带有前缀）获取数据。这是我的代码：
if __name__ == '__main__':
username = "xxxx"
password = "xxxx"
url = "http://xxxx.xxx:80/xxxx/xxxx/"
_es = Elasticsearch([url], http_auth=(username, password))
if _es.ping():
print('Connect')
print(_es)
res = _es.search(index='index1', body={"query": {"match_all": {}}})
print(res)
else:
print('It could not connect!')
实际上，我可以 ping _e，并且 _es 将是一个弹性对象，如下所示：
<Elasticsearch([{'主机': 'xxxx.xxx', 'url_prefix': 'xxxx/xxxx/', '端口': 80}])>
另外，为了验证我的URL、端口和前缀，我在Postman中检查了它，我可以正确获取Json格式的数据。但是当我在Python中运行代码时，我收到以下错误：
Connect
<Elasticsearch([{'host': 'xxxx.xxx', 'url_prefix': 'xxxx/xxxx/', 'port': 80}])>
Traceback (most recent call last):
File "/----.py", line 29, in <module>
res = _es.search(index='index1', body={"query": {"match_all": {}}})
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/client/utils.py", line 139, …