为什么Presto比Spark SQL更快?Presto和Spark SQL在计算架构和内存管理方面有什么区别?
我通过多进程池为每个子进程实例化一个 grpc 服务器。当我使用多个客户端访问服务器时,发现以下两个问题?
顺便说一下,我的开发环境是:
[OS]
ProductName: Mac OS X
ProductVersion: 10.14.6
BuildVersion: 18G5033
[packages]
grpcio = '1.30.0'
grpcio-tools = '1.30.0'
multiprocess = "0.70.10"
grpcio-status = "1.30.0"
googleapis-common-protos = "1.52.0"
[requires]
python_version = "3.8.3"
Run Code Online (Sandbox Code Playgroud)
这是服务器输出:
[PID 83287] Binding to 'localhost:52909'
[PID 83288] Starting new server.
[PID 83289] Starting new server.
[PID 83290] Starting new server.
[PID 83291] Starting new server.
[PID 83292] Starting new server.
[PID 83293] Starting new server.
[PID 83294] Starting new server.
[PID 83295] Starting …Run Code Online (Sandbox Code Playgroud) 我尝试使用 pypy3 安装 pyarrow,但出现一些错误。
基本信息如下:
macOS 10.15.7
Xcode 12.3
python version 3.7.9
pypy3 version 7.3.3
pyarrow version 0.17.1
cmd is 'pip_pypy3 install pyarrow==0.17.1'
Run Code Online (Sandbox Code Playgroud)
日志中的一些关键信息和错误内容:
...
Requirement already satisfied: numpy>=1.14 in /usr/local/Cellar/pypy3/7.3.3/libexec/site-packages (from pyarrow==0.17.1) (1.19.5)
...
cmake -DPYTHON_EXECUTABLE=/usr/local/Cellar/pypy3/7.3.3/bin/pypy3 -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release /private/var/folders/7p/d9yrtx8s2h94h9bh3x801zmr0000gn/T/pip-install-jagh2frg/pyarrow_522bc325fbd74d9ebdf84f29e3a66c0c
...
2021-01-10T21:19:27,670 -- Found Python3: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 (found version "3.7.9") found components: Interpreter Development NumPy Development.Module Development.Embed
2021-01-10T21:19:27,713 -- Found Python3Alt: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7
2021-01-10T21:19:28,226 CMake Warning (dev) at …Run Code Online (Sandbox Code Playgroud) python ×2
cmake ×1
grpc ×1
grpc-python ×1
multiprocess ×1
presto ×1
pyarrow ×1
pypy ×1
python-3.x ×1