有人可以建议如何使用函数elasticsearch.helpers.streaming_bulk而不是elasticsearch.helpers.bulk将数据索引到elasticsearch中.
如果我只是改变streaming_bulk而不是批量,则没有任何内容被索引,所以我想它需要以不同的形式使用.
下面的代码从500个elemens的块中的CSV文件创建索引,类型和索引数据到elasticsearch.它工作正常,但我在徘徊是否有可能提高性能.这就是我想尝试使用streaming_bulk函数的原因.
目前我需要10分钟为100MB的CSV文档索引100万行.我使用两台机器,Centos 6.6,8 CPU-s,x86_64,CPU MHz:2499.902,Mem:15.574G总计.不确定它会更快.
es = elasticsearch.Elasticsearch([{'host': 'uxmachine-test', 'port': 9200}])
index_name = 'new_index'
type_name = 'new_type'
mapping = json.loads(open(config["index_mapping"]).read()) #read mapping from json file
es.indices.create(index_name)
es.indices.put_mapping(index=index_name, doc_type=type_name, body=mapping)
with open(file_to_index, 'rb') as csvfile:
reader = csv.reader(csvfile) #read documents for indexing from CSV file, more than million rows
content = {"_index": index_name, "_type": type_name}
batch_chunks = []
iterator = 0
for row in reader:
var = transform_row_for_indexing(row,fields, index_name, type_name,id_name,id_increment)
id_increment = id_increment + 1
#var = transform_row_for_indexing(row,fields, index_name, …尝试使用Swarm独立版和docker-compose版本"2"在不同的虚拟机上运行群集应用程序.设置覆盖网络.但是想强制某些容器在特定主机上运行.
在文档中有以下建议,但使用此参数我根本无法启动任何容器:
environment:
- "constraint:node==node-1"
ERROR: for elasticsearch1 Cannot create container for service elasticsearch1: Unable to find a node that satisfies the following conditions
[available container slots]
[node==node-1]
我们应该将主机注册为node-1 node-2 ...还是默认完成.
[root@ux-test14 ~]# docker node ls
Error response from daemon: 404 page not found
[root@ux-test14 ~]# docker run swarm list
[root@ux-test14 ~]#
[root@ux-test14 ~]# docker info
Containers: 8
Running: 6
Paused: 0
Stopped: 2
Images: 8
Server Version: swarm/1.2.5
Role: primary
Strategy: spread
Filters: health, port, containerslots, dependency, affinity, constraint …有我正在用某种功能一一处理的文件列表：
list_of_files = [file_1, file_2, file_3, .... file_n]
# each file we are processing in function function_x(file)
for file in list_of_files:
function_x(file)
但处理一个文件需要太长时间，所以我想并行处理 4 个文件，当其中任何一个文件完成后，继续下一个表单 list_of_files