标签: amazon-elasticsearch

我设置了AWS Kinesis Firehose Stream以将数据提供给AWS ElasticSearch集群,我可以通过将文档发送到Firehose Stream来成功插入文档,Firehose Stream将它们加载到ElasticSearch中.

但我希望能够在将文档id发送到Firehose Stream时手动指定/设置文档的值.我成功使用AWS PHP SDK将数据发送到Firehose,我无法弄清楚是否有办法手动设置文档id.

$firehoseParams = [
    'DeliveryStreamName' => 'myStreamName', // REQUIRED
    'Record' => [ // REQUIRED
        'Data' => '{"json_encoded": "data", ...}', // REQUIRED
    ],
];
$firehoseResult = $this->_firehoseClient->putRecord($firehoseParams);

我试过在JSON数据中设置id,_id和esDocumentId值都无济于事.

有人有主意吗？

我无法找到如何使用amazon sdk与亚马逊弹性搜索集群进行交互的示例.有人能指出我的例子吗？我找到了像AWSCredentials和AWSElasticsearchClient这样的javadocs,只是寻找如何将所有内容粘合在一起的具体示例.我正在寻找如何做的事情:

- 创建索引

- 索引文件

- 删除文件

只是基本的操作.我假设有一种"亚马逊"方式来创建客户端并与群集交互,因为它们提供了一个sdk和一个用于签名请求的凭证对象.

我无法使用浏览器访问AWS Elasticsearch Kibana.

我在我的VPC中设置了一个完全如此处所述的Elasticsearch实例;

https://aws.amazon.com/blogs/aws/amazon-elasticsearch-service-now-supports-vpc/

我使用默认的IAM访问策略模板,该模板基本上是所有当前的IAM配置文件(*)

我的EC2 webapp(xenforo论坛)很愉快地联系在一起并且正在消失.

我想通过浏览器从家用PC访问我的elasticsearch域kibana端点.

我附加到群集配置的安全组包括允许从我的家庭宽带固定IP地址入站的所有TCP的规则.

我登录AWS控制台,点击elasticsearch域概述中的Kibana链接,然后......没有,超时.

我已经阅读了有关此事的所有内容.没有快乐 - 除非我应该签署我的https请求,这看起来很复杂,我的理解是IP访问应该可以配置安全组吗？

任何人都可以澄清吗？

我正在尝试连接到AWS Elasticsearch,但我总是收到以下错误:

Exception in thread "main" org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: []
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:278)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:197)
at org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106)
at org.elasticsearch.client.support.AbstractClient.index(AbstractClient.java:98)
at org.elasticsearch.client.transport.TransportClient.index(TransportClient.java:334)
at org.elasticsearch.action.index.IndexRequestBuilder.doExecute(IndexRequestBuilder.java:313)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)
at com.c_backendcrawler.utility.ElasticSearch.uploadObject(ElasticSearch.java:25)
at com.c_backendcrawler.Start.main(Start.java:34)

我的代码如下:

 //Create Client
    Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name", "zencubes-search").put("node.name","Darkhawk").build();
    TransportClient transportClient = new TransportClient(settings);
    transportClient.addTransportAddress(new InetSocketTransportAddress(
            "x.x.x.x",9300));
    return transportClient;

AWS Elasticsearch的输出:

    {
status: 200,
name: "Darkhawk",
cluster_name: "817880037706:zencubes-search",
version: {
number: "1.5.2",
build_hash: "62ff9868b4c8a0c45860bebb259e21980778ab1c",
build_timestamp: "2015-04-27T09:21:06Z",
build_snapshot: false,
lucene_version: "4.10.4"
},
tagline: "You Know, for Search" …

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html提到可以通过以下方式禁用自动创建索引:

通过在所有节点的配置文件中将action.auto_create_index设置为false,可以禁用自动索引创建.

如何在Elasticsearch中使用Java AWSElasticsearchClient类或以任何其他方式将其作为服务来完成？

从我所做的研究看来,由于HTTP仅支持Amazon Elasticsearch,因此两者无法协同工作.

希望有人可以澄清一下,实际上它不是Spring Data Elasticsearch所能实现的.

我正在尝试使用Lambda函数进行S3 Put事件通知.一旦我在S3存储桶中添加/添加任何新的JSON文件,就应该调用我的Lambda函数.我遇到的挑战是没有足够的文档来实现Java中的这种Lambda函数.我发现的大部分文档都是针对Node.js的

我想要,我的Lambda函数应该被调用,然后在Lambda函数中,我想使用添加的json,然后将该JSON发送到AWS ES服务.

但是我应该为此使用哪些类？有人对此有任何想法吗？S3 abd ES全部设置并运行.lambda的自动生成代码是`

@Override
public Object handleRequest(S3Event input, Context context) {
    context.getLogger().log("Input: " + input);

    // TODO: implement your handler
    return null;
}

接下来是什么？？

我正在尝试在AWS上创建一个VPC控制的弹性搜索服务.问题是我在运行以下代码时不断收到错误:'ValidationException:在继续操作之前,必须启用服务链接角色以授予Amazon ES访问VPC的权限'.

const AWS = require('aws-sdk');
AWS.config.update({region:'<aws-datacenter>'});
const accessPolicies = {
  Statement: [{
    Effect: "Allow",
    Principal: {
      AWS: "*"
    },
    Action: "es:*",
    Resource: "arn:aws:es:<dc>:<accountid>:domain/<domain-name/*"
  }]
};
const params = {
  DomainName: '<domain>',
  /* required */
  AccessPolicies: JSON.stringify(accessPolicies),
  AdvancedOptions: {
    EBSEnabled: "true",
    VolumeType: "io1",
    VolumeSize: "100",
    Iops: "1000"
  },
  EBSOptions: {
    EBSEnabled: true,
    Iops: 1000,
    VolumeSize: 100,
    VolumeType: "io1"
  },
  ElasticsearchClusterConfig: {
    DedicatedMasterCount: 3,
    DedicatedMasterEnabled: true,
    DedicatedMasterType: "m4.large.elasticsearch",
    InstanceCount: 2,
    InstanceType: 'm4.xlarge.elasticsearch',
    ZoneAwarenessEnabled: true
  },
  ElasticsearchVersion: '5.5',
  SnapshotOptions: {
    AutomatedSnapshotStartHour: 3 …

我正在将 AWS ElasticSearch 与 Cognito 结合使用，并在尝试登录时收到以下错误

Sorry! Something went wrong during authentication between Kibana and Amazon Cognito. What happened? User: x:x:x::xx:x is not authorized to perform: es:ESHttpGet

我的 AWS ES 集群访问策略看起来像

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::xxxxxxxxx:role/service-role/CognitoAccessForAmazonES"
      },
      "Action": "es:ESHttp*",
      "Resource": "arn:aws:es:xxxxxxx:xxxxxxx:domain/elastic-blah/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": "xx.xx.xx.xx/32"
        }
      }
    }
  ]
}

ElasticSearch 新手。我按照本指南进行设置：https ://john.soban.ski/boto3-ec2-to-amazon-elasticsearch.html

我在那里运行了“connect_to_es.py”脚本，奇怪的是它第一次工作，但在随后的运行中，它开始抛出此错误：

Traceback (most recent call last):
  File "../connect_to_es.py", line 21, in <module>
    print(json.dumps(es.info(), indent=4, sort_keys=True))
  File "/home/ubuntu/projects/.venv/lib/python3.8/site-packages/elasticsearch/client/utils.py", line 168, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/home/ubuntu/projects/.venv/lib/python3.8/site-packages/elasticsearch/client/__init__.py", line 294, in info
    return self.transport.perform_request(
  File "/home/ubuntu/projects/.venv/lib/python3.8/site-packages/elasticsearch/transport.py", line 413, in perform_request
    _ProductChecker.raise_error(self._verified_elasticsearch)
  File "/home/ubuntu/projects/.venv/lib/python3.8/site-packages/elasticsearch/transport.py", line 630, in raise_error
    raise UnsupportedProductError(message)
elasticsearch.exceptions.UnsupportedProductError: The client noticed that the server is not a supported distribution of Elasticsearch

我的elasticsearch python库版本是7.14，我在AWS上的elasticsearch运行的是7.10。对这里发生的事情有什么想法吗？

代码副本：

from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth
import …