EMR的外部配置单元metastore

Question

我正在使用默认配置单元元存储创建一个EMR集群，此后，我使用一些属性覆盖了hive-site.xml，这些属性将aws rds实例指向配置单元存储，一切都很好，但是重新启动配置单元服务器后，我无法将RDS用作配置单元metastore。它仍然在EMR创建的默认配置单元元存储中使用。

Answer 1

You can override the default configurations for applications by supplying a configuration object for applications when you create a cluster. The configuration object is referenced as a JSON file. Configuration objects consist of a classification, properties, and optional nested configurations. Properties are the settings you want to change in that file. You can specify multiple classifications for multiple applications in a single JSON object.

For overriding hive-site.xml with your external mysql metastore information, create a configuration file called hiveConfiguration.json containing edits to hive-site.xml:

[
    {
      "Classification": "hive-site",
      "Properties": {
        "javax.jdo.option.ConnectionURL": "jdbc:mysql:\/\/hostname:3306\/hive?createDatabaseIfNotExist=true",
        "javax.jdo.option.ConnectionDriverName": "org.mariadb.jdbc.Driver",
        "javax.jdo.option.ConnectionUserName": "username",
        "javax.jdo.option.ConnectionPassword": "password"
      }
    }
]

Use hiveConfiguration.json with the following AWS CLI command to create the cluster:

aws emr create-cluster --release-label emr-5.11.0 --instance-type m3.xlarge --instance-count 2 \
--applications Name=Hive --configurations ./hiveConfiguration.json --use-default-roles

Reference :

https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hive-metastore-external.html