将AWS Dynamodb备份到S3

Question

将AWS Dynamodb备份到S3

Ali*_*Ali 31 backup amazon-s3 elastic-map-reduce amazon-dynamodb

在亚马逊文档 http://aws.amazon.com/dynamodb/以及其他地方已经建议你可以使用Elastic Map Reduce备份你的dynamodb表,
我对这如何工作有一个大致的了解,但我不能找到任何指南或教程,

所以我的问题是如何自动化dynamodb备份(使用EMR)？

到目前为止,我认为我需要使用map函数创建一个"流"作业,该函数从dynamodb读取数据,并将reduce写入S3,我相信这些可以用Python(或java或其他几种语言)编写.

任何评论,澄清,代码示例,更正都表示赞赏.

Answer 1

Ali*_*Ali 35

随着AWS Data Pipeline的引入,以及用于dynamodb到S3备份的现成模板,最简单的方法是在Data Pipeline [link]中安排备份,

如果您有特殊需求(数据转换,非常精细的颗粒控制......),请考虑@greg的答案

请注意,默认情况下,m3.xlarge实例用于数据管道.我每6个小时运行一次备份,结果每天花费> 1美元 - 几乎是AWS总费用的一半.对于企业而言,它是完全值得的,但对于你几乎不想花很多钱的宠物项目,应该考虑它. (4认同)

Answer 2

gre*_*reg 14

使用MapReduce和DynamoDB有一些很好的指南.我前几天跟着这个,并且数据输出到S3的时间相当轻松.我认为最好的办法是创建一个执行备份任务的配置单元脚本,将其保存在S3存储桶中,然后使用AWS API为您的语言实用地启动新的EMR作业流程,完成备份.您可以将其设置为cron作业.

将数据从Dynamo导出到S3的配置单元脚本示例:

CREATE EXTERNAL TABLE my_table_dynamodb (
    company_id string
    ,id string
    ,name string
    ,city string
    ,state string
    ,postal_code string)
 STORED BY 'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler'
 TBLPROPERTIES ("dynamodb.table.name"="my_table","dynamodb.column.mapping" = "id:id,name:name,city:city,state:state,postal_code:postal_code");

CREATE EXTERNAL TABLE my_table_s3 (
    ,id string
    ,name string
    ,city string
    ,state string
    ,postal_code string)
 ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
 LOCATION 's3://yourBucket/backup_path/dynamo/my_table';

 INSERT OVERWRITE TABLE my_table_s3
 SELECT * from my_table_dynamodb;

Run Code Online (Sandbox Code Playgroud)

以下是一个PHP脚本示例,它将启动新的EMR作业流程:

$emr = new AmazonEMR();

$response = $emr->run_job_flow(
            'My Test Job',
            array(
                "TerminationProtected" => "false",
                "HadoopVersion" => "0.20.205",
                "Ec2KeyName" => "my-key",
                "KeepJobFlowAliveWhenNoSteps" => "false",
                "InstanceGroups" => array(
                    array(
                        "Name" => "Master Instance Group",
                        "Market" => "ON_DEMAND",
                        "InstanceType" => "m1.small",
                        "InstanceCount" => 1,
                        "InstanceRole" => "MASTER",
                    ),
                    array(
                        "Name" => "Core Instance Group",
                        "Market" => "ON_DEMAND",
                        "InstanceType" => "m1.small",
                        "InstanceCount" => 1,
                        "InstanceRole" => "CORE",
                    ),
                ),
            ),
            array(
                "Name" => "My Test Job",
                "AmiVersion" => "latest",
                "Steps" => array(
                    array(
                        "HadoopJarStep" => array(
                            "Args" => array(
                                "s3://us-east-1.elasticmapreduce/libs/hive/hive-script",
                                "--base-path",
                                "s3://us-east-1.elasticmapreduce/libs/hive/",
                                "--install-hive",
                                "--hive-versions",
                                "0.7.1.3",
                            ),
                            "Jar" => "s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar",
                        ),
                        "Name" => "Setup Hive",
                        "ActionOnFailure" => "TERMINATE_JOB_FLOW",
                    ),
                    array(
                        "HadoopJarStep" => array(
                            "Args" => array(
                                "s3://us-east-1.elasticmapreduce/libs/hive/hive-script",
                                "--base-path",
                                "s3://us-east-1.elasticmapreduce/libs/hive/",
                                "--hive-versions",
                                "0.7.1.3",
                                "--run-hive-script",
                                "--args",
                                "-f",
                                "s3n://myBucket/hive_scripts/hive_script.hql",
                                "-d",
                                "INPUT=Var_Value1",
                                "-d",
                                "LIB=Var_Value2",
                                "-d",
                                "OUTPUT=Var_Value3",
                            ),
                            "Jar" => "s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar",
                        ),
                        "Name" => "Run Hive Script",
                        "ActionOnFailure" => "CANCEL_AND_WAIT",
                    ),
                ),
                "LogUri" => "s3n://myBucket/logs",
            )
        );

}

Run Code Online (Sandbox Code Playgroud)