使用PowerShell和AWS加载数据本地INFILE

Question

过去两天我一直在搜索谷歌,试着了解我是否可以这样做.我在aws上托管了一个MySQL instace,我需要每小时加载一个小的CSV文件.LOAD DATA LOCAL INFILE似乎是票,但我似乎找不到任何能告诉我如何使用PowerShell从我的机器发送到AWS MySQL实例的地方.有人可以帮助我使用代码或者至少指出一些例子吗？

我可能很接近,我终于找到了一个视频:

[void][System.Reflection.Assembly]::LoadWithPartialName("MySql.Data")
$csvPath = 'D:/ETL_Data/TwitchTVData.csv'
$srvr = "myRDS.rds.amazonaws.com"
$DB = "TwitchTV_Data"
$uid = "User"
$pwd = "PWD"
$conStr = @"
server=$($srvr);port=3306; database=$();uid=$($uid);pwd=$($pwd)
"@
$conObj = New-Object MySql.Data.MySqlClient.MySqlConnection 
$conObj.ConnectionString = $conStr
$conObj.Open()
$queryStr = "LOAD DATA FROM $($csvPath)
    INTO TABLE TwitchTV_Top10_Game_Data
    FIELDS TERMINATED BY ','
    LINES TERMINATED BY '\n'
    (Game_Name, Viewers, Channels, Snapshot);"
$cmd = New-Object MySql.Data.MySqlClient.MySqlCommand($queryStr, $conn)
$conObj.Close()

我刚收到身份验证错误.所以现在我必须解决这个问题.

---

编辑:

我将原始代码修改为

[void][System.Reflection.Assembly]::LoadWithPartialName("MySql.Data")
$csvPath = 'D:\ETL_Data\TwitchTVData.csv'
$srvr = "myrds.rds.amazonaws.com"
$db = "TwitchTVData"
$uid = "user"
$pwd = "pwd"
$conStr = @"
server=$($srvr);port=3306; database=$($db);uid=$($uid);pwd=$($pwd)
"@
$conObj = New-Object MySql.Data.MySqlClient.MySqlConnection 
$conObj.ConnectionString = $conStr
$conObj.Open()

$csv = Import-Csv $csvPath | select -First 3

foreach ($record in $csv) {
    $queryStr = "INSERT INTO TwitchTV_Top10_Game_Data ($columns) VALUES ('$($record.Game_Name)','$($record.Viewers)','$($record.Channels)','$($record.Snapshot)');"
    $cmd = New-Object MySql.Data.MySqlClient.MySqlCommand($queryStr, $conObj)
}

$conObj.Close()

这会执行,但不会执行任何操作.没有数据进入表中,但没有错误.

以下答案如下:

$cred = Get-Credentials
$csvPath = 'D:\ETL_Data\TwitchTVData.csv'

$con = Connect-MySqlServer -Credential $cred -ComputerName myrds.us-west-2.rds.amazonaws.com -Port 3306

#Define our column headings (needed for the Insert statement)
$columns = "Game_Name,Viewers,Channels,Snapshot"

#import our csv
$csv = Import-Csv $csvPath | select -First 3

foreach ($record in $csv) {
    $query = "INSERT INTO TwitchTV_Top10_Game_Data ($columns) VALUES ('$($record.Game_Name)','$($record.Viewers)','$($record.Channels)','$($record.Snapshot)');"

    "Executing : $query"
    Invoke-MySqlQuery -Query $query -Verbose
    Start-Sleep -Milliseconds 250
}

这给出了评论中提到的错误.

Answer 1

所以,我带着一个兔子洞去了WAYYY.

但我找到了答案,但是我的不使用PowerShell,但mysql实用程序mysqldbimport,你可以在这里下载.(链接到我的Sql Utilities 1.6.5)

首先,您需要确保向数据库添加网络规则,以允许它修复3306到您的VM 的mySQL端口.确保您的设置至少如下所示:

现在,启动MySQL Utilities Console,这是你想要的命令和语法.

mysqldbimport --server=yourUserName:YourPassword@YourDBName.us-east-1.rds.amazonaws.com:3306 c:\PathTo\YourFile.csv

如果您需要端点名称(我YourDBName.us-east.etc上面提到的,请在此处获取).

我希望这有帮助!我将进一步尝试,看看我是否也可以找出PowerShell语法.

使用PowerShell执行此操作

我是谁让朋友挂？它也在PowerShell中工作......有一些警告.这是如何做:

1. 下载MySql ADO连接器
2. 从这篇博文中下载Adam Betram的真棒MySQL PowerShell模块

因此,要下载并导入Adam的模块,请运行以下命令:

invoke-WebRequest  -Uri https://github.com/adbertram/MySQL/archive/master.zip -OutFile  'C:\temp\MySQL.zip'
expand-archive C:\temp\MySQL.zip c:\somepath\MySql
import-module c:\somepath\MySql\MySQL-master\MySQL.psm1

现在,连接到您的Amazon MySQL实例

$cred = Get-Credential
$con = Connect-MySqlServer -ComputerName yourDB-east-1.rds.amazonaws.com -Credential $cred -Port 3306

现在,如果您需要创建表,首先在这样的哈希表中定义列标题和数据类型.

$table = @{"DeviceType"="VARCHAR (10)"
"DriveLetter"="VARCHAR (10)"
"DriveType  "="VARCHAR (50)"
"FreeSpace"="VARCHAR (10)"
"LastLoggedonUser"="VARCHAR (10)"
"MachineName"="VARCHAR (10)"
"Manufacturer"="VARCHAR (10)"
"Model"="VARCHAR (10)"
"SpaceInMB"="VARCHAR (10)"
}
#run this command to create this table
 New-MySqlTable -Table newtable01 -Database test -Column $table -Debug

如果有效,您将看到以下内容

Tables_in_test
--------------
newtable02    

我有一个.csv文件只是挂在我的硬盘驱动器上,其中有一些关于计算机硬盘的信息,所以我决定导入它,因为它是星期五晚上,孩子们终于睡着了,还有什么爸爸会为了好玩而做什么？

所以当你导入它们时,你基本上要解析每一行并运行一个INSERT INTO命令......但PowerShell是一个巨大的痛苦,因为它不尊重列顺序,所以你必须列出列名称...然后属性...按顺序.它很臭.

这是代码的其余部分.当然,您将修改以匹配您自己的列标题.

#Define our column headings (needed for the Insert statement)
$columns="MachineName,Model,Manufacturer,LastLoggedonUser,DriveLetter,DriveType,DeviceType,SpaceInMB,FreeSpace"

#import our csv
$csv = import-csv C:\temp\input2.csv | select -First 3

 ForEach ($record in $csv){
    $query = "INSERT INTO newtable01 ($columns) VALUES ('$($record.MachineName)','$($record.Model)',`
             '$($record.Manufacturer)','$($record.'Last Logged on User')','$($record.DriveLetter)',`
             '$($record.DriveType)','$($record.DeviceType)','$($record.'SpaceInMB ')','$($record.'FreeSpace(NA)')');"

 "about to exec:$query"
 Invoke-MySqlQuery -Query $query -Verbose
 start-sleep -Milliseconds 250

}

哪个在行动中看起来像这样

about to exec:INSERT INTO newtable01 (MachineName,Model,Manufacturer,LastLoggedonUser,DriveLetter,DriveType,DeviceType,SpaceInMB,FreeSpace) VALUES ('BLAHBLAH','Optiplex980',
             'Dell','DUDEBRO','C','WDCWD5000BEKT','Desktop','476837','NULL');
VERBOSE: 0 records found
about to exec:INSERT INTO newtable01 (MachineName,Model,Manufacturer,LastLoggedonUser,DriveLetter,DriveType,DeviceType,SpaceInMB,FreeSpace) VALUES ('BLAHBLAH','Optiplex980',
             'Dell','SOMEGUY','C','TOSHIBAMK2561GSYN','Desktop','238122','NULL');
VERBOSE: 0 records found
about to exec:INSERT INTO newtable01 (MachineName,Model,Manufacturer,LastLoggedonUser,DriveLetter,DriveType,DeviceType,SpaceInMB,FreeSpace) VALUES ('BLAHBLAH','Optiplex980',
             'Dell','HAMSANDWICH','C','ST9250410ASATADevice','Desktop','238','122');
VERBOSE: 0 records found

剩下的就是跑去Invoke-MYSqlQuery -Query "Select * from newtable01 limit 5" | format-table看它确实有效......