将最新的csv文件导入ssis中的sql server

Question

我有一个文件夹,我用时间戳每半小时收到.csv文件.现在,我需要从可用文件中获取最新文件并将其导入sql server.

例如

在我的源文件夹中,我有

test_01112012_120122.csv
test_01112012_123022.csv
test_01112012_123555.csv

现在我需要获取最新文件并在SSIS的帮助下将该文件导入sql server.

谢谢你的
光彩

Answer 1

即使您使用SSIS作为导入工具,也需要来自@garry Vass的代码或类似的代码.

在SSIS中,您需要将连接字符串更新为平面文件连接管理器以指向新文件.因此,您需要确定最新文件是什么.

查找最新文件

无论您是通过文件属性(Garry的代码)还是文件名的切片和切块来实现,都将取决于您的业务规则.它始终是最近修改过的文件(属性)还是需要基于被解释为序列的文件名.如果它test_01112012_120122.csv有错误并且内容被更新,这很重要.修改日期将更改,但文件名不会更改,并且这些更改不会被移植回数据库.

我建议你创建2个String类型的变量,并将其作用域命名为RootFolder和CurrentFile.或者,如果要限制特定类型,可以创建一个名为FileMask的文件*.csv.RootFolder将是您希望在其中查找文件的基本文件夹 C:\ssisdata\MyProject. CurrentFile将从完全限定路径的脚本中为最近修改的文件分配一个值.我发现在这一点上将设计时值分配给CurrentFile很有帮助,通常是集合中最旧的文件.

将脚本任务拖到控制流上并设置为ReadOnlyVariable User :: RootFolder(可选择User :: FileMask).您的ReadWriteVariable将是User :: CurrentFile.

这个脚本会进入public partial class ScriptMain: ...大括号内

    /// <summary>
    /// This verbose script identifies the most recently modified file of type fileMask
    /// living in RootFolder and assigns that to a DTS level variable.
    /// </summary>
    public void Main()
    {
        string fileMask = "*.csv";
        string mostRecentFile = string.Empty;
        string rootFolder = string.Empty;

        // Assign values from the DTS variables collection.
        // This is case sensitive. User:: is not required
        // but you must convert it from the Object type to a strong type
        rootFolder = Dts.Variables["User::RootFolder"].Value.ToString();

        // Repeat the above pattern to assign a value to fileMask if you wish
        // to make it a more flexible approach

        // Determine the most recent file, this could be null
        System.IO.FileInfo candidate = ScriptMain.GetLatestFile(rootFolder, fileMask);

        if (candidate != null)
        {
            mostRecentFile = candidate.FullName;
        }

        // Push the results back onto the variable
        Dts.Variables["CurrentFile"].Value = mostRecentFile;

        Dts.TaskResult = (int)ScriptResults.Success;
    }

    /// <summary>
    /// Find the most recent file matching a pattern
    /// </summary>
    /// <param name="directoryName">Folder to begin searching in</param>
    /// <param name="fileExtension">Extension to search, e.g. *.csv</param>
    /// <returns></returns>
    private static System.IO.FileInfo GetLatestFile(string directoryName, string fileExtension)
    {
        System.IO.DirectoryInfo directoryInfo = new System.IO.DirectoryInfo(directoryName);

        System.IO.FileInfo mostRecent = null;

        // Change the SearchOption to AllDirectories if you need to search subfolders
        System.IO.FileInfo[] legacyArray = directoryInfo.GetFiles(fileExtension, System.IO.SearchOption.TopDirectoryOnly);
        foreach (System.IO.FileInfo current in legacyArray)
        {
            if (mostRecent == null)
            {
                mostRecent = current;
            }

            if (current.LastWriteTimeUtc >= mostRecent.LastWriteTimeUtc)
            {
                mostRecent = current;
            }
        }

        return mostRecent;

        // To make the below code work, you'd need to edit the properties of the project
        // change the TargetFramework to probably 3.5 or 4. Not sure
        // Current error is the OrderByDescending doesn't exist for 2.0 framework
        //return directoryInfo.GetFiles(fileExtension)
        //     .OrderByDescending(q => q.LastWriteTimeUtc)
        //     .FirstOrDefault();
    }

    #region ScriptResults declaration
    /// <summary>
    /// This enum provides a convenient shorthand within the scope of this class for setting the
    /// result of the script.
    /// 
    /// This code was generated automatically.
    /// </summary>
    enum ScriptResults
    {
        Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
        Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
    };
    #endregion

}

更新连接管理器

此时,我们的脚本已为CurrentFile变量赋值.下一步是告诉SSIS我们需要使用该文件.在CSV的Connection Manager中,您需要为ConnectionString设置表达式(F4或右键单击并选择"属性").您要分配的值是我们的CurrentFile变量以及表达的方式@[User::CurrentFile]

最后,这些屏幕截图基于即将发布的SQL Server 2012,因此图标可能看起来不同但功能保持不变.