标签: partitioning

在haskell中,如何生成集合的平衡分区？

假设我有一组{1,3,4,6,9},该组将是一个平衡的分区s1{9,3}和s2{6,4,1},看到s1-s2是1.

我创建了一个分区类型为day的表，并已从命令行工具将time_partitioning_expiration设置为1209600秒（14天）bq。我通过bq show在表上运行来验证设置是否正确，我可以看到

"timePartitioning": {
    "expirationMs": "1209600000", 
    "type": "DAY"
  }, 
  "type": "TABLE"

但是，分区中似乎有一些我希望删除的数据。

SELECT 
  count(*) as c,
  _partitiontime as pDate
FROM [poc.reporting]
group by pDate
; 

1   373800  2016-07-17 00:00:00 UTC  
2   640800  2016-07-18 00:00:00 UTC  
3   373800  2016-07-16 00:00:00 UTC  

我的理解是，设置time_partition_expiration将在内部删除/删除分区吗？如果是这样，为什么我没有删除分区？

我正在尝试按照此处的建议创建一个用于存储时间序列数据的垂直分区表。

到目前为止，我的架构如下所示：

CREATE TABLE events 
(
     topic text, 
     t timestamp, 
     value integer, 
     primary key(topic, t)
);

CREATE TABLE events_2014 
(
     primary key (topic, t), 
     check (t between '2014-01-01' and '2015-01-01') 
) INHERITS (events);

现在我正在尝试创建一个INSTEAD OF INSERT触发器，以便可以在events表中插入事件，并且该行将最终出现在正确的子表中。但是文档说INSTEAD OF INSERT触发器只能在视图上创建，不能在表（或子表）上创建：

CREATE OR REPLACE FUNCTION insert_events () RETURNS TRIGGER AS $insert_events$ BEGIN
  IF new.t between '2014-01-01' and '2015-01-01' THEN
    INSERT INTO events_2014 SELECT new.*;
  ...
  END IF
  RETURN NULL;
END;
$insert_events$ LANGUAGE PLPGSQL;

CREATE TRIGGER …

使用AWS :: Glue :: Table，您可以像此处那样建立Athena表。雅典娜支持在S3中基于文件夹结构对数据进行分区。我想从我的Glue模板对Athena表进行分区。

从AWS Glue Table TableInput看来，我可以使用它PartitionKeys来对数据进行分区，但是当我尝试使用以下模板时，Athena失败并且无法获取任何数据。

Resources:
  ...

  MyGlueTable:
    Type: AWS::Glue::Table
    Properties:
      DatabaseName: !Ref MyGlueDatabase
      CatalogId: !Ref AWS::AccountId
      TableInput:
        Name: my-glue-table
        Parameters: { "classification" : "json" }
        PartitionKeys:
          - {Name: dt, Type: string}
        StorageDescriptor:
          Location: "s3://elasticmapreduce/samples/hive-ads/tables/impressions/"
          InputFormat: "org.apache.hadoop.mapred.TextInputFormat"
          OutputFormat: "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"
          SerdeInfo:
            Parameters: { "separatorChar" : "," }
            SerializationLibrary: "org.apache.hive.hcatalog.data.JsonSerDe"
          StoredAsSubDirectories: false
          Columns:
            - {Name: requestBeginTime, Type: string}
            - {Name: adId, Type: string}
            - …

我有一个当前按年份拆分的分区方案设置。我正在尝试为明年创建一个新分区：

ALTER PARTITION SCHEME psYearlyPartition_Log

    NEXT USED Year7FileGroup;

ALTER PARTITION FUNCTION pfYearlyPartition_Log()

    SPLIT RANGE('20190101');

然而，这运行了很多小时并且没有完成。有谁知道为什么会发生这种情况，因为似乎还没有任何 2019 年的数据？

最后一个分区是从 2016-01-01 到今天。

我在这篇文章的底部运行了查询，结果如下：

 CREATE PARTITION FUNCTION [pfYearlyPartition](datetime) AS RANGE LEFT FOR VALUES('2012-01-01T00:00:00','2013-01-01T00:00:00','2014-01-01T00:00:00','2015-01-01T00:00:00','2016-01-01T00:00:00');

CREATE PARTITION SCHEME [psYearlyPartition_Table1] AS PARTTITION [pfYearlyPartition] TO ([YEAR1FileGroup],[YEAR2FileGroup],[YEAR3FileGroup],[YEAR4FileGroup],[YEAR5FileGroup],[YEAR6FileGroup],[YEAR11FileGroup]);
CREATE PARTITION SCHEME [psYearlyPartition_Table2] AS PARTTITION [pfYearlyPartition] TO ([YEAR1FileGroup],[YEAR2FileGroup],[YEAR3FileGroup],[YEAR4FileGroup],[YEAR5FileGroup],[YEAR6FileGroup],[YEAR11FileGroup]);
CREATE PARTITION SCHEME [psYearlyPartition_Table3] AS PARTTITION [pfYearlyPartition] TO ([YEAR1FileGroup],[YEAR2FileGroup],[YEAR3FileGroup],[YEAR4FileGroup],[YEAR5FileGroup],[YEAR6FileGroup],[YEAR11FileGroup]);
CREATE PARTITION SCHEME [psYearlyPartition_Table4] AS PARTTITION [pfYearlyPartition] TO ([YEAR1FileGroup],[YEAR2FileGroup],[YEAR3FileGroup],[YEAR4FileGroup],[YEAR5FileGroup],[YEAR6FileGroup],[YEAR11FileGroup]);
CREATE PARTITION SCHEME [psYearlyPartition_Table5] AS PARTTITION [pfYearlyPartition] TO ([YEAR1FileGroup],[YEAR2FileGroup],[YEAR3FileGroup],[YEAR4FileGroup],[YEAR5FileGroup],[YEAR6FileGroup],[YEAR11FileGroup]);
CREATE PARTITION SCHEME [psYearlyPartition_Table6] AS …