在 MySQL 中构建日志记录应用程序数据的最佳方法是什么

Question

我可能想多了，但是：

我有一个 API；我想记录对 API 的调用以进行统计/调试。理想情况下，读取速度应该很快，但不是必需的（主要由我读取）。但是，写入速度应该不错，因为我不想减慢 API 的速度。

我们将从头开始，但 API（和日志记录）将持续数年。数据越旧，它的用处就越小。在某些时候，我们可能希望完全删除旧数据（例如，删除超过三年的数据）。

目前正在运行 MySql 5.5。

构建表格的最佳方法是什么？按年对它进行分区，只是弥补未来分区的负载？

PARTITION BY RANGE( YEAR(date) ) (
    PARTITION p0 VALUES LESS THAN (2016),
    PARTITION p1 VALUES LESS THAN (2017),
    PARTITION p2 VALUES LESS THAN (2018),
    PARTITION p3 VALUES LESS THAN (2019),
    PARTITION p4 VALUES LESS THAN MAXVALUE
);

或者有没有办法创建自动滚动分区？

还是我只是按年份编制索引？

Answer 1

就在这里。我会将分区名称更改为更有意义的名称。

\n

假设数据库中的表如下所示mydb：

\n

CREATE TABLE mytable\n(\n    id INT NOT NULL AUTO_INCREMENT,\n    date DATETIME,\n    PRIMARY KEY (id,date),\n    KEY (date)\n)\nPARTITION BY RANGE( YEAR(date) ) (\n    PARTITION p2015 VALUES LESS THAN (2016),\n    PARTITION p2016 VALUES LESS THAN (2017),\n    PARTITION p2017 VALUES LESS THAN (2018),\n    PARTITION p2018 VALUES LESS THAN (2019),\n    PARTITION p9999 VALUES LESS THAN MAXVALUE\n);\n

\n

添加年份为 2019 的分区

\n

ALTER TABLE mytable DROP PARTITION p9999;\nALTER TABLE mytable ADD PARTITION\n    (PARTITION p2019 VALUES LESS THAN (2020)),\n    (PARTITION p9999 VALUES LESS THAN MAXVALUE)\n;\n

\n

删除最旧的分区

\n

ALTER TABLE mytable DROP PARTITION p2016;\n

\n

想要一个存储过程来执行以下操作吗？

\n

\n
• 今年之前删除分区
\n
• 追加 4 年后的新分区
\n

\n

这是代码

\n

DELIMITER $$\n\nDROP PROCEDURE IF EXISTS `mydb`.`sp_mytable_Rotate` $$\nCREATE DEFINER=`root`@`localhost` PROCEDURE `mydb`.`sp_mytable_Rotate` ()\nThis_Stored_Procedure:BEGIN\n\n  SET @year0 = YEAR(NOW());\n  SET @year4 = @year0 + 4;\n  SET @year5 = @year0 + 5;\n  SET @OldPartitionToKeep = CONCAT(\'p\',@year0);\n  SET @NewPartitionToMake = CONCAT(\'p\',@year4);\n\n  SELECT COUNT(1) INTO @zapcount FROM information_schema.partitions\n  WHERE table_schema = DATABASE() AND table_name = \'mytable\'\n  AND partition_name < @OldPartitionToKeep;\n\n  IF @zapcount = 0 THEN LEAVE This_Stored_Procedure; END IF;\n\n  SELECT GROUP_CONCAT(partition_name) INTO @partitions_to_drop\n  FROM information_schema.partitions  WHERE table_schema = DATABASE()\n  AND table_name = \'mytable\'  AND partition_name < @OldPartitionToKeep;\n  SET @partitions_to_drop= CONCAT(@partitions_to_drop,\',p9999\');\n\n  SET @sql = CONCAT(\'ALTER TABLE mytable DROP PARTITION \',@partitions_to_drop);\n  PREPARE s FROM @sql; EXECUTE s; DEALLOCATE PREPARE s;\n\n  SET @sql = \'ALTER TABLE mytable ADD PARTITION \';\n  SET @sql = CONCAT(@sql,\'(PARTITION \',@NewPartitionToMake,\' VALUES LESS THAN (\',@year5,\'))\');\n  PREPARE s FROM @sql; EXECUTE s; DEALLOCATE PREPARE s;\n\n  SET @sql = \'ALTER TABLE mytable ADD PARTITION (PARTITION p9999 VALUES LESS THAN MAXVALUE)\');\n  PREPARE s FROM @sql; EXECUTE s; DEALLOCATE PREPARE s;\n\nEND $$\n\nDELIMITER ;\n

\n

从这里开始，只需调用它

\n

USE mydb\nCALL sp_mytable_Rotate();\n

\n

1 月 1 日，您可以在轮换之前使用 mysqldump 备份旧分区的数据，如下所示：

\n

MYSQL_USER=root\nMYSQL_PASS=rootpassword\nMYSQL_CONN="-u${MYSQL_USER} -p${MYSQL_PASS}"\nTHISYEAR=`date +%Y`\n(( LASTYEAR = THISYEAR - 1 ))\nBACKUP_FILE=Data_From_${LASTYEAR}.sql\nBACKUP_GZIP=${BACKUP_FILE}.gz\nmysqldump ${MYSQL_CONN} mydb mytable --where="date<\'%{THISYEAR}-01-01\'"| gzip > ${BACKUP_GZIP}\nmysql ${MYSQL_CONN} -Dmydb -ANe"CALL sp_mytable_Rotate()"\n

\n

根据V\xc3\xa9race\的建议，如果要将数据备份到表中，ARCHIVE表可以压缩数据并且没有额外的索引开销。

\n

MYSQL_USER=root\nMYSQL_PASS=rootpassword\nMYSQL_CONN="-u${MYSQL_USER} -p${MYSQL_PASS}"\nTHISYEAR=`date +%Y`\n(( LASTYEAR = THISYEAR - 1 ))\nBACKUP_TABLE=mydata_${LASTYEAR}\nMKDT="MAKEDATE(${THISYEAR},1)"\nSQL="CREATE TABLE ${BACKUP_TABLE} ENGINE=ARCHIVE SELECT * FROM mytable WHERE 0=1;"\nSQL="${SQL} INSERT INTO ${BACKUP_TABLE} SELECT * FROM mytable WHERE date < ${MKDT}"\nSQL="${SQL} CALL sp_mytable_Rotate();"\nmysql ${MYSQL_CONN} -Dmydb -ANe"${SQL}"\n

\n

然后，您可以将ARCHIVE表的文件.frm移动到指定的备份卷。.ARZ

\n

试一试 ！！！

\n