MongoDB SECONDARY在夜间成为RECOVERING

Question

我正在运行一个由3个成员组成的传统MongoDB副本集(数据中心A中的member1,数据中心B中的member2和member3).member1是当前的PRIMARY,我正在添加成员2和3 rs.add().他们正在执行初始同步并很快成为SECONDARY.一切都很好,两个成员的复制延迟是0秒,直到凌晨2点.

现在:每天凌晨2点,两个成员都会进入RECOVERING状态并完全停止复制,这导致我rs.printSlaveReplicationInfo()在早上时间进行查看会导致数小时的复制延迟.凌晨2点左右,我没有大量的插入或维护任务.

我在PRIMARY上获得以下日志条目:

2015-10-09T01:59:38.914+0200 [initandlisten] connection accepted from 192.168.227.209:59905 #11954 (37 connections now open)
2015-10-09T01:59:55.751+0200 [conn11111] warning: Collection dropped or state deleted during yield of CollectionScan
2015-10-09T01:59:55.869+0200 [conn11111] warning: Collection dropped or state deleted during yield of CollectionScan
2015-10-09T01:59:55.870+0200 [conn11111] getmore local.oplog.rs cursorid:1155433944036 ntoreturn:0 keyUpdates:0 numYields:1 locks(micros) r:32168 nreturned:0 reslen:20 134ms
2015-10-09T01:59:55.872+0200 [conn11111] end connection 192.168.227.209:58972 (36 connections now open)

而且,更有趣的是,我在两个SECONDARY上获得以下日志条目:

2015-10-09T01:59:55.873+0200 [rsBackgroundSync] repl: old cursor isDead, will initiate a new one
2015-10-09T01:59:55.873+0200 [rsBackgroundSync] replSet syncing to: member1:27017
2015-10-09T01:59:56.065+0200 [rsBackgroundSync] replSet error RS102 too stale to catch up, at least from member1:27017
2015-10-09T01:59:56.066+0200 [rsBackgroundSync] replSet our last optime : Oct  9 01:59:23 5617035b:17f
2015-10-09T01:59:56.066+0200 [rsBackgroundSync] replSet oldest at member1:27017 : Oct  9 01:59:23 5617035b:1af
2015-10-09T01:59:56.066+0200 [rsBackgroundSync] replSet See http://dochub.mongodb.org/core/resyncingaverystalereplicasetmember
2015-10-09T01:59:56.066+0200 [rsBackgroundSync] replSet error RS102 too stale to catch up
2015-10-09T01:59:56.066+0200 [rsBackgroundSync] replSet RECOVERING

这也是引人注目的 - 每天凌晨2点左右,oplog的开始"重置":

configured oplog size:   990MB
log length start to end: 19485secs (5.41hrs)
oplog first event time:  Fri Oct 09 2015 02:00:33 GMT+0200 (CEST)
oplog last event time:   Fri Oct 09 2015 07:25:18 GMT+0200 (CEST)
now:                     Fri Oct 09 2015 07:25:26 GMT+0200 (CEST)

我不确定这是否与该问题有某种关联.我也想知道这么小的延迟(Oct 9 01:59:23 5617035b:17f< - > Oct 9 01:59:23 5617035b:1af)会让会员变得陈旧.

这也可能是服务器(VM主机)时间问题还是完全不同的东西？(为什么第一个oplog事件每晚都被"重置"而不是"转移"到像NOW减去24小时的时间戳？)我可以做些什么来调查和避免？

Answer 1

增加 oplog 大小应该可以解决这个问题（根据我们的评论）。

给遇到此问题的其他人的一些参考

• 可能需要更大 Oplog 大小的工作负载
• 错误：replSet 错误 RS102 太陈旧，无法赶上link1和link2