在 CircleCI 上运行 Jest 测试时如何防止 postgres 死锁？

Question

当我在 CircleCI 上运行我的测试时，它会多次记录以下消息，最终测试失败，因为由于死锁，没有任何数据库方法可以检索数据：

{
  "message": "Error running raw sql query in pool.",
  "stack": "error: deadlock detected\n    at Connection.Object.<anonymous>.Connection.parseE (/home/circleci/backend/node_modules/pg/lib/connection.js:567:11)\n    at Connection.Object.<anonymous>.Connection.parseMessage (/home/circleci/-backend/node_modules/pg/lib/connection.js:391:17)\n    at Socket.<anonymous> (/home/circleci/backend/node_modules/pg/lib/connection.js:129:22)\n    at emitOne (events.js:116:13)\n    at Socket.emit (events.js:211:7)\n    at addChunk (_stream_readable.js:263:12)\n    at readableAddChunk (_stream_readable.js:250:11)\n    at Socket.Readable.push (_stream_readable.js:208:10)\n    at TCP.onread (net.js:597:20)",
  "name": "error",
  "length": 316,
  "severity": "ERROR",
  "code": "40P01",
  "detail": "Process 1000 waits for AccessExclusiveLock on relation 17925 of database 16384; blocked by process 986.\nProcess 986 waits for RowShareLock on relation 17870 of database 16384; blocked by process 1000.",
  "hint": "See server log for query details.",
  "file": "deadlock.c",
  "line": "1140",
  "routine": "DeadLockReport",
  "level": "error",
  "timestamp": "2018-10-15T20:54:29.221Z"
}

这是我运行的测试命令： jest --logHeapUsage --forceExit --runInBand

• 我也试过这个： jest --logHeapUsage --forceExit --maxWorkers=2

几乎所有的测试都运行某种数据库功能。当我们添加更多测试时，这个问题才开始出现。有没有其他人遇到过同样的问题？

Answer 1

根据错误消息，我们由于 RowShareLock 而出现死锁；

这意味着两个事务（我们称它们为 transactionOne 和 transactionTwo）已锁定另一个事务所需的资源

例子：

transactionOne locks record in UserTable with userId = 1
transactionTwo locks record in UserTable with userId = 2

transactionOne attempts to update in UserTable for userId = 2, but since it is locked by another transaction - it waits for the lock to be released

transactionTwo attempts to update in UserTable for userId = 1, but since it is locked by another transaction - it waits for the lock to be released

Now the SQL engine detects that there is a deadlock and randomly picks one of the transactions and terminates it.

Lets say the SQL engine picks transactionOne and terminates it. This will result in the exception that is posted in the question.

transactionTwo is now allowed to perform an update in UserTable for user with userId = 1. 
transactionTwo completes with success

SQL 引擎检测死锁的速度非常快，并且异常会立即出现。

这就是僵局的原因。死锁可能有不同的根本原因。

我看到你使用 pg 插件。确保在事务中正确使用它：pg node-postgres transactions

我怀疑一些不同的根本原因及其解决方案：

原因 1：针对同一数据库实例运行多个测试

可能是不同的 ci 管道针对同一个 Postgres 实例执行相同的测试

解决方案：

这是最不可能发生的情况，但 CI 管道应该在每次运行时配置自己单独的 Postgres 实例。

原因 2：未使用适当的 catch("ROLLBACK") 处理事务

这意味着某些交易可能会保持活动状态并阻止其他交易。

解决方案：所有事务都应该有适当的错误处理。

transactionOne locks record in UserTable with userId = 1
transactionTwo locks record in UserTable with userId = 2

transactionOne attempts to update in UserTable for userId = 2, but since it is locked by another transaction - it waits for the lock to be released

transactionTwo attempts to update in UserTable for userId = 1, but since it is locked by another transaction - it waits for the lock to be released

Now the SQL engine detects that there is a deadlock and randomly picks one of the transactions and terminates it.

Lets say the SQL engine picks transactionOne and terminates it. This will result in the exception that is posted in the question.

transactionTwo is now allowed to perform an update in UserTable for user with userId = 1. 
transactionTwo completes with success

原因3：并发。例如： 测试是并行运行的，它们会导致死锁。

我们正在编写可扩展的应用程序。这意味着僵局是不可避免的。我们必须为此做好准备并妥善处理。

解决方案：使用“让我们再试一次”策略。当我们检测到代码中存在死锁异常时，我们只需重试有限次。十多年来，这种方法已经在我所有的生产应用程序中得到了验证。

使用辅助函数的解决方案：

  const client = await pool.connect()
  try {
    await client.query('BEGIN')

    //do what you have to do

    await client.query('COMMIT')
  } catch (e) {
    await client.query('ROLLBACK')
    throw e
  } finally {
    client.release()
  }

如果代码变得复杂/嵌套。我们可以尝试使用高阶函数用另一种解决方案来实现

//Sample deadlock wrapper
const handleDeadLocks = async (action, currentAttepmt = 1 , maxAttepmts = 3) {
  try {
    return await action();
  } catch (e) {
    //detect it is a deadlock. Not 100% sure whether this is deterministic enough
    const isDeadlock = e.stack?.includes("deadlock detected");
    const nextAttempt = currentAttepmt + 1;
    if (isDeadlock && nextAttempt <= maxAttepmts) {
      //try again
      return await handleDeadLocks(action, nextAttempt, maxAttepmts); 
    } else {
      throw e;
    }

  }
}

//our db access functions
const updateUserProfile = async (input) => {
  return handleDeadLocks(async () => {
    //do our db calls
  });
};