MySQL连接器python 35资源暂时不可用于大型查询？

Question

我试图使用MySQL连接器来替代pymysql,因为它支持在一个查询多个语句对有些更新我不得不作出(这是我与其他问题),但是它没有发送过甚的我的其他使用案例大选择陈述.

我有一个动态生成的Select语句,它检索与传入的任何指定值匹配的所有行; 例如Select * from table where col_a in (val_1, val_2.... val_350,000)

我的select语句一直出现同样的错误:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/mysql/connector/network.py", line 212, in send_compressed
    self.sock.sendall(zip_packet)
BlockingIOError: [Errno 35] Resource temporarily unavailable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/threading.py", line 921, in _bootstrap_inner
    self.run()
  File "/Users/maldeiri/raw_data_processing/sql_retriever.py", line 22, in run
    self.mysql_cursor.execute(self.sql_statement)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/mysql/connector/cursor.py", line 515, in execute
    self._handle_result(self._connection.cmd_query(stmt))
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/mysql/connector/connection.py", line 488, in cmd_query
    result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/mysql/connector/connection.py", line 261, in _send_cmd
    packet_number)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/mysql/connector/network.py", line 215, in send_compressed
    errno=2055, values=(self.get_address(), _strioerror(err)))
mysql.connector.errors.OperationalError: 2055: Lost connection to MySQL server at 'database_end_point:3306', system error: 35 Resource temporarily unavailable

如果我有compress = True或False,则会发生这种情况.我也不认为这是服务器端问题,因为我提到完全相同的Select语句似乎与pymysql运行相同的代码和机器.

我有什么想法可以解决这个问题吗？

Answer 1

不要构建那个可怕的IN(...)，而是将值放入表中，每行一个。

然后对真实表进行操作JOIN以获取所需的行。（确保col_a在真实表中对其进行索引；不必在额外表中对其进行索引。）

如果庞大的列表可能有重复数据，您可能应该首先对列表进行重复数据删除。看看 Python 是否可以足够轻松地完成它。如果没有，您可以将这一列作为插入它们的PRIMARY KEY操作。INSERT IGNORE或者，

CREATE TABLE t (val) ENGINE=MyISAM;
INSERT or LOAD DATA ... (no dedupping)
SELECT rt.* FROM real_table
    JOIN ( SELECT DISTINCT val FROM t )  ON rt.val = t.val;