Jog*_*esh 5 python sockets sftp timeout paramiko
我是Python新手,通过在线参考文献编写了这个脚本。该脚本基本上连接到网络外部的 sftp 服务器,并列出该服务器中的一个目录来收集 .xml 文件(从旧到最新)并循环中 1 个文件传输。
几分钟后,脚本在达到连接超时或没有要传输的文件时结束会话。休眠 5 秒,然后再次连接以获取下一组文件,就像近实时一样。在我们开始每隔 10 -15 分钟出现“套接字已关闭错误”之前,该脚本可以正常工作几个月。该脚本将按预期正常运行并开始传输文件,然后突然挂起 2-3 分钟,最终出现以下错误。
有时,当脚本连接到 sftp 服务器并开始传输文件时,在几个文件之后,将再次发送相同的错误
错误: 返回 self._send(s, m) 文件“C:\ProgramData\Anaconda3\lib\site-packages\paramiko\channel.py”,第 1198 行,在 _send 中引发 socket.error(“套接字已关闭”)OSError :套接字已关闭
import os
import shutil
import paramiko
from time import sleep
from datetime import datetime
import fnmatch
from lxml import etree
import lxml
localpath=r"D:/Imported_Files/"
logpath=r"D:/Imported_Files/log/"
temppath=r"D:/Imported_Files/file_rename_temp/"
while True:
#########
try: #### providing server credentials and connection to the sftp server with username and private key
host = "Hostname"
port = 22
transport = paramiko.Transport((host, port))
username = "username"
mykey = paramiko.RSAKey.from_private_key_file("C:/<PATH>",password='#########')
transport.connect(username = username, pkey = mykey)
sftp = paramiko.SFTPClient.from_transport(transport)
except Exception as e:
print(str(e))
sleep(30)
continue
try:
sftp.listdir()
sftp.chdir("outbox")
sftp.listdir("")
file_list=[x.filename for x in sorted(sftp.listdir_attr(),key= lambda f: f.st_mtime)] ## listing directory and get oldest files first in the list to process
file_list
except Exception as e: #### continue if there is an exception
print(str(e))
sleep(30)
continue
dttxt=str(datetime.now().strftime('%Y%m%d'))
for file in file_list: #### getting only files with .xml extension
if fnmatch.fnmatch(file,"*.xml"):
tempfilepath=temppath+file
localfilepath=localpath+file
file_info=sftp.stat(file)
file_mod_time=datetime.fromtimestamp(file_info.st_mtime) ### assigning modified timestamp of file to variable
file_acc_time=datetime.fromtimestamp(file_info.st_atime) ### assigning create timestamp of file to variable
try:
sftp.get(file,tempfilepath) ### performing sftp of the selected file from the list
except:
file_error_log = open(logpath+"files_not_processed"+dttxt+".log", "a") #### writing info to log
file_error_log.write("Failed:"+file+"\n")
file_error_log.close()
print("getting file "+file+" failed!")
continue
try:
sftp.remove(file) #### removing the file from sftp server after successful transfer
except:
print("deleteing file "+file+" failed")
os.remove(tempfilepath)
print("exception, moving on to next file")
file_error_ftp_remove = open(logpath+"files_not_deleted_ftp"+dttxt+".log", "a")
file_error_ftp_remove.write("Failed:"+file+"\n")
file_error_ftp_remove.close()
continue
try:
root = lxml.etree.parse(tempfilepath) #### parsing the file to extract a tag from .xml
system_load_id=root.find('system_load_id')
if system_load_id.text==None:
system_load_id_text=""
else:
system_load_id_text=system_load_id.text
new_filename=localpath+os.path.splitext(os.path.basename(tempfilepath))[0]+"-"+system_load_id_text+os.path.splitext(os.path.basename(localfilepath))[1]
sleep(0.3)
os.rename(tempfilepath, new_filename)
except:
sleep(0.3)
os.rename(tempfilepath, localpath+file)
print('Cant parse xml, hence moving the file as it is')
pass
########### file moved to final location after parsing .xml. writing to log and processing next file in the list
file_processed_log = open(logpath+"files_processed"+ str(datetime.now().strftime('%Y%m%d'))+".log", "a")
file_processed_log.write(str(datetime.now().strftime('%Y%m%d %H:%M:%S'))+" : "+file+","+str(file_mod_time)+","+str(file_acc_time)+"\n")
file_processed_log.close()
print(datetime.now())
sleep(5) ######## 1 session complete , sleeping and connecting to server again to get the next set of files
Run Code Online (Sandbox Code Playgroud)
问题并不一致。有时,我们会遇到这样的错误,例如一天 5 次,有时一天 100 多次
我在网上进行了研究,不确定问题出在哪里,因为该脚本运行良好几个月,每天近乎实时地处理 1000 个文件,没有任何问题
归档时间: |
|
查看次数: |
3797 次 |
最近记录: |