以下程序:
import multiprocessing,operator
f = operator.itemgetter(0)
# def f(*a): return operator.itemgetter(0)(*a)
if __name__ == '__main__':
multiprocessing.Pool(1).map(f, ["ab"])
Run Code Online (Sandbox Code Playgroud)
失败,出现以下错误:
Process PoolWorker-1:
Traceback (most recent call last):
File "/usr/lib/python3.2/multiprocessing/process.py", line 267, in _bootstrap
self.run()
File "/usr/lib/python3.2/multiprocessing/process.py", line 116, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.2/multiprocessing/pool.py", line 102, in worker
task = get()
File "/usr/lib/python3.2/multiprocessing/queues.py", line 382, in get
return recv()
TypeError: itemgetter expected 1 arguments, got 0
Run Code Online (Sandbox Code Playgroud)
为什么我会得到错误(在Linux x64上的cPython 2.7和3.2上),如果我取消注释第三行,为什么它会消失?
我想运行一个使用协同程序和多线程请求URL的服务.但是我无法将协同程序传递给执行程序中的工作程序.有关此问题的最小示例,请参阅下面的代码:
import time
import asyncio
import concurrent.futures
EXECUTOR = concurrent.futures.ThreadPoolExecutor(max_workers=5)
async def async_request(loop):
await asyncio.sleep(3)
def sync_request(_):
time.sleep(3)
async def main(loop):
futures = [loop.run_in_executor(EXECUTOR, async_request,loop)
for x in range(10)]
await asyncio.wait(futures)
loop = asyncio.get_event_loop()
loop.run_until_complete(main(loop))
Run Code Online (Sandbox Code Playgroud)
导致以下错误:
Traceback (most recent call last):
File "co_test.py", line 17, in <module>
loop.run_until_complete(main(loop))
File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
return future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "co_test.py", line 10, in main
futures …Run Code Online (Sandbox Code Playgroud) 我正在编写一些python代码来进行一些计算并将结果写入文件.这是我目前的代码:
for name, group in data.groupby('Date'):
df = lot_of_numpy_calculations(group)
with open('result.csv', 'a') as f:
df.to_csv(f, header=False, index=False)
Run Code Online (Sandbox Code Playgroud)
有时计算和写入都需要.我在python中阅读了一篇关于async的文章,但我不知道如何实现它.有没有一种简单的方法来优化这个循环,以便它不会等到写完成并开始下一次迭代?
我试图在多线程程序中将信号从子线程发送到主线程(不能使用多进程)。不幸的是,即使在用尽所有在线可用的阅读材料(我可以找到)之后,我也无法清楚地了解如何这样做。我是信号和python的初学者,所以请耐心等待并像对新手一样解释。我不能在进程中使用 join 方法,因为我希望两个线程同时运行。这是我在这里找到的与该主题相关的代码 - http://pymotw.com/2/signal/#signals-and-threads ,它对我来说并不真正有效。
import signal
import threading
import os
import time
def signal_handler(num, stack):
print 'Received signal %d in %s' % (num, threading.currentThread())
signal.signal(signal.SIGUSR1, signal_handler)
def wait_for_signal():
print 'Waiting for signal in', threading.currentThread()
signal.pause()
print 'Done waiting'
# Start a thread that will not receive the signal
receiver = threading.Thread(target=wait_for_signal, name='receiver')
receiver.start()
time.sleep(0.1)
def send_signal():
print 'Sending signal in', threading.currentThread()
os.kill(os.getpid(), signal.SIGUSR1)
sender = threading.Thread(target=send_signal, name='sender')
sender.start()
sender.join()
# Wait for the thread to see the signal …Run Code Online (Sandbox Code Playgroud) 我会尝试发布一个最小的工作示例,但不幸的是这个问题只需要很多部分,所以我尽可能地将其剥离.
首先,我使用一个简单的脚本来模拟通过函数调用按键.这是从这里调整的.
import ctypes
SendInput = ctypes.windll.user32.SendInput
PUL = ctypes.POINTER(ctypes.c_ulong)
class KeyBdInput(ctypes.Structure):
_fields_ = [("wVk", ctypes.c_ushort),
("wScan", ctypes.c_ushort),
("dwFlags", ctypes.c_ulong),
("time", ctypes.c_ulong),
("dwExtraInfo", PUL)]
class HardwareInput(ctypes.Structure):
_fields_ = [("uMsg", ctypes.c_ulong),
("wParamL", ctypes.c_short),
("wParamH", ctypes.c_ushort)]
class MouseInput(ctypes.Structure):
_fields_ = [("dx", ctypes.c_long),
("dy", ctypes.c_long),
("mouseData", ctypes.c_ulong),
("dwFlags", ctypes.c_ulong),
("time",ctypes.c_ulong),
("dwExtraInfo", PUL)]
class Input_I(ctypes.Union):
_fields_ = [("ki", KeyBdInput),
("mi", MouseInput),
("hi", HardwareInput)]
class Input(ctypes.Structure):
_fields_ = [("type", ctypes.c_ulong),
("ii", Input_I)]
def getKeyCode(unicodeKey):
k = unicodeKey
curKeyCode = 0
if k == …Run Code Online (Sandbox Code Playgroud) python windows keyboard multithreading python-multithreading
python 的多处理队列的文档:https : //docs.python.org/2/library/multiprocessing.html#multiprocessing.Queue
不像 queue.Queue 那样清楚:https ://docs.python.org/3/library/queue.html
当 maxsize 参数没有提供给构造函数时,关于队列的大小是否是“无限的”(例如,在程序可以设法分配内存的任何可能范围内)。
是这样吗?
python python-multithreading python-3.x python-multiprocessing python-3.6
背景:
我有一些复杂的强化学习算法,我想在多个线程中运行。
问题
尝试调用sess.run线程时,我收到以下错误消息:
RuntimeError: The Session graph is empty. Add operations to the graph before calling run().
重现错误的代码:
import tensorflow as tf
import threading
def thread_function(sess, i):
inn = [1.3, 4.5]
A = tf.placeholder(dtype=float, shape=(None), name="input")
P = tf.Print(A, [A])
Q = tf.add(A, P)
sess.run(Q, feed_dict={A: inn})
def main(sess):
thread_list = []
for i in range(0, 4):
t = threading.Thread(target=thread_function, args=(sess, i))
thread_list.append(t)
t.start()
for t in thread_list:
t.join()
if __name__ == '__main__':
sess = tf.Session()
main(sess) …Run Code Online (Sandbox Code Playgroud) python multithreading python-multithreading python-3.x tensorflow
我有以下问题:给定一个运行 fastapi 的后端,它有一个流端点,用于更新前端,我想在每次调用更新后端状态的函数时发送这些更新(可以是计划的作业或已命中并导致状态更新的不同端点)。
我想要实现的一个简单版本是:
from fastapi import FastAPI
from starlette.responses import StreamingResponse
class State:
def __init__(self):
self.messages = []
def update(self, new_messages):
self.messages = new_messages
# HERE: notify waiting stream endpoint
app = FastAPI()
state = State()
@app.get('/stream')
def stream():
def event_stream():
while True:
# HERE lies the question: wait for state to be update
for message in state.messages:
yield 'data: {}\n\n'.format(json.dumps(message))
return StreamingResponse(event_stream(), media_type="text/event-stream")
Run Code Online (Sandbox Code Playgroud)
我希望它永远运行下去。每次状态更新时,都会event_stream解除阻塞并发送消息。
我看过线程和异步,但我有一种感觉,我缺少一些关于如何在 python 中执行此操作的简单概念。
python python-multithreading server-sent-events python-asyncio fastapi
下面的代码通过将总完成时间返回几乎为零来按预期执行,因为它不会等待线程完成每个作业。
import concurrent.futures
import time
start = time.perf_counter()
def do_something(seconds):
print(f'Sleeping {seconds} second(s)...')
time.sleep(seconds)
return f'Done Sleeping...{seconds}'
executor= concurrent.futures.ThreadPoolExecutor()
secs = [10, 4, 3, 2, 1]
fs = [executor.submit(do_something, sec) for sec in secs]
finish = time.perf_counter()
print(f'Finished in {round(finish-start, 2)} second(s)')
Run Code Online (Sandbox Code Playgroud)
但是使用with命令它确实等待:
with concurrent.futures.ThreadPoolExecutor() as executor:
secs = [10, 4, 3, 2, 1]
fs = [executor.submit(do_something, sec) for sec in secs]
Run Code Online (Sandbox Code Playgroud)
为什么?with多线程具有这种行为的原因是什么?
我正在创建一个线程管理器类,该类将执行任务作为线程处理并将结果传递给下一个流程步骤。该流程在接收任务的第一次执行时正常工作,但第二次执行失败并出现以下错误:
...python3.8/concurrent/futures/thread.py", line 179, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
Run Code Online (Sandbox Code Playgroud)
任务来自Cmd.cmdloop用户输入 - 因此,脚本是持久的,不会关闭。相反,run将被多次调用,因为从用户那里收到输入。
我已经实现了一个ThreadPoolExecutor来处理工作负载并尝试按时间顺序收集结果,concurrent.futures.as_completed以便每个项目按照完成的顺序处理到下一步。
run下面的方法非常适合第一次执行,但在第二次执行同一任务(在第一次执行期间成功)时返回错误。
def run ( self, _executor=None, _futures={}, ) -> bool :
task = self.pipeline.get( )
with _executor or self.__default_executor as executor :
_futures = { executor.submit ( task.target.execute, ), }
for future in concurrent.futures.as_completed ( _futures, ) :
print( future.result ( ) )
return True …Run Code Online (Sandbox Code Playgroud) python-multithreading python-3.x threadpoolexecutor concurrent.futures python-3.8
python ×7
python-3.x ×5
async-await ×1
fastapi ×1
keyboard ×1
python-2.7 ×1
python-3.6 ×1
python-3.8 ×1
signals ×1
tensorflow ×1
windows ×1