在多核机器Linux OS中,当进程调度程序将一个进程迁移到另一个cpu时

Question

在我的程序中,其rss为65G,当通话时fork,sys_clone->dup_mm->copy_page_range将消耗超过2秒.在这种情况下,一个cpu在执行fork时会100%sys,同时,一个线程在fork完成之前无法获得cpu时间.机器有16个CPU,其他CPU空闲.

所以我的问题是一个cpu忙于fork,为什么调度程序不会将等待这个cpu的进程迁移到其他空闲cpu？一般来说,调度程序何时以及如何在cpus之间迁移进程？

我搜索这个网站,现有的线程无法回答我的问题.

• Linux调度程序如何在多核处理器上调度进程？
• 多核处理器可以同时运行多个进程吗？

Answer 1

rss为65G，调用fork时sys_clone->dup_mm->copy_page_range会消耗2秒以上

在执行fork（或clone）操作时，应将现有进程的 vmas 复制到新进程的 vmas 中。dup_mm函数（kernel/fork.c）创建新的mm并进行实际的复制。没有直接调用copy_page_range，但我认为静态函数dup_mmap可能被内联到dup_mm并且它调用了copy_page_range.

在new和 olddup_mmap中都有几把锁被锁定：mmoldmm

356         down_write(&oldmm->mmap_sem);

获取mmap_sem读取器/写入器信号量后，所有 mmap 上都会有一个循环来复制其元信息：

381         for (mpnt = oldmm->mmap; mpnt; mpnt = mpnt->vm_next) 

仅在循环（在您的情况下很长）之后mmap_sem才解锁：

465 out:
468         up_write(&oldmm->mmap_sem);

虽然 rwlockmmap_sep被作者关闭，但任何其他读者或作者都无法对oldmm.

一个线程在 fork 完成之前无法获得 cpu 时间所以我的问题是一个 cpu 在 fork 上忙，为什么调度程序不将在此 cpu 上等待的进程迁移到其他空闲 cpu？

您确定其他线程已准备好运行并且不想对 mmap 执行任何操作，例如：

• 映射新的东西或取消映射不需要的东西，
• 增大或缩小其堆 ( brk)，
• 增加它的堆栈，
• 页面错误
• 或许多其他活动...？

Actually, the wait-cpu thread is my IO thread, which send/receive package from client, in my observation, the package always exist, but the IO thread cannot receive it.

You should check stack of your wait-cpu thread (there is even SysRq for this), and kind of I/O. mmaping of file is the variant of I/O which will be blocked on mmap_sem by fork.

Also you can check the "last used CPU" of the wait-cpu thread, e.g. in the top monitoring utility, by enabling the thread view (H key) and adding "Last used CPU" column to output (fj in older; f scroll to P, enter in newer). I think it is possible that your wait-cpu thread already was on the other CPU, just not allowed (not ready) to run.

If you are using fork only to make exec, it can be useful to:

• either switch to vfork+exec (or just to posix_spawn). vfork will suspend your process (but may not suspend your other threads, it is dangerous) until new process will do exec or exit, but execing may be faster than waiting for 65 GB of mmaps to be copied.
• or not doing fork from the multithreaded process with several active threads and multi-GB virtual memory. You can create small (without multi-GB mmaped) helper process, communicate with it using ipc or sockets or pipes and ask it to fork and do everything you want.