The last program that was written using thread is rewritten using the multiprocessing module. There are something I found in this exercise
import time
import logging
import multiprocessing
from threading import Thread, Lock
from queue import Queue
dl_job = { 'a.jpg': 3,
'b.jpg': 4,
'c.jpg': 2,
'd.jpg': 6}
cov_job = { 'a.jpg' : 15,
'b.jpg' : 4,
'c.jpg' : 8,
'd.jpg' : 9}
FORMAT='[%(processName)s %(process)d %(threadName)s, %(asctime)s, %(levelname)s] %(message)s'
logging.basicConfig(format=FORMAT, level=logging.DEBUG)
class Dl_picture():
def __init__(self):
#self.dl_queue = Queue()
#self.cov_queue = Queue()
self.remain_pic = len(dl_job)
#self.remain_pic_lock = Lock()
#self.cov_end = 0
self.cov_end = multiprocessing.Value('i', 0)
return
def download(self, dl_queue, cov_queue, remain_pic_lock):
while not dl_queue.empty():
try:
pic = dl_queue.get(block=False)
logging.info('Downloading %s Time %s' % (pic, dl_job[pic]))
time.sleep(dl_job[pic])
dl_queue.task_done()
with remain_pic_lock:
self.remain_pic -= 1
cov_queue.put(pic)
if self.remain_pic == 0:
cov_queue.put(None)
except:
logging.info('Queue retreive error')
else:
logging.info("I can't get any task from queue!")
def convert(self, cov_queue):
while True:
if not self.cov_end.value:
pic = cov_queue.get()
if pic:
logging.info('Converting %s Time %s' % (pic, cov_job[pic]))
time.sleep(cov_job[pic])
cov_queue.task_done()
else:
logging.info("I don't have picture to convert!")
self.cov_end.value = 1
else:
logging.info("I don't need to wait for picture to convert, exit")
break
if __name__ == '__main__':
dl_queue = Queue()
cov_queue = multiprocessing.JoinableQueue()
remain_pic_lock = Lock()
t = Dl_picture()
for pic, tn in dl_job.items():
dl_queue.put(pic)
s = time.perf_counter()
thread_arr = []
num_thread = 5
for _ in range(num_thread):
td = Thread(target=t.download, args=(dl_queue, cov_queue, remain_pic_lock))
td.start()
thread_arr.append(td)
num_process = 4
p_arr = []
for _ in range(num_process):
p = multiprocessing.Process(target=t.convert, args=(cov_queue,))
p.start()
p_arr.append(p)
for i in range(num_process):
p_arr[i].join()
e = time.perf_counter()
logging.info('Total time used %s seconds' % (e - s))
The most noticeably adjustment is that since the Process class inside the multiprocessing module accepts parameter values which need to be picklable. And when you pass a class function into the target parameter, the original class __init__ definition has objects from thread and queue module which are not picklable. This leads to the changes of moving the queue and lock out of it. Then pass them into the creation of the thread variables as arguments.
The second thing to change is the use of the Queue class in the multiprocessing model for the convert function since the original queue model is not picklable.
Another thing that needs to change is the self.cov_end variable. This is the most crucial change for the program to run correctly. The important difference between a processes and threads is that processes don’t share memory while threads do. So in order for the variable to tell all the processes that there is no more image to convert and can exit. We will need a variable that is shared by all the processes. The line self.cov_end = multiprocessing.Value(‘i’, 0) is for that. Lastly I also change the logging format so processname and processid will be logged.
The output of the program is shown below and notice threads have the same processid while processes have different id.
[MainProcess 336864 Thread-1, 2019-07-13 16:59:08,380, INFO] Downloading a.jpg Time 3 [MainProcess 336864 Thread-2, 2019-07-13 16:59:08,386, INFO] Downloading b.jpg Time 4 [MainProcess 336864 Thread-3, 2019-07-13 16:59:08,387, INFO] Downloading c.jpg Time 2 [MainProcess 336864 Thread-4, 2019-07-13 16:59:08,388, INFO] Downloading d.jpg Time 6 [MainProcess 336864 Thread-5, 2019-07-13 16:59:08,389, INFO] I can't get any task from queue! [MainProcess 336864 Thread-3, 2019-07-13 16:59:10,390, INFO] I can't get any task from queue! [Process-2 338060 MainThread, 2019-07-13 16:59:10,395, INFO] Converting c.jpg Time 8 [MainProcess 336864 Thread-1, 2019-07-13 16:59:11,386, INFO] I can't get any task from queue! [Process-1 337120 MainThread, 2019-07-13 16:59:11,387, INFO] Converting a.jpg Time 15 [MainProcess 336864 Thread-2, 2019-07-13 16:59:12,387, INFO] I can't get any task from queue! [Process-3 338248 MainThread, 2019-07-13 16:59:12,389, INFO] Converting b.jpg Time 4 [MainProcess 336864 Thread-4, 2019-07-13 16:59:14,391, INFO] I can't get any task from queue! [Process-4 338388 MainThread, 2019-07-13 16:59:14,393, INFO] Converting d.jpg Time 9 [Process-3 338248 MainThread, 2019-07-13 16:59:16,392, INFO] I don't have picture to convert! [Process-3 338248 MainThread, 2019-07-13 16:59:16,393, INFO] I don't need to wait for picture to convert, exit [Process-2 338060 MainThread, 2019-07-13 16:59:18,397, INFO] I don't need to wait for picture to convert, exit [Process-4 338388 MainThread, 2019-07-13 16:59:23,394, INFO] I don't need to wait for picture to convert, exit [Process-1 337120 MainThread, 2019-07-13 16:59:26,390, INFO] I don't need to wait for picture to convert, exit [MainProcess 336864 MainThread, 2019-07-13 16:59:26,427, INFO] Total time used 18.048427699999998 seconds