In python, redis-py is used with multiprocessing modules, why is each process a different fd?… here is a solution to the problem.
In python, redis-py is used with multiprocessing modules, why is each process a different fd?
In python, using redis-py with multiprocessing modules, why is each process a different fd?
Test code:
# xiaorui.cc
import time
import multiprocessing
import redis
r = redis. Redis(host='127.0.0.1', port=6379, db=0)
def func(msg):
for i in xrange(30):
time.sleep(1)
print r.keys()
return "done " + msg
if __name__ == "__main__":
pool = multiprocessing. Pool(processes=4)
result = []
for i in xrange(4):
msg = "hello %d" %(i)
result.append(pool.apply_async(func, (msg, )))
pool.close()
pool.join()
for res in result:
print res.get()
print "Sub-process(es) done."
Test results:
[[email protected] ~]$ ps aux f|grep a.py
508 11704 5.0 0.0 421096 11664 pts/11 Sl+ 17:50 0:00 | \_ python a.py
508 11709 0.0 0.0 193760 7464 pts/11 S+ 17:50 0:00 | \_ python a.py
508 11710 0.0 0.0 193760 7468 pts/11 S+ 17:50 0:00 | \_ python a.py
508 11711 0.0 0.0 193760 7468 pts/11 S+ 17:50 0:00 | \_ python a.py
508 11712 0.0 0.0 193760 7476 pts/11 S+ 17:50 0:00 | \_ python a.py
508 11720 0.0 0.0 103248 832 pts/12 S+ 17:50 0:00 \_ grep a.py
[[email protected] ~]$ sudo lsof -p 11709|grep 6379
python 11709 ruifengyun 4u IPv4 4173927407 0t0 TCP localhost:51433->localhost:6379 (ESTABLISHED)
[[email protected] ~]$ sudo lsof -p 11710|grep 6379
python 11710 ruifengyun 4u IPv4 4173927417 0t0 TCP localhost:51435->localhost:6379 (ESTABLISHED)
[[email protected] ~]$ sudo lsof -p 11711|grep 6379
python 11711 ruifengyun 4u IPv4 4173927411 0t0 TCP localhost:51434->localhost:6379 (ESTABLISHED)
[[email protected] ~]$ sudo lsof -p 11712|grep 6379
python 11712 ruifengyun 4u IPv4 4173927416 0t0 TCP localhost:51436->localhost:6379 (ESTABLISHED)
Why is the Redis Conn FD different for each process? In my opinion, different Redis FDs will only occur if Redis Connect is created in lazy mode.
Also, all child processes are shared R objects (Redis Conect FD).
Solution
Each process needs to connect to Redis separately. And a single outbound port will only be used by a single process. So each process has its own port.