Python, global state, multiprocessing and other ways to hang yourself

C and C++ were supposed to be the “dangerous” languages. There’s Stroustrup’s quote, for example, widely circulated on the internet. However, Stroustrup clarifies that his statement applies to all powerful languages. I find Python to be a powerful language that, by design, protects you from “simple” dangers, but lets you wander into more complex dangers without much warning.

Stroustrup’s statement in more detail is (from his website):

“C makes it easy to shoot yourself in the foot; C++ makes it harder, but when you do it blows your whole leg off”. Yes, I said something like that (in 1986 or so). What people tend to miss, is that what I said there about C++ is to a varying extent true for all powerful languages. As you protect people from simple dangers, they get themselves into new and less obvious problems.

The surprising thing about Python I’d like to briefly discuss here is how the multiprocessing module can silently fail when dealing with shared state.

What I effectively did was have a variable declared in the parent process, passed to the child process which then modified it. Python happily lets you do this, but changes to the variable are not seen by the parent process. As this answer to the question on stackoverflow explains:

When you use multiprocessing to open a second process, an entirely new instance of Python, with its own global state, is created. That global state is not shared, so changes made by child processes to global variables will be invisible to the parent process.

I was thrown though, by two additional layers of complexity. Firstly as the example code below shows, I shared the state in a somewhat hidden manner – I passed an instance method to the new process. I should have realized that this implicitly shares the original object – via self – with the new process.

Secondly, when I printed out the ids of the relevant object in the parent and child processes the ids came out to be the same. As the documents explain:

CPython implementation detail: This is the address of the object in memory.

The same id (= memory address) business threw me for a bit. Then I heard a faint voice in my head mumbling things about ‘virtual addressing’ and ‘copy-on-write’ and ‘os.fork()’. So what’s going on here? A little bit of perusing on stack overflow allows us to collect the appropriate details.

As mentioned above, Python, because of some implementation reasons (keyword: Global Interpreter Lock – GIL) uses os.fork() to achieve true multiprocessing via the multiprocessing module. fork() creates an exact copy of the old process and starts it in a new one. This means, that, for everything to be consistent, the original pointers in the program need to keep pointing to the same things, otherwise the new process will fall apart. But WAIT! This is chaos! Now we have two identical running processes writing and reading the same memory locations! Not quite.

Modern OSes use virtual addressing. Basically the address values (pointers) you see inside your program are not actual physical memory locations, but pointers to an index table (virtual addresses) that in turn contains pointers to the actual physical memory locations. Because of this indirection, you can have the same virtual address point to different physical addresses IF the virtual addresses belong to index tables of separate processes.

In our case, this explains the same id() value and the fact that when the child process modified the object with the same id value, it was actually accessing a different physical object, which explains why it’s parent doppelganger now diverges.

For completeness, we should mention copy-on-write. What this means is that the OS cleverly manages things such that initially the virtual addresses actually point to the original physical addresses – as if you copied the address table from the original process. This allows the two processes to share memory while reading (and saves a bunch of copying). Once either of the processes writes to a memory location, however, a bunch of copying is done and the relevant values now reside in a new memory location and one of the virtual tables are updated to reflect this.

Which brings us to the question: what the heck am I doing worrying about addresses in Python?! Shouldn’t this stuff just work? Isn’t that why I’m incurring the performance penalty, so I can have neater code and not worry about the low level details? Well, nothing’s perfect and keep in mind Stroustrup’s saw, I guess.

Also, never pass up a learning opportunity, much of which only comes through the school of hard knocks. It also gives a dopamine rush you’ll not believe.  I wonder what kind of shenanigans you can get into doing concurrent programming in Lisp, hmm …

So, what about the other ways to hang yourself, as promised in the title? Oh, that was just clickbait, sorry. This is all I got.

Code follows:

import Queue as Q
from multiprocessing import Process, Queue
import time

def describe_set(s):
  return 'id: {}, contents: {}'.format(id(s), s)

class SetManager(object):
  def __init__(self):
    self.my_set = set()
    print('Child process: {}'.format(describe_set(self.my_set)))

  def add_to_set(self, item):
    print('Adding {}'.format(item))
    self.my_set.add(item)
    print('Child process: {}'.format(describe_set(self.my_set)))

def test1():
  print('\n\nBasic test, no multiprocessing')

  sm = SetManager()
  print('Parent process: {}'.format(describe_set(sm.my_set)))

  sm.add_to_set(1)
  print('Parent process: {}'.format(describe_set(sm.my_set)))

class SetManager2(SetManager):
  def __init__(self, q, q_reply):
    super(SetManager2, self).__init__()
    # SetManager.__init__(self)
    self.keep_running = True

    self.q, self.q_reply = q, q_reply
    self.sp = Process(target=self.loop, args=())
    self.sp.start()

  def loop(self):
    while self.keep_running:
      try:
        msg = self.q.get(timeout=1)
        if msg == 'quit':
          self.keep_running = False
        elif msg == 'peek':
          self.q_reply.put(list(self.my_set))
        else:
          self.add_to_set(msg)
      except Q.Empty:
        pass
      time.sleep(.1)

def test2():
  print('\n\nMultiprocessing with method set off in new thread')

  q, q_reply = Queue(), Queue()

  sm = SetManager2(q, q_reply)
  print('Parent process: {}'.format(describe_set(sm.my_set)))

  sm.q.put(1)
  time.sleep(1)
  print('Parent process: {}'.format(describe_set(sm.my_set)))

  sm.q.put(2)
  time.sleep(1)
  print('Parent process: {}'.format(describe_set(sm.my_set)))

  q.put('peek')
  time.sleep(1)
  print('Reply from child process: {}'.format(describe_set(q_reply.get())))

  sm.q.put('quit')
  sm.sp.join()

class SetManager3(SetManager2):
  def __init__(self, q, q_reply):
    super(SetManager2, self).__init__()
    self.keep_running = True
    self.q = q
    self.q_reply = q_reply

def start_set_manager3_in_process(q, q_reply):
  sm = SetManager3(q, q_reply)
  sm.loop()

def test3():
  print('\n\nMultiprocessing with object created in new thread')

  q, q_reply = Queue(), Queue()

  sp = Process(target=start_set_manager3_in_process, args=(q, q_reply))
  sp.start()
  # print('Parent process: items are: {}'.format(sm.my_set))

  q.put(1)
  time.sleep(1)

  q.put(2)
  time.sleep(2)

  q.put('peek')
  time.sleep(1)
  print('Reply from child process: {}'.format(describe_set(q_reply.get())))

  q.put('quit')
  sp.join()

test1()
test2()
test3()
Advertisements

3 thoughts on “Python, global state, multiprocessing and other ways to hang yourself

  1. homes says:

    Thanks for this, the documentation for Python multiprocessing is tough to work through even with stackexchange, etc. One (potentially dumb) question that I’m a little stumped on: what does the child process actually do when it starts? It starts a new process, loads a new interpreter, does all the same imports, etc., but it also obviously never runs anything in the “if __name__==__main__:” block even though it still thinks its own current module’s name is __main__. I think I get how to use it practically but I want to be sure I’m not missing anything.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s