Python Odness

Posted:	2008-06-16 20:30
Tags:	Python

You probably already know that variables in Python are effectively passed by reference:

a = ['this', 'is', 'a', 'list']

def change(arg):
    arg.append('with')
    arg.append('new')
    arg.append('variables')
    print arg is a

change(a)
print ' '.join(a)

This program would result in the variable a being changed even though the appending of the extra words happens in the scope of the change() function. As you can see from the first print statement this is because the arg variable really is the same thing as the a variable, not a copy of it as would happen in C++.

The output from the program is:

True
this is a list with new variables

This behaviour is fairly understandable but I want to show you something you might not expect but which you really should be aware of. Consider this code, what would you expect to be printed the first time this function was called?

def test(a=[], b={}):
    print a, b
    a.append('a')
    b[len(a)] = 'b'

If you guessed [], {} then you would be correct. Try calling it again. What do you expect the second time?

The answer is might surprise you. It turns out to be [a], {1: 'b'} because even though you are calling test() with no arguments, the previous call changed the values of the empty [] and {} to different values. The second time the function was called the definition was effectively:

def test(a=['a'], b={1: 'b'}):
    print a, b
    a.append('a')
    b[len(a)] = 'b'

As you keep calling test() the list and dictionary keep getting bigger. Now you might be suprised by this and wonder what would happen if you pass ordinary variables into the function. Well it appends a to the first one and adds a new key to the second one but this time because the variables aren't the actual ones which make up the function definition the function definition isn't changed. If you call test() again the variables from the function definition will go on incrementing. Here's the full source:

def test(a=[], b={}):
    print a, b
    a.append('a')
    b[len(a)] = 'b'

test()
test()
test()
print
c = []
d = {}
test(c, d)
print 'Output: ',  c, d
test()

And the full output:

[] {}
['a'] {1: 'b'}
['a', 'a'] {1: 'b', 2: 'b'}

[] {}
Output:  ['a'] {1: 'b'}
['a', 'a', 'a'] {1: 'b', 2: 'b', 3: 'b'}

So, how do you avoid this problem?

1. Never use mutable variables as defaults to functions.

Instead you can do something like this:

def test(a=None, b=None):
    if a is None:
        a = []
    if b is None:
        b = {}

This is very safe and always works.

2. Make copies of the variables:

   def test(a=[], b={}):
       a = a[:]
       b = b.copy()
   

By creating local variables with the same names as the defaults you can no longer (easily) accidentally change the variables defined as defaults. The drawback is that you make copies of any arguments which are passed to the function which then uses up more memory.

3. Be very careful not to actually change the values of the defaults.

This isn't a particularly good idea because someone else looking at your code might not be aware of what you are doing and might change the value without realising.

I'd imagine quite a few libraries which allow mutable variables as default values might have this particular problem. In my own code I try to only use None as a default option. Do other people have any handy workarounds?

(view source)