The Digital Cat - decoratorshttps://www.thedigitalcatonline.com/2022-09-18T23:00:00+00:00Adventures of a curious cat in the land of programmingFirst-class objects in Python - Higher-order functions, wrappers, and factories2021-03-09T16:00:00+00:002022-09-18T23:00:00+00:00Leonardo Giordanitag:www.thedigitalcatonline.com,2021-03-09:/blog/2021/03/09/first-class-objects-in-python/<p>My new book "First-class objects in Python" is out! Grab your <strong>FREE</strong> copy <a href="https://www.thedigitalcat.academy/freebie-first-class-objects">here</a>!</p><div class="imageblock"><img src="/images/first-class-objects-in-python.jpg"></div>Python Mocks: a gentle introduction - Part 22016-09-27T09:00:00+00:002016-09-27T09:00:00+00:00Leonardo Giordanitag:www.thedigitalcatonline.com,2016-09-27:/blog/2016/09/27/python-mocks-a-gentle-introduction-part-2/<p>Second part of the series on Python mocks: how to patch code</p><p>In the <a href="https://www.thedigitalcatonline.com/blog/2016/03/06/python-mocks-a-gentle-introduction-part-1/">first post</a> I introduced you to Python mocks, objects that can imitate other objects and work as placeholders, replacing external systems during unit testing. I described the basic behaviour of mock objects, the <code>return_value</code> and <code>side_effect</code> attributes, and the <code>assert_called_with()</code> method.</p>
<p>In this post I will briefly review the remaining <code>assert_*</code> methods and some interesting attributes that allow to check the calls received by the mock object. Then I will introduce and exemplify patching, which is a very important topic in testing.</p>
<h2 id="other-assertions-and-attributes">Other assertions and attributes<a class="headerlink" href="#other-assertions-and-attributes" title="Permanent link">¶</a></h2>
<p>The <a href="https://docs.python.org/dev/library/unittest.mock.html">official documentation</a> of the mock library lists many other assertion, namely <code>assert_called_once_with()</code>, <code>assert_any_call()</code>, <code>assert_has_calls()</code>, <code>assert_not_called()</code>. If you grasped how <code>assert_called_with()</code> works, you will have no troubles in understanding how those other behave. Be sure to check the documentation to get a full description of what mock object can assert about their history after being used by your code.</p>
<p>Together with those methods, mock objects also provide some useful attributes, two of which have been already reviewed in the first post. The remaining attributes are as expected mostly related to calls, and are <code>called</code>, <code>call_count</code>, <code>call_args</code>, <code>call_args_list</code>, <code>method_calls</code>, <code>mock_calls</code>. While these also are very well described in the official documentation, I want to point out the two <code>method_calls</code> and <code>mock_calls</code> attributes, that store the detailed list of methods which are called on the mock, and the <code>call_args_list</code> attribute that lists the parameters of every call.</p>
<p>Do not forget that methods called on a mock object are mocks themselves, so you may first access the main mock object to get information about the called methods, and then access those methods to get the arguments they received.</p>
<h2 id="patching">Patching<a class="headerlink" href="#patching" title="Permanent link">¶</a></h2>
<p>Mocks are very simple to introduce in your tests whenever your objects accept classes or instances from outside. In that case, as described, you just have to instantiate the <code>Mock</code> class and pass the resulting object to your system. However, when the external classes instantiated by your library are hardcoded this simple trick does not work. In this case you have no chance to pass a mock object instead of the real one.</p>
<p>This is exactly the case addressed by patching. Patching, in a testing framework, means to replace a globally reachable object with a mock, thus achieving the target of having the code run unmodified, while part of it has been hot swapped, that is, replaced at run time.</p>
<h2 id="a-warm-up-example">A warm-up example<a class="headerlink" href="#a-warm-up-example" title="Permanent link">¶</a></h2>
<p>Let us start with a very simple example. Patching can be complex to grasp at the beginning so it is better to learn it with trivial code. If you do not have it yet, create the testing environment <code>mockplayground</code> with the instruction given in the previous post.</p>
<p>I want to develop a simple class that returns information about a given file. The class shall be instantiated with the file name, which can be a relative path.</p>
<p>For the sake of brevity I will not show you every step of the TDD development of the class. Remember that TDD requires you to write a test and then implement the code, but sometimes this could be too fine grained, so do not use the TDD rules without thinking.</p>
<p>The tests for the initialization of the class are</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">fileinfo</span> <span class="kn">import</span> <span class="n">FileInfo</span>
<span class="k">def</span> <span class="nf">test_init</span><span class="p">():</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s1">'somefile.ext'</span>
<span class="n">fi</span> <span class="o">=</span> <span class="n">FileInfo</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">fi</span><span class="o">.</span><span class="n">filename</span> <span class="o">==</span> <span class="n">filename</span>
<span class="k">def</span> <span class="nf">test_init</span><span class="p">():</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s1">'somefile.ext'</span>
<span class="n">relative_path</span> <span class="o">=</span> <span class="s1">'../</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="n">fi</span> <span class="o">=</span> <span class="n">FileInfo</span><span class="p">(</span><span class="n">relative_path</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">fi</span><span class="o">.</span><span class="n">filename</span> <span class="o">==</span> <span class="n">filename</span>
</code></pre></div>
<p>You can put them into the <code>tests/test_fileinfo.py</code> file. The code that makes the tests pass could be something like</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">os</span>
<span class="k">class</span> <span class="nc">FileInfo</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">original_path</span> <span class="o">=</span> <span class="n">path</span>
<span class="bp">self</span><span class="o">.</span><span class="n">filename</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">basename</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
</code></pre></div>
<p>Up to now I didn't introduce any new feature. Now I want the <code>get_info()</code> function to return a tuple with the file name, the original path the class was instantiated with, and the absolute path of the file.</p>
<p>You immediately realise that you have an issue in writing the test. There is no way to easily test something as "the absolute path", since the outcome of the function called in the test is supposed to vary with the path of the test itself. Let us write part of the test</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">test_get_info</span><span class="p">():</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s1">'somefile.ext'</span>
<span class="n">original_path</span> <span class="o">=</span> <span class="s1">'../</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="n">fi</span> <span class="o">=</span> <span class="n">FileInfo</span><span class="p">(</span><span class="n">original_path</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">fi</span><span class="o">.</span><span class="n">get_info</span><span class="p">()</span> <span class="o">==</span> <span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">original_path</span><span class="p">,</span> <span class="s1">'???'</span><span class="p">)</span>
</code></pre></div>
<p>where the '???' string highlights that I cannot put something sensible to test the absolute path of the file.</p>
<p>Patching is the way to solve this problem. You know that the function will use some code to get the absolute path of the file. So in the scope of the test only you can replace that code with different code and perform the test. Since the replacement code has a known outcome writing the test is now possible.</p>
<p>Patching, thus, means to inform Python that in some scope you want a globally accessible module/object replaced by a mock. Let's see how we can use it in our example</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">unittest.mock</span> <span class="kn">import</span> <span class="n">patch</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">test_get_info</span><span class="p">():</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s1">'somefile.ext'</span>
<span class="n">original_path</span> <span class="o">=</span> <span class="s1">'../</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="k">with</span> <span class="n">patch</span><span class="p">(</span><span class="s1">'os.path.abspath'</span><span class="p">)</span> <span class="k">as</span> <span class="n">abspath_mock</span><span class="p">:</span>
<span class="n">test_abspath</span> <span class="o">=</span> <span class="s1">'some/abs/path'</span>
<span class="n">abspath_mock</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">test_abspath</span>
<span class="n">fi</span> <span class="o">=</span> <span class="n">FileInfo</span><span class="p">(</span><span class="n">original_path</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">fi</span><span class="o">.</span><span class="n">get_info</span><span class="p">()</span> <span class="o">==</span> <span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">original_path</span><span class="p">,</span> <span class="n">test_abspath</span><span class="p">)</span>
</code></pre></div>
<p>Remember that if you are using Python 2 you installed the <code>mock</code> module with <code>pip</code>, so your import statement becomes <code>from mock import patch</code>.</p>
<p>You clearly see the context in which the patching happens, as it is enclosed in a <code>with</code> statement. Inside this statement the module <code>os.path.abspath</code> will be replaced by a mock created by the function <code>patch</code> and called <code>abspath_mock</code>. We can now give the function a <code>return_value</code> as we did with standard mocks in the first post and run the test.</p>
<p>The code that make the test pass is</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">FileInfo</span><span class="p">:</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">get_info</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">filename</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">original_path</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">abspath</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">filename</span><span class="p">)</span>
</code></pre></div>
<p>Obviously to write the test you have to know that you are going to use the <code>os.path.abspath</code> function, so patching is somehow a "less pure" practice in TDD. In pure OOP/TDD you are only concerned with the external behaviour of the object, and not with its internal structure. This example, however, shows that you have to cope with some real world issues, and patching is a clean way to do it.</p>
<h2 id="the-patching-decorator">The patching decorator<a class="headerlink" href="#the-patching-decorator" title="Permanent link">¶</a></h2>
<p>The <code>patch</code> function we imported from the <code>unittest.mock</code> module is very powerful, and can be used as a function decorator as well. When used in this fashion you need to change the decorated function to accept a mock as last argument.</p>
<div class="highlight"><pre><span></span><code><span class="nd">@patch</span><span class="p">(</span><span class="s1">'os.path.abspath'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_get_info</span><span class="p">(</span><span class="n">abspath_mock</span><span class="p">):</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s1">'somefile.ext'</span>
<span class="n">original_path</span> <span class="o">=</span> <span class="s1">'../</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="n">test_abspath</span> <span class="o">=</span> <span class="s1">'some/abs/path'</span>
<span class="n">abspath_mock</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">test_abspath</span>
<span class="n">fi</span> <span class="o">=</span> <span class="n">FileInfo</span><span class="p">(</span><span class="n">original_path</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">fi</span><span class="o">.</span><span class="n">get_info</span><span class="p">()</span> <span class="o">==</span> <span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">original_path</span><span class="p">,</span> <span class="n">test_abspath</span><span class="p">)</span>
</code></pre></div>
<p>As you can see the <code>patch</code> decorator works like a big <code>with</code> statement for the whole function. Obviously in this way you replace the target function <code>os.path.abspath</code> in the scope of the whole function. It is then up to you to decide if you need to use <code>patch</code> as a decorator or in a <code>with</code> block.</p>
<h2 id="multiple-patches">Multiple patches<a class="headerlink" href="#multiple-patches" title="Permanent link">¶</a></h2>
<p>We can also patch more that one object. Say for example that we want to change the above test to check that the outcome of the <code>FileInfo.get_info()</code> method also contains the size of the file. To get the size of a file in Python we may use the <code>os.path.getsize()</code> function, which returns the size of the file in bytes.</p>
<p>So now we have to patch <code>os.path.getsize</code> as well, and this can be done with another <code>patch</code> decorator.</p>
<div class="highlight"><pre><span></span><code><span class="nd">@patch</span><span class="p">(</span><span class="s1">'os.path.getsize'</span><span class="p">)</span>
<span class="nd">@patch</span><span class="p">(</span><span class="s1">'os.path.abspath'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_get_info</span><span class="p">(</span><span class="n">abspath_mock</span><span class="p">,</span> <span class="n">getsize_mock</span><span class="p">):</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s1">'somefile.ext'</span>
<span class="n">original_path</span> <span class="o">=</span> <span class="s1">'../</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="n">test_abspath</span> <span class="o">=</span> <span class="s1">'some/abs/path'</span>
<span class="n">abspath_mock</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">test_abspath</span>
<span class="n">test_size</span> <span class="o">=</span> <span class="mi">1234</span>
<span class="n">getsize_mock</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">test_size</span>
<span class="n">fi</span> <span class="o">=</span> <span class="n">FileInfo</span><span class="p">(</span><span class="n">original_path</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">fi</span><span class="o">.</span><span class="n">get_info</span><span class="p">()</span> <span class="o">==</span> <span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">original_path</span><span class="p">,</span> <span class="n">test_abspath</span><span class="p">,</span> <span class="n">test_size</span><span class="p">)</span>
</code></pre></div>
<p>Please notice that the decorator which is nearest to the function is applied first. Always remember that the decorator syntax with <code>@</code> is a shortcut to replace the function with the output of the decorator, so two decorators result in</p>
<div class="highlight"><pre><span></span><code><span class="nd">@decorator1</span>
<span class="nd">@decorator2</span>
<span class="k">def</span> <span class="nf">myfunction</span><span class="p">():</span>
<span class="k">pass</span>
</code></pre></div>
<p>which is a shorcut for</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">myfunction</span><span class="p">():</span>
<span class="k">pass</span>
<span class="n">myfunction</span> <span class="o">=</span> <span class="n">decorator1</span><span class="p">(</span><span class="n">decorator2</span><span class="p">(</span><span class="n">myfunction</span><span class="p">))</span>
</code></pre></div>
<p>This explains why, in the test code, the function receives first <code>abspath_mock</code> and then <code>getsize_mock</code>. The first decorator applied to the function is the patch of <code>os.path.abspath</code>, which appends the mock that we call <code>abspath_mock</code>. Then the patch of <code>os.path.getsize</code> is applied and this appends its own mock.</p>
<p>The code that makes the test pass is</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">FileInfo</span><span class="p">:</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">get_info</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">filename</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">original_path</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">abspath</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">filename</span><span class="p">),</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">getsize</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">filename</span><span class="p">)</span>
</code></pre></div>
<p>We can write the above test using two <code>with</code> statements as well</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">test_get_info</span><span class="p">():</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s1">'somefile.ext'</span>
<span class="n">original_path</span> <span class="o">=</span> <span class="s1">'../</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="k">with</span> <span class="n">patch</span><span class="p">(</span><span class="s1">'os.path.abspath'</span><span class="p">)</span> <span class="k">as</span> <span class="n">abspath_mock</span><span class="p">:</span>
<span class="n">test_abspath</span> <span class="o">=</span> <span class="s1">'some/abs/path'</span>
<span class="n">abspath_mock</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">test_abspath</span>
<span class="k">with</span> <span class="n">patch</span><span class="p">(</span><span class="s1">'os.path.getsize'</span><span class="p">)</span> <span class="k">as</span> <span class="n">getsize_mock</span><span class="p">:</span>
<span class="n">test_size</span> <span class="o">=</span> <span class="mi">1234</span>
<span class="n">getsize_mock</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">test_size</span>
<span class="n">fi</span> <span class="o">=</span> <span class="n">FileInfo</span><span class="p">(</span><span class="n">original_path</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">fi</span><span class="o">.</span><span class="n">get_info</span><span class="p">()</span> <span class="o">==</span> <span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">original_path</span><span class="p">,</span> <span class="n">test_abspath</span><span class="p">,</span> <span class="n">test_size</span><span class="p">)</span>
</code></pre></div>
<p>Using more than one <code>with</code> statement, however, makes the code difficult to read, in my opinion, so in general I prefer to avoid complex <code>with</code> trees if I do not need a limited scope of the patching.</p>
<h2 id="patching-immutable-objects">Patching immutable objects<a class="headerlink" href="#patching-immutable-objects" title="Permanent link">¶</a></h2>
<p>The most widespread version of Python is CPython, which is written, as the name suggests, in C. Part of the standard library is also written in C, while the rest is written in Python itself.</p>
<p>The objects (classes, modules, functions, etc) that are implemented in C are shared between interpreters, which is something that you can do embedding the Python interpreter in a C program, for example. This requires those objects to be immutable, so that you cannot alter them at runtime from a single interpreter.</p>
<p>For an example of this immutability just check the following code</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">a</span> <span class="o">=</span> <span class="mi">1</span>
<span class="gp">>>> </span><span class="n">a</span><span class="o">.</span><span class="n">conjugate</span> <span class="o">=</span> <span class="mi">5</span>
<span class="gt">Traceback (most recent call last):</span>
File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>, in <span class="n"><module></span>
<span class="gr">AttributeError</span>: <span class="n">'int' object attribute 'conjugate' is read-only</span>
</code></pre></div>
<p>Here I'm trying to replace a method with an integer, which is pointless, but nevertheless shows the issue we are facing.</p>
<p>What has this immutability to do with patching? What <code>patch</code> does is actually to temporarily replace an attibute of an object (method of a class, class of a module, etc), so if that object is immutable the patching action fails.</p>
<p>A typical example of this problem is the <code>datetime</code> module, which is also one of the best candidates for patching, since the output of time functions is by definition time-varying.</p>
<p>Let me show the problem with a simple class that logs operations. The class is the following (you can put it into a file called <code>logger.py</code>)</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">datetime</span>
<span class="k">class</span> <span class="nc">Logger</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">messages</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">def</span> <span class="nf">log</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">messages</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">(),</span> <span class="n">message</span><span class="p">))</span>
</code></pre></div>
<p>This is pretty simple, but testing this code is problematic, because the <code>log()</code> method produces results that depend on the actual execution time.</p>
<p>If we try to write a test patching <code>datetime.datetime.now</code> we have a bitter surprise. This is the test code, that you can put in <code>tests/test_logger.py</code></p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">unittest.mock</span> <span class="kn">import</span> <span class="n">patch</span>
<span class="kn">from</span> <span class="nn">logger</span> <span class="kn">import</span> <span class="n">Logger</span>
<span class="k">def</span> <span class="nf">test_init</span><span class="p">():</span>
<span class="n">l</span> <span class="o">=</span> <span class="n">Logger</span><span class="p">()</span>
<span class="k">assert</span> <span class="n">l</span><span class="o">.</span><span class="n">messages</span> <span class="o">==</span> <span class="p">[]</span>
<span class="nd">@patch</span><span class="p">(</span><span class="s1">'datetime.datetime.now'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_log</span><span class="p">(</span><span class="n">mock_now</span><span class="p">):</span>
<span class="n">test_now</span> <span class="o">=</span> <span class="mi">123</span>
<span class="n">test_message</span> <span class="o">=</span> <span class="s2">"A test message"</span>
<span class="n">mock_now</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">test_now</span>
<span class="n">l</span> <span class="o">=</span> <span class="n">Logger</span><span class="p">()</span>
<span class="n">l</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="n">test_message</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">l</span><span class="o">.</span><span class="n">messages</span> <span class="o">==</span> <span class="p">[(</span><span class="n">test_now</span><span class="p">,</span> <span class="n">test_message</span><span class="p">)]</span>
</code></pre></div>
<p>and the execution of pytest returns a <code>TypeError: can't set attributes of built-in/extension type 'datetime.datetime'</code>, which is exactly a problem of immutability.</p>
<p>There are several ways to address this problem, but all of them leverage the fact that, when you import of subclass an immutable object what you get is a "copy" of that is now mutable.</p>
<p>The easiest example in this case is the module <code>datetime</code> itself. In the <code>test_log</code> function we try to patch directly the <code>datetime.datetime.now</code> object, affecting the builtin module <code>datetime</code>. The file <code>logger.py</code>, however, does import <code>datetime</code>, so that the latter becomes a local symbol in the <code>logger</code> module. This is exactly the key for our patching. Let us change the code to</p>
<div class="highlight"><pre><span></span><code><span class="nd">@patch</span><span class="p">(</span><span class="s1">'logger.datetime.datetime'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_log</span><span class="p">(</span><span class="n">mock_datetime</span><span class="p">):</span>
<span class="n">test_now</span> <span class="o">=</span> <span class="mi">123</span>
<span class="n">test_message</span> <span class="o">=</span> <span class="s2">"A test message"</span>
<span class="n">mock_datetime</span><span class="o">.</span><span class="n">now</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">test_now</span>
<span class="n">l</span> <span class="o">=</span> <span class="n">Logger</span><span class="p">()</span>
<span class="n">l</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="n">test_message</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">l</span><span class="o">.</span><span class="n">messages</span> <span class="o">==</span> <span class="p">[(</span><span class="n">test_now</span><span class="p">,</span> <span class="n">test_message</span><span class="p">)]</span>
</code></pre></div>
<p>As you see running the test now the patching works. What we did was to patch <code>logger.datetime.datetime</code> instead of <code>datetime.datetime.now</code>. Two things changed, thus, in our test. First, we are patching the module imported in the <code>logger.py</code> file and not the module provided globally by the Python interpreter. Second, we have to patch the whole module because this is what is imported by the <code>logger.py</code> file. If you try to patch <code>logger.datetime.datetime.now</code> you will find that it is still immutable.</p>
<p>Another possible solution to this problem is to create a function that invokes the immutable object and returns its value. This last function can be easily patched, because it just uses the builtin objects and thus is not immutable. This solution, however, requires to change the source code to allow testing, which is far from being desirable. Obviously it is better to introduce a small change in the code and have it tested than to leave it untested, but whenever is possible I avoid solutions that introduce code which wouldn't be required without tests.</p>
<h2 id="final-words">Final words<a class="headerlink" href="#final-words" title="Permanent link">¶</a></h2>
<p>In this second part of this small series on Python testing we reviewed the patching mechanism and run through some of its subtleties. Patching is a really effective technique, and patch-based tests can be found in many different packages. Take your time to become confident with mocks and patching, since they will be one of your main tools while working with Python and any other object-oriented language.</p>
<p>As always, I strongly recommend finding some time to read the <a href="https://docs.python.org/dev/library/unittest.mock.html">official documentation</a> of the mock library.</p>
<h2 id="feedback">Feedback<a class="headerlink" href="#feedback" title="Permanent link">¶</a></h2>
<p>The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections</p>Abstract Base Classes in Python2016-04-03T11:00:00+01:002016-04-03T11:00:00+01:00Leonardo Giordanitag:www.thedigitalcatonline.com,2016-04-03:/blog/2016/04/03/abstract-base-classes-in-python/<p>What are Abstract Base Classes in Python and why are they useful?</p><p>With the introduction of Abstract Base Classes, Python once again shows its nature of a very innovative and flexible language. It is interesting to see how such a remarkable feature has been introduced into the language by a pure Python module. This demonstrates that Python is built in a way that is very open to changes, thanks to its foundations in pure polymorphism based on delegation.</p>
<p>Many Python programmers overlooked Abstract Base Classes and the classes in the <code>collections</code> module, which are one of the simplest and useful applications of the concept. Sure enough, this is not a feature that you will use every day or that will change the way you are programming in Python. But neither is it something you shall discard before understanding what it brings into the language, and what sort of problems it can solve for you.</p>
<h2 id="eafp">EAFP<a class="headerlink" href="#eafp" title="Permanent link">¶</a></h2>
<p>Python is a dynamically-typed object-oriented language strongly based on delegation, so its approach to problems is intrinsically polymorphic. This means that Python deals mostly with the behaviour of objects and not with their structure. The well-known EAFP protocol (it's Easier to Ask Forgiveness than Permission) comes from this approach. This code</p>
<div class="highlight"><pre><span></span><code><span class="k">try</span><span class="p">:</span>
<span class="n">someobj</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="k">except</span> <span class="ne">TypeError</span><span class="p">:</span>
<span class="c1"># object is not subscriptable</span>
<span class="o">...</span>
</code></pre></div>
<p>does not check if the object is a <code>list</code> or a <code>dictionary</code> (both would allow the <code>[1]</code> notation), but if the object can be accessed by key (or index). When you accept a parameter in a function Python does not specify the type (leaving aside type hints) because you are not interested in accepting a given type or one of its derived types. You are interested in accepting something that provides the methods you will use.</p>
<p>The behaviour, in an object-oriented environment, is the run-time interface of the object. This is different from the static interface, which is the collection of the methods provided by the object. The run-time interface is the actual interface the object shows when it is used, and this encompasses the methods provided by its class, but also methods provided by parent classes, the metaclass and other entry points provided by <code>__getattr__</code>.</p>
<h2 id="complex-checks">Complex checks<a class="headerlink" href="#complex-checks" title="Permanent link">¶</a></h2>
<p>Sometimes, however, you need to perform complex checks, such as "it behaves like a list". How can you test this condition? You could test if the incoming object has some standard methods, but this is not only incomplete but also wrong. For example, I could write the following test</p>
<div class="highlight"><pre><span></span><code><span class="k">try</span><span class="p">:</span>
<span class="n">obj</span><span class="o">.</span><span class="n">append</span>
<span class="n">obj</span><span class="o">.</span><span class="n">count</span>
<span class="n">obj</span><span class="o">.</span><span class="n">extend</span>
<span class="n">obj</span><span class="o">.</span><span class="n">index</span>
<span class="n">obj</span><span class="o">.</span><span class="n">insert</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="k">except</span> <span class="ne">AttributeError</span><span class="p">:</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
</code></pre></div>
<p>which tries to cover the methods a list-like object shall provide. This test, however, accepts objects that do not really behave like a list such as</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">FakeList</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">fakemethod</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">pass</span>
<span class="k">def</span> <span class="fm">__getattr__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="k">if</span> <span class="n">name</span> <span class="ow">in</span> <span class="p">[</span><span class="s1">'append'</span><span class="p">,</span> <span class="s1">'count'</span><span class="p">,</span> <span class="s1">'extend'</span><span class="p">,</span> <span class="s1">'index'</span><span class="p">,</span> <span class="s1">'insert'</span><span class="p">,</span> <span class="o">...</span><span class="p">]:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">fakemethod</span>
</code></pre></div>
<p>It is unlikely that you will write such a class, but this shows you one of the potential pitfalls of the previous test, which is wrong because it tries to rely on the structure instead of testing the behaviour. The temptation to rely on <code>isinstance()</code> is big</p>
<div class="highlight"><pre><span></span><code><span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">someobj</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span>
<span class="o">...</span>
</code></pre></div>
<p>If possible this approach is worse than before, since it tests the exact type. Even if <code>isinstance()</code> and <code>issubclass()</code> are smart enough to walk through the hierarchy of parent classes, this excludes evary class that behaves like a list does not inherit from it.</p>
<h2 id="back-to-delegation">Back to delegation<a class="headerlink" href="#back-to-delegation" title="Permanent link">¶</a></h2>
<p>The idea proposed in <a href="http://legacy.python.org/dev/peps/pep-3119/">PEP 3119</a> to face this problem is very elegant, and leverages the very nature of Python: that of being strongly based on delegation. The solution, implemented in Python 3 and backported to Python 2.7, changes the nature of the two <code>isinstance()</code> and <code>issubclass()</code> builtins. Now the first thing that <code>isinstance()</code> does is to call the <code>__instancecheck__()</code> method of the queried class, basically giving it the chance to answer the call with a different algorithm than the standard one. The same happens for <code>issubclass()</code>, which becomes <code>__subclasscheck__()</code>. So the following code</p>
<div class="highlight"><pre><span></span><code><span class="nb">issubclass</span><span class="p">(</span><span class="n">myclass</span><span class="p">,</span> <span class="n">someclass</span><span class="p">)</span>
</code></pre></div>
<p>does no more perform a pure external check of the relationship between <code>someclass</code> and <code>myclass</code>. The first thing that <code>issubclass()</code> does now is the following</p>
<div class="highlight"><pre><span></span><code><span class="n">someclass</span><span class="o">.</span><span class="fm">__subclasscheck__</span><span class="p">(</span><span class="n">myclass</span><span class="p">)</span>
</code></pre></div>
<p>This is very natural because, after all, <code>someclass</code> is the best source of judgement about being a subclass of itself.</p>
<h2 id="a-new-type-of-subclass">A new type of subclass<a class="headerlink" href="#a-new-type-of-subclass" title="Permanent link">¶</a></h2>
<p>With the introduction of delegation-based instance and subclass checks, Python provides a new type of subclass, and thus a new way to relate classes together. Now a subclass may be a real subclass, obtained using inheritance</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">ChildClass</span><span class="p">(</span><span class="n">ParentClass</span><span class="p">):</span>
<span class="k">pass</span>
</code></pre></div>
<p>or can be a <em>virtual</em> subclass, obtained through registration</p>
<div class="highlight"><pre><span></span><code><span class="n">ParentClass</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">ChildClass</span><span class="p">)</span>
</code></pre></div>
<p>The difference between a real and a virtual subclass is very simple: a real subclass knows its relationship with the parent class through its <code>__bases__</code> attribute, and can thus implicitly delegate the resolution of missing methods. A virtual subclass knows nothing about the class that registered it, and nowhere in the subclass will you find something that links it to the parent class. Thus, a virtual parent class is useful only as a categorization.</p>
<h2 id="abstract-base-classes">Abstract Base Classes<a class="headerlink" href="#abstract-base-classes" title="Permanent link">¶</a></h2>
<p>Classes that can register other classes, thus becoming virtual parents of those, are called in Python Abstract Base Classes, or ABCs.</p>
<p>The name of this new language element is important. ABCs are first of all classes, just like any other class you can create in Python, and they can be subclassed in the usual way to create taxonomies. They are also meant to be base classes, that is classes that represent fundamental behaviours or categories. Last, they are abstract. This has a very precise meaning in Python and will be the subject of the last part of this post.</p>
<p>The classes provided by the <code>collections</code> module are Abstract Base Classes, and they set themselves as virtual parents of some base types in the same module. If you check the <code>_collections_abc.py</code> file in your Python 3 installation (for example in <code>/usr/lib/python3.4/_collections_abc.py</code>) you will find code like this</p>
<div class="highlight"><pre><span></span><code><span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="n">Sequence</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="nb">tuple</span><span class="p">)</span>
<span class="n">Sequence</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="nb">str</span><span class="p">)</span>
<span class="n">Sequence</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="nb">range</span><span class="p">)</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="n">MutableSequence</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="nb">list</span><span class="p">)</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
</code></pre></div>
<p>Where the <code>Sequence</code> and the <code>MutableSequence</code> ABCs register some built-in types of Python.</p>
<p>It is very important to understand that registering a class does not imply any form of check about methods or attributes. Registering is just the <em>promise</em> that a given behaviour is provided by the registered class.</p>
<p>To demonstrate this let me provide you a very simple example made using one of the <code>collections</code> classes</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">collections</span>
<span class="gp">>>> </span><span class="k">class</span> <span class="nc">MyClass</span><span class="p">():</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">...</span>
<span class="gp">>>> </span><span class="nb">issubclass</span><span class="p">(</span><span class="n">MyClass</span><span class="p">,</span> <span class="n">collections</span><span class="o">.</span><span class="n">Sequence</span><span class="p">)</span>
<span class="go">False</span>
<span class="gp">>>> </span><span class="n">collections</span><span class="o">.</span><span class="n">Sequence</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">MyClass</span><span class="p">)</span>
<span class="go"><class '__main__.MyClass'></span>
<span class="gp">>>> </span><span class="nb">issubclass</span><span class="p">(</span><span class="n">MyClass</span><span class="p">,</span> <span class="n">collections</span><span class="o">.</span><span class="n">Sequence</span><span class="p">)</span>
<span class="go">True</span>
<span class="gp">>>></span>
</code></pre></div>
<p>As you can see, the <code>MyClass</code> class is initially not recognized as a subclass of <code>collections.Sequence</code>, but after the registration <code>issubclass()</code> returns <code>True</code>, even if the class is still empty.</p>
<h2 id="how-to-create-abcs">How to create ABCs<a class="headerlink" href="#how-to-create-abcs" title="Permanent link">¶</a></h2>
<p>The example given by the official documentation is very simple and clear</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">abc</span> <span class="kn">import</span> <span class="n">ABCMeta</span>
<span class="k">class</span> <span class="nc">MyABC</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">ABCMeta</span><span class="p">):</span>
<span class="k">pass</span>
<span class="n">MyABC</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="nb">tuple</span><span class="p">)</span>
<span class="k">assert</span> <span class="nb">issubclass</span><span class="p">(</span><span class="nb">tuple</span><span class="p">,</span> <span class="n">MyABC</span><span class="p">)</span>
<span class="k">assert</span> <span class="nb">isinstance</span><span class="p">((),</span> <span class="n">MyABC</span><span class="p">)</span>
</code></pre></div>
<p>All you need to do is to create a class and use the <code>ABCMeta</code> metaclass provided by the <code>abc</code> module and you will obtain a class that has the <code>register()</code> method and a suitable implementation of <code>__subclasscheck__()</code> and <code>__instancecheck__()</code>. Checking again the <code>_collections_abc.py</code> file you can see that this is exactly the way the <code>collections</code> classes are implemented</p>
<div class="highlight"><pre><span></span><code><span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="k">class</span> <span class="nc">Hashable</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">ABCMeta</span><span class="p">):</span>
<span class="o">...</span>
<span class="k">class</span> <span class="nc">Iterable</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">ABCMeta</span><span class="p">):</span>
<span class="o">...</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
</code></pre></div>
<h2 id="are-you-scared-of-metaclasses">Are you scared of metaclasses?<a class="headerlink" href="#are-you-scared-of-metaclasses" title="Permanent link">¶</a></h2>
<p>Metaclasses are a strange topic in Python. Most of the times the advice given to the novice is "Don't use them", like they were an error of the language and something that shall be avoided.</p>
<p>I don't think so. As a matter of facts I definitely disagree with such position, for many reasons.</p>
<p>First of all, if you are programming in Python its better for you to understand everything Python provides you, both the good and the bad parts. Programming languages are tools, and you shall know their strengths and their limitations. Most of the times what we call "limitations" are just features that become a restraint just because we are not aware of them. The C language, for example, is not object-oriented. Is this a strength or a limitation? Python provides you a very powerful inspection mechanism. Is this a strength or a limitations? I could give countless other examples.</p>
<p>Second, powerful features are the one you should know better. After all, we use a language for the unique features it provides, not for the features it shares with other languages. I use Python because of its powerful polymorphism implementation, not because of loops and inheritance. Those are provided by Java and C++, too, for example. I write a device driver in C because of the closeness to the machine language and its speed, not because of the <code>int</code> and <code>float</code> types, which are provided by many other languages. So, since powerful features are what let the language do what others cannot, those are the ones you have to master.</p>
<p>Third, if a feature of a language is a design error, and it can be, you need to understand why it is an error, and how you can avoid using it. Before ES6, JavaScript had some issues with scopes, given by the behaviour of the <code>var</code> keyword. You cannot afford being ignorant about those limitations, otherwise your software will be buggy. So, while studying the <code>for</code> loop in JavaScript took me a couple of minutes (it is a C-like for loop, after all), I spent a lot of time in dealing with <code>var</code>, which is the dangerous button of the whole device.</p>
<p>Back to Python. Metaclasses are not a last-minute feature put into the language just for fun. They are the foundation of Python itself, and the relationship between <code>object</code> and <code>type</code> is something so beautiful that it is a pity that basically no one talks about it. So, plase stop complaining against metaclasses and telling people that they are dangerous or complex.</p>
<p>Metaclasses are part of the language. And they are not complex to understand.</p>
<h2 id="why-metaclasses-for-abcs">Why metaclasses for ABCs?<a class="headerlink" href="#why-metaclasses-for-abcs" title="Permanent link">¶</a></h2>
<p>If you program in Python you should be somehow familiar with classes and instances. You know that when you build an instance you use a class (like a blueprint) and that the class can put things into the instance. For example</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Class definition</span>
<span class="k">class</span> <span class="nc">Child</span><span class="p">():</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">answer</span> <span class="o">=</span> <span class="mi">42</span>
<span class="c1"># Link instance and class</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">Child</span><span class="p">()</span>
<span class="c1"># Use the instance</span>
<span class="k">assert</span> <span class="n">c</span><span class="o">.</span><span class="n">answer</span> <span class="o">==</span> <span class="mi">42</span>
</code></pre></div>
<p>Now, when you build a class you use a metaclass (like a blueprint) and the metaclass can put things into the class.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Metaclass definition</span>
<span class="k">class</span> <span class="nc">NewType</span><span class="p">(</span><span class="nb">type</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">bases</span><span class="p">,</span> <span class="n">namespace</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">answer</span> <span class="o">=</span> <span class="mi">42</span>
<span class="c1"># Link class and metaclass</span>
<span class="k">class</span> <span class="nc">Child</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">NewType</span><span class="p">):</span> <span class="k">pass</span>
<span class="c1"># Use the class</span>
<span class="k">assert</span> <span class="n">Child</span><span class="o">.</span><span class="n">answer</span> <span class="o">==</span> <span class="mi">42</span>
</code></pre></div>
<p>Sounds complex? Not at all, in my opinion. If you check the two examples you will see that they are exactly the same thing, the first referring to the instance-class relationship, the second to the class-metaclass one.</p>
<p>This is all you need to understand metaclasses. When you build a class you need to put things into it, for example you need to put the <code>__getattribute__</code> or the <code>__new__()</code> methods. This is done by the metaclass, which is usually <code>type</code> for every class. Indeed, if you check the <code>__class__</code> attribute into a class you get exactly this</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="nb">int</span>
<span class="go"><class 'int'></span>
<span class="gp">>>> </span><span class="nb">int</span><span class="o">.</span><span class="vm">__class__</span>
<span class="go"><class 'type'></span>
<span class="gp">>>></span>
</code></pre></div>
<h2 id="metaclasses-and-mro">Metaclasses and MRO<a class="headerlink" href="#metaclasses-and-mro" title="Permanent link">¶</a></h2>
<p>A slightly advanced annotation: when I say that the metaclass <em>puts</em> the methods into the class I'm simplifying the whole thing. As a matter of fact, like a class provides methods to the instance at runtime through the <code>__class__</code> attribute and the MRO protocol, the metaclass provides methods for the class. Attributes, instead, are put inside the class by the <code>__new__</code> or <code>__init__</code> methods of the metaclass.</p>
<p>Let us review the MRO mechanism for instances and classes first. When you call a method on an instance Python automatically looks for that method in the instance first, then in the parent class and in every class in its hierarchy.</p>
<p>So in this example</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">GrandParent</span><span class="p">():</span> <span class="k">pass</span>
<span class="k">class</span> <span class="nc">Parent</span><span class="p">(</span><span class="n">GrandParent</span><span class="p">):</span> <span class="k">pass</span>
<span class="k">class</span> <span class="nc">Child</span><span class="p">(</span><span class="n">Parent</span><span class="p">):</span> <span class="k">pass</span>
</code></pre></div>
<p>calling the method <code>get_name()</code> on an instance of <code>Child</code> will look for it first into the <code>Child</code> class, then into <code>Parent</code> and <code>GrandParent</code>, in this order. Finally, it will check <code>object</code>.</p>
<p>What happens to the MRO when a class of this hierarchy defines a different metaclass? For example</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">NewType</span><span class="p">(</span><span class="nb">type</span><span class="p">):</span> <span class="k">pass</span>
<span class="k">class</span> <span class="nc">GrandParent</span><span class="p">():</span> <span class="k">pass</span>
<span class="k">class</span> <span class="nc">Parent</span><span class="p">(</span><span class="n">GrandParent</span><span class="p">):</span> <span class="k">pass</span>
<span class="k">class</span> <span class="nc">Child</span><span class="p">(</span><span class="n">Parent</span><span class="p">,</span> <span class="n">metaclass</span><span class="o">=</span><span class="n">NewType</span><span class="p">):</span> <span class="k">pass</span>
</code></pre></div>
<p>In this case everything works as usual, but after checking <code>object</code> the MRO will also check the <code>NewType</code> metaclass (and its ancestors).</p>
<p>So, metaclasses can act as mixins, and they are queried only at the end of the usual MRO. This is exactly what happens using multiple inheritance if <code>NewType</code> were a standard parent class that does not have <code>Parent</code> or <code>GrandParent</code> as ancestors.</p>
<p>Metaclasses are not part of the MRO however, since the MRO just deals with standard inheritance. If you check the MRO of the <code>Child</code> class, you will see that the metaclass is not included</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">Child</span><span class="o">.</span><span class="n">mro</span><span class="p">()</span>
<span class="go">[<class '__main__.Child'>, <class '__main__.Parent'>, <class '__main__.GrandParent'>, <class 'object'>]</span>
<span class="gp">>>> </span><span class="n">Child</span><span class="o">.</span><span class="vm">__class__</span>
<span class="go"><class '__main__.NewType'></span>
<span class="gp">>>> </span>
</code></pre></div>
<h2 id="abstract-methods">Abstract methods<a class="headerlink" href="#abstract-methods" title="Permanent link">¶</a></h2>
<p>Why are ABC called <code>abstract</code>? ABCs can be instantiated, so they are after all not pure interfaces (like Java ones, for example)</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">abc</span>
<span class="gp">>>> </span><span class="k">class</span> <span class="nc">MyABC</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">abc</span><span class="o">.</span><span class="n">ABCMeta</span><span class="p">):</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="n">m</span> <span class="o">=</span> <span class="n">MyABC</span><span class="p">()</span>
<span class="gp">>>> </span>
</code></pre></div>
<p>They may however define some methods as <em>abstract</em>, using the <code>abc.abstractmethod</code> decorator, which prevents the class from being instantiated if the method is not implemented. Let me give you a simple example: I define an Abstract Base Class with and abstract method</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">MyABC</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">abc</span><span class="o">.</span><span class="n">ABCMeta</span><span class="p">):</span>
<span class="nd">@abc</span><span class="o">.</span><span class="n">abstractmethod</span>
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">pass</span>
</code></pre></div>
<p>and try to instantiate it. Python complains</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">m</span> <span class="o">=</span> <span class="n">MyABC</span><span class="p">()</span>
<span class="gt">Traceback (most recent call last):</span>
File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>, in <span class="n"><module></span>
<span class="gr">TypeError</span>: <span class="n">Can't instantiate abstract class MyABC with abstract methods get</span>
</code></pre></div>
<p>I am forced to create a new class that inherits from <code>MyABC</code> and implements the method</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">Concrete</span><span class="p">(</span><span class="n">MyABC</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="mi">1</span>
</code></pre></div>
<p>Now I can instantiate the class.</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">c</span> <span class="o">=</span> <span class="n">Concrete</span><span class="p">()</span>
<span class="gp">>>> </span>
</code></pre></div>
<p>Check the official documentation of Abstract Base Classes for a complete description of <code>@abstractmethod</code> and <code>@abstractproperty</code>.</p>
<h2 id="what-about-the-behaviour">What about the behaviour?<a class="headerlink" href="#what-about-the-behaviour" title="Permanent link">¶</a></h2>
<p>So, what happened to the Python tenet "Check the behaviour and not the structure"? With collections, after all, we dropped the EAFP protocol, going back to a Look Before You Leap approach. Are we going against the philosophy at the very base of the language?</p>
<p>It is very interesting to see what Guido van Rossum, creator of the Python language, says about this in <a href="http://legacy.python.org/dev/peps/pep-3119/">PEP 3119</a>: <em>Invocation means interacting with an object by invoking its methods. Usually this is combined with polymorphism, so that invoking a given method may run different code depending on the type of an object. Inspection means the ability for external code (outside of the object's methods) to examine the type or properties of that object, and make decisions on how to treat that object based on that information. [...] In classical OOP theory, invocation is the preferred usage pattern, and inspection is actively discouraged, being considered a relic of an earlier, procedural programming style. However, in practice this view is simply too dogmatic and inflexible, and leads to a kind of design rigidity that is very much at odds with the dynamic nature of a language like Python.</em></p>
<p>So the point is that forcing the use of a pure polymorphic approach sometimes can lead to solutions that are too complex or even incorrect. The key words here, in my opinion, are "dogmatic", "inflexible", and "rigidity", opposed to "dynamic nature". I really like this flexibility in a language and in its author.</p>
<p>Writing <code>if isinstance(obj, collections.Sequence)</code> is not EAFP, neither is any conditional test you may write. Nevertheless, no one would replace conditional tests with a pure EAFP approach, simply because sometimes those tests are more readable. This is the exact purpose of collections in Python and ABCs in general: to allow parts of the code to be simpler.</p>
<h2 id="final-words">Final words<a class="headerlink" href="#final-words" title="Permanent link">¶</a></h2>
<p>I hope this post helped you understand that Abstract Base Classes, and in particular the standard collections, are useful and easy to understand. Metaclasses are also not that scary and dangerous, even if using them obviously requires some skill.</p>
<h2 id="sources">Sources<a class="headerlink" href="#sources" title="Permanent link">¶</a></h2>
<p>The official documentation of the <code>abc</code> module is very well written. <a href="https://docs.python.org/3.5/library/abc.html">Here</a> you find the version for Python 3.5. I also suggest to read the original <a href="https://www.python.org/dev/peps/pep-3119">PEP 3119</a> and the related <a href="https://www.python.org/dev/peps/pep-3141">PEP 3141</a> for a deeper understanding of the topic.</p>
<h2 id="feedback">Feedback<a class="headerlink" href="#feedback" title="Permanent link">¶</a></h2>
<p>The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections.</p>Python Mocks: a gentle introduction - Part 12016-03-06T18:00:00+01:002019-02-27T23:30:00+00:00Leonardo Giordanitag:www.thedigitalcatonline.com,2016-03-06:/blog/2016/03/06/python-mocks-a-gentle-introduction-part-1/<p>An introduction to mocks in Python: basic concepts and how to use them for tests</p><p>As already stressed in the two introductory posts on TDD (you can find them <a href="/categories/tdd/">here</a>) testing requires to write some code that uses the functions and objects you are going to develop. This means that you need to isolate a given (external) function that is part of your public API and demonstrate that it works with standard inputs and in edge cases.</p>
<p>For example, if you are going to develop an object that stores percentages (such as for example poll results), you should test the following conditions: the class can store a standard percentage such as 42%, the class shall give an error if you try to store a negative percentage, the class shall give an error if you store a percentage greater than 100%.</p>
<p>Tests shall be <em>idempotent</em> and <em>isolated</em>. Idempotent in mathematics and computer science identifies a process that can be run multiple times without changing the status of the system. Isolated means that a test shall not change its behaviour depending on previous executions of itself, nor depend on the previous execution (or missing execution) of other tests.</p>
<p>Such restrictions, which guarantee that your tests are not passing due to a temporary configuration of the system or the order in which they are run, can raise big issues when dealing with external libraries and systems, or with intrinsically mutable concepts such as time. In the testing discipline, such issues are mostly faced using mocks, that is objects that pretend to be other objects.</p>
<p>In this series of posts I am going to review the Python <code>mock</code> library and exemplify its use. I will not cover everything you may do with mock, obviously, but hopefully I'll give you the information you need to start using this powerful library.</p>
<h2 id="installation">Installation<a class="headerlink" href="#installation" title="Permanent link">¶</a></h2>
<p>First of all, <code>mock</code> is a Python library which development started around 2008. It was selected to be included in the standard library as of Python 3.3, which however does not prevent you to use other libraries if you prefer.</p>
<p>Python 3 users, thus, are not required to take any step, while for Python 2 projects you are still required to issue a <code>pip install mock</code> to install it into the system or the current virtualenv.</p>
<p>You may find the official documentation <a href="https://docs.python.org/dev/library/unittest.mock.html">here</a>. It is very detailed, and as always I strongly recommend taking your time to run through it.</p>
<h2 id="basic-concepts">Basic concepts<a class="headerlink" href="#basic-concepts" title="Permanent link">¶</a></h2>
<p>A mock, in the testing lingo, is an object that simulates the behaviour of another (more complex) object. When you (unit)test an object of your library you need sometimes to access other systems your object want to connect to, but you do not really want to be forced to run them, for several reasons.</p>
<p>The first one is that connecting with external systems means having a complex testing environment, that is you are dropping the isolation requirement of you tests. If your object wants to connect with a website, for example, you are forced to have a running Internet connection, and if the remote website is down you cannot test your library.</p>
<p>The second reason is that the setup of an external system is usually slow in comparison with the speed of unit tests. We expect to run hundred of tests in seconds, and if we have to fetch information from a remote server for each of them the time easily increases by several orders of magnitude. Remember: having slow tests means that you cannot run them while you develop, which in turn means that you will not really use them for TDD.</p>
<p>The third reason is more subtle, and has to to with the mutable nature of an external system, thus I'll postpone the discussion of this issue for the moment.</p>
<p>Let us try and work with a mock in Python and see what it can do. First of all fire up a Python shell or a <a href="http://jupyter.org">Jupyter Notebook</a> and import the library </p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">unittest</span> <span class="kn">import</span> <span class="n">mock</span>
</code></pre></div>
<p>If you are using Python 2 you have to install it and use</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">mock</span>
</code></pre></div>
<p>The main object that the library provides is <code>Mock</code> and you can instantiate it without any argument</p>
<div class="highlight"><pre><span></span><code><span class="n">m</span> <span class="o">=</span> <span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">()</span>
</code></pre></div>
<p>This object has the peculiar property of creating methods and attributes on the fly when you require them. Let us first look inside the object to take a glance of what it provides</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="nb">dir</span><span class="p">(</span><span class="n">m</span><span class="p">)</span>
<span class="go">['assert_any_call', 'assert_called_once_with', 'assert_called_with', 'assert_has_calls', 'attach_mock', 'call_args', 'call_args_list', 'call_count', 'called', 'configure_mock', 'method_calls', 'mock_add_spec', 'mock_calls', 'reset_mock', 'return_value', 'side_effect']</span>
</code></pre></div>
<p>As you can see there are some methods which are already defined into the <code>Mock</code> object. Let us read a non-existent attribute</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span>
<span class="go"><Mock name='mock.some_attribute' id='140222043808432'></span>
<span class="gp">>>> </span><span class="nb">dir</span><span class="p">(</span><span class="n">m</span><span class="p">)</span>
<span class="go">['assert_any_call', 'assert_called_once_with', 'assert_called_with', 'assert_has_calls', 'attach_mock', 'call_args', 'call_args_list', 'call_count', 'called', 'configure_mock', 'method_calls', 'mock_add_spec', 'mock_calls', 'reset_mock', 'return_value', 'side_effect', 'some_attribute']</span>
</code></pre></div>
<p>Well, as you can see this class is somehow different from what you are accustomed to. First of all its instances do not raise an <code>AttributeError</code> when asked for a non-existent attribute, but they happily return another instance of <code>Mock</code> itself. Second, the attribute you tried to access has now been created inside the object and accessing it returns the same mock object as before.</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span>
<span class="go"><Mock name='mock.some_attribute' id='140222043808432'></span>
</code></pre></div>
<p>Mock objects are callables, which means that they may act both as attributes and as methods. If you try to call the mock it just returns you another mock with a name that includes parentheses to signal its callable nature</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="p">()</span>
<span class="go"><Mock name='mock.some_attribute()' id='140247621475856'></span>
</code></pre></div>
<p>As you can understand, such objects are the perfect tool to mimic other objects or systems, since they may expose any API without raising exceptions. To use them in tests, however, we need them to behave just like the original, which implies returning sensible values or performing operations.</p>
<h2 id="return-value">Return value<a class="headerlink" href="#return-value" title="Permanent link">¶</a></h2>
<p>The simplest thing a mock can do for you is to return a given value every time you call it. This is configured setting the <code>return_value</code> attribute of a mock object</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some</span> <span class="n">attribute</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="mi">42</span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some</span> <span class="n">attribute</span><span class="p">()</span>
<span class="go">42</span>
</code></pre></div>
<p>Now the object does not return a mock object any more, instead it just returns the static value stored in the <code>return_value</code> attribute. Obviously you can also store a callable such as a function or an object, and the method will return it, but it will not run it. Let me give you an example</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="k">def</span> <span class="nf">print_answer</span><span class="p">():</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="s2">"42"</span><span class="p">)</span>
<span class="gp">... </span>
<span class="gp">>>> </span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">print_answer</span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="p">()</span>
<span class="go"><function print_answer at 0x7f8df1e3f400></span>
</code></pre></div>
<p>As you can see calling <code>some_attribute()</code> just returns the value stored in <code>return_value</code>, that is the function itself. To return values that come from a function we have to use a slightly more complex attribute of mock objects called <code>side_effect</code>.</p>
<h2 id="side-effect">Side effect<a class="headerlink" href="#side-effect" title="Permanent link">¶</a></h2>
<p>The <code>side_effect</code> parameter of mock objects is a very powerful tool. It accepts three different flavours of objects, callables, iterables, and exceptions, and changes its behaviour accordingly.</p>
<p>If you pass an exception the mock will raise it</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="o">.</span><span class="n">side_effect</span> <span class="o">=</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'A custom value error'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="p">()</span>
<span class="gt">Traceback (most recent call last):</span>
File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>, in <span class="n"><module></span>
File <span class="nb">"/usr/lib/python3.4/unittest/mock.py"</span>, line <span class="m">902</span>, in <span class="n">__call__</span>
<span class="w"> </span><span class="k">return</span> <span class="n">_mock_self</span><span class="o">.</span><span class="n">_mock_call</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
File <span class="nb">"/usr/lib/python3.4/unittest/mock.py"</span>, line <span class="m">958</span>, in <span class="n">_mock_call</span>
<span class="w"> </span><span class="k">raise</span> <span class="n">effect</span>
<span class="gr">ValueError</span>: <span class="n">A custom value error</span>
</code></pre></div>
<p>If you pass an iterable, such as for example a generator, or a plain list, tuple, or similar objects, the mock will yield the values of that iterable, i.e. return every value contained in the iterable on subsequent calls of the mock. Let me give you an example</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="o">.</span><span class="n">side_effect</span> <span class="o">=</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="p">()</span>
<span class="go">0</span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="p">()</span>
<span class="go">1</span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="p">()</span>
<span class="go">2</span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="p">()</span>
<span class="gt">Traceback (most recent call last):</span>
File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>, in <span class="n"><module></span>
File <span class="nb">"/usr/lib/python3.4/unittest/mock.py"</span>, line <span class="m">902</span>, in <span class="n">__call__</span>
<span class="w"> </span><span class="k">return</span> <span class="n">_mock_self</span><span class="o">.</span><span class="n">_mock_call</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
File <span class="nb">"/usr/lib/python3.4/unittest/mock.py"</span>, line <span class="m">961</span>, in <span class="n">_mock_call</span>
<span class="w"> </span><span class="n">result</span> <span class="o">=</span> <span class="nb">next</span><span class="p">(</span><span class="n">effect</span><span class="p">)</span>
<span class="gr">StopIteration</span>
</code></pre></div>
<p>As promised, the mock just returns every object found in the iterable (in this case a <code>range</code> object) once at a time until the generator is exhausted. According to the iterator protocol (see <a href="https://www.thedigitalcatonline.com/blog/2013/03/25/python-generators-from-iterators-to-cooperative-multitasking/">this post</a>) once every item has been returned the object raises the <code>StopIteration</code> exception, which means that you can correctly use it in a loop.</p>
<p>The last and perhaps most used case is that of passing a callable to <code>side_effect</code>, which shamelessly executes it with its own same parameters. This is very powerful, especially if you stop thinking about "functions" and start considering "callables". Indeed, <code>side_effect</code> also accepts a class and calls it, that is it can instantiate objects. Let us consider a simple example with a function without arguments</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="k">def</span> <span class="nf">print_answer</span><span class="p">():</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="s2">"42"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="o">.</span><span class="n">side_effect</span> <span class="o">=</span> <span class="n">print_answer</span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="p">()</span>
<span class="go">42</span>
</code></pre></div>
<p>A slightly more complex example: a function with arguments</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="k">def</span> <span class="nf">print_number</span><span class="p">(</span><span class="n">num</span><span class="p">):</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="s2">"Number:"</span><span class="p">,</span> <span class="n">num</span><span class="p">)</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="o">.</span><span class="n">side_effect</span> <span class="o">=</span> <span class="n">print_number</span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="o">.</span><span class="n">side_effect</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
<span class="go">Number: 5</span>
</code></pre></div>
<p>And finally an example with a class</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="k">class</span> <span class="nc">Number</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="gp">... </span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="gp">... </span> <span class="bp">self</span><span class="o">.</span><span class="n">_value</span> <span class="o">=</span> <span class="n">value</span>
<span class="gp">... </span> <span class="k">def</span> <span class="nf">print_value</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="s2">"Value:"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_value</span><span class="p">)</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="o">.</span><span class="n">side_effect</span> <span class="o">=</span> <span class="n">Number</span>
<span class="gp">>>> </span><span class="n">n</span> <span class="o">=</span> <span class="n">m</span><span class="o">.</span><span class="n">some_attribute</span><span class="o">.</span><span class="n">side_effect</span><span class="p">(</span><span class="mi">26</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">n</span>
<span class="go"><__main__.Number object at 0x7f8df1aa4470></span>
<span class="gp">>>> </span><span class="n">n</span><span class="o">.</span><span class="n">print_value</span><span class="p">()</span>
<span class="go">Value: 26</span>
</code></pre></div>
<h2 id="testing-with-mocks">Testing with mocks<a class="headerlink" href="#testing-with-mocks" title="Permanent link">¶</a></h2>
<p>Now we know how to build a mock and how to give it a static return value or make it call a callable object. It is time to see how to use a mock in a test and what facilities do mocks provide. I'm going to use <a href="http://pytest.org">pytest</a> as a testing framework. You can find a quick introduction to pytest and TDD <a href="/categories/tdd/">here</a>).</p>
<h3 id="setup">Setup<a class="headerlink" href="#setup" title="Permanent link">¶</a></h3>
<p>If you want to quickly setup a pytest playground you may execute this code in a terminal (you need to have Python 3 and virtualenv installed in your system) </p>
<div class="highlight"><pre><span></span><code>mkdir<span class="w"> </span>mockplayground
<span class="nb">cd</span><span class="w"> </span>mockplayground
virtualenv<span class="w"> </span>venv3<span class="w"> </span>-p<span class="w"> </span>python3
<span class="nb">source</span><span class="w"> </span>venv3/bin/activate
pip<span class="w"> </span>install<span class="w"> </span>--upgrade<span class="w"> </span>pip
pip<span class="w"> </span>install<span class="w"> </span>pytest
<span class="nb">echo</span><span class="w"> </span><span class="s2">"[pytest]"</span><span class="w"> </span>>><span class="w"> </span>pytest.ini
<span class="nb">echo</span><span class="w"> </span><span class="s2">"norecursedirs=venv*"</span><span class="w"> </span>>><span class="w"> </span>pytest.ini
mkdir<span class="w"> </span>tests
touch<span class="w"> </span>myobj.py
touch<span class="w"> </span>tests/test_mock.py
<span class="nv">PYTHONPATH</span><span class="o">=</span><span class="s2">"."</span><span class="w"> </span>py.test
</code></pre></div>
<p>The <code>PYTHONPATH</code> environment variable is an easy way to avoid having to setup a whole Python project to just test some simple code.</p>
<h3 id="the-three-test-types">The three test types<a class="headerlink" href="#the-three-test-types" title="Permanent link">¶</a></h3>
<p>According to Sandy Metz we need to test only three types of messages (calls) between objects:</p>
<ul>
<li>Incoming queries (assertion on result)</li>
<li>Incoming commands (assertion on direct public side effects)</li>
<li>Outgoing commands (expectation on call and arguments)</li>
</ul>
<p>You can see the original talk <a href="https://www.youtube.com/watch?v=URSWYvyc42M">here</a> or read the slides <a href="https://speakerdeck.com/skmetz/magic-tricks-of-testing-railsconf">here</a>. The final table is shown in slide number 176.</p>
<p>As you can see when dealing with external objects we are only interested in knowing IF a method was called and WHAT PARAMETERS the caller passed to the object. We are not testing if the remote object returns the correct result, this is faked by the mock, which indeed returns exactly the result we need.</p>
<p>So the purpose of the methods provided by mock objects is to allow us to check what methods we called on the mock itself and what parameters we used in the call.</p>
<h3 id="asserting-calls">Asserting calls<a class="headerlink" href="#asserting-calls" title="Permanent link">¶</a></h3>
<p>To show how to use Python mocks in testing I will follow the TDD methodology, writing tests first and then writing the code that makes the tests pass. In this post I want to give you a simple overview of the mock objects, so I will not implement a real world use case, and the code will be very trivial. In the second part of this series I will test and implement a real class, in order to show some more interesting use cases.</p>
<p>The first thing we are usually interested in when dealing with an external object is to know that a given method has been called on it. Python mocks provide the <code>assert_called_with()</code> method to check this condition.</p>
<p>The use case we are going to test is the following. <em>We instantiate the <code>myobj.MyObj</code> class, which requires an external object. The class shall call the <code>connect()</code> method of the external object without any parameter.</em></p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">unittest</span> <span class="kn">import</span> <span class="n">mock</span>
<span class="kn">import</span> <span class="nn">myobj</span>
<span class="k">def</span> <span class="nf">test_instantiation</span><span class="p">():</span>
<span class="n">external_obj</span> <span class="o">=</span> <span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">()</span>
<span class="n">myobj</span><span class="o">.</span><span class="n">MyObj</span><span class="p">(</span><span class="n">external_obj</span><span class="p">)</span>
<span class="n">external_obj</span><span class="o">.</span><span class="n">connect</span><span class="o">.</span><span class="n">assert_called_with</span><span class="p">()</span>
</code></pre></div>
<p>The <code>myobj.MyObj</code> class, in this simple example, needs to connect to an external object, for example a remote repository or a database. The only thing we need to know for testing purposes is if the class called the <code>connect()</code> method of the external object without any parameter.</p>
<p>So the first thing we do in this test is to instantiate the mock object. This is a fake version of the external object, and its only purpose is to accept calls from the <code>MyObj</code> object under test and return sensible values. Then we instantiate the <code>MyObj</code> class passing the external object. We expect the class to call the <code>connect()</code> method so we express this expectation calling <code>external_obj.connect.assert_called_with()</code>.</p>
<p>What happens behind the scenes? The <code>MyObj</code> class receives the external object and somewhere is its initialization process calls the <code>connect()</code> method of the mock object and this creates the method itself as a mock object. This new mock records the parameters used to call it and the subsequent call to <code>assert_called_with()</code> checks that the method was called and that no parameters were passed.</p>
<p>Running pytest the test obviously fails.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nv">PYTHONPATH</span><span class="o">=</span><span class="s2">"."</span><span class="w"> </span>py.test
<span class="o">==========================================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">==========================================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.4.3+,<span class="w"> </span>pytest-2.9.0,<span class="w"> </span>py-1.4.31,<span class="w"> </span>pluggy-0.3.1
rootdir:<span class="w"> </span>/home/leo/devel/mockplayground,<span class="w"> </span>inifile:<span class="w"> </span>pytest.ini
collected<span class="w"> </span><span class="m">1</span><span class="w"> </span>items<span class="w"> </span>
tests/test_mock.py<span class="w"> </span><span class="nv">F</span>
<span class="o">===============================================</span><span class="w"> </span><span class="nv">FAILURES</span><span class="w"> </span><span class="o">================================================</span>
___________________________________________<span class="w"> </span>test_instantiation<span class="w"> </span>__________________________________________
<span class="w"> </span>def<span class="w"> </span>test_instantiation<span class="o">()</span>:
<span class="w"> </span><span class="nv">external_obj</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>mock.Mock<span class="o">()</span>
><span class="w"> </span>myobj.MyObj<span class="o">(</span>external_obj<span class="o">)</span>
E<span class="w"> </span>AttributeError:<span class="w"> </span><span class="s1">'module'</span><span class="w"> </span>object<span class="w"> </span>has<span class="w"> </span>no<span class="w"> </span>attribute<span class="w"> </span><span class="s1">'MyObj'</span>
tests/test_mock.py:6:<span class="w"> </span><span class="nv">AttributeError</span>
<span class="o">=======================================</span><span class="w"> </span><span class="m">1</span><span class="w"> </span>failed<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.03<span class="w"> </span><span class="nv">seconds</span><span class="w"> </span><span class="o">========================================</span>
$
</code></pre></div>
<p>Putting this code in <code>myobj.py</code> is enough to make the test pass</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">MyObj</span><span class="p">():</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">repo</span><span class="p">):</span>
<span class="n">repo</span><span class="o">.</span><span class="n">connect</span><span class="p">()</span>
</code></pre></div>
<p>As you can see, the <code>__init__()</code> method actually calls <code>repo.connect()</code>, where <code>repo</code> is expected to be a full-featured external object that provides a given API. In this case (for the moment) the API is just its <code>connect()</code> method. Calling <code>repo.connect()</code> when <code>repo</code> is a mock object silently creates the method as a mock object, as shown before.</p>
<p>The <code>assert_called_with()</code> method also allows us to check the parameters we passed when calling. To show this let us pretend that we expect the <code>MyObj.setup()</code> method to call <code>setup(cache=True, max_connections=256)</code> on the external object. As you can see we pass a couple of arguments (namely <code>cache</code> and <code>max_connections</code>) to the called method, and we want to be sure that the call was exactly in this form. The new test is thus</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">test_setup</span><span class="p">():</span>
<span class="n">external_obj</span> <span class="o">=</span> <span class="n">mock</span><span class="o">.</span><span class="n">Mock</span><span class="p">()</span>
<span class="n">obj</span> <span class="o">=</span> <span class="n">myobj</span><span class="o">.</span><span class="n">MyObj</span><span class="p">(</span><span class="n">external_obj</span><span class="p">)</span>
<span class="n">obj</span><span class="o">.</span><span class="n">setup</span><span class="p">()</span>
<span class="n">external_obj</span><span class="o">.</span><span class="n">setup</span><span class="o">.</span><span class="n">assert_called_with</span><span class="p">(</span><span class="n">cache</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">max_connections</span><span class="o">=</span><span class="mi">256</span><span class="p">)</span>
</code></pre></div>
<p>As usual the first run fails. Be sure to check this, it is part of the TDD methodology. You must have a test that DOES NOT PASS, then write some code that make it pass.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nv">PYTHONPATH</span><span class="o">=</span><span class="s2">"."</span><span class="w"> </span>py.test
<span class="o">==========================================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">==========================================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.4.3+,<span class="w"> </span>pytest-2.9.0,<span class="w"> </span>py-1.4.31,<span class="w"> </span>pluggy-0.3.1
rootdir:<span class="w"> </span>/home/leo/devel/mockplayground,<span class="w"> </span>inifile:<span class="w"> </span>pytest.ini
collected<span class="w"> </span><span class="m">2</span><span class="w"> </span>items<span class="w"> </span>
tests/test_mock.py<span class="w"> </span>.F
<span class="o">===============================================</span><span class="w"> </span><span class="nv">FAILURES</span><span class="w"> </span><span class="o">================================================</span>
______________________________________________<span class="w"> </span>test_setup<span class="w"> </span>_______________________________________________
<span class="w"> </span>def<span class="w"> </span>test_setup<span class="o">()</span>:
<span class="w"> </span><span class="nv">external_obj</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>mock.Mock<span class="o">()</span>
<span class="w"> </span><span class="nv">obj</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>myobj.MyObj<span class="o">(</span>external_obj<span class="o">)</span>
><span class="w"> </span>obj.setup<span class="o">()</span>
E<span class="w"> </span>AttributeError:<span class="w"> </span><span class="s1">'MyObj'</span><span class="w"> </span>object<span class="w"> </span>has<span class="w"> </span>no<span class="w"> </span>attribute<span class="w"> </span><span class="s1">'setup'</span>
tests/test_mock.py:14:<span class="w"> </span><span class="nv">AttributeError</span>
<span class="o">==================================</span><span class="w"> </span><span class="m">1</span><span class="w"> </span>failed,<span class="w"> </span><span class="m">1</span><span class="w"> </span>passed<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">0</span>.03<span class="w"> </span><span class="nv">seconds</span><span class="w"> </span><span class="o">===================================</span>
$
</code></pre></div>
<p>To show you what type of check the mock object provides let me implement a partially correct solution</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">MyObj</span><span class="p">():</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">repo</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_repo</span> <span class="o">=</span> <span class="n">repo</span>
<span class="n">repo</span><span class="o">.</span><span class="n">connect</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">setup</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_repo</span><span class="o">.</span><span class="n">setup</span><span class="p">(</span><span class="n">cache</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</code></pre></div>
<p>As you can see the external object has been stored in <code>self._repo</code> and the call to <code>self._repo.setup()</code> is not exactly what the test expects, lacking the <code>max_connections</code> parameter. Running pytest we obtain the following result (I removed most of the pytest output)</p>
<div class="highlight"><pre><span></span><code>E<span class="w"> </span>AssertionError:<span class="w"> </span>Expected<span class="w"> </span>call:<span class="w"> </span>setup<span class="o">(</span><span class="nv">cache</span><span class="o">=</span>True,<span class="w"> </span><span class="nv">max_connections</span><span class="o">=</span><span class="m">256</span><span class="o">)</span>
E<span class="w"> </span>Actual<span class="w"> </span>call:<span class="w"> </span>setup<span class="o">(</span><span class="nv">cache</span><span class="o">=</span>True<span class="o">)</span>
</code></pre></div>
<p>and you see that the error message is very clear about what we expected and what happened in our code.</p>
<p>As you can read in the official documentation, the <code>Mock</code> object also provides the following methods and attributes: <code>assert_called_once_with</code>, <code>assert_any_call</code>, <code>assert_has_calls</code>, <code>assert_not_called</code>, <code>called</code>, <code>call_count</code>. Each of them explores a different aspect of the mock behaviour concerning calls, make sure to check their description and the examples provided along.</p>
<h2 id="final-words">Final words<a class="headerlink" href="#final-words" title="Permanent link">¶</a></h2>
<p>In this first part of the series I described the behaviour of mock objects and the methods they provide to simulate return values and to test calls. They are a very powerful tool that allows you to avoid creating complex and slow tests that depend on external facilities to run, thus missing the main purpose of tests, which is that of <em>continuously</em> helping you to check your code.</p>
<p>In the next issue of the series I will explore the automatic creation of mock methods from a given object and the very important patching mechanism provided by the <code>patch</code> decorator and context manager.</p>
<h2 id="updates">Updates<a class="headerlink" href="#updates" title="Permanent link">¶</a></h2>
<p>2019-02-27: <a href="https://github.com/schneeemensch">schneeemensch</a> spotted a mistake in the code of the "Return value" section, where <code>m.some_attribute.side_effect()</code> was used instead of <code>m.some_attribute()</code>. Thanks!</p>
<h2 id="feedback">Feedback<a class="headerlink" href="#feedback" title="Permanent link">¶</a></h2>
<p>The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections.</p>Python decorators: metaprogramming with style2015-04-23T13:00:00+01:002015-04-23T13:00:00+01:00Leonardo Giordanitag:www.thedigitalcatonline.com,2015-04-23:/blog/2015/04/23/python-decorators-metaprogramming-with-style/<p>This post is the result of a lot of personal research on Python decorators, meta- and functional programming. I want however to thank Bruce Eckel and the people behind the open source book "Python 3 Patterns, Recipes and Idioms" for a lot of precious information on the subject. See the Resources section at the end of the post to check their work.</p>
<h2 id="is-python-functional">Is Python functional?<a class="headerlink" href="#is-python-functional" title="Permanent link">¶</a></h2>
<p>Well, no. Python is a strong object-oriented programming language and is not really going to mix OOP and functional like, for example, Scala (which is a very good language, by the way).</p>
<p>However, Python provides <em>some</em> features taken from functional programming. <a href="https://www.thedigitalcatonline.com/blog/2013/03/25/python-generators-from-iterators-to-cooperative-multitasking/">Generators and iterators</a> are one of them, and Python is not the only non pure functional programming language to have them in their toolbox.</p>
<p>Perhaps the most distinguishing feature of <a href="https://en.wikipedia.org/wiki/Functional_programming">functional languages</a> is that functions are first-class citizens (or first-class objects). This means that functions can be passed as an argument to other functions or can be returned by them. Functions, in functional languages, are just one of the data types available (even if this is a very rough simplification).</p>
<p>Python has three important features that allows it to provide a functional behaviour: references, function objects and callables.</p>
<h3 id="references">References<a class="headerlink" href="#references" title="Permanent link">¶</a></h3>
<p>Python variables share a common nature: they are all references. This means that variables are not typed per se, being pure memory addresses, and that functions do not declare the incoming data type for arguments (leaving aside gradual typing). Python polymorphism is based on delegation, and incoming function arguments are expected to provide a given behaviour, not a given structure.</p>
<p>Python functions are thus ready to accept every type of data that can be referenced, and functions can.</p>
<p>Read <a href="https://www.thedigitalcatonline.com/blog/2014/08/21/python-3-oop-part-4-polymorphism/">this post</a> to dive into delegation-based polymorphism and references in Python.</p>
<h2 id="functions-objects">Functions objects<a class="headerlink" href="#functions-objects" title="Permanent link">¶</a></h2>
<p>Since Python pushes the object-oriented paradigm to its maximum, it makes a point of always following the tenet <em>everything is an object</em>. So Python functions are objects as you can see from this simple example</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="k">def</span> <span class="nf">f</span><span class="p">():</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="go"><class 'function'></span>
<span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="nb">type</span><span class="p">(</span><span class="n">f</span><span class="p">))</span>
<span class="go"><class 'type'></span>
<span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="n">f</span><span class="p">)</span><span class="o">.</span><span class="vm">__bases__</span>
<span class="go">(<class 'object'>,)</span>
<span class="gp">>>></span>
</code></pre></div>
<p>Given that, Python does nothing special to treat functions like first-class citizens, it simply recognizes that they are objects just like any other thing.</p>
<h3 id="callables">Callables<a class="headerlink" href="#callables" title="Permanent link">¶</a></h3>
<p>While Python has the well-defined <code>function</code> class seen in the above example, it relies more on the presence of the <code>__call__</code> method. That is, in Python any object can act as a function, provided that it has this method, which is invoked when the object is "called".</p>
<p>This will be crucial for the discussion about decorators, so be sure that you remember that we are usually more interested in <em>callable objects</em> and not only in functions, which, obviously, are a particular type of callable objects (or simply <em>callables</em>).</p>
<p>The fact that functions are callables can also be shown with some simple code</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="k">def</span> <span class="nf">f</span><span class="p">():</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="n">f</span><span class="o">.</span><span class="fm">__call__</span>
<span class="go"><method-wrapper '__call__' of function object at 0xb6709fa4></span>
</code></pre></div>
<h2 id="metaprogramming">Metaprogramming<a class="headerlink" href="#metaprogramming" title="Permanent link">¶</a></h2>
<p>While this is not a post on languages theory, it is worth spending a couple of words about metaprogramming. Usually "programming" can be seen as the task of applying transformations to data. Data and functions can be put together by an object-oriented approach, but they still are two different things. But you soon realize that, as you may run some code to change data, you may also run some code to change the code itself.</p>
<p>In low level languages this can be very simple, since at machine level everything is a sequence of bytes, and changing data or code does not make any difference. One of the most simple examples that I recall from my x86 Assembly years is the very simple self obfuscation code found is some computer viruses. The code was encrypted with a <a href="https://en.wikipedia.org/wiki/XOR_cipher">XOR cipher</a> and the first thing the code itself did upon execution was to decrypt its own code and then run it. The purpose of such tricks was (and is) to obfuscate the code such that it would be difficult for an antivirus to find the virus code and remove it. This is a very primitive form of metaprogramming, since it recognizes that for Assembly language there is no real distinction between code and data.</p>
<p>In higher lever languages such as Python achieving metaprogramming is no more a matter of changing byte values. It requires the language to treat its own structures as data. Every time we are trying to alter the behaviour of a language part we are actually metaprogramming. The first example that usually comes to mind are metaclasses (probably due to the "meta" word in their name), which are actually a way to change the default behaviour of the class creation process. Classes (part of the language) are created by another part of the language (metaclasses).</p>
<h2 id="decorators">Decorators<a class="headerlink" href="#decorators" title="Permanent link">¶</a></h2>
<p><a href="https://www.thedigitalcatonline.com/blog/2014/09/01/python-3-oop-part-5-metaclasses/">Metaclasses</a> are often perceived as a very tricky and dangerous thing to play with, and indeed they are seldom required in Python, with the most notable exception (no pun intended) being the <a href="https://www.thedigitalcatonline.com/blog/2014/09/04/python-3-oop-part-6-abstract-base-classes/">Abstract Base Classes</a> provided by the <code>collections</code> module.</p>
<p>Decorators, on the other side, are a feature loved by many experienced programmers and after their introduction the community has developed a big set of very interesting use cases.</p>
<p>I think that the first approach to decorators is often difficult for beginners because the functional version of decorators are indeed a bit complex to understand. Luckily, Python allows us to write decorators using classes too, which make the whole thing really easy to understand and write, I think.</p>
<p>So I will now review Python decorators starting from their rationale, then looking at class-based decorators without arguments, class-based decorators with arguments, and finally moving to function-based decorators.</p>
<h2 id="rationale">Rationale<a class="headerlink" href="#rationale" title="Permanent link">¶</a></h2>
<p>What are decorators, and why should you learn how to define and use them? </p>
<p>Well, decorators are a way to change the behaviour of a function or a class, so they are actually a way of metaprogramming, but they make it a lot more accessible than metaclasses do. Decorators are, in my opinion, a very <em>natural</em> way of altering functions and classes.</p>
<p>Moreover, with the addition of some syntactic sugar, they are a very compact way to both make changes and signal that those changes have been made.</p>
<p>The best syntactic form of a decorator is the following</p>
<div class="highlight"><pre><span></span><code><span class="nd">@dec</span>
<span class="k">def</span> <span class="nf">func</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwds</span><span class="p">):</span>
<span class="k">pass</span>
</code></pre></div>
<p>where <code>dec</code> is the name of the decorator and the function <code>func</code> is said to be <em>decorated</em> by it. As you can see any reader can quickly identify that the function has a special label attached, thus being altered in its behaviour.</p>
<p>This form, however, is just a simplification of the more generic form</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">func</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwds</span><span class="p">):</span>
<span class="k">pass</span>
<span class="n">func</span> <span class="o">=</span> <span class="n">dec</span><span class="p">(</span><span class="n">func</span><span class="p">)</span>
</code></pre></div>
<p>But what actually ARE the changes that you may want do to functions or classes? Let us stick for the moment to a very simple task: adding attributes. This is by no means a meaningless task, since there are many practical use cases that make use of it. Let us first test how we can add attributes to functions in plain Python</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="k">def</span> <span class="nf">func</span><span class="p">():</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="n">func</span><span class="o">.</span><span class="n">attr</span> <span class="o">=</span> <span class="s2">"a custom function attribute"</span>
<span class="gp">>>> </span><span class="n">func</span><span class="o">.</span><span class="n">attr</span>
<span class="go">'a custom function attribute'</span>
</code></pre></div>
<p>and to classes</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="k">class</span> <span class="nc">SomeClass</span><span class="p">:</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="n">SomeClass</span><span class="o">.</span><span class="n">attr</span> <span class="o">=</span> <span class="s2">"a custom class attribute"</span>
<span class="gp">>>> </span><span class="n">SomeClass</span><span class="o">.</span><span class="n">attr</span>
<span class="go">'a custom class attribute'</span>
<span class="gp">>>> </span><span class="n">s</span> <span class="o">=</span> <span class="n">SomeClass</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">s</span><span class="o">.</span><span class="n">attr</span>
<span class="go">'a custom class attribute'</span>
</code></pre></div>
<p>As you can see adding attributes to a class correctly results in a class attribute, which is thus shared by any instance of that class (check <a href="https://www.thedigitalcatonline.com/blog/2014/08/20/python-3-oop-part-2-classes-and-members/">this post</a> for some explanations about class attributes and sharing).</p>
<h2 id="class-based-decorators-without-arguments">Class-based decorators without arguments<a class="headerlink" href="#class-based-decorators-without-arguments" title="Permanent link">¶</a></h2>
<p>As already explained, Python allows you to call any object (as you do with functions) as long as it provides the <code>__call__()</code> method. So to write a class-based decorator you just need to create an object that defines such a method.</p>
<p>When used as a decorator, a class is instantiated at decoration time, that is when the function is defined, and called when the function is called.</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">CustomAttr</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">obj</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">attr</span> <span class="o">=</span> <span class="s2">"a custom function attribute"</span>
<span class="bp">self</span><span class="o">.</span><span class="n">obj</span> <span class="o">=</span> <span class="n">obj</span>
<span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">obj</span><span class="p">()</span>
</code></pre></div>
<p>As you can see there is already a lot of things that shall be clarified. First of all the class, when used as a decorator, is initialized with the object that is going to be decorated, here called <code>obj</code> (most of the time it is just called <code>f</code> for function, but you know that this is only a special case).</p>
<p>While the <code>__init__()</code> method is called at decoration time, the <code>__call__()</code> method of the decorator is called instead of the same method of the decorated object. In this case (decorator without arguments), the <code>__call__()</code> method of the decorator does not receive any argument. In this example we just "redirect" the call to the original function, which was stored during the initialization step.</p>
<p>So you see that in this case we have two different moments in which we may alter the behaviour of the decorated objects. The first is at its definition and the second is when it is actually called.</p>
<p>The decorator can be applied with the simple syntax shown in a previous section</p>
<div class="highlight"><pre><span></span><code><span class="nd">@CustomAttr</span>
<span class="k">def</span> <span class="nf">func</span><span class="p">():</span>
<span class="k">pass</span>
</code></pre></div>
<p>When Python parses the file and defines the function <code>func</code> the code it executes under the hood is</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">func</span><span class="p">():</span>
<span class="k">pass</span>
<span class="n">func</span> <span class="o">=</span> <span class="n">CustomAttr</span><span class="p">(</span><span class="n">func</span><span class="p">)</span>
</code></pre></div>
<p>according to the definition of decorator. This is why the class shall accept the decorated object as a parameter in its <code>__init__()</code> method.</p>
<p>Note that in this case the <code>func</code> object we obtain after the decoration is no more a function but a <code>CustomAttr</code> object</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">func</span>
<span class="go"><__main__.CustomAttr object at 0xb6f5ea8c></span>
</code></pre></div>
<p>and this is why in the <code>__init__()</code> method I attached the <code>attr</code> attribute to the class instance <code>self</code> and not to <code>obj</code>, so that now this works</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">func</span><span class="o">.</span><span class="n">attr</span>
<span class="go">'a custom function attribute'</span>
</code></pre></div>
<p>This replacement is also the reason why you shall also redefine <code>__call__()</code>. When you write <code>func()</code> you are not executing the function but calling the instance of <code>CustomAttr</code> returned by the decoration.</p>
<h2 id="class-based-decorators-with-arguments">Class-based decorators with arguments<a class="headerlink" href="#class-based-decorators-with-arguments" title="Permanent link">¶</a></h2>
<p>This case is the most natural step beyond the previous one. Once you started metaprogramming, you want to do it with style, and the first thing to do is to add parametrization. Adding parameters to decorators has the only purpose of generalizing the metaprogramming code, just like when you write parametrized functions instead of hardcoding values.</p>
<p>There is a big caveat here. Class-based decorators with arguments behave in a slightly different way to their counterpart without arguments. Specifically, the <code>__call__()</code> method is run during the decoration and not during the call.</p>
<p>Let us first review the syntax</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">CustomAttrArg</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">value</span> <span class="o">=</span> <span class="n">value</span>
<span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">obj</span><span class="p">):</span>
<span class="n">obj</span><span class="o">.</span><span class="n">attr</span> <span class="o">=</span> <span class="s2">"a custom function attribute with value </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">value</span><span class="p">)</span>
<span class="k">return</span> <span class="n">obj</span>
<span class="nd">@CustomAttrArg</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">func</span><span class="p">():</span>
<span class="k">pass</span>
</code></pre></div>
<p>Now the <code>__init__()</code> method shall accept some arguments, with the standard rules of Python functions for named and default arguments. The <code>__call__()</code> method receives the decorated object, which in the previous case was passed to the <code>__init__()</code> method.</p>
<p>The biggest change, however is that <code>__call__()</code> is not run when you call the decorated object, but immediately after <code>__init__()</code> during the decoration phase. This results in the following difference: while in the previous case the decorated object was no more itself, but an instance of the decorator, now the decorated objects becomes the return value of the <code>__call__()</code> method.</p>
<p>Remember that, when you call the decorated object, you are now actually calling what you get from <code>__call__()</code> so be sure of returning something meaningful.</p>
<p>In the above example I stored one single argument in <code>__init__()</code>, and this argument is passed to the decorator when applying it to the function. Then, in <code>__call__()</code>, I set the attribute of the decorated object, using the stored argument, and return the object itself. This is important, since I have to return a callable object.</p>
<p>This means that, if you have to do something complex with the decorated object, you may just define a local function that makes use of it and return this function. Let us see a very simple example</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">CustomAttrArg</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">value</span> <span class="o">=</span> <span class="n">value</span>
<span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">obj</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">wrap</span><span class="p">():</span>
<span class="c1"># Here you can do complex stuff</span>
<span class="n">obj</span><span class="p">()</span>
<span class="c1"># Here you can do complex stuff</span>
<span class="k">return</span> <span class="n">wrap</span>
<span class="nd">@CustomAttrArg</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">func</span><span class="p">():</span>
<span class="k">pass</span>
</code></pre></div>
<p>Here the returned object is no more the decorated one, but a new function <code>wrap()</code> defined locally. It is interesting to show how Python identifies it</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="nd">@CustomAttrArg</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="gp">... </span><span class="k">def</span> <span class="nf">func</span><span class="p">():</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">... </span>
<span class="gp">>>> </span><span class="n">func</span>
<span class="go"><function CustomAttrArg.__call__.<locals>.wrap at 0xb70185cc></span>
</code></pre></div>
<p>This pattern enables you to do every sort of things with the decorated object. Not only to change it (adding attributes, for example), but also pre- or post- filtering its results. You may start to understand that what we called metaprogramming may be very useful for everyday tasks, and not only for some obscure wizardry.</p>
<h3 id="decorators-and-prototypes">Decorators and prototypes<a class="headerlink" href="#decorators-and-prototypes" title="Permanent link">¶</a></h3>
<p>If you write a class-based decorator with arguments, you are in charge of returning a callable object of choice. There is no assumption on the returned object, even if the usual case is that the returned object has the same prototype as the decorated one.</p>
<p>This means that if the decorated object accepts zero arguments (like in my example), you usually return a callable that accepts zero arguments. This is however by no means enforced by the language, and through this you may push the metaprogramming technique a bit. I'm not providing examples of this technique in this post, however.</p>
<h2 id="function-based-decorators">Function-based decorators<a class="headerlink" href="#function-based-decorators" title="Permanent link">¶</a></h2>
<p>Function-based decorators are very simple for simple cases and a bit trickier for complex ones. The problem is that their syntax can be difficult to grasp at first sight if you never saw a decorator. They are indeed not very different from the class-based decorators with arguments case, as they define a local function that wraps the decorated object and return it.</p>
<p>The case without arguments is always the simplest one</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">decorate</span><span class="p">(</span><span class="n">f</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">wrap</span><span class="p">():</span>
<span class="n">f</span><span class="p">()</span>
<span class="k">return</span> <span class="n">wrap</span>
<span class="nd">@decorate</span>
<span class="k">def</span> <span class="nf">func</span><span class="p">():</span>
<span class="k">pass</span>
</code></pre></div>
<p>This behaves like the equivalent case with classes. The function is passed as an argument to the <code>decorate()</code> function by the decoration process that calls it passing the decorated object. When you actually call the function, however, you are actually calling <code>wrap()</code>.</p>
<p>As happens for class-based decorators, the parametrization changes the calling procedure. This is the code</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">decorate</span><span class="p">(</span><span class="n">arg1</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">wrap</span><span class="p">(</span><span class="n">f</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">_wrap</span><span class="p">(</span><span class="n">arg</span><span class="p">):</span>
<span class="n">f</span><span class="p">(</span><span class="n">arg</span> <span class="o">+</span> <span class="n">arg1</span><span class="p">)</span>
<span class="k">return</span> <span class="n">_wrap</span>
<span class="k">return</span> <span class="n">wrap</span>
<span class="nd">@decorate</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">func</span><span class="p">(</span><span class="n">arg</span><span class="p">):</span>
<span class="k">pass</span>
</code></pre></div>
<p>As you see it is not really straightforward, and this is the reason I preferred to discuss it as the last case. Recall what we learned about class-based decorators: the first call to the <code>decorate()</code> function happens when the decorator is called with an argument. Thus <code>@decorate(1)</code> calls <code>decorate()</code> passing <code>1</code> as <code>arg1</code>, and this function returns the <code>wrap()</code> local function.</p>
<p>This second function accepts another function as an argument, and indeed it is used in the actual decoration process, which can be represented by the code <code>func = wrap(func)</code>. This <code>wrap()</code> function, being used to decorate <code>func()</code>, wants to return a compatible object, that is in this case a function that accepts a single argument. This is why, in turn, <code>wrap()</code> defines and returns a <code>_wrap()</code> local function, which eventually uses both the argument passed to <code>func()</code> and the argument passed to the decorator.</p>
<p>So the process may be summarized as follows (I will improperly call <code>func_dec</code> the decorated function to show what is happening)</p>
<ul>
<li><code>@decorator(1)</code> returns a <code>wrap()</code> function (that knows the argument)</li>
<li><code>func</code> is redefined as <code>func_dec = wrap(func)</code> becoming <code>_wrap()</code></li>
<li>When you call <code>func_dec(arg)</code> Python executes <code>_wrap(arg)</code> which calls the original <code>func()</code> passing <code>1 + arg</code> as argument</li>
</ul>
<p>Obviously the power of the decorator concept is that you are not dealing with <code>func_dec()</code> but with <code>func()</code> itself, and all the "magic" happens under the hood.</p>
<p>If you feel uncomfortable with function-based decorators don't worry, as they are indeed a bit awkward. I usually stick to function based decorators for the simple cases (setting class attributes, for example), and move to class-based ones when the code is more complex.</p>
<h2 id="example">Example<a class="headerlink" href="#example" title="Permanent link">¶</a></h2>
<p>A good example of the power of decorators which comes out of the box is <code>functools.total_ordering</code>. The <code>functools</code> module provides a lot of interesting tools to push the functional approach in Python, most notably <code>partial()</code> and <code>partialmethod()</code>, which are however out of the scope of this post.</p>
<p>The <code>total_ordering</code> decorator (documented <a href="https://docs.python.org/3.4/library/functools.html#functools.total_ordering">here</a>) wants to make an object provide a full set of comparison ordering methods starting from a small set of them. Comparison methods are those methods Python calls when two objects are compared. For example when you write <code>a == b</code>, Python executes <code>a.__eq__(b)</code> and the same happens for the other five operators <code>></code> (<code>__gt__</code>), <code><</code> (<code>__lt__</code>), <code>>=</code> (<code>__ge__</code>), <code><=</code> (<code>__le__</code>) and <code>!=</code> (<code>__ne__</code>).</p>
<p>Mathematically all those operators may be expressed using only one of them and the <code>__eq__()</code> method, for example <code>__ne__</code> is <code>!__eq__</code> and <code>__lt__</code> is <code>__le__ and !__eq__</code>. This decorator makes use of this fact to provide the missing methods for the decorated object. A quick example</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">Person</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
<span class="k">def</span> <span class="fm">__eq__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">==</span> <span class="n">other</span><span class="o">.</span><span class="n">name</span>
<span class="k">def</span> <span class="fm">__lt__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o"><</span> <span class="n">other</span><span class="o">.</span><span class="n">name</span>
</code></pre></div>
<p>This is a simple class that defines the <code>==</code> and <code><</code> comparison methods.</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">p1</span> <span class="o">=</span> <span class="n">Person</span><span class="p">(</span><span class="s1">'Bob'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">p2</span> <span class="o">=</span> <span class="n">Person</span><span class="p">(</span><span class="s1">'Alice'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">p1</span> <span class="o">==</span> <span class="n">p2</span>
<span class="go">False</span>
<span class="gp">>>> </span><span class="n">p1</span> <span class="o"><</span> <span class="n">p2</span>
<span class="go">False</span>
<span class="gp">>>> </span><span class="n">p1</span> <span class="o">>=</span> <span class="n">p2</span>
<span class="gt">Traceback (most recent call last):</span>
File <span class="nb">"/home/leo/prova.py"</span>, line <span class="m">103</span>, in <span class="n"><module></span>
<span class="w"> </span><span class="n">p1</span> <span class="o">>=</span> <span class="n">p2</span>
<span class="gr">TypeError</span>: <span class="n">unorderable types: Person() >= Person()</span>
</code></pre></div>
<p>A big warning: Python doesn't complain if you try to perform the <code>></code> and <code>!=</code> comparisons but lacking the dedicated methods it does perform a "standard" comparison. This means that, as the documentation states <a href="https://docs.python.org/2/reference/datamodel.html#object.__lt__">here</a>, "There are no implied relationships among the comparison operators. The truth of x==y does not imply that x!=y is false."</p>
<p>With the <code>total_ordering</code> decorator, however, all six comparisons become available</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">functools</span>
<span class="nd">@functools</span><span class="o">.</span><span class="n">total_ordering</span>
<span class="k">class</span> <span class="nc">Person</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
<span class="k">def</span> <span class="fm">__eq__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">==</span> <span class="n">other</span><span class="o">.</span><span class="n">name</span>
<span class="k">def</span> <span class="fm">__lt__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o"><</span> <span class="n">other</span><span class="o">.</span><span class="n">name</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="n">p1</span> <span class="o">=</span> <span class="n">Person</span><span class="p">(</span><span class="s1">'Bob'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">p2</span> <span class="o">=</span> <span class="n">Person</span><span class="p">(</span><span class="s1">'Alice'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">p1</span> <span class="o">==</span> <span class="n">p2</span>
<span class="go">False</span>
<span class="gp">>>> </span><span class="n">p1</span> <span class="o">!=</span> <span class="n">p2</span>
<span class="go">True</span>
<span class="gp">>>> </span><span class="n">p1</span> <span class="o">></span> <span class="n">p2</span>
<span class="go">True</span>
<span class="gp">>>> </span><span class="n">p1</span> <span class="o"><</span> <span class="n">p2</span>
<span class="go">False</span>
<span class="gp">>>> </span><span class="n">p1</span> <span class="o">>=</span> <span class="n">p2</span>
<span class="go">True</span>
<span class="gp">>>> </span><span class="n">p1</span> <span class="o"><=</span> <span class="n">p2</span>
<span class="go">False</span>
</code></pre></div>
<h2 id="final-words">Final words<a class="headerlink" href="#final-words" title="Permanent link">¶</a></h2>
<p>Decorators are a very powerful tool, and they are worth learning. This may be for you just the first step into the amazing world of metaprogramming or just an advanced technique you may use to simplify your code. Whatever you do with them be sure to understand the difference between the two cases (with and without arguments) and don't avoid function-based decorators just because their syntax is a bit complex.</p>
<h2 id="resources">Resources<a class="headerlink" href="#resources" title="Permanent link">¶</a></h2>
<p>Many thanks to Bruce Eckel for <a href="http://www.artima.com/weblogs/viewpost.jsp?thread=240808">his three posts</a> which have been (I think) the source for the page on <a href="http://python-3-patterns-idioms-test.readthedocs.org/en/latest/PythonDecorators.html">Python 3 Patterns, Recipes and Idioms</a> (the latter is still work in progress). Update: as Bruce stated "the Python3 Patterns book is kind of a failed project". So beware that information contained there is not going to be updated. It is however still a good starting point for Python studies and investigations.</p>
<p>A good source of advanced decorators can be found at the <a href="https://wiki.python.org/moin/PythonDecoratorLibrary">Python Decorator Library</a>, and a lot of stuff may be found on Stackoverflow under the <a href="https://stackoverflow.com/questions/tagged/python-decorators">python-decorators tag</a>.</p>
<p><a href="https://github.com/GrahamDumpleton">Graham Dumpleton</a> wrote a very interesting a in-depth analysis of Python decorators <a href="https://github.com/GrahamDumpleton/wrapt/tree/develop/blog">here</a>.</p>
<h2 id="feedback">Feedback<a class="headerlink" href="#feedback" title="Permanent link">¶</a></h2>
<p>The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections.</p>Advanced use of Python decorators and metaclasses2014-10-14T09:43:08+02:002014-10-14T09:43:08+02:00Leonardo Giordanitag:www.thedigitalcatonline.com,2014-10-14:/blog/2014/10/14/decorators-and-metaclasses/<h2 id="abstract">Abstract<a class="headerlink" href="#abstract" title="Permanent link">¶</a></h2>
<p>While introducing people to Python metaclasses I realized that sometimes the big problem of the most powerful Python features is that programmers do not perceive how they may simplify their usual tasks. Therefore, features like metaclasses are considered a fancy but rather unuseful addition to a standard OOP language, instead of a real game changer.</p>
<p>This post wants to show how to use metaclasses and decorators to create a powerful class that can be inherited and customized by easily adding decorated methods.</p>
<h2 id="metaclasses-and-decorators-a-match-made-in-space">Metaclasses and decorators: a match made in space<a class="headerlink" href="#metaclasses-and-decorators-a-match-made-in-space" title="Permanent link">¶</a></h2>
<p>Metaclasses are a complex topic, and most of the times even advanced programmers do not see a wide range of practical uses for them. Chances are that this is the part of Python (or other languages that support metaclasses, like Smalltalk and Ruby) that fits the least the "standard" object-oriented patterns or solutions found in C++ and Java, just to mention two big players.</p>
<p>Indeed metaclasess usually come in play when programming advanced libraries or frameworks, where a lot of automation must be provided. For example, Django Forms system heavily relies on metaclasses to provide all its magic.</p>
<p>We also have to note, however, that we usually call "magic" or "tricks" all those techniques we are not familiar with, and as a result in Python many things are called this way, being its implementation often peculiar compared to other languages.</p>
<p>Time to bring some spice into your programming: let's practice some Python wizardry and exploit the power of the language!</p>
<p>In this post I want to show you an interesting joint use of decorators and metaclasses. I will show you how to use decorators to mark methods so that they can be automatically used by the class when performing a given operation.</p>
<p>More in detail, I will implement a class that can be called on a string to "process" it, and show you how to implement different "filters" through simple decorated methods. What I want to obtain is something like this:</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">MyStringProcessor</span><span class="p">(</span><span class="n">StringProcessor</span><span class="p">):</span>
<span class="nd">@stringfilter</span>
<span class="k">def</span> <span class="nf">capitalize</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="nd">@stringfilter</span>
<span class="k">def</span> <span class="nf">remove_double_spaces</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="n">msp</span> <span class="o">=</span> <span class="n">MyStringProcessor</span><span class="p">()</span>
<span class="s2">"A test string"</span> <span class="o">==</span> <span class="n">msp</span><span class="p">(</span><span class="s2">"a test string"</span><span class="p">)</span>
</code></pre></div>
<p>The module defines a <code>StringProcessor</code> class that I can inherit and customize adding methods that have a standard signature <code>(self, str)</code> and are decorated with <code>@stringfilter</code>. This class can later be instantiated and the instance used to directly process a string and return the result. Internally the class automatically executes all the decorated methods in succession. I also would like the class to obey the order I defined the filters: first defined, first executed.</p>
<h2 id="the-hitchhikers-guide-to-metaclasses">The Hitchhiker's Guide To Metaclasses<a class="headerlink" href="#the-hitchhikers-guide-to-metaclasses" title="Permanent link">¶</a></h2>
<p>How can metaclasses help to reach this target?</p>
<p>Simply put, metaclasses are classes that are instantiated to get classes. That means that whenever I use a class, for example to instantiate it, first Python <em>builds</em> that class using the metaclass and the class definition we wrote. For example, you know that you can find the class members in the <code>__dict__</code> attribute: this attribute is created by the standard metaclass, which is <code>type</code>.</p>
<p>Given that, a metaclass is a good starting point for us to insert some code to identify a subset of functions inside the definition of the class. In other words, we want the output of the metaclass (that is, the class) be built exactly as happens in the standard case, but with an addition: a separate list of all the methods decorated with <code>@stringfilter</code>.</p>
<p>You know that a class has a <em>namespace</em>, that is a dictionary of what was defined inside the class. So, when the standard <code>type</code> metaclass is used to create a class, the class body is parsed and a <code>dict()</code> object is used to collect the namespace.</p>
<p>We are however interested in preserving the order of definition and a Python dictionary is an unordered structure, so we take advantage of the <code>__prepare__</code> hook introduced in the class creation process with Python 3. This function, if present in the metaclass, is used to preprocess the class and to return the structure used to host the namespace. So, following the example found in the official documentation, we start defining a metaclass like</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">FilterClass</span><span class="p">(</span><span class="nb">type</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__prepare__</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">bases</span><span class="p">,</span> <span class="o">**</span><span class="n">kwds</span><span class="p">):</span>
<span class="k">return</span> <span class="n">collections</span><span class="o">.</span><span class="n">OrderedDict</span><span class="p">()</span>
</code></pre></div>
<p>This way, when the class will be created, an <code>OrderedDict</code> will be used to host the namespace, allowing us to keep the definition order. Please note that the signature <code>__prepare__(name, bases, **kwds)</code> is enforced by the language. If you want the method to get the metaclass as a first argument (because the code of the method needs it) you have to change the signature to <code>__prepare__(metacls, name, bases, **kwds)</code> and decorate it with <code>@classmethod</code>.</p>
<p>The second function we want to define in our metaclass is <code>__new__</code>. Just like happens for the instantiation of classes, this method is invoked by Python to get a new instance of the metaclass, and is run before <code>__init__</code>. Its signature has to be <code>__new__(metacls, name, bases, namespace, **kwds)</code> and the result shall be an instance of the metaclass. As for its normal class counterpart (after all a metaclass is a class), <code>__new__()</code> usually wraps the same method of the parent class, <code>type</code> in this case, adding its own customizations.</p>
<p>The customization we need is the creation of a list of methods that are marked in some way (the decorated filters). Say for simplicity's sake that the decorated methods have an attribute <code>_filter</code>.</p>
<p>The full metaclass is then</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">FilterClass</span><span class="p">(</span><span class="nb">type</span><span class="p">):</span>
<span class="nd">@classmethod</span>
<span class="k">def</span> <span class="fm">__prepare__</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">bases</span><span class="p">,</span> <span class="o">**</span><span class="n">kwds</span><span class="p">):</span>
<span class="k">return</span> <span class="n">collections</span><span class="o">.</span><span class="n">OrderedDict</span><span class="p">()</span>
<span class="k">def</span> <span class="fm">__new__</span><span class="p">(</span><span class="n">metacls</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">bases</span><span class="p">,</span> <span class="n">namespace</span><span class="p">,</span> <span class="o">**</span><span class="n">kwds</span><span class="p">):</span>
<span class="n">result</span> <span class="o">=</span> <span class="nb">type</span><span class="o">.</span><span class="fm">__new__</span><span class="p">(</span><span class="n">metacls</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">bases</span><span class="p">,</span> <span class="nb">dict</span><span class="p">(</span><span class="n">namespace</span><span class="p">))</span>
<span class="n">result</span><span class="o">.</span><span class="n">_filters</span> <span class="o">=</span> <span class="p">[</span>
<span class="n">value</span> <span class="k">for</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">namespace</span><span class="o">.</span><span class="n">values</span><span class="p">()</span> <span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="s1">'_filter'</span><span class="p">)]</span>
<span class="k">return</span> <span class="n">result</span>
</code></pre></div>
<p>Now we have to find a way to mark all filter methods with a <code>_filter</code> attribute.</p>
<h2 id="the-anatomy-of-purple-decorators">The Anatomy of Purple Decorators<a class="headerlink" href="#the-anatomy-of-purple-decorators" title="Permanent link">¶</a></h2>
<p><strong>decorate</strong>: <em>to add something to an object or place, especially in order to make it more attractive</em> (Cambridge Dictionary)</p>
<p>Decorators are, as the name suggests, the best way to augment functions or methods. Remember that a decorator is basically a callable that accepts another callable, processes it, and returns it.</p>
<p>Used in conjunction with metaclasses, decorators are a very powerful and expressive way to implement advanced behaviours in our code. In this case we may easily use them to add an attribute to decorated methods, one of the most basic tasks for a decorator.</p>
<p>I decided to implement the <code>@stringfilter</code> decorator as a function, even if I usually prefer implementing them as classes. The reason is that decorator classes behave differently when used to implement decorators without arguments rather than decorators with arguments. In this case this difference would force to write some complex code and an explanation of that would be overkill now. In a future post on dectorators you will find all the gory details, but in the meantime you may check the three Bruce Eckel posts listed in the references section.</p>
<p>The decorator is very simple:</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">stringfilter</span><span class="p">(</span><span class="n">func</span><span class="p">):</span>
<span class="n">func</span><span class="o">.</span><span class="n">_filter</span> <span class="o">=</span> <span class="kc">True</span>
<span class="k">return</span> <span class="n">func</span>
</code></pre></div>
<p>As you can see the decorator just creates an attribute called <code>_filter</code> into the function (remember that functions are objects). The actual value of this attribute is not important in this case, since we are just interested in telling apart class members that contain it.</p>
<h2 id="the-dynamics-of-a-callable-object">The Dynamics of a Callable Object<a class="headerlink" href="#the-dynamics-of-a-callable-object" title="Permanent link">¶</a></h2>
<p>We are used to think about functions as special language components that may be "called" or executed. In Python functions are objects, just like everything else, and the feature that allows them to be executed comes from the presence of the <code>__call__()</code> method. Python is polymorphic by design and based on delegation, so (almost) everything that happens in the code relies on some features of the target object.</p>
<p>The result of this generalization is that every object that contains the <code>__call__()</code> method may be executed like a function, and gains the name of <em>callable object</em>.</p>
<p>The <code>StringProcessor</code> class shall thus contain this method and perform there the string processing with all the contained filters. The code is</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">StringProcessor</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">FilterClass</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">string</span><span class="p">):</span>
<span class="n">_string</span> <span class="o">=</span> <span class="n">string</span>
<span class="k">for</span> <span class="n">_filter</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_filters</span><span class="p">:</span>
<span class="n">_string</span> <span class="o">=</span> <span class="n">_filter</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">_string</span><span class="p">)</span>
<span class="k">return</span> <span class="n">_string</span>
</code></pre></div>
<p>A quick review of this simple function shows that it accepts the string as an argument, stores it in a local variable and loops over the filters, executing each of them on the local string, that is on the result of the previous filter.</p>
<p>The filter functions are extracted from the <code>self._filters</code> list, that is compiled by the <code>FilterClass</code> metaclass we already discussed.</p>
<p>What we need to do now is to inherit from <code>StringProcessor</code> to get the metaclass machinery and the <code>__call__()</code> method, and to define as many methods as needed, decorating them with the <code>@stringfilter</code> decorator.</p>
<p>Note that, thanks to the decorator and the metaclass, you may have other methods in your class that do not interfere with the string processing as long as they are not decorated with the decorator under consideration.</p>
<p>An example derived class may be the following</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">MyStringProcessor</span><span class="p">(</span><span class="n">StringProcessor</span><span class="p">):</span>
<span class="nd">@stringfilter</span>
<span class="k">def</span> <span class="nf">capitalize</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">string</span><span class="p">):</span>
<span class="k">return</span> <span class="n">string</span><span class="o">.</span><span class="n">capitalize</span><span class="p">()</span>
<span class="nd">@stringfilter</span>
<span class="k">def</span> <span class="nf">remove_double_spaces</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">string</span><span class="p">):</span>
<span class="k">return</span> <span class="n">string</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">' '</span><span class="p">,</span> <span class="s1">' '</span><span class="p">)</span>
</code></pre></div>
<p>The two <code>capitalize()</code> and <code>remove_double_spaces()</code> methods have been decorated, so they will be applied in order to any string passed when calling the class. A quick example of this last class is</p>
<div class="highlight"><pre><span></span><code><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">strproc</span>
<span class="gp">>>> </span><span class="n">msp</span> <span class="o">=</span> <span class="n">strproc</span><span class="o">.</span><span class="n">MyStringProcessor</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">input_string</span> <span class="o">=</span> <span class="s2">"a test string"</span>
<span class="gp">>>> </span><span class="n">output_string</span> <span class="o">=</span> <span class="n">msp</span><span class="p">(</span><span class="n">input_string</span><span class="p">)</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s2">"INPUT STRING:"</span><span class="p">,</span> <span class="n">input_string</span><span class="p">)</span>
<span class="go">INPUT STRING: a test string</span>
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="s2">"OUTPUT STRING:"</span><span class="p">,</span> <span class="n">output_string</span><span class="p">)</span>
<span class="go">OUTPUT STRING: A test string</span>
<span class="gp">>>> </span>
</code></pre></div>
<p>That's it!</p>
<h2 id="final-words">Final words<a class="headerlink" href="#final-words" title="Permanent link">¶</a></h2>
<p>There are obviously other ways to accomplish the same task, and this post wanted just to give a practical example of what metaclasses are good for, and why I think that they should be part of any Python programmer's arsenal.</p>
<p>[Update] Some developers <a href="https://www.reddit.com/r/Python/comments/2jbi2f/advanced_use_of_python_decorators_and_metaclasses/">on Reddit</a> and Linkedin raised objections to the content of the post mainly about the fact that the example may be perfectly implemented without metaclasses and about the dangerous nature of metaclasses. Since I try to learn from everyone, I thank them for their suggestions.</p>
<p>It is especially interesting to know that some developers consider the use of metaclasses a risky business, because they hide a lot of the structure of the class and the underlying machinery. This is true, so (as you should do for other technologies), think carefully about the reasons that drive you to use metaclasses, and be sure you know them well.</p>
<h2 id="book-trivia">Book Trivia<a class="headerlink" href="#book-trivia" title="Permanent link">¶</a></h2>
<p>Section titles come from the following books: <em>A Match Made in Space - George McFly</em>, <em>The Hitchhiker's Guide To the Galaxy - Various Authors</em>, <em>The Anatomy of Purple Dragons - Unknown</em>, <em>The Dynamics of an Asteroid - James Moriarty</em>.</p>
<h2 id="source-code">Source code<a class="headerlink" href="#source-code" title="Permanent link">¶</a></h2>
<p>The <a href="/code/metaclasses/strproc.py">strproc.py</a> file contains the full source code used in this post.</p>
<h2 id="online-resources">Online resources<a class="headerlink" href="#online-resources" title="Permanent link">¶</a></h2>
<p>The following resources may be useful.</p>
<h4 id="metaclasses">Metaclasses<a class="headerlink" href="#metaclasses" title="Permanent link">¶</a></h4>
<ul>
<li>Python 3 official documentation: <a href="https://docs.python.org/3.4/reference/datamodel.html#customizing-class-creation">customizing class creation</a>.</li>
<li>Python 3 OOP Part 5 - Metaclasses <a href="https://www.thedigitalcatonline.com/blog/2014/09/01/python-3-oop-part-5-metaclasses/">on this blog</a>.</li>
<li><a href="http://python-3-patterns-idioms-test.readthedocs.org/en/latest/Metaprogramming.html">Metaprogramming examples and patterns</a> (still using some Python 2 code but useful).</li>
</ul>
<h4 id="decorators">Decorators<a class="headerlink" href="#decorators" title="Permanent link">¶</a></h4>
<ul>
<li><a href="http://www.artima.com/weblogs/viewpost.jsp?thread=240808">Bruce Eckel</a> on decorators (series of three posts, 6 years old but still valid).</li>
<li><a href="http://simeonfranklin.com/blog/2012/jul/1/python-decorators-in-12-steps/">A different approach</a> on explaining decorators.</li>
<li><a href="http://www.jeffknupp.com/blog/2013/11/29/improve-your-python-decorators-explained/">Jeff Knupp</a> goes deep inside the concept of function.</li>
</ul>
<h4 id="callable-objects">Callable objects<a class="headerlink" href="#callable-objects" title="Permanent link">¶</a></h4>
<ul>
<li><a href="http://www.rafekettler.com/magicmethods.html#callable">Rafe Kettler</a> provides a very detaild guide on Python "magic" methods.</li>
</ul>
<h2 id="updates">Updates<a class="headerlink" href="#updates" title="Permanent link">¶</a></h2>
<p>2014-10-17: <a href="https://github.com/thermokarst">Matthew Dillon</a> and <a href="https://github.com/dmcdougall">Damon McDougall</a> spotted two typos. Thank you!</p>
<p>2014-10-17: <a href="https://www.reddit.com/user/ionelmc">ionelmc</a> suggested two corrections <a href="https://www.reddit.com/r/Python/comments/2jbi2f/advanced_use_of_python_decorators_and_metaclasses/cla696y">here</a> and <a href="https://www.reddit.com/r/Python/comments/2jbi2f/advanced_use_of_python_decorators_and_metaclasses/cla6o77">here</a>. Both are correct so I implemented them. The second one is more about style, but fits well the introductory purpose of the post. Thanks!</p>
<h2 id="feedback">Feedback<a class="headerlink" href="#feedback" title="Permanent link">¶</a></h2>
<p>The <a href="https://github.com/TheDigitalCatOnline/blog_source/issues">GitHub issues</a> page is the best place to submit corrections.</p>