Python language: the with statement and context managers

What ?

The with statement is used to wrap the execution of a code block, in such a manner that
  • a runtime context is set up, before the code block executes.
  • the runtime context is teared down, after the code block has executed (no matter if an exception is raised inside of the code block).
To do this, the with statement depends on a context manager, which handles the set up and tear down operations.

Why ?

The with statement is a generalization of the try-finally construct.
From the python docs:
If finally is present, it specifies a ‘cleanup’ handler. The try clause is executed, including any except and else clauses. If an exception occurs in any of the clauses and is not handled, the exception is temporarily saved. The finally clause is executed. If there is a saved exception, it is re-raised at the end of the finally clause. If the finally clause raises another exception or executes a return or break statement, the saved exception is discarded:
def f():
    try:
        1/0
    finally:
        return 42

>>> f()
42
The exception information is not available to the program during execution of the finally clause.
The with statement behaves in the same way, guaranteeing that the 'cleanup handler' is always executed, even if an exception is raised. Compared to try-finally, usage of the with statement comes with some benefits:
  • by implementing the context manager, you can improve code reusability.
  • if an exception was raised by your code block, the context manager's __exit__() method - which handles tearing down the runtime context  - has access to this exception. When returning control to the with statement, exit() can either suppress the exception (= by returning true) or let the with statement re-raise the exception (= default).

How ?

There are two forms of the with statement. In the first, the context object does not return/yield a context-specific object to work with.
with context :
    do_something()
In the second, the context provides us an object to be used within the context.
with context as variable:
    do_something(variable)
The context manager can be implemented in two different ways: as a class or as a generator function. See the links below for details and examples:

Context Manager as a Class

This is the most transparant way to implement a context manager: create a class in which has an __enter__() and __exit__() method. Here's an example of a lock with memcached:
class Lock():
   def __init__(self, name):
       self.key = "lock.%s" % name

   def __enter__(self):
       value = cache.add(self.key, "1", 60000)
       return value

   def __exit__(self, exc_type, exc_val, exc_tb):
       cache.delete(self.key)
       return False
The Lock context manager can be used like this: 
with Lock("myname") as lock_acquired:
    if lock_acquired:  # test the yielded bool, to see if the lock is acquired
        doSomeExpensiveStuff()

Context Manager as a Generator Function

In this case, the @contextmanager decorator is used to transform a simple generator function into a context manager. All function code before the yield statement is responsible for the setup (same as __enter__), all code after the yield statement is responsible for the tear down (same as __exit__).

See http://www.itmaybeahack.com/book/python-2.6/html/p03/p03c07_contexts.html#defining-a-context-manager-function for more details.

Our locking class could be rewritten like this: 
@contextmanager
def lock(name):
    key = "lock.%s" % name  # setup starts here
    lock = cache.add(key, "1", 60000)
    yield lock  # yield the lock result (True or False) to the with stmt
    if lock:  # teardown starts here
        cache.delete(key)
The Lock context manager can be used like this: 
with lock("myname") as lock_acquired:
    if lock_acquired:  # test the yielded bool, to see if the lock is acquired
        doSomeExpensiveStuff()
See also these examples of memcached locking using generators:

File access

In Python 2.5, the file object has been equipped with __enter__ and __exit__ methods out of the box; the former simply returns the file object itself, and the latter closes the file.

So to open a file, process its contents, and make sure to close it, this is all the code you need: 
with open("x.txt") as f:
    data = f.read()
    # do something with data

Comments

Post a Comment

Popular posts from this blog

Handling control characters (escaping) in python for json and mysql

JSON loads() and dumps(): how control characters are handled The json lib escapes control characters when you dump(s) a python object into a json string. import json aa = "String with\ttab and\nnewline" print "1. Py string with control chars, unescaped: '%s'" % aa aa = json.dumps(aa, indent = 2, encoding="utf8") # dumping py-obj into json-string will cause control chars to be escaped print "\n2. JSON Encoded String with control chars, escaped: '%s'" % aa aa = json.loads(aa) # loading json-string into py-obj will leave control chars unescaped print "\n3. Py object (= string) with control chars, unescaped: '%s'" % aa Output: 1. Py string with control chars, unescaped: 'String with tab and newline' 2. JSON Encoded String with control chars, escaped: '"String with\ttab and\nnewline"' 3. Py object (= string) with control chars, unescaped: 'String with tab and newline' Note that json.l...
Read more

python port sniffer with pcapy and impacket

The code below will sniff all network traffic (use a filter to limit the sniffer to only certain packets) during 1000 milliseconds and print some info about it. It requires 2 python modules: pcapy for capturing the packets impacket for analyzing the packet content Note that this sample uses pcapy's dispatch() method and relies on its ability to return automatically after a timeout expires. This functionality is not available on all platforms (e.g. not on linux). import os, logging, socket import pcapy, impacket, impacket.ImpactDecoder ETHERNET_MAX_FRAME_SIZE = 1518 PROMISC_MODE = 0 class Packet(object): def __init__(self, bytes, sniffed_bytes, timestamp, proto_id, src_ip, tgt_ip, src_port, tgt_port, msgs): """ @brief Initialize data. """ super(Packet, self).__init__() self.bytes = bytes self.sniffed_bytes = sniffed_bytes self.src_ip = src_ip self.tgt_ip = tgt_ip self.src_port = src_port self.tgt_port = tgt_port ...
Read more

Django field, form and model validation process

Django's form and field validation process can be best understood by looking at the code in django's BaseForm class (in django.forms.forms) - see below. See also this blog post for a graphic overview. The form.full_clean() method is called by django at the start of the validation process (by form.is_valid(), usually immediately after the view receives the posted data). full_clean() performs a 3 phase validation , with each phase corresponding to a call of one of the protected methods below: self._clean_fields() : Field validation This method iterates over the form fields: For each field, it calls the Widget's value_from_datadict() method, passing the complete post data in a QueryDict. The Widget's task is to retrieve its own value from this QueryDict and return it. The returned value is then passed to the Field's own clean() method. This method does the initial validation (e.g. checking if the value is not empty), without touching the db. If the Field's...
Read more