Memcached Internals


http://martinfitzpatrick.name/article/control-memcached-from-the-command-line
http://work.tinou.com/2011/04/memcached-for-dummies.html :
  • "Memcached is LRU per slab class, but not globally LRU"

  • "Which brings me to another point, items/chunks are not actively reclaimed/expired.  Memached does not have a background thread that explicitly expires items, reclaiming used chunks for new items.  When a slab needs a chunk and there are no more pages, Memcache will look at the queue tail for items to evict.  Memcache will make a best effort to evict expired items (items you've explicitly set to expire after some time).  In scenario 1, item 2, an expired item, is evicted.  However, in scenario 2, item 1, which has not yet expired, will be evicted, even though item 4 would seem like the better candidate.  But since item 4 is not near the tail, Memcached stops looking and just expires item 1."
http://highscalability.com/strategy-break-memcache-dog-pile
Reader comment:
  • "I tackled this problem a little differently, though I have to say I like the stale key approach. For cases where showing cached data past its expiration is unacceptable though (for example, when data needs to be expired, like cached shopping cart contents), I'll describe my solution.

    Basically, on a cache get failure, any requesting clients would generate a rand(1,1000) and cache it to a lock key using memcached's add(). Then, each client would read back the lock key value and compare it to the random value they tried to set. Since add() only sets a key if it doesn't already exist, only one client will acquire this lock (the others will read back that client's random value and not their own).

    Clients that didn't successfully acquire the lock would sleep for a specified time (few hundred ms) and go back and retry the cache get. If the data still wasn't cached, they'd attempt to acquire the lock again, and repeat until either they successfully read refreshed data from the cache, or got the lock and refreshed themselves.

    The lock key would expire quickly in case one client aborted before the refresh was successful, to avoid hanging the other clients."
* video: http://www.infoq.com/presentations/lutke-rockstar-memcaching

Comments

Post a Comment

Popular posts from this blog

Handling control characters (escaping) in python for json and mysql

JSON loads() and dumps(): how control characters are handled The json lib escapes control characters when you dump(s) a python object into a json string. import json aa = "String with\ttab and\nnewline" print "1. Py string with control chars, unescaped: '%s'" % aa aa = json.dumps(aa, indent = 2, encoding="utf8") # dumping py-obj into json-string will cause control chars to be escaped print "\n2. JSON Encoded String with control chars, escaped: '%s'" % aa aa = json.loads(aa) # loading json-string into py-obj will leave control chars unescaped print "\n3. Py object (= string) with control chars, unescaped: '%s'" % aa Output: 1. Py string with control chars, unescaped: 'String with tab and newline' 2. JSON Encoded String with control chars, escaped: '"String with\ttab and\nnewline"' 3. Py object (= string) with control chars, unescaped: 'String with tab and newline' Note that json.l...
Read more

python port sniffer with pcapy and impacket

The code below will sniff all network traffic (use a filter to limit the sniffer to only certain packets) during 1000 milliseconds and print some info about it. It requires 2 python modules: pcapy for capturing the packets impacket for analyzing the packet content Note that this sample uses pcapy's dispatch() method and relies on its ability to return automatically after a timeout expires. This functionality is not available on all platforms (e.g. not on linux). import os, logging, socket import pcapy, impacket, impacket.ImpactDecoder ETHERNET_MAX_FRAME_SIZE = 1518 PROMISC_MODE = 0 class Packet(object): def __init__(self, bytes, sniffed_bytes, timestamp, proto_id, src_ip, tgt_ip, src_port, tgt_port, msgs): """ @brief Initialize data. """ super(Packet, self).__init__() self.bytes = bytes self.sniffed_bytes = sniffed_bytes self.src_ip = src_ip self.tgt_ip = tgt_ip self.src_port = src_port self.tgt_port = tgt_port ...
Read more

Django field, form and model validation process

Django's form and field validation process can be best understood by looking at the code in django's BaseForm class (in django.forms.forms) - see below. See also this blog post for a graphic overview. The form.full_clean() method is called by django at the start of the validation process (by form.is_valid(), usually immediately after the view receives the posted data). full_clean() performs a 3 phase validation , with each phase corresponding to a call of one of the protected methods below: self._clean_fields() : Field validation This method iterates over the form fields: For each field, it calls the Widget's value_from_datadict() method, passing the complete post data in a QueryDict. The Widget's task is to retrieve its own value from this QueryDict and return it. The returned value is then passed to the Field's own clean() method. This method does the initial validation (e.g. checking if the value is not empty), without touching the db. If the Field's...
Read more