Python

Finding a missing print server

I have a TrendNet TE100-P1P print server. It's a little $30 gizmo with an Ethernet jack on one end and a parallel printer port on the other, so that you can turn a printer into a network printer without hanging it off a computer. Bought it because my new motherboard didn't have a parallel port. The other benefit is that now the printer works when my main desktop computer is off.

Anyway, we had a power failure, and the printer stopped working. No "lexmark" found in local DNS. No DHCP lease for the TrendNet gizmo.

The gizmo was still there, but I couldn't find it. There's no UI on the device, just the network jack. So I wrote a little script (below) to portscan my LAN for anything that answered on port 631. That found it.

Turns out that when the power goes out, the print server reconfigures its network settings from DHCP to fixed. It keeps its last network settings, but the DHCP / DNS server no longer has any record of them, so you can't find it unless you remember what its last IP was.

Here's the script. Because I had 700+ addresses to check, and it sometimes takes a few seconds for a failed socket connection to timeout, I used 250 threads to make it faster. (I initally tried using a separate thread for each IP, but I hit an OS limit at 255 threads, and decided to use a fixed-size pool and two shared work queues.)

#!/usr/bin/env python

"""Hunting for the TrendNet print server"""

import socket
import threading
import Queue

socket.setdefaulttimeout(5)

port = 631
num_threads = 250

class ConnectThread(threading.Thread):
    def __init__(self, in_queue, out_queue):
        threading.Thread.__init__(self)
        self.in_queue = in_queue
        self.out_queue = out_queue
        self.setDaemon(True)

    def run(self):
        while True:
            ip = self.in_queue.get()
            try:
                sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
                sock.connect((ip, port))
            except socket.error:
                self.out_queue.put((ip, False))
            else:
                self.out_queue.put((ip, True))
            self.in_queue.task_done()

def main():
    job_queue = Queue.Queue()
    result_queue = Queue.Queue()
    count = 0
    for net in range(3):
        for addr in range(1, 255 + 1):
            ip = "192.168.%d.%d" % (net, addr)
            count += 1
            job_queue.put(ip)
    for unused in xrange(num_threads):
        ConnectThread(job_queue, result_queue).start()
    job_queue.join()

    for unused in xrange(count):
        ip, status = result_queue.get()
        if status:
            print ip
    

if __name__ == "__main__":
    main()

Linux
Programming
Python

Comments (0)

Permalink

Drawing rectangles in PyGTK: GDK vs. Cairo

Someone on the PyGTK mailing list just asked which is faster for drawing rectangles, GDK or Cairo.

I wasn't sure, so I wrote a silly little test program.  Note that it's not quite apples-to-apples as the Cairo rectangles have variable transparency while the GDK rectangles are opaque.

On my box, running Gentoo Linux and the latest stable versions of everything, the Cairo version draws 500 rectangles in about 0.01 to 0.03 seconds, while the GDK version takes about 0.13 to 0.14 seconds.  So Cairo is faster.

To run this, just cut-and-paste into an editor window, save the file as rectangles.py, and run "python rectangles.py"

#!/usr/bin/env python

"""GTK rectangle drawing speed test, GDK vs. Cairo

David Ripton 2008-12-03
MIT license
"""

import time
import random

import gtk

NUM_RECTS = 500

handle_id = None

def main():
    window = gtk.Window()
    window.connect("destroy", gtk.main_quit)
    window.set_default_size(800, 600)
    vbox = gtk.VBox()
    window.add(vbox)
    area = gtk.DrawingArea()
    vbox.pack_start(area)
    gdk_button = gtk.Button("GDK")
    gdk_button.connect("clicked", on_gdk_button_clicked, area)
    vbox.pack_start(gdk_button, expand=False)
    cairo_button = gtk.Button("Cairo")
    cairo_button.connect("clicked", on_cairo_button_clicked, area)
    vbox.pack_start(cairo_button, expand=False)
    window.show_all()
    gtk.main()

def on_gdk_button_clicked(button, area):
    global handle_id
    if handle_id is not None:
        area.disconnect(handle_id)
    handle_id = area.connect("expose-event", on_area_exposed_gdk)
    area.queue_draw()

def on_cairo_button_clicked(button, area):
    global handle_id
    if handle_id is not None:
        area.disconnect(handle_id)
    handle_id = area.connect("expose-event", on_area_exposed_cairo)
    area.queue_draw()

def on_area_exposed_gdk(area, event):
    t0 = time.time()
    width, height = area.window.get_size()
    colormap = area.get_colormap()
    gc = area.get_style().fg_gc[gtk.STATE_NORMAL]
    for ii in xrange(NUM_RECTS):
        r = random.randrange(0, 65535 + 1)
        g = random.randrange(0, 65535 + 1)
        b = random.randrange(0, 65535 + 1)
        gc.foreground = colormap.alloc_color(r, g, b)
        x = random.randrange(0, width)
        y = random.randrange(0, height)
        w = random.randrange(0, width - x)
        h = random.randrange(0, height - y)
        area.window.draw_rectangle(gc, True, x, y, w, h)
    t1 = time.time()
    print "gdk drew %d rectangles in %f seconds" % (NUM_RECTS, t1-t0)

def on_area_exposed_cairo(area, event):
    t0 = time.time()
    cr = area.window.cairo_create()
    width, height = area.window.get_size()
    for ii in xrange(NUM_RECTS):
        r = random.random()
        g = random.random()
        b = random.random()
        a = random.random()
        cr.set_source_rgba(r, g, b, a)
        x = random.randrange(0, width)
        y = random.randrange(0, height)
        w = random.randrange(0, width - x)
        h = random.randrange(0, height - y)
        cr.rectangle(x, y, w, h)
        cr.fill()
    t1 = time.time()
    print "cairo drew %d rectangles in %f seconds" % (NUM_RECTS, t1-t0)

if __name__ == "__main__":
    main()

screen shot of the cairo rectangles


Rant: It is way too hard to post code with WordPress. First I tried the "code" button, but all my indentation was destroyed (bad for any code, fatal for Python). Then I tried the "b-quote" button; same effect. Then I switched to HTML mode and hand-inserted a "pre" tag, which preserved the indentation. But WordPress then proceded to vandalize my code with "smart" quotes. (Have you ever seen anything with "smart" in its name that actually was?) Luckily it was simple to find and install the wpuntexturize plugin, a few lines of PHP that eradicate Moron Quotes. But why the hell are they there in the first place, let alone enabled by default, let alone enabled by default inside a "pre" tag?

Programming
Python
Rant

Comments (0)

Permalink

AmpChat WxPython support

Stephen Waterbury added a WxPython client to my AmpChat example program. (Mentioned earlier in this post.)

His Mercurial repository is here.

This is still just example code, but now it's an example of multiple things:

  • how to use Twisted AMP
  • how to use Twisted with PyGTK (trivial because gtk2reactor rocks)
  • how to make WxPython's mainloop run in parallel with Twisted's using threads
  • how to write the same simple GUI program in both PyGTK and WxPython
  • how to use Mercurial to contribute to a project without commit rights in the original repo

Anyone want to contribute a PyQT or Tk version?

Programming
Python

Comments (0)

Permalink

Frustrating Python 2.6 warnings

I've been running Python 2.6 (released October 1) as the only Python on my Gentoo desktop for a few weeks. In most ways, it's the Best Python Ever. It's got per-user site-packages. It's got the multiprocessing module so you can use lots of CPU cores without thinking too hard. It's got abstract base classes. It's got binary literals. It's got ast.literal_eval, so you can safely evaluate static dicts without worrying about security. It's got other cool stuff that you can read about here.

But it constantly spews annoying deprecation warnings like this:

dripton@al ~/src/Slugathon/slugathon $ ./Connect.py
/usr/lib/python2.6/site-packages/twisted/internet/_sslverify.py:4: DeprecationWarning: the md5 module is deprecated; use hashlib instead
import itertools, md5

Or this:
dripton@al ~/src/projecteuler $ py.test test_euler86.py
===================================== test process starts =====================================
executable: /usr/bin/python (2.6.0-final-0)
/usr/lib/python2.6/site-packages/py/process/cmdexec.py:27: DeprecationWarning: The popen2 module is deprecated. Use the subprocess module.
import popen2
using py lib: /usr/lib/python2.6/site-packages/py

or even this:
dripton@al ~/src/Colossus-git/Colossus $ ant
/usr/lib/python2.6/site-packages/java_config_2/EnvironmentManager.py:15: DeprecationWarning: the sets module is deprecated
from sets import Set
/usr/lib/python2.6/site-packages/java_config_2/EnvironmentManager.py:15: DeprecationWarning: the sets module is deprecated
from sets import Set
/usr/lib/python2.6/site-packages/java_config_2/EnvironmentManager.py:15: DeprecationWarning: the sets module is deprecated
from sets import Set
/usr/lib/python2.6/site-packages/java_config_2/EnvironmentManager.py:15: DeprecationWarning: the sets module is deprecated
from sets import Set
Buildfile: build.xml
...
BUILD SUCCESSFUL
Total time: 5 seconds

If you're reading carefully, you'll notice one important fact: none of those warnings were in my code. I imported twisted, and got warnings because twisted used md5. I ran py.test, and got warnings because py.test used popen2. I ran ant, which last time I checked was a Java program, and got warnings because some Python script wrapped around the Java used sets.Set.

Of course, there's a good reason for these warnings. Python 3.0 is coming, and many of the features that trigger deprecation warnings in 2.6 will be gone in 3.0. (Well, no important functionality will truly go away, but where there are two ways to do something, sometimes they'll pick one and axe the other.) This is useful and important, but will cause pain. And the pain has already started.

Of course, I know how to silence the warnings. Temporarily:
dripton@al ~/src/Slugathon/slugathon $ python -Wignore Connect.py

Or permanently:
$ sudo vim /usr/lib/python2.6/sets.py
#import warnings
#warnings.warn("the sets module is deprecated", DeprecationWarning,
# stacklevel=2)

But it's annoying to have to do that. Especially when you didn't even realize you were running a Python program, as in the ant example above.

I predict lots of whining about this when 2.6 goes more mainstream (say, when it becomes default in Ubuntu).

Python

Comments (0)

Permalink

I'm no longer maintaining Colossus

I started Colossus in December 1997, almost 11 years ago. Yesterday I turned over leadership of the project to Clemens Katzer. That's a long time to maintain a software project, and it feels weird to me that I finally quit.

Colossus was my first large game project, my first large Java program, my first large GUI program, my first experience working on a complex game AI, and my first time managing a significant multi-programmer software project. Somehow it worked out pretty well, for the first few years. We had a pretty full-featured, stable Titan game, with somewhat dumb but working AI, and support for lots of cool variants. We had a rotating team of up to half a dozen developers at a time working on the game, and it was fun.

But then I added network play, and I didn't do a very good job. At the time there were no Java remote method libraries that had the features we needed, so I wrote everything myself with a simple string socket protocol, with lots of receiver threads. But I wasn't rigorous enough with thread synchronization, which meant walking the line between deadlock and corrupted game data. And gaining acceptable performance meant that a lot of logic that existed on the server side needed to be reproduced on the client side, but excessive coupling in the code base meant this couldn't be done cleanly and a lot of code got duplicated. I tried and tried to fix the mess, but couldn't put Humpty-Dumpty together again.

I finally decided to start over, in a better programming language (Python), with a better networking framework (Twisted), with a better overall design paradigm (game events flowing from the server to the client, using the Observer pattern to reduce coupling and allow complete reuse of the core game logic between the client and server) and with safe sane single-threaded code. Thus was born Slugathon. Unfortunately, as a new father I didn't have nearly as much free time as I did back when I started Colossus, and so Slugathon still isn't finished.

With Slugathon unfinished I felt obligated to keep maintaining Colossus, but I didn't really have my heart in it. I figured my job was to keep the project alive until a successor showed up to take it over. We had several guys on the team who were technically qualified to take over, but none seemed likely to put in the necessary amount of time on a consistent basis. Then Clemens joined, and not only submitted code for his pet features (the fun part that everyone wants to do) but started watching the bug tracker like a hawk, and talking to users about their bugs, and making special test builds so that users could test that their obscure hard-to-reproduce bugs were fixed, and adding documentation of which features went into which releases, and all the other not-fun crap that 90% of small volunteer open source projects can't find anyone to do. Around the same time, longtime contributor Peter Becker did a giant refactoring that reduced the amount of code duplication from disgusting to merely gross. And Clemens switched the server side from listener threads to NIO, reducing the total thread count from insane to merely scary.

Which meant I was no longer needed on Colossus. So I'm committing to releasing a stable, network-playable version of Slugathon by the end of 2008. And hoping that the Colossus team can continue cleaning up the mess I left them. Hopefully someday soon we'll have two stable networked Titan games instead of zero.

(By the way, I played a couple of four-human 2-AI networked Abyssal6 Colossus games this morning. It was great fun, even though both games had technical difficulties.)

Games
Programming
Python

Comments (0)

Permalink

Simple Linux backup solution

My desktop computer, which runs Gentoo Linux, has a 120 GB drive (used for my / and /boot and swap partitions) and a 480 GB drive (used for /home).  I decided to add another drive for backups.  Ended up with a 720 GB drive, which is bigger than the others combined with room to spare, and only cost me about $100.

(Backing up to another hard drive on the same box is easy, and fast, and protects from a single drive failing, but does not protect against the computer being stolen or the house burning down.  So I still need to push really important stuff, like source code, offsite.)

I mount the new drive as /mnt/backup.  I use rsync to backup everything, minus a few excluded directories, to /mnt/backup/{TIMESTAMP}.  I tell rsync to use hard links when a file is the same as the file in the previous backup.  That way I have a full backup tree per day, but the incremental hard disk space used is only equivalent to the size of the files that actually changed that day.

I control the backup with this Python script (sh or perl would work too, but I like Python), which lives at /usr/local/bin/backup.py:

#!/usr/bin/python

"""Backup most of a filesystem to a backup disk, using rsync."""

import subprocess
import time
import os

SOURCE = "/"
DESTINATION = "/mnt/backup"
EXCLUDES = [
    "/proc",
    "/sys",
    "/lost+found",
    "/mnt",
    "/media",
    "/tmp",
    "/var/tmp",
    "/var/run",
    "/var/lock",
]
RSYNC = "/usr/bin/rsync"

def find_latest_destdir():
    latest = 0
    for fn in os.listdir(DESTINATION):
        if fn.isdigit() and len(fn) == 14:
            timestamp = int(fn)
            latest = max(timestamp, latest)
    if latest:
        return str(latest)
    return None

def main():
    cmd = [RSYNC]
    cmd.append("-ab")
    for exclude in EXCLUDES:
        cmd.append("--exclude=%s" % exclude)
    latest = find_latest_destdir()
    if latest:
        cmd.append("--link-dest=%s" % (os.path.join(DESTINATION, latest)))
    cmd.append(SOURCE)
    timestamp = time.strftime("%Y%m%d%H%M%S")
    cmd.append(os.path.join(DESTINATION, timestamp))
    print cmd
    returncode = subprocess.call(cmd)


if __name__ == "__main__":
    main()

And here's the crontab line that runs it every night at 3 a.m, added with "sudo crontab -e":

0 3  * * *      /usr/local/bin/backup.py

Automated nightly backups are now so easy (thanks to tools like rsync, and the cheapness of hard drives) that there's really no excuse not to do them.

Linux
Python

Comments (0)

Permalink

Another Python benchmark

The previous benchmark was too fast, and mostly only measured startup time. Here's a slower one. Project Euler Problem 10

Basically, find the sum of all primes up to a million. Best of 5 runs for each Python version.  Basically the standard Sieve of Eratosthenes in a loop.

python2.4 13.131s
python2.5 12.67s
python2.6 11.517s
python3.0 13.764s
jython2.2.1 11.542s
pypy-c (r56024, translate.py –gc=hybrid –thread targetpypystandalone –faassen –allworkingmodules), 12.774s
python2.5 with psyco: 1.926s
python2.5 using gmpy.is_prime 2.441s
python2.5, psyco, gmpy.is_prime 1.762s

So on this one all the Python versions are pretty close, with a nice steady improvement from 2.4 to 2.5 to 2.6. Jython and pypy are both in the same ballpark.  psyco makes it run about 6 times as fast. Using gmpy's is_prime function (written in C) instead of my Python sieve has a similar result.

Python

Comments (0)

Permalink

Quick Python version speed comparison

Python 2.6 beta 1 and 3.0 beta 1 were both released Wednesday. This gave me the urge to run a quick speed comparison on all the versions of Python installed on my Linux box.

The program in question is my solution to Project Euler problem 100. (Source code not provided, because that would be a spoiler.)

$ time python2.4 euler100.py > /dev/null

real 0m0.010s
user 0m0.006s
sys 0m0.005s

$ time python2.5 euler100.py > /dev/null

real 0m0.013s
user 0m0.010s
sys 0m0.003s

$ time python2.6 euler100.py > /dev/null

real 0m0.012s
user 0m0.009s
sys 0m0.003s

$ time python3.0 euler100.py > /dev/null

real 0m0.024s
user 0m0.021s
sys 0m0.004s

$ time pypy-c euler100.py > /dev/null
debug: WARNING: library path not found, using compiled-in sys.path

real 0m0.042s
user 0m0.009s
sys 0m0.033s

$ time jython euler100.py > /dev/null
Traceback (innermost last):
File "euler100.py", line 58, in ?
File "euler100.py", line 49, in main
OverflowError: float too large to convert

real 0m3.372s
user 0m2.707s
sys 0m0.092s

Executive summary: Python 2.4 through 2.6 are all about the same speed, for this test. 3.0 is about twice as slow. PyPy is four times as slow. And Jython is way slow and blows up converting big numbers.

Of course, this is just one data point. I should scale this test up to run all my Project Euler solutions that don't rely on external libraries. (It would probably take significant work to get libraries like gmpy to work with 3.0 or PyPy.)

Python

Comments (0)

Permalink

Installing goocanvas on RHEL 5

We have some code at work that uses the old GnomeCanvas. I wanted to update it to a Cairo-using canvas to get antialiasing and better font scaling. Looked at CrCanvas, FooCanvas, GooCanvas, and HippoCanvas. Decided to use GooCanvas, based on docs and examples and Python bindings and license.

Unfortunately, the latest GooCanvas (0.10) wants Cairo 1.4, and Red Hat Enterprise 5 (which we have to use at work) is stuck at Cairo 1.2, and half of the Gnome stack seems to depend on Cairo.

The solution was pretty simple: use GooCanvas 0.9. Feels weird to deliberately install an old version of something, but when you're running Red Hat, sometimes you just have to pretend it's still 2006.

After a few hours of working with GooCanvas (from Python), I'm pretty happy with it. The install went well except that it put the Python libraries in /usr/local/lib/python2.4 instead of /usr/lib/python2.4 by default. I got a bit confused by SvgSurface (which is for output not input), but ended up using an SVG example that comes with 0.10 to make a custom SVG-based component. (The alternative was converting SVGs to PNGs on startup and then every time the scale was changed, which would have worked but seemed a bit gross.)

I see that gtkmm (C++ bindings for GTK+) is now using Cairo behind the standard GTK DrawingArea widget by default, so you don't need to scrounge up a third-party Cairo canvas. Great idea. I hope it spreads to the other GTK+ bindings.

Linux
Python

Comments (0)

Permalink

Twisted AMP chat example

I wrote a simple little PyGTK chat program a couple of months ago, to learn the Twisted AMP protocol.

Someone just asked about bidirectional AMP on the Twisted mailing list, which reminded me that I should publicize this example a bit.

It's in a Mercurial repository (browseable over the web if you don't want to bother with Mercurial) at
http://ripton.net/hg/ampchat

Programming
Python

Comments (1)

Permalink