home
> blog
>> 638

translate to German (by SYSTRAN)

Weblog RDF feed, Atom feed

Fri Jun 25 04:25:54 2010 GMT: Insomniac

Sun Jun 20 19:09:34 2010 GMT: netbsd 5.1rc on Linux KVM

Sun Jun 20 19:04:48 2010 GMT: Giant PyXMPP Memory Leak

Shortly after last week's update of edge.cmeerw.net it became apparent that there was a giant memory leak somewhere in PyXMPP. Interestingly, when JabRSS was running on Ubuntu 8.04, it only leaked a few MB per day (and I had actually hoped that the upgrade would improve things), but after the upgrade, the leak increased about tenfold to more than 100 MB per day.

Looking at how PyXMPP has been implemented, the likely culprit appears to be the python libxml2 library (or the way it's being used). As debugging these things is extremely difficult, I finally decided to glue together my own little XMPP library based on Python's ElementTree. Yes, it does smell like not-invented-here syndrome, but I am a bit fed up with the quality of PyXMPP (although it looked fine on the surface) and did't want to take the risk of trying another Python XMPP library (xmpppy appears to be the other alternative).

When developing an XMPP library, although it's largely just XML, there are two things that make it a bit more complicated than it should be.

The first issue is that you need to be able to parse the XML stream incrementally, but ideally you will still want to have some DOM-like interface. PyXMPP had to implement some really ugly hacks to get that part working - ElementTree is a lot better in this respect.

The second issue is that the XMPP spec is quite strict on how namespaces should be represented in the outgoing XML stream. This again caused huge problems when you try to use a standard XML library - and ElementTree isn't ideal for this use case either, but it's not as bad as the hacks required for libxml2.

To sum it up, JabRSS is now already running with my new (and extremely lightweight) xmpplify library. I am sure, there are still a few rough edges around, but at least memory usage appears to be stable so far.

BTW, one final note: establishing the XMPP connection (including authentication) appears to be quite complex as well with around half a dozen messages required.

Wed Jun 16 06:03:37 2010 GMT: Free INGATE VServers

Maybe I am already too late to enter the free VServer giveaway, but let's see... (update: looks like I missed it)

Sat Jun 12 20:56:11 2010 GMT: Running Ubuntu Lucid Lynx with Kernel 2.6.26

I had hoped to be able to upgrade my virtual server to Ubuntu Lucid Lynx today. It started quite well as the "apt-get dist-upgrade" ran without any problems, but after the first reboot, the machine didn't come back up. A quick search on the net revealed that Lucid Lynx doesn't support any older kernels (in this case, my virtual server hoster provides Debian's stable 2.6.26 kernel).

So what do you do with a server that refuses to boot and you only have minimal access to the server? Clearly, there must be a way to work around this.

It turns out that the main issue is that some packages used during the boot process (mountall, plymouth, udev) rely on syscalls that are only available on more recent kernels: mainly pipe2 and the SOCK_CLOEXEC flag for the socket syscall. Fortunately, these can be mapped to the standard pipe syscall and the SOCK_CLOEXEC flag can simply be stripped away using a LD_PRELOAD library:

#define _GNU_SOURCE
 
#include <unistd.h>
#include <dlfcn.h>
#include <sys/socket.h>
#include <stdio.h>
 
int (*socket_fwd)(int, int, int) = NULL;
 
static void __attribute__ ((constructor)) init()
{
    socket_fwd = dlsym(RTLD_NEXT, "socket");
}
 
int pipe2(int pipedes[2], int flags)
{
    return pipe(pipedes);
}
 
int socket (int domain, int type, int protocol)
{
    return socket_fwd(domain, type & ~SOCK_CLOEXEC, protocol);
}

Compiling that into a shared library and modifying the "exec mountall" call in /etc/init/mountall.conf as follows

mount -o remount,rw /dev/hda1 /
LD_PRELOAD=/liblucidcompat.so /sbin/mountall $force_fsck $fsck_fix
ifup lo

seems to somehow "work around" the problem - well, there are still a couple of errors, but it appears to boot without any major problems. I am sure some hotplugging functionality won't work now, but I don't expect to see any hotplugging on a Xen virtualised server anyway.

BTW, don't try this at home - you could end up with a broken system...

Sun Jun 06 11:29:19 2010 GMT: JabRSS Migration to PyXMPP

I have just moved JabRSS to use PyXMPP instead of PyXPCOM/JabXPCOM/jabberoo. While the previous set up worked quite well, it got incresingly difficult to get PyXPCOM built stand-alone, so moving to PyXMPP will hopefully simplify things as it comes as a standard package for Debian/Ubuntu.

So far, the service appears to work as usual, but only time will tell if there are any hidden problems...

Sat May 29 13:35:22 2010 GMT: PyXMPP Bug

I have just been playing a bit with PyXMPP and already found a critical bug. Fortunately, it should be quite easy (almost trivial) to fix it...

Sun May 02 09:47:48 2010 GMT: IPv6 Reverse DNS

Fri Apr 30 21:02:57 2010 GMT: Native IPv6 Connectivity

Wed Apr 14 08:04:14 2010 GMT: Fix for Berkeley DB's SQLite Concurrency Issue

Thu Apr 08 19:45:38 2010 GMT: Concurrency Issue in Berkeley DB/SQLite

Sun Apr 04 21:21:18 2010 GMT: Ubuntu Lucid Lynx