waf-devel

Escaping the Google cave

2012-01-28T05:12:00.000-08:00

1. The problem with user tracking

A few years ago, a user complained that the Waf project was using Googlecode, and threatened to not use Waf if it remained hosted on the Google server. I thought that it was paranoid at that time, and I just forgot about that request...

Now a few years have passed, and it is a bit late to move to Github. Also, tons of websites are now hosted by Google, and it would be impossible to avoid all of them. But this is not my main concern. Rather, I have taken the bad habit of logging in on my Google or Facebook accounts more often than I cleared all my cookies and I am getting targeted content too often.

For instance I started to notice that it was much harder to obtain information from Google search. I would frequently find Python in all my search results. Searching for Java programming techniques would lead me to more Python sites. Searching for Scons or CMake would only lead me back to Waf. I would also get ads related to the contents of my emails. In other words, the Google tracking had started to create a convenient place where I would always find familiar information.

Since I was a child, I have always had the important feeling or belief that there exists a world independent of me, a reality that is worth exploring (solipsists may disagree). It is a virtue to try to know the world as it is and not as one would like it to be. The web is interesting because it gives an opportunity of getting other views easily, and to explore a world that is not limited or bound by a particular view, and I would like to keep it this way.

2. Filtering tracking websites

First of all, I believe that the "do not track" cookie is one of the most idiotic invention created recently. If I were creating a website and if someone set that cookie, I would really love to track that someone in the sneakiest way possible. This is equivalent to wandering around with a big sign reading "kick me" stuck on your back.

One of the first things I have tried is to avoid tracking by blocking the scripts that report the pages that I am visiting, for example google analytics. For this it is simple to edit the file /etc/hosts:

127.0.0.1 www.google-analytics.com
127.0.0.1 google-analytics.com
127.0.0.1 ssl.google-analytics.com

There are many more addresses to exclude however, and it does not prevent google from reading your mail. It is enough to log in once to Googlemail to get targeted ads and personalized contents again.

The filtering approach is also imperfect, for example, if it becomes widespread, a few websites will start breaking if the tracking is blocked. It will be easy to test in javascript what hosts are blocked to create a fingerprint of the user. This goes back to the principles of information theory, if you have a secret, it will leak eventually however hard you try to keep it.

3. Setting up multiple identities

Trying to filter the websites is just too complicated to do, and removing http cookies, flash cookies, visited pages, website preferences and user-agent is just too much of a hassle. Websites may also try cache timing attacks to get more information on you anyway.

Tor is nice but limited in terms of feature and speed (no flash, use the Tor browser, etc). Virtual machines are convenient but use a lot of resources, for example flash and javascript are too slow to be usable. I keep them for untrusted websites (and with flash disabled anyway).

The best success I have had so far is by setting up multiple Linux user accounts and multiple identities. I keep my current account for normal stateless activities, and use the other accounts for stateful operations. For example, I created a user account named "google" for all googlemail and googlecode-related activities:

# useradd google -p users
# mkdir /home/google
# echo "export DISPLAY=:0" >> /home/google/.bashrc
# chown google /home/google/.bashrc /home/google

The current user account must allow the windows for each other user accounts to be displayed on the current Xorg session. Make sure to always use xhost +local:accountname:

echo "xhost + local:google" >> ~/.profile

To obtain any sound, it is necessary to tweak the pulseaudio settings. First, the file /etc/pulse/default.pa must be copied to ~/.pulse/default.pa and modified to allow connections from other user accounts:

> diff -urN /etc/pulse/default.pa ~/.pulse/default.pa 
--- /etc/pulse/default.pa       2011-10-30 03:59:03.000000000 +0100
+++ .pulse/default.pa   2011-12-01 00:34:34.537118644 +0100
@@ -158,3 +158,6 @@
 ### Make some devices default
 #set-default-sink output
 #set-default-source input
+
+load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1
+

Then a new file must be added to each other user account, for example /home/google/.pulse/client.conf:

default-server = 127.0.0.1

After that, a web browser can be started easily

sudo su
su - google
firefox

To make certain that I do not confuse the accounts (the web browser completion already helps a lot), I am also using different web browsers with different versions, with different extensions (noflash, firebug, etc), with different themes and with different language settings. For example, looking at ads in Serbian or in Russian is fun. The firefox themes (Personas) even have animations to help remember what window is what.

Happy new year Debian

2012-01-05T11:53:00.000-08:00

The Debian folks have actually started a crusade against projects using Waf. The archives containing the Waf file are now rejected. I can of course support such a policy, after all, shipping a C file generated by Bison or Flex without the grammar source file is similar to distributing proprietary software. I do not question this.

But the binary file in question is a perfectly legal python script containing additional source code in a compressed form. There is no state machine, no obfuscation of the contents, and the source is not modified. This has nothing to do with minified javascript since the variable names and the code structure are not modified.

It may be difficult to inspect the file in question, but then how about the 1MB configure scripts that are redistributed with most Autotool-based projects? The configure file is machine-generated, it contains fragments of original source files, and make it dead easy to conceal malware. If the policy were applied properly (ensure security and prevent machine-generated contents), then Autotool-based packages containing a configure script should be removed as well.

There are very good reasons of shipping the Waf file with the projects, for example one is to provide the users with a way to discard installation and interpreter issues. This is exactly what the configure script does. Then, the workaround suggested by a Debian folk consists in unpacking the Waf file, which means that people have the choice of getting their projects removed from Debian or to cripple their packages by removing support for compilation with Python 2.5, 2.4 and potentially 2.3.

Was not Debian supposed to promote and help open-source projects without making a discrimination?

Or maybe it is that some people have too much time on their hands and cannot break critical elements of the system anymore.

So, how do you get the KDE source code?

2011-12-25T10:23:00.000-08:00

I finally found some time to start working on my personal application Semantik again. I started to work on Waf a long time ago to simplify the development of KDE applications.

I had not checked out the KDE website for a while, and it has apparently changed quite a lot. It was probably designed by usability experts, judging by the blue-on-blue color scheme. This may irritate the eyes, but trap of web links will fry your brain. Here is for example the links I have followed when looking for the latest/development source code:

The little information contained on the website is hidden very well. At this point I lost hope to get the development code and I opted for the released code. The download button is of course called "get kde software". The source code is only a few clicks away!

The kde.org website was clearly made to maximize user clicks. The only thing it is missing at this point is ads with fake download buttons (do not click on this link by the way).

The Gnome website is only slightly better. It reminds me of the apple website which I used when I needed a mac mini. There are lots of corporate fonts and pictures with rounded corners, so that at least I can understand that I am reading the wrong page if I am a developer. Trying to attract developers by pretending they can find love also had the opposite effect on me.

Why cannot those developer websites imitate the design of the Samba website instead? The information is displayed clearly and features virtually no link traps.

Python 3 is not ready for production

2011-12-17T01:03:00.000-08:00

What do you get when you try to install python3.2.2 on Linux?

> python3.2
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python 3.2.2 (default, Dec 17 2011, 10:04:46) 
[GCC 4.6.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
File "/etc/pythonstart", line 5, in <module>
import atexit
ImportError: No module named atexit

It is the first time I am hear the PYTHONHOME variable is necessary. So what happens if we set it?

> export PYTHONHOME=/usr/local/lib64/python3.2/
> python3.2
Fatal Python error: Py_Initialize: Unable to get the locale encoding
LookupError: no codec search functions registered: can't find encoding
Aborted

Alright, encoding issues. Python 3 does not work on Linux anymore. Let's try to uninstall it?

> sudo make uninstall
make: *** No rule to make target `uninstall'.  Stop

And now the broken files must be removed by hand. I guess I will wait for Python 3.2.3, after all threading is broken in both python 2.7.2 and 3.2

wscript file headers

2011-07-03T04:52:00.000-07:00

A few users have asked why the wscript files usually include the following header if they are not meant to be executed directly:

#! /usr/bin/env python
# encoding: utf-8

Several benefits have been observed over the time, and it appears that this header is a best practice:

Most text editors - especially on unix-like systems - will read the first line beginning by #! to apply proper syntax highlighting. Your text editor may well display the file properly, but others may not.
If the wscript files are ever meant to be executed, then:
1. The process will not hang when called as ./wscript (the 'import' program takes screenshots)
2. The /usr/bin/env will redirect the interpreter to the adequate python executable (python can be installed in /usr/local or /opt for example)
The encoding declaration will prevent special characters such as €, é or í from causing errors

When a wscript file is created for the first time, it may have to be reloaded to force the text editor to take the file type into account. In Vim at least, a mapping can be added to save the hassle of re-opening the file:

au BufNewFile,BufRead wscript* set filetype=python

Running Waf on different Python interpreters

2011-05-22T04:47:00.000-07:00

IronPython 2.7 now provides a subprocess module, and the good news is that Waf will be able to run on it. The changes are only in Waf trunk for now, they will be available in Waf 1.6.5 once a remaining issue is resolved (the logging module behaves differently on IronPython).

A few of you may be curious to know how well Waf performs on different python interpreters. Different propjects may have different requirements, so the tests are always biased in one way or another. Among all the tests considered so far, the genbench script provides a good balance between project size, process execution, regexps and parsing. The numbers collected represent the best run in in five executions:

./genbench.py /tmp/build 50 100 15 5
cd /tmp/build
waf clean; time waf build -p -j3

On Windows XP, the IronPython runtime is fairly close to the Pypy one. And Pypy 1.5 did not complete normally (file descriptor leak).

Windows XP
cPython 2.7	IronPython 2.7	Pypy 1.5
1m54	2m14	2m05

IronPython requires .NET 4, and it would not run on Linux using Mono 2.10.2. It is no surprize that the test completes faster as spawning new processes (fork) is slower on Windows. Pypy 1.5 could not even complete normally once in about 10 runs.

Linux
python 2.6.5	python 3.2	jython 2.5.2
1m41	1m41	5m30

For the moment, the best interpreter remains cPython for both portability and performance.

Debugging tools

2011-04-30T03:29:00.000-07:00

Trying to trace or to debug a build almost always incurs a performance penalty. The most important tool is included by default but only enabled when calling waf with the verbose option (waf -v). Others are present in the waflib/extras folder and are not included by default. They can be added to the waf file or downloaded by calling "waf update --files=parallel_debug,why"

The module "why" provides some more output to help understanding why files are being compiled. It requires a fresh recompilation though, and inflates the cache files significantly as it stores some data between the builds. The output will display the task signatures and the reason of the execution:

12:26:20 /comp/waf/demos/c> ./waf build --zones=task
Waf: Entering directory `/disk/comp/waf/demos/c/build'
12:26:21 task -> Task must run                      : c3490b5e8498da72aef5a9f645d72276 39da3e173d11cdbb273cda14856dbb6a
12:26:21 task -> * Source file or manual dependency : 3d3fde4f5fc0124efdba8e94e1d216f8 5e4ef70cc1618412ef9975baab0157c1
[12/15] c: program/main.c -> build/program/main.c.1.o
Waf: Leaving directory `/disk/comp/waf/demos/c/build'
'build' finished successfully (0.089s)

For performance debugging, the module parallel_debug creates pictures representing when tasks are executed during the build. The diagrams show how well the build can be distributed, and therefore how fast the build can be. Generating the diagrams incurs a minor performance penalty, which is the reason why this tool is not included by default in the waf file.

It has been updated recently to improve the layout, now tooltips are present in the svg output file. Follow the links to consult the diagrams of the Samba 4 developer build for 2 processors and for 8 processors.

Colors in cmd

2011-04-30T00:37:00.000-07:00

The cmd window looks terrible compared to rxvt/xterm ones, and unlike others, cmd is unable to interpret ANSI escape codes. The module ansiterm has been been bringing happy colors in cmd and other consoles since 2008. It wraps sys.stdout/sys.stderr in an object which intercepts the escape codes to provide the coloring by using a few functions from the windll module.

The progress bar, which was broken for a long time, is even displayed properly (waf 1.6.4):

The display in msys it also known to work in waf 1.6:

Right now, one issue remain in Windows Vista. The platform-specific and underdocumented functions from the module windll make it very difficult to understand and fix the problems though

windll.kernel32.FillConsoleOutputCharacterA

New python projects resembling ansiterm.py have appeared recently, here are for example:

* Colorama
* Colorconsole

Both also split the small amount of code into several files, which make the re-use only more difficult, and neither seems to support clearing single lines, which is mandatory for displaying progress bars properly.

Python 3.2, and the build system kit

2011-01-08T08:50:00.000-08:00

The new "concurrent.futures" module from Python 3.2 will now make it even easier to execute tasks concurrently. We never miss an opportunity to remove some code so we had to try it, some functionality being similar to the contents of the Runner module.

First, we could not remove a much code as we hoped: fewer than 20 lines, most of the code being in the synchronization and the exception handling anyway. Then we met a surprising performance degradation linked to the creation of circular references.

Finally, the Python executor appeared to increase the builds by a few seconds on our build benchmarks, so we finally gave up.

The executor API looks good though, and many developers will be tempted to try to create new python-based build systems from it. Executing tasks represents only a very small part of what one expects in a build system though:

A system for handling commands and command-line options
A system of (task) order and dependencies
A handler for exceptions and user errors
A usable system (a very difficult problem)
An extension system for new programming languages
A portable system (not limited to Python 3.2)
Adequate performance (it is so easy to over synchronize or to write bad algorithms)

Developers seem to underestimate the amount of work required to make a system work for more than one or two projects (Waf has had nearly 11000 revisions in 6 years). Specialized and incomplete build scripts are still popping up. Just to name a few python-based ones:

The fragmentation seems unavoidable, and the situation is probably going to be similar to that of operating systems or programming languages, with a few big players and tons of niche projects.

If it must happen, then let it happen properly, at least. Starting in version 1.6.2, the Waf framework features a new build system kit which enables the creation of new build systems. By re-using the tested and maintained components from the Waf framework, much more time can be spent on more interesting problems such as creating an intuitive XML/YAML/JSON schema (if you believe that IDEs should edit your scripts) or creating a domain-specific programming language (make-like, cmake-like, ... ?), or extracting commands and dependencies to create derivated files (Makefiles, Visual studio, ...)

The few examples illustrate the possibilities offered by the Waf framework:

overview: Create a file using the Waf components to perform simple builds that do not use wscript files
parser: Add a parser for a domain-specific language, and create a file able to build targets from specific files
noscript: Create a file able to build targets without using a script file, finding the source files from header dependencies
nostate: Create a make-like build system that does not use build files (timestamps only)
makefile_dumper: Use the Waf API to dump the build decription into a makefile

Have a look and think of the kittens before trying to start a build system from scratch.

Make your own build system with the Waf framework

2010-12-05T10:11:00.000-08:00

Are you dissatisfied with Python? Do you prefer XML, YAML or JSON for build scripts? Or maybe you think you can create a better domain-specific language for build scripts? Before trying to invent new build system from scratch, try re-using the Waf framework.

Waf 1.6 features a new customization system for the Waf files, which means that the Waf libraries can be re-used easily without forcing the use of wscript files. Here is how to create a waf file that will print "Hello, world!" and stop:

waf configure 
  --prelude=$'\tprint("Hello, world!")\n\tsys.exit(0)\n'

The resulting waf file will then exhibit the following lines of code:

if __name__ == '__main__':
    print("Hello, world!")
    sys.exit(0)
    from waflib import Scripting
    Scripting.waf_entry_point(cwd, VERSION, wafdir)

Writing code in the prelude section is a bit complicated, but the execution can be delegated to a separate waf tool. For example, with a tool named bbdlib featuring a function named "start":

./waf-light configure build  --tools=$PWD/bbdlib.py 
  --prelude=$'\tfrom waflib.extras import bbdlib\n\tbbdlib.start(cwd, VERSION, wafdir)\n\tsys.exit(0)'

The file bbdlib.py will be included in the resulting waf file as waflib/extras/bbdlib.py, so the resulting code has to use an import. Also, the original parameters (current working directory, waf version and waf directory) are propagated as they can be useful in the execution:

if __name__ == '__main__':
    from waflib.extras import bbdlib
    bbdlib.start(cwd, VERSION, wafdir)
    sys.exit(0)
    from waflib import Scripting
    Scripting.waf_entry_point(cwd, VERSION, wafdir)

Here is for example a script that may be used to read files named "bbit":

import os, sys, imp
from waflib import Context, Options, Configure, Utils, Logs

def start(cwd, version, wafdir):
    try:
        os.stat(cwd + '/bbit')
    except:
        print('call from a folder containing "bbit"')
        sys.exit(1)

    Logs.init_log()
    Context.waf_dir = wafdir
    Context.top_dir = Context.run_dir = cwd
    Context.out_dir = os.path.join(cwd, 'build')
    Context.g_module = imp.new_module('wscript')
    Context.g_module.root_path = os.path.join(cwd, 'bbit')
    Context.Context.recurse = lambda x, y: getattr(
         Context.g_module, x.cmd, Utils.nada)(x)

    Context.g_module.configure = lambda ctx: ctx.load('g++')
    Context.g_module.build = lambda bld: bld.objects(
        source='main.c')

    opt = Options.OptionsContext().execute()

    do_config = 'configure' in sys.argv
    try:
        os.stat(cwd + '/build')
    except:
        do_config = True
    if do_config:
        Context.create_context('configure').execute()

    if 'clean' in sys.argv:
        Context.create_context('clean').execute()
    if 'build' in sys.argv:
        Context.create_context('build').execute()

Here are a few points to keep in mind:

A module simulating a wscript file is created dynamically to simulate the execution of a top-level wscript file. This script attempts to re-use as much code as possible, but some configuration tests illustrate how to bypass all restrictions.
Although a few variables are shared by several classes (Context.top_dir), the initialization is not required. One might want to subclass the BuildContext class and start with a nearly empty script.
The commands configure, clean and build are implemented for the illustration. In practice it may be easier to create command subclasses for specific purposes.

The complete code source for this example can be found in the custom build section of the waf source distribution. If you start writing a Waf-based build system, feel free to notify us in the comments!

Top-level source files

2010-12-05T06:35:00.000-08:00

A really weird scenario was discussed on IRC recently: declare a C++ program by referencing only the main.cpp file, and have the build system add other source files by looking at the includes. For example:

main.cpp includes foo.h
foo.h has a corresponding foo.cpp file
foo.cpp includes a bar.h file
the file bar.h has a corresponding bar.cpp file

The program to be built declares main.cpp as unique source file, but the build system must detect and add the files foo.cpp and bar.cpp automatically.

The request was a bit surprising, because in large projects it is usually the norm to declare the source files explicitly to avoid confusing other developers. Even globs are used rarely in practice, as they conceal the exact source files to use.

For a single developer working on his project it probably does not matter anyhow, so an example was added in the playground area. Here is the link to the example.

Code generators and unknown files

2010-11-25T04:54:00.000-08:00

Code generators may lead to complicated scenarios, especially when they create files which are not known in advance. The usual process consists in adding specific task classes which will read the files after the execution, create the corresponding node objects, and update the node signatures.

When the generated files are kept in the source directory (and redistributed with the source code), a more simple (and less correct) solution is still possible. In the following example, a code generator will create C files in the folder named "generated":

def options(opt):
    opt.load('compiler_c')
def configure(cnf):
    cnf.load('compiler_c')
def build(bld):
    from waflib import Build
    bld.post_mode = Build.POST_LAZY

    def create_files(tsk):
        out = tsk.generator.path.make_node('generated')
        from waflib import Utils
        for x in out.ant_glob('*.c'):
            x.delete()
        import random
        for x in range(2):
            num = random.randint(0, 2**31)
            k = out.make_node('test%d.c' % num)
            k.write('int k%d = %d;' % (num, num))
    bld(rule=create_files, source='wscript', name='codegen')

    bld.add_group()

    bld.program(features='find_them',
          source=['main.c'], target='app')

from waflib.TaskGen import feature, before
from waflib import Utils
@feature('find_them')
@before('process_source')
def list_the_source_files(self):
    self.source = Utils.to_list(self.source) +
         self.path.ant_glob('generated/*.c')

The first rule create_files will remove any C file from the generated directory, and add new ones with random names. The generated files, and main.c will be compiled into a program, but only after the files are correctly created.

The POST_LAZY scheme will force the build groups to be processed sequentially, so the total of tasks to execute will be incorrect. For example, the progress bar will reach 100% each time a build group is processed. For example, the task counter will jump from 1 to 5:

12:52:14 /playground/codegen> waf clean build
'clean' finished successfully (0.005s)
Waf: Entering directory `/playground/codegen/build'
[1/1] codegen: wscript
[2/5] c: main.c -> build/main.c.1.o                                                                                                                                             
[3/5] c: generated/test2017594948.c -> build/generated/test2017594948.c.1.o                                                                                                     
[4/5] c: generated/test277184706.c -> build/generated/test277184706.c.1.o                                                                                                       
[5/5] cprogram: build/main.c.1.o build/generated/test2017594948.c.1.o build/generated/test277184706.c.1.o -> build/app                                                          
Waf: Leaving directory `/playground/codegen/build'                                                                                                                
'build' finished successfully (0.235s)

The feature find_them is will complete the program source list after code generator is executed, as adding source=bld.ant_glob('**/*.c') would only read old files.

The code source for this example has been added to the playground folder.

Order in archive and compression ratio (2)

2010-09-23T11:43:00.000-07:00

This is a follow-up on the previous entry on observing the results of changing the file order when creating compressed archives. Shuffling the files before adding them to the archive and compressing will give files of different sizes. The following diagrams represent the distributions of the file sizes after creating 560000 archives randomly.

The distribution for tar.gz files seems to follow a normal law. On this test, the gap between the smallest file and the biggest file is about 4%:

The distribution for bzip2 files is dependent on the input files. Different files will produce different curves. Also, the gap between the smallest and biggest files was found to be less than 0.5%:

The amount of bytes of the files compressed using xz (tar cJvf) is an even number (69700, 69702, 69704, ...). This is causing the discontinuity in the following curve (and the optical illusion). Also, the gap between the smallest and biggest files was more than 0.9%.

Changing the file order in the archive does change the size of the compressed archives, but the question that comes naturally is "by how much?". The amount of files and the compression speed make it a bit difficult to perform an exhaustive search (67!). To make a oneself a better idea of the bounds, using a local search such as simple hill-climbing search is therefore a good idea. Here are the results after compressing about 50000 files:

Algorithm	Min size	Max size	Gap
gzip	78868	86856	10%
bzip2	69818	70227	0.5%
xz	68876	70248	2%

On the bzip2 search, the hill climbing search did not increase the bounds substantially. This suggests that the search got trapped in local minimums, which is not really surprizing given the shape of the distribution. Since Waf uses bzip2 compression, it will be interesting to try another kind of local search to improve the bounds, and try to understand what makes some archives more compressible than others.

Order in archive and compression ratio (1)

2010-09-09T12:54:00.000-07:00

The following example was recently added to the Waf directory to experiment with lots of tasks created by make-like rules. The tasks from the example will perform the following operations:

Create compressed archives from the same files taken in random order (shuffled)
Measure the compressed file sizes, and add the values into a main data file
Compute the file size distribution from the main data file
Create a gnuplot script to represent the distribution
Use the gnuplot script to create pictures

The files used in the example are the python files from Waf, and the archives are created in the tar format, which are compressed by gzip or bzip2. After creating a bit more than 300000 compressed files, the distribution of the gzip files will look like the following:

The file sizes range from 80282 bytes to 83605 bytes. A significant file reduction can then be obtained by carefully changing the input file order.

For the same number of compressed files, the distribution of the bzip2 files will be the following:

The files created range from 68532 to 68807 bytes, so the size variation is not significant in this example. Yet, the shape of the distribution is much more interesting, and its causes remain mysterious. For example, the same shape may be obtained by compressing the concatenation of the file contents, so the tar file format can be discarded. Also, the distribution curve keeps its shape by adding more data points.

Have a try yourself by downloading waf 1.6 from trunk and the example.

Dia cannot say 'Yes'

2010-02-09T15:56:00.000-08:00

I have been using Dia for some time to create diagrams for the Waf book. After all these years, it seems that Dia still cannot say "Yes":

Single file Python apps

2010-02-02T11:41:00.000-08:00

Various applications exist to ease software redistribution and packaging, for example:
klik
cpan
easyinstall
gems

Yet, redistribution is even easier when the application is platform-independent (scripts) and when the application is directly executable.

A naive approach for Python software is to concatenate all the dependent scripts into a single scripts. In the case of Waf, the tools are actually plugins which are not meant to be loaded all at once. Also, the resulting script would have a pretty huge size.

The best idea so far is to create an archive (in the tar file format) and to encode it as a base64 string hidden into the final script. The script, containing only a few routines for decompressing the archive would decode the string and unpack the library into a hidden folder when executed. All the program logic would then follow from the library files uncompressed.

The base64 encoding is safe in the sense that any binary string as input will be transformed in a string containing only letters and a few symbols (64 in total). Yet this operation increases the file size by about 33%.

The ascii85 encoding (used in pdf and postscript files) produces less safe symbols such as quotes and backslashes (85 characters in total), with a better cost (25%) and only a few more lines of python code.

The best of all encodings is to avoid having to encode the binary stream at all, but the python interpreter would not accept such files. The cPython interpreter at least ignores all characters located in comments between # and the following newline (\n and \r), which enables us to store binary data in commented lines. By using this system, the size increase for the binary data is about 2%.

The system for the waf coding is therefore:
1. make a compressed archive of all the files and obtain a binary string
2. find suitable 2-character escape sequences for the newline characters by scanning the binary string
3. replace the forbidden characters by the escape sequences
4. store the escape sequences, and write the binary string in a commented line

Upon execution, the library will be unpacked by following these steps:
1. open the script being executed, and read the binary string
2. replace the escape sequences by the newline characters
3. unpack the files from the binary string to obtain the library

In the case of Jython, the interpreter is a bit different, and will first read the whole file before parsing the Python code (the parser used by Jython seems to require that). Jython will then validate the characters present in the comment sections and throw a syntax error when trying to execute the Waf file.

Fortunately, changing the encoding declaration from 'utf-8' to 'iso8859-1' solved the problem entirely. While quite a few character sequences are forbidden in 'utf-8', binary data makes no such problem in 'iso8859-1'.

The last problem in Jython is the lack of bzip2 support. For now to obtain a compatible waf script, it is necessary to build Waf by using the gzip compression algorithm:
./waf-light --make-waf --zip-type=gz

Although the waf directory is only for preparing the Waf files, replacing the contents of the folder wafadmin and changing the name of the final application may help building and redistributing other Python applications as well.

Eclipse password recovery (cvs, subversion, ..)

2009-07-13T07:43:00.000-07:00

Eclipse stores all the passwords under a file named .keyring, and it makes somewhat difficult to recover them.

Or at least, it only seems difficult to do so. Using a plugin to enable beanshell code execution such as Eclipse-shell, the information may be recovered easily in a few lines of code:

import org.eclipse.core.internal.runtime.auth.AuthorizationDatabase;
import java.lang.reflect.Field;
import java.io.File;

File file = new File("c:/Eclipse/configuration/org.eclipse.core.runtime/.keyring");
String path = file.getAbsolutePath();
AuthorizationDatabase aut = new AuthorizationDatabase(path, "");
Field f = aut.getClass().getDeclaredField("authorizationInfo");
f.setAccessible(true);
f.get(aut);

Waf on jython 2.5 - conclusion

2009-06-30T15:14:00.000-07:00

Executing Waf on Jython is possible, but complicated and slow

After discarding many possible causes (threading, garbage collection), the problem has finally been located in the Jython subprocess execution

Since os.fork might allocate too much memory, Jython uses ProcessBuilder for executing external programs (new in Java 1.5).

Update: Java itself does not seem to be slow at launching processes:


~/tmp/pb > javac Fu.java
~/tmp/pb > time java Fu
1.63user 3.87system 0:05.89elapsed 93%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+64outputs (1major+567624minor)pagefaults 0swaps

The code of Fu.java is the following:


import java.io.*;
import java.util.*;
public class Fu {
    public static void main(String args[]) throws IOException {
        for (int i = 0; i < 1000; ++i) {
            ProcessBuilder p = new ProcessBuilder("ls", "-l", "/tmp");
            p.start();
        }
    }
}

Waf on Jython 2.5

2009-06-19T13:46:00.000-07:00

Jython 2.5 has been released a few days ago, and since Waf somehow supports it, I wanted to have a closer look. Jython is an implementation of the Python programming language for the Java virtual machine.

The supposed benefits of Jython are:

A different threading model able to use multicore processors
The use of the heavily optimized Java virtual machine
Using waf for web applications (Maven) where people usually do not have cPython

Unfortunately, Jython seems to also use the java primitives for parsing the scripts, and executing the waf script will not work. The main Waf script contains a compressed data island in a comment section to ease redistribution. When cPython rightfully skips the comments (except \r characters), Jython has to validate all unicode characters.

For now the solution to execute waf on Jython is to use the waf directory and the waf-light script:
~/jython/bin/jython waf-light

A quick benchmark can be created by using the script genbench.py located in the waf distribution:
utils/genbench.py /tmp/build 50 100 15 5

When executed, the benchmark will compile 5000 c++ files and link them by calling operating systems processes. The test was executed on a laptop having more than enough RAM and a dual core processor.

The cPython command was:
waf distclean configure build clean build clean build -p -j5
The average time of execution was 2m15 (std dev: 5 secs)

The Jython command was:
~/jython/bin/jython waf-light distclean configure build clean build clean build -p -j5
The average time of execution was 5m15 (std dev: 10 secs)

From a developer perspective, the java virtual machine and the jython start-up (7 seconds) slows down the edit-compile-run cyle, which is quite un pleasant (on cpython, waf starts instantly).

I will have a look at the profiles to understand the source of such a difference between cPython and Jython. For now Jython is not an interesting option for Waf users, and it probably will not make other applications such as Scons faster any time soon either.