Cfengine 3.1.2 extended change log

Highlights

Cfengine 3.1.2 was released December 9th, 2010. The theme of the release is major efficiency improvements, but some bug fixes are also included. Looking in the ChangeLog-file in the root of the tarball, we find the following goodies.

  • cf-promises is almost gone
  • potential for much less network load when checking for policy updates
  • ps-command runs less frequently
  • faster access to classes

We will have a look at how to take advantage of the new features in the following.

Caching policy checks

The biggest improvements in version 3.1.2 comes from caching the outcome of cf-promises runs. The main task of cf-promises is to verify that the syntax of the policy is correct (but it also does more advanced analysis). The other components of Cfengine reading the policy (e.g. cf-agent, cf-execd, cf-serverd, etc.) will require verification from cf-promises before even touching the policy themselves. In the default schedule of cf-execd, cf-agent runs every five minutes. This means cf-promises is run at least every five minutes as well.

The amount of time it takes to run cf-promises depends on a number of factors, but perhaps most importantly on the size and complexity of the checked policy. My workstation, using a fairly small policy, seems to spend a bit less than one second on running cf-promises.

root@host:# time cf-promises
real    0m0.730s

Using an enterprise-size policy, this check will probably take multiple seconds to complete. Using the default schedule for cf-agent, this means about 30 seconds of server time per hour per server. Now, if we assume 1000 servers in the organisation, we see that a total of more than 8 hours of precious server time is getting lost just to this simple thing every hour. Even though Cfengine is the most efficient configuration management solution on the planet, there seems to be room for improvement here.

How often does the configuration management policy of an organisation change? Hopefully not every day. Perhaps every week, or month? Assuming once a week, it means that cf-promises is run against the same configuration 12*24*7 = 2016 times a row! How about reducing this to once?

This is exactly what Cfengine 3.1.2 brings, simply by touching a file WORKDIR/masterfiles/cf_promises_validated. This file is created by cf-agent or any other Cfengine component after it has successfully verified the policy with cf-promises. Also, before running cf-promises, the components check if any file either

is newer than the file cf_promises_validated (based on modification time). If not, the run of cf-promises is skipped, and 8 hours of server time is reclaimed every hour!

Reducing network load

The performance improvement by caching the outcome of runs of cf-promises is done automatically by Cfengine, without any need of policy change. However, in the most common architecture of Cfengine, there is still much to be gained by doing a tiny modification to the policy. We assume the hosts pull policy updates from the WORKDIR/masterfiles directory on distribution server(s), as seen in the picture (shamelessly copied from the Cfengine community site).

The trick is to try to copy our new cf_promises_validated file (based on modification time) before trying to copy anything from the remote masterfiles directory itself. Note that it resides in masterfiles on the distribution server(s), so the access policy should need only minor modifications, if any. The end hosts check if the file got copied using if-repaired classes, and only copy the rest of masterfiles if so. The important thing to note here, is that the distribution servers must also run the policy, i.e. copy from their WORKDIR/masterfiles to WORKDIR/inputs locally and run cf-agent regularly in order to update cf_promises_validated if anything changed in WORKDIR/masterfiles to signal the end hosts of the change. This is however the normal way to do it. The following policy snippet demonstrates the more efficient copy.

# Assumptions:
# *) the class “am_distribution_server” is set on distribution servers
# *) the string variable “distribution_server” contains the address of the distribution server

files:

!am_distribution_server::
“/var/cfengine/inputs/cf_promises_validated”
comment => “Check whether a validation stamp is available for a new policy update”,
copy_from => u_remote_cp(“/var/cfengine/masterfiles/cf_promises_validated”,”$(distribution_server)”),
classes => u_if_repaired(“validated_updates_ready”);

am_distribution_server|validated_updates_ready::
# distribution server should always put masterfiles in inputs in order to check new policy

“/var/cfengine/inputs”
comment => “Copy policy updates from master source on distribution server if a new validation was acquired”,
copy_from => u_remote_cp(“/var/cfengine/masterfiles”,”$(distribution_server)”),
depth_search => u_recurse(“inf”);

download above policy

The bodies used are found in the Cfengine Open-Promise Body Library, but they are prefixed by “u_” to make the update-policy (usually part of failsafe.cf) self-containing.

Using this method, we reduce the (possible recursive) copy of the masterfiles directory containing tens or hundreds of files to one file. Reclaim those bytes!

Internal optimisations

As you might know, Cfengine relies on the ps command in order to acquire information about the running processes. Process information interfaces are not portable across Unices, so the ps command was deemed the only viable interface. Process information is required for example when making promises to start a process if it is not running, like the following.

processes:
“httpd”
restart_class => “start_apache”;

“mysql”
restart_class => “start_mysql”;

commands:
start_apache::
“/etc/init.d/apache2 restart”;

start_mysql::
“/etc/init.d/mysql restart”;

download above policy

Previously, these processes-promises would result in two executions of ps. However, running commands has significant overhead, and should thus be done only when strictly necessary. Cfengine 3.1.2 caches the output of a ps execution and reuses it within the same bundle, so ps will only run once in Cfengine 3.1.2 using the above snippet. The reason for not using a global cache for the whole run of cf-agent is to try to avoid staleness, as process information may change quite frequently.

Another internal optimisation introduced is to index class access though a simple hash table. The hash function used is simply to pick the first character of the class name (so “aix” hashes to 0, “linux” hashes to 11, etc.). Because classes seem to be distributed quite well alphabetically, this should be quite efficient. There is a trade-off between the computational burden of the hash function and the quality of the probability distribution it produces. This hash function is very fast, and should have an decent distribution on classes. As a flat list of classes was used previously, the optimisation should have significant effect on all access to classes, including class evaluation, definition, and reporting.

Get it!

As usual, Cfengine 3.1.2 is provided not only as a source tarball, but also prepackaged for the most popular Linux distributions by logging into the Engine Room (free registration required). Users of the following distributions enjoy free packages, both 32- and 64-bit versions.

  • CentOS 5
  • Debian 5 and 6
  • Fedora 14
  • Red Hat Enterprise Linux 3, 4, 5 and 6
  • Suse 9, 10 and 11
  • Ubuntu 8, 9 and 10

Note that most distributions also maintain a Cfengine 3 package, but this is usually older and may not be built in a uniform way.

Please leave a comment if you found this useful, or have suggestions for improvements.

Enjoy!

Posted in CFEngine | Tagged , , , , , , , , , , , , , , , , , | 7 Comments

Amarok mp3, wma and flac codecs

Amarok LogoAmarok is an innovative and configurable music player running on top of the K Desktop Environment (KDE) libraries, making it available for both Linux and Windows.

Unfortunately, non-free codecs such as mp3 and wma are not included in the default installation due to licensing restrictions. Additionally, the flac playback is not always working properly — songs may lag in the beginning, and 24-bit flac is not working at all — the playback is extremely noisy and crackling. However, these issues are not caused by Amarok itself, but rather xine-lib, the default back-end to the KDE multimedia framework Phonon.

We will install the GStreamer back-end with the necessary plugins and configure Amarok to use it instead of xine-lib to solve our issues. The generic instructions should apply to all Amarok 2 installations, and they have been tested on Ubuntu 10.4 LTS. In the end we will cover some additional tips to make Amarok really shine.

Amarok with GStreamer back-end

  1. Install. We need to install a few packages. On Ubuntu, run the following command in a terminal.

    sudo apt-get install amarok phonon-backend-gstreamer gstreamer0.10-plugins-ugly

  2. Set up. Open Amarok and go to Settings-> Configure Amarok->Playback->Configure Phonon->Backend. Mark GStreamer and click Prefer until it is at the top, followed by OK. Restart Amarok.
    Phonon GStreamer Backend
  3. Play!

Amarok tips

  • No Local Collection.
    On Ubuntu 10.4, the Local Collection in AmAmarok - No Local Collectionarok 2 may show up empty even though local music is indeed available. This is a bug in Ubuntu yet to be fixed. A workaround is to install the package “mysql-server-core-5.1” (e.g. run “sudo apt-get install mysql-server-core-5.1” from a terminal), and restart Amarok, and the local collection will be displayed correctly.
  • Audio Device Not Working.
    If you get a message from Phonon when starting Amarok, Phonon Audio Warning when Starting Amaroksaying that the audio playback device is not working (showed to the left), just use the default one. Go to Settings-> Configure Amarok->Playback->Configure Phonon -> Device Preference -> Audio Output -> Music. Mark “default” and click Prefer until it reaches the top, followed by OK, and the message will go away.
  • Keyboard shortcuts. Having keyboard shortcuts to the mostly used actions is very useful. Next song, previous song, play/pause are nice to have shortcuts for. To set it up, go to Settings -> Configure Shortcuts in Amarok. Search for “next”, click on the “Global” cell, then “Custom” and type in the key combination of your liking (I use Win+B) — see the screenshot below. Global means that the combination will work even when Amarok is running in the background. Follow the same recipe for any action you want a shortcut to.
  • Which file type? If Amarok refuses to play some file, you can use the Unix “file” command to find out the exact type of that file. For example, “file audio.flac” will show if “audio.flac” is a 16- or 24-bit flac file.

    $ file audio.flac
    audio.flac: FLAC audio bitstream data, 24 bit, stereo, 48 kHz, 13914720 samples

  • Internet Services. Additional features may easily be added to Amarok by enabling internet services. Go to Settings -> Configure Amarok -> Internet Services to see them. In particular, last.fm is a handy service which recommends new music to you. The recommendations are based on the songs you play and the collection of other people playing the same songs.

Happy playing with Amarok!

Posted in Software configuration | Tagged , , , , , , , , , , , , , , , , | 3 Comments

Compile for Windows on Linux

Introduction

In an earlier post, we saw how to obtain gcc on windows, using the MinGW-w64 suite. However, users familiar to gcc are often using one of the operating systems of the Unix family, such as Linux. As amazing as it may first seem, the MinGW-w64 project allows users to compile native Windows binaries on Linux. This concept of targeting a different platform than the compiler is running on is however not new, and is known as cross-compilation.

Cross-compiling Windows binaries on Linux may have many benefits to it.

  • Increased compilation speed. Linux is generally faster than Windows with the mingw toolchain.
  • Reduced operating system complexity. On cross-platform projects that are also built on Linux, we can get one less operating system to maintain.
  • Access to Unix build tools. Build tools such as make, autoconf, automake and Unix utilities as grep, sed, and cat, to mention a few, become available for use in Windows builds as well. Even though projects such as MSYS port a few of these utilities to Windows, the performance is generally lower, and the versions are older and less supported than the native Unix counterparts. Also, if you already have a build environment set up under Linux, you don’t have to set it up again on Windows, but just use the existing one.
  • Lower license costs. As we know, Windows costs in terms of license fees. Building on Linux, developers do not need to have  a Windows installation on their machines, but maybe just a central Windows installation for testing purposes.

How It Works

On a Linux build environment, a gcc that compiles native binaries is usually installed in “/usr/bin”. Native headers and libraries are in turn found in “/usr/include” and “/usr/lib”, respectively. We can see that all these directories are rooted in “/usr”.

Any number of cross-compiler environments can be installed on the same system, as long as they are rooted in different directories. In our example, we will use “/opt/mingw32” and “/opt/mingw64” as root directories for the new build environments. Now, we would perhaps expect to find “/opt/mingw32/bin/gcc” and “/opt/mingw64/bin/gcc”, but we instead see “/opt/mingw32/bin/i686-w64-mingw32-gcc” and “/opt/mingw64/bin/x86_64-w64-mingw32-gcc”. The reason for this is that we  (and configure scripts) should be able to pick the “right” gcc, even if  we have multiple compilers in the PATH environment variable. If they were all named gcc, cross-compiling would easily become messy.

Cross-World Hello

  1. Go to the MinGW-w64 download page. We need two toolchains – one for targeting win32 and another for targeting win64.Open “Toolchains targetting Win32” , followed by “Automated Builds”, “mingw-builds” and a recent version (e.g. mingw-w64-bin_x86_64-linux_20131228.tar.bz2).

    Now do the same for “Toolchains targetting Win64” (e.g. mingw-w64-bin_x86_64-linux_20131228.tar.bz2).

    There are some notes on the package naming convention below to help you pick the right one. Also note that the direct links above might be to older versions when you read this — so please check the directory structure for updates.

  2. Unpack the first archive to /opt/mingw32 and the second to /opt/mingw64.
  3. In a text editor (e.g. gedit or nano), paste in the little greeting-code and save it to /tmp/hello.c
    #include <stdio.h>
    int main()
    {
    printf("Hello World!\n");
    return 0;
    }
  4. Compile it for both 32- and 64-bit Windows with the following commands.
    /opt/mingw32/bin/i686-w64-mingw32-gcc /tmp/hello.c -o /tmp/hello-w32.exe
    /opt/mingw64/bin/x86_64-w64-mingw32-gcc /tmp/hello.c -o /tmp/hello-w64.exe
  5. Run “hello-w32.exe” on 32-bit Windows, and “hello-w64.exe” on 64-bit Windows.

 

In order to build useful applications, it is convenient to use existing libraries such as the OpenSSL library on Windows.

Package Naming Conventions

As we saw on the MinGW-w64 download page, there are a lot of available packages with only subtle and perhaps confusing name differences. The automatically built packages have the following generic naming pattern.

mingw-TARGET-bin_HOST_DATE.PKG

  • TARGET states which platform we want the compiled binaries to run, and can be either “w32” (32-bit Windows) or “w64” (64-bit Windows).
  • HOST gives the host system, that is, the system on which the compiler binaries themselves are run. Thus, we are cross-compiling if HOST is different from TARGET. If we have a Intel 32-bit Linux distribution, we can pick a HOST value of “i686-linux”, from a 64-bit Linux host we would choose “x86_64-linux”, and from 32-bit Windows we can choose “i686-mingw”.
  • DATE is the date, in the form YYYYMMDD, when the automatic build was created.
  • PKG is the compressed archive format, such as “zip”, “tar.bz2” and such. Generally, zip archives contain binaries that run on Windows, all other archives contains binaries that run on Linux.

Running the Binaries

Using Wine, we can even test the binaries directly from Linux. However, this only works on 32-bit Windows binaries and is not perfect due to bugs and missing features in Wine itself. After downloading and installing Wine for our distribution, we can test our program above by running “wine hello-w32.exe”.
Note that 64-bit Windows can run 32-bit binaries due to an emulation layer called Windows 32-bit On Windows 64-bit, but native binaries are more efficient.

Posted in GCC Windows | Tagged , , , , , , , , , | 44 Comments

Dell PowerEdge R200 Virtualization

The Dell PowerEdge R200 is a reasonably priced 1U rack server. With 8GB RAM and a 250GB hard drive, its price tag is around £370 at the time of writing. Not bad for a rack server. Most of the low price is achieved by picking a more inexpensive processor than the standard Intel Xeon server processors, which are priced at about £250 alone. Workstation processors such as the Celeron or E-series keeps this server affordable, but yet well performing.

For applications with relatively low load, this server may be the perfect host for virtual machines. Or at least so it seems. Some of the R-series servers are actually delivered with processors that do not support Intel’s hardware virtualization (VT-x) technology! I bought a R200 server with an Intel Pentium E5400 processor. According to the Intel page, it seems like some models of these are support virtualization, but the one from Dell does not.

After five phone calls and two emails to Dell, I was informed that the server did support virtualization, but I had to buy a Xeon processor with a price tag around £300 to get virtualization support. They also quoted their own tech center page.

The PowerEdge R200 server must be configured with Intel Xeon processors to run Hyper-V. This server can also be configured with Intel Celeron®, Intel Pentium® Dual-Core, and Intel Core 2 Duo processors—none of which supports Intel VT hardware-assisted virtualization.

Fortunately, this is wrong! I bought a Intel Pentium E6300 processor for about £50. Probably any recent Intel processor with LGA775 socket will work, but remember to check the Intel pages for VT-x support. Replacing the existing processor is very easy, and is no problem if you have touched the internals of a computer some time in your life. Remember that virtualization support must be enabled in the BIOS (press F2 after turning the server on). Furthermore, Dell support informed me that the server warranty is still valid for the components that are not replaced.

So now I have a working rack server running virtual machines, costing about £420. I am very happy with it, I just wish I didn’t have to spend days figuring out how to fix it in the first place. Hopefully, this post will help some spend less time on this issue.

Posted in Hardware | Tagged , , , , , , , , , , , , , | Leave a comment

gcc for Windows

Introduction

Most Unix and Linux users know the GNU Compiler Collection (gcc). This software compiler system is being used to build a lot of software, including modern operating systems such as Linux and BSD, which means it is well tested and maintained. Additionally, gcc is available under GPL v3 – the most popular free software license. This gives everyone the right to examine and modify its source code and also download it free of charge.

gcc can provide many benefits to developers targeting the Windows platform. In contrary to Visual Studio and most other compiler tools for Windows, gcc is Open Source Software completely free of charge and supported by a large community. Projects targeting multiple operating systems will also save time by using just one compiler system, and if you already know gcc you don’t have to spend time learning new compilers.

Compiling for 64-bit Windows

There are multiple ports of gcc to Windows, but MinGW (Minimalist GNU for Windows) seems to get the most traction. But currently MinGW can only build 32-bit binaries. In order to be able to target both 32-bit (x86) and 64-bit (x64) Windows versions, we will use a flavour of it called MinGW-w64 (aka. mingw64). In addition to the compiler binaries, the package contains headers (e.g. stdlib.h, strings.h, etc.) and libraries necessary to build applications.

Hello World

It is time to get our hands dirty and compile a C-version of Hello World. We will create one for 32-bit Windows and one for 64-bit Windows. The build platform used is a 32-bit Windows XP, but any version of Windows newer than Windows 2000, including Windows Vista and Windows 7 should give the same results. 64-bit versions of Windows will work as well.

  1. Go to the MinGW-w64 download page. We need two toolchains – one for targeting win32 and another for targeting win64. New packages are frequently uploaded, but we just pick the newest personal  build (I had some bad experiences with the automated builds). Open “Toolchains targetting Win32″ and “Toolchains targetting Win64″, followed by “Personal Builds”. At the time of writing, we can open for example “sezero_20110510” and pick “mingw-w32-bin_i686-mingw_20110510_sezero.zip” and “mingw-w64-bin_i686-mingw_20110510_sezero.zip“, but also see MinGW-w64 package naming conventions.
  2. Unpack the first package to “c:\mingw32” and the second to “c:\mingw64”.
  3. Open a text editor (e.g. Notepad), paste in the famous world greeting and save it under “c:\tmp\hello.c”.
    #include <stdio.h>
    int main()
    {
      printf("Hello World!\n");
      return 0;
    }
  4. Open a Command Prompt, cd to “c:\tmp” and compile with the following commands. See the related cross-compilation post for an explanation of the compiler names.
    "c:\mingw32\bin\i686-w64-mingw32-gcc.exe hello.c -ohello-w32.exe"
    "c:\mingw64\bin\x86_64-w64-mingw32-gcc.exe hello.c -ohello-w64.exe"
  5. On 32-bit windows (aka. x86), you can only run “hello-w32.exe”, while you can actually run both on 64-bit Windows (aka. x64) because of an emulation layer called Windows 32-bit On Windows 64-bit. It is however much more efficient to run native binaries, that is, “hello-w64.exe” on 64-bit Windows versions.

Gcc compile and run of Hello World on Windows.

Build Environment

When building larger projects, we usually want a Unix-like shell in order to run additional tools like configure, make and automake. On Windows, we can use MSYS (Minimal SYStem) to get a shell and the basic commands often used in build-scripts, like grep and sed. This means we can compile code on Windows that will run on Windows, which seems like the natural way to go.

However, note that with the MinGW-w64 suite, it is possible to compile code on Linux that will run on Windows. This is also known as cross-compilation, and will greatly increase the build speed but also allow to use any native Linux shell, rendering MSYS unnecessary. In another post, we consider how to cross-compile Windows binaries on Linux.

Furthermore, real-world applications depend on common libraries. There is a related article on how to compile and use the OpenSSL library on Windows.

mingw-w64-bin_i686-mingw_20101003_sezero.zipmingw-w64-bin_i686-mingw_20101003_sezero.zip
Posted in GCC Windows | Tagged , , , , , , , , | 21 Comments