Measure

The time of five events are measured:

• [1] (printk timing) A very early kernel message, “ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.AGP_._PRT]”
  This message is buffered until syslog starts, so its date information in /var/log/messages can not be used. The following event allows a conversion between the two timings.
• [2] (printk timing and date from syslog) The first message when the kernel executes userland: “Adding 2047888k swap on /dev/hda5″
• [3] (date from syslog) first userland daemon message: “acpid: starting up with proc fs”
• [4] (date from syslog) first X program message: (seahorse makes a call to gpgme)
• [5] (date) a program run by the GNOME session messenger stores the output of date +%S.

make clean was performed before each recompile.

System

Gentoo Linux, 64bit, 2.6.30-gentoo-r4 kernel, quite normal End-user system with the GNOME Desktop. HP Pavilion dv8000 single-cpu 2GHz, 1GB RAM.

Optimizing with -O3, -O2 or -Os?

People from Gentoo, GCC, the LKML and many others will tell you that using -O3 is not a good idea. Furthermore, O3 it is known to make the kernel instable.

So let’s try it anyway, and for the fun of it, add -march=native after -m64. We also allow GCC to uninline functions marked as inline.

I can look at the time differences (in seconds, y-axis) between the events (deltas), relative to one optimization (I chose -Os), and absolute times (zero being system boot):

A average over 3 boot was taken, and the error bars show the standard deviation. Usually, you should take a dot that lies out 3 times the standard deviation seriously. If you now want to complain that you can’t see the dots in the graph, you are missing the point: The standard deviation is way larger than the difference between the averages.

=> It makes absolutely no difference whether you compile the kernel -O3, -O2 or -Os.

But: I ran into trouble with -O3, as the cpufreq sysfs-interface hung (blocked io).

Preemption

I keep the -Os kernel.

Three options are given for the kernel preemption model: No forced preemption (“preempt”),  Voluntary Kernel Preemption (default), preemptible kernel (“no-preempt”). Get more information in menuconfig or the kernel documentation.

Again, the same diagrams as above:

One might have the feeling that a preemptible kernel is slower during kernel boot, but faster when executing the userland startup. But again, the standard deviation is way to big, one can not draw a definite conclusion, none of the options out-performs the other.

To make the point clear, the difference between the average boots is less than a second (and yes, this system boots in 60 seconds into the GNOME desktop).

Pretty boring so far, but how would a significant result look like, then?

Scheduler

I keep the preemptible, -Os kernel.

The kernel got a new scheduler a while ago, CFQ (Completely fair scheduler). (This also gave us the beautiful program ionice).

There are 4 schedulers available, they can be selected by the kernel parameter “elevator=…”: Anticipatory, Deadline, CFQ and No-op. Get more information about them in menuconfig or the kernel documentation.

Now that’s a real difference: CFQ boots around 9 seconds faster than the others, the standard deviation is around less than 2 seconds.

=> CFQ is superior to the other schedulers in this scenario.

“Unfortunately”, CFQ is already the default scheduler, so the default kernel is already the best choice. Good to know.

All used kernel configs are available as a tar file. You can play with the numbers using the gnumeric file.

Watching the disk light, a program like readahead (keeps file content in memory) should be the way to improve startup speed. In my setup, using readahead-list did not change the boot up time.

Parallel boot services (RC_PARALLEL_STARTUP) managed to squeeze out 2-3 seconds to a final boot time of around 58 seconds. At the end you learn the best tools … bootchart for example.

sed  '/mypattern/d' file

Deletes all lines that contain the regular expression “mypattern”.

In-place replacement:

sed --in-place '/mypattern/d' file1

sed also supports making backups with e.g. –in-place=.backup .

while ! cat /proc/loadavg |cut -f2 -d ' '|grep '0\.00'; do sleep 10m; done && poweroff

Alternatively:

while ps 12345; do sleep 10m; done && poweroff

for i in 1 2 3 4; do grep -v $i; done

So I wrote this chaining tool that does the job in the obvious way:

chain.sh

#!/bin/bash
commands_file=$1
a=$(mktemp)
b=$(mktemp)
cat > $a
while read line; do
$line < $a > $b
mv $b $a
done < $commands_file
cat $a
rm $a

With this, you can do:

for i in 1 2 3 4; do
echo grep -v $i >> mycommands.txt
done
script/chain.sh mycommands.txt

lang=”bash”

Steps:

1. create ssh-key on new collaborators computer (ssh-keygen)
2. create github user
3. add public-key to github user account
4. let the project owner add you as project member
5. git clone git@…

Where 1 < 3, 1 < 2 < 4, 2 < 5 if that means anything to you ;-)

* it works with git remote

So, what I did was wrap maven in a script that colorizes the output. The result looks like this:

This is how to do it:
create the file ~/bin/mvn with the following content (mvn (0 KB))

#!/bin/bash
/usr/bin/mvn $* | sed -e 's/Tests run: \([^,]*\), Failures: \([^,]*\), Errors: \([^,]*\), Skipped: \([^,]*\)/Tests run: \1, Failures: \2, Errors: \3, Skipped: \4/g' \
-e 's/\(\[WARN\].*\)/\1/g' \
-e 's/\(\[INFO\].*\)/\1/g' \
-e 's/\(\[ERROR\].*\)/\1/g'

replace /usr/bin/mvn with wherever your mvn is (find out with which mvn).
Double-check that you get the encoding right, the weird sign should be hexadecimal 1b. Use a editor that supports encodings for copy+paste.
Give your ~/bin/mvn execute permissions (chmod +x ~/bin/mvn)

Try it out: call ~/bin/mvn in one of your maven projects

What is left to do is to register your new mvn-wrapper so it is called everytime.
Add the following line to your ~/.bashrc :

alias mvn=~/bin/mvn

Execute the command you just added in your running shells so you can use it there too.

You are done.

Changing the distributed mvn startup script

Even better, but a little more intrusive (you need root rights):
change the distributed mvn file. It is a bash script actually.
In gentoo:

 $ which mvn
/usr/bin/mvn
$ file /usr/bin/mvn
/usr/bin/mvn: symbolic link to `/usr/share/maven-bin-2.0/bin/mvn'
$ file /usr/share/maven-bin-2.0/bin/mvn
/usr/share/maven-bin-2.0/bin/mvn: Bourne shell script text executable
$ 

Change the last call in that file to:

exec "$JAVACMD" \
$MAVEN_OPTS \
-classpath "${M2_HOME}"/boot/classworlds-*.jar \
"-Dclassworlds.conf=${M2_HOME}/bin/m2.conf" \
"-Dmaven.home=${M2_HOME}"  \
${CLASSWORLDS_LAUNCHER} $QUOTED_ARGS |
sed -e 's/Tests run: \([^,]*\), Failures: \([^,]*\), Errors: \([^,]*\), Skipped: \([^,]*\)/Tests run: \1, Failures: \2, Errors: \3, Skipped: \4/g' \
-e 's/\(\[WARN\].*\)/\1/g' \
-e 's/\(\[INFO\].*\)/\1/g' \
-e 's/\(\[ERROR\].*\)/\1/g'

That way you have it there directly. Maybe the gentoo devs could add a color use-flag. That’d be fun…

timeoutwrapper.sh

#!/bin/bash
# License: BSD License
# author: Ram
# author: Johannes Buchner
TIMEOUT=$1
shift
COMMAND="$*"
if [[ "$TIMEOUT" == "" ]] || [[ "$COMMAND" == "" ]]; then
echo "USAGE: $0 <seconds to wait> <command with arguments>"
exit
fi
$COMMAND &
PRODPID=$!
export PRODPID
# record own PID
export PID=$$
# define exit function
exit_timeout() {
echo "Timeout. Checking for unfinished."
for i in ${PRODPID} ; do
ps -p $i |grep -v "PID TTY"
if [ $? == 0 ] ; then
# process still alive
echo "Sending SIGTERM to process $i"
kill $i
fi
done
# timeout exit
exit
}
# Handler for signal USR1 for the timer
trap exit_timeout SIGUSR1
# starting timer in subshell. It sends a SIGUSR1 to the father if it timeouts.
export TIMEOUT
(sleep $TIMEOUT ; kill -SIGUSR1 $PID) &
# record PID of timer
TPID=$!
# wait for all production processes to finish
wait ${PRODPID}
# Normal exit
echo "All processes ended normal"
# kill timer
kill $TPID

Usage examples:

$ bash timeoutwrapper.sh
USAGE: timeoutwrapper.sh <seconds to wait> <command with arguments>
$ bash timeoutwrapper.sh 1 sleep 3
Timeout. Checking for unfinished.
26612 pts/3    00:00:00 sleep
Sending SIGTERM to process 26612
$ bash timeoutwrapper.sh 4 sleep 3
All processes ended normal
$ bash timeoutwrapper.sh 2 du -sh .
Timeout. Checking for unfinished.
26622 pts/3    00:00:00 du
Sending SIGTERM to process 26622
$

PS: I found the script here and adopted it
http://redflo.de/tiki-index.php?page=Bash+script+with+timeout+function

Anyway, consider you have a ssh server in the internet, and two computers behind different firewalls that only let traffic out. But you want to connect from computer A to computer B.
So you could connect to the ssh server from both computers, but you would need some software that reverses your TCP flow. One of the channels has to be the client and one the server. That might even be possible just using pipes and ssh -L.

Ok, so I thought I was clever until I found the -R option.
What it does is to connects to the ssh server and offers a local service (port) remotely. (-L offers a remote service (port) locally.)
Consequent. And works nicely too!