Jimmy's weblog


Booting Grml via iPXE

Filed under: — jimmy @ 12:39 am

iPXE is a network boot firmware that allows you to boot ISO images over http. It’s actually part of the Grml ISO and you can find it under Addons in the boot menu. Alternatively you can download it from the iPXE website. All you need is an ISO image and the memdisk binary stored on a webserver. The ISO image needs to support the memdiskfind approach, more details can be found on the syslinux website. Grml supported it for a while, unfortunately it was broken in the last stable release but it’s working in the daily built images again.
To boot over http you need to run iPXE and press Ctrl-B after an IP address was assigned. This will give you a shell where you need to type in the following commands:

initrd http://download.grml.org/devel/grml64-small_20160415_netboot_xyz.iso
chain http://download.grml.org/devel/memdisk iso

The first one will download the ISO and the next one will download memdisk and chainboot the ISO, which is the daily build of Grml-small on the 15th of April. There’s also another ISO if you prefer Grml-full:

initrd http://download.grml.org/devel/grml64-full_20160415_netboot_xyz.iso
chain http://download.grml.org/devel/memdisk iso

If you want to boot newer daily builds, just replace the first line with the correct URL.

There’s also an easier way using iPXE menus and it only requires one command instead of two:

chain http://download.grml.org/devel/menu.ipxe

This will present you a menu where you can choose which flavor to boot.
The file menu.ipxe contains:

set menu-default small

menu iPXE boot menu for Grml
item --key f full    Boot Grml Full
item --key s small   Boot Grml Small

choose --default ${menu-default} selected
goto ${selected}

echo Booting Grml Full over http
initrd http://download.grml.org/devel/grml64-full_20160415_netboot_xyz.iso
chain http://download.grml.org/devel/memdisk iso

echo Booting Grml Small over http
initrd http://download.grml.org/devel/grml64-small_20160415_netboot_xyz.iso
chain http://download.grml.org/devel/memdisk iso

If you want to test this on your own webserver just copy your choice of ISOs, the memdisk binary and the file menu.ipxe to the root directory.


Installing jenkins-debian-glue on a Rackspace Cloud Server in ~5 minutes

Filed under: — jimmy @ 11:33 pm

jenkins-debian-glue allows you to build Debian & Ubuntu packages directly from the Jenkins Continuous Integration system. There’s a good documentation on how to deploy it automatically on cloud servers. I just want to add one extra remark on how to do it on a Rackspace Cloud Server. Because otherwise the setup may fail because of a missing full qualified domain name (FDQN).

Use FQDN as "Server Name" when creating the server instance, e.g. jenkins.jimmy.co.at

Thanks to the guys in the rackspace channel on IRC for pointing me in the right direction!
This way /etc/hosts and /etc/hostname will contain the FQDN and the automatic setup of jenkins-debian-glue will work without any problems. I suggest to correct the hostname in /etc/hostname to the short name, e.g. jenkins, because Debian and Ubuntu always use the short name in /etc/hostname and I’m not sure if something is going to break otherwise! AFAIK this is not true for Red Hat/CentOS.
BTW: /bin/hostname -s works also with FQDN in /etc/hostname.
I’ve tested this setup with a Standard and a Performance 1 instance, with Debian 7 and with Ubuntu 12.04 LTS. Whole installation including creating server instance took about 5 minutes.
Have fun building Debian & Ubuntu packages :)


I’m looking for a new job :-)

Filed under: — jimmy @ 11:55 pm

Dear readers,

Today I’m not posting about Linux or IBM Hardware but to ask for your help. I’m going to sell my company and relocate to California in the beginning of 2014. Therefore I’m looking for a cool job in the Bay Area, Debian jobs preferred. So if you know about such a job I greatly appreciate your mail (jimmy@g-tec.co.at) :)
(in other words: system administrator, developer or devops jobs closely related to FLOSS)
Thank you.


My new X230… Backup of the factory default state

Filed under: — jimmy @ 1:46 am

Before I started to modify the original setup I wanted to backup a couple of things because I knew, that most of the steps are not straightforward and I might need to get back (To be honest, I had two other Lenovo machines where I tested the steps before I applied them on my X230 ;-) )
I started with creating the recovery media and Lenovo supports to use usb flash media instead of cd recordables, too. But something told me that I should not fully trust the software to create a bootable usb flash drive. Long story short: The software creates an unbootable flash drive if you do not prepare the flash drive before you start the program :(
Have a look at the instructions at Lenovo’s support site. It’s not only a missing boot flag which I set afterwards, there seem to be other constraints, too. The flash drive, which did not work, had a strange partition table, too. On the working flash drive it looked like this:

Disk /dev/sdc: 15.8 GB, 15804137472 bytes
255 heads, 63 sectors/track, 1921 cylinders, total 30867456 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0×00000000

Device Boot Start End Blocks Id System
/dev/sdc1 * 2048 30867455 15432704 7 HPFS/NTFS/exFAT

I assume that the starting cylinder is important.

Usually you can create recovery media only once but there’s a hack to do it again, which is very useful if things go wrong. Open command prompt and type:

echo 0 > Q:\FactoryRecovery\RECOVERY.INI:Done

It’s also a good idea to save the mbr and partition table, e.g.:

dd if=/dev/sda of=mbr-backup-sda.img bs=1 count=512
sfdisk -d /dev/sda > sfdisk-sda-dump.txt
sfdisk -l /dev/sda > sfdisk-sda.txt

(Basically the first command is sufficient but I prefer to have different backups)


My new X230… things have changed :)

Filed under: — jimmy @ 6:34 pm

My first Lenovo notebook was a X60s, which is still running quite well but 32 bit is not enough ;-) It came with WinXP preinstalled because there was no option to order it with Linux or without OS. All other notebooks that I used afterwards came without OS so there was no other OS to take care about. My X230 came with Win7 preinstalled so I decided to compare the actual installation with my first one eight years ago (I don’t really need the preinstalled OS so I might as well just make a clean installation but I was interested about the possibilities for our customer installations).
So i will write some posts about certain aspects of the original setup. First one will be about backuping up the factory default state because there the troubles started ;-)


Got my Lenovo X230 without camera :(

Filed under: — jimmy @ 7:51 pm

Take care when ordering your notebook in the Lenovo webshop, you might miss a camera, too. I configured mine and I wanted a builtin camera. Then I decided that the 3×3 antennas would be better than the 2×2 and so I changed the configuration. What I did not see was, that the box for the camera options, where you choose if you want a builtin camera or none, just disappeared and the camera gets unselected. When checking the config again before hitting the order button I checked every option but there was no line like “camera: none” so it looked good for me. When the notebook arrived there was no camera builtin. Reason: The third antenna in the 3×3 config needs the space where the camera would be. Unfortunately there was no hint or warning in the webshop :(
Of course I was able to send it back and order a new one but this process is very time consuming. This happened a couple of weeks ago and in the meantime I got my new X230 but the bug is still not fixed in the shop although I reported it :(
(BTW: I ordered in the shop of Austria but I also checked the US shop which has a similiar problem)

I hope this post helps somebody else not to run into the same problem.


Cisco WAP4410N and Gigabit Performance

Filed under: — jimmy @ 10:42 pm

In the last days I made some tests with 802.11n performance with Cisco WAP121 and WAP4410N access points to find out how much throughput I can achieve. Therefore Gigabit ethernet is required and the WAP4410N comes with a Gigabit interface (the WAP121 does not, JFYI). So I connected the AP with the included ethernet cable and forced it to use 1000 Mbps. Suddenly the link went down and never came up again. I checked the switch and everything was configured properly. Just for fun I took a look at the supplied cable and it said Cat.5, not Cat.5e. So Cisco includes a cable that does not work with Gigabit *LOL*
(Yes, under some circumstances a Cat.5 cable might work but I never tried that myself. It’s better to pay 2 Cents more for Cat.5e or Cat.6 ;-) )


IBM x3100 M4 with ServeRAID C100: Testing Debian Compatibility

Filed under: — jimmy @ 2:22 pm

When IBM announced their new entry server I first checked the specs and found out, that they are going back to fakeraid controllers in the low cost machines. I was really glad a couple of years ago when those controllers disappeared but now they are back again :( Furthermore it first seemed, that the controller was not supported in the kernel of debian squeeze. So I ordered one of the first x3100 M4 to find out myself. It turned out, that there’s absolutely no problem with the squeeze kernel (Didn’t find those reports, who said that the controller is not detected, again, so I hope this false information is lost forever or at least hard to find).
Depending on the BIOS setting (modus AHCI or RAID) the controller shows up as:

00:1f.2 SATA controller: Intel Corporation Cougar Point 6 port SATA AHCI Controller (rev 05)


00:1f.2 RAID bus controller: Intel Corporation 82801 SATA RAID Controller (rev 05)

Although I already wasted a lot of time in my life with fakeraid controllers I decided to test the ServeRAID C100, too. Here are the lessons I’ve learned:

  • Never create the RAID in the OS because you might choose the wrong format and thus you will not see the RAID in the BIOS or IMM2 and lose the ability to monitor the health status. Create it in the BIOS or with the CLI tool.
  • If you want to install with the debian installer then you need to add option “dmraid=true” in expert mode
  • grub2 in squeeze has a bug that prevents installation on fakeraid (dirty hack: install grub from sid)
  • Booting from dmraid is broken in Debian: Bug #603319 with a dirty hack
  • I hope this posting will help you, if you really need to configure such a machine. It’s also possible and supported to upgrade the x3100 M4 with a HW-RAID controller, e.g. ServeRAID M5015 (you need the Hot-Swap Backplane, too).


    Growing Hardware RAID-5 with arcconf

    Filed under: — jimmy @ 8:26 pm

    Maybe you read my previous post “Growing Hardware RAID-5 with LVM by adding a physical disk” where I described how to add another disk to a RAID-5 with the MegaCLI tool. For older ServeRAID controllers you have to use the arcconf tool instead, e.g. ServeRAID-8k, which was shipped in the first x3650 models. There’s plenty of documentation about arcconf but how to resize a raid was hard to find. The MODIFY command is used for resizing and let’s assume that we have 3 disks in the array and add another one (same as in my previous post)

    ./arcconf modify 1 from 1 to MAX 5 0 0 0 1 0 2 0 3

    The syntax of arrconf is:

    ./arcconf MODIFY <Controller#> FROM <LogicalDrive#> TO [Options] <Size> <RAID#> <CHANNEL# DRIVE#> [CHANNEL# DRIVE#] [noprompt]

    So in our case it was Controller# 1 and LogicalDrive# 1 and MAX means to use all capacity that will be available. Raidlevel will still be level 5 and the harddisks are [0 0], [0 1], [0 2] and the new [0 3].
    To watch the status of the raid migration, use

    ./arcconf getstatus 1

    JFYI: To add another 146 GB SAS (15k) to the raid it took about 12 hours to rebuild the array.


    Growing Hardware RAID-5 with LVM by adding a physical disk

    Filed under: — jimmy @ 1:00 am

    Sometimes you’re running out of space and all you can do is adding additional disks. Although it’s a simple task in theory, there a some risky steps involved. So you should backup your volumes before. In my case I had a RAID-5 in an IBM x3650 M3 Server with a ServeRAID M5015 Controller. There was one volume with three 146 GB SAS disks and on top LVM. The root filesystem was on this volume, too, which doesn’t make things easier ;-)
    First Step: Insert a new 146 GB SAS disk to extend the volume. The first risky step is to rebuild the RAID-5 with four disks instead of three. So you may want to upgrade the firmware of the controller first and make a backup of the LVM volumes. Of course you should not save the backup on the same server ;-)
    I configured a partimage server on another machine and ran partimage client on the server with the new disk:

    partimage -z2 -domMc -s save /dev/mapper/main-rootfs fileserver.partimg.bz2

    (Please lookup the options in the manpage yourself)
    The command for the ServeRaid controller was not easy to find but here we go:

    /opt/MegaRAID/MegaCli/MegaCli64 -LDRecon -Start -r5 -add -PhysDrv [252:3]  -L0 -a0

    252:3 stands for the fourth drive. You can find out more about available drives running:

    /opt/MegaRAID/MegaCli/MegaCli64 -PDList -a0

    To watch the rebuild status run:

    /opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -L0 -a0

    As we want to resize the root filesystem we need to reboot the machine so that programs like fdisk see the new size of the array. Afterwards you can resize the partition or simply delete and recreate it. Next step is to resize the LVM physical volume and again a reboot is needed so that pvresize recognizes the new partition size. So these are the next steps where things might get wrong, too:

    pvresize -v /dev/sda2
    lvresize -l +34871 /dev/main/rootfs
    resize2fs /dev/main/rootfs

    /dev/sda2 was my LVM partition (sda1 was for /boot) and thanks to the new disk I had 34871 extends available (146 GB). The resize commands work with active volumes and mounted filesystem so you don’t have to stop any services. Only the two or three reboots (if you want to make a final reboot after all changes) interrupt the services but this should be ok.


    Upgrading Samba from plaintext passwords to encrypted passwords

    Filed under: — jimmy @ 5:23 pm

    Recently I had to upgrade a samba installation which was using plaintext passwords. This is not as rare as one might think because some companies couldn’t upgrade from win95 PCs as those were coupled to machines which didn’t support newer versions. For plaintext passwords samba uses /etc/passwd and /etc/shadow to find out usernames and passwords. With encrypted passwords samba uses its own database with the tdbsam backend. There was also a smbpasswd backend which is now deprecated. But it’s very handy if you want to upgrade to encrypted passwords without having every user to enter her password again to fill the new database. There is the global option “update encrypted = yes” which replaces a plaintext password with an encrypted one when the user logs in. The plaintext password is written to the smbpasswd file defined in smb.conf (This does not work with the tdbsam backend but you can migrate from smbpasswd to tdbsam easily afterwards). So before we let the users login again we have to generate a smbpasswd file. One way to achive this is to run this command:

    cat /etc/passwd | mksmbpasswd >> /etc/samba/smbpasswd

    (assuming that “passdb backend = smbpasswd:/etc/samba/smbpasswd” is in your smb.conf)
    After all users have logged in (check smbpasswd file for filled in passwords) you can disable plaintext auth.

    Watch out for bug in WinXP SP3 with plaintext auth!
    When I made this migration it happened that this procedure was working great for some clients where others completely failed to login. I found out that Win7 clients and WinXP clients with SP2 worked, but WinXP clients with SP3 were failing. The reason is simply a bug caused by patch KB2536276. You can find the full story in bugzilla of samba: Windows security patch KB2536276 prevents access to shares
    In my case I was lost and had to ask the users to type in their passwords again but there seems to be a workaround if you read the last comments in the bugreport (changing domain name)


    Grazer Linuxtage 2011

    Filed under: — jimmy @ 5:28 pm

    Hope to see you there :-)

    Grazer Linuxtage 2011


    Using IBM Bootable Media Creator on Debian

    Filed under: — jimmy @ 1:03 am

    IBM provides a tool called “Bootable Media Creator” (bomc) which creates bootable media – like an usb key – that allows you to easily install all firmware updates even if there is no OS installed on the server. You just select your server model, e.g. x3650, and the tool downloads all available updates. This saves a lot of time.
    Formerly I used bomc on a SLES server but now I wanted to try if it would work on my Debian notebook, since the tool is available as a single binary and not as a rpm package as usual.
    It started up and seemed to work but unfortunately after downloading some packages it stopped with an error. Long story short: I just copied the SuSE-release file to my /etc on my Debian notebook and all errors were gone :-)
    My /etc/SuSE-release looks like this:

    SUSE Linux Enterprise Desktop 10 (i586)
    VERSION = 10

    Bomc can be downloaded at the IBM site


    Dirty dirvish hack to use “localhost” as client option

    Filed under: — jimmy @ 11:03 am

    When using dirvish you usually have to use the local hostname for the client option in default.conf. In other words, the output of the command “hostname”. If it’s different than the client option than dirvish assumes that it’s a remote transfer and uses ssh to connect. So you can not use “localhost” which would be rather straightforward ;-(
    Lately I needed dirvish on a sytem with heartbeat where two systems share one configuration but have different hostnames. Because I was so angry about this IMO stupid check with “hostname” I made a dirty hack…
    Line 258 in /usr/sbin/dirvish looks like this:

    chomp($$Options{Server} = `hostname`);

    I changed it to:

    chomp($$Options{Server} = localhost);

    This way dirvish assumes localhost as local tranfer and everything else as remote. The check itself can be found in line 520.
    Remember, this hack is really evil. There are several better ways to solve the problem.


    Linuxtage 2010

    Filed under: — jimmy @ 8:18 am

    Only one day left. I’ll be there at the grml-booth, which is a must see, again. See you there :-)

    Grazer Linuxtage 2010


    Linuxtage Graz 2009

    Filed under: — jimmy @ 8:05 am

    See you there ;-)

    Grazer Linuxtage 2009


    OpenOffice and Freefonts

    Filed under: — jimmy @ 2:55 pm

    Today I tried an upgrade of my debian unstable machine. Didn’t do this for several months, so I expected some problems, but this one was really hard to solve. I already had openoffice 2.4 installed, so there weren’t many changes, but after the complete upgrade the free fonts disappeared (FreeSans, FreeMono, …). The font drop-down menu showed three blank lines and I was also able to type in the word “FreeSans” into the box, but all I got were blank spaces. I googled around for two hours and finally I found a hint about the ttf-freefont package: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=472830. I don’t understand exactly what’s wrong, but installing version 20060501cvs-12 from snapshot.debian.net solved the problem and the fonts get now rendered correctly.


    IBM Server x3350

    Filed under: — jimmy @ 9:35 pm

    A few months ago I bought the new x3350 Server. Previously the x3250 was my standard model for small solutions. But I was always missing some important features which are now available with the x3350. These are:

    • Hot-swap redundant power supplies
    • Dual Gigabit Ethernet for bonding
    • Light path diagnostics

    Thank you IBM for exactly implementing my wishlist ;-)

    When I installed Debian etch on the machine everything worked fine except the broadcom NICs. They were not detected so I had to use a newer kernel.
    The problem was solved when “Etch and a half” was released because the 2.6.24 kernel includes a newer tg3 module.


    grml 1.1 and IBM Server x206m

    Filed under: — jimmy @ 11:06 am

    Remember the x206m and its “problematic” SAS/SATA Controller? ;-) Last night I tested the aic94xx driver again, which is also shipped with grml. Former releases of the driver worked with SAS but not with SATA. grml 1.1 comes with kernel 2.6.23, so worth a try. Unfortunately it’s still not clear if the firmware for the driver is licensed under GPL or Adaptec’s license. So it’s currently not included in the grml release but you can download it from here. Copy the file to /lib/firmware, unload and load the aic94xx module again. Watch the syslog and you will see, that the harddisks will be detected. Also tested it with a 500 GB IBM SATA disk. Works, too! :-)

    So the next step is to upgrade my x206m to etch and find out if it will work with the standard etch kernel, too.


    The forgotten posting: Debian etch on IBM x3200 and x3250

    Filed under: — jimmy @ 6:28 pm

    Yes, this posting is called “forgotten posting” because I just realized that I never posted it, although i should had been done a long time ago :-(

    A lot of people had problems with the Adaptec Controller in the x206m model. Fortunately, the successor x3200 has a LSI controller on board which works perfectly when running Debian etch on it. It uses the Fusion MPT drivers and best of all: It’s not a fakeraid controller, it’s real HW-RAID (thx to mika for pointing me in the right direction, because I never thought about it :-) ).

    Notes for using HW-RAID: There’s a small cmdline utility on the IBM CDs called “cfggen”. With this tool you can manage your RAID volumes. The userland utility (daemon) mpt-statusd monitors your RAID volumes (it’s included in Debian etch).

    Notes about Debian woody: My x3200 was running a long time with Debian woody. You just need to install a newer kernel so that the controller gets recognized.

    All of the information above applies to the x3250, too. It’s the rack version of the x3200.