update: 20100503: It's been brought to my attention that there might have been a regression in one of the libraries, mesa, maybe, so that master might not be as reliable a reference point for this guide. so, just in case, here are the commit hashes that were used from the different projects:

• dri2proto: c34ce137fdb21fc9a52bb8d5a0c25e3c5d79e687
• drm: 6293152eb065016a2e5e4fcd047c2db5c2fb0f36
• linux-2.6 (nouveau master) f108a544dbc9b64b0b42cfc6e21bc4f5ffd38998
• mesa: dd3b98bdf67df9a5b1127bf7d50bbb6691597bfb
• xf86-video-nouveau: 13c10430ba8f7b0edff3ad8aae4a97672eea4a8f

for the non-git-savvy: git checkout -b mybranch <hash>

we now return to your regular broadcasting...

A week or two ago I figured it was about time that I took the plung and tried out the latest nouveau drivers from experimental on my laptop. Unfortunately, it meant bidding adieu to compiz (apart from the bling it does actually provide some pretty damned useful features). I thought I'd give it a shot at least for a while on principal, since I've always had to think of the nvidia drivers as a necessary evil, and their lack of proper xrandr support is really, really annoying.

Since then I haven't noticed any problems, apart from the following message after every suspend/resume:

[  931.213239] Uhhuh. NMI received for unknown reason 80 on CPU 0.
[  931.213240] You have some hardware problem, likely on the PCI bus.
[  931.213242] Dazed and confused, but trying to continue

No idea what's causing it, but otherwise the system is stable and functional. If anyone knows what can be done about this, I'd appreciate hearing from you.

My graphics card, by the way:

01:00.0 VGA compatible controller: nVidia Corporation G72M [Quadro NVS 110M/GeForce Go 7300] (rev a1)

In the time since then, the nouveau driver has made its way into unstable, and a lot of distros (ubuntu, fedora, ...) are even using it as the default driver. So before going any further with the 3d/compiz stuff, I highly recommend that you get the packaged version working first. This will not only make sure that the basic driver works, but it will also make recovery/rollback a lot easier. Since it's all packaged in the standard repositories, it's not much harder than installing a few packages and editing a few files... so come on give it a try

Getting the 2d driver working in Debian sid

Really, this is so easy it's hardly worth writing up, but i'm running with the -v flag this morning...

Make sure you don't have any reference to nvidia in /etc/modules or other places that might cause it to be automatically loaded. Then remove the proprietary glx bindings, which would otherwise interfere with the nouveau driver.

sudo apt-get remove nvidia-glx

Next, you'll need to install a kernel that includes updated drm support. Normally this would require a kernel >= 2.6.33.1, but the debian kernel maintainers have graciously backported this code to the 2.6.32-4 kernel package. note this isn't the default kernel version in testing/unstable at the time of writing so you have to explicitly install it along with the nouveau driver.

sudo apt-get install linux-image-2.6.32-4-amd64 xserver-xorg-video-nouveau

If you are already running this kernel and using the proprietary nvidia drivers, via module-assistant or otherwise, you /might/ need to uninstall them first. On my system I left them installed in the old kernel version but did not install them on the new kernel version.

Required xorg configuration

I don't believe that nouveau is picked by default if available at the time of writing, so you have to have the following in your xorg.conf:

sudo sh -c "cat > /etc/X11/xorg.conf" << EOF
Section "Device"
    Identifier     "nVidia Corporation G72M [Quadro NVS 110M/GeForce Go 7300]"
    Driver         "nouveau"
EndSection
EOF

This was all I needed to do, apart from rebooting into the new kernel. For those who don't want to venture outside the packaging system, this is about as far as you can go for the time being. Though really, if you're bothering to read this far along, you're probably more interested in what comes next

Getting 3d working (without voiding your distro warranty)

So I thought I'd give the 3d-accelerated gallium a go. Their wiki page certainly makes one think twice, but at this point I was seriously fiending to get my bling back (well, the compiz scale plugin, anyway) and figured it wouldn't hurt too much to try.

Via google and trawling around in various forums I found a number of how-to documents for getting Gallium3d on debian/ubuntu systems, but most of them recommended doing things in a way that wasn't easily reversible and that could totally hose your system in a not-so-hypothetical worst-case scenario. It's pretty hard to roll back a make install done into /usr, for example, and subsequent package upgrades can leave your system even worse off as they will probably clobber half of what you just did, leaving the other half totally broken. And if you have a laptop which requires the gui for networking (i.e. network-manager), then fixing/rolling-back the changes is not entirely trivial, since you'd likely have to re-download the various drm/mesa/xorg packages.

So enter the tried and true unix utility stow (apt-get install stow, of course). stow is a utility that lets you install software into arbitrary folders, and then manage the system-wide installation of the software via a farm of symlinks pointing at the installed location. It's probably more well known among the sysadmin crowd than the desktop user crowd, but it's incredibly useful for the task at hand as it will let us easily install multiple copies of different software in a way that's very easy to update and/or roll-back. RTFM stow(1) for more information.

So, on to the juicy stuff...

Install needed software, remove problematic packages, setup, etc

At least on my system, I had to remove libdrm-dev to keep things from accidentally linking against it instead of the updated version that we install later on below. There's probably a way to override this with CFLAGS but I figure this is simpler.

sudo apt-get remove libdrm-dev
sudo apt-get install stow build-essential xorg-dev git-core libtool mesa-common-dev automake autoconf 
mkdir ~/nouveau

Compile a linux-2.6.34 RC kernel

cd ~/nouveau
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
cd linux-2.6
git remote add -f nouveau git://anongit.freedesktop.org/nouveau/linux-2.6
git checkout -b nouveau-master nouveau/master
make menuconfig

Enable device drivers -> staging drivers -> nouveau. Make any other changes you might want to make (I just took defaults for everything else).

make deb-pkg

The linux-firmware-free debian package ships some of the same stuff from the generated linux-firmware-image package. I decided to opt for the upstream firmware, though I'm not sure that this is necessary:

sudo apt-get remove linux-firmware-free
sudo dpkg -i ~/nouveau/linux-image*.deb ~/nouveau/linux-firmware-image*.deb
sudo update-grub

Install a copy of dri2proto

cd ~/nouveau
git clone git://anongit.freedesktop.org/xorg/proto/dri2proto
cd dri2proto
./autogen.sh --prefix=/usr/local/stow/dri2proto-20100428
sudo mkdir -p /usr/local/stow/dri2proto-20100428
sudo make install
sudo stow -d /usr/local/stow dri2proto-20100428

Install a cutting-edge version of libdrm

cd ~/nouveau
git clone git://anongit.freedesktop.org/git/mesa/drm/
cd drm
./autogen.sh --prefix=/usr/local/stow/drm-20100428 --disable-intel --disable-radeon --enable-nouveau-experimental-api
make
sudo mkdir -p /usr/local/stow/drm-20100428
sudo make install
sudo stow  -d /usr/local/stow  drm-20100428

Likewise for mesa

cd ~/nouveau
git clone git://anongit.freedesktop.org/git/mesa/mesa
cd mesa
./autogen.sh --prefix=/usr/local/stow/mesa-20100428  --enable-debug --enable-glx-tls --disable-asm --with-dri-drivers= --enable-gallium-nouveau --disable-gallium-intel --disable-gallium-radeon --disable-gallium-svga --with-state-trackers=glx,dri --with-demos=xdemos,demos,trivial,tests
make
sudo mkdir -p /usr/local/stow/mesa-20100428 
sudo make install
sudo stow -d /usr/local/stow mesa-20100428

And now the xorg driver...

cd ~/nouveau
git clone git://anongit.freedesktop.org/git/nouveau/xf86-video-nouveau/
cd xf86-video-nouveau
./autogen.sh --prefix=/usr/local/stow/xf86-video-nouveau-20100428
make
sudo mkdir -p /usr/local/stow/xf86-video-nouveau-20100428
sudo make install
sudo stow -d /usr/local/stow xf86-video-nouveau-20100428

Configuration

Configure xorg to look in alternate location for xorg modules and set up DRI

sudo sh -c "cat > /etc/X11/xorg.conf" << EOF

Section "Files"
    ModulePath "/usr/local/lib/xorg/modules,/usr/lib/xorg/modules"
EndSection

Section "ServerFlags"
    Option "GlxVisuals" "all"
Endsection

Section "Device"
    Identifier     "nVidia Corporation G72M [Quadro NVS 110M/GeForce Go 7300]"
    Driver         "nouveau"
EndSection

EOF

I've /heard/ that disabling TV out is a necessity on some cards, though I haven't seen this myself:

sudo sh -c "cat > /etc/modprobe.d/nouveau.conf" << EOF
# not entirely sure this is necessary, but heard it helped with vsync
options nouveau tv_disable=1
EOF

Finally, the one thing that can not be done in /usr/local is making the nouveau DRI driver available. From what I could tell the path that mesa uses to find the DRI libraries is hard-coded (corrections welcome). Since you won't already have this file (otherwise, why would you be following along here?), just drop in a symlink to the file in /usr/local:

sudo ln -s /usr/local/lib/dri/nouveau_dri.so /usr/lib/dri/

Finally, reboot, and hope for the best

Riding the bleeding edge: how to upgrade later on

Again, stow really shows its usefullness here. Using drm as an example case, upgrading an individual component would look something like:

cd ~/nouveau/drm
git clean -dfx
git pull
today=`date +%Y%m%d`
./autogen.sh --prefix=/usr/local/stow/drm-$today --disable-intel --disable-radeon --enable-nouveau-experimental-api
make
sudo mkdir -p /usr/local/stow/drm-$today
sudo make install
sudo stow -d /usr/local/stow -D drm-20100428
sudo stow -d /usr/local/stow  drm-$today

Note how the old version is still installed, so you can switch components around pretty freely.

Abort! Abort!: How to roll back changes

In case your system becomes totally unusable, or you've decided that you just want to go back to the safe and cozy world of what's provided by the debian packages, simply do the following:

• Remove the ModulePath directive from your xorg.conf
• Remove the nouveau_dri.so symlink from /usr/lib/dri
• Possibly(?) boot into your previous kernel (I read grumblings about ABI breakage on lkml which might mean that you can't use the same libdrm with the old/new drivers, though I haven't had to find out).

Assuming that you're using the nouveau driver from unstable, this should be all that is necessary.

Conclusion

Overall impression

I am very, very happy with the resulting setup. I was led to believe that this whole thing was much more unstable and unusable than it actually is! While there are certainly some glitches and missing features, I feel comfortable enough using it that I don't have plans to switch back. But if I find myself in a situation where I can't use the new setup, rolling back shouldn't be a problem either. I figure I'm happy with what I have now and it will only get better.

What works, at least to a "kinda works" level.

• A standard compiz installation
• Basic 3d support
• Xvideo
• Display port switching with fn-F7 (but see below)
• Suspend/Resume (most of the time, anyway)

What still doesn't work

• Suspend/Resume doesn't /always/ work, I wouldn't call it 100% reliable anyway
• I've had an occasional graphical lockup of the system requiring reboot
• Switching display ports only partially works (one of the two displays will usually be corrupted, though the other will work ok).
• Docking and undocking combined with suspend/resume and switching displays seems to trigger some hard lockups.
• Haven't tried extensive use of 3d-apps yet.
• Graphics glitches/artifacts
• A number of things will make compiz crash. When this happens it automatically falls back to metacity, so it's not a huge problem--just pretty annoying.
  • gnome-display-properties
  • using the compiz "Resize Window" plugin with "Normal" (change it to "Rectangle")
  • switching from lcd to external monitor will sometimes cause it to crash
  • I would guess other plugins might be problematic as well, though at least what I had enabled previously seems to work okay.
• Any kind of GPU power saving features (graphics card runs pretty hot, anyway).

Further reading, references

• DRM installation guide
• Kernel modesetting tips
• Gallium3D howto
• Tips for lockup recovery
• An earlier ubuntu how-to

Thanks

A couple shoutouts to those who helped me with this on the way:

• all the developers involved in the respective projects... a big Thank You!
• ymanton, mwk, calim, paerley on #nouveau for helping me troubleshoot kernel/libdrm issues.
• KiBi on #debian-x for all the work getting nouveau into unstable.

Feedback / Corrections

Unforunately, I've had to disable comments for the time being due to overwhelming quantities of spam. So please drop me a line via email or on IRC (seanius on oftc/freenode).

this computer is in use and can not be unlocked

this weekend while giving a bit of TLC to the php packages, i thought i'd finally get around to tackling #490023 and similar bugs, which meant learning about how to use the triggers feature provided by dpkg.

unfortunately documentation for this feature was in fairly short supply, and after extensive searching (and by extensive searching i mean typing "dpkg triggers howto" into a google search) i had a couple short manpages (dpkg-trigger(1) and deb-triggers(5)) and an overly verbose and possibly out of date /usr/share/doc/dpkg/triggers.txt.gz. none of these documents really gave a clear "big-picture" for how to get going either.

so, as is often the case, i got distracted from my intended task and ended up on a little side-project putting together a nice howto/tutorial for how to integrate dpkg triggers into a package. note that there may be misunderstandings or inaccuracies in the document, as it's based on an afternoon's worth of hacking and q&a directed at #debian-dpkg (thanks to Guillem and Raphaël for fielding those questions). so if i'm off on anything, please feel free to let me know!

to get started:

git clone http://git.debian.org/git/users/seanius/dpkg-triggers-example.git

and for those who are too lazy even for that, the README follows:

The debian package triggertest

This package is a dead-simple "triggers in a nutshell". It shows a few different ways that triggers can be used in practice, to hopefully cover most of the general use cases for triggers.

The purpose of triggers

Triggers are used to ensure that during the standard package-management process certain operations always run, but not more than necessary. For example:

• during an upgrade many packages might provide updated texinfo(5) documentation, and triggers are used to ensure that update-info-dir(8) is run without having to be run several unnecessary times from the maintainer scripts of package in question.

• if a system has both apache2 and PHP's apache2 engine installed, apache2 will need to be restarted whenever new php extensions are installed or removed. Without triggers it would be very disruptive (and inefficient) to explicitly stop and start apache by each PHP extension.

• update-initramfs(8) should be run once and only once after kernels are updated or certain other packages are installed/upgraded that would want to have a new initramfs generated.

• other expensive operations that are desirable to be aggregated and consolidated during installs/upgrades, such as update-texmf(8) or scrollkeeper-update.

How triggers work in a nutshell

Trigger-using packages can be classified in two behavioral categories:

• Consumers: packages which declare triggers and thus can be "triggered"
• Producers: packages which activate triggers (explicitly or implicitly)

When a consumer is triggered, its postinst script is run with the arguments:

postinst triggered "<trigger1> ... <triggerN>"

i.e. $2 contains an iterable list of activated triggers.

Note that if a consumer is going to be normally configured (i.e. it is also being updated), then no triggering may occur and thus the standard control flow of the maintainer scripts should still take care to handle this.

A "trigger" is declared for the consumer by shipping a file in DEBIAN/triggers (in the case of debhelper based packages, ./debian/consumer-package.triggers will JDTRT). Useful RTFM: deb-triggers(5).

Example:

interest /path/to/a/directory
interest my-second-trigger

This declares that the package in question is interested in two triggers.

The first trigger has a leading path separator in its name, which instructs dpkg that any filesystem modifications underneath this directory by other packages should cause the respective consumer(s) to be triggered.

The second name is completely arbitrary (and also global, so you should probably pick something reasonably specific and identifiable), and will only cause a trigger to be activated when a producer

1. ships its own triggers file with a corresponding "activate my-second-trigger", or
2. invokes the trigger explicitly in a maintainer script by calling "dpkg-trigger my-second-trigger".

Another perhaps useful RTFM: dpkg-trigger(1).

Examples contained in this source package

This source package produces a single "consumer" package and a number of different "producer" packages. The consumer is triggered in various ways, such as:

• Filesystem updates to a given directory from any other package
• "activate foo" from a producer's triggers file
• Activation from a producer's explicit call to dpkg-triggers

In this case the triggers correspond to an "update-foo" and "update-bar" command, which do nothing other than echo to the console that they are being run. In practice they'd correspond to updating some kind of database or restarting a daemon.

More information

For more information about triggers, see /usr/share/doc/dpkg/triggers.txt.gz, which will make a lot more sense having a working example like this package as reference. Especially useful is "Transition hints for existing packages", which can be used if you need to gracefully handle migrating in support for triggers across other packages not directly under your control.

Corrections, suggestions, etc

Are appreciated and should be sent to the author, below.

-- Sean Finney seanius@debian.org Sat, 19 Sep 2009 15:57:48 +0200

thanks largely to Peter Palfrader and Bernd Zeimetz, the debian patch-tracker is back alive and now hosted as an official debian.org service: http://patch-tracker.debian.org.

in addition to the new host, i've made a few very basic optimizations that will hopefully reduce the strain when someone browses through some of the larger patch sets (ehem, openoffice.org, i'm looking at you).

if you notice any problems with the service, please drop me a line.

continuing on the trend of finding novel ways to make life more interesting with git, here's the description from a new hook:

# prepare-commit-msg hook for debian package git repositories
#
# this script scans the diff that is going into a commit, and automatically
# injects some "proposed" comments based on what it finds in the diff.  this
# can be used to avoid a few extra keystrokes when performing some of the
# more standard/boring tasks.

(see below for how to fetch the script)

this one definitely falls in the "carrot" category, as it encourages properly isolated (and thus automatically identifiable) changes. the proposed comments are then given to the standard editor, so one can easily amend them, append "Closes:" lines, etc.

some sample use-cases currently implemented:

• changelog entries

  • detect when a new version is introduced in debian/changelog

    prepare release information for <dist>/<vers>
    
    

  • detect when an existing version is updated in debian/changelog

    release information for <dist>/<vers>
    
    

• debconf translations

  • detect when one or more po files are modified

    updates to (<lang1>,<lang2>...) debconf translations
    
    
     * <lang1>: <translator for lang1>
     * <lang2>: <translator for lang1>
    
    

• updates in debian/patches

  • detect when a new patch is introduced

    new debian patch <patchname>
    
    
    <patch header (anything before the diff)>
    
    

  • detect when an existing patch is modified

    update debian patch <patchname>
    
    
    <patch header>
    
    

  • detect when several patches are imported simultaneously

    import <n> files in debian/patches
    
    
    <patch1>:
      <patch header>
    <patch2>:
      <patch header>
    <non-patch-file (series/etc)>:
      no patch description/comments found
    ...
    
    

  • detect when several patches are created/modified simultaneously

    create/update <n> files in debian/patches
    ...
    
    

as always, comments/feedback/suggestions/etc welcome

using this new hook

note this is the same repo as the previous hooks i've blogged about, so if you already have that set up you can skip the clone and instead just pull in the changes. also note that this is in your local repo, not the remote one.

to set it up:

REPO_PATH=/path/to/your/repo.git
HOOK_REPO_PATH=/somewhere/you/want/to/put/it
git clone git://git.debian.org/users/seanius/vcs-hooks/git-hooks.git $HOOK_REPO_PATH
ln -sf $HOOK_REPO_PATH/debian/git-hooks/prepare-commit-msg-guess-message.py $REPO_PATH/.git/hooks/prepare-commit-msg

there aren't currently any configurable options in this hook.

previously i threw together a small hook to make it a bit easier to avoid duplicate work while maintaining packages, as well as easily keep the BTS up to date with relevant information.

now i've commonized the code just a bit and have a second hook which can be used to maintain certian (what i believe to be) good practices for keeping packages in git. admittedly, it's a bit more "stick" than "carrot" with respect to streamlining workflows, but i feel the justifications and the resulting benefits are worth it.

so the hook does basically two things, either of which can be customized and/or disabled.

prevent "non-debian" changes on a "debian" branch

assuming that there are seperate branches for "debian" packaging and for "upstream" development, this hook prevents upstream-style changes on the debian branches. that is to say changes to files outside ./debian are not permitted on a "debian" branch, unless of course you're merging from an upstream branch.

instead, changes to the source in a debian branch should be managed by quilt-style "feature patches", or a more advanced branch topology using some kind of feature branches (topgit or similar).

prevent changelog modifications from being mixed in with other commits

this one might be a bit more controversial for some. basically, the idea is that:

• stuff is fixed in a commit
• there is a log message for this commit, which contains a description of the fix
• the changelog update is redundant information to this log
• it's impossible to merge/cherry-pick/revert the fix later if the changelog gets tangled into the commit.
• there are nice tools (git dch) for managing debian/changelog updates anyway.

therefore, this hook "declines" commits which modify debian/changelog, unless it is the only file being changed.

using this new hook

note this is the same repo as the other hook, so if you already have that set up you can skip the clone and instead just pull in the changes.

to set it up:

REPO_PATH=/path/to/your/repo.git
HOOK_REPO_PATH=/somewhere/you/want/to/put/it
git clone git://git.debian.org/users/seanius/vcs-hooks/git-hooks.git $HOOK_REPO_PATH
ln -sf $HOOK_REPO_PATH/debian/git-hooks/pre-receive-fileset-fascism.py $REPO_PATH/hooks/pre-receive

the config options for controlling this hook (these are git config options just like the other hook):

# hooks.debianbranches (default: 'debian-.*')
#    a regular expression which indicates which branches are "debian" branches.
#    in the context of this hook such branches are not allowed to have changes
#    in files outside of the ./debian directory.
# hooks.sacredchangelog (default: True)
#    if set to True, debian/changelog can not be changed in a commit that
#    also modifies other files.  this helps ensure changes that are easily
#    merged/cherry-picked/reverted.

examples of the hook in action

rejecting a commit that has an entangled changelog:

rangda[~/debian/php] git push                                  :)
Counting objects: 9, done.
Delta compression using 2 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 442 bytes, done.
Total 5 (delta 4), reused 0 (delta 0)
fileset-fascism rejecting commit 2fab3269b3d9daedd7013a576483a11ccb4cb86a
    debian/changelog must be changed seperately from other files.
    changed files in this commit:
        debian/changelog
        debian/control
error: hooks/pre-receive exited with error code 1
To ssh://git.debian.org/git/pkg-php/php.git
 ! [remote rejected] debian-sid -> debian-sid (pre-receive hook declined)
error: failed to push some refs to 'ssh://git.debian.org/git/pkg-php/php.git'

rejecting a commit that has non ./debian changes:

rangda[~/debian/php] git push                                  :)
Counting objects: 5, done.
Delta compression using 2 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 309 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
fileset-fascism rejecting commit d96669c0780e33b250af81404e263fc181fe0547
    non ./debian changes on a debian branch in this commit.
error: hooks/pre-receive exited with error code 1
To ssh://git.debian.org/git/pkg-php/php.git
 ! [remote rejected] debian-sid -> debian-sid (pre-receive hook declined)
error: failed to push some refs to 'ssh://git.debian.org/git/pkg-php/php.git'

as always, comments/feedback/suggestions/etc welcome

(note that i've just tried enabling the comments feature of ikiwiki, so feel free to use it )

(update: okay so i apparently failed to enable comments earlier... but it should be working now )

compiz(-fusion) 0.8.2 uploaded to unstable

for all you lovers of desktop bling, the latest stable release of compiz-fusion has been uploaded to unstable. new features of note:

• new kde4 based decorator and kconfig4 module for the kde4.2 desktop
• better stability for suspend/resume (that might be a driver improvement though)
• gnomecompat support for importing keyboard shortcuts
• fix for xdamage/refresh problems on some nvidia cards
• zomg more bling plugins

new one-off script for building compiz packages

the biggest problem with trying to maintain compiz packages is that they're for some reason distributed as almost a dozen different source projects upstream, but extremely interdependant wrt their API. therefore updating the packages to a new upstream version is a Big Pain. i've numbed that pain a bit with a new helper script, which should amongst other things make it easier to manage backports to lenny, which i've already seen people requesting.

compiz 0.8.2 lenny packages available for testing

..which brings us to our next topic. before i upload them to somewhere a little more official, i figure i could put a call out to teh lazywebs to do some QA work and make sure that it doesn't totally explode on your system (unless you're using the explode animation plugin, in which case it's functioning properly)

sources.list entry to try it out on an amd64 lenny system:

deb http://people.debian.org/~seanius/compiz/lenny-backports/amd64 ./

sources.list entry to try it out on an i386 lenny system:

deb http://people.debian.org/~seanius/compiz/lenny-backports/i386 ./

and then apt-get install compiz compiz-fusion-plugins-main (etc)

so give it a try and let me know how it goes!

pkg-php moving to git

with some initial legwork from Mark Hershberger we've migrated to a git repository for the php packaging vcs. due to the long and convoluted history of our svn repository, we decided it wasn't worth the effort of trying to get every single svn commit mapped into the new repo. as a compromise to having to start from scratch, we used the existing branches/tags on top of a series of upstream tarball imports to get an accurate representation of the release history.

new git<->bts integration hook for debian packaging

i've been dabbling with a new hook to make for more useful integration from a git-buildpackage oriented workflow. in this workflow the changelog is often the last thing prepared, possibly days or even weeks after a fix has been committed for a bug. therefore the standard tagpending and typical changelog-scanning hooks aren't incredibly useful.

instead, this hook scans for "Closes:" meta-info in the commits, similar to how "git dch" does, and then sends a notification and/or control commands to the bts. here's an example. note that it also mentions the branch that recieved the fix in the notification, so it's easier to see which branches have a fix at a quick glance.

how to use this for your own git repo:

REPO_PATH=/path/to/your/repo.git
HOOK_REPO_PATH=/somewhere/you/want/to/put/it
git clone git://git.debian.org/users/seanius/vcs-hooks/git-hooks.git $HOOK_REPO_PATH
ln -sf $HOOK_REPO_PATH/git-post-receive-url-notifications.py $REPO_PATH/hooks/post-receive

there's a number of configurable options (take a look at the top of the file for some fairly verbose comments), but it should work with some fairly reasonable defaults out of the box.

it can also be configured to do the more traditional changelog-scanning, but i'm finding it to be a better workflow to avoid combining changelog entries with the actual fixes (makes them harder to cherry-pick later), and a notification with a link to the changelog entry really isn't that useful beyond what "bts tag nnn pending" can already do. plus i don't like the typical duplication involved in manually managing the changelog. i find it better to generate the changelog via git-dch and automatically get all the "closes:" tags, and then just do a bit of editorial touchups before preparing/releasing the upload.

so give it a try if you like, feel free to send feedback/fixes/etc. for example i think the utf-8 support might still be a bit dodgy, as well as possibly the automatic gitweb url detection and changelog scanning.

now that ps3-utils is packaged, and it seems spufs didn't take much to get to work, i'd like to turn back to the installation procedure, which was admittedly manual and not exactly as "seamless" as i think it could be.

the first step as i see it is to get the bootloader working both generally and also specifically for the installer. by "working", i mean the following:

• that it can be used to start the installer
• that it can be used to boot the system, of course
• that it can be built within debian

the third of these being the most complicated, as will be detailed below.

how a linux (kboot) "otheros" bootloader works on the ps3

okay, so i'm not intimately familiar with the powerpc architecture, nor with all the details of cross-compilation, but this is as much as i can tell.

• the file "otheros.bld" which you use to start the installation is a compressed linux image with a built in ramdisk initrd.
• it is a custom generated image made by kboot, which includes specialized versions of busybox and kexec-tools among others.
• it is compiled with a locally bootstrapped compiler and uclibc library.
• the initrd boots to a kboot script/prompt, which will then find and load the "real" or "system installed" kernel off of the disk.
• you can also specify a different kernel to use, different boot params, etc.
• you can also optionally get to a limited shell environment to poke around.

how kboot can be used on the installer

including support for this "otheros.bld" bootloader on the installer is fairly easy from a technical point of view, whether it's a usb disk or netinst cdrom image. basically, in either case the media should have a directory called "ps3/otheros", in which the file is placed and named "otheros.bld". when running the PS3 in game-os mode, this is where it looks when you tell it to "scan for otheros".

how kboot can be used/maintained, generally speaking

beyond a kboot.conf file, nothing really needs to be done on the host os to use kboot. of course, it would be nice to be able to issue updates for an image, which should be possible with a package that makes use of ps3-flash-util from the ps3-utils package. nothing too difficult/complicated there.

how kboot can be used/built within debian

this is the tricky part. in additional to the standard Free Software guidelines which debian is so (in)famous for following, we also have some basic QA rules that are needed from a distribution release management / security perspective.

one of these rules is that generally speaking it is considered taboo to include embedded copies of other software within a package, if that software is already available in debian. for example, i don't think it's very likely that the ftp-masters will look very kindly on some new software package that includes a copy of the entire kernel source tree, the gcc compiler, (uc)libc, busybox, coreutils, udev, etc.

it may be that a certain subset will require special patches, in which case some convincing/justification will be in order, but for the big ones (i.e. the kernel) there will almost certainly need to be modifications made.

ps3-kboot in ubuntu

there are already source packages available for ps3-kboot in ubuntu. apart from the fact that i can't stand working with packages that use cdbs, this package can't be used because the package maintainers' solution to the previously mentioned problem is basically to ignore that it exists.

it also seems that there's quite a bit of customization in this source package with respect to the initrd generation, which is probably worth some review... so i will likely still use this package as a base point of reference.

ps3-utils

is packaged and uploaded now, and will be in debian after making it through the initial processing of NEW. i ended up using the ubuntu ps3pf-utils package as the starting point, as it was the most recent package and was in reasonably good shape.

for tracking purposes: the here is the original ITP, and i've made the packaging generally available at an initial git repo for ps3-utils. if more people get interested/involved this location might change of to something more "official" looking.

bluetooth

i stumbled across another odd behaviour between the ps3 "game OS" and linux. it seems that if you have bluetooth daemons running under linux, your controllers will be disabled the next time you boot into the "game OS". i asked on a ps3/cell related mailing list that i recently joined, and got a confirmation that this was a problem. the suggested workaround at the moment is to disable the bluetooth daemons in linux while they find out what the problem is. i've also found that rebooting the ps3 an extra time from game OS to game OS again seems to resolve the problem (or maybe it fixes itself after a short while, i'm not entirely sure, but have eventually been able to use the controllers again).

so i've been looking into the state of things with running debian on my ps3, and while talking to some folks on #cell, it was brought to my attention that there was a problem or two with the libspe2 packages, used for programming SPU using applications.

for those not in the loop, the Cell Broadband Engine chip which powers the ps3 is a multi-core processor, with two general purpose PPU's and 6 specialized SPU's available. Without spufs support, the SPU's aren't available, and the real number-crunching capabilities of the ps3 aren't either.

more information on the nitty-gritty of spufs is available here

anyway, the problems are fixed now

still, as of now there's still some one-time setup after installation. namely:

    echo spu    /spu    spufs   defaults    0   0 >> /etc/fstab

(and then mount it or just reboot)

there really should be some way to have this setup automatically, either from the installer or through some kind of spufs support package. i'll take it up on the mailing lists and see what the consensus would be for this.