Codestrom - The Wandering Programmer

Fixing libtool on Solaris / OpenSolaris

If you're building a lot of software on Solaris/OpenSolaris by hand.. You may have missed one small thing that could later come back and bite you.. If you're reading this you may already have seen libtool end up killing your build with something like blah bla ELFCLASS32

ld: fatal: file /usr/sfw/lib/libstdc++.so: wrong ELF class: ELFCLASS32

You'll notice it only happens with -m64.. So what to do?  Well the easiest fix and I guess standard policy for Debian and not OSUNIX is to remove all *.la files.  When libtool finds a la file it gets so excited that it wants to include the full path to the lib.  In this case the .la file it found was for 32bit, but -m64 is pretty clear.  By deleting all *.la files we let the linker be smarter than libtool and avoid these stupid errors entirely.

Quote about love

Love becomes so profound after you lose it and wonder if you'll ever feel this way again.  My advice.. Don't ever give up and try not to lose it..

Christopher Bergström

blobs, firmware and microcode in opensolaris/osunix

I've been working for months on OSUNIX (OpenSolaris technology distribution) to slowly refactor out the dependency on closed binaries.  The other day during the lastest build (onnv_115) the kernel package fails.  (No big surprise here really)  However, solving the failure wasn't nearly as interesting as digging into some of the files around the failure.

As it turns out.. There's a few binary files with binary license in this island called onnv-gate.  Looking a bit more into what others are doing..

OpenBSD may or may not be sane, but it's a project I have a lot of respect for.  To cite one of their developers and possibly answer my own question..

Reyk Floeter explained, "/there is a major difference between binary blobs and firmware images; the blobs are loaded as code into the OS kernel, but the firmware runs directly on the device on crappy embedded micro CPUs./"

So there's a distinction being made here for open software and open hardware.  Blobs possibly being a security risk, but firmware not executing directly on the host cpu and thus falling into a difference category or lower risk.

So then on there's the edge case of microcode updates.  Which generally happen during a bios upgrade, but for convenience can be loaded each time on boot..

usr/src/cmd/ucodeadm/amd-ucode.bin
usr/src/cmd/ucodeadm/intel-ucode.txt
i1480-usb-0.0.bin

Some possibly interesting links to help distinguish the thin line between tin hat and todo item.

http://kerneltrap.org/OpenBSD/That_Which_We_Call_Free
http://www.openbsd.org/lyrics.html#39
http://lkml.indiana.edu/hypermail/linux/kernel/0903.1/00335.html

So my goal today is to have everything be entirely open software.  In the future maybe this moves and the goal to have entirely open hardware is also possible.  I write these words now and wonder how I'll reflect back in the years to come.

Why can't RPM spec files be coverted to ebuilds

In evaluating converting the jpackage spec repository to ebuilds in an automated way I came across the following obstacles

    1) SourceNN: variables don't frequently contain a valid URL
    2) sub %packages
        # ebuilds have one inherit strength in that they typically map one tarball and one package per ebuild.  With this you are maintaining one source package per binary package.  To add or remove features USE flags were invented.  Since most end users are not compiling from source binary distributions invented ways to sub divide the packaged files into multiple binary packages.  The only benefit to this division of packages is saving bandwidth and space.

    3) Dependencies are loosely coupled and not globally tracked.
    4) ebuilds are uniquely mapped category + package and specs are just mapped by package.
        #Using the existing Group: would be possible, but more complicated than just adding a Category: variable

With this comes a couple questions...

1) How many Windows, Sun or IBM software packages come with separate package just for the docs?  (Yes some packages allow you not to install docs on the initial install)

2) When your XP updates are you asked if you want the docs and demo to go with it? No, you see a progress bar and just wait.  So there are two sides to this bandwidth and disk space question. 

    Remote server
    End user

I don't think most open source projects remote server is hitting a bottleneck for disk space or bandwidth.  Most successful projects are effectively using various forms of server and geo load balancing to distribute and localize.

So then comes the more questions...

3) Who is the end user and how do we invest our time so that they have the most efficient packaging as possible.

There are hundreds of thousands of software packages, but only a handful of package formats.  (RPM, dpkg, xpkg.. etc)  In my evaluation for which binary package to choose for OSUNIX I came across some very interesting numbers.  *If* distributions really wanted to save space and bandwidth they'd quit using gzip and bzip2.  lzma and xz (lzma2) showed in my early tests to significantly save space and were faster to extract.   Using xar for our oxe packages we won on size and speed vs dpkg and rpm even though xar compresses per file.

4) How often do people install the docs?

You may wonder why I care, but this is about saving time and giving the greatest benefit to the developers and end users.  There's 649 packages I wanted to use from the jpackage repo.  Roughly 2145 of the sub packages were either javadoc, manual or demo related.  How and why were the other packages being divided up?  ebuild package maintainers may have a lot of bad habits, but it's clear that there's no panacea to packaging.  It all goes back to having stronger policies in place that are based on logic and actually solving problems. </rant>

jpackage for opensolaris

Since OSUNIX is using ebuilds with pkgcore to build all packages it works really well with python/c/c++ code, but for java it's just plain ugly.  What to do?  I'm faced with the choice to either mangle and fight with what gentoo java devs are doing or to find some other sensible path.  So why not use jpackage?

1) It won't have any gentoo'isms
2) Distro agnostic
3) They are working on things like jdk6
4) maven2 already builds

So why not help a group that's on the right path?

They're using Koji which *really* sucks.. So if you can't get their cvs to work..

setenv CVSROOT :pserver:anonymous@jpackage.org:/home/projects/jpackage/cvs
cvs login (hit enter)
mkdir jpackage ; cd jpackage
cvs -z3 checkout .

From there you'll have a cvs checkout of their current repo.  All the stuff I'm interested in is in the **/devel directory.  So hopefully I can script convert this from spec files to ebuilds.  Then we'll have everything jpackage has!  (and a way to convert pkg5 spec files over to ebuild as well)

Icedtea6 on osunix aka fully (open)solaris

Small upfront disclaimer...

These instructions aren't really clean, organized and sometimes specific to a 64bit build.  (IOW don't complain if you see some dirty hacks)  Corrections welcome.. drop me a comment and I'll fix..

Things needed to build and get working in order in order.. gcj, ecj, jamvm-1.5.3 (read gotchas below) , classpath, ant.   From this you'll have enough to at least bootstrap the first part of icedtea6.

Prerequisites

Xorg - might as well build the whole thing.. until someone makes a --without-awt you'll need it..
Cups - Building cups is beyond the scope of this doc, but you'll need cups.h http://ftp.easysw.com/pub/cups/1.3.10/cups-1.3.10-source.tar.bz2
xalan.jar wget http://godel.cs.bilgi.edu.tr/apache/xml/xalan-j/xalan-j_2_7_1-bin.zip
serializer.jar wget http://godel.cs.bilgi.edu.tr/apache/xml/xalan-j/xalan-j_2_7_1-bin.zip
wget http://www.apache.org/dist/xerces/j/Xerces-J-bin.2.9.1.zip
Freetype, jpeg, gif and png libraries as well

Step 1 gcj

gcc is really evil, but we need to start somewhere..

This should give an idea of how I built gcc-4.4 prelease..

http://hg.osunix.org/osunix-overlay/src/tip/sys-devel/gcc/gcc-4.4-r2.ebuild

# Note : don't set target/build/host env variables
just change --enable-languages=c,c++ to--enable-languages=c,c++,java

# Note : gij is broken on (open)solaris so may as well delete it to avoid wasting your time..

Step 2 ecj

Pull ecj.jar

wget "http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/R-3.4-200806172000/ecj-3.4.jar&url=http://eclipse.mirror.kangaroot.net/eclipse/downloads/drops/R-3.4-200806172000/ecj-3.4.jar&mirror_id=468"

gcj -findirect-dispatch -m64 -o native_ecj --main=org.eclipse.jdt.internal.compiler.batch.Main ecj.jar

# Now you'll have a native 64bit ecj that you can build other things with..

Step 3 jamvm

Change a couple solaris x86 lines in configure to x86_64
mv -f solaris x86 sub directory to x86_64
CFLAGS=-m64 LDFLAGS=-m64

# Some jamvm gotchas..
# 1.5.2 has a 64K stack size by default.. When generating some files you'll eventually hit a stackoverflow unless you add -Xss256K or higher..  That's why 1.5.3 should work out of the box.. Also.. if you do hit a stackoverflow the generated file is probably corrupted.
# jvmtiEnterTrace.cpp won't be generated correctly.. I'm not certain the point of failure, but you'll need to download it from somewhere
# After the bootstrap ant needs to find build.properties file.  If using jamvm it doesn't seem to find it and the build will fail.  Switching to the just bootstrapped java should fix it.

Step 4 GNU Classpath

More evil gnu saves the day..
# You can download the prebuilt one, but the location of ecj.jar for it will not be the same and thus later mess things up.  Adjust --with-ecj-jar=/usr/local/src/ecj.jar to your location..

./configure --prefix=/usr/local/classpath --with-glibj-zip=/usr/local/classpath/share/classpath/glibj.zip --with-pic --disable-gconf-peer --enable-jni --with-ecj-jar=/usr/local/src/ecj.jar --without-x --disable-plugin --disable-gtk-peer  --disable-examples --disable-gjdoc 'CFLAGS=-m64' 'LDFLAGS=-m64'

Step 5 ant (which is really evil)

I personally loath ant.. So with that out of the way we have to get it working..

# Note : disable junit first http://ant.apache.org/faq.html#170-requires-junit
# Before the icedtea6 was bootstrapped I dirty hacked the ant shell script to set JAVA_HOME inside it so I didn't have to export to the env.  After icedtea6 is removed you must remember to remove this.

/usr/local/src/jamvm/bin/jamvm -classpath /usr/local/src/apache-ant-1.7.1/lib/ant-launcher.jar -Dant.home=/usr/share/ant-core -Dbuild.compiler=gcj org.apache.tools.ant.launch.Launcher

./dist/bin/ant -Dant.install=/usr/local/src/ant install

Step 6 icedtea6 (yay!)

# If you're using (t)csh.. bash and normal shell users feel free to adjust

setenv ALT_CUPS_HEADERS_PATH /usr/include
setenv ALT_FREETYPE_LIB_PATH /usr/lib/amd64
setenv ALT_FREETYPE_HEADERS_PATH /usr/include/freetype2
setenv XARCH '-m64'
setenv ARCH amd64
setenv ARCH_DATA_MODEL 64
setenv MAKE /usr/bin/gmake
# Note : If you set to 5.10 which is what Sun Studio Express currently is it'll not pick up the most recent cflags in some places.
setenv ENFORCE_COMPILER_REV '5.9'
setenv ENFORCE_C_COMPILER_REV '5.9'
setenv ALT_COMPILER_PATH /opt/SunStudioExpress/bin
setenv NO_DOCS true

mkdir mybuild ; cd mybuild

# Note : adjust accordingly

../configure --prefix=/usr/jdk/icedtea6 --disable-cacao --disable-liveconnect --disable-pulse-java --with-gcj-home=/usr/lib/gcj-4.4.0-10 --with-ecj=/usr/bin/ecj  --with-ecj-jar=/usr/local/src/ecj.jar --with-xalan2-jar=/usr/local/src/xalan-j_2_7_1/xalan.jar --with-xalan2-serializer-jar=/usr/local/src/xalan-j_2_7_1/serializer.jar --with-xerces2-jar=/usr/local/src/xerces-2_9_1/xercesImpl.jar --without-rhino --disable-visualvm --with-ant-home=/usr/local/src/ant --with-java=/usr/local/src/jamvm/bin/jamvm --with-pkgversion=Icedtea6-1.4.1  'CC=/opt/SunStudioExpress/bin/cc' 'CFLAGS=-m64 -I/usr/include/cups -I/usr/local/classpath/include' 'LDFLAGS=-m64'

# depending on your version you may also need
# --with-libgcj-jar=/usr/share/java/libgcj-4.4.0.jar

# The big list of problems and stuff..
1) Where you point JAVA_HOME should have a jdk like structure.  There's the gcj-jdk redhat package that sets this up, but it didn't build/work so I did it manually.. ymmv

# Note : The dirty hacks in no particular order...
cd /usr/lib/gcj-4.4.0-10/jre/lib/
ln -sf /usr/local/classpath/share/classpath/glibj.zip rt.jar

# Note : can't find Invocable?
cp ./lib/rt/javax/script/Invocable.class ./lib/hotspot-tools/javax/script/
/usr/gcc/temp/bin/gjar cf bootstrap/jdk1.7.0/jre/lib/tools.jar -C lib/hotspot-tools com -C lib/hotspot-tools sun -C lib/hotspot-tools org -C lib/hotspot-tools java -C lib/hotspot-tools javax

# The arch targets are all screwed up and confused because of amd64
cd /usr/local/src/icedtea6-1.4.1/mybuild/openjdk-ecj/control/build/
ln -sf solaris-amd64 solaris-i586

# Test fails.. Just make sure it doesn't..
cd /usr/local/src/icedtea6-1.4.1/mybuild/openjdk-ecj/jdk/make/sun/xawt

--- Makefile.old    2009-05-01 08:38:56.261848810 -0700
+++ Makefile  2009-05-01 08:31:33.127737691 -0700
@@ -248,6 +248,7 @@
    fi
    @if [ "$(DOCOMPARE)$(suffix $@)" = "true.64" ]; then \
      $(ECHO) COMPARING $@ and $(STORED_SIZES_TMPL_$(PLATFORM)_$(LIBARCH)); \
+      cp $@ $(STORED_SIZES_TMPL_$(PLATFORM)_$(LIBARCH)); \
      $(DIFF) $@ $(STORED_SIZES_TMPL_$(PLATFORM)_$(LIBARCH)); \
    fi 

# You'll have some missing symbols from the mapfiles.. Removing those lets the build go further

Java_sun_java2d_xr_XRSurfaceData_XRIsDrawableValid /usr/local/src/icedtea6-1.4.1/mybuild/openjdk-ecj/control/build/solaris-amd64/tmp/sun/sun.awt.X11/xawt/mapfile-vers
Java_sun_java2d_xr_XRSurfaceData_flushNativeSurface /usr/local/src/icedtea6-1.4.1/mybuild/openjdk-ecj/control/build/solaris-amd64/tmp/sun/sun.awt.X11/xawt/mapfile-vers
Java_sun_java2d_xr_XRSurfaceData_XRSetGradientPaint /usr/local/src/icedtea6-1.4.1/mybuild/openjdk-ecj/control/build/solaris-amd64/tmp/sun/sun.awt.X11/xawt/mapfile-vers
Java_sun_java2d_xr_XRSurfaceData_setInvalid /usr/local/src/icedtea6-1.4.1/mybuild/openjdk-ecj/control/build/solaris-amd64/tmp/sun/sun.awt.X11/xawt/mapfile-vers

---
Java_sun_java2d_x11_X11SurfaceData_XResetClip /usr/local/src/icedtea6-1.4.1/mybuild/openjdk-ecj/control/build/solaris-amd64/tmp/sun/sun.awt/headless/mapfile-vers
Java_sun_java2d_x11_X11SurfaceData_XSetClip /usr/local/src/icedtea6-1.4.1/mybuild/openjdk-ecj/control/build/solaris-amd64/tmp/sun/sun.awt/headless/mapfile-vers
Java_sun_java2d_x11_X11SurfaceData_setInvalid /usr/local/src/icedtea6-1.4.1/mybuild/openjdk-ecj/control/build/solaris-amd64/tmp/sun/sun.awt/headless/mapfile-vers
Java_sun_java2d_x11_X11SurfaceData_XCreateGC /usr/local/src/icedtea6-1.4.1/mybuild/openjdk-ecj/control/build/solaris-amd64/tmp/sun/sun.awt/headless/mapfile-vers
Java_sun_java2d_x11_X11SurfaceData_isDrawableValid /usr/local/src/icedtea6-1.4.1/mybuild/openjdk-ecj/control/build/solaris-amd64/tmp/sun/sun.awt/headless/mapfile-vers
Java_sun_java2d_x11_X11SurfaceData_initOps /usr/local/src/icedtea6-1.4.1/mybuild/openjdk-ecj/control/build/solaris-amd64/tmp/sun/sun.awt/headless/mapfile-vers
---
Java_sun_java2d_xr_XRSurfaceData_initOps /usr/local/src/icedtea6-1.4.1/mybuild/openjdk-ecj/control/build/solaris-amd64/tmp/sun/sun.awt.X11/xawt/mapfile-vers

# Icedtea6 assume gnu cp and ln
ln -sfv can be converted to ln -sf
cp -a can be converted to cp -pPrf

# First time around you'll need to steal libjvm.so
mkdir -p /usr/local/src/icedtea6-1.4.1/mybuild/bootstrap/jdk1.6.0/jre/lib/amd64/server
cp ../../jamvm-1.5.3/src/.libs/libjvm.so /usr/local/src/icedtea6-1.4.1/mybuild/bootstrap/jdk1.6.0/jre/lib/amd64/server

# Manual symlink... (ugly hack)
/usr/local/src/icedtea6-1.4.1/mybuild/bootstrap # ln -sf /usr/local/src/icedtea6-1.4.1/mybuild/bootstrap/openjdk-ecj/control/build/solaris-i586 icedtea

# Since this is amd64 only..  bin/* won't have any commands.. they are all in bin/amd64/*
# cd location/of/your/bin
ln -sf amd64/* .

# On OpenJDK bootstrap use icedtea6 libjvm.so and the other jre bits

cp -rf ./openjdk-ecj/control/build/solaris-amd64/j2sdk-image/jre /usr/local/src/icedtea6-1.4.1/mybuild/bootstrap/jdk1.6.0/

# Didn't get copied over?
cp /usr/local/src/icedtea6-1.4.1/mybuild/bootstrap/jdk1.7.0/jre/lib/tools.jar /usr/local/src/icedtea6-1.4.1/mybuild/bootstrap/jdk1.6.0/lib/

# This needs to be changed to gas and use -64 as flag to build 64bit..
# As a quick fix you can cd to that directory and run gas manually

mybuild/openjdk/hotspot/make/solaris/makefiles/sparcWorks.make
AS   = /usr/ccs/bin/as

# ./test_gamma breaks stuff so once again disable the test

--- openjdk/hotspot/make/solaris/Makefile.old   2009-05-03 05:24:28.591298714 -0700
+++ openjdk/hotspot/make/solaris/Makefile       2009-05-03 05:24:45.635642249 -0700
@@ -236,35 +236,30 @@

 $(TARGETS_C2):  $(SUBDIRS_C2)
        cd $(OSNAME)_$(BUILDARCH)_compiler2/$@ && $(MAKE) $(MFLAGS)
-       cd $(OSNAME)_$(BUILDARCH)_compiler2/$@ && ./test_gamma
 ifdef INSTALL
        cd $(OSNAME)_$(BUILDARCH)_compiler2/$@ && $(MAKE) $(MFLAGS) install
 endif

 $(TARGETS_TIERED):  $(SUBDIRS_TIERED)
        cd $(OSNAME)_$(BUILDARCH)_tiered/$(patsubst %tiered,%,$@) && $(MAKE) $(MFLAGS)
-       cd $(OSNAME)_$(BUILDARCH)_tiered/$(patsubst %tiered,%,$@) && ./test_gamma
 ifdef INSTALL
        cd $(OSNAME)_$(BUILDARCH)_tiered/$(patsubst %tiered,%,$@) && $(MAKE) $(MFLAGS) install
 endif

 $(TARGETS_C1):  $(SUBDIRS_C1)
        cd $(OSNAME)_$(BUILDARCH)_compiler1/$(patsubst %1,%,$@) && $(MAKE) $(MFLAGS)
-       cd $(OSNAME)_$(BUILDARCH)_compiler1/$(patsubst %1,%,$@) && ./test_gamma
 ifdef INSTALL
        cd $(OSNAME)_$(BUILDARCH)_compiler1/$(patsubst %1,%,$@) && $(MAKE) $(MFLAGS) install
 endif

 $(TARGETS_CORE):  $(SUBDIRS_CORE)
        cd $(OSNAME)_$(BUILDARCH)_core/$(patsubst %core,%,$@) && $(MAKE) $(MFLAGS)
-       cd $(OSNAME)_$(BUILDARCH)_core/$(patsubst %core,%,$@) && ./test_gamma
 ifdef INSTALL
        cd $(OSNAME)_$(BUILDARCH)_core/$(patsubst %core,%,$@) && $(MAKE) $(MFLAGS) install
 endif

 $(TARGETS_KERNEL):  $(SUBDIRS_KERNEL)
        cd $(OSNAME)_$(BUILDARCH)_kernel/$(patsubst %kernel,%,$@) && $(MAKE) $(MFLAGS)
-       cd $(OSNAME)_$(BUILDARCH)_kernel/$(patsubst %kernel,%,$@) && ./test_gamma
 ifdef INSTALL
        cd $(OSNAME)_$(BUILDARCH)_kernel/$(patsubst %kernel,%,$@) && $(MAKE) $(MFLAGS) install
 endif

# More mapfiles..

Java_sun_java2d_xr_XRSurfaceData_XRIsDrawableValid /usr/local/src/icedtea6-1.4.1/mybuild/openjdk/control/build/solaris-amd64/tmp/sun/sun.awt.X11/xawt/mapfile-vers
Java_sun_java2d_xr_XRSurfaceData_flushNativeSurface /usr/local/src/icedtea6-1.4.1/mybuild/openjdk/control/build/solaris-amd64/tmp/sun/sun.awt.X11/xawt/mapfile-vers
Java_sun_java2d_xr_XRSurfaceData_XRSetGradientPaint /usr/local/src/icedtea6-1.4.1/mybuild/openjdk/control/build/solaris-amd64/tmp/sun/sun.awt.X11/xawt/mapfile-vers
Java_sun_java2d_xr_XRSurfaceData_setInvalid /usr/local/src/icedtea6-1.4.1/mybuild/openjdk/control/build/solaris-amd64/tmp/sun/sun.awt.X11/xawt/mapfile-vers
Java_sun_java2d_xr_XRSurfaceData_initOps /usr/local/src/icedtea6-1.4.1/mybuild/openjdk/control/build/solaris-amd64/tmp/sun/sun.awt.X11/xawt/mapfile-vers

---
Java_sun_java2d_x11_X11SurfaceData_XResetClip /usr/local/src/icedtea6-1.4.1/mybuild/openjdk/control/build/solaris-amd64/tmp/sun/sun.awt/headless/mapfile-vers
Java_sun_java2d_x11_X11SurfaceData_XSetClip /usr/local/src/icedtea6-1.4.1/mybuild/openjdk/control/build/solaris-amd64/tmp/sun/sun.awt/headless/mapfile-vers
Java_sun_java2d_x11_X11SurfaceData_setInvalid /usr/local/src/icedtea6-1.4.1/mybuild/openjdk/control/build/solaris-amd64/tmp/sun/sun.awt/headless/mapfile-vers
Java_sun_java2d_x11_X11SurfaceData_XCreateGC /usr/local/src/icedtea6-1.4.1/mybuild/openjdk/control/build/solaris-amd64/tmp/sun/sun.awt/headless/mapfile-vers
Java_sun_java2d_x11_X11SurfaceData_isDrawableValid /usr/local/src/icedtea6-1.4.1/mybuild/openjdk/control/build/solaris-amd64/tmp/sun/sun.awt/headless/mapfile-vers
----

Java_sun_java2d_x11_X11SurfaceData_initOps /usr/local/src/icedtea6-1.4.1/mybuild/openjdk/control/build/solaris-amd64/tmp/sun/sun.awt/headless/mapfile-vers

Towards the end one last dirty hack...
cp -prf openjdk/control/build/solaris-amd64/* openjdk/control/build/solaris-i586/

# Congrats! From here you should have icedtea6 served

----
References:
https://c1.complang.tuwien.ac.at/cacaowiki/OpenJDK

Andrew John Hughes - ebuilds
http://overlays.gentoo.org/proj/java/browser/java-overlay/dev-java/icedtea6

Big thanks to everyone in #openjdk on oftc