Codestrom - The Wandering Programmer

dedup rumors and now finally a patch

I'll let the logs speak for themselves.. Rumor is in snv_128 we may actually see zfs dedup!  Sweet!

Author: Jeff Bonwick <Jeff.Bonwick@Sun.COM>
Repository: /hg/onnv/onnv-gate
Latest revision: e2081f5023069120b7a160a1826f5bf80151828f
Total changesets: 1
Log message:
PSARC 2009/571 ZFS Deduplication Properties
6677093 zfs should have dedup capability

Files:
	create: usr/src/uts/common/fs/zfs/ddt.c
	create: usr/src/uts/common/fs/zfs/ddt_zap.c
	create: usr/src/uts/common/fs/zfs/sys/ddt.h
	create: usr/src/uts/common/fs/zfs/zle.c
	update: usr/src/cmd/filebench/Makefile.com
	update: usr/src/cmd/mdb/common/modules/zfs/zfs.c
	update: usr/src/cmd/sgs/Makefile.var
	update: usr/src/cmd/zdb/Makefile.com
	update: usr/src/cmd/zdb/zdb.c
	update: usr/src/cmd/zdb/zdb_il.c
	update: usr/src/cmd/zpool/zpool_main.c
	update: usr/src/cmd/ztest/ztest.c
	update: usr/src/common/avl/avl.c
	update: usr/src/common/zfs/zfs_fletcher.c
	update: usr/src/common/zfs/zfs_prop.c
	update: usr/src/common/zfs/zfs_prop.h
	update: usr/src/common/zfs/zpool_prop.c
	update: usr/src/common/zfs/zprop_common.c
	update: usr/src/grub/capability
	update: usr/src/grub/grub-0.97/stage2/fsys_zfs.c
	update: usr/src/grub/grub-0.97/stage2/zfs-include/spa.h
	update: usr/src/grub/grub-0.97/stage2/zfs-include/zap_impl.h
	update: usr/src/grub/grub-0.97/stage2/zfs-include/zfs.h
	update: usr/src/lib/libzfs/common/libzfs_dataset.c
	update: usr/src/lib/libzfs/common/libzfs_pool.c
	update: usr/src/lib/libzpool/common/llib-lzpool
	update: usr/src/uts/common/Makefile.files
	update: usr/src/uts/common/fs/vfs.c
	update: usr/src/uts/common/fs/zfs/arc.c
	update: usr/src/uts/common/fs/zfs/bplist.c
	update: usr/src/uts/common/fs/zfs/dbuf.c
	update: usr/src/uts/common/fs/zfs/dmu.c
	update: usr/src/uts/common/fs/zfs/dmu_objset.c
	update: usr/src/uts/common/fs/zfs/dmu_send.c
	update: usr/src/uts/common/fs/zfs/dmu_traverse.c
	update: usr/src/uts/common/fs/zfs/dmu_tx.c
	update: usr/src/uts/common/fs/zfs/dnode_sync.c
	update: usr/src/uts/common/fs/zfs/dsl_dataset.c
	update: usr/src/uts/common/fs/zfs/dsl_deleg.c
	update: usr/src/uts/common/fs/zfs/dsl_dir.c
	update: usr/src/uts/common/fs/zfs/dsl_pool.c
	update: usr/src/uts/common/fs/zfs/dsl_prop.c
	update: usr/src/uts/common/fs/zfs/dsl_scrub.c
	update: usr/src/uts/common/fs/zfs/dsl_synctask.c
	update: usr/src/uts/common/fs/zfs/lzjb.c
	update: usr/src/uts/common/fs/zfs/metaslab.c
	update: usr/src/uts/common/fs/zfs/sha256.c
	update: usr/src/uts/common/fs/zfs/spa.c
	update: usr/src/uts/common/fs/zfs/spa_history.c
	update: usr/src/uts/common/fs/zfs/spa_misc.c
	update: usr/src/uts/common/fs/zfs/space_map.c
	update: usr/src/uts/common/fs/zfs/sys/arc.h
	update: usr/src/uts/common/fs/zfs/sys/bplist.h
	update: usr/src/uts/common/fs/zfs/sys/dbuf.h
	update: usr/src/uts/common/fs/zfs/sys/dmu.h
	update: usr/src/uts/common/fs/zfs/sys/dmu_objset.h
	update: usr/src/uts/common/fs/zfs/sys/dmu_traverse.h
	update: usr/src/uts/common/fs/zfs/sys/dsl_dataset.h
	update: usr/src/uts/common/fs/zfs/sys/dsl_pool.h
	update: usr/src/uts/common/fs/zfs/sys/metaslab.h
	update: usr/src/uts/common/fs/zfs/sys/metaslab_impl.h
	update: usr/src/uts/common/fs/zfs/sys/spa.h
	update: usr/src/uts/common/fs/zfs/sys/spa_impl.h
	update: usr/src/uts/common/fs/zfs/sys/txg.h
	update: usr/src/uts/common/fs/zfs/sys/txg_impl.h
	update: usr/src/uts/common/fs/zfs/sys/uberblock.h
	update: usr/src/uts/common/fs/zfs/sys/vdev.h
	update: usr/src/uts/common/fs/zfs/sys/vdev_impl.h
	update: usr/src/uts/common/fs/zfs/sys/zap.h
	update: usr/src/uts/common/fs/zfs/sys/zap_impl.h
	update: usr/src/uts/common/fs/zfs/sys/zap_leaf.h
	update: usr/src/uts/common/fs/zfs/sys/zil.h
	update: usr/src/uts/common/fs/zfs/sys/zil_impl.h
	update: usr/src/uts/common/fs/zfs/sys/zio.h
	update: usr/src/uts/common/fs/zfs/sys/zio_checksum.h
	update: usr/src/uts/common/fs/zfs/sys/zio_compress.h
	update: usr/src/uts/common/fs/zfs/sys/zio_impl.h
	update: usr/src/uts/common/fs/zfs/txg.c
	update: usr/src/uts/common/fs/zfs/vdev.c
	update: usr/src/uts/common/fs/zfs/vdev_mirror.c
	update: usr/src/uts/common/fs/zfs/vdev_queue.c
	update: usr/src/uts/common/fs/zfs/vdev_raidz.c
	update: usr/src/uts/common/fs/zfs/zap.c
	update: usr/src/uts/common/fs/zfs/zap_leaf.c
	update: usr/src/uts/common/fs/zfs/zap_micro.c
	update: usr/src/uts/common/fs/zfs/zfs_fm.c
	update: usr/src/uts/common/fs/zfs/zfs_ioctl.c
	update: usr/src/uts/common/fs/zfs/zfs_log.c
	update: usr/src/uts/common/fs/zfs/zfs_replay.c
	update: usr/src/uts/common/fs/zfs/zfs_vnops.c
	update: usr/src/uts/common/fs/zfs/zil.c
	update: usr/src/uts/common/fs/zfs/zio.c
	update: usr/src/uts/common/fs/zfs/zio_checksum.c
	update: usr/src/uts/common/fs/zfs/zio_compress.c
	update: usr/src/uts/common/fs/zfs/zio_inject.c
	update: usr/src/uts/common/fs/zfs/zvol.c
	update: usr/src/uts/common/sys/avl.h
	update: usr/src/uts/common/sys/fs/zfs.h
	update: usr/src/uts/intel/zfs/spa_boot.c
	update: usr/src/uts/sparc/zfs/spa_boot.c

FAQ zfs deduplication

Q. When will ZFS have deduplication?

A. Just my view on ZFS + deduplication based on what I know..

• There is some legal bla bla between Sun and green-bytes that's tying up the IP around dedup... (someone knock some sense into green-bytes please) 
• there's an acquisition that's got all sorts of delays.. which may very well delay the thing with green-bytes as well
• Is it in the best interest of Oracle for ZFS to have dedup before btrfs?  (I doubt this is a reason, but it's a question to ask)
• Nobody has written the code.. patches welcome :)

Drop me a comment if I'm missing something


IANAL


./C

btrfs review

I've done a comparison of the feature sets for btrfs vs zfs, but an ex Sun employee really lays it down in a bit more technical details.  Superb job!

http://lwn.net/Articles/342892/

As someone who is very dedicated to the multitude of benefits OpenSolaris technology *currently* has over Linux I must say it'll be very interested if we end up with btrfs on OpenSolaris.  I would vastly prefer this to having ZFS on Linux.. Why.. because even though I *love* Linux (I do love the community)  I still really miss that proper engineering.. maybe in the end it's a tradeoff for long term sustainability.. I can live with that.. maybe linux is the prototype/poc which then later pushes the evolution back into "enterprise" (I use this word loosely) operating systems..

So I made a flippant entry a while back of why btrfs should be ported to OpenSolaris... Now looking deeper at the facts I think it becomes more and more a good business choice.  The problem as far as I know though is that it's tightly being coupled with the Linux kernel and maybe intentionally not being documented well.  Both of those combined would make it rather difficult to independently clean room implement.  Maybe we should kick off the btrfs documentation project.. and the BSD/OpenSolaris folks can implement it..

Any game to help out?

OpenSolaris vim vimrc file

I needed a quick vimrc today for OpenSolaris and instead of my typical one.. Just pulled one off the net and then made a few changes to it..

* Backspace / Delete should work as is normal in Linux

* Color syntax

* Remembers the cursor position

* Doesn't have that stupid visual mouse mode automatically set

.... other stuff you'd normally find in Linux..

Download my vimrc

HOWTO get virtualbox to redirect to a console COM1 / ttya for debbuging

This is a short HOWTO on getting virtualbox (vbox) to redirect output to COM1 aka ttya so you can connect via a "serial" connection outside of the virtual machine.  A few things you can use this for is logging the console output on boot or login from your normal work terminal of choice.

To start..

1) If you're on an opensolaris host.. You'll need socat or nc.  (Thanks go to Triskelios for pointing out nc will work as well)

If you're on ips

pkg install SUNWnetcat

or build socat-1.7 (2.x doesn't build and I haven't as of today had a chance to file a bug or send a patch.. someone should do this *cough* )

http://www.dest-unreach.org/socat/

(Not sure if it'll build with sun cc.. I was lazy and used gcc this time..)

2) Next configure your serial connection for the virtual machine..

enable serial ports

COM1

use host-pipe

create pipe

# then put the path ( I used /tmp/vboxlog )

3) When the virtual machine starts you'll see /tmp/vboxlog now exists

socat unix-client:/tmp/vboxlog stdout

or

nc -U /tmp/vboxlog

or

socat unix-client:/tmp/vboxlog stdout | tee boot.log

4) Modify your kernel line for grub to look like..

kernel /platform/i86pc/multiboot -B console=ttya,ttya-mode="115200,8,n,1,-"

----

From here you should be in business to catch the console output for the virtual machine when it boots.  Once you stop seeing output you can just connect without tee and login like normal..

I couldn't find this in the documentation anywhere so hope it help.. Let me know..

Small rants about lawyers and other developers

Lately OSUNIX has had two minor challenges.

1. Legal pre-review for CDDL libc + GPL code
2. Other developers making promises and not keeping them

I must say that for the first problem it's really coming down to lawyer bullshit.  From any lay perspective is *very* clear that libc + GPL userland code isn't going to be a derived work from the GPL userland code.  It's been requested that copyright holder for all the CDDL and GPL code linking together be sent for a risk assessment.  If knowing the open source landscape I really wonder to myself wtf this will accomplish. Why you may ask?

• SFLC is really the only lawfirm enforcing or suing others over GPL issues
• SFLC represents FSF which is probably the majority copyright holder of any critical GPL software

Basically SFLC on one side is requesting the exact list of software which if used by another client (FSF which they represent) would be the ones who would want to enforce the GPL.  I ask myself..  I mean seriously... is it just me or is it not obvious that lawyers are creating work for themselves here so that they appear to be doing something meaningful.  The real solution is for someone wearing the lawyer badge to step up and give a clear answer.  Then if anyone doesn't like it to call bullshit and or forever shutup and let developers get back to *cough* work..

Speaking of work... my 2nd small rant..

I don't care if this is open source or you're getting paid a lot.  If you say you're going to help.. help.. if you say you'll get something done.. get it done..  We all should be working as a team and able to count on each other.

GPLv2 and CDDL libc special exception

So this morning I've had two nagging questions bugging me..

Does an open source CDDL libc fall under a special exception of the GPL?

Is OSUNIX and Nexenta on the verge of dying because of legal issues?

Disclaimer: I'm not a lawyer and anything here shouldn't be taken as legal advice.

I'm rambling my ideas and woes as the project leader for OSUNIX which could along with other good distributions have a big problem soon.  (On this list.. Nexenta, Belenix and various other OpenSolaris technology containing distributions)  By technology I mean in this case specifically libc.  So it's been casually mentioned that "linking" GPL userland code + CDDL libc would make a derivative work.  So where to start.. The spirit of the GPL was best summed up by these three things below.

In Professor Moglen's mysql-affidavit [1]  he lists "three primary conditions"  of the GPL:

1. Redistribution must itself occur under GPL and only GPL, with no additional license conditions.
2. Redistribution must include "source code," the human-readable form of computer programs that allows programmers to understand and modify computer programs for themselves, as opposed to "object code," which is the "machine language" version of computer programs that is very difficult for programmers to understand or modify.
3. Redistribution must include a copy of the GPL, so that users are aware of their rights to use, copy, modify and distribute, and so that anyone engaged in redistribution is also aware of the conditions under which redistribution is permitted.

While the CDDL license is a more "liberal" license it's still incompatible with the GPL because of section 2.b and point #1 from above.  (someone correct me if I'm wrong)  However, GPLv2 [2] has a "special exception" in Section 3.c for major or system components.

"However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. "

So here's where it gets a bit fuzzy.. The compiler will generally include a small amount of "code" into your GPL code so that it can interact with the runtime and also of course your code will run on top of a kernel of some sorts.  However, section 3.c also has "only if you received the program in object code or executable form" which at the time it was written covered all the known special exceptions it needed to.  I guess they never imagined one of those proprietary operating systems would one day open source and do so under a GPL incompatible license.  So here's the two problems in a nutshell.  CDDL isn't GPL compatible and also CDDL libc which comes in source as well as binary form may not fall under the special exception.  So lets stop and ask ourselves one simple question.

So lets try to think as some draconian lawyer for a minute and also ask.. Is linking Application Foo with a system library that's so core as libc really making a derived work?  Umm.. and what about the clear exception for libstdc++ [3]  which was targeted specifically at a compiler, but in spirit shows that a c++ or c library is really a core system library that is special and should be under a "special exception"

In the spirit of the GPL would the original authors have factored in this special exception had they known about it?  I guess we'll soon find out since bdale of SPI has asked the SFLC to comment on this very matter.

One thing you have have noticed is that I didn't include OpenSolaris in my list of operating systems from above.  Why?  Well.. they don't actually distribute the source along with their core operating sytem under CDDL. They distribute it in binary only form under the OpenSolaris binary license.  By doing so they could very strongly argue they fall under this special exception.


What could this mean for the rest of us?

Good possibilities

• It's finally concluded that CDDL libc falls into the special exception category

Bad possibilities

• Nexenta / OSUNIX could die, be sued or receive some notice to cease and desist distributing GPL licensed code that links agianst CDDL libc
  

Unlikely (?) possibilities

• The exception is clarified to allow CDDL libc into this category
• FSF, Debian or someone really bold files a lawsuit (oh. I'd love this and all the bad PR for anyone filing that lawsuit which would certainly backlash against them)
  

As a side note.. Anyone within kicking range of a lawyer that writes these licenses..  Please give him/her a good one in the butt, because while I'm strongly in favor of open source and enforcing all source code be distributed. I see it as completely stupid that if source code *is* distributed that there should in any way shape or form be any problem with the packages being shipped together.  Yes of course we live in a world of Novell and MS agreements, patent trolls, and various other despots, but can't I dream?


[1] http://www.gnu.org/press/mysql-affidavit.html

[2] http://www.gnu.org/licenses/gpl-2.0.html

[3] http://www.gnu.org/licenses/gpl-faq.html#LibGCCException

Fixing libtool on Solaris / OpenSolaris

If you're building a lot of software on Solaris/OpenSolaris by hand.. You may have missed one small thing that could later come back and bite you.. If you're reading this you may already have seen libtool end up killing your build with something like blah bla ELFCLASS32

ld: fatal: file /usr/sfw/lib/libstdc++.so: wrong ELF class: ELFCLASS32

You'll notice it only happens with -m64.. So what to do?  Well the easiest fix and I guess standard policy for Debian and not OSUNIX is to remove all *.la files.  When libtool finds a la file it gets so excited that it wants to include the full path to the lib.  In this case the .la file it found was for 32bit, but -m64 is pretty clear.  By deleting all *.la files we let the linker be smarter than libtool and avoid these stupid errors entirely.

blobs, firmware and microcode in opensolaris/osunix

I've been working for months on OSUNIX (OpenSolaris technology distribution) to slowly refactor out the dependency on closed binaries.  The other day during the lastest build (onnv_115) the kernel package fails.  (No big surprise here really)  However, solving the failure wasn't nearly as interesting as digging into some of the files around the failure.

As it turns out.. There's a few binary files with binary license in this island called onnv-gate.  Looking a bit more into what others are doing..

OpenBSD may or may not be sane, but it's a project I have a lot of respect for.  To cite one of their developers and possibly answer my own question..

Reyk Floeter explained, "/there is a major difference between binary blobs and firmware images; the blobs are loaded as code into the OS kernel, but the firmware runs directly on the device on crappy embedded micro CPUs./"

So there's a distinction being made here for open software and open hardware.  Blobs possibly being a security risk, but firmware not executing directly on the host cpu and thus falling into a difference category or lower risk.

So then on there's the edge case of microcode updates.  Which generally happen during a bios upgrade, but for convenience can be loaded each time on boot..

usr/src/cmd/ucodeadm/amd-ucode.bin
usr/src/cmd/ucodeadm/intel-ucode.txt
i1480-usb-0.0.bin

Some possibly interesting links to help distinguish the thin line between tin hat and todo item.

http://kerneltrap.org/OpenBSD/That_Which_We_Call_Free
http://www.openbsd.org/lyrics.html#39
http://lkml.indiana.edu/hypermail/linux/kernel/0903.1/00335.html

So my goal today is to have everything be entirely open software.  In the future maybe this moves and the goal to have entirely open hardware is also possible.  I write these words now and wonder how I'll reflect back in the years to come.

Why can't RPM spec files be coverted to ebuilds

In evaluating converting the jpackage spec repository to ebuilds in an automated way I came across the following obstacles

    1) SourceNN: variables don't frequently contain a valid URL
    2) sub %packages
        # ebuilds have one inherit strength in that they typically map one tarball and one package per ebuild.  With this you are maintaining one source package per binary package.  To add or remove features USE flags were invented.  Since most end users are not compiling from source binary distributions invented ways to sub divide the packaged files into multiple binary packages.  The only benefit to this division of packages is saving bandwidth and space.

    3) Dependencies are loosely coupled and not globally tracked.
    4) ebuilds are uniquely mapped category + package and specs are just mapped by package.
        #Using the existing Group: would be possible, but more complicated than just adding a Category: variable

With this comes a couple questions...

1) How many Windows, Sun or IBM software packages come with separate package just for the docs?  (Yes some packages allow you not to install docs on the initial install)

2) When your XP updates are you asked if you want the docs and demo to go with it? No, you see a progress bar and just wait.  So there are two sides to this bandwidth and disk space question. 

    Remote server
    End user

I don't think most open source projects remote server is hitting a bottleneck for disk space or bandwidth.  Most successful projects are effectively using various forms of server and geo load balancing to distribute and localize.

So then comes the more questions...

3) Who is the end user and how do we invest our time so that they have the most efficient packaging as possible.

There are hundreds of thousands of software packages, but only a handful of package formats.  (RPM, dpkg, xpkg.. etc)  In my evaluation for which binary package to choose for OSUNIX I came across some very interesting numbers.  *If* distributions really wanted to save space and bandwidth they'd quit using gzip and bzip2.  lzma and xz (lzma2) showed in my early tests to significantly save space and were faster to extract.   Using xar for our oxe packages we won on size and speed vs dpkg and rpm even though xar compresses per file.

4) How often do people install the docs?

You may wonder why I care, but this is about saving time and giving the greatest benefit to the developers and end users.  There's 649 packages I wanted to use from the jpackage repo.  Roughly 2145 of the sub packages were either javadoc, manual or demo related.  How and why were the other packages being divided up?  ebuild package maintainers may have a lot of bad habits, but it's clear that there's no panacea to packaging.  It all goes back to having stronger policies in place that are based on logic and actually solving problems. </rant>