Codestrom - The Wandering Programmer

dedup rumors and now finally a patch

I'll let the logs speak for themselves.. Rumor is in snv_128 we may actually see zfs dedup!  Sweet!

Author: Jeff Bonwick <Jeff.Bonwick@Sun.COM>
Repository: /hg/onnv/onnv-gate
Latest revision: e2081f5023069120b7a160a1826f5bf80151828f
Total changesets: 1
Log message:
PSARC 2009/571 ZFS Deduplication Properties
6677093 zfs should have dedup capability

Files:
	create: usr/src/uts/common/fs/zfs/ddt.c
	create: usr/src/uts/common/fs/zfs/ddt_zap.c
	create: usr/src/uts/common/fs/zfs/sys/ddt.h
	create: usr/src/uts/common/fs/zfs/zle.c
	update: usr/src/cmd/filebench/Makefile.com
	update: usr/src/cmd/mdb/common/modules/zfs/zfs.c
	update: usr/src/cmd/sgs/Makefile.var
	update: usr/src/cmd/zdb/Makefile.com
	update: usr/src/cmd/zdb/zdb.c
	update: usr/src/cmd/zdb/zdb_il.c
	update: usr/src/cmd/zpool/zpool_main.c
	update: usr/src/cmd/ztest/ztest.c
	update: usr/src/common/avl/avl.c
	update: usr/src/common/zfs/zfs_fletcher.c
	update: usr/src/common/zfs/zfs_prop.c
	update: usr/src/common/zfs/zfs_prop.h
	update: usr/src/common/zfs/zpool_prop.c
	update: usr/src/common/zfs/zprop_common.c
	update: usr/src/grub/capability
	update: usr/src/grub/grub-0.97/stage2/fsys_zfs.c
	update: usr/src/grub/grub-0.97/stage2/zfs-include/spa.h
	update: usr/src/grub/grub-0.97/stage2/zfs-include/zap_impl.h
	update: usr/src/grub/grub-0.97/stage2/zfs-include/zfs.h
	update: usr/src/lib/libzfs/common/libzfs_dataset.c
	update: usr/src/lib/libzfs/common/libzfs_pool.c
	update: usr/src/lib/libzpool/common/llib-lzpool
	update: usr/src/uts/common/Makefile.files
	update: usr/src/uts/common/fs/vfs.c
	update: usr/src/uts/common/fs/zfs/arc.c
	update: usr/src/uts/common/fs/zfs/bplist.c
	update: usr/src/uts/common/fs/zfs/dbuf.c
	update: usr/src/uts/common/fs/zfs/dmu.c
	update: usr/src/uts/common/fs/zfs/dmu_objset.c
	update: usr/src/uts/common/fs/zfs/dmu_send.c
	update: usr/src/uts/common/fs/zfs/dmu_traverse.c
	update: usr/src/uts/common/fs/zfs/dmu_tx.c
	update: usr/src/uts/common/fs/zfs/dnode_sync.c
	update: usr/src/uts/common/fs/zfs/dsl_dataset.c
	update: usr/src/uts/common/fs/zfs/dsl_deleg.c
	update: usr/src/uts/common/fs/zfs/dsl_dir.c
	update: usr/src/uts/common/fs/zfs/dsl_pool.c
	update: usr/src/uts/common/fs/zfs/dsl_prop.c
	update: usr/src/uts/common/fs/zfs/dsl_scrub.c
	update: usr/src/uts/common/fs/zfs/dsl_synctask.c
	update: usr/src/uts/common/fs/zfs/lzjb.c
	update: usr/src/uts/common/fs/zfs/metaslab.c
	update: usr/src/uts/common/fs/zfs/sha256.c
	update: usr/src/uts/common/fs/zfs/spa.c
	update: usr/src/uts/common/fs/zfs/spa_history.c
	update: usr/src/uts/common/fs/zfs/spa_misc.c
	update: usr/src/uts/common/fs/zfs/space_map.c
	update: usr/src/uts/common/fs/zfs/sys/arc.h
	update: usr/src/uts/common/fs/zfs/sys/bplist.h
	update: usr/src/uts/common/fs/zfs/sys/dbuf.h
	update: usr/src/uts/common/fs/zfs/sys/dmu.h
	update: usr/src/uts/common/fs/zfs/sys/dmu_objset.h
	update: usr/src/uts/common/fs/zfs/sys/dmu_traverse.h
	update: usr/src/uts/common/fs/zfs/sys/dsl_dataset.h
	update: usr/src/uts/common/fs/zfs/sys/dsl_pool.h
	update: usr/src/uts/common/fs/zfs/sys/metaslab.h
	update: usr/src/uts/common/fs/zfs/sys/metaslab_impl.h
	update: usr/src/uts/common/fs/zfs/sys/spa.h
	update: usr/src/uts/common/fs/zfs/sys/spa_impl.h
	update: usr/src/uts/common/fs/zfs/sys/txg.h
	update: usr/src/uts/common/fs/zfs/sys/txg_impl.h
	update: usr/src/uts/common/fs/zfs/sys/uberblock.h
	update: usr/src/uts/common/fs/zfs/sys/vdev.h
	update: usr/src/uts/common/fs/zfs/sys/vdev_impl.h
	update: usr/src/uts/common/fs/zfs/sys/zap.h
	update: usr/src/uts/common/fs/zfs/sys/zap_impl.h
	update: usr/src/uts/common/fs/zfs/sys/zap_leaf.h
	update: usr/src/uts/common/fs/zfs/sys/zil.h
	update: usr/src/uts/common/fs/zfs/sys/zil_impl.h
	update: usr/src/uts/common/fs/zfs/sys/zio.h
	update: usr/src/uts/common/fs/zfs/sys/zio_checksum.h
	update: usr/src/uts/common/fs/zfs/sys/zio_compress.h
	update: usr/src/uts/common/fs/zfs/sys/zio_impl.h
	update: usr/src/uts/common/fs/zfs/txg.c
	update: usr/src/uts/common/fs/zfs/vdev.c
	update: usr/src/uts/common/fs/zfs/vdev_mirror.c
	update: usr/src/uts/common/fs/zfs/vdev_queue.c
	update: usr/src/uts/common/fs/zfs/vdev_raidz.c
	update: usr/src/uts/common/fs/zfs/zap.c
	update: usr/src/uts/common/fs/zfs/zap_leaf.c
	update: usr/src/uts/common/fs/zfs/zap_micro.c
	update: usr/src/uts/common/fs/zfs/zfs_fm.c
	update: usr/src/uts/common/fs/zfs/zfs_ioctl.c
	update: usr/src/uts/common/fs/zfs/zfs_log.c
	update: usr/src/uts/common/fs/zfs/zfs_replay.c
	update: usr/src/uts/common/fs/zfs/zfs_vnops.c
	update: usr/src/uts/common/fs/zfs/zil.c
	update: usr/src/uts/common/fs/zfs/zio.c
	update: usr/src/uts/common/fs/zfs/zio_checksum.c
	update: usr/src/uts/common/fs/zfs/zio_compress.c
	update: usr/src/uts/common/fs/zfs/zio_inject.c
	update: usr/src/uts/common/fs/zfs/zvol.c
	update: usr/src/uts/common/sys/avl.h
	update: usr/src/uts/common/sys/fs/zfs.h
	update: usr/src/uts/intel/zfs/spa_boot.c
	update: usr/src/uts/sparc/zfs/spa_boot.c

zendesk + jira + jetty 6 (impossible)

So if you're like me and you want zendesk + jira integration please forget the first part of this document. It's rubbish..

The versions needed between jira 3, jetty 6 and zendesk plugin won't work..  So just trying to save you all some time..

Links for jars just for reference..

http://downloads.agilos.org/zendesk-jira-plugin-1.2.jar

http://maven.restlet.org/org/restlet/org.restlet/1.2-M2/org.restlet-1.2-M2.jar

and finally

http://www.slf4j.org/download.html

Last, but not least this zendesk + jira installation doc may be helpful to you

http://agilos.org/display/zen/ZenDesk+JIRA+plugin

I've submitted a trouble ticket with zendesk to get the installation guide links updated so hopefully this document is soon irrelevant

Is Sun violating the GPL?

My OSUNIX project underwent months and months of delays from "passionate people" at Debian, but now that's seemingly resolved.  From that came a letter from the SFLC about the simultaneous distribution of software that links GPLv2 grub code to mixed CDDL code in libc.

With this in mind we've been working with GRUB developers to get zfs boot support in grub2.. afaik it's there and ready for testing.. (along with some other juicy features)  So hte question is.. When will Sun start to ship grub2?

------

Here's the copy/pasted text..

Dear Christopher,
You asked me to review the OSUNIX project’s proposed distribution of the OSUNIX
operating system, and to give you my opinion on whether the distribution presented
any license-conflict issues. I do not believe that the distribution you described to me
would cause OSUNIX or its licensees to violate the license of any of the distribution’s
component software. In this letter I will describe what I understand to be the licensing
structure of the proposed distribution, and why the differently licensed components are
each distributed in compliance with their respective licenses. If I have misunderstood the
facts, please correct me as soon as possible so that I can revise my opinion appropriately.
You told me that you plan to distribute OSUNIX primarily via an ISO file. Since
OSUNIX is based on OpenSolaris, the majority of the software in that ISO is licensed
under the Sun Common Development and Distribution License (CDDL). However, the
ISO will include two notable exceptions to this general rule. The first is the OpenSolaris
C library: while most of the code for the library is available under the CDDL, the binary
you ship in the ISO is licensed under the OpenSolaris Binary License (OBL) because
it contains wide-character support software licensed by Sun from a third party. The
OBL, though allowing redistribution as a part of OpenSolaris code (in this case the
C library), is essentially a proprietary rather than a free or open source license. The
second exception is grub2, a Free Software Foundation-copyrighted program distributed
under the GNU General Public License, version 3 (GPLv3). In your distribution of
OSUNIX, grub2 links with the OBL-licensed C library.
GPLv3 and CDDL are generally considered to be “incompatible” licenses, meaning
that it is impermissible (in this case, because of restrictions present in both licenses) to
combine code under both licenses into a single work. That resulting work would be a
“derivative work” under copyright law, and both licenses require that derivative works
be licensed according to the same terms as the original work. Since this restriction could
not be satisfied for both licenses simultaneously, the licenses are incompatible. (CDDL
allows “Larger Works” based on CDDL and non-CDDL code to be licensed under a non-
CDDL license so long as “the requirements of [the CDDL] are fulfilled for the CDDL
software.” This does not resolve the incompatibility with GPLv3, because some of those
requirements conflict with GPLv3’s.) GPLv3 is, of course, also incompatible in this way
with proprietary licenses such as the OBL.
The proposed OSUNIX ISO contains no incompatible combination of CDDL- and
GPLv3-licensed software. Both licenses allow for covered software to be aggregated with separate software under other licenses, and do not place restrictions on this kind
of aggregation. The CDDL and GPLv3 components of the OSUNIX distribution you
have described do not combine more intimately than this.
However, the process of compiling a program that links against a separate library, as
you plan to do with grub2 and the OpenSolaris C library, produces a binary program
which may be considered a derivative work of both the program and the library. Be-
cause GPLv3 is not compatible with the OBL, this sort of combination is generally
impermissible under GPLv3.
But the combination of grub2 and the OpenSolaris C library is allowed, because the
C library is a “System Library” within the meaning of GPLv3, and as such is exempt
from GPLv3’s source code-provision requirements. Section 1 of GPLv3 defines the
“Corresponding Source” that must be provided with every distribution of a program
based on GPLv3-licensed code. This definition explicitly excludes “System Libraries,”
whose relevant definition can be paraphrased as: anything that is normally packaged
with a major component of the operating system (e.g. the kernel) and serves only
to implement a standard interface (e.g. for a programming language) for which an
implementation is available to the public in source code form.
Even this paraphrased definition can be difficult to follow, but it’s application to OS-
UNIX can be summarized as follows: because the C library is essential to major portions
of the OSUNIX operating system, and its purpose is to define an interface to the de
facto-standard C programming language (which is also implemented in other freely-
licensed libraries, such as glibc), you can link a GPLv3 program against it without
distributing source for the library.
As an aside, this result is different than it would have been under version 2 of the
GPL. In GPLv2, the “system library exception” did not apply to programs that were
distributed with the operating system. This limitation does not exist in GPLv3.
Because OSUNIX does not combine (but merely aggregates) CDDL- and GPLv3-licensed
software, and because its combination of grub2 and the OpenSolaris C library is allowed
under GPLv3’s “System Library” exception, I believe that the distribution you’ve de-
scribed is permissible under the licenses of all its component software.

zfs dedup where art thou

So the other day I posted a FAQ on zfs deduplication but I guess I'm not an authority on the topic fo anyone to believe me..

Doesn't it seem "strange" to anyone else besides me that the zfs dedup presentation at the Australian kernel conference isn't online yet?  Are Sun users really so stupid to buy into the comment that it's still in production?  No.. the truth is Larry's interview from a few days ago can appear on youtube within days or hours, but zfs dedup is caught up in some legal whatever.

Drop me or Max Bruning a comment if you want zfs dedup done in OpenSolaris.  I'm currently seeking supporters and donations.  All donations will be tax deductible via the OSUNIX project.  Link forthcoming..

chipotle where art thou

So I'm spending  a few days visiting the University of Delaware.. One of my friends said they wanted mexican food and my first thought was of Chipotle..  A quick search online revealed that the first state I guess is the last state to get a Chipotle.. *sigh*  There is one 26.5 miles away from where I am right now, but still I thought that's a bit far for just a burrito.. We went instead to Santa Fe a little restaurant on main street.. All I can say is that for anyone who actually *knows* what a burrito is *supposed* to taste like should steer clear.  I would rate it as very bad and overpriced.  With that in mind.. Help me to bring a Chipotle to the poor little unloved and small state of Delaware.. At the very least we will save some unsuspecting university students from a very bad dinner.

I'm kinda serious.. Please sign the petition..

http://www.petitiononline.com/chipotde/petition-sign.html

FAQ zfs deduplication

Q. When will ZFS have deduplication?

A. Just my view on ZFS + deduplication based on what I know..

• There is some legal bla bla between Sun and green-bytes that's tying up the IP around dedup... (someone knock some sense into green-bytes please) 
• there's an acquisition that's got all sorts of delays.. which may very well delay the thing with green-bytes as well
• Is it in the best interest of Oracle for ZFS to have dedup before btrfs?  (I doubt this is a reason, but it's a question to ask)
• Nobody has written the code.. patches welcome :)

Drop me a comment if I'm missing something


IANAL


./C

Microsoft's view on copyright

So today I'm browsing around looking for miscellaneous things to read on this "lazy" Sunday afternoon.  Since I'm stepping into the role as the new CTO of PathScale I thought I should at least spend some more time looking into open source business models.  That's when someone pointed me at Open Source Is Not a Business Model (CAOS 9) .  Being curious as I am I thought I'd try to get more opinions or feedback on this case study..  Half way down a quick google search I noticed there was a 71 page PDF.  Well thanks to Microsoft [1] the 451 Group's article is free for the whole world?

I'm not sure if Microsoft has a special place in their heart for open source business models, but gosh I could only hope they make this as freely accessible to their management.  Which begs another question.. What is Microsoft's view on copyright infringement?

"Authorized Reproduction: Public reproduction or republication — including citations from
analysts’ reports — of the Group’s material(s) is forbidden, except with the expressed written
permission of the Group. To inquire about authorized reproduction, individual named users
should contact their respective account manager(s)."

[1] I'll be letting the copyright owner know about this and if in fact it's not supposed to be publicly available the link will probably be removed.

btrfs review

I've done a comparison of the feature sets for btrfs vs zfs, but an ex Sun employee really lays it down in a bit more technical details.  Superb job!

http://lwn.net/Articles/342892/

As someone who is very dedicated to the multitude of benefits OpenSolaris technology *currently* has over Linux I must say it'll be very interested if we end up with btrfs on OpenSolaris.  I would vastly prefer this to having ZFS on Linux.. Why.. because even though I *love* Linux (I do love the community)  I still really miss that proper engineering.. maybe in the end it's a tradeoff for long term sustainability.. I can live with that.. maybe linux is the prototype/poc which then later pushes the evolution back into "enterprise" (I use this word loosely) operating systems..

So I made a flippant entry a while back of why btrfs should be ported to OpenSolaris... Now looking deeper at the facts I think it becomes more and more a good business choice.  The problem as far as I know though is that it's tightly being coupled with the Linux kernel and maybe intentionally not being documented well.  Both of those combined would make it rather difficult to independently clean room implement.  Maybe we should kick off the btrfs documentation project.. and the BSD/OpenSolaris folks can implement it..

Any game to help out?

Introducing OSUNIX lightening talk at OSCON

OSCON went great!  I can't say thanks enough for John Plocher giving the Introduction to OSUNIX lightening talk.  For those of you interested or who missed it.  Here's the PDF version.  If you need the original Open Office document feel free to ping me on irc or email.Download Osunix-introduction-final

Free events for OSCON

Here's a rough list of Free things around OSCON that I'm interested, but thought I'd share since others may be interested as well..

Tonight Friday 17th

    19:00+ Marriott lobby and then dinner
    http://www.communityleadershipsummit.com/wiki/index.php/Friday_night_plans
    Maybe hang out in Tanq bar until everyone arrives.
    Note : May be a lot of floss foundation people here since the community leader summit has overshadowed there un conference

Saturday 18th :
    9:00-17:00
    http://www.communityleadershipsummit.com/
    This is the unconference before OSCON.  Should be interesting and a good chance to network with others

Sunday 19th :
    9:30-17:00
    http://www.communityleadershipsummit.com/
    This is the unconference before OSCON.  Should be interesting and a good chance to network with others

    9:30-21:00
    PgDay is here... Come join!
    http://wiki.postgresql.org/wiki/PgDaySanJose2009#Details
    http://postgresql.meetup.com/1/calendar/10626647/  

    19:00+
    Party for the PostgreSQL 8.4 release!!
    Must RSVP http://postgresql.meetup.com/1/calendar/10718082/


Monday 20th -
    19:00 - 20:00 - Crossbow Network Virtualization BOF
    Convergence of Compute and Network Services,Location: Meeting Room B1/B4
    If you're still using the native OSX on your Mac come here and I'll help you install the latest OpenSolaris with Crossbow!
   
Tuesday 21st -
    Open/recovery??


Wednesday 22nd -
    9:00-14:00(?) unclear when this "ends"
    Understanding Legal Issues in Open Source
    http://en.oreilly.com/oscon2009/public/schedule/detail/10440
    For anyone with outstanding legal questions this may be your chance to bug those at the SFLC to get answers.

    All day - Expo Hall opens for OSCON.. Come check out the OpenSolaris booth!!!
    All day - Visit the LinuxFund booth to find out about the big party on Wednesday!
    20:00-24:00 - LinuxFund party - must visit booth for details

Thursday 23rd
   
    All day - Expo Hall opens for OSCON.. Missed the OpenSolaris booth on Wednesday one more chance
    13:30 - 15:20 - Lightening talks - Be there or be square!
    19:00 - Women in FLOSS and Allies Gathering,Moderated by: Kirrily Robert, Location: Meeting Room B1/B4
    20:00 - BayPIGgies July Meeting Location: Meeting Room B2


BOF sessions
    http://en.oreilly.com/oscon2009/public/cfp/68

Other free events
    http://en.oreilly.com/oscon2009/public/content/free

Questions, want to meet up or just want to yell at me?

    Christopher - 415-269-8386