Codestrom - The Wandering Programmer

dedup rumors and now finally a patch

I'll let the logs speak for themselves.. Rumor is in snv_128 we may actually see zfs dedup!  Sweet!

Author: Jeff Bonwick <Jeff.Bonwick@Sun.COM>
Repository: /hg/onnv/onnv-gate
Latest revision: e2081f5023069120b7a160a1826f5bf80151828f
Total changesets: 1
Log message:
PSARC 2009/571 ZFS Deduplication Properties
6677093 zfs should have dedup capability

Files:
	create: usr/src/uts/common/fs/zfs/ddt.c
	create: usr/src/uts/common/fs/zfs/ddt_zap.c
	create: usr/src/uts/common/fs/zfs/sys/ddt.h
	create: usr/src/uts/common/fs/zfs/zle.c
	update: usr/src/cmd/filebench/Makefile.com
	update: usr/src/cmd/mdb/common/modules/zfs/zfs.c
	update: usr/src/cmd/sgs/Makefile.var
	update: usr/src/cmd/zdb/Makefile.com
	update: usr/src/cmd/zdb/zdb.c
	update: usr/src/cmd/zdb/zdb_il.c
	update: usr/src/cmd/zpool/zpool_main.c
	update: usr/src/cmd/ztest/ztest.c
	update: usr/src/common/avl/avl.c
	update: usr/src/common/zfs/zfs_fletcher.c
	update: usr/src/common/zfs/zfs_prop.c
	update: usr/src/common/zfs/zfs_prop.h
	update: usr/src/common/zfs/zpool_prop.c
	update: usr/src/common/zfs/zprop_common.c
	update: usr/src/grub/capability
	update: usr/src/grub/grub-0.97/stage2/fsys_zfs.c
	update: usr/src/grub/grub-0.97/stage2/zfs-include/spa.h
	update: usr/src/grub/grub-0.97/stage2/zfs-include/zap_impl.h
	update: usr/src/grub/grub-0.97/stage2/zfs-include/zfs.h
	update: usr/src/lib/libzfs/common/libzfs_dataset.c
	update: usr/src/lib/libzfs/common/libzfs_pool.c
	update: usr/src/lib/libzpool/common/llib-lzpool
	update: usr/src/uts/common/Makefile.files
	update: usr/src/uts/common/fs/vfs.c
	update: usr/src/uts/common/fs/zfs/arc.c
	update: usr/src/uts/common/fs/zfs/bplist.c
	update: usr/src/uts/common/fs/zfs/dbuf.c
	update: usr/src/uts/common/fs/zfs/dmu.c
	update: usr/src/uts/common/fs/zfs/dmu_objset.c
	update: usr/src/uts/common/fs/zfs/dmu_send.c
	update: usr/src/uts/common/fs/zfs/dmu_traverse.c
	update: usr/src/uts/common/fs/zfs/dmu_tx.c
	update: usr/src/uts/common/fs/zfs/dnode_sync.c
	update: usr/src/uts/common/fs/zfs/dsl_dataset.c
	update: usr/src/uts/common/fs/zfs/dsl_deleg.c
	update: usr/src/uts/common/fs/zfs/dsl_dir.c
	update: usr/src/uts/common/fs/zfs/dsl_pool.c
	update: usr/src/uts/common/fs/zfs/dsl_prop.c
	update: usr/src/uts/common/fs/zfs/dsl_scrub.c
	update: usr/src/uts/common/fs/zfs/dsl_synctask.c
	update: usr/src/uts/common/fs/zfs/lzjb.c
	update: usr/src/uts/common/fs/zfs/metaslab.c
	update: usr/src/uts/common/fs/zfs/sha256.c
	update: usr/src/uts/common/fs/zfs/spa.c
	update: usr/src/uts/common/fs/zfs/spa_history.c
	update: usr/src/uts/common/fs/zfs/spa_misc.c
	update: usr/src/uts/common/fs/zfs/space_map.c
	update: usr/src/uts/common/fs/zfs/sys/arc.h
	update: usr/src/uts/common/fs/zfs/sys/bplist.h
	update: usr/src/uts/common/fs/zfs/sys/dbuf.h
	update: usr/src/uts/common/fs/zfs/sys/dmu.h
	update: usr/src/uts/common/fs/zfs/sys/dmu_objset.h
	update: usr/src/uts/common/fs/zfs/sys/dmu_traverse.h
	update: usr/src/uts/common/fs/zfs/sys/dsl_dataset.h
	update: usr/src/uts/common/fs/zfs/sys/dsl_pool.h
	update: usr/src/uts/common/fs/zfs/sys/metaslab.h
	update: usr/src/uts/common/fs/zfs/sys/metaslab_impl.h
	update: usr/src/uts/common/fs/zfs/sys/spa.h
	update: usr/src/uts/common/fs/zfs/sys/spa_impl.h
	update: usr/src/uts/common/fs/zfs/sys/txg.h
	update: usr/src/uts/common/fs/zfs/sys/txg_impl.h
	update: usr/src/uts/common/fs/zfs/sys/uberblock.h
	update: usr/src/uts/common/fs/zfs/sys/vdev.h
	update: usr/src/uts/common/fs/zfs/sys/vdev_impl.h
	update: usr/src/uts/common/fs/zfs/sys/zap.h
	update: usr/src/uts/common/fs/zfs/sys/zap_impl.h
	update: usr/src/uts/common/fs/zfs/sys/zap_leaf.h
	update: usr/src/uts/common/fs/zfs/sys/zil.h
	update: usr/src/uts/common/fs/zfs/sys/zil_impl.h
	update: usr/src/uts/common/fs/zfs/sys/zio.h
	update: usr/src/uts/common/fs/zfs/sys/zio_checksum.h
	update: usr/src/uts/common/fs/zfs/sys/zio_compress.h
	update: usr/src/uts/common/fs/zfs/sys/zio_impl.h
	update: usr/src/uts/common/fs/zfs/txg.c
	update: usr/src/uts/common/fs/zfs/vdev.c
	update: usr/src/uts/common/fs/zfs/vdev_mirror.c
	update: usr/src/uts/common/fs/zfs/vdev_queue.c
	update: usr/src/uts/common/fs/zfs/vdev_raidz.c
	update: usr/src/uts/common/fs/zfs/zap.c
	update: usr/src/uts/common/fs/zfs/zap_leaf.c
	update: usr/src/uts/common/fs/zfs/zap_micro.c
	update: usr/src/uts/common/fs/zfs/zfs_fm.c
	update: usr/src/uts/common/fs/zfs/zfs_ioctl.c
	update: usr/src/uts/common/fs/zfs/zfs_log.c
	update: usr/src/uts/common/fs/zfs/zfs_replay.c
	update: usr/src/uts/common/fs/zfs/zfs_vnops.c
	update: usr/src/uts/common/fs/zfs/zil.c
	update: usr/src/uts/common/fs/zfs/zio.c
	update: usr/src/uts/common/fs/zfs/zio_checksum.c
	update: usr/src/uts/common/fs/zfs/zio_compress.c
	update: usr/src/uts/common/fs/zfs/zio_inject.c
	update: usr/src/uts/common/fs/zfs/zvol.c
	update: usr/src/uts/common/sys/avl.h
	update: usr/src/uts/common/sys/fs/zfs.h
	update: usr/src/uts/intel/zfs/spa_boot.c
	update: usr/src/uts/sparc/zfs/spa_boot.c

zfs dedup where art thou

So the other day I posted a FAQ on zfs deduplication but I guess I'm not an authority on the topic fo anyone to believe me..

Doesn't it seem "strange" to anyone else besides me that the zfs dedup presentation at the Australian kernel conference isn't online yet?  Are Sun users really so stupid to buy into the comment that it's still in production?  No.. the truth is Larry's interview from a few days ago can appear on youtube within days or hours, but zfs dedup is caught up in some legal whatever.

Drop me or Max Bruning a comment if you want zfs dedup done in OpenSolaris.  I'm currently seeking supporters and donations.  All donations will be tax deductible via the OSUNIX project.  Link forthcoming..

FAQ zfs deduplication

Q. When will ZFS have deduplication?

A. Just my view on ZFS + deduplication based on what I know..

• There is some legal bla bla between Sun and green-bytes that's tying up the IP around dedup... (someone knock some sense into green-bytes please) 
• there's an acquisition that's got all sorts of delays.. which may very well delay the thing with green-bytes as well
• Is it in the best interest of Oracle for ZFS to have dedup before btrfs?  (I doubt this is a reason, but it's a question to ask)
• Nobody has written the code.. patches welcome :)

Drop me a comment if I'm missing something


IANAL


./C

btrfs review

I've done a comparison of the feature sets for btrfs vs zfs, but an ex Sun employee really lays it down in a bit more technical details.  Superb job!

http://lwn.net/Articles/342892/

As someone who is very dedicated to the multitude of benefits OpenSolaris technology *currently* has over Linux I must say it'll be very interested if we end up with btrfs on OpenSolaris.  I would vastly prefer this to having ZFS on Linux.. Why.. because even though I *love* Linux (I do love the community)  I still really miss that proper engineering.. maybe in the end it's a tradeoff for long term sustainability.. I can live with that.. maybe linux is the prototype/poc which then later pushes the evolution back into "enterprise" (I use this word loosely) operating systems..

So I made a flippant entry a while back of why btrfs should be ported to OpenSolaris... Now looking deeper at the facts I think it becomes more and more a good business choice.  The problem as far as I know though is that it's tightly being coupled with the Linux kernel and maybe intentionally not being documented well.  Both of those combined would make it rather difficult to independently clean room implement.  Maybe we should kick off the btrfs documentation project.. and the BSD/OpenSolaris folks can implement it..

Any game to help out?

zfs root

ZFS root the bottom line

So to clear any doubt about it.  zfs root is *rock solid*.  zfs root not only allows multiple BE (boot environments), but other advantages.  You can't currently use compression or raidz(2) with zfs root, but those are being planned currently.. With any luck they'll be accepted as google summer of code projects :)

zfs vs btrfs comparison

To be completely honest this is a high level best effort feature comparison between zfs and btrfs.  (I'm trying to be as unbiased as possible)  While zfs has years of being used in production I think btrfs is quickly adding a large comparable feature set.  Nothing is perfect or instant so corrections and updates are *most* welcome. (This includes how I've worded things)  Check my about page for contact details.  As I have time I'll try to publish more detailed descriptions of each file system.  (Selfish note: I'm also interested to porting btrfs or zfs under a more liberal license if anyone wants to sponsor the effort or needs details.)

Feature	ZFS	BTRFS
COW (copy on write)	yes	yes
Snapshots	yes	yes
Performance degradation at near 98-100% disk usage	yes	Currently does kernel oopse, but being worked on and honestly will probably have similar issue. (Someone ping me when this is fixed)
Block level compression	yes	yes. Currenly only available as a mount option.  *Maintenance nightmare*
Full on disk encryption (file, attributes, name.. etc)	Being worked on, but not production ready.  Sun recently revised their release schedule from May to probably some time in 2009.  (I'm just guessing on this timeline)  Using solaris lofi encryption is possible in the mean-time.	Planned, but no certain schedule.  Is limited to file bodies, but ecryptfs may also offer additional options
Spindle write balances	yes	Planned - no schedule
Online resizing	no (being worked on)	yes
Online defragmentation	no (ties into the same code as pool resizing so being worked on)	yes
shared library interface	libzfs	none.. no plans, but may be possible?
Alternative compression capability	yes, but would need to link into the kernel	yes, but would need to link into the kernel
Aggressive file caching	yes	various kernel options.. (Unclear to the author what these options are or exactly do.. check /proc/sys/vm for whatever hackery)
Ditto block	yes (multiple copies of metadata and files logically distributed across all spindles.  Number of copies of the default is tunable.)	metadata only
write checksums	yes	yes
Built-in raid (More notes below)	0,1,10, 5 with fixes write hole (raidz), and some variant of 6 called raidz2	0,1 and 10
Automatically turns off write cache when not controlling full disk	yes	no
Easy way to transport pools/volumes between hosts	yes (zfs send/recieve)	Mirroring will be available, but I don't think this is the same as zfs send/receive
Primary sponsoring company	Sun™ (I think Sun™ should encourage the community to be more involved?)	Oracle™, RedHat™, HP™, IBM™, Intel™, Novell™and Linux community
License	CDDL	GPL2
ACL	yes (pure ACL.. all files have ACL) with userland support	yes, but can be turned off? (but..  the gnu utils chmod and others may not have full support acl yet.)
ACL model	NFSv4 ACL	Posix
Endian independent	Yes.. swap a drive been sparc or intel as you wish	No
separate zil	Yes	No
separate l2arc	Yes	No
zfs supports case-insensitive filesystems for Windows compatibility	Yes	No
enforce utf-8-only filenames	Yes	No
direct io support	Yes	Writes yes. Reads No - Planned
POSIX compliant concurrent writes (single writer locks)	Yes	No
Extended attributes	Yes	Yes
Quotas	Yes	Yes
Favorite Quote from lead developer	"When I take on a new problem is that I'm not interested in how it's been done before. I only want to know, of all the constraints people tend to assume, which ones are actually fundamental and which ones are just habit?" Jeff Bonwick	"something lame that gets us 85% of the way there is fine" Chris Mason
**Warning some opinion** RAID/volume management	Inside ZFS	Managed outside the filesystem Is lvm really good at managing raid? Does it make the /problem/ more simple or add layers to the administration which are a pita

Christopher Bergström © 2009 - Please don't reproduce without asking first.