Merge ~ahasenack/ubuntu/+source/tdb:disco-tdb-1.3.18 into ubuntu/+source/tdb:ubuntu/devel

[Andreas Hasenack]

PPA with builds: ppa:ahasenack/samba-4.10
https://launchpad.net/~ahasenack/+archive/ubuntu/samba-4.10/

Bileto ticket:
https://bileto.ubuntu.com/#/ticket/3672

It was green, but recent uploads removed the dep8 test history. I have to copy the packages over again.
This is part of the effort to remove python2 from the desktop iso.

That requires samba to be updated to 4.10 (bug #1818518), and that update requires a new tdb, which is that this branch is about.

We are going ahead of Debian.

Upstream switched from python 2 to using python 3 by default, so that required a few changes to the ./configure call in d/rules, and more cleanups in dh_clean.

I don't know why debian is skipping generating a symbols file for the python{,3}-tdb packages (the -N option):
override_dh_makeshlibs:
        dh_makeshlibs -Npython-tdb -Npython3-tdb -- -c4

I also don't know why debian is manually building a static library in override_dh_auto_install, but it was not being listed in the corresponding d/libtdb-dev.install file, which I "fixed". I can bring this up with debian if you want.

I didn't clean up the commit tree, because there has been some back and forth here about dropping or not the py2 packages entirely. Foundations wanted the python-* (py2) packages entirely dropped, but at least python-tdb is still needed by bzr-git (Recommends). I'll keep building it, but we might be asked to drop it and somehow fix bzr-git. Anyway, I think that can be done later if needed.

[Andreas Hasenack (ahasenack)]

From IRC:
<cjwatson> bzr-git's tdb thing is used by LP
<cjwatson> for git-to-bzr code imports
<cjwatson> this doesn't mean you should block on it, since we maintain our own copies of the modules we need, just FYI on what sorts of things it's used for

[Christian Ehrhardt  (paelzer)]

probably bikeshedding, but all the changes in the changelog are "due to" the New upstream version.
I'd therefore have expected to find them indented as

   * New upstream version: 1.3.18 (LP: #1818520)
     - d/p/40_test_transaction_expand_non_fatal.diff: refresh to get rid of
       fuzz
     - ...

up to you

[Andreas Hasenack (ahasenack)]

I can improve that

[Christian Ehrhardt  (paelzer)]

waf cleanup is fine, but given how much you list don't you want to convert it to just
  $ find ... -name .pyc -delete
or something like that?
Considering that you also clear directories maybe:
 $ find -name ".pyc" -o -name "__pycache__" -exec rm {} \;

[Christian Ehrhardt  (paelzer)]

Other than the nit picks mentioned the commits LGTM, I'll go check the build log now ...

[Christian Ehrhardt  (paelzer)]

I wondered if [1] would cause trouble for the py2 packages, but after reviewing what ends up in those packages I think you are good.

[1]: https://gitlab.com/samba-team/samba/commit/a07279b78a83d7e62eb0c58a2ac989407a40fb6b

[Christian Ehrhardt  (paelzer)]

I'm done parsing the upstream changes towards 1.3.18 and the build log not much showing up there that needs work.

The one thing I wondered is (probably unimportant):
   dh_gencontrol
dpkg-gencontrol: warning: package python3-tdb: substitution variable ${python3:Versions} unused, but is defined
dpkg-gencontrol: warning: package python3-tdb: substitution variable ${python3:Versions} unused, but is defined
dpkg-gencontrol: warning: package python-tdb: substitution variable ${python:Versions} unused, but is defined
dpkg-gencontrol: warning: package python-tdb: substitution variable ${python:Versions} unused, but is defined

The field [1] is optional after all.
An entry would look loke that on the py3 packages (py2 isn't needed since py2.7 is the only one)
XS-Python-Version: ${python3:Versions}

OTOH it might future transitions easier (or harder??)
See also man dh_python3 for that.

Since I have not seen you iterating over python3 versions in d/rules it might be safer to explicitly define this to be for the current py3.7.
I have seen builds crash in -devel in times there were two py3 (e.g. 3.6&3.7) in -release, and as far as i understand the flag here it might help with that.

And adding those lines should be easier than modifying more of d/rules to build potentially multiply pytohn modules iterating over `py3versions -vr`

[1]: https://www.debian.org/doc/packaging-manuals/python-policy/ch-module_packages.html

[Christian Ehrhardt  (paelzer)]

All that said, nothing I found is critical - the changes LGTM.
Do you want to change the MP state to ack now or only after a re-review once you are finished?
(I must admit I squashed the commits for reviewability)

review: Approve

[Andreas Hasenack (ahasenack)]

I'll look at the python issues you pointed at, and maybe use that for the other packages too. Let's leave the MP state as is for now.

[Andreas Hasenack (ahasenack)]

Even when using X-Python3-Version: ${python3:Versions} in d/control, I still get the warning, specifically in the binary packages. I'll remove it for now, pending a better understanding of how this works.

[Andreas Hasenack (ahasenack)]

- changelog updated
- I also squashed the revert commit I had

tdb will build py2 and py3 packages. py2 because of bzr-git, even though the FFe bug said it would have been ok to drop python-tdb (py2). We can still do that later.

I'll move on to the other MPs now, get all consistent.

[Andreas Hasenack (ahasenack)]

Tagged and uploaded:

$ git push pkg upload/1.3.18-0ubuntu1
Enumerating objects: 395, done.
Counting objects: 100% (395/395), done.
Delta compression using up to 4 threads
Compressing objects: 100% (208/208), done.
Writing objects: 100% (301/301), 438.24 KiB | 245.00 KiB/s, done.
Total 301 (delta 101), reused 277 (delta 86)
To ssh://git.launchpad.net/~usd-import-team/ubuntu/+source/tdb
 * [new tag] upload/1.3.18-0ubuntu1 -> upload/1.3.18-0ubuntu1

$ dput ubuntu ../tdb_1.3.18-0ubuntu1_source.changes
Checking signature on .changes
gpg: ../tdb_1.3.18-0ubuntu1_source.changes: Valid signature from AC983EB5BF6BCBA9
Checking signature on .dsc
gpg: ../tdb_1.3.18-0ubuntu1.dsc: Valid signature from AC983EB5BF6BCBA9
Uploading to ubuntu (via ftp to upload.ubuntu.com):
  Uploading tdb_1.3.18-0ubuntu1.dsc: done.
  Uploading tdb_1.3.18-0ubuntu1.debian.tar.xz: done.
  Uploading tdb_1.3.18-0ubuntu1_source.buildinfo: done.
  Uploading tdb_1.3.18-0ubuntu1_source.changes: done.
Successfully uploaded packages.

[Andreas Hasenack (ahasenack)]

Now uploading with the orig tarball included:

$ dput -f ubuntu ../tdb_1.3.18-0ubuntu1_source.changes
Checking signature on .changes
gpg: ../tdb_1.3.18-0ubuntu1_source.changes: Valid signature from AC983EB5BF6BCBA9
Checking signature on .dsc
gpg: ../tdb_1.3.18-0ubuntu1.dsc: Valid signature from AC983EB5BF6BCBA9
Uploading to ubuntu (via ftp to upload.ubuntu.com):
  Uploading tdb_1.3.18-0ubuntu1.dsc: done.
  Uploading tdb_1.3.18.orig.tar.gz: done.
  Uploading tdb_1.3.18-0ubuntu1.debian.tar.xz: done.
  Uploading tdb_1.3.18-0ubuntu1_source.buildinfo: done.
  Uploading tdb_1.3.18-0ubuntu1_source.changes: done.
Successfully uploaded packages.

diff --git a/ABI/tdb-1.3.17.sigs b/ABI/tdb-1.3.17.sigs
 new file mode 100644
index 0000000..e2b0427
--- /dev/null
+++ b/ABI/tdb-1.3.17.sigs
@@ -0,0 +1,73 @@
+tdb_add_flags: void (struct tdb_context *, unsigned int)
+tdb_append: int (struct tdb_context *, TDB_DATA, TDB_DATA)
+tdb_chainlock: int (struct tdb_context *, TDB_DATA)
+tdb_chainlock_mark: int (struct tdb_context *, TDB_DATA)
+tdb_chainlock_nonblock: int (struct tdb_context *, TDB_DATA)
+tdb_chainlock_read: int (struct tdb_context *, TDB_DATA)
+tdb_chainlock_read_nonblock: int (struct tdb_context *, TDB_DATA)
+tdb_chainlock_unmark: int (struct tdb_context *, TDB_DATA)
+tdb_chainunlock: int (struct tdb_context *, TDB_DATA)
+tdb_chainunlock_read: int (struct tdb_context *, TDB_DATA)
+tdb_check: int (struct tdb_context *, int (*)(TDB_DATA, TDB_DATA, void *), void *)
+tdb_close: int (struct tdb_context *)
+tdb_delete: int (struct tdb_context *, TDB_DATA)
+tdb_dump_all: void (struct tdb_context *)
+tdb_enable_seqnum: void (struct tdb_context *)
+tdb_error: enum TDB_ERROR (struct tdb_context *)
+tdb_errorstr: const char *(struct tdb_context *)
+tdb_exists: int (struct tdb_context *, TDB_DATA)
+tdb_fd: int (struct tdb_context *)
+tdb_fetch: TDB_DATA (struct tdb_context *, TDB_DATA)
+tdb_firstkey: TDB_DATA (struct tdb_context *)
+tdb_freelist_size: int (struct tdb_context *)
+tdb_get_flags: int (struct tdb_context *)
+tdb_get_logging_private: void *(struct tdb_context *)
+tdb_get_seqnum: int (struct tdb_context *)
+tdb_hash_size: int (struct tdb_context *)
+tdb_increment_seqnum_nonblock: void (struct tdb_context *)
+tdb_jenkins_hash: unsigned int (TDB_DATA *)
+tdb_lock_nonblock: int (struct tdb_context *, int, int)
+tdb_lockall: int (struct tdb_context *)
+tdb_lockall_mark: int (struct tdb_context *)
+tdb_lockall_nonblock: int (struct tdb_context *)
+tdb_lockall_read: int (struct tdb_context *)
+tdb_lockall_read_nonblock: int (struct tdb_context *)
+tdb_lockall_unmark: int (struct tdb_context *)
+tdb_log_fn: tdb_log_func (struct tdb_context *)
+tdb_map_size: size_t (struct tdb_context *)
+tdb_name: const char *(struct tdb_context *)
+tdb_nextkey: TDB_DATA (struct tdb_context *, TDB_DATA)
+tdb_null: dptr = 0xXXXX, dsize = 0
+tdb_open: struct tdb_context *(const char *, int, int, int, mode_t)
+tdb_open_ex: struct tdb_context *(const char *, int, int, int, mode_t, const struct tdb_logging_context *, tdb_hash_func)
+tdb_parse_record: int (struct tdb_context *, TDB_DATA, int (*)(TDB_DATA, TDB_DATA, void *), void *)
+tdb_printfreelist: int (struct tdb_context *)
+tdb_remove_flags: void (struct tdb_context *, unsigned int)
+tdb_reopen: int (struct tdb_context *)
+tdb_reopen_all: int (int)
+tdb_repack: int (struct tdb_context *)
+tdb_rescue: int (struct tdb_context *, void (*)(TDB_DATA, TDB_DATA, void *), void *)
+tdb_runtime_check_for_robust_mutexes: bool (void)
+tdb_set_logging_function: void (struct tdb_context *, const struct tdb_logging_context *)
+tdb_set_max_dead: void (struct tdb_context *, int)
+tdb_setalarm_sigptr: void (struct tdb_context *, volatile sig_atomic_t *)
+tdb_store: int (struct tdb_context *, TDB_DATA, TDB_DATA, int)
+tdb_storev: int (struct tdb_context *, TDB_DATA, const TDB_DATA *, int, int)
+tdb_summary: char *(struct tdb_context *)
+tdb_transaction_active: bool (struct tdb_context *)
+tdb_transaction_cancel: int (struct tdb_context *)
+tdb_transaction_commit: int (struct tdb_context *)
+tdb_transaction_prepare_commit: int (struct tdb_context *)
+tdb_transaction_start: int (struct tdb_context *)
+tdb_transaction_start_nonblock: int (struct tdb_context *)
+tdb_transaction_write_lock_mark: int (struct tdb_context *)
+tdb_transaction_write_lock_unmark: int (struct tdb_context *)
+tdb_traverse: int (struct tdb_context *, tdb_traverse_func, void *)
+tdb_traverse_chain: int (struct tdb_context *, unsigned int, tdb_traverse_func, void *)
+tdb_traverse_key_chain: int (struct tdb_context *, TDB_DATA, tdb_traverse_func, void *)
+tdb_traverse_read: int (struct tdb_context *, tdb_traverse_func, void *)
+tdb_unlock: int (struct tdb_context *, int, int)
+tdb_unlockall: int (struct tdb_context *)
+tdb_unlockall_read: int (struct tdb_context *)
+tdb_validate_freelist: int (struct tdb_context *, int *)
+tdb_wipe_all: int (struct tdb_context *)
diff --git a/ABI/tdb-1.3.18.sigs b/ABI/tdb-1.3.18.sigs
 new file mode 100644
index 0000000..e2b0427
--- /dev/null
+++ b/ABI/tdb-1.3.18.sigs
@@ -0,0 +1,73 @@
+tdb_add_flags: void (struct tdb_context *, unsigned int)
+tdb_append: int (struct tdb_context *, TDB_DATA, TDB_DATA)
+tdb_chainlock: int (struct tdb_context *, TDB_DATA)
+tdb_chainlock_mark: int (struct tdb_context *, TDB_DATA)
+tdb_chainlock_nonblock: int (struct tdb_context *, TDB_DATA)
+tdb_chainlock_read: int (struct tdb_context *, TDB_DATA)
+tdb_chainlock_read_nonblock: int (struct tdb_context *, TDB_DATA)
+tdb_chainlock_unmark: int (struct tdb_context *, TDB_DATA)
+tdb_chainunlock: int (struct tdb_context *, TDB_DATA)
+tdb_chainunlock_read: int (struct tdb_context *, TDB_DATA)
+tdb_check: int (struct tdb_context *, int (*)(TDB_DATA, TDB_DATA, void *), void *)
+tdb_close: int (struct tdb_context *)
+tdb_delete: int (struct tdb_context *, TDB_DATA)
+tdb_dump_all: void (struct tdb_context *)
+tdb_enable_seqnum: void (struct tdb_context *)
+tdb_error: enum TDB_ERROR (struct tdb_context *)
+tdb_errorstr: const char *(struct tdb_context *)
+tdb_exists: int (struct tdb_context *, TDB_DATA)
+tdb_fd: int (struct tdb_context *)
+tdb_fetch: TDB_DATA (struct tdb_context *, TDB_DATA)
+tdb_firstkey: TDB_DATA (struct tdb_context *)
+tdb_freelist_size: int (struct tdb_context *)
+tdb_get_flags: int (struct tdb_context *)
+tdb_get_logging_private: void *(struct tdb_context *)
+tdb_get_seqnum: int (struct tdb_context *)
+tdb_hash_size: int (struct tdb_context *)
+tdb_increment_seqnum_nonblock: void (struct tdb_context *)
+tdb_jenkins_hash: unsigned int (TDB_DATA *)
+tdb_lock_nonblock: int (struct tdb_context *, int, int)
+tdb_lockall: int (struct tdb_context *)
+tdb_lockall_mark: int (struct tdb_context *)
+tdb_lockall_nonblock: int (struct tdb_context *)
+tdb_lockall_read: int (struct tdb_context *)
+tdb_lockall_read_nonblock: int (struct tdb_context *)
+tdb_lockall_unmark: int (struct tdb_context *)
+tdb_log_fn: tdb_log_func (struct tdb_context *)
+tdb_map_size: size_t (struct tdb_context *)
+tdb_name: const char *(struct tdb_context *)
+tdb_nextkey: TDB_DATA (struct tdb_context *, TDB_DATA)
+tdb_null: dptr = 0xXXXX, dsize = 0
+tdb_open: struct tdb_context *(const char *, int, int, int, mode_t)
+tdb_open_ex: struct tdb_context *(const char *, int, int, int, mode_t, const struct tdb_logging_context *, tdb_hash_func)
+tdb_parse_record: int (struct tdb_context *, TDB_DATA, int (*)(TDB_DATA, TDB_DATA, void *), void *)
+tdb_printfreelist: int (struct tdb_context *)
+tdb_remove_flags: void (struct tdb_context *, unsigned int)
+tdb_reopen: int (struct tdb_context *)
+tdb_reopen_all: int (int)
+tdb_repack: int (struct tdb_context *)
+tdb_rescue: int (struct tdb_context *, void (*)(TDB_DATA, TDB_DATA, void *), void *)
+tdb_runtime_check_for_robust_mutexes: bool (void)
+tdb_set_logging_function: void (struct tdb_context *, const struct tdb_logging_context *)
+tdb_set_max_dead: void (struct tdb_context *, int)
+tdb_setalarm_sigptr: void (struct tdb_context *, volatile sig_atomic_t *)
+tdb_store: int (struct tdb_context *, TDB_DATA, TDB_DATA, int)
+tdb_storev: int (struct tdb_context *, TDB_DATA, const TDB_DATA *, int, int)
+tdb_summary: char *(struct tdb_context *)
+tdb_transaction_active: bool (struct tdb_context *)
+tdb_transaction_cancel: int (struct tdb_context *)
+tdb_transaction_commit: int (struct tdb_context *)
+tdb_transaction_prepare_commit: int (struct tdb_context *)
+tdb_transaction_start: int (struct tdb_context *)
+tdb_transaction_start_nonblock: int (struct tdb_context *)
+tdb_transaction_write_lock_mark: int (struct tdb_context *)
+tdb_transaction_write_lock_unmark: int (struct tdb_context *)
+tdb_traverse: int (struct tdb_context *, tdb_traverse_func, void *)
+tdb_traverse_chain: int (struct tdb_context *, unsigned int, tdb_traverse_func, void *)
+tdb_traverse_key_chain: int (struct tdb_context *, TDB_DATA, tdb_traverse_func, void *)
+tdb_traverse_read: int (struct tdb_context *, tdb_traverse_func, void *)
+tdb_unlock: int (struct tdb_context *, int, int)
+tdb_unlockall: int (struct tdb_context *)
+tdb_unlockall_read: int (struct tdb_context *)
+tdb_validate_freelist: int (struct tdb_context *, int *)
+tdb_wipe_all: int (struct tdb_context *)
diff --git a/Makefile b/Makefile
index fe44ff6..8fd56c8 100644
--- a/Makefile
+++ b/Makefile
@@ -1,6 +1,8 @@
 # simple makefile wrapper to run waf
 
-WAF=WAF_MAKE=1 PATH=buildtools/bin:../../buildtools/bin:$$PATH waf
+WAF_BIN=`PATH=buildtools/bin:../../buildtools/bin:$$PATH which waf`
+WAF_BINARY=$(PYTHON) $(WAF_BIN)
+WAF=PYTHONHASHSEED=1 WAF_MAKE=1 $(WAF_BINARY)
 
 all:
 	$(WAF) build
diff --git a/_tdb_text.py b/_tdb_text.py
index c823bf8..f3caa53 100644
--- a/_tdb_text.py
+++ b/_tdb_text.py
@@ -4,7 +4,6 @@
 # Published under the GNU LGPLv3 or later
 
 import sys
-import functools
 
 import tdb
 
diff --git a/buildtools/bin/waf b/buildtools/bin/waf
index 1b0f466..3ee4d5b 100755
--- a/buildtools/bin/waf
+++ b/buildtools/bin/waf
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
-# encoding: ISO-8859-1
-# Thomas Nagy, 2005-2010
-
+#!/usr/bin/env python3
+# encoding: latin-1
+# Thomas Nagy, 2005-2018
+#
 """
 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions
@@ -30,25 +30,24 @@ IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 POSSIBILITY OF SUCH DAMAGE.
 """
 
-import os, sys
-if sys.hexversion<0x203000f: raise ImportError("Waf requires Python >= 2.3")
-
-if 'PSYCOWAF' in os.environ:
-	try:import psyco;psyco.full()
-	except:pass
+import os, sys, inspect
 
-VERSION="1.5.19"
+VERSION="2.0.8"
 REVISION="x"
+GIT="x"
 INSTALL="x"
 C1='x'
 C2='x'
+C3='x'
 cwd = os.getcwd()
 join = os.path.join
 
+if sys.hexversion<0x206000f:
+	raise ImportError('Python >= 2.6 is required to create the waf file')
+
 WAF='waf'
 def b(x):
 	return x
-
 if sys.hexversion>0x300000f:
 	WAF='waf3'
 	def b(x):
@@ -58,20 +57,111 @@ def err(m):
 	print(('\033[91mError: %s\033[0m' % m))
 	sys.exit(1)
 
-def test(dir):
-	try: os.stat(join(dir, 'wafadmin')); return os.path.abspath(dir)
+def unpack_wafdir(dir, src):
+	f = open(src,'rb')
+	c = 'corrupt archive (%d)'
+	while 1:
+		line = f.readline()
+		if not line: err('run waf-light from a folder containing waflib')
+		if line == b('#==>\n'):
+			txt = f.readline()
+			if not txt: err(c % 1)
+			if f.readline() != b('#<==\n'): err(c % 2)
+			break
+	if not txt: err(c % 3)
+	txt = txt[1:-1].replace(b(C1), b('\n')).replace(b(C2), b('\r')).replace(b(C3), b('\x00'))
+
+	import shutil, tarfile
+	try: shutil.rmtree(dir)
 	except OSError: pass
+	try:
+		for x in ('Tools', 'extras'):
+			os.makedirs(join(dir, 'waflib', x))
+	except OSError:
+		err("Cannot unpack waf lib into %s\nMove waf in a writable directory" % dir)
+
+	os.chdir(dir)
+	tmp = 't.bz2'
+	t = open(tmp,'wb')
+	try: t.write(txt)
+	finally: t.close()
+
+	try:
+		t = tarfile.open(tmp)
+	except:
+		try:
+			os.system('bunzip2 t.bz2')
+			t = tarfile.open('t')
+			tmp = 't'
+		except:
+			os.chdir(cwd)
+			try: shutil.rmtree(dir)
+			except OSError: pass
+			err("Waf cannot be unpacked, check that bzip2 support is present")
+
+	try:
+		for x in t: t.extract(x)
+	finally:
+		t.close()
+
+	for x in ('Tools', 'extras'):
+		os.chmod(join('waflib',x), 493)
+
+	if sys.hexversion<0x300000f:
+		sys.path = [join(dir, 'waflib')] + sys.path
+		import fixpy2
+		fixpy2.fixdir(dir)
+
+	os.remove(tmp)
+	os.chdir(cwd)
+
+	try: dir = unicode(dir, 'mbcs')
+	except: pass
+	try:
+		from ctypes import windll
+		windll.kernel32.SetFileAttributesW(dir, 2)
+	except:
+		pass
+
+def test(dir):
+	try:
+		os.stat(join(dir, 'waflib'))
+		return os.path.abspath(dir)
+	except OSError:
+		pass
 
 def find_lib():
-	return os.path.abspath(os.path.join(os.path.dirname(__file__), '../../third_party/waf'))
+	path = '../../third_party/waf'
+	paths = [path, path+'/waflib']
+	return [os.path.abspath(os.path.join(os.path.dirname(__file__), x)) for x in paths]
 
 wafdir = find_lib()
-w = join(wafdir, 'wafadmin')
-t = join(w, 'Tools')
-f = join(w, '3rdparty')
-sys.path = [w, t, f] + sys.path
+for p in wafdir:
+	sys.path.insert(0, p)
 
 if __name__ == '__main__':
-	import Scripting
-	Scripting.prepare(t, cwd, VERSION, wafdir)
+	#import extras.compat15#PRELUDE
+	import sys
+
+	from waflib.Tools import ccroot, c, ar, compiler_c, gcc
+	sys.modules['cc'] = c
+	sys.modules['ccroot'] = ccroot
+	sys.modules['ar'] = ar
+	sys.modules['compiler_cc'] = compiler_c
+	sys.modules['gcc'] = gcc
+
+	from waflib import Options
+	Options.lockfile = os.environ.get('WAFLOCK', '.lock-wscript')
+	if os.path.isfile(Options.lockfile) and os.stat(Options.lockfile).st_size == 0:
+		os.environ['NOCLIMB'] = "1"
+	# there is a single top-level, but libraries must build independently
+	os.environ['NO_LOCK_IN_TOP'] = "1"
+
+	from waflib import Task
+	class o(object):
+		display = None
+	Task.classes['cc_link'] = o
+
+	from waflib import Scripting
+	Scripting.waf_entry_point(cwd, VERSION, wafdir[0])
 
diff --git a/buildtools/examples/run_on_target.py b/buildtools/examples/run_on_target.py
index 8322759..79c5730 100755
--- a/buildtools/examples/run_on_target.py
+++ b/buildtools/examples/run_on_target.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 #
 # Sample run-on-target script
diff --git a/buildtools/scripts/abi_gen.sh b/buildtools/scripts/abi_gen.sh
index 787718c..6dd6d32 100755
--- a/buildtools/scripts/abi_gen.sh
+++ b/buildtools/scripts/abi_gen.sh
@@ -17,5 +17,5 @@ done
 ) > $GDBSCRIPT
 
 # forcing the terminal avoids a problem on Fedora12
-TERM=none gdb -batch -x $GDBSCRIPT "$SHAREDLIB" < /dev/null
+TERM=none gdb -n -batch -x $GDBSCRIPT "$SHAREDLIB" < /dev/null
 rm -f $GDBSCRIPT
diff --git a/buildtools/wafsamba/configure_file.py b/buildtools/wafsamba/configure_file.py
index e28282b..6ad4354 100644
--- a/buildtools/wafsamba/configure_file.py
+++ b/buildtools/wafsamba/configure_file.py
@@ -1,7 +1,9 @@
 # handle substitution of variables in .in files
 
-import re, os
-import Build, sys, Logs
+import sys
+import re
+import os
+from waflib import Build, Logs
 from samba_utils import SUBST_VARS_RECURSIVE
 
 def subst_at_vars(task):
diff --git a/buildtools/wafsamba/generic_cc.py b/buildtools/wafsamba/generic_cc.py
index 504e902..1352c54 100644
--- a/buildtools/wafsamba/generic_cc.py
+++ b/buildtools/wafsamba/generic_cc.py
@@ -3,69 +3,68 @@
 # based on suncc.py from waf
 
 import os, optparse
-import Utils, Options, Configure
-import ccroot, ar
-from Configure import conftest
+from waflib import Errors
+from waflib.Tools import ccroot, ar
+from waflib.Configure import conf
 
-from compiler_cc import c_compiler
+#
+# Let waflib provide useful defaults, but
+# provide generic_cc as last resort fallback on
+# all platforms
+#
+from waflib.Tools.compiler_c import c_compiler
+for key in c_compiler.keys():
+    c_compiler[key].append('generic_cc')
 
-c_compiler['default'] = ['gcc', 'generic_cc']
-c_compiler['hpux'] = ['gcc', 'generic_cc']
-
-@conftest
+@conf
 def find_generic_cc(conf):
     v = conf.env
     cc = None
-    if v['CC']: cc = v['CC']
-    elif 'CC' in conf.environ: cc = conf.environ['CC']
-    if not cc: cc = conf.find_program('cc', var='CC')
-    if not cc: conf.fatal('generic_cc was not found')
-    cc = conf.cmd_to_list(cc)
-    v['CC']  = cc
-    v['CC_NAME'] = 'generic'
-
-@conftest
-def generic_cc_common_flags(conf):
-    v = conf.env
+    if v.CC:
+        cc = v.CC
+    elif 'CC' in conf.environ:
+        cc = conf.environ['CC']
+    if not cc:
+        cc = conf.find_program('cc', var='CC')
+    if not cc:
+        conf.fatal('generic_cc was not found')
 
-    v['CC_SRC_F']            = ''
-    v['CC_TGT_F']            = ['-c', '-o', '']
-    v['CPPPATH_ST']          = '-I%s' # template for adding include paths
+    try:
+        conf.cmd_and_log(cc + ['--version'])
+    except Errors.WafError:
+        conf.fatal('%r --version could not be executed' % cc)
 
-    # linker
-    if not v['LINK_CC']: v['LINK_CC'] = v['CC']
-    v['CCLNK_SRC_F']         = ''
-    v['CCLNK_TGT_F']         = ['-o', '']
+    v.CC = cc
+    v.CC_NAME = 'generic_cc'
 
-    v['LIB_ST']              = '-l%s' # template for adding libs
-    v['LIBPATH_ST']          = '-L%s' # template for adding libpaths
-    v['STATICLIB_ST']        = '-l%s'
-    v['STATICLIBPATH_ST']    = '-L%s'
-    v['CCDEFINES_ST']        = '-D%s'
+@conf
+def generic_cc_common_flags(conf):
+    v = conf.env
 
-#    v['SONAME_ST']           = '-Wl,-h -Wl,%s'
-#    v['SHLIB_MARKER']        = '-Bdynamic'
-#    v['STATICLIB_MARKER']    = '-Bstatic'
+    v.CC_SRC_F            = ''
+    v.CC_TGT_F            = ['-c', '-o']
+    v.CPPPATH_ST          = '-I%s'
+    v.DEFINES_ST          = '-D%s'
 
-    # program
-    v['program_PATTERN']     = '%s'
+    if not v.LINK_CC:
+        v.LINK_CC = v.CC
 
-    # shared library
-#    v['shlib_CCFLAGS']       = ['-Kpic', '-DPIC']
-#    v['shlib_LINKFLAGS']     = ['-G']
-    v['shlib_PATTERN']       = 'lib%s.so'
+    v.CCLNK_SRC_F         = ''
+    v.CCLNK_TGT_F         = ['-o']
 
-    # static lib
-#    v['staticlib_LINKFLAGS'] = ['-Bstatic']
-#    v['staticlib_PATTERN']   = 'lib%s.a'
+    v.LIB_ST              = '-l%s' # template for adding libs
+    v.LIBPATH_ST          = '-L%s' # template for adding libpaths
+    v.STLIB_ST            = '-l%s'
+    v.STLIBPATH_ST        = '-L%s'
 
-detect = '''
-find_generic_cc
-find_cpp
-find_ar
-generic_cc_common_flags
-cc_load_tools
-cc_add_flags
-link_add_flags
-'''
+    v.cprogram_PATTERN    = '%s'
+    v.cshlib_PATTERN      = 'lib%s.so'
+    v.cstlib_PATTERN      = 'lib%s.a'
 
+def configure(conf):
+    conf.find_generic_cc()
+    conf.find_ar()
+    conf.generic_cc_common_flags()
+    conf.cc_load_tools()
+    conf.cc_add_flags()
+    conf.link_add_flags()
diff --git a/buildtools/wafsamba/hpuxcc.py b/buildtools/wafsamba/hpuxcc.py
 deleted file mode 100644
index c263556..0000000
--- a/buildtools/wafsamba/hpuxcc.py
+++ /dev/null
@@ -1,56 +0,0 @@
-# compiler definition for HPUX
-# based on suncc.py from waf
-
-import os, optparse, sys
-import Utils, Options, Configure
-import ccroot, ar
-from Configure import conftest
-import gcc
-
-
-@conftest
-def gcc_modifier_hpux(conf):
-    v=conf.env
-    v['CCFLAGS_DEBUG']=['-g']
-    v['CCFLAGS_RELEASE']=['-O2']
-    v['CC_SRC_F']=''
-    v['CC_TGT_F']=['-c','-o','']
-    v['CPPPATH_ST']='-I%s'
-    if not v['LINK_CC']:v['LINK_CC']=v['CC']
-    v['CCLNK_SRC_F']=''
-    v['CCLNK_TGT_F']=['-o','']
-    v['LIB_ST']='-l%s'
-    v['LIBPATH_ST']='-L%s'
-    v['STATICLIB_ST']='-l%s'
-    v['STATICLIBPATH_ST']='-L%s'
-    v['RPATH_ST']='-Wl,-rpath,%s'
-    v['CCDEFINES_ST']='-D%s'
-    v['SONAME_ST']='-Wl,-h,%s'
-    v['SHLIB_MARKER']=[]
-#    v['STATICLIB_MARKER']='-Wl,-Bstatic'
-    v['FULLSTATIC_MARKER']='-static'
-    v['program_PATTERN']='%s'
-    v['shlib_CCFLAGS']=['-fPIC','-DPIC']
-    v['shlib_LINKFLAGS']=['-shared']
-    v['shlib_PATTERN']='lib%s.sl'
-#   v['staticlib_LINKFLAGS']=['-Wl,-Bstatic']
-    v['staticlib_PATTERN']='lib%s.a'
-
-gcc.gcc_modifier_hpux = gcc_modifier_hpux
-
-from TaskGen import feature, after
-@feature('cprogram', 'cshlib')
-@after('apply_link', 'apply_lib_vars', 'apply_obj_vars')
-def hpux_addfullpath(self):
-  if sys.platform == 'hp-ux11':
-    link = getattr(self, 'link_task', None)
-    if link:
-        lst = link.env.LINKFLAGS
-        buf = []
-        for x in lst:
-           if x.startswith('-L'):
-               p2 = x[2:]
-               if not os.path.isabs(p2):
-                   x = x[:2] + self.bld.srcnode.abspath(link.env) + "/../" + x[2:].lstrip('.')
-           buf.append(x)
-        link.env.LINKFLAGS = buf
diff --git a/buildtools/wafsamba/irixcc.py b/buildtools/wafsamba/irixcc.py
 deleted file mode 100644
index f3cb451..0000000
--- a/buildtools/wafsamba/irixcc.py
+++ /dev/null
@@ -1,79 +0,0 @@
-
-# compiler definition for irix/MIPSpro cc compiler
-# based on suncc.py from waf
-
-import os, optparse
-import Utils, Options, Configure
-import ccroot, ar
-from Configure import conftest
-
-from compiler_cc import c_compiler
-
-c_compiler['irix'] = ['gcc', 'irixcc']
-
-@conftest
-def find_irixcc(conf):
-    v = conf.env
-    cc = None
-    if v['CC']: cc = v['CC']
-    elif 'CC' in conf.environ: cc = conf.environ['CC']
-    if not cc: cc = conf.find_program('cc', var='CC')
-    if not cc: conf.fatal('irixcc was not found')
-    cc = conf.cmd_to_list(cc)
-
-    try:
-        if Utils.cmd_output(cc + ['-c99'] + ['-version']) != '':
-            conf.fatal('irixcc %r was not found' % cc)
-    except ValueError:
-        conf.fatal('irixcc -v could not be executed')
-
-    conf.env.append_unique('CCFLAGS', '-c99')
-
-    v['CC']  = cc
-    v['CC_NAME'] = 'irix'
-
-@conftest
-def irixcc_common_flags(conf):
-    v = conf.env
-
-    v['CC_SRC_F']            = ''
-    v['CC_TGT_F']            = ['-c', '-o', '']
-    v['CPPPATH_ST']          = '-I%s' # template for adding include paths
-
-    # linker
-    if not v['LINK_CC']: v['LINK_CC'] = v['CC']
-    v['CCLNK_SRC_F']         = ''
-    v['CCLNK_TGT_F']         = ['-o', '']
-
-    v['LIB_ST']              = '-l%s' # template for adding libs
-    v['LIBPATH_ST']          = '-L%s' # template for adding libpaths
-    v['STATICLIB_ST']        = '-l%s'
-    v['STATICLIBPATH_ST']    = '-L%s'
-    v['CCDEFINES_ST']        = '-D%s'
-
-#    v['SONAME_ST']           = '-Wl,-h -Wl,%s'
-#    v['SHLIB_MARKER']        = '-Bdynamic'
-#    v['STATICLIB_MARKER']    = '-Bstatic'
-
-    # program
-    v['program_PATTERN']     = '%s'
-
-    # shared library
-#    v['shlib_CCFLAGS']       = ['-Kpic', '-DPIC']
-#    v['shlib_LINKFLAGS']     = ['-G']
-    v['shlib_PATTERN']       = 'lib%s.so'
-
-    # static lib
-#    v['staticlib_LINKFLAGS'] = ['-Bstatic']
-#    v['staticlib_PATTERN']   = 'lib%s.a'
-
-detect = '''
-find_irixcc
-find_cpp
-find_ar
-irixcc_common_flags
-cc_load_tools
-cc_add_flags
-link_add_flags
-'''
-
diff --git a/buildtools/wafsamba/nothreads.py b/buildtools/wafsamba/nothreads.py
 deleted file mode 100644
index 9bd33e8..0000000
--- a/buildtools/wafsamba/nothreads.py
+++ /dev/null
@@ -1,219 +0,0 @@
-# encoding: utf-8
-# Thomas Nagy, 2005-2008 (ita)
-
-# this replaces the core of Runner.py in waf with a varient that works
-# on systems with completely broken threading (such as Python 2.5.x on
-# AIX). For simplicity we enable this when JOBS=1, which is triggered
-# by the compatibility makefile used for the waf build. That also ensures
-# this code is tested, as it means it is used in the build farm, and by
-# anyone using 'make' to build Samba with waf
-
-"Execute the tasks"
-
-import sys, random, threading
-try: from Queue import Queue
-except ImportError: from queue import Queue
-import Utils, Options
-from Constants import EXCEPTION, CRASHED, MAXJOBS, ASK_LATER, SKIPPED, SKIP_ME, SUCCESS
-
-GAP = 15
-
-run_old = threading.Thread.run
-def run(*args, **kwargs):
-    try:
-        run_old(*args, **kwargs)
-    except (KeyboardInterrupt, SystemExit):
-        raise
-    except:
-        sys.excepthook(*sys.exc_info())
-threading.Thread.run = run
-
-
-class TaskConsumer(object):
-    consumers = 1
-
-def process(tsk):
-    m = tsk.master
-    if m.stop:
-        m.out.put(tsk)
-        return
-
-    try:
-        tsk.generator.bld.printout(tsk.display())
-        if tsk.__class__.stat: ret = tsk.__class__.stat(tsk)
-        # actual call to task's run() function
-        else: ret = tsk.call_run()
-    except Exception as e:
-        tsk.err_msg = Utils.ex_stack()
-        tsk.hasrun = EXCEPTION
-
-        # TODO cleanup
-        m.error_handler(tsk)
-        m.out.put(tsk)
-        return
-
-    if ret:
-        tsk.err_code = ret
-        tsk.hasrun = CRASHED
-    else:
-        try:
-            tsk.post_run()
-        except Utils.WafError:
-            pass
-        except Exception:
-            tsk.err_msg = Utils.ex_stack()
-            tsk.hasrun = EXCEPTION
-        else:
-            tsk.hasrun = SUCCESS
-    if tsk.hasrun != SUCCESS:
-        m.error_handler(tsk)
-
-    m.out.put(tsk)
-
-class Parallel(object):
-    """
-    keep the consumer threads busy, and avoid consuming cpu cycles
-    when no more tasks can be added (end of the build, etc)
-    """
-    def __init__(self, bld, j=2):
-
-        # number of consumers
-        self.numjobs = j
-
-        self.manager = bld.task_manager
-        self.manager.current_group = 0
-
-        self.total = self.manager.total()
-
-        # tasks waiting to be processed - IMPORTANT
-        self.outstanding = []
-        self.maxjobs = MAXJOBS
-
-        # tasks that are awaiting for another task to complete
-        self.frozen = []
-
-        # tasks returned by the consumers
-        self.out = Queue(0)
-
-        self.count = 0 # tasks not in the producer area
-
-        self.processed = 1 # progress indicator
-
-        self.stop = False # error condition to stop the build
-        self.error = False # error flag
-
-    def get_next(self):
-        "override this method to schedule the tasks in a particular order"
-        if not self.outstanding:
-            return None
-        return self.outstanding.pop(0)
-
-    def postpone(self, tsk):
-        "override this method to schedule the tasks in a particular order"
-        # TODO consider using a deque instead
-        if random.randint(0, 1):
-            self.frozen.insert(0, tsk)
-        else:
-            self.frozen.append(tsk)
-
-    def refill_task_list(self):
-        "called to set the next group of tasks"
-
-        while self.count > self.numjobs + GAP or self.count >= self.maxjobs:
-            self.get_out()
-
-        while not self.outstanding:
-            if self.count:
-                self.get_out()
-
-            if self.frozen:
-                self.outstanding += self.frozen
-                self.frozen = []
-            elif not self.count:
-                (jobs, tmp) = self.manager.get_next_set()
-                if jobs is not None:
-                    self.maxjobs = jobs
-                if tmp:
-                    self.outstanding += tmp
-                break
-
-    def get_out(self):
-        "the tasks that are put to execute are all collected using get_out"
-        ret = self.out.get()
-        self.manager.add_finished(ret)
-        if not self.stop and getattr(ret, 'more_tasks', None):
-            self.outstanding += ret.more_tasks
-            self.total += len(ret.more_tasks)
-        self.count -= 1
-
-    def error_handler(self, tsk):
-        "by default, errors make the build stop (not thread safe so be careful)"
-        if not Options.options.keep:
-            self.stop = True
-        self.error = True
-
-    def start(self):
-        "execute the tasks"
-
-        while not self.stop:
-
-            self.refill_task_list()
-
-            # consider the next task
-            tsk = self.get_next()
-            if not tsk:
-                if self.count:
-                    # tasks may add new ones after they are run
-                    continue
-                else:
-                    # no tasks to run, no tasks running, time to exit
-                    break
-
-            if tsk.hasrun:
-                # if the task is marked as "run", just skip it
-                self.processed += 1
-                self.manager.add_finished(tsk)
-                continue
-
-            try:
-                st = tsk.runnable_status()
-            except Exception as e:
-                self.processed += 1
-                if self.stop and not Options.options.keep:
-                    tsk.hasrun = SKIPPED
-                    self.manager.add_finished(tsk)
-                    continue
-                self.error_handler(tsk)
-                self.manager.add_finished(tsk)
-                tsk.hasrun = EXCEPTION
-                tsk.err_msg = Utils.ex_stack()
-                continue
-
-            if st == ASK_LATER:
-                self.postpone(tsk)
-            elif st == SKIP_ME:
-                self.processed += 1
-                tsk.hasrun = SKIPPED
-                self.manager.add_finished(tsk)
-            else:
-                # run me: put the task in ready queue
-                tsk.position = (self.processed, self.total)
-                self.count += 1
-                self.processed += 1
-                tsk.master = self
-
-                process(tsk)
-
-        # self.count represents the tasks that have been made available to the consumer threads
-        # collect all the tasks after an error else the message may be incomplete
-        while self.error and self.count:
-            self.get_out()
-
-        #print loop
-        assert (self.count == 0 or self.stop)
-
-
-# enable nothreads
-import Runner
-Runner.process = process
-Runner.Parallel = Parallel
diff --git a/buildtools/wafsamba/pkgconfig.py b/buildtools/wafsamba/pkgconfig.py
index 999bad4..b83d5f3 100644
--- a/buildtools/wafsamba/pkgconfig.py
+++ b/buildtools/wafsamba/pkgconfig.py
@@ -1,7 +1,7 @@
 # handle substitution of variables in pc files
 
 import os, re, sys
-import Build, Logs
+from waflib import Build, Logs
 from samba_utils import SUBST_VARS_RECURSIVE, TO_LIST
 
 def subst_at_vars(task):
@@ -52,7 +52,7 @@ def PKG_CONFIG_FILES(bld, pc_files, vnum=None, extra_name=None):
                                 rule=subst_at_vars,
                                 source=f+'.in',
                                 target=target)
-        bld.add_manual_dependency(bld.path.find_or_declare(f), bld.env['PREFIX'])
+        bld.add_manual_dependency(bld.path.find_or_declare(f), bld.env['PREFIX'].encode('utf8'))
         t.vars = []
         if t.env.RPATH_ON_INSTALL:
             t.env.LIB_RPATH = t.env.RPATH_ST % t.env.LIBDIR
diff --git a/buildtools/wafsamba/samba3.py b/buildtools/wafsamba/samba3.py
index 44daff9..5aab250 100644
--- a/buildtools/wafsamba/samba3.py
+++ b/buildtools/wafsamba/samba3.py
@@ -1,12 +1,11 @@
 # a waf tool to add autoconf-like macros to the configure section
 # and for SAMBA_ macros for building libraries, binaries etc
 
-import Options, Build, os
-from samba_utils import os_path_relpath, TO_LIST, samba_add_onoff_option
+import os
+from waflib import Build
+from samba_utils import os_path_relpath, TO_LIST
 from samba_autoconf import library_flags
 
-Options.Handler.SAMBA3_ADD_OPTION = samba_add_onoff_option
-
 def SAMBA3_IS_STATIC_MODULE(bld, module):
     '''Check whether module is in static list'''
     if module in bld.env['static_modules']:
@@ -32,7 +31,7 @@ def s3_fix_kwargs(bld, kwargs):
     '''fix the build arguments for s3 build rules to include the
     necessary includes, subdir and cflags options '''
     s3dir = os.path.join(bld.env.srcdir, 'source3')
-    s3reldir = os_path_relpath(s3dir, bld.curdir)
+    s3reldir = os_path_relpath(s3dir, bld.path.abspath())
 
     # the extra_includes list is relative to the source3 directory
     extra_includes = [ '.', 'include', 'lib' ]
diff --git a/buildtools/wafsamba/samba_abi.py b/buildtools/wafsamba/samba_abi.py
index 196b468..5e7686d 100644
--- a/buildtools/wafsamba/samba_abi.py
+++ b/buildtools/wafsamba/samba_abi.py
@@ -1,7 +1,13 @@
 # functions for handling ABI checking of libraries
 
-import Options, Utils, os, Logs, samba_utils, sys, Task, fnmatch, re, Build
-from TaskGen import feature, before, after
+import os
+import sys
+import re
+import fnmatch
+
+from waflib import Options, Utils, Logs, Task, Build, Errors
+from waflib.TaskGen import feature, before, after
+from wafsamba import samba_utils
 
 # these type maps cope with platform specific names for common types
 # please add new type mappings into the list below
@@ -10,7 +16,7 @@ abi_type_maps = {
     'struct __va_list_tag *' : 'va_list'
     }
 
-version_key = lambda x: map(int, x.split("."))
+version_key = lambda x: list(map(int, x.split(".")))
 
 def normalise_signature(sig):
     '''normalise a signature from gdb'''
@@ -79,7 +85,7 @@ def abi_check_task(self):
     libpath = self.inputs[0].abspath(self.env)
     libname = os.path.basename(libpath)
 
-    sigs = Utils.cmd_output([abi_gen, libpath])
+    sigs = samba_utils.get_string(Utils.cmd_output([abi_gen, libpath]))
     parsed_sigs = parse_sigs(sigs, self.ABI_MATCH)
 
     sig_file = self.ABI_FILE
@@ -87,7 +93,7 @@ def abi_check_task(self):
     old_sigs = samba_utils.load_file(sig_file)
     if old_sigs is None or Options.options.ABI_UPDATE:
         if not save_sigs(sig_file, parsed_sigs):
-            raise Utils.WafError('Failed to save ABI file "%s"' % sig_file)
+            raise Errors.WafError('Failed to save ABI file "%s"' % sig_file)
         Logs.warn('Generated ABI signatures %s' % sig_file)
         return
 
@@ -112,14 +118,14 @@ def abi_check_task(self):
             got_error = True
 
     if got_error:
-        raise Utils.WafError('ABI for %s has changed - please fix library version then build with --abi-update\nSee http://wiki.samba.org/index.php/Waf#ABI_Checking for more information\nIf you have not changed any ABI, and your platform always gives this error, please configure with --abi-check-disable to skip this check' % libname)
+        raise Errors.WafError('ABI for %s has changed - please fix library version then build with --abi-update\nSee http://wiki.samba.org/index.php/Waf#ABI_Checking for more information\nIf you have not changed any ABI, and your platform always gives this error, please configure with --abi-check-disable to skip this check' % libname)
 
 
-t = Task.task_type_from_func('abi_check', abi_check_task, color='BLUE', ext_in='.bin')
+t = Task.task_factory('abi_check', abi_check_task, color='BLUE', ext_in='.bin')
 t.quiet = True
 # allow "waf --abi-check" to force re-checking the ABI
 if '--abi-check' in sys.argv:
-    Task.always_run(t)
+    t.always_run = True
 
 @after('apply_link')
 @feature('abi_check')
@@ -184,18 +190,20 @@ def abi_write_vscript(f, libname, current_version, versions, symmap, abi_match):
         f.write("}%s;\n\n" % last_key)
         last_key = " %s" % symver
     f.write("%s {\n" % current_version)
-    local_abi = filter(lambda x: x[0] == '!', abi_match)
-    global_abi = filter(lambda x: x[0] != '!', abi_match)
+    local_abi = list(filter(lambda x: x[0] == '!', abi_match))
+    global_abi = list(filter(lambda x: x[0] != '!', abi_match))
     f.write("\tglobal:\n")
     if len(global_abi) > 0:
         for x in global_abi:
             f.write("\t\t%s;\n" % x)
     else:
         f.write("\t\t*;\n")
-    if abi_match != ["*"]:
-        f.write("\tlocal:\n")
-        for x in local_abi:
-            f.write("\t\t%s;\n" % x[1:])
+    # Always hide symbols that must be local if exist
+    local_abi.extend(["!_end", "!__bss_start", "!_edata"])
+    f.write("\tlocal:\n")
+    for x in local_abi:
+        f.write("\t\t%s;\n" % x[1:])
+    if global_abi != ["*"]:
         if len(global_abi) > 0:
             f.write("\t\t*;\n")
     f.write("};\n")
diff --git a/buildtools/wafsamba/samba_autoconf.py b/buildtools/wafsamba/samba_autoconf.py
index c4391d0..ee1fc23 100644
--- a/buildtools/wafsamba/samba_autoconf.py
+++ b/buildtools/wafsamba/samba_autoconf.py
@@ -1,9 +1,10 @@
 # a waf tool to add autoconf-like macros to the configure section
 
 import os, sys
-import Build, Options, preproc, Logs
-from Configure import conf
-from TaskGen import feature
+from waflib import Build, Options, Logs, Context
+from waflib.Configure import conf
+from waflib.TaskGen import feature
+from waflib.Tools import c_preproc as preproc
 from samba_utils import TO_LIST, GET_TARGET_TYPE, SET_TARGET_TYPE, unique_list, mkdir_p
 
 missing_headers = set()
@@ -18,7 +19,7 @@ def DEFINE(conf, d, v, add_to_cflags=False, quote=False):
     '''define a config option'''
     conf.define(d, v, quote=quote)
     if add_to_cflags:
-        conf.env.append_value('CCDEFINES', d + '=' + str(v))
+        conf.env.append_value('CFLAGS', '-D%s=%s' % (d, str(v)))
 
 def hlist_to_string(conf, headers=None):
     '''convert a headers list to a set of #include lines'''
@@ -44,11 +45,11 @@ def COMPOUND_START(conf, msg):
     if v != [] and v != 0:
         conf.env.in_compound = v + 1
         return
-    conf.check_message_1(msg)
-    conf.saved_check_message_1 = conf.check_message_1
-    conf.check_message_1 = null_check_message_1
-    conf.saved_check_message_2 = conf.check_message_2
-    conf.check_message_2 = null_check_message_2
+    conf.start_msg(msg)
+    conf.saved_check_message_1 = conf.start_msg
+    conf.start_msg = null_check_message_1
+    conf.saved_check_message_2 = conf.end_msg
+    conf.end_msg = null_check_message_2
     conf.env.in_compound = 1
 
 
@@ -58,9 +59,9 @@ def COMPOUND_END(conf, result):
     conf.env.in_compound -= 1
     if conf.env.in_compound != 0:
         return
-    conf.check_message_1 = conf.saved_check_message_1
-    conf.check_message_2 = conf.saved_check_message_2
-    p = conf.check_message_2
+    conf.start_msg = conf.saved_check_message_1
+    conf.end_msg = conf.saved_check_message_2
+    p = conf.end_msg
     if result is True:
         p('ok')
     elif not result:
@@ -96,10 +97,10 @@ def CHECK_HEADER(conf, h, add_headers=False, lib=None):
     hdrs = hlist_to_string(conf, headers=h)
     if lib is None:
         lib = ""
-    ret = conf.check(fragment='%s\nint main(void) { return 0; }' % hdrs,
+    ret = conf.check(fragment='%s\nint main(void) { return 0; }\n' % hdrs,
                      type='nolink',
                      execute=0,
-                     ccflags=ccflags,
+                     cflags=ccflags,
                      mandatory=False,
                      includes=cpppath,
                      uselib=lib.upper(),
@@ -250,7 +251,10 @@ def CHECK_FUNC(conf, f, link=True, lib=None, headers=None):
 
     ret = False
 
-    conf.COMPOUND_START('Checking for %s' % f)
+    in_lib_str = ""
+    if lib:
+        in_lib_str = " in %s" % lib
+    conf.COMPOUND_START('Checking for %s%s' % (f, in_lib_str))
 
     if link is None or link:
         ret = CHECK_CODE(conf,
@@ -322,7 +326,7 @@ def CHECK_SIZEOF(conf, vars, headers=None, define=None, critical=True):
         ret = False
         if v_define is None:
             v_define = 'SIZEOF_%s' % v.upper().replace(' ', '_')
-        for size in list((1, 2, 4, 8, 16, 32)):
+        for size in list((1, 2, 4, 8, 16, 32, 64)):
             if CHECK_CODE(conf,
                       'static int test_array[1 - 2 * !(((long int)(sizeof(%s))) <= %d)];' % (v, size),
                       define=v_define,
@@ -383,12 +387,10 @@ def CHECK_CODE(conf, code, define,
     else:
         execute = 0
 
-    defs = conf.get_config_header()
-
     if addmain:
-        fragment='%s\n%s\n int main(void) { %s; return 0; }\n' % (defs, hdrs, code)
+        fragment='%s\n int main(void) { %s; return 0; }\n' % (hdrs, code)
     else:
-        fragment='%s\n%s\n%s\n' % (defs, hdrs, code)
+        fragment='%s\n%s\n' % (hdrs, code)
 
     if msg is None:
         msg="Checking for %s" % define
@@ -398,15 +400,11 @@ def CHECK_CODE(conf, code, define,
     # Be strict when relying on a compiler check
     # Some compilers (e.g. xlc) ignore non-supported features as warnings
     if strict:
-        extra_cflags = None
-        if conf.env["CC_NAME"] == "gcc":
-            extra_cflags = "-Werror"
-        elif conf.env["CC_NAME"] == "xlc":
-            extra_cflags = "-qhalt=w"
-        cflags.append(extra_cflags)
+        if 'WERROR_CFLAGS' in conf.env:
+            cflags.extend(conf.env['WERROR_CFLAGS'])
 
     if local_include:
-        cflags.append('-I%s' % conf.curdir)
+        cflags.append('-I%s' % conf.path.abspath())
 
     if not link:
         type='nolink'
@@ -431,11 +429,11 @@ def CHECK_CODE(conf, code, define,
 
     conf.COMPOUND_START(msg)
 
-    ret = conf.check(fragment=fragment,
+    try:
+        ret = conf.check(fragment=fragment,
                      execute=execute,
                      define_name = define,
-                     mandatory = mandatory,
-                     ccflags=cflags,
+                     cflags=cflags,
                      ldflags=ldflags,
                      includes=includes,
                      uselib=uselib,
@@ -444,22 +442,30 @@ def CHECK_CODE(conf, code, define,
                      quote=quote,
                      exec_args=exec_args,
                      define_ret=define_ret)
-    if not ret and CONFIG_SET(conf, define):
-        # sometimes conf.check() returns false, but it
-        # sets the define. Maybe a waf bug?
-        ret = True
-    if ret:
+    except Exception:
+        if always:
+            conf.DEFINE(define, 0)
+        else:
+            conf.undefine(define)
+        conf.COMPOUND_END(False)
+        if mandatory:
+            raise
+        return False
+    else:
+        # Success is indicated by ret but we should unset
+        # defines set by WAF's c_config.check() because it
+        # defines it to int(ret) and we want to undefine it
+        if not ret:
+            conf.undefine(define)
+            conf.COMPOUND_END(False)
+            return False
         if not define_ret:
             conf.DEFINE(define, 1)
             conf.COMPOUND_END(True)
         else:
-            conf.COMPOUND_END(conf.env[define])
+            conf.DEFINE(define, ret, quote=quote)
+            conf.COMPOUND_END(ret)
         return True
-    if always:
-        conf.DEFINE(define, 0)
-    conf.COMPOUND_END(False)
-    return False
-
 
 
 @conf
@@ -490,8 +496,9 @@ def CHECK_CFLAGS(conf, cflags, fragment='int main(void) { return 0; }\n'):
         check_cflags.extend(conf.env['WERROR_CFLAGS'])
     return conf.check(fragment=fragment,
                       execute=0,
+                      mandatory=False,
                       type='nolink',
-                      ccflags=check_cflags,
+                      cflags=check_cflags,
                       msg="Checking compiler accepts %s" % cflags)
 
 @conf
@@ -547,12 +554,15 @@ def library_flags(self, libs):
         # note that we do not add the -I and -L in here, as that is added by the waf
         # core. Adding it here would just change the order that it is put on the link line
         # which can cause system paths to be added before internal libraries
-        extra_ccflags = TO_LIST(getattr(self.env, 'CCFLAGS_%s' % lib.upper(), []))
+        extra_ccflags = TO_LIST(getattr(self.env, 'CFLAGS_%s' % lib.upper(), []))
         extra_ldflags = TO_LIST(getattr(self.env, 'LDFLAGS_%s' % lib.upper(), []))
         extra_cpppath = TO_LIST(getattr(self.env, 'CPPPATH_%s' % lib.upper(), []))
         ccflags.extend(extra_ccflags)
         ldflags.extend(extra_ldflags)
         cpppath.extend(extra_cpppath)
+
+        extra_cpppath = TO_LIST(getattr(self.env, 'INCLUDES_%s' % lib.upper(), []))
+        cpppath.extend(extra_cpppath)
     if 'EXTRA_LDFLAGS' in self.env:
         ldflags.extend(self.env['EXTRA_LDFLAGS'])
 
@@ -585,9 +595,9 @@ int foo()
 
         (ccflags, ldflags, cpppath) = library_flags(conf, lib)
         if shlib:
-            res = conf.check(features='c cshlib', fragment=fragment, lib=lib, uselib_store=lib, ccflags=ccflags, ldflags=ldflags, uselib=lib.upper(), mandatory=False)
+            res = conf.check(features='c cshlib', fragment=fragment, lib=lib, uselib_store=lib, cflags=ccflags, ldflags=ldflags, uselib=lib.upper(), mandatory=False)
         else:
-            res = conf.check(lib=lib, uselib_store=lib, ccflags=ccflags, ldflags=ldflags, uselib=lib.upper(), mandatory=False)
+            res = conf.check(lib=lib, uselib_store=lib, cflags=ccflags, ldflags=ldflags, uselib=lib.upper(), mandatory=False)
 
         if not res:
             if mandatory:
@@ -661,8 +671,8 @@ def CHECK_FUNCS_IN(conf, list, library, mandatory=False, checklibc=False,
 @conf
 def IN_LAUNCH_DIR(conf):
     '''return True if this rule is being run from the launch directory'''
-    return os.path.realpath(conf.curdir) == os.path.realpath(Options.launch_dir)
-Options.Handler.IN_LAUNCH_DIR = IN_LAUNCH_DIR
+    return os.path.realpath(conf.path.abspath()) == os.path.realpath(Context.launch_dir)
+Options.OptionsContext.IN_LAUNCH_DIR = IN_LAUNCH_DIR
 
 
 @conf
@@ -674,23 +684,42 @@ def SAMBA_CONFIG_H(conf, path=None):
         return
 
     # we need to build real code that can't be optimized away to test
-    if conf.check(fragment='''
-        #include <stdio.h>
-
-        int main(void)
-        {
-            char t[100000];
-            while (fgets(t, sizeof(t), stdin));
-            return 0;
-        }
-        ''',
-        execute=0,
-        ccflags='-fstack-protector',
-        ldflags='-fstack-protector',
-        mandatory=False,
-        msg='Checking if toolchain accepts -fstack-protector'):
-            conf.ADD_CFLAGS('-fstack-protector')
-            conf.ADD_LDFLAGS('-fstack-protector')
+    stack_protect_list = ['-fstack-protector-strong', '-fstack-protector']
+    for stack_protect_flag in stack_protect_list:
+        flag_supported = conf.check(fragment='''
+                                    #include <stdio.h>
+
+                                    int main(void)
+                                    {
+                                        char t[100000];
+                                        while (fgets(t, sizeof(t), stdin));
+                                        return 0;
+                                    }
+                                    ''',
+                                    execute=0,
+                                    cflags=[ '-Werror', '-Wp,-D_FORTIFY_SOURCE=2', stack_protect_flag],
+                                    mandatory=False,
+                                    msg='Checking if compiler accepts %s' % (stack_protect_flag))
+        if flag_supported:
+            conf.ADD_CFLAGS('%s' % (stack_protect_flag))
+            break
+
+    flag_supported = conf.check(fragment='''
+                                #include <stdio.h>
+
+                                int main(void)
+                                {
+                                    char t[100000];
+                                    while (fgets(t, sizeof(t), stdin));
+                                    return 0;
+                                }
+                                ''',
+                                execute=0,
+                                cflags=[ '-Werror', '-fstack-clash-protection'],
+                                mandatory=False,
+                                msg='Checking if compiler accepts -fstack-clash-protection')
+    if flag_supported:
+        conf.ADD_CFLAGS('-fstack-clash-protection')
 
     if Options.options.debug:
         conf.ADD_CFLAGS('-g', testflags=True)
@@ -774,9 +803,12 @@ int main(void) {
         conf.env['EXTRA_LDFLAGS'].extend(conf.env['ADDITIONAL_LDFLAGS'])
 
     if path is None:
-        conf.write_config_header('config.h', top=True)
+        conf.write_config_header('default/config.h', top=True, remove=False)
     else:
-        conf.write_config_header(path)
+        conf.write_config_header(os.path.join(conf.variant, path), remove=False)
+    for key in conf.env.define_key:
+        conf.undefine(key, from_env=False)
+    conf.env.define_key = []
     conf.SAMBA_CROSS_CHECK_COMPLETE()
 
 
@@ -863,9 +895,6 @@ def CHECK_CC_ENV(conf):
     The build farm sometimes puts a space at the start"""
     if os.environ.get('CC'):
         conf.env.CC = TO_LIST(os.environ.get('CC'))
-        if len(conf.env.CC) == 1:
-            # make for nicer logs if just a single command
-            conf.env.CC = conf.env.CC[0]
 
 
 @conf
@@ -875,7 +904,7 @@ def SETUP_CONFIGURE_CACHE(conf, enable):
         # when -C is chosen, we will use a private cache and will
         # not look into system includes. This roughtly matches what
         # autoconf does with -C
-        cache_path = os.path.join(conf.blddir, '.confcache')
+        cache_path = os.path.join(conf.bldnode.abspath(), '.confcache')
         mkdir_p(cache_path)
         Options.cache_global = os.environ['WAFCACHE'] = cache_path
     else:
@@ -899,6 +928,3 @@ def SAMBA_CHECK_UNDEFINED_SYMBOL_FLAGS(conf):
         if conf.CHECK_LDFLAGS(['-undefined', 'dynamic_lookup']):
             conf.env.undefined_ignore_ldflags = ['-undefined', 'dynamic_lookup']
 
-@conf
-def CHECK_CFG(self, *k, **kw):
-    return self.check_cfg(*k, **kw)
diff --git a/buildtools/wafsamba/samba_autoproto.py b/buildtools/wafsamba/samba_autoproto.py
index b2b5233..ace434f 100644
--- a/buildtools/wafsamba/samba_autoproto.py
+++ b/buildtools/wafsamba/samba_autoproto.py
@@ -1,13 +1,13 @@
 # waf build tool for building automatic prototypes from C source
 
 import os
-import Build
+from waflib import Build
 from samba_utils import SET_TARGET_TYPE, os_path_relpath
 
 def SAMBA_AUTOPROTO(bld, header, source):
     '''rule for samba prototype generation'''
     bld.SET_BUILD_GROUP('prototypes')
-    relpath = os_path_relpath(bld.curdir, bld.srcnode.abspath())
+    relpath = os_path_relpath(bld.path.abspath(), bld.srcnode.abspath())
     name = os.path.join(relpath, header)
     SET_TARGET_TYPE(bld, name, 'PROTOTYPE')
     t = bld(
@@ -16,7 +16,7 @@ def SAMBA_AUTOPROTO(bld, header, source):
         target = header,
         update_outputs=True,
         ext_out='.c',
-        before ='cc',
+        before ='c',
         rule = '${PERL} "${SCRIPT}/mkproto.pl" --srcdir=.. --builddir=. --public=/dev/null --private="${TGT}" ${SRC}'
         )
     t.env.SCRIPT = os.path.join(bld.srcnode.abspath(), 'source4/script')
diff --git a/buildtools/wafsamba/samba_bundled.py b/buildtools/wafsamba/samba_bundled.py
index 253d604..60ce7da 100644
--- a/buildtools/wafsamba/samba_bundled.py
+++ b/buildtools/wafsamba/samba_bundled.py
@@ -1,9 +1,9 @@
 # functions to support bundled libraries
 
 import sys
-import Build, Options, Logs
-from Configure import conf
-from samba_utils import TO_LIST
+from waflib import Build, Options, Logs
+from waflib.Configure import conf
+from wafsamba import samba_utils
 
 def PRIVATE_NAME(bld, name, private_extension, private_library):
     '''possibly rename a library to include a bundled extension'''
@@ -51,19 +51,19 @@ Build.BuildContext.BUILTIN_LIBRARY = BUILTIN_LIBRARY
 
 def BUILTIN_DEFAULT(opt, builtins):
     '''set a comma separated default list of builtin libraries for this package'''
-    if 'BUILTIN_LIBRARIES_DEFAULT' in Options.options:
+    if 'BUILTIN_LIBRARIES_DEFAULT' in Options.options.__dict__:
         return
-    Options.options['BUILTIN_LIBRARIES_DEFAULT'] = builtins
-Options.Handler.BUILTIN_DEFAULT = BUILTIN_DEFAULT
+    Options.options.__dict__['BUILTIN_LIBRARIES_DEFAULT'] = builtins
+Options.OptionsContext.BUILTIN_DEFAULT = BUILTIN_DEFAULT
 
 
 def PRIVATE_EXTENSION_DEFAULT(opt, extension, noextension=''):
     '''set a default private library extension'''
-    if 'PRIVATE_EXTENSION_DEFAULT' in Options.options:
+    if 'PRIVATE_EXTENSION_DEFAULT' in Options.options.__dict__:
         return
-    Options.options['PRIVATE_EXTENSION_DEFAULT'] = extension
-    Options.options['PRIVATE_EXTENSION_EXCEPTION'] = noextension
-Options.Handler.PRIVATE_EXTENSION_DEFAULT = PRIVATE_EXTENSION_DEFAULT
+    Options.options.__dict__['PRIVATE_EXTENSION_DEFAULT'] = extension
+    Options.options.__dict__['PRIVATE_EXTENSION_EXCEPTION'] = noextension
+Options.OptionsContext.PRIVATE_EXTENSION_DEFAULT = PRIVATE_EXTENSION_DEFAULT
 
 
 def minimum_library_version(conf, libname, default):
@@ -139,7 +139,7 @@ def CHECK_BUNDLED_SYSTEM(conf, libname, minversion='0.0.0',
     # We always do a logic validation of 'onlyif' first
     missing = []
     if onlyif:
-        for l in TO_LIST(onlyif):
+        for l in samba_utils.TO_LIST(onlyif):
             f = 'FOUND_SYSTEMLIB_%s' % l
             if not f in conf.env:
                 Logs.error('ERROR: CHECK_BUNDLED_SYSTEM(%s) - ' % (libname) +
diff --git a/buildtools/wafsamba/samba_conftests.py b/buildtools/wafsamba/samba_conftests.py
index b52727b..ef632ba 100644
--- a/buildtools/wafsamba/samba_conftests.py
+++ b/buildtools/wafsamba/samba_conftests.py
@@ -2,34 +2,35 @@
 # to test for commonly needed configuration options
 
 import os, shutil, re
-import Build, Configure, Utils, Options, Logs
-from Configure import conf
-from samba_utils import TO_LIST, ADD_LD_LIBRARY_PATH
+from waflib import Build, Configure, Utils, Options, Logs, Errors
+from waflib.Configure import conf
+from samba_utils import TO_LIST, ADD_LD_LIBRARY_PATH, get_string
 
 
 def add_option(self, *k, **kw):
     '''syntax help: provide the "match" attribute to opt.add_option() so that folders can be added to specific config tests'''
+    Options.OptionsContext.parser = self
     match = kw.get('match', [])
     if match:
         del kw['match']
     opt = self.parser.add_option(*k, **kw)
     opt.match = match
     return opt
-Options.Handler.add_option = add_option
+Options.OptionsContext.add_option = add_option
 
 @conf
 def check(self, *k, **kw):
     '''Override the waf defaults to inject --with-directory options'''
 
     if not 'env' in kw:
-        kw['env'] = self.env.copy()
+        kw['env'] = self.env.derive()
 
     # match the configuration test with specific options, for example:
     # --with-libiconv -> Options.options.iconv_open -> "Checking for library iconv"
     additional_dirs = []
     if 'msg' in kw:
         msg = kw['msg']
-        for x in Options.Handler.parser.parser.option_list:
+        for x in Options.OptionsContext.parser.parser.option_list:
              if getattr(x, 'match', None) and msg in x.match:
                  d = getattr(Options.options, x.dest, '')
                  if d:
@@ -46,12 +47,12 @@ def check(self, *k, **kw):
     add_options_dir(additional_dirs, kw['env'])
 
     self.validate_c(kw)
-    self.check_message_1(kw['msg'])
+    self.start_msg(kw['msg'])
     ret = None
     try:
         ret = self.run_c_code(*k, **kw)
     except Configure.ConfigurationError as e:
-        self.check_message_2(kw['errmsg'], 'YELLOW')
+        self.end_msg(kw['errmsg'], 'YELLOW')
         if 'mandatory' in kw and kw['mandatory']:
             if Logs.verbose > 1:
                 raise
@@ -59,7 +60,7 @@ def check(self, *k, **kw):
                 self.fatal('the configuration failed (see %r)' % self.log.name)
     else:
         kw['success'] = ret
-        self.check_message_2(self.ret_msg(kw['okmsg'], kw))
+        self.end_msg(self.ret_msg(kw['okmsg'], kw))
 
         # success! keep the CPPPATH/LIBPATH
         add_options_dir(additional_dirs, self.env)
@@ -85,7 +86,7 @@ def CHECK_LARGEFILE(conf, define='HAVE_LARGEFILE'):
     '''see what we need for largefile support'''
     getconf_cflags = conf.CHECK_COMMAND(['getconf', 'LFS_CFLAGS']);
     if getconf_cflags is not False:
-        if (conf.CHECK_CODE('return !(sizeof(off_t) >= 8)',
+        if (conf.CHECK_CODE('if (sizeof(off_t) < 8) return 1',
                             define='WORKING_GETCONF_LFS_CFLAGS',
                             execute=True,
                             cflags=getconf_cflags,
@@ -100,13 +101,13 @@ def CHECK_LARGEFILE(conf, define='HAVE_LARGEFILE'):
                     else:
                         conf.DEFINE(flag_split[0], flag_split[1])
 
-    if conf.CHECK_CODE('return !(sizeof(off_t) >= 8)',
+    if conf.CHECK_CODE('if (sizeof(off_t) < 8) return 1',
                        define,
                        execute=True,
                        msg='Checking for large file support without additional flags'):
         return True
 
-    if conf.CHECK_CODE('return !(sizeof(off_t) >= 8)',
+    if conf.CHECK_CODE('if (sizeof(off_t) < 8) return 1',
                        define,
                        execute=True,
                        cflags='-D_FILE_OFFSET_BITS=64',
@@ -114,7 +115,7 @@ def CHECK_LARGEFILE(conf, define='HAVE_LARGEFILE'):
         conf.DEFINE('_FILE_OFFSET_BITS', 64)
         return True
 
-    if conf.CHECK_CODE('return !(sizeof(off_t) >= 8)',
+    if conf.CHECK_CODE('if (sizeof(off_t) < 8) return 1',
                        define,
                        execute=True,
                        cflags='-D_LARGE_FILES',
@@ -162,7 +163,7 @@ def find_config_dir(conf):
     '''find a directory to run tests in'''
     k = 0
     while k < 10000:
-        dir = os.path.join(conf.blddir, '.conf_check_%d' % k)
+        dir = os.path.join(conf.bldnode.abspath(), '.conf_check_%d' % k)
         try:
             shutil.rmtree(dir)
         except OSError:
@@ -257,7 +258,8 @@ int foo(int v) {
     environ[0] = 1;
     ldb_module = PyImport_ImportModule("ldb");
     return v * 2;
-}'''
+}
+'''
     return conf.check(features='c cshlib',uselib='PYEMBED',fragment=snip,msg=msg, mandatory=False)
 
 # this one is quite complex, and should probably be broken up
@@ -337,7 +339,8 @@ def CHECK_LIBRARY_SUPPORT(conf, rpath=False, version_script=False, msg=None):
 
     # we need to run the program, try to get its result
     args = conf.SAMBA_CROSS_ARGS(msg=msg)
-    proc = Utils.pproc.Popen([lastprog] + args, stdout=Utils.pproc.PIPE, stderr=Utils.pproc.PIPE)
+    proc = Utils.subprocess.Popen([lastprog] + args,
+            stdout=Utils.subprocess.PIPE, stderr=Utils.subprocess.PIPE)
     (out, err) = proc.communicate()
     w = conf.log.write
     w(str(out))
@@ -364,7 +367,7 @@ def CHECK_PERL_MANPAGE(conf, msg=None, section=None):
         else:
             msg = "perl manpage generation"
 
-    conf.check_message_1(msg)
+    conf.start_msg(msg)
 
     dir = find_config_dir(conf)
 
@@ -381,28 +384,28 @@ WriteMakefile(
 """)
     back = os.path.abspath('.')
     os.chdir(bdir)
-    proc = Utils.pproc.Popen(['perl', 'Makefile.PL'],
-                             stdout=Utils.pproc.PIPE,
-                             stderr=Utils.pproc.PIPE)
+    proc = Utils.subprocess.Popen(['perl', 'Makefile.PL'],
+                             stdout=Utils.subprocess.PIPE,
+                             stderr=Utils.subprocess.PIPE)
     (out, err) = proc.communicate()
     os.chdir(back)
 
     ret = (proc.returncode == 0)
     if not ret:
-        conf.check_message_2('not found', color='YELLOW')
+        conf.end_msg('not found', color='YELLOW')
         return
 
     if section:
         man = Utils.readf(os.path.join(bdir,'Makefile'))
         m = re.search('MAN%sEXT\s+=\s+(\w+)' % section, man)
         if not m:
-            conf.check_message_2('not found', color='YELLOW')
+            conf.end_msg('not found', color='YELLOW')
             return
         ext = m.group(1)
-        conf.check_message_2(ext)
+        conf.end_msg(ext)
         return ext
 
-    conf.check_message_2('ok')
+    conf.end_msg('ok')
     return True
 
 
@@ -416,7 +419,7 @@ def CHECK_COMMAND(conf, cmd, msg=None, define=None, on_target=True, boolean=Fals
     if on_target:
         cmd.extend(conf.SAMBA_CROSS_ARGS(msg=msg))
     try:
-        ret = Utils.cmd_output(cmd)
+        ret = get_string(Utils.cmd_output(cmd))
     except:
         conf.COMPOUND_END(False)
         return False
@@ -461,7 +464,7 @@ def CHECK_INLINE(conf):
         ret = conf.CHECK_CODE('''
         typedef int foo_t;
         static %s foo_t static_foo () {return 0; }
-        %s foo_t foo () {return 0; }''' % (i, i),
+        %s foo_t foo () {return 0; }\n''' % (i, i),
                               define='INLINE_MACRO',
                               addmain=False,
                               link=False)
@@ -486,13 +489,13 @@ def CHECK_XSLTPROC_MANPAGES(conf):
         return False
 
     s='http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl'
-    conf.CHECK_COMMAND('%s --nonet %s 2> /dev/null' % (conf.env.XSLTPROC, s),
+    conf.CHECK_COMMAND('%s --nonet %s 2> /dev/null' % (conf.env.get_flat('XSLTPROC'), s),
                              msg='Checking for stylesheet %s' % s,
                              define='XSLTPROC_MANPAGES', on_target=False,
                              boolean=True)
     if not conf.CONFIG_SET('XSLTPROC_MANPAGES'):
-        print "A local copy of the docbook.xsl wasn't found on your system" \
-              " consider installing package like docbook-xsl"
+        print("A local copy of the docbook.xsl wasn't found on your system" \
+              " consider installing package like docbook-xsl")
 
 #
 # Determine the standard libpath for the used compiler,
@@ -506,12 +509,12 @@ def CHECK_STANDARD_LIBPATH(conf):
     # at least gcc and clang support this:
     try:
         cmd = conf.env.CC + ['-print-search-dirs']
-        out = Utils.cmd_output(cmd).split('\n')
+        out = get_string(Utils.cmd_output(cmd)).split('\n')
     except ValueError:
         # option not supported by compiler - use a standard list of directories
         dirlist = [ '/usr/lib', '/usr/lib64' ]
     except:
-        raise Utils.WafError('Unexpected error running "%s"' % (cmd))
+        raise Errors.WafError('Unexpected error running "%s"' % (cmd))
     else:
         dirlist = []
         for line in out:
diff --git a/buildtools/wafsamba/samba_cross.py b/buildtools/wafsamba/samba_cross.py
index b8f2000..8863c2c 100644
--- a/buildtools/wafsamba/samba_cross.py
+++ b/buildtools/wafsamba/samba_cross.py
@@ -1,8 +1,9 @@
 # functions for handling cross-compilation
 
 import os, sys, re, shlex
-import Utils, Logs, Options
-from Configure import conf
+from waflib import Utils, Logs, Options, Errors, Context
+from waflib.Configure import conf
+from wafsamba import samba_utils
 
 real_Popen = None
 
@@ -81,12 +82,12 @@ def cross_answer(ca_file, msg):
                     f.close()
                     return (int(m.group(1)), m.group(2))
                 else:
-                    raise Utils.WafError("Bad answer format '%s' in %s" % (line, ca_file))
+                    raise Errors.WafError("Bad answer format '%s' in %s" % (line, ca_file))
     f.close()
     return ANSWER_UNKNOWN
 
 
-class cross_Popen(Utils.pproc.Popen):
+class cross_Popen(Utils.subprocess.Popen):
     '''cross-compilation wrapper for Popen'''
     def __init__(*k, **kw):
         (obj, args) = k
@@ -118,10 +119,10 @@ class cross_Popen(Utils.pproc.Popen):
             newargs.extend(args[0:i])
             if use_answers:
                 p = real_Popen(newargs,
-                               stdout=Utils.pproc.PIPE,
-                               stderr=Utils.pproc.PIPE)
+                               stdout=Utils.subprocess.PIPE,
+                               stderr=Utils.subprocess.PIPE)
                 ce_out, ce_err = p.communicate()
-                ans = (p.returncode, ce_out)
+                ans = (p.returncode, samba_utils.get_string(ce_out))
                 add_answer(ca_file, msg, ans)
             else:
                 args = newargs
@@ -144,8 +145,8 @@ def SAMBA_CROSS_ARGS(conf, msg=None):
 
     global real_Popen
     if real_Popen is None:
-        real_Popen  = Utils.pproc.Popen
-        Utils.pproc.Popen = cross_Popen
+        real_Popen  = Utils.subprocess.Popen
+        Utils.subprocess.Popen = cross_Popen
 
     ret = []
 
@@ -154,11 +155,11 @@ def SAMBA_CROSS_ARGS(conf, msg=None):
 
     if conf.env.CROSS_ANSWERS:
         if msg is None:
-            raise Utils.WafError("Cannot have NULL msg in cross-answers")
-        ret.extend(['--cross-answers', os.path.join(Options.launch_dir, conf.env.CROSS_ANSWERS), msg])
+            raise Errors.WafError("Cannot have NULL msg in cross-answers")
+        ret.extend(['--cross-answers', os.path.join(Context.launch_dir, conf.env.CROSS_ANSWERS), msg])
 
     if ret == []:
-        raise Utils.WafError("Cannot cross-compile without either --cross-execute or --cross-answers")
+        raise Errors.WafError("Cannot cross-compile without either --cross-execute or --cross-answers")
 
     return ret
 
@@ -167,5 +168,5 @@ def SAMBA_CROSS_CHECK_COMPLETE(conf):
     '''check if we have some unanswered questions'''
     global cross_answers_incomplete
     if conf.env.CROSS_COMPILE and cross_answers_incomplete:
-        raise Utils.WafError("Cross answers file %s is incomplete" % conf.env.CROSS_ANSWERS)
+        raise Errors.WafError("Cross answers file %s is incomplete" % conf.env.CROSS_ANSWERS)
     return True
diff --git a/buildtools/wafsamba/samba_deps.py b/buildtools/wafsamba/samba_deps.py
index 978a5e9..f8c3880 100644
--- a/buildtools/wafsamba/samba_deps.py
+++ b/buildtools/wafsamba/samba_deps.py
@@ -2,9 +2,10 @@
 
 import os, sys, re, time
 
-import Build, Environment, Options, Logs, Utils
-from Logs import debug
-from Configure import conf
+from waflib import Build, Options, Logs, Utils, Errors
+from waflib.Logs import debug
+from waflib.Configure import conf
+from waflib import ConfigSet
 
 from samba_bundled import BUILTIN_LIBRARY
 from samba_utils import LOCAL_CACHE, TO_LIST, get_tgt_list, unique_list, os_path_relpath
@@ -85,7 +86,7 @@ def build_dependencies(self):
         # extra link flags from pkg_config
         libs = self.final_syslibs.copy()
 
-        (ccflags, ldflags, cpppath) = library_flags(self, list(libs))
+        (cflags, ldflags, cpppath) = library_flags(self, list(libs))
         new_ldflags        = getattr(self, 'samba_ldflags', [])[:]
         new_ldflags.extend(ldflags)
         self.ldflags       = new_ldflags
@@ -102,7 +103,7 @@ def build_dependencies(self):
               self.sname, self.uselib, self.uselib_local, self.add_objects)
 
     if self.samba_type in ['SUBSYSTEM']:
-        # this is needed for the ccflags of libs that come from pkg_config
+        # this is needed for the cflags of libs that come from pkg_config
         self.uselib = list(self.final_syslibs)
         self.uselib.extend(list(self.direct_syslibs))
         for lib in self.final_libs:
@@ -235,7 +236,7 @@ def add_init_functions(self):
         if sentinel == 'NULL':
             proto = "extern void __%s_dummy_module_proto(void)" % (sname)
             cflags.append('-DSTATIC_%s_MODULES_PROTO=%s' % (sname, proto))
-        self.ccflags = cflags
+        self.cflags = cflags
         return
 
     for m in modules:
@@ -257,7 +258,7 @@ def add_init_functions(self):
                 proto += '_MODULE_PROTO(%s)' % f
             proto += "extern void __%s_dummy_module_proto(void)" % (m)
             cflags.append('-DSTATIC_%s_MODULES_PROTO=%s' % (m, proto))
-    self.ccflags = cflags
+    self.cflags = cflags
 
 
 def check_duplicate_sources(bld, tgt_list):
@@ -271,6 +272,9 @@ def check_duplicate_sources(bld, tgt_list):
         tpath = os.path.normpath(os_path_relpath(t.path.abspath(bld.env), t.env.BUILD_DIRECTORY + '/default'))
         obj_sources = set()
         for s in source_list:
+            if not isinstance(s, str):
+                print('strange path in check_duplicate_sources %r' % s)
+                s = s.abspath()
             p = os.path.normpath(os.path.join(tpath, s))
             if p in obj_sources:
                 Logs.error("ERROR: source %s appears twice in target '%s'" % (p, t.sname))
@@ -299,7 +303,7 @@ def check_duplicate_sources(bld, tgt_list):
             Logs.warn("WARNING: source %s is in more than one target: %s" % (s, subsystems[s].keys()))
         for tname in subsystems[s]:
             if len(subsystems[s][tname]) > 1:
-                raise Utils.WafError("ERROR: source %s is in more than one subsystem of target '%s': %s" % (s, tname, subsystems[s][tname]))
+                raise Errors.WafError("ERROR: source %s is in more than one subsystem of target '%s': %s" % (s, tname, subsystems[s][tname]))
 
     return True
 
@@ -372,7 +376,7 @@ def add_samba_attributes(bld, tgt_list):
         t.samba_abspath = t.path.abspath(bld.env)
         t.samba_deps_extended = t.samba_deps[:]
         t.samba_includes_extended = TO_LIST(t.samba_includes)[:]
-        t.ccflags = getattr(t, 'samba_cflags', '')
+        t.cflags = getattr(t, 'samba_cflags', '')
 
 def replace_grouping_libraries(bld, tgt_list):
     '''replace dependencies based on grouping libraries
@@ -715,6 +719,11 @@ def reduce_objects(bld, tgt_list):
                 if t.sname in rely_on:
                     dup = dup.difference(rely_on[t.sname])
                 if dup:
+                    # Do not remove duplicates of BUILTINS
+                    d = next(iter(dup))
+                    if BUILTIN_LIBRARY(bld, d):
+                        continue
+
                     debug('deps: removing dups from %s of type %s: %s also in %s %s',
                           t.sname, t.samba_type, dup, t2.samba_type, l)
                     new = new.difference(dup)
@@ -951,7 +960,7 @@ savedeps_inputs  = ['samba_deps', 'samba_includes', 'local_include', 'local_incl
                     'source', 'grouping_library', 'samba_ldflags', 'allow_undefined_symbols',
                     'use_global_deps', 'global_include' ]
 savedeps_outputs = ['uselib', 'uselib_local', 'add_objects', 'includes',
-                    'ccflags', 'ldflags', 'samba_deps_extended', 'final_libs']
+                    'cflags', 'ldflags', 'samba_deps_extended', 'final_libs']
 savedeps_outenv  = ['INC_PATHS']
 savedeps_envvars = ['NONSHARED_BINARIES', 'GLOBAL_DEPENDENCIES', 'EXTRA_CFLAGS', 'EXTRA_LDFLAGS', 'EXTRA_INCLUDES' ]
 savedeps_caches  = ['GLOBAL_DEPENDENCIES', 'TARGET_TYPE', 'INIT_FUNCTIONS', 'SYSLIB_DEPS']
@@ -960,7 +969,7 @@ savedeps_files   = ['buildtools/wafsamba/samba_deps.py']
 def save_samba_deps(bld, tgt_list):
     '''save the dependency calculations between builds, to make
        further builds faster'''
-    denv = Environment.Environment()
+    denv = ConfigSet.ConfigSet()
 
     denv.version = savedeps_version
     denv.savedeps_inputs = savedeps_inputs
@@ -1007,15 +1016,15 @@ def save_samba_deps(bld, tgt_list):
         if tdeps != {}:
             denv.outenv[t.sname] = tdeps
 
-    depsfile = os.path.join(bld.bdir, "sambadeps")
+    depsfile = os.path.join(bld.cache_dir, "sambadeps")
     denv.store_fast(depsfile)
 
 
 
 def load_samba_deps(bld, tgt_list):
     '''load a previous set of build dependencies if possible'''
-    depsfile = os.path.join(bld.bdir, "sambadeps")
-    denv = Environment.Environment()
+    depsfile = os.path.join(bld.cache_dir, "sambadeps")
+    denv = ConfigSet.ConfigSet()
     try:
         debug('deps: checking saved dependencies')
         denv.load_fast(depsfile)
diff --git a/buildtools/wafsamba/samba_dist.py b/buildtools/wafsamba/samba_dist.py
index 8d51632..6af7bb4 100644
--- a/buildtools/wafsamba/samba_dist.py
+++ b/buildtools/wafsamba/samba_dist.py
@@ -2,13 +2,41 @@
 # uses git ls-files to get file lists
 
 import os, sys, tarfile
-import Utils, Scripting, Logs, Options
-from Configure import conf
-from samba_utils import os_path_relpath
+from waflib import Utils, Scripting, Logs, Options
+from waflib.Configure import conf
+from samba_utils import os_path_relpath, get_string
+from waflib import Context
 
 dist_dirs = None
 dist_files = None
 dist_blacklist = ""
+dist_archive = None
+
+class Dist(Context.Context):
+    # TODO remove
+    cmd = 'dist'
+    fun = 'dist'
+    def execute(self):
+        Context.g_module.dist()
+
+class DistCheck(Scripting.DistCheck):
+    fun = 'distcheck'
+    cmd = 'distcheck'
+    def execute(self):
+        Options.options.distcheck_args = ''
+        if Context.g_module.distcheck is Scripting.distcheck:
+            # default
+            Context.g_module.distcheck(self)
+        else:
+            Context.g_module.distcheck()
+        Context.g_module.dist()
+        self.check()
+    def get_arch_name(self):
+        global dist_archive
+        return dist_archive
+    def make_distcheck_cmd(self, tmpdir):
+        waf = os.path.abspath(sys.argv[0])
+        return [sys.executable, waf, 'configure', 'build', 'install', 'uninstall', '--destdir=' + tmpdir]
 
 def add_symlink(tar, fname, abspath, basedir):
     '''handle symlinks to directories that may move during packaging'''
@@ -69,7 +97,7 @@ def add_tarfile(tar, fname, abspath, basedir):
     tinfo.gid   = 0
     tinfo.uname = 'root'
     tinfo.gname = 'root'
-    fh = open(abspath)
+    fh = open(abspath, "rb")
     tar.addfile(tinfo, fileobj=fh)
     fh.close()
 
@@ -91,7 +119,7 @@ def vcs_dir_contents(path):
         repo = os.path.dirname(repo)
     if repo == "/":
         raise Exception("unsupported or no vcs for %s" % path)
-    return Utils.cmd_output(ls_files_cmd, cwd=cwd, env=env).split()
+    return get_string(Utils.cmd_output(ls_files_cmd, cwd=cwd, env=env)).split('\n')
 
 
 def dist(appname='', version=''):
@@ -136,12 +164,14 @@ def dist(appname='', version=''):
 
     if not isinstance(appname, str) or not appname:
         # this copes with a mismatch in the calling arguments for dist()
-        appname = Utils.g_module.APPNAME
-        version = Utils.g_module.VERSION
+        appname = Context.g_module.APPNAME
+        version = Context.g_module.VERSION
     if not version:
-        version = Utils.g_module.VERSION
+        version = Context.g_module.VERSION
 
-    srcdir = os.path.normpath(os.path.join(os.path.dirname(Utils.g_module.root_path), Utils.g_module.srcdir))
+    srcdir = os.path.normpath(
+                os.path.join(os.path.dirname(Context.g_module.root_path),
+                    Context.g_module.top))
 
     if not dist_dirs:
         Logs.error('You must use samba_dist.DIST_DIRS() to set which directories to package')
@@ -218,6 +248,9 @@ def dist(appname='', version=''):
     else:
         Logs.info('Created %s' % dist_name)
 
+    # TODO use the ctx object instead
+    global dist_archive
+    dist_archive = dist_name
     return dist_name
 
 
diff --git a/buildtools/wafsamba/samba_git.py b/buildtools/wafsamba/samba_git.py
index c58a579..09a204f 100644
--- a/buildtools/wafsamba/samba_git.py
+++ b/buildtools/wafsamba/samba_git.py
@@ -4,7 +4,7 @@ import subprocess
 def find_git(env=None):
     """Find the git binary."""
     if env is not None and 'GIT' in env:
-        return env['GIT']
+        return env.get_flat('GIT')
 
     # Get version from GIT
     if os.path.exists("/usr/bin/git"):
diff --git a/buildtools/wafsamba/samba_headers.py b/buildtools/wafsamba/samba_headers.py
index 0a80082..a268c01 100644
--- a/buildtools/wafsamba/samba_headers.py
+++ b/buildtools/wafsamba/samba_headers.py
@@ -1,7 +1,7 @@
 # specialist handling of header files for Samba
 
 import os, re, sys, fnmatch
-import Build, Logs, Utils
+from waflib import Build, Logs, Utils, Errors
 from samba_utils import TO_LIST, os_path_relpath
 
 
@@ -99,7 +99,7 @@ def create_public_header(task):
         os.unlink(tgt)
         sys.stderr.write("%s:%u:Error: unable to resolve public header %s (maybe try one of %s)\n" % (
             os.path.relpath(src, os.getcwd()), linenumber, hpath, suggested))
-        raise Utils.WafError("Unable to resolve header path '%s' in public header '%s' in directory %s" % (
+        raise Errors.WafError("Unable to resolve header path '%s' in public header '%s' in directory %s" % (
             hpath, relsrc, task.env.RELPATH))
     infile.close()
     outfile.close()
@@ -148,11 +148,12 @@ def PUBLIC_HEADERS(bld, public_headers, header_path=None, public_headers_install
         else:
             h_name =  h
             inst_name = os.path.basename(h)
-        relpath1 = os_path_relpath(bld.srcnode.abspath(), bld.curdir)
-        relpath2 = os_path_relpath(bld.curdir, bld.srcnode.abspath())
+        curdir = bld.path.abspath()
+        relpath1 = os_path_relpath(bld.srcnode.abspath(), curdir)
+        relpath2 = os_path_relpath(curdir, bld.srcnode.abspath())
         targetdir = os.path.normpath(os.path.join(relpath1, bld.env.build_public_headers, inst_path))
-        if not os.path.exists(os.path.join(bld.curdir, targetdir)):
-            raise Utils.WafError("missing source directory %s for public header %s" % (targetdir, inst_name))
+        if not os.path.exists(os.path.join(curdir, targetdir)):
+            raise Errors.WafError("missing source directory %s for public header %s" % (targetdir, inst_name))
         target = os.path.join(targetdir, inst_name)
 
         # the source path of the header, relative to the top of the source tree
diff --git a/buildtools/wafsamba/samba_install.py b/buildtools/wafsamba/samba_install.py
index 21035bf..47bc0cb 100644
--- a/buildtools/wafsamba/samba_install.py
+++ b/buildtools/wafsamba/samba_install.py
@@ -4,8 +4,8 @@
 # library use
 
 import os
-import Utils
-from TaskGen import feature, before, after
+from waflib import Utils, Errors
+from waflib.TaskGen import feature, before, after
 from samba_utils import LIB_PATH, MODE_755, install_rpath, build_rpath
 
 @feature('install_bin')
@@ -45,7 +45,7 @@ def install_binary(self):
 
     # tell waf to install the right binary
     bld.install_as(os.path.join(install_path, orig_target),
-                   os.path.join(self.path.abspath(bld.env), self.target),
+                   self.path.find_or_declare(self.target),
                    chmod=MODE_755)
 
 
@@ -143,8 +143,9 @@ def install_library(self):
 
         # tell waf to install the library
         bld.install_as(os.path.join(install_path, install_name),
-                       os.path.join(self.path.abspath(bld.env), inst_name),
+                       self.path.find_or_declare(inst_name),
                        chmod=MODE_755)
+
         if install_link and install_link != install_name:
             # and the symlink if needed
             bld.symlink_as(os.path.join(install_path, install_link), os.path.basename(install_name))
@@ -227,7 +228,7 @@ def symlink_bin(self):
         return
 
     if not self.link_task.outputs or not self.link_task.outputs[0]:
-        raise Utils.WafError('no outputs found for %s in symlink_bin' % self.name)
+        raise Errors.WafError('no outputs found for %s in symlink_bin' % self.name)
     binpath = self.link_task.outputs[0].abspath(self.env)
     bldpath = os.path.join(self.bld.env.BUILD_DIRECTORY, self.link_task.outputs[0].name)
 
diff --git a/buildtools/wafsamba/samba_optimisation.py b/buildtools/wafsamba/samba_optimisation.py
 deleted file mode 100644
index 5008f83..0000000
--- a/buildtools/wafsamba/samba_optimisation.py
+++ /dev/null
@@ -1,269 +0,0 @@
-# This file contains waf optimisations for Samba
-
-# most of these optimisations are possible because of the restricted build environment
-# that Samba has. For example, Samba doesn't attempt to cope with Win32 paths during the
-# build, and Samba doesn't need build varients
-
-# overall this makes some build tasks quite a bit faster
-
-import os
-import Build, Utils, Node
-from TaskGen import feature, after, before
-import preproc
-
-@feature('c', 'cc', 'cxx')
-@after('apply_type_vars', 'apply_lib_vars', 'apply_core')
-def apply_incpaths(self):
-    lst = []
-
-    try:
-        kak = self.bld.kak
-    except AttributeError:
-        kak = self.bld.kak = {}
-
-    # TODO move the uselib processing out of here
-    for lib in self.to_list(self.uselib):
-        for path in self.env['CPPPATH_' + lib]:
-            if not path in lst:
-                lst.append(path)
-    if preproc.go_absolute:
-        for path in preproc.standard_includes:
-            if not path in lst:
-                lst.append(path)
-
-    for path in self.to_list(self.includes):
-        if not path in lst:
-            if preproc.go_absolute or path[0] != '/':  # os.path.isabs(path):
-                lst.append(path)
-            else:
-                self.env.prepend_value('CPPPATH', path)
-
-    for path in lst:
-        node = None
-        if path[0] == '/': # os.path.isabs(path):
-            if preproc.go_absolute:
-                node = self.bld.root.find_dir(path)
-        elif path[0] == '#':
-            node = self.bld.srcnode
-            if len(path) > 1:
-                try:
-                    node = kak[path]
-                except KeyError:
-                    kak[path] = node = node.find_dir(path[1:])
-        else:
-            try:
-                node = kak[(self.path.id, path)]
-            except KeyError:
-                kak[(self.path.id, path)] = node = self.path.find_dir(path)
-
-        if node:
-            self.env.append_value('INC_PATHS', node)
-
-@feature('c', 'cc')
-@after('apply_incpaths')
-def apply_obj_vars_cc(self):
-    """after apply_incpaths for INC_PATHS"""
-    env = self.env
-    app = env.append_unique
-    cpppath_st = env['CPPPATH_ST']
-
-    lss = env['_CCINCFLAGS']
-
-    try:
-         cac = self.bld.cac
-    except AttributeError:
-         cac = self.bld.cac = {}
-
-    # local flags come first
-    # set the user-defined includes paths
-    for i in env['INC_PATHS']:
-
-        try:
-            lss.extend(cac[i.id])
-        except KeyError:
-
-            cac[i.id] = [cpppath_st % i.bldpath(env), cpppath_st % i.srcpath(env)]
-            lss.extend(cac[i.id])
-
-    env['_CCINCFLAGS'] = lss
-    # set the library include paths
-    for i in env['CPPPATH']:
-        app('_CCINCFLAGS', cpppath_st % i)
-
-import Node, Environment
-
-def vari(self):
-    return "default"
-Environment.Environment.variant = vari
-
-def variant(self, env):
-    if not env: return 0
-    elif self.id & 3 == Node.FILE: return 0
-    else: return "default"
-Node.Node.variant = variant
-
-
-import TaskGen, Task
-
-def create_task(self, name, src=None, tgt=None):
-    task = Task.TaskBase.classes[name](self.env, generator=self)
-    if src:
-        task.set_inputs(src)
-    if tgt:
-        task.set_outputs(tgt)
-    return task
-TaskGen.task_gen.create_task = create_task
-
-def hash_constraints(self):
-    a = self.attr
-    sum = hash((str(a('before', '')),
-            str(a('after', '')),
-            str(a('ext_in', '')),
-            str(a('ext_out', '')),
-            self.__class__.maxjobs))
-    return sum
-Task.TaskBase.hash_constraints = hash_constraints
-
-def hash_env_vars(self, env, vars_lst):
-    idx = str(id(env)) + str(vars_lst)
-    try:
-        return self.cache_sig_vars[idx]
-    except KeyError:
-        pass
-
-    m = Utils.md5()
-    m.update(''.join([str(env[a]) for a in vars_lst]))
-
-    ret = self.cache_sig_vars[idx] = m.digest()
-    return ret
-Build.BuildContext.hash_env_vars = hash_env_vars
-
-
-def store_fast(self, filename):
-    file = open(filename, 'wb')
-    data = self.get_merged_dict()
-    try:
-        Build.cPickle.dump(data, file, -1)
-    finally:
-        file.close()
-Environment.Environment.store_fast = store_fast
-
-def load_fast(self, filename):
-    file = open(filename, 'rb')
-    try:
-        data = Build.cPickle.load(file)
-    finally:
-        file.close()
-    self.table.update(data)
-Environment.Environment.load_fast = load_fast
-
-def is_this_a_static_lib(self, name):
-    try:
-        cache = self.cache_is_this_a_static_lib
-    except AttributeError:
-        cache = self.cache_is_this_a_static_lib = {}
-    try:
-        return cache[name]
-    except KeyError:
-        ret = cache[name] = 'cstaticlib' in self.bld.get_tgen_by_name(name).features
-        return ret
-TaskGen.task_gen.is_this_a_static_lib = is_this_a_static_lib
-
-def shared_ancestors(self):
-    try:
-        cache = self.cache_is_this_a_static_lib
-    except AttributeError:
-        cache = self.cache_is_this_a_static_lib = {}
-    try:
-        return cache[id(self)]
-    except KeyError:
-
-        ret = []
-        if 'cshlib' in self.features: # or 'cprogram' in self.features:
-            if getattr(self, 'uselib_local', None):
-                lst = self.to_list(self.uselib_local)
-                ret = [x for x in lst if not self.is_this_a_static_lib(x)]
-        cache[id(self)] = ret
-        return ret
-TaskGen.task_gen.shared_ancestors = shared_ancestors
-
-@feature('c', 'cc', 'cxx')
-@after('apply_link', 'init_cc', 'init_cxx', 'apply_core')
-def apply_lib_vars(self):
-    """after apply_link because of 'link_task'
-    after default_cc because of the attribute 'uselib'"""
-
-    # after 'apply_core' in case if 'cc' if there is no link
-
-    env = self.env
-    app = env.append_value
-    seen_libpaths = set([])
-
-    # OPTIMIZATION 1: skip uselib variables already added (700ms)
-    seen_uselib = set([])
-
-    # 1. the case of the libs defined in the project (visit ancestors first)
-    # the ancestors external libraries (uselib) will be prepended
-    self.uselib = self.to_list(self.uselib)
-    names = self.to_list(self.uselib_local)
-
-    seen = set([])
-    tmp = Utils.deque(names) # consume a copy of the list of names
-    while tmp:
-        lib_name = tmp.popleft()
-        # visit dependencies only once
-        if lib_name in seen:
-            continue
-
-        y = self.get_tgen_by_name(lib_name)
-        if not y:
-            raise Utils.WafError('object %r was not found in uselib_local (required by %r)' % (lib_name, self.name))
-        y.post()
-        seen.add(lib_name)
-
-        # OPTIMIZATION 2: pre-compute ancestors shared libraries (100ms)
-        tmp.extend(y.shared_ancestors())
-
-        # link task and flags
-        if getattr(y, 'link_task', None):
-
-            link_name = y.target[y.target.rfind('/') + 1:]
-            if 'cstaticlib' in y.features:
-                app('STATICLIB', link_name)
-            elif 'cshlib' in y.features or 'cprogram' in y.features:
-                # WARNING some linkers can link against programs
-                app('LIB', link_name)
-
-            # the order
-            self.link_task.set_run_after(y.link_task)
-
-            # for the recompilation
-            dep_nodes = getattr(self.link_task, 'dep_nodes', [])
-            self.link_task.dep_nodes = dep_nodes + y.link_task.outputs
-
-            # OPTIMIZATION 3: reduce the amount of function calls
-            # add the link path too
-            par = y.link_task.outputs[0].parent
-            if id(par) not in seen_libpaths:
-                seen_libpaths.add(id(par))
-                tmp_path = par.bldpath(self.env)
-                if not tmp_path in env['LIBPATH']:
-                    env.prepend_value('LIBPATH', tmp_path)
-
-
-        # add ancestors uselib too - but only propagate those that have no staticlib
-        for v in self.to_list(y.uselib):
-            if v not in seen_uselib:
-                seen_uselib.add(v)
-                if not env['STATICLIB_' + v]:
-                    if not v in self.uselib:
-                        self.uselib.insert(0, v)
-
-    # 2. the case of the libs defined outside
-    for x in self.uselib:
-        for v in self.p_flag_vars:
-            val = self.env[v + '_' + x]
-            if val:
-                self.env.append_value(v, val)
-
-
diff --git a/buildtools/wafsamba/samba_patterns.py b/buildtools/wafsamba/samba_patterns.py
index 2b93937..d0fe965 100644
--- a/buildtools/wafsamba/samba_patterns.py
+++ b/buildtools/wafsamba/samba_patterns.py
@@ -1,6 +1,7 @@
 # a waf tool to add extension based build patterns for Samba
 
-import Build
+import sys
+from waflib import Build
 from wafsamba import samba_version_file
 
 def write_version_header(task):
@@ -146,13 +147,19 @@ def write_build_options_section(fp, keys, section):
     fp.write("\n")
 
 def write_build_options(task):
-    tbl = task.env['defines']
+    tbl = task.env
     keys_option_with = []
     keys_option_utmp = []
     keys_option_have = []
     keys_header_sys = []
     keys_header_other = []
     keys_misc = []
+    if sys.hexversion>0x300000f:
+        trans_table = bytes.maketrans(b'.-()', b'____')
+    else:
+        import string
+        trans_table = string.maketrans('.-()', '____')
+
     for key in tbl:
         if key.startswith("HAVE_UT_UT_") or key.find("UTMP") >= 0:
             keys_option_utmp.append(key)
@@ -169,7 +176,7 @@ def write_build_options(task):
             l = key.split("(")
             keys_misc.append(l[0])
         else:
-            keys_misc.append(key)
+            keys_misc.append(key.translate(trans_table))
 
     tgt = task.outputs[0].bldpath(task.env)
     f = open(tgt, 'w')
diff --git a/buildtools/wafsamba/samba_perl.py b/buildtools/wafsamba/samba_perl.py
index 2e9a53a..e019acb 100644
--- a/buildtools/wafsamba/samba_perl.py
+++ b/buildtools/wafsamba/samba_perl.py
@@ -1,6 +1,6 @@
-import Utils
-from Configure import conf
-
+from waflib import Utils
+from waflib.Configure import conf
+from samba_utils import get_string
 done = {}
 
 @conf
@@ -9,13 +9,16 @@ def SAMBA_CHECK_PERL(conf, mandatory=True, version=(5,0,0)):
         return
     done["done"] = True
     conf.find_program('perl', var='PERL', mandatory=mandatory)
-    conf.check_tool('perl')
+    conf.load('perl')
     path_perl = conf.find_program('perl')
     conf.env.PERL_SPECIFIED = (conf.env.PERL != path_perl)
     conf.check_perl_version(version)
 
     def read_perl_config_var(cmd):
-        return Utils.to_list(Utils.cmd_output([conf.env.PERL, '-MConfig', '-e', cmd]))
+        output = Utils.cmd_output([conf.env.get_flat('PERL'), '-MConfig', '-e', cmd])
+        if not isinstance(output, str):
+            output = get_string(output)
+        return Utils.to_list(output)
 
     def check_perl_config_var(var):
         conf.start_msg("Checking for perl $Config{%s}:" % var)
diff --git a/buildtools/wafsamba/samba_pidl.py b/buildtools/wafsamba/samba_pidl.py
index 9651e4d..3fecfa9 100644
--- a/buildtools/wafsamba/samba_pidl.py
+++ b/buildtools/wafsamba/samba_pidl.py
@@ -1,8 +1,8 @@
 # waf build tool for building IDL files with pidl
 
 import os
-import Build
-from TaskGen import feature, before
+from waflib import Build, Utils
+from waflib.TaskGen import feature, before
 from samba_utils import SET_TARGET_TYPE, TO_LIST, LOCAL_CACHE
 
 def SAMBA_PIDL(bld, pname, source,
@@ -76,9 +76,9 @@ def SAMBA_PIDL(bld, pname, source,
         else:
             cc = 'CC="%s"' % bld.CONFIG_GET("CC")
 
-    t = bld(rule='cd .. && %s %s ${PERL} "${PIDL}" --quiet ${OPTIONS} --outputdir ${OUTPUTDIR} -- "${SRC[0].abspath(env)}"' % (cpp, cc),
+    t = bld(rule='cd ${PIDL_LAUNCH_DIR} && %s %s ${PERL} ${PIDL} --quiet ${OPTIONS} --outputdir ${OUTPUTDIR} -- "${IDLSRC}"' % (cpp, cc),
             ext_out    = '.c',
-            before     = 'cc',
+            before     = 'c',
             update_outputs = True,
             shell      = True,
             source     = source,
@@ -86,18 +86,22 @@ def SAMBA_PIDL(bld, pname, source,
             name       = name,
             samba_type = 'PIDL')
 
-    # prime the list of nodes we are dependent on with the cached pidl sources
-    t.allnodes = pidl_src_nodes
 
-    t.env.PIDL = os.path.join(bld.srcnode.abspath(), 'pidl/pidl')
+    t.env.PIDL_LAUNCH_DIR = bld.srcnode.path_from(bld.bldnode)
+    pnode = bld.srcnode.find_resource('pidl/pidl')
+    t.env.PIDL = pnode.path_from(bld.srcnode)
     t.env.OPTIONS = TO_LIST(options)
-    t.env.OUTPUTDIR = bld.bldnode.name + '/' + bld.path.find_dir(output_dir).bldpath(t.env)
+    snode = t.path.find_resource(source[0])
+    t.env.IDLSRC = snode.path_from(bld.srcnode)
+    t.env.OUTPUTDIR = bld.bldnode.path_from(bld.srcnode) + '/' + bld.path.find_dir(output_dir).path_from(bld.srcnode)
+
+    bld.add_manual_dependency(snode, pidl_src_nodes)
 
     if generate_tables and table_header_idx is not None:
         pidl_headers = LOCAL_CACHE(bld, 'PIDL_HEADERS')
         pidl_headers[name] = [bld.path.find_or_declare(out_files[table_header_idx])]
 
-    t.more_includes = '#' + bld.path.relpath_gen(bld.srcnode)
+    t.more_includes = '#' + bld.path.path_from(bld.srcnode)
 Build.BuildContext.SAMBA_PIDL = SAMBA_PIDL
 
 
@@ -117,13 +121,15 @@ Build.BuildContext.SAMBA_PIDL_LIST = SAMBA_PIDL_LIST
 @before('exec_rule')
 def collect(self):
     pidl_headers = LOCAL_CACHE(self.bld, 'PIDL_HEADERS')
+    # The first source is tables.pl itself
+    self.source = Utils.to_list(self.source)
     for (name, hd) in pidl_headers.items():
         y = self.bld.get_tgen_by_name(name)
         self.bld.ASSERT(y is not None, 'Failed to find PIDL header %s' % name)
         y.post()
         for node in hd:
             self.bld.ASSERT(node is not None, 'Got None as build node generating PIDL table for %s' % name)
-            self.source += " " + node.relpath_gen(self.path)
+            self.source.append(node)
 
 
 def SAMBA_PIDL_TABLES(bld, name, target):
@@ -131,9 +137,9 @@ def SAMBA_PIDL_TABLES(bld, name, target):
     bld.SET_BUILD_GROUP('main')
     t = bld(
             features = 'collect',
-            rule     = '${PERL} ${SRC} --output ${TGT} | sed "s|default/||" > ${TGT}',
+            rule     = '${PERL} ${SRC} > ${TGT}',
             ext_out  = '.c',
-            before   = 'cc',
+            before   = 'c',
             update_outputs = True,
             shell    = True,
             source   = '../../librpc/tables.pl',
diff --git a/buildtools/wafsamba/samba_python.py b/buildtools/wafsamba/samba_python.py
index cb99fe9..fac0e34 100644
--- a/buildtools/wafsamba/samba_python.py
+++ b/buildtools/wafsamba/samba_python.py
@@ -1,11 +1,11 @@
 # waf build tool for building IDL files with pidl
 
-import os
-import Build, Logs, Utils, Configure
-from Configure import conf
+import os, sys
+from waflib import Build, Logs, Utils, Configure, Errors
+from waflib.Configure import conf
 
 @conf
-def SAMBA_CHECK_PYTHON(conf, mandatory=True, version=(2,4,2)):
+def SAMBA_CHECK_PYTHON(conf, mandatory=True, version=(2,6,0)):
     # enable tool to build python extensions
     if conf.env.HAVE_PYTHON_H:
         conf.check_python_version(version)
@@ -14,23 +14,25 @@ def SAMBA_CHECK_PYTHON(conf, mandatory=True, version=(2,4,2)):
     interpreters = []
 
     if conf.env['EXTRA_PYTHON']:
-        conf.all_envs['extrapython'] = conf.env.copy()
+        conf.all_envs['extrapython'] = conf.env.derive()
         conf.setenv('extrapython')
         conf.env['PYTHON'] = conf.env['EXTRA_PYTHON']
         conf.env['IS_EXTRA_PYTHON'] = 'yes'
         conf.find_program('python', var='PYTHON', mandatory=True)
-        conf.check_tool('python')
+        conf.load('python')
         try:
-            conf.check_python_version((3, 3, 0))
+            conf.check_python_version(version)
         except Exception:
-            Logs.warn('extra-python needs to be Python 3.3 or later')
+            Logs.warn('extra-python needs to be Python %s.%s.%s or later' %
+                      (version[0], version[1], version[2]))
             raise
         interpreters.append(conf.env['PYTHON'])
         conf.setenv('default')
 
-    conf.find_program('python', var='PYTHON', mandatory=mandatory)
-    conf.check_tool('python')
-    path_python = conf.find_program('python')
+    conf.find_program('python3', var='PYTHON', mandatory=mandatory)
+    conf.load('python')
+    path_python = conf.find_program('python3')
+
     conf.env.PYTHON_SPECIFIED = (conf.env.PYTHON != path_python)
     conf.check_python_version(version)
 
@@ -42,14 +44,16 @@ def SAMBA_CHECK_PYTHON(conf, mandatory=True, version=(2,4,2)):
 def SAMBA_CHECK_PYTHON_HEADERS(conf, mandatory=True):
     if conf.env.disable_python:
         if mandatory:
-            raise Utils.WafError("Cannot check for python headers when "
+            raise Errors.WafError("Cannot check for python headers when "
                                  "--disable-python specified")
 
         conf.msg("python headers", "Check disabled due to --disable-python")
         # we don't want PYTHONDIR in config.h, as otherwise changing
         # --prefix causes a complete rebuild
-        del(conf.env.defines['PYTHONDIR'])
-        del(conf.env.defines['PYTHONARCHDIR'])
+        conf.env.DEFINES = [x for x in conf.env.DEFINES
+            if not x.startswith('PYTHONDIR=')
+            and not x.startswith('PYTHONARCHDIR=')]
+
         return
 
     if conf.env["python_headers_checked"] == []:
@@ -64,21 +68,22 @@ def SAMBA_CHECK_PYTHON_HEADERS(conf, mandatory=True):
         if conf.env['EXTRA_PYTHON']:
             extraversion = conf.all_envs['extrapython']['PYTHON_VERSION']
             if extraversion == conf.env['PYTHON_VERSION']:
-                raise Utils.WafError("extrapython %s is same as main python %s" % (
+                raise Errors.WafError("extrapython %s is same as main python %s" % (
                     extraversion, conf.env['PYTHON_VERSION']))
     else:
         conf.msg("python headers", "using cache")
 
     # we don't want PYTHONDIR in config.h, as otherwise changing
     # --prefix causes a complete rebuild
-    del(conf.env.defines['PYTHONDIR'])
-    del(conf.env.defines['PYTHONARCHDIR'])
+    conf.env.DEFINES = [x for x in conf.env.DEFINES
+        if not x.startswith('PYTHONDIR=')
+        and not x.startswith('PYTHONARCHDIR=')]
 
 def _check_python_headers(conf, mandatory):
     try:
-        Configure.ConfigurationError
-        conf.check_python_headers(mandatory=mandatory)
-    except Configure.ConfigurationError:
+        conf.errors.ConfigurationError
+        conf.check_python_headers()
+    except conf.errors.ConfigurationError:
         if mandatory:
              raise
 
@@ -95,6 +100,11 @@ def _check_python_headers(conf, mandatory):
             conf.env.append_unique('LIBPATH_PYEMBED', lib[2:]) # strip '-L'
             conf.env['LINKFLAGS_PYEMBED'].remove(lib)
 
+    # same as in waf 1.5, keep only '-fno-strict-aliasing'
+    # and ignore defines such as NDEBUG _FORTIFY_SOURCE=2
+    conf.env.DEFINES_PYEXT = []
+    conf.env.CFLAGS_PYEXT = ['-fno-strict-aliasing']
+
     return
 
 def PYTHON_BUILD_IS_ENABLED(self):
@@ -145,7 +155,7 @@ def SAMBA_PYTHON(bld, name,
     source = bld.EXPAND_VARIABLES(source, vars=vars)
 
     if realname is not None:
-        link_name = 'python_modules/%s' % realname
+        link_name = 'python/%s' % realname
     else:
         link_name = None
 
diff --git a/buildtools/wafsamba/samba_third_party.py b/buildtools/wafsamba/samba_third_party.py
index 1144f81..e0dd3e1 100644
--- a/buildtools/wafsamba/samba_third_party.py
+++ b/buildtools/wafsamba/samba_third_party.py
@@ -1,12 +1,12 @@
 # functions to support third party libraries
 
 import os
-import Utils, Build
-from Configure import conf
+from waflib import Utils, Build, Context
+from waflib.Configure import conf
 
 @conf
 def CHECK_FOR_THIRD_PARTY(conf):
-    return os.path.exists(os.path.join(Utils.g_module.srcdir, 'third_party'))
+    return os.path.exists(os.path.join(Context.g_module.top, 'third_party'))
 
 Build.BuildContext.CHECK_FOR_THIRD_PARTY = CHECK_FOR_THIRD_PARTY
 
@@ -36,18 +36,18 @@ Build.BuildContext.CHECK_POPT = CHECK_POPT
 
 @conf
 def CHECK_CMOCKA(conf):
-    return conf.CHECK_BUNDLED_SYSTEM_PKG('cmocka', minversion='1.1.1')
+    return conf.CHECK_BUNDLED_SYSTEM_PKG('cmocka', minversion='1.1.3')
 
 Build.BuildContext.CHECK_CMOCKA = CHECK_CMOCKA
 
 @conf
 def CHECK_SOCKET_WRAPPER(conf):
-    return conf.CHECK_BUNDLED_SYSTEM_PKG('socket_wrapper', minversion='1.1.9')
+    return conf.CHECK_BUNDLED_SYSTEM_PKG('socket_wrapper', minversion='1.2.1')
 Build.BuildContext.CHECK_SOCKET_WRAPPER = CHECK_SOCKET_WRAPPER
 
 @conf
 def CHECK_NSS_WRAPPER(conf):
-    return conf.CHECK_BUNDLED_SYSTEM_PKG('nss_wrapper', minversion='1.1.3')
+    return conf.CHECK_BUNDLED_SYSTEM_PKG('nss_wrapper', minversion='1.1.5')
 Build.BuildContext.CHECK_NSS_WRAPPER = CHECK_NSS_WRAPPER
 
 @conf
@@ -62,5 +62,5 @@ Build.BuildContext.CHECK_UID_WRAPPER = CHECK_UID_WRAPPER
 
 @conf
 def CHECK_PAM_WRAPPER(conf):
-    return conf.CHECK_BUNDLED_SYSTEM_PKG('pam_wrapper', minversion='1.0.4')
+    return conf.CHECK_BUNDLED_SYSTEM_PKG('pam_wrapper', minversion='1.0.7')
 Build.BuildContext.CHECK_PAM_WRAPPER = CHECK_PAM_WRAPPER
diff --git a/buildtools/wafsamba/samba_utils.py b/buildtools/wafsamba/samba_utils.py
index 0f95c12..ad97de1 100644
--- a/buildtools/wafsamba/samba_utils.py
+++ b/buildtools/wafsamba/samba_utils.py
@@ -1,30 +1,92 @@
 # a waf tool to add autoconf-like macros to the configure section
 # and for SAMBA_ macros for building libraries, binaries etc
 
-import os, sys, re, fnmatch, shlex
+import errno
+import os, sys, re, fnmatch, shlex, inspect
 from optparse import SUPPRESS_HELP
-import Build, Options, Utils, Task, Logs, Configure
-from TaskGen import feature, before, after
-from Configure import conf, ConfigurationContext
-from Logs import debug
+from waflib import Build, Options, Utils, Task, Logs, Configure, Errors, Context
+from waflib import Scripting
+from waflib.TaskGen import feature, before, after
+from waflib.Configure import ConfigurationContext
+from waflib.Logs import debug
+from waflib import ConfigSet
+from waflib.Build import CACHE_SUFFIX
 
 # TODO: make this a --option
 LIB_PATH="shared"
 
 
+PY3 = sys.version_info[0] == 3
+
+if PY3:
+
+    # helper function to get a string from a variable that maybe 'str' or
+    # 'bytes' if 'bytes' then it is decoded using 'utf8'. If 'str' is passed
+    # it is returned unchanged
+    # Using this function is PY2/PY3 code should ensure in most cases
+    # the PY2 code runs unchanged in PY2 whereas the code in PY3 possibly
+    # decodes the variable (see PY2 implementation of this function below)
+    def get_string(bytesorstring):
+        tmp = bytesorstring
+        if isinstance(bytesorstring, bytes):
+            tmp = bytesorstring.decode('utf8')
+        elif not isinstance(bytesorstring, str):
+            raise ValueError('Expected byte of string for %s:%s' % (type(bytesorstring), bytesorstring))
+        return tmp
+
+else:
+
+    # Helper function to return string.
+    # if 'str' or 'unicode' passed in they are returned unchanged
+    # otherwise an exception is generated
+    # Using this function is PY2/PY3 code should ensure in most cases
+    # the PY2 code runs unchanged in PY2 whereas the code in PY3 possibly
+    # decodes the variable (see PY3 implementation of this function above)
+    def get_string(bytesorstring):
+        tmp = bytesorstring
+        if not(isinstance(bytesorstring, str) or isinstance(bytesorstring, unicode)):
+            raise ValueError('Expected str or unicode for %s:%s' % (type(bytesorstring), bytesorstring))
+        return tmp
+
 # sigh, python octal constants are a mess
 MODE_644 = int('644', 8)
+MODE_744 = int('744', 8)
 MODE_755 = int('755', 8)
+MODE_777 = int('777', 8)
+
+def conf(f):
+    # override in order to propagate the argument "mandatory"
+    def fun(*k, **kw):
+        mandatory = True
+        if 'mandatory' in kw:
+            mandatory = kw['mandatory']
+            del kw['mandatory']
+
+        try:
+            return f(*k, **kw)
+        except Errors.ConfigurationError:
+            if mandatory:
+                raise
+
+    fun.__name__ = f.__name__
+    if 'mandatory' in inspect.getsource(f):
+        fun = f
+
+    setattr(Configure.ConfigurationContext, f.__name__, fun)
+    setattr(Build.BuildContext, f.__name__, fun)
+    return f
+Configure.conf = conf
+Configure.conftest = conf
 
 @conf
 def SET_TARGET_TYPE(ctx, target, value):
     '''set the target type of a target'''
     cache = LOCAL_CACHE(ctx, 'TARGET_TYPE')
     if target in cache and cache[target] != 'EMPTY':
-        Logs.error("ERROR: Target '%s' in directory %s re-defined as %s - was %s" % (target, ctx.curdir, value, cache[target]))
+        Logs.error("ERROR: Target '%s' in directory %s re-defined as %s - was %s" % (target, ctx.path.abspath(), value, cache[target]))
         sys.exit(1)
     LOCAL_CACHE_SET(ctx, 'TARGET_TYPE', target, value)
-    debug("task_gen: Target '%s' created of type '%s' in %s" % (target, value, ctx.curdir))
+    debug("task_gen: Target '%s' created of type '%s' in %s" % (target, value, ctx.path.abspath()))
     return True
 
 
@@ -101,7 +163,7 @@ def LOCAL_CACHE_SET(ctx, cachename, key, value):
 def ASSERT(ctx, expression, msg):
     '''a build assert call'''
     if not expression:
-        raise Utils.WafError("ERROR: %s\n" % msg)
+        raise Errors.WafError("ERROR: %s\n" % msg)
 Build.BuildContext.ASSERT = ASSERT
 
 
@@ -122,9 +184,9 @@ def dict_concat(d1, d2):
 
 def ADD_COMMAND(opt, name, function):
     '''add a new top level command to waf'''
-    Utils.g_module.__dict__[name] = function
+    Context.g_module.__dict__[name] = function
     opt.name = function
-Options.Handler.ADD_COMMAND = ADD_COMMAND
+Options.OptionsContext.ADD_COMMAND = ADD_COMMAND
 
 
 @feature('c', 'cc', 'cshlib', 'cprogram')
@@ -199,8 +261,10 @@ def subst_vars_error(string, env):
         if re.match('\$\{\w+\}', v):
             vname = v[2:-1]
             if not vname in env:
-                raise KeyError("Failed to find variable %s in %s" % (vname, string))
+                raise KeyError("Failed to find variable %s in %s in env %s <%s>" % (vname, string, env.__class__, str(env)))
             v = env[vname]
+            if isinstance(v, list):
+                v = ' '.join(v)
         out.append(v)
     return ''.join(out)
 
@@ -212,51 +276,6 @@ def SUBST_ENV_VAR(ctx, varname):
 Build.BuildContext.SUBST_ENV_VAR = SUBST_ENV_VAR
 
 
-def ENFORCE_GROUP_ORDERING(bld):
-    '''enforce group ordering for the project. This
-       makes the group ordering apply only when you specify
-       a target with --target'''
-    if Options.options.compile_targets:
-        @feature('*')
-        @before('exec_rule', 'apply_core', 'collect')
-        def force_previous_groups(self):
-            if getattr(self.bld, 'enforced_group_ordering', False):
-                return
-            self.bld.enforced_group_ordering = True
-
-            def group_name(g):
-                tm = self.bld.task_manager
-                return [x for x in tm.groups_names if id(tm.groups_names[x]) == id(g)][0]
-
-            my_id = id(self)
-            bld = self.bld
-            stop = None
-            for g in bld.task_manager.groups:
-                for t in g.tasks_gen:
-                    if id(t) == my_id:
-                        stop = id(g)
-                        debug('group: Forcing up to group %s for target %s',
-                              group_name(g), self.name or self.target)
-                        break
-                if stop is not None:
-                    break
-            if stop is None:
-                return
-
-            for i in xrange(len(bld.task_manager.groups)):
-                g = bld.task_manager.groups[i]
-                bld.task_manager.current_group = i
-                if id(g) == stop:
-                    break
-                debug('group: Forcing group %s', group_name(g))
-                for t in g.tasks_gen:
-                    if not getattr(t, 'forced_groups', False):
-                        debug('group: Posting %s', t.name or t.target)
-                        t.forced_groups = True
-                        t.post()
-Build.BuildContext.ENFORCE_GROUP_ORDERING = ENFORCE_GROUP_ORDERING
-
-
 def recursive_dirlist(dir, relbase, pattern=None):
     '''recursive directory list'''
     ret = []
@@ -271,6 +290,18 @@ def recursive_dirlist(dir, relbase, pattern=None):
     return ret
 
 
+def symlink(src, dst, force=True):
+    """Can create symlink by force"""
+    try:
+        os.symlink(src, dst)
+    except OSError as exc:
+        if exc.errno == errno.EEXIST and force:
+            os.remove(dst)
+            os.symlink(src, dst)
+        else:
+            raise
+
+
 def mkdir_p(dir):
     '''like mkdir -p'''
     if not dir:
@@ -312,8 +343,7 @@ def EXPAND_VARIABLES(ctx, varstr, vars=None):
     if not isinstance(varstr, str):
         return varstr
 
-    import Environment
-    env = Environment.Environment()
+    env = ConfigSet.ConfigSet()
     ret = varstr
     # substitute on user supplied dict if avaiilable
     if vars is not None:
@@ -345,16 +375,18 @@ def RUN_COMMAND(cmd,
         return os.WEXITSTATUS(status)
     if os.WIFSIGNALED(status):
         return - os.WTERMSIG(status)
-    Logs.error("Unknown exit reason %d for command: %s" (status, cmd))
+    Logs.error("Unknown exit reason %d for command: %s" % (status, cmd))
     return -1
 
 
 def RUN_PYTHON_TESTS(testfiles, pythonpath=None, extra_env=None):
     env = LOAD_ENVIRONMENT()
     if pythonpath is None:
-        pythonpath = os.path.join(Utils.g_module.blddir, 'python')
+        pythonpath = os.path.join(Context.g_module.out, 'python')
     result = 0
     for interp in env.python_interpreters:
+        if not isinstance(interp, str):
+            interp = ' '.join(interp)
         for testfile in testfiles:
             cmd = "PYTHONPATH=%s %s %s" % (pythonpath, interp, testfile)
             if extra_env:
@@ -374,16 +406,15 @@ try:
     # Even if hashlib.md5 exists, it may be unusable.
     # Try to use MD5 function. In FIPS mode this will cause an exception
     # and we'll get to the replacement code
-    foo = md5('abcd')
+    foo = md5(b'abcd')
 except:
     try:
         import md5
         # repeat the same check here, mere success of import is not enough.
         # Try to use MD5 function. In FIPS mode this will cause an exception
-        foo = md5.md5('abcd')
+        foo = md5.md5(b'abcd')
     except:
-        import Constants
-        Constants.SIG_NIL = hash('abcd')
+        Context.SIG_NIL = hash('abcd')
         class replace_md5(object):
             def __init__(self):
                 self.val = None
@@ -409,20 +440,20 @@ except:
 def LOAD_ENVIRONMENT():
     '''load the configuration environment, allowing access to env vars
        from new commands'''
-    import Environment
-    env = Environment.Environment()
+    env = ConfigSet.ConfigSet()
     try:
-        env.load('.lock-wscript')
-        env.load(env.blddir + '/c4che/default.cache.py')
-    except:
+        p = os.path.join(Context.g_module.out, 'c4che/default'+CACHE_SUFFIX)
+        env.load(p)
+    except (OSError, IOError):
         pass
     return env
 
 
 def IS_NEWER(bld, file1, file2):
     '''return True if file1 is newer than file2'''
-    t1 = os.stat(os.path.join(bld.curdir, file1)).st_mtime
-    t2 = os.stat(os.path.join(bld.curdir, file2)).st_mtime
+    curdir = bld.path.abspath()
+    t1 = os.stat(os.path.join(curdir, file1)).st_mtime
+    t2 = os.stat(os.path.join(curdir, file2)).st_mtime
     return t1 > t2
 Build.BuildContext.IS_NEWER = IS_NEWER
 
@@ -432,47 +463,46 @@ def RECURSE(ctx, directory):
     '''recurse into a directory, relative to the curdir or top level'''
     try:
         visited_dirs = ctx.visited_dirs
-    except:
+    except AttributeError:
         visited_dirs = ctx.visited_dirs = set()
-    d = os.path.join(ctx.curdir, directory)
+    d = os.path.join(ctx.path.abspath(), directory)
     if os.path.exists(d):
         abspath = os.path.abspath(d)
     else:
-        abspath = os.path.abspath(os.path.join(Utils.g_module.srcdir, directory))
+        abspath = os.path.abspath(os.path.join(Context.g_module.top, directory))
     ctxclass = ctx.__class__.__name__
     key = ctxclass + ':' + abspath
     if key in visited_dirs:
         # already done it
         return
     visited_dirs.add(key)
-    relpath = os_path_relpath(abspath, ctx.curdir)
-    if ctxclass == 'Handler':
-        return ctx.sub_options(relpath)
-    if ctxclass == 'ConfigurationContext':
-        return ctx.sub_config(relpath)
-    if ctxclass == 'BuildContext':
-        return ctx.add_subdirs(relpath)
-    Logs.error('Unknown RECURSE context class', ctxclass)
+    relpath = os_path_relpath(abspath, ctx.path.abspath())
+    if ctxclass in ['tmp', 'OptionsContext', 'ConfigurationContext', 'BuildContext']:
+        return ctx.recurse(relpath)
+    if 'waflib.extras.compat15' in sys.modules:
+        return ctx.recurse(relpath)
+    Logs.error('Unknown RECURSE context class: {}'.format(ctxclass))
     raise
-Options.Handler.RECURSE = RECURSE
+Options.OptionsContext.RECURSE = RECURSE
 Build.BuildContext.RECURSE = RECURSE
 
 
-def CHECK_MAKEFLAGS(bld):
+def CHECK_MAKEFLAGS(options):
     '''check for MAKEFLAGS environment variable in case we are being
     called from a Makefile try to honor a few make command line flags'''
     if not 'WAF_MAKE' in os.environ:
         return
     makeflags = os.environ.get('MAKEFLAGS')
     if makeflags is None:
-        return
+        makeflags = ""
     jobs_set = False
+    jobs = None
     # we need to use shlex.split to cope with the escaping of spaces
     # in makeflags
     for opt in shlex.split(makeflags):
         # options can come either as -x or as x
         if opt[0:2] == 'V=':
-            Options.options.verbose = Logs.verbose = int(opt[2:])
+            options.verbose = Logs.verbose = int(opt[2:])
             if Logs.verbose > 0:
                 Logs.zones = ['runner']
             if Logs.verbose > 2:
@@ -486,22 +516,53 @@ def CHECK_MAKEFLAGS(bld):
             # this is also how "make test TESTS=testpattern" works, and
             # "make VERBOSE=1" as well as things like "make SYMBOLCHECK=1"
             loc = opt.find('=')
-            setattr(Options.options, opt[0:loc], opt[loc+1:])
+            setattr(options, opt[0:loc], opt[loc+1:])
         elif opt[0] != '-':
             for v in opt:
-                if v == 'j':
+                if re.search(r'j[0-9]*$', v):
                     jobs_set = True
+                    jobs = opt.strip('j')
                 elif v == 'k':
-                    Options.options.keep = True
-        elif opt == '-j':
+                    options.keep = True
+        elif re.search(r'-j[0-9]*$', opt):
             jobs_set = True
+            jobs = opt.strip('-j')
         elif opt == '-k':
-            Options.options.keep = True
+            options.keep = True
     if not jobs_set:
         # default to one job
-        Options.options.jobs = 1
-
-Build.BuildContext.CHECK_MAKEFLAGS = CHECK_MAKEFLAGS
+        options.jobs = 1
+    elif jobs_set and jobs:
+        options.jobs = int(jobs)
+
+waflib_options_parse_cmd_args = Options.OptionsContext.parse_cmd_args
+def wafsamba_options_parse_cmd_args(self, _args=None, cwd=None, allow_unknown=False):
+    (options, commands, envvars) = \
+        waflib_options_parse_cmd_args(self,
+                                      _args=_args,
+                                      cwd=cwd,
+                                      allow_unknown=allow_unknown)
+    CHECK_MAKEFLAGS(options)
+    if options.jobs == 1:
+        #
+        # waflib.Runner.Parallel processes jobs inline if the possible number
+        # of jobs is just 1. But (at least in waf <= 2.0.12) it still calls
+        # create a waflib.Runner.Spawner() which creates a single
+        # waflib.Runner.Consumer() thread that tries to process jobs from the
+        # queue.
+        #
+        # This has strange effects, which are not noticed typically,
+        # but at least on AIX python has broken threading and fails
+        # in random ways.
+        #
+        # So we just add a dummy Spawner class.
+        class NoOpSpawner(object):
+            def __init__(self, master):
+                return
+        from waflib import Runner
+        Runner.Spawner = NoOpSpawner
+    return options, commands, envvars
+Options.OptionsContext.parse_cmd_args = wafsamba_options_parse_cmd_args
 
 option_groups = {}
 
@@ -513,7 +574,7 @@ def option_group(opt, name):
     gr = opt.add_option_group(name)
     option_groups[name] = gr
     return gr
-Options.Handler.option_group = option_group
+Options.OptionsContext.option_group = option_group
 
 
 def save_file(filename, contents, create_dir=False):
@@ -542,9 +603,9 @@ def load_file(filename):
 
 def reconfigure(ctx):
     '''rerun configure if necessary'''
-    import Configure, samba_wildcard, Scripting
-    if not os.path.exists(".lock-wscript"):
-        raise Utils.WafError('configure has not been run')
+    if not os.path.exists(os.environ.get('WAFLOCK', '.lock-wscript')):
+        raise Errors.WafError('configure has not been run')
+    import samba_wildcard
     bld = samba_wildcard.fake_build_environment()
     Configure.autoconfig = True
     Scripting.check_configured(bld)
@@ -561,7 +622,7 @@ def map_shlib_extension(ctx, name, python=False):
     if python:
         return ctx.env.pyext_PATTERN % root1
     else:
-        (root2, ext2) = os.path.splitext(ctx.env.shlib_PATTERN)
+        (root2, ext2) = os.path.splitext(ctx.env.cshlib_PATTERN)
     return root1+ext2
 Build.BuildContext.map_shlib_extension = map_shlib_extension
 
@@ -583,7 +644,7 @@ def make_libname(ctx, name, nolibprefix=False, version=None, python=False):
     if python:
         libname = apply_pattern(name, ctx.env.pyext_PATTERN)
     else:
-        libname = apply_pattern(name, ctx.env.shlib_PATTERN)
+        libname = apply_pattern(name, ctx.env.cshlib_PATTERN)
     if nolibprefix and libname[0:3] == 'lib':
         libname = libname[3:]
     if version:
@@ -617,7 +678,7 @@ def get_tgt_list(bld):
         tgt_list.append(t)
     return tgt_list
 
-from Constants import WSCRIPT_FILE
+from waflib.Context import WSCRIPT_FILE
 def PROCESS_SEPARATE_RULE(self, rule):
     ''' cause waf to process additional script based on `rule'.
         You should have file named wscript_<stage>_rule in the current directory
@@ -628,15 +689,21 @@ def PROCESS_SEPARATE_RULE(self, rule):
         stage = 'configure'
     elif isinstance(self, Build.BuildContext):
         stage = 'build'
-    file_path = os.path.join(self.curdir, WSCRIPT_FILE+'_'+stage+'_'+rule)
-    txt = load_file(file_path)
-    if txt:
-        dc = {'ctx': self}
-        if getattr(self.__class__, 'pre_recurse', None):
-            dc = self.pre_recurse(txt, file_path, self.curdir)
-        exec(compile(txt, file_path, 'exec'), dc)
-        if getattr(self.__class__, 'post_recurse', None):
-            dc = self.post_recurse(txt, file_path, self.curdir)
+    file_path = os.path.join(self.path.abspath(), WSCRIPT_FILE+'_'+stage+'_'+rule)
+    node = self.root.find_node(file_path)
+    if node:
+        try:
+            cache = self.recurse_cache
+        except AttributeError:
+            cache = self.recurse_cache = {}
+        if node not in cache:
+            cache[node] = True
+            self.pre_recurse(node)
+            try:
+                function_code = node.read('rU', None)
+                exec(compile(function_code, node.abspath(), 'exec'), self.exec_dict)
+            finally:
+                self.post_recurse(node)
 
 Build.BuildContext.PROCESS_SEPARATE_RULE = PROCESS_SEPARATE_RULE
 ConfigurationContext.PROCESS_SEPARATE_RULE = PROCESS_SEPARATE_RULE
@@ -693,4 +760,4 @@ def samba_add_onoff_option(opt, option, help=(), dest=None, default=True,
                    default=default)
     opt.add_option(without_val, help=SUPPRESS_HELP, action="store_false",
                    dest=dest)
-Options.Handler.samba_add_onoff_option = samba_add_onoff_option
+Options.OptionsContext.samba_add_onoff_option = samba_add_onoff_option
diff --git a/buildtools/wafsamba/samba_version.py b/buildtools/wafsamba/samba_version.py
index be26439..f0e7b4d 100644
--- a/buildtools/wafsamba/samba_version.py
+++ b/buildtools/wafsamba/samba_version.py
@@ -1,5 +1,5 @@
-import os
-import Utils
+import os, sys
+from waflib import Utils, Context
 import samba_utils
 from samba_git import find_git
 
@@ -14,7 +14,7 @@ def git_version_summary(path, env=None):
     environ = dict(os.environ)
     environ["GIT_DIR"] = '%s/.git' % path
     environ["GIT_WORK_TREE"] = path
-    git = Utils.cmd_output(env.GIT + ' show --pretty=format:"%h%n%ct%n%H%n%cd" --stat HEAD', silent=True, env=environ)
+    git = samba_utils.get_string(Utils.cmd_output(env.GIT + ' show --pretty=format:"%h%n%ct%n%H%n%cd" --stat HEAD', silent=True, env=environ))
 
     lines = git.splitlines()
     if not lines or len(lines) < 4:
@@ -198,7 +198,10 @@ also accepted as dictionary entries here
         for name in sorted(self.vcs_fields.keys()):
             string+="#define SAMBA_VERSION_%s " % name
             value = self.vcs_fields[name]
-            if isinstance(value, basestring):
+            string_types = str
+            if sys.version_info[0] < 3:
+                string_types = basestring
+            if isinstance(value, string_types):
                 string += "\"%s\"" % value
             elif type(value) is int:
                 string += "%d" % value
@@ -260,5 +263,5 @@ def load_version(env=None, is_install=True):
         env = samba_utils.LOAD_ENVIRONMENT()
 
     version = samba_version_file("./VERSION", ".", env, is_install=is_install)
-    Utils.g_module.VERSION = version.STRING
+    Context.g_module.VERSION = version.STRING
     return version
diff --git a/buildtools/wafsamba/samba_waf18.py b/buildtools/wafsamba/samba_waf18.py
 new file mode 100644
index 0000000..cc310fb
--- /dev/null
+++ b/buildtools/wafsamba/samba_waf18.py
@@ -0,0 +1,429 @@
+# compatibility layer for building with more recent waf versions
+
+import os, shlex, sys
+from waflib import Build, Configure, Node, Utils, Options, Logs, TaskGen
+from waflib import ConfigSet
+from waflib.TaskGen import feature, after
+from waflib.Configure import conf, ConfigurationContext
+
+from waflib.Tools.flex import decide_ext
+
+# This version of flexfun runs in tsk.get_cwd() as opposed to the
+# bld.variant_dir: since input paths adjusted against tsk.get_cwd(), we have to
+# use tsk.get_cwd() for the work directory as well.
+def flexfun(tsk):
+    env = tsk.env
+    bld = tsk.generator.bld
+    def to_list(xx):
+        if isinstance(xx, str):
+            return [xx]
+        return xx
+    tsk.last_cmd = lst = []
+    lst.extend(to_list(env.FLEX))
+    lst.extend(to_list(env.FLEXFLAGS))
+    inputs = [a.path_from(tsk.get_cwd()) for a in tsk.inputs]
+    if env.FLEX_MSYS:
+        inputs = [x.replace(os.sep, '/') for x in inputs]
+    lst.extend(inputs)
+    lst = [x for x in lst if x]
+    txt = bld.cmd_and_log(lst, cwd=tsk.get_cwd(), env=env.env or None, quiet=0)
+    tsk.outputs[0].write(txt.replace('\r\n', '\n').replace('\r', '\n')) # issue #1207
+
+TaskGen.declare_chain(
+    name = 'flex',
+    rule = flexfun, # issue #854
+    ext_in = '.l',
+    decider = decide_ext,
+)
+
+
+for y in (Build.BuildContext, Build.CleanContext, Build.InstallContext, Build.UninstallContext, Build.ListContext):
+    class tmp(y):
+        variant = 'default'
+
+def abspath(self, env=None):
+    if env and hasattr(self, 'children'):
+        return self.get_bld().abspath()
+    return self.old_abspath()
+Node.Node.old_abspath = Node.Node.abspath
+Node.Node.abspath = abspath
+
+def bldpath(self, env=None):
+    return self.abspath()
+    #return self.path_from(self.ctx.bldnode.parent)
+Node.Node.bldpath = bldpath
+
+def srcpath(self, env=None):
+    return self.abspath()
+    #return self.path_from(self.ctx.bldnode.parent)
+Node.Node.srcpath = srcpath
+
+def store_fast(self, filename):
+    file = open(filename, 'wb')
+    data = self.get_merged_dict()
+    try:
+        Build.cPickle.dump(data, file, -1)
+    finally:
+        file.close()
+ConfigSet.ConfigSet.store_fast = store_fast
+
+def load_fast(self, filename):
+    file = open(filename, 'rb')
+    try:
+        data = Build.cPickle.load(file)
+    finally:
+        file.close()
+    self.table.update(data)
+ConfigSet.ConfigSet.load_fast = load_fast
+
+@feature('c', 'cxx', 'd', 'asm', 'fc', 'includes')
+@after('propagate_uselib_vars', 'process_source')
+def apply_incpaths(self):
+    lst = self.to_incnodes(self.to_list(getattr(self, 'includes', [])) + self.env['INCLUDES'])
+    self.includes_nodes = lst
+    cwdx = getattr(self.bld, 'cwdx', self.bld.bldnode)
+    self.env['INCPATHS'] = [x.path_from(cwdx) for x in lst]
+
+@conf
+def define(self, key, val, quote=True, comment=None):
+   assert key and isinstance(key, str)
+
+   if val is None:
+       val = ()
+   elif isinstance(val, bool):
+       val = int(val)
+
+   # waf 1.5
+   self.env[key] = val
+
+   if isinstance(val, int) or isinstance(val, float):
+           s = '%s=%s'
+   else:
+           s = quote and '%s="%s"' or '%s=%s'
+   app = s % (key, str(val))
+
+   ban = key + '='
+   lst = self.env.DEFINES
+   for x in lst:
+           if x.startswith(ban):
+                   lst[lst.index(x)] = app
+                   break
+   else:
+           self.env.append_value('DEFINES', app)
+
+   self.env.append_unique('define_key', key)
+
+# compat15 removes this but we want to keep it
+@conf
+def undefine(self, key, from_env=True, comment=None):
+    assert key and isinstance(key, str)
+
+    ban = key + '='
+    self.env.DEFINES = [x for x in self.env.DEFINES if not x.startswith(ban)]
+    self.env.append_unique('define_key', key)
+    # waf 1.5
+    if from_env:
+        self.env[key] = ()
+
+class ConfigurationContext(Configure.ConfigurationContext):
+    def init_dirs(self):
+        self.setenv('default')
+        self.env.merge_config_header = True
+        return super(ConfigurationContext, self).init_dirs()
+
+def find_program_samba(self, *k, **kw):
+    kw['mandatory'] = False
+    ret = self.find_program_old(*k, **kw)
+    return ret
+Configure.ConfigurationContext.find_program_old = Configure.ConfigurationContext.find_program
+Configure.ConfigurationContext.find_program = find_program_samba
+
+Build.BuildContext.ENFORCE_GROUP_ORDERING = Utils.nada
+Build.BuildContext.AUTOCLEANUP_STALE_FILES = Utils.nada
+
+@conf
+def check(self, *k, **kw):
+    '''Override the waf defaults to inject --with-directory options'''
+
+    # match the configuration test with speficic options, for example:
+    # --with-libiconv -> Options.options.iconv_open -> "Checking for library iconv"
+    self.validate_c(kw)
+
+    additional_dirs = []
+    if 'msg' in kw:
+        msg = kw['msg']
+        for x in Options.OptionsContext.parser.parser.option_list:
+             if getattr(x, 'match', None) and msg in x.match:
+                 d = getattr(Options.options, x.dest, '')
+                 if d:
+                     additional_dirs.append(d)
+
+    # we add the additional dirs twice: once for the test data, and again if the compilation test suceeds below
+    def add_options_dir(dirs, env):
+        for x in dirs:
+             if not x in env.CPPPATH:
+                 env.CPPPATH = [os.path.join(x, 'include')] + env.CPPPATH
+             if not x in env.LIBPATH:
+                 env.LIBPATH = [os.path.join(x, 'lib')] + env.LIBPATH
+
+    add_options_dir(additional_dirs, kw['env'])
+
+    self.start_msg(kw['msg'], **kw)
+    ret = None
+    try:
+        ret = self.run_build(*k, **kw)
+    except self.errors.ConfigurationError:
+        self.end_msg(kw['errmsg'], 'YELLOW', **kw)
+        if Logs.verbose > 1:
+            raise
+        else:
+            self.fatal('The configuration failed')
+    else:
+        kw['success'] = ret
+        # success! time for brandy
+        add_options_dir(additional_dirs, self.env)
+
+    ret = self.post_check(*k, **kw)
+    if not ret:
+        self.end_msg(kw['errmsg'], 'YELLOW', **kw)
+        self.fatal('The configuration failed %r' % ret)
+    else:
+        self.end_msg(self.ret_msg(kw['okmsg'], kw), **kw)
+    return ret
+
+@conf
+def CHECK_LIBRARY_SUPPORT(conf, rpath=False, version_script=False, msg=None):
+    '''see if the platform supports building libraries'''
+
+    if msg is None:
+        if rpath:
+            msg = "rpath library support"
+        else:
+            msg = "building library support"
+
+    def build(bld):
+        lib_node = bld.srcnode.make_node('libdir/liblc1.c')
+        lib_node.parent.mkdir()
+        lib_node.write('int lib_func(void) { return 42; }\n', 'w')
+        main_node = bld.srcnode.make_node('main.c')
+        main_node.write('int main(void) {return !(lib_func() == 42);}', 'w')
+        linkflags = []
+        if version_script:
+            script = bld.srcnode.make_node('ldscript')
+            script.write('TEST_1.0A2 { global: *; };\n', 'w')
+            linkflags.append('-Wl,--version-script=%s' % script.abspath())
+        bld(features='c cshlib', source=lib_node, target='lib1', linkflags=linkflags, name='lib1')
+        o = bld(features='c cprogram', source=main_node, target='prog1', uselib_local='lib1')
+        if rpath:
+            o.rpath = [lib_node.parent.abspath()]
+        def run_app(self):
+             args = conf.SAMBA_CROSS_ARGS(msg=msg)
+             env = dict(os.environ)
+             env['LD_LIBRARY_PATH'] = self.inputs[0].parent.abspath() + os.pathsep + env.get('LD_LIBRARY_PATH', '')
+             self.generator.bld.cmd_and_log([self.inputs[0].abspath()] + args, env=env)
+        o.post()
+        bld(rule=run_app, source=o.link_task.outputs[0])
+
+    # ok, so it builds
+    try:
+        conf.check(build_fun=build, msg='Checking for %s' % msg)
+    except conf.errors.ConfigurationError:
+        return False
+    return True
+
+@conf
+def CHECK_NEED_LC(conf, msg):
+    '''check if we need -lc'''
+    def build(bld):
+        lib_node = bld.srcnode.make_node('libdir/liblc1.c')
+        lib_node.parent.mkdir()
+        lib_node.write('#include <stdio.h>\nint lib_func(void) { FILE *f = fopen("foo", "r");}\n', 'w')
+        bld(features='c cshlib', source=[lib_node], linkflags=conf.env.EXTRA_LDFLAGS, target='liblc')
+    try:
+        conf.check(build_fun=build, msg=msg, okmsg='-lc is unnecessary', errmsg='-lc is necessary')
+    except conf.errors.ConfigurationError:
+        return False
+    return True
+
+# already implemented on "waf -v"
+def order(bld, tgt_list):
+    return True
+Build.BuildContext.check_group_ordering = order
+
+@conf
+def CHECK_CFG(self, *k, **kw):
+    if 'args' in kw:
+        kw['args'] = shlex.split(kw['args'])
+    if not 'mandatory' in kw:
+        kw['mandatory'] = False
+    kw['global_define'] = True
+    return self.check_cfg(*k, **kw)
+
+def cmd_output(cmd, **kw):
+
+    silent = False
+    if 'silent' in kw:
+        silent = kw['silent']
+        del(kw['silent'])
+
+    if 'e' in kw:
+        tmp = kw['e']
+        del(kw['e'])
+        kw['env'] = tmp
+
+    kw['shell'] = isinstance(cmd, str)
+    kw['stdout'] = Utils.subprocess.PIPE
+    if silent:
+        kw['stderr'] = Utils.subprocess.PIPE
+
+    try:
+        p = Utils.subprocess.Popen(cmd, **kw)
+        output = p.communicate()[0]
+    except OSError as e:
+        raise ValueError(str(e))
+
+    if p.returncode:
+        if not silent:
+            msg = "command execution failed: %s -> %r" % (cmd, str(output))
+            raise ValueError(msg)
+        output = ''
+    return output
+Utils.cmd_output = cmd_output
+
+
[e-mail]('c', 'cxx', 'd')
[e-mail]('apply_incpaths', 'propagate_uselib_vars')
[e-mail]('apply_link', 'process_source')
+def apply_uselib_local(self):
+    """
+    process the uselib_local attribute
+    execute after apply_link because of the execution order set on 'link_task'
+    """
+    env = self.env
+    from waflib.Tools.ccroot import stlink_task
+
+    # 1. the case of the libs defined in the project (visit ancestors first)
+    # the ancestors external libraries (uselib) will be prepended
+    self.uselib = self.to_list(getattr(self, 'uselib', []))
+    self.includes = self.to_list(getattr(self, 'includes', []))
+    names = self.to_list(getattr(self, 'uselib_local', []))
+    get = self.bld.get_tgen_by_name
+    seen = set()
+    seen_uselib = set()
+    tmp = Utils.deque(names) # consume a copy of the list of names
+    if tmp:
+        if Logs.verbose:
+            Logs.warn('compat: "uselib_local" is deprecated, replace by "use"')
+    while tmp:
+        lib_name = tmp.popleft()
+        # visit dependencies only once
+        if lib_name in seen:
+            continue
+
+        y = get(lib_name)
+        y.post()
+        seen.add(lib_name)
+
+        # object has ancestors to process (shared libraries): add them to the end of the list
+        if getattr(y, 'uselib_local', None):
+            for x in self.to_list(getattr(y, 'uselib_local', [])):
+                obj = get(x)
+                obj.post()
+                if getattr(obj, 'link_task', None):
+                    if not isinstance(obj.link_task, stlink_task):
+                        tmp.append(x)
+
+        # link task and flags
+        if getattr(y, 'link_task', None):
+
+            link_name = y.target[y.target.rfind(os.sep) + 1:]
+            if isinstance(y.link_task, stlink_task):
+                env.append_value('STLIB', [link_name])
+            else:
+                # some linkers can link against programs
+                env.append_value('LIB', [link_name])
+
+            # the order
+            self.link_task.set_run_after(y.link_task)
+
+            # for the recompilation
+            self.link_task.dep_nodes += y.link_task.outputs
+
+            # add the link path too
+            tmp_path = y.link_task.outputs[0].parent.bldpath()
+            if not tmp_path in env['LIBPATH']:
+                env.prepend_value('LIBPATH', [tmp_path])
+
+        # add ancestors uselib too - but only propagate those that have no staticlib defined
+        for v in self.to_list(getattr(y, 'uselib', [])):
+            if v not in seen_uselib:
+                seen_uselib.add(v)
+                if not env['STLIB_' + v]:
+                    if not v in self.uselib:
+                        self.uselib.insert(0, v)
+
+        # if the library task generator provides 'export_includes', add to the include path
+        # the export_includes must be a list of paths relative to the other library
+        if getattr(y, 'export_includes', None):
+            self.includes.extend(y.to_incnodes(y.export_includes))
+
[e-mail]('cprogram', 'cxxprogram', 'cstlib', 'cxxstlib', 'cshlib', 'cxxshlib', 'dprogram', 'dstlib', 'dshlib')
[e-mail]('apply_link')
+def apply_objdeps(self):
+    "add the .o files produced by some other object files in the same manner as uselib_local"
+    names = getattr(self, 'add_objects', [])
+    if not names:
+        return
+    names = self.to_list(names)
+
+    get = self.bld.get_tgen_by_name
+    seen = []
+    while names:
+        x = names[0]
+
+        # visit dependencies only once
+        if x in seen:
+            names = names[1:]
+            continue
+
+        # object does not exist ?
+        y = get(x)
+
+        # object has ancestors to process first ? update the list of names
+        if getattr(y, 'add_objects', None):
+            added = 0
+            lst = y.to_list(y.add_objects)
+            lst.reverse()
+            for u in lst:
+                if u in seen:
+                    continue
+                added = 1
+                names = [u]+names
+            if added:
+                continue # list of names modified, loop
+
+        # safe to process the current object
+        y.post()
+        seen.append(x)
+
+        for t in getattr(y, 'compiled_tasks', []):
+            self.link_task.inputs.extend(t.outputs)
+
[e-mail]('apply_link')
+def process_obj_files(self):
+    if not hasattr(self, 'obj_files'):
+        return
+    for x in self.obj_files:
+        node = self.path.find_resource(x)
+        self.link_task.inputs.append(node)
+
[e-mail]_method
+def add_obj_file(self, file):
+    """Small example on how to link object files as if they were source
+    obj = bld.create_obj('cc')
+    obj.add_obj_file('foo.o')"""
+    if not hasattr(self, 'obj_files'):
+        self.obj_files = []
+    if not 'process_obj_files' in self.meths:
+        self.meths.append('process_obj_files')
+    self.obj_files.append(file)
diff --git a/buildtools/wafsamba/samba_wildcard.py b/buildtools/wafsamba/samba_wildcard.py
index ed3e0c2..6173ce8 100644
--- a/buildtools/wafsamba/samba_wildcard.py
+++ b/buildtools/wafsamba/samba_wildcard.py
@@ -1,15 +1,15 @@
 # based on playground/evil in the waf svn tree
 
 import os, datetime, fnmatch
-import Scripting, Utils, Options, Logs, Environment
-from Constants import SRCDIR, BLDDIR
+from waflib import Scripting, Utils, Options, Logs, Errors
+from waflib import ConfigSet, Context
 from samba_utils import LOCAL_CACHE, os_path_relpath
 
 def run_task(t, k):
     '''run a single build task'''
     ret = t.run()
     if ret:
-        raise Utils.WafError("Failed to build %s: %u" % (k, ret))
+        raise Errors.WafError("Failed to build %s: %u" % (k, ret))
 
 
 def run_named_build_task(cmd):
@@ -45,7 +45,7 @@ def run_named_build_task(cmd):
 
 
     if not found:
-        raise Utils.WafError("Unable to find build target matching %s" % cmd)
+        raise Errors.WafError("Unable to find build target matching %s" % cmd)
 
 
 def rewrite_compile_targets():
@@ -125,7 +125,7 @@ def wildcard_main(missing_cmd_fn):
 def fake_build_environment(info=True, flush=False):
     """create all the tasks for the project, but do not run the build
     return the build context in use"""
-    bld = getattr(Utils.g_module, 'build_context', Utils.Context)()
+    bld = getattr(Context.g_module, 'build_context', Utils.Context)()
     bld = Scripting.check_configured(bld)
 
     Options.commands['install'] = False
@@ -134,16 +134,15 @@ def fake_build_environment(info=True, flush=False):
     bld.is_install = 0 # False
 
     try:
-        proj = Environment.Environment(Options.lockfile)
+        proj = ConfigSet.ConfigSet(Options.lockfile)
     except IOError:
-        raise Utils.WafError("Project not configured (run 'waf configure' first)")
+        raise Errors.WafError("Project not configured (run 'waf configure' first)")
 
-    bld.load_dirs(proj[SRCDIR], proj[BLDDIR])
     bld.load_envs()
 
     if info:
         Logs.info("Waf: Entering directory `%s'" % bld.bldnode.abspath())
-    bld.add_subdirs([os.path.split(Utils.g_module.root_path)[0]])
+    bld.add_subdirs([os.path.split(Context.g_module.root_path)[0]])
 
     bld.pre_build()
     if flush:
diff --git a/buildtools/wafsamba/stale_files.py b/buildtools/wafsamba/stale_files.py
index 2dd08e1..175f573 100644
--- a/buildtools/wafsamba/stale_files.py
+++ b/buildtools/wafsamba/stale_files.py
@@ -14,7 +14,9 @@ nodes/tasks, in which case the method will have to be modified
 to exclude some folders for example.
 """
 
-import Logs, Build, os, samba_utils, Options, Utils
+from waflib import Logs, Build, Options, Utils, Errors
+import os
+from wafsamba import samba_utils
 from Runner import Parallel
 
 old_refill_task_list = Parallel.refill_task_list
@@ -46,7 +48,7 @@ def replace_refill_task_list(self):
 
     # paranoia
     if bin_base[-4:] != '/bin':
-        raise Utils.WafError("Invalid bin base: %s" % bin_base)
+        raise Errors.WafError("Invalid bin base: %s" % bin_base)
 
     # obtain the expected list of files
     expected = []
diff --git a/buildtools/wafsamba/symbols.py b/buildtools/wafsamba/symbols.py
index 7ff4bac..3eca3d4 100644
--- a/buildtools/wafsamba/symbols.py
+++ b/buildtools/wafsamba/symbols.py
@@ -2,8 +2,8 @@
 # using nm, producing a set of exposed defined/undefined symbols
 
 import os, re, subprocess
-import Utils, Build, Options, Logs
-from Logs import debug
+from waflib import Utils, Build, Options, Logs, Errors
+from waflib.Logs import debug
 from samba_utils import TO_LIST, LOCAL_CACHE, get_tgt_list, os_path_relpath
 
 # these are the data structures used in symbols.py:
@@ -59,12 +59,12 @@ def symbols_extract(bld, objfiles, dynamic=False):
 
     for line in nmpipe:
         line = line.strip()
-        if line.endswith(':'):
+        if line.endswith(b':'):
             filename = line[:-1]
             ret[filename] = { "PUBLIC": set(), "UNDEFINED" : set() }
             continue
-        cols = line.split(" ")
-        if cols == ['']:
+        cols = line.split(b" ")
+        if cols == [b'']:
             continue
         # see if the line starts with an address
         if len(cols) == 3:
@@ -73,10 +73,10 @@ def symbols_extract(bld, objfiles, dynamic=False):
         else:
             symbol_type = cols[0]
             symbol = cols[1]
-        if symbol_type in "BDGTRVWSi":
+        if symbol_type in b"BDGTRVWSi":
             # its a public symbol
             ret[filename]["PUBLIC"].add(symbol)
-        elif symbol_type in "U":
+        elif symbol_type in b"U":
             ret[filename]["UNDEFINED"].add(symbol)
 
     # add to the cache
@@ -106,10 +106,10 @@ def find_ldd_path(bld, libname, binary):
     lddpipe = subprocess.Popen(['ldd', binary], stdout=subprocess.PIPE).stdout
     for line in lddpipe:
         line = line.strip()
-        cols = line.split(" ")
-        if len(cols) < 3 or cols[1] != "=>":
+        cols = line.split(b" ")
+        if len(cols) < 3 or cols[1] != b"=>":
             continue
-        if cols[0].startswith("libc."):
+        if cols[0].startswith(b"libc."):
             # save this one too
             bld.env.libc_path = cols[2]
         if cols[0].startswith(libname):
@@ -119,8 +119,9 @@ def find_ldd_path(bld, libname, binary):
 
 
 # some regular expressions for parsing readelf output
-re_sharedlib = re.compile('Shared library: \[(.*)\]')
-re_rpath     = re.compile('Library rpath: \[(.*)\]')
+re_sharedlib = re.compile(b'Shared library: \[(.*)\]')
+# output from readelf could be `Library rpath` or `Libray runpath`
+re_rpath     = re.compile(b'Library (rpath|runpath): \[(.*)\]')
 
 def get_libs(bld, binname):
     '''find the list of linked libraries for any binary or library
@@ -146,7 +147,8 @@ def get_libs(bld, binname):
             libs.add(m.group(1))
         m = re_rpath.search(line)
         if m:
-            rpath.extend(m.group(1).split(":"))
+            # output from Popen is always bytestr even in py3
+            rpath.extend(m.group(2).split(b":"))
 
     ret = set()
     for lib in libs:
@@ -410,7 +412,7 @@ def check_library_deps(bld, t):
             if dep2 == name and t.in_library != t2.in_library:
                 Logs.warn("WARNING: mutual dependency %s <=> %s" % (name, real_name(t2.sname)))
                 Logs.warn("Libraries should match. %s != %s" % (t.in_library, t2.in_library))
-                # raise Utils.WafError("illegal mutual dependency")
+                # raise Errors.WafError("illegal mutual dependency")
 
 
 def check_syslib_collisions(bld, tgt_list):
@@ -430,7 +432,7 @@ def check_syslib_collisions(bld, tgt_list):
                 Logs.error("ERROR: Target '%s' has symbols '%s' which is also in syslib '%s'" % (t.sname, common, lib))
                 has_error = True
     if has_error:
-        raise Utils.WafError("symbols in common with system libraries")
+        raise Errors.WafError("symbols in common with system libraries")
 
 
 def check_dependencies(bld, t):
@@ -546,7 +548,7 @@ def symbols_whyneeded(task):
 
     why = Options.options.WHYNEEDED.split(":")
     if len(why) != 2:
-        raise Utils.WafError("usage: WHYNEEDED=TARGET:DEPENDENCY")
+        raise Errors.WafError("usage: WHYNEEDED=TARGET:DEPENDENCY")
     target = why[0]
     subsystem = why[1]
 
@@ -579,7 +581,7 @@ def report_duplicate(bld, binname, sym, libs, fail_on_error):
         else:
             libnames.append(lib)
     if fail_on_error:
-        raise Utils.WafError("%s: Symbol %s linked in multiple libraries %s" % (binname, sym, libnames))
+        raise Errors.WafError("%s: Symbol %s linked in multiple libraries %s" % (binname, sym, libnames))
     else:
         print("%s: Symbol %s linked in multiple libraries %s" % (binname, sym, libnames))
 
diff --git a/buildtools/wafsamba/test_duplicate_symbol.sh b/buildtools/wafsamba/test_duplicate_symbol.sh
index 89a4027..46f44a6 100755
--- a/buildtools/wafsamba/test_duplicate_symbol.sh
+++ b/buildtools/wafsamba/test_duplicate_symbol.sh
@@ -5,7 +5,7 @@
 
 subunit_start_test duplicate_symbols
 
-if ./buildtools/bin/waf build --dup-symbol-check; then
+if $PYTHON ./buildtools/bin/waf build --dup-symbol-check; then
 	subunit_pass_test duplicate_symbols
 else
 	echo | subunit_fail_test duplicate_symbols
diff --git a/buildtools/wafsamba/tests/test_abi.py b/buildtools/wafsamba/tests/test_abi.py
index bba78c1..d6bdb04 100644
--- a/buildtools/wafsamba/tests/test_abi.py
+++ b/buildtools/wafsamba/tests/test_abi.py
@@ -21,7 +21,7 @@ from wafsamba.samba_abi import (
     normalise_signature,
     )
 
-from cStringIO import StringIO
+from samba.compat import StringIO
 
 
 class NormaliseSignatureTests(TestCase):
@@ -66,6 +66,10 @@ class WriteVscriptTests(TestCase):
 1.0 {
 \tglobal:
 \t\t*;
+\tlocal:
+\t\t_end;
+\t\t__bss_start;
+\t\t_edata;
 };
 """)
 
@@ -84,6 +88,10 @@ MYLIB_0.1 {
 1.0 {
 \tglobal:
 \t\t*;
+\tlocal:
+\t\t_end;
+\t\t__bss_start;
+\t\t_edata;
 };
 """)
 
@@ -99,6 +107,9 @@ MYLIB_0.1 {
 \t\t*;
 \tlocal:
 \t\texc_*;
+\t\t_end;
+\t\t__bss_start;
+\t\t_edata;
 };
 """)
 
@@ -115,6 +126,9 @@ MYLIB_0.1 {
 \t\tpub_*;
 \tlocal:
 \t\texc_*;
+\t\t_end;
+\t\t__bss_start;
+\t\t_edata;
 \t\t*;
 };
 """)
diff --git a/buildtools/wafsamba/tru64cc.py b/buildtools/wafsamba/tru64cc.py
 deleted file mode 100644
index e1bbb1d..0000000
--- a/buildtools/wafsamba/tru64cc.py
+++ /dev/null
@@ -1,77 +0,0 @@
-
-# compiler definition for tru64/OSF1 cc compiler
-# based on suncc.py from waf
-
-import os, optparse
-import Utils, Options, Configure
-import ccroot, ar
-from Configure import conftest
-
-from compiler_cc import c_compiler
-
-c_compiler['osf1V'] = ['gcc', 'tru64cc']
-
-@conftest
-def find_tru64cc(conf):
-    v = conf.env
-    cc = None
-    if v['CC']: cc = v['CC']
-    elif 'CC' in conf.environ: cc = conf.environ['CC']
-    if not cc: cc = conf.find_program('cc', var='CC')
-    if not cc: conf.fatal('tru64cc was not found')
-    cc = conf.cmd_to_list(cc)
-
-    try:
-        if not Utils.cmd_output(cc + ['-V']):
-            conf.fatal('tru64cc %r was not found' % cc)
-    except ValueError:
-        conf.fatal('tru64cc -V could not be executed')
-
-    v['CC']  = cc
-    v['CC_NAME'] = 'tru64'
-
-@conftest
-def tru64cc_common_flags(conf):
-    v = conf.env
-
-    v['CC_SRC_F']            = ''
-    v['CC_TGT_F']            = ['-c', '-o', '']
-    v['CPPPATH_ST']          = '-I%s' # template for adding include paths
-
-    # linker
-    if not v['LINK_CC']: v['LINK_CC'] = v['CC']
-    v['CCLNK_SRC_F']         = ''
-    v['CCLNK_TGT_F']         = ['-o', '']
-
-    v['LIB_ST']              = '-l%s' # template for adding libs
-    v['LIBPATH_ST']          = '-L%s' # template for adding libpaths
-    v['STATICLIB_ST']        = '-l%s'
-    v['STATICLIBPATH_ST']    = '-L%s'
-    v['CCDEFINES_ST']        = '-D%s'
-
-#    v['SONAME_ST']           = '-Wl,-h -Wl,%s'
-#    v['SHLIB_MARKER']        = '-Bdynamic'
-#    v['STATICLIB_MARKER']    = '-Bstatic'
-
-    # program
-    v['program_PATTERN']     = '%s'
-
-    # shared library
-#    v['shlib_CCFLAGS']       = ['-Kpic', '-DPIC']
-    v['shlib_LINKFLAGS']     = ['-shared']
-    v['shlib_PATTERN']       = 'lib%s.so'
-
-    # static lib
-#    v['staticlib_LINKFLAGS'] = ['-Bstatic']
-#    v['staticlib_PATTERN']   = 'lib%s.a'
-
-detect = '''
-find_tru64cc
-find_cpp
-find_ar
-tru64cc_common_flags
-cc_load_tools
-cc_add_flags
-link_add_flags
-'''
-
diff --git a/buildtools/wafsamba/wafsamba.py b/buildtools/wafsamba/wafsamba.py
index 12d5421..1b98e1c 100644
--- a/buildtools/wafsamba/wafsamba.py
+++ b/buildtools/wafsamba/wafsamba.py
@@ -1,15 +1,16 @@
 # a waf tool to add autoconf-like macros to the configure section
 # and for SAMBA_ macros for building libraries, binaries etc
 
-import Build, os, sys, Options, Task, Utils, cc, TaskGen, fnmatch, re, shutil, Logs, Constants
-from Configure import conf
-from Logs import debug
+import os, sys, re, shutil, fnmatch
+from waflib import Build, Options, Task, Utils, TaskGen, Logs, Context, Errors
+from waflib.Configure import conf
+from waflib.Logs import debug
 from samba_utils import SUBST_VARS_RECURSIVE
 TaskGen.task_gen.apply_verif = Utils.nada
 
 # bring in the other samba modules
-from samba_optimisation import *
 from samba_utils import *
+from samba_utils import symlink
 from samba_version import *
 from samba_autoconf import *
 from samba_patterns import *
@@ -25,27 +26,19 @@ import samba_install
 import samba_conftests
 import samba_abi
 import samba_headers
-import tru64cc
-import irixcc
-import hpuxcc
 import generic_cc
 import samba_dist
 import samba_wildcard
-import stale_files
 import symbols
 import pkgconfig
 import configure_file
-
-# some systems have broken threading in python
-if os.environ.get('WAF_NOTHREADS') == '1':
-    import nothreads
+import samba_waf18
 
 LIB_PATH="shared"
 
 os.environ['PYTHONUNBUFFERED'] = '1'
 
-
-if Constants.HEXVERSION < 0x105019:
+if Context.HEXVERSION not in (0x2000800,):
     Logs.error('''
 Please use the version of waf that comes with Samba, not
 a system installed version. See http://wiki.samba.org/index.php/Waf
@@ -55,26 +48,25 @@ Alternatively, please run ./configure and make as usual. That will
 call the right version of waf.''')
     sys.exit(1)
 
-
 @conf
 def SAMBA_BUILD_ENV(conf):
     '''create the samba build environment'''
-    conf.env.BUILD_DIRECTORY = conf.blddir
-    mkdir_p(os.path.join(conf.blddir, LIB_PATH))
-    mkdir_p(os.path.join(conf.blddir, LIB_PATH, "private"))
-    mkdir_p(os.path.join(conf.blddir, "modules"))
-    mkdir_p(os.path.join(conf.blddir, 'python/samba/dcerpc'))
+    conf.env.BUILD_DIRECTORY = conf.bldnode.abspath()
+    mkdir_p(os.path.join(conf.env.BUILD_DIRECTORY, LIB_PATH))
+    mkdir_p(os.path.join(conf.env.BUILD_DIRECTORY, LIB_PATH, "private"))
+    mkdir_p(os.path.join(conf.env.BUILD_DIRECTORY, "modules"))
+    mkdir_p(os.path.join(conf.env.BUILD_DIRECTORY, 'python/samba/dcerpc'))
     # this allows all of the bin/shared and bin/python targets
     # to be expressed in terms of build directory paths
-    mkdir_p(os.path.join(conf.blddir, 'default'))
-    for (source, target) in [('shared', 'shared'), ('modules', 'modules'), ('python', 'python_modules')]:
-        link_target = os.path.join(conf.blddir, 'default/' + target)
+    mkdir_p(os.path.join(conf.env.BUILD_DIRECTORY, 'default'))
+    for (source, target) in [('shared', 'shared'), ('modules', 'modules'), ('python', 'python')]:
+        link_target = os.path.join(conf.env.BUILD_DIRECTORY, 'default/' + target)
         if not os.path.lexists(link_target):
-            os.symlink('../' + source, link_target)
+            symlink('../' + source, link_target)
 
     # get perl to put the blib files in the build directory
-    blib_bld = os.path.join(conf.blddir, 'default/pidl/blib')
-    blib_src = os.path.join(conf.srcdir, 'pidl/blib')
+    blib_bld = os.path.join(conf.env.BUILD_DIRECTORY, 'default/pidl/blib')
+    blib_src = os.path.join(conf.srcnode.abspath(), 'pidl/blib')
     mkdir_p(blib_bld + '/man1')
     mkdir_p(blib_bld + '/man3')
     if os.path.islink(blib_src):
@@ -148,7 +140,7 @@ def SAMBA_LIBRARY(bld, libname, source,
         public_headers = None
 
     if private_library and public_headers:
-        raise Utils.WafError("private library '%s' must not have public header files" %
+        raise Errors.WafError("private library '%s' must not have public header files" %
                              libname)
 
     if LIB_MUST_BE_PRIVATE(bld, libname):
@@ -225,13 +217,13 @@ def SAMBA_LIBRARY(bld, libname, source,
     # we don't want any public libraries without version numbers
     if (not private_library and target_type != 'PYTHON' and not realname):
         if vnum is None and soname is None:
-            raise Utils.WafError("public library '%s' must have a vnum" %
+            raise Errors.WafError("public library '%s' must have a vnum" %
                     libname)
         if pc_files is None:
-            raise Utils.WafError("public library '%s' must have pkg-config file" %
+            raise Errors.WafError("public library '%s' must have pkg-config file" %
                        libname)
         if public_headers is None and not bld.env['IS_EXTRA_PYTHON']:
-            raise Utils.WafError("public library '%s' must have header files" %
+            raise Errors.WafError("public library '%s' must have header files" %
                        libname)
 
     if bundled_name is not None:
@@ -273,7 +265,7 @@ def SAMBA_LIBRARY(bld, libname, source,
     vscript = None
     if bld.env.HAVE_LD_VERSION_SCRIPT:
         if private_library:
-            version = "%s_%s" % (Utils.g_module.APPNAME, Utils.g_module.VERSION)
+            version = "%s_%s" % (Context.g_module.APPNAME, Context.g_module.VERSION)
         elif vnum:
             version = "%s_%s" % (libname, vnum)
         else:
@@ -282,17 +274,17 @@ def SAMBA_LIBRARY(bld, libname, source,
             vscript = "%s.vscript" % libname
             bld.ABI_VSCRIPT(version_libname, abi_directory, version, vscript,
                             abi_match)
-            fullname = apply_pattern(bundled_name, bld.env.shlib_PATTERN)
+            fullname = apply_pattern(bundled_name, bld.env.cshlib_PATTERN)
             fullpath = bld.path.find_or_declare(fullname)
             vscriptpath = bld.path.find_or_declare(vscript)
             if not fullpath:
-                raise Utils.WafError("unable to find fullpath for %s" % fullname)
+                raise Errors.WafError("unable to find fullpath for %s" % fullname)
             if not vscriptpath:
-                raise Utils.WafError("unable to find vscript path for %s" % vscript)
+                raise Errors.WafError("unable to find vscript path for %s" % vscript)
             bld.add_manual_dependency(fullpath, vscriptpath)
             if bld.is_install:
                 # also make the .inst file depend on the vscript
-                instname = apply_pattern(bundled_name + '.inst', bld.env.shlib_PATTERN)
+                instname = apply_pattern(bundled_name + '.inst', bld.env.cshlib_PATTERN)
                 bld.add_manual_dependency(bld.path.find_or_declare(instname), bld.path.find_or_declare(vscript))
             vscript = os.path.join(bld.path.abspath(bld.env), vscript)
 
@@ -327,10 +319,12 @@ def SAMBA_LIBRARY(bld, libname, source,
         link_name = 'shared/%s' % realname
 
     if link_name:
+        if 'waflib.extras.compat15' in sys.modules:
+            link_name = 'default/' + link_name
         t.link_name = link_name
 
     if pc_files is not None and not private_library:
-        if pyembed and bld.env['IS_EXTRA_PYTHON']:
+        if pyembed:
             bld.PKG_CONFIG_FILES(pc_files, vnum=vnum, extra_name=bld.env['PYTHON_SO_ABI_FLAG'])
         else:
             bld.PKG_CONFIG_FILES(pc_files, vnum=vnum)
@@ -674,7 +668,7 @@ def SAMBA_GENERATOR(bld, name, rule, source='', target='',
         target=target,
         shell=isinstance(rule, str),
         update_outputs=True,
-        before='cc',
+        before='c',
         ext_out='.c',
         samba_type='GENERATOR',
         dep_vars = dep_vars,
@@ -728,22 +722,6 @@ Build.BuildContext.SET_BUILD_GROUP = SET_BUILD_GROUP
 
 
 
-@conf
-def ENABLE_TIMESTAMP_DEPENDENCIES(conf):
-    """use timestamps instead of file contents for deps
-    this currently doesn't work"""
-    def h_file(filename):
-        import stat
-        st = os.stat(filename)
-        if stat.S_ISDIR(st[stat.ST_MODE]): raise IOError('not a file')
-        m = Utils.md5()
-        m.update(str(st.st_mtime))
-        m.update(str(st.st_size))
-        m.update(filename)
-        return m.digest()
-    Utils.h_file = h_file
-
-
 def SAMBA_SCRIPT(bld, name, pattern, installdir, installname=None):
     '''used to copy scripts from the source tree into the build directory
        for use by selftest'''
@@ -758,14 +736,14 @@ def SAMBA_SCRIPT(bld, name, pattern, installdir, installname=None):
         target = os.path.join(installdir, iname)
         tgtdir = os.path.dirname(os.path.join(bld.srcnode.abspath(bld.env), '..', target))
         mkdir_p(tgtdir)
-        link_src = os.path.normpath(os.path.join(bld.curdir, s))
+        link_src = os.path.normpath(os.path.join(bld.path.abspath(), s))
         link_dst = os.path.join(tgtdir, os.path.basename(iname))
         if os.path.islink(link_dst) and os.readlink(link_dst) == link_src:
             continue
-        if os.path.exists(link_dst):
+        if os.path.islink(link_dst):
             os.unlink(link_dst)
         Logs.info("symlink: %s -> %s/%s" % (s, installdir, iname))
-        os.symlink(link_src, link_dst)
+        symlink(link_src, link_dst)
 Build.BuildContext.SAMBA_SCRIPT = SAMBA_SCRIPT
 
 
@@ -779,10 +757,10 @@ def copy_and_fix_python_path(task):
         replacement="""sys.path.insert(0, "%s")
 sys.path.insert(1, "%s")""" % (task.env["PYTHONARCHDIR"], task.env["PYTHONDIR"])
 
-    if task.env["PYTHON"][0] == "/":
-        replacement_shebang = "#!%s\n" % task.env["PYTHON"]
+    if task.env["PYTHON"][0].startswith("/"):
+        replacement_shebang = "#!%s\n" % task.env["PYTHON"][0]
     else:
-        replacement_shebang = "#!/usr/bin/env %s\n" % task.env["PYTHON"]
+        replacement_shebang = "#!/usr/bin/env %s\n" % task.env["PYTHON"][0]
 
     installed_location=task.outputs[0].bldpath(task.env)
     source_file = open(task.inputs[0].srcpath(task.env))
@@ -790,7 +768,7 @@ sys.path.insert(1, "%s")""" % (task.env["PYTHONARCHDIR"], task.env["PYTHONDIR"])
     lineno = 0
     for line in source_file:
         newline = line
-        if (lineno == 0 and task.env["PYTHON_SPECIFIED"] is True and
+        if (lineno == 0 and
                 line[:2] == "#!"):
             newline = replacement_shebang
         elif pattern in line:
@@ -798,7 +776,7 @@ sys.path.insert(1, "%s")""" % (task.env["PYTHONARCHDIR"], task.env["PYTHONDIR"])
         installed_file.write(newline)
         lineno = lineno + 1
     installed_file.close()
-    os.chmod(installed_location, 0755)
+    os.chmod(installed_location, 0o755)
     return 0
 
 def copy_and_fix_perl_path(task):
@@ -826,7 +804,7 @@ def copy_and_fix_perl_path(task):
         installed_file.write(newline)
         lineno = lineno + 1
     installed_file.close()
-    os.chmod(installed_location, 0755)
+    os.chmod(installed_location, 0o755)
     return 0
 
 
@@ -834,6 +812,8 @@ def install_file(bld, destdir, file, chmod=MODE_644, flat=False,
                  python_fixup=False, perl_fixup=False,
                  destname=None, base_name=None):
     '''install a file'''
+    if not isinstance(file, str):
+        file = file.abspath()
     destdir = bld.EXPAND_VARIABLES(destdir)
     if not destname:
         destname = file
@@ -898,16 +878,19 @@ def INSTALL_DIR(bld, path, chmod=0o755, env=None):
     if not path:
         return []
 
-    destpath = bld.get_install_path(path, env)
+    destpath = bld.EXPAND_VARIABLES(path)
+    if Options.options.destdir:
+        destpath = os.path.join(Options.options.destdir, destpath.lstrip(os.sep))
 
     if bld.is_install > 0:
         if not os.path.isdir(destpath):
             try:
+                Logs.info('* create %s', destpath)
                 os.makedirs(destpath)
                 os.chmod(destpath, chmod)
             except OSError as e:
                 if not os.path.isdir(destpath):
-                    raise Utils.WafError("Cannot create the folder '%s' (error: %s)" % (path, e))
+                    raise Errors.WafError("Cannot create the folder '%s' (error: %s)" % (path, e))
 Build.BuildContext.INSTALL_DIR = INSTALL_DIR
 
 def INSTALL_DIRS(bld, destdir, dirs, chmod=0o755, env=None):
@@ -938,7 +921,7 @@ def SAMBAMANPAGES(bld, manpages, extra_source=None):
     '''build and install manual pages'''
     bld.env.SAMBA_EXPAND_XSL = bld.srcnode.abspath() + '/docs-xml/xslt/expand-sambadoc.xsl'
     bld.env.SAMBA_MAN_XSL = bld.srcnode.abspath() + '/docs-xml/xslt/man.xsl'
-    bld.env.SAMBA_CATALOG = bld.srcnode.abspath() + '/bin/default/docs-xml/build/catalog.xml'
+    bld.env.SAMBA_CATALOG = bld.bldnode.abspath() + '/docs-xml/build/catalog.xml'
     bld.env.SAMBA_CATALOGS = 'file:///etc/xml/catalog file:///usr/local/share/xml/catalog file://' + bld.env.SAMBA_CATALOG
 
     for m in manpages.split():
@@ -958,59 +941,6 @@ def SAMBAMANPAGES(bld, manpages, extra_source=None):
         bld.INSTALL_FILES('${MANDIR}/man%s' % m[-1], m, flat=True)
 Build.BuildContext.SAMBAMANPAGES = SAMBAMANPAGES
 
-#############################################################
-# give a nicer display when building different types of files
-def progress_display(self, msg, fname):
-    col1 = Logs.colors(self.color)
-    col2 = Logs.colors.NORMAL
-    total = self.position[1]
-    n = len(str(total))
-    fs = '[%%%dd/%%%dd] %s %%s%%s%%s\n' % (n, n, msg)
-    return fs % (self.position[0], self.position[1], col1, fname, col2)
-
-def link_display(self):
-    if Options.options.progress_bar != 0:
-        return Task.Task.old_display(self)
-    fname = self.outputs[0].bldpath(self.env)
-    return progress_display(self, 'Linking', fname)
-Task.TaskBase.classes['cc_link'].display = link_display
-
-def samba_display(self):
-    if Options.options.progress_bar != 0:
-        return Task.Task.old_display(self)
-
-    targets    = LOCAL_CACHE(self, 'TARGET_TYPE')
-    if self.name in targets:
-        target_type = targets[self.name]
-        type_map = { 'GENERATOR' : 'Generating',
-                     'PROTOTYPE' : 'Generating'
-                     }
-        if target_type in type_map:
-            return progress_display(self, type_map[target_type], self.name)
-
-    if len(self.inputs) == 0:
-        return Task.Task.old_display(self)
-
-    fname = self.inputs[0].bldpath(self.env)
-    if fname[0:3] == '../':
-        fname = fname[3:]
-    ext_loc = fname.rfind('.')
-    if ext_loc == -1:
-        return Task.Task.old_display(self)
-    ext = fname[ext_loc:]
-
-    ext_map = { '.idl' : 'Compiling IDL',
-                '.et'  : 'Compiling ERRTABLE',
-                '.asn1': 'Compiling ASN1',
-                '.c'   : 'Compiling' }
-    if ext in ext_map:
-        return progress_display(self, ext_map[ext], fname)
-    return Task.Task.old_display(self)
-
-Task.TaskBase.classes['Task'].old_display = Task.TaskBase.classes['Task'].display
-Task.TaskBase.classes['Task'].display = samba_display
-
-
 @after('apply_link')
 @feature('cshlib')
 def apply_bundle_remove_dynamiclib_patch(self):
diff --git a/buildtools/wafsamba/wscript b/buildtools/wafsamba/wscript
index 0eef330..ab19859 100644
--- a/buildtools/wafsamba/wscript
+++ b/buildtools/wafsamba/wscript
@@ -3,7 +3,8 @@
 # this is a base set of waf rules that everything else pulls in first
 
 import os, sys
-import wafsamba, Configure, Logs, Options, Utils
+from waflib import Configure, Logs, Options, Utils, Context, Errors
+import wafsamba
 from samba_utils import os_path_relpath
 from optparse import SUPPRESS_HELP
 
@@ -14,12 +15,17 @@ from optparse import SUPPRESS_HELP
 # are resolved related to WAFCACHE. It will need a lot of testing
 # before it is enabled by default.
 if '--enable-auto-reconfigure' in sys.argv:
-    Configure.autoconfig = True
+    Configure.autoconfig = 'clobber'
 
-def set_options(opt):
-    opt.tool_options('compiler_cc')
+def default_value(option, default=''):
+    if option in Options.options.__dict__:
+        return Options.options.__dict__[option]
+    return default
 
-    opt.tool_options('gnu_dirs')
+def options(opt):
+    opt.load('compiler_cc')
+
+    opt.load('gnu_dirs')
 
     gr = opt.option_group('library handling options')
 
@@ -31,17 +37,17 @@ def set_options(opt):
                    help=("comma separated list of normally public libraries to build instead as private libraries. May include !LIBNAME to disable making a library private. Can be 'NONE' or 'ALL' [auto]"),
                    action="store", dest='PRIVATE_LIBS', default='')
 
-    extension_default = Options.options['PRIVATE_EXTENSION_DEFAULT']
+    extension_default = default_value('PRIVATE_EXTENSION_DEFAULT')
     gr.add_option('--private-library-extension',
                    help=("name extension for private libraries [%s]" % extension_default),
                    action="store", dest='PRIVATE_EXTENSION', default=extension_default)
 
-    extension_exception = Options.options['PRIVATE_EXTENSION_EXCEPTION']
+    extension_exception = default_value('PRIVATE_EXTENSION_EXCEPTION')
     gr.add_option('--private-extension-exception',
                    help=("comma separated list of libraries to not apply extension to [%s]" % extension_exception),
                    action="store", dest='PRIVATE_EXTENSION_EXCEPTION', default=extension_exception)
 
-    builtin_default = Options.options['BUILTIN_LIBRARIES_DEFAULT']
+    builtin_default = default_value('BUILTIN_LIBRARIES_DEFAULT')
     gr.add_option('--builtin-libraries',
                    help=("command separated list of libraries to build directly into binaries [%s]" % builtin_default),
                    action="store", dest='BUILTIN_LIBRARIES', default=builtin_default)
@@ -71,7 +77,7 @@ def set_options(opt):
                    action="store", dest='MODULESDIR', default='${PREFIX}/modules')
 
     opt.add_option('--with-privatelibdir',
-                   help=("private library directory [PREFIX/lib/%s]" % Utils.g_module.APPNAME),
+                   help=("private library directory [PREFIX/lib/%s]" % Context.g_module.APPNAME),
                    action="store", dest='PRIVATELIBDIR', default=None)
 
     opt.add_option('--with-libiconv',
@@ -109,9 +115,6 @@ def set_options(opt):
     gr.add_option('--enable-gccdeps',
                    help=("Enable use of gcc -MD dependency module"),
                    action="store_true", dest='enable_gccdeps', default=True)
-    gr.add_option('--timestamp-dependencies',
-                   help=("use file timestamps instead of content for build dependencies (BROKEN)"),
-                   action="store_true", dest='timestamp_dependencies', default=False)
     gr.add_option('--pedantic',
 		   help=("Enable even more compiler warnings"),
 		   action='store_true', dest='pedantic', default=False)
@@ -210,23 +213,19 @@ def set_options(opt):
 @Utils.run_once
 def configure(conf):
     conf.env.hlist = []
-    conf.env.srcdir = conf.srcdir
+    conf.env.srcdir = conf.srcnode.abspath()
 
     conf.define('SRCDIR', conf.env['srcdir'])
 
-    if Options.options.timestamp_dependencies:
-        conf.ENABLE_TIMESTAMP_DEPENDENCIES()
-
     conf.SETUP_CONFIGURE_CACHE(Options.options.enable_configure_cache)
 
     # load our local waf extensions
-    conf.check_tool('gnu_dirs')
-    conf.check_tool('wafsamba')
-    conf.check_tool('print_commands')
+    conf.load('gnu_dirs')
+    conf.load('wafsamba')
 
     conf.CHECK_CC_ENV()
 
-    conf.check_tool('compiler_cc')
+    conf.load('compiler_c')
 
     conf.CHECK_STANDARD_LIBPATH()
 
@@ -236,31 +235,10 @@ def configure(conf):
     # older gcc versions (< 4.4) does not work with gccdeps, so we have to see if the .d file is generated
     if Options.options.enable_gccdeps:
         # stale file removal - the configuration may pick up the old .pyc file
-        p = os.path.join(conf.srcdir, 'buildtools/wafsamba/gccdeps.pyc')
+        p = os.path.join(conf.env.srcdir, 'buildtools/wafsamba/gccdeps.pyc')
         if os.path.exists(p):
             os.remove(p)
-
-        from TaskGen import feature, after
-        @feature('testd')
-        @after('apply_core')
-        def check_d(self):
-            tsk = self.compiled_tasks[0]
-            tsk.outputs.append(tsk.outputs[0].change_ext('.d'))
-
-        import Task
-        cc = Task.TaskBase.classes['cc']
-        oldmeth = cc.run
-
-        cc.run = Task.compile_fun_noshell('cc', '${CC} ${CCFLAGS} ${CPPFLAGS} ${_CCINCFLAGS} ${_CCDEFFLAGS} ${CC_SRC_F}${SRC} ${CC_TGT_F}${TGT[0].abspath(env)}')[0]
-        try:
-            try:
-                conf.check(features='c testd', fragment='int main() {return 0;}\n', ccflags=['-MD'], mandatory=True, msg='Check for -MD')
-            except:
-                pass
-            else:
-                conf.check_tool('gccdeps', tooldir=conf.srcdir + "/buildtools/wafsamba")
-        finally:
-            cc.run = oldmeth
+        conf.load('gccdeps')
 
     # make the install paths available in environment
     conf.env.LIBDIR = Options.options.LIBDIR or '${PREFIX}/lib'
@@ -330,15 +308,16 @@ def configure(conf):
               "-qhalt=w",     # IBM xlc
               "-w2",           # Tru64
              ]:
-        if conf.CHECK_CFLAGS([f], '''
-'''):
+        if conf.CHECK_CFLAGS([f]):
             if not 'WERROR_CFLAGS' in conf.env:
                 conf.env['WERROR_CFLAGS'] = []
             conf.env['WERROR_CFLAGS'].extend([f])
             break
 
     # check which compiler/linker flags are needed for rpath support
-    if not conf.CHECK_LDFLAGS(['-Wl,-rpath,.']) and conf.CHECK_LDFLAGS(['-Wl,-R,.']):
+    if conf.CHECK_LDFLAGS(['-Wl,-rpath,.']):
+        conf.env['RPATH_ST'] = '-Wl,-rpath,%s'
+    elif conf.CHECK_LDFLAGS(['-Wl,-R,.']):
         conf.env['RPATH_ST'] = '-Wl,-R,%s'
 
     # check for rpath
@@ -348,7 +327,7 @@ def configure(conf):
         conf.env.RPATH_ON_INSTALL = (conf.env.RPATH_ON_BUILD and
                                      not Options.options.disable_rpath_install)
         if not conf.env.PRIVATELIBDIR:
-            conf.env.PRIVATELIBDIR = '%s/%s' % (conf.env.LIBDIR, Utils.g_module.APPNAME)
+            conf.env.PRIVATELIBDIR = '%s/%s' % (conf.env.LIBDIR, Context.g_module.APPNAME)
         conf.env.RPATH_ON_INSTALL_PRIVATE = (
             not Options.options.disable_rpath_private_install)
     else:
@@ -370,10 +349,10 @@ def configure(conf):
     else:
         conf.env.HAVE_LD_VERSION_SCRIPT = False
 
-    if conf.CHECK_CFLAGS(['-fvisibility=hidden'] + conf.env.WERROR_CFLAGS):
+    if conf.CHECK_CFLAGS(['-fvisibility=hidden']):
         conf.env.VISIBILITY_CFLAGS = '-fvisibility=hidden'
         conf.CHECK_CODE('''int main(void) { return 0; }
-                           __attribute__((visibility("default"))) void vis_foo2(void) {}''',
+                           __attribute__((visibility("default"))) void vis_foo2(void) {}\n''',
                         cflags=conf.env.VISIBILITY_CFLAGS,
                         strict=True,
                         define='HAVE_VISIBILITY_ATTR', addmain=False)
@@ -466,6 +445,15 @@ def configure(conf):
     conf.DEFINE('_GNU_SOURCE', 1, add_to_cflags=True)
     conf.DEFINE('_XOPEN_SOURCE_EXTENDED', 1, add_to_cflags=True)
 
+    #
+    # Needs to be defined before std*.h and string*.h are included
+    # As Python.h already brings string.h we need it in CFLAGS.
+    # See memset_s() details here:
+    # https://en.cppreference.com/w/c/string/byte/memset
+    #
+    if conf.CHECK_CFLAGS(['-D__STDC_WANT_LIB_EXT1__=1']):
+        conf.ADD_CFLAGS('-D__STDC_WANT_LIB_EXT1__=1')
+
     # on Tru64 certain features are only available with _OSF_SOURCE set to 1
     # and _XOPEN_SOURCE set to 600
     if conf.env['SYSTEM_UNAME_SYSNAME'] == 'OSF1':
@@ -501,7 +489,7 @@ struct foo bar = { .y = 'X', .x = 1 };
 
     # see if we need special largefile flags
     if not conf.CHECK_LARGEFILE():
-        raise Utils.WafError('Samba requires large file support support, but not available on this platform: sizeof(off_t) < 8')
+        raise Errors.WafError('Samba requires large file support support, but not available on this platform: sizeof(off_t) < 8')
 
     if conf.env.HAVE_STDDEF_H and conf.env.HAVE_STDLIB_H:
         conf.DEFINE('STDC_HEADERS', 1)
@@ -512,7 +500,7 @@ struct foo bar = { .y = 'X', .x = 1 };
         conf.DEFINE('TIME_WITH_SYS_TIME', 1)
 
     # cope with different extensions for libraries
-    (root, ext) = os.path.splitext(conf.env.shlib_PATTERN)
+    (root, ext) = os.path.splitext(conf.env.cshlib_PATTERN)
     if ext[0] == '.':
         conf.define('SHLIBEXT', ext[1:], quote=True)
     else:
@@ -534,7 +522,7 @@ struct foo bar = { .y = 'X', .x = 1 };
                         #if !defined(LITTLE) || !defined(B) || LITTLE != B
                         #error Not little endian.
                         #endif
-                        int main(void) { return 0; }""",
+                        int main(void) { return 0; }\n""",
                             addmain=False,
                             headers="endian.h sys/endian.h",
                             define="HAVE_LITTLE_ENDIAN")
@@ -553,7 +541,7 @@ struct foo bar = { .y = 'X', .x = 1 };
                         #if !defined(BIG) || !defined(B) || BIG != B
                         #error Not big endian.
                         #endif
-                        int main(void) { return 0; }""",
+                        int main(void) { return 0; }\n""",
                             addmain=False,
                             headers="endian.h sys/endian.h",
                             define="HAVE_BIG_ENDIAN")
@@ -576,7 +564,7 @@ struct foo bar = { .y = 'X', .x = 1 };
     # Extra sanity check.
     if conf.CONFIG_SET("HAVE_BIG_ENDIAN") == conf.CONFIG_SET("HAVE_LITTLE_ENDIAN"):
         Logs.error("Failed endian determination.  The PDP-11 is back?")
-	sys.exit(1)
+        sys.exit(1)
     else:
         if conf.CONFIG_SET("HAVE_BIG_ENDIAN"):
             conf.DEFINE('WORDS_BIGENDIAN', 1)
@@ -607,12 +595,13 @@ struct foo bar = { .y = 'X', .x = 1 };
 
 def build(bld):
     # give a more useful message if the source directory has moved
-    relpath = os_path_relpath(bld.curdir, bld.srcnode.abspath())
+    curdir = bld.path.abspath()
+    srcdir = bld.srcnode.abspath()
+    relpath = os_path_relpath(curdir, srcdir)
     if relpath.find('../') != -1:
-        Logs.error('bld.curdir %s is not a child of %s' % (bld.curdir, bld.srcnode.abspath()))
-        raise Utils.WafError('''The top source directory has moved. Please run distclean and reconfigure''')
+        Logs.error('bld.path %s is not a child of %s' % (curdir, srcdir))
+        raise Errors.WafError('''The top source directory has moved. Please run distclean and reconfigure''')
 
-    bld.CHECK_MAKEFLAGS()
     bld.SETUP_BUILD_GROUPS()
     bld.ENFORCE_GROUP_ORDERING()
     bld.CHECK_PROJECT_RULES()
diff --git a/common/dump.c b/common/dump.c
index 73286b8..adcf591 100644
--- a/common/dump.c
+++ b/common/dump.c
@@ -60,6 +60,7 @@ static tdb_off_t tdb_dump_record(struct tdb_context *tdb, int hash,
 
 static int tdb_dump_chain(struct tdb_context *tdb, int i)
 {
+	struct tdb_chainwalk_ctx chainwalk;
 	tdb_off_t rec_ptr, top;
 
 	if (i == -1) {
@@ -74,11 +75,19 @@ static int tdb_dump_chain(struct tdb_context *tdb, int i)
 	if (tdb_ofs_read(tdb, top, &rec_ptr) == -1)
 		return tdb_unlock(tdb, i, F_WRLCK);
 
+	tdb_chainwalk_init(&chainwalk, rec_ptr);
+
 	if (rec_ptr)
 		printf("hash=%d\n", i);
 
 	while (rec_ptr) {
+		bool ok;
 		rec_ptr = tdb_dump_record(tdb, i, rec_ptr);
+		ok = tdb_chainwalk_check(tdb, &chainwalk, rec_ptr);
+		if (!ok) {
+			printf("circular hash chain %d\n", i);
+			break;
+		}
 	}
 
 	return tdb_unlock(tdb, i, F_WRLCK);
@@ -86,7 +95,7 @@ static int tdb_dump_chain(struct tdb_context *tdb, int i)
 
 _PUBLIC_ void tdb_dump_all(struct tdb_context *tdb)
 {
-	int i;
+	uint32_t i;
 	for (i=0;i<tdb->hash_size;i++) {
 		tdb_dump_chain(tdb, i);
 	}
diff --git a/common/freelist.c b/common/freelist.c
index 86fac2f..37a4c16 100644
--- a/common/freelist.c
+++ b/common/freelist.c
@@ -27,12 +27,6 @@
 
 #include "tdb_private.h"
 
-/* 'right' merges can involve O(n^2) cost when combined with a
-   traverse, so they are disabled until we find a way to do them in
-   O(1) time
-*/
-#define USE_RIGHT_MERGES 0
-
 /* read a freelist record and check for simple errors */
 int tdb_rec_free_read(struct tdb_context *tdb, tdb_off_t off, struct tdb_record *rec)
 {
@@ -61,30 +55,6 @@ int tdb_rec_free_read(struct tdb_context *tdb, tdb_off_t off, struct tdb_record 
 	return 0;
 }
 
-
-#if USE_RIGHT_MERGES
-/* Remove an element from the freelist.  Must have alloc lock. */
-static int remove_from_freelist(struct tdb_context *tdb, tdb_off_t off, tdb_off_t next)
-{
-	tdb_off_t last_ptr, i;
-
-	/* read in the freelist top */
-	last_ptr = FREELIST_TOP;
-	while (tdb_ofs_read(tdb, last_ptr, &i) != -1 && i != 0) {
-		if (i == off) {
-			/* We've found it! */
-			return tdb_ofs_write(tdb, last_ptr, &next);
-		}
-		/* Follow chain (next offset is at start of record) */
-		last_ptr = i;
-	}
-	tdb->ecode = TDB_ERR_CORRUPT;
-	TDB_LOG((tdb, TDB_DEBUG_FATAL,"remove_from_freelist: not on list at off=%u\n", off));
-	return -1;
-}
-#endif
-
-
 /* update a record tailer (must hold allocation lock) */
 static int update_tailer(struct tdb_context *tdb, tdb_off_t offset,
 			 const struct tdb_record *rec)
@@ -199,7 +169,7 @@ static int merge_with_left_record(struct tdb_context *tdb,
  *   0 if left was not a free record
  *   1 if left was free and successfully merged.
  *
- * The currend record is handed in with pointer and fully read record.
+ * The current record is handed in with pointer and fully read record.
  *
  * The left record pointer and struct can be retrieved as result
  * in lp and lr;
@@ -318,33 +288,6 @@ int tdb_free(struct tdb_context *tdb, tdb_off_t offset, struct tdb_record *rec)
 		goto fail;
 	}
 
-#if USE_RIGHT_MERGES
-	/* Look right first (I'm an Australian, dammit) */
-	if (offset + sizeof(*rec) + rec->rec_len + sizeof(*rec) <= tdb->map_size) {
-		tdb_off_t right = offset + sizeof(*rec) + rec->rec_len;
-		struct tdb_record r;
-
-		if (tdb->methods->tdb_read(tdb, right, &r, sizeof(r), DOCONV()) == -1) {
-			TDB_LOG((tdb, TDB_DEBUG_FATAL, "tdb_free: right read failed at %u\n", right));
-			goto left;
-		}
-
-		/* If it's free, expand to include it. */
-		if (r.magic == TDB_FREE_MAGIC) {
-			if (remove_from_freelist(tdb, right, r.next) == -1) {
-				TDB_LOG((tdb, TDB_DEBUG_FATAL, "tdb_free: right free failed at %u\n", right));
-				goto left;
-			}
-			rec->rec_len += sizeof(r) + r.rec_len;
-			if (update_tailer(tdb, offset, rec) == -1) {
-				TDB_LOG((tdb, TDB_DEBUG_FATAL, "tdb_free: update_tailer failed at %u\n", offset));
-				goto fail;
-			}
-		}
-	}
-left:
-#endif
-
 	ret = check_merge_with_left_record(tdb, offset, rec, NULL, NULL);
 	if (ret == -1) {
 		goto fail;
@@ -444,6 +387,8 @@ static tdb_off_t tdb_allocate_from_freelist(
 	struct tdb_context *tdb, tdb_len_t length, struct tdb_record *rec)
 {
 	tdb_off_t rec_ptr, last_ptr, newrec_ptr;
+	struct tdb_chainwalk_ctx chainwalk;
+	bool modified;
 	struct {
 		tdb_off_t rec_ptr, last_ptr;
 		tdb_len_t rec_len;
@@ -466,6 +411,9 @@ static tdb_off_t tdb_allocate_from_freelist(
 	if (tdb_ofs_read(tdb, FREELIST_TOP, &rec_ptr) == -1)
 		return 0;
 
+	modified = false;
+	tdb_chainwalk_init(&chainwalk, rec_ptr);
+
 	bestfit.rec_ptr = 0;
 	bestfit.last_ptr = 0;
 	bestfit.rec_len = 0;
@@ -526,6 +474,8 @@ static tdb_off_t tdb_allocate_from_freelist(
 				merge_created_candidate = true;
 			}
 
+			modified = true;
+
 			continue;
 		}
 
@@ -542,6 +492,14 @@ static tdb_off_t tdb_allocate_from_freelist(
 		last_ptr = rec_ptr;
 		rec_ptr = rec->next;
 
+		if (!modified) {
+			bool ok;
+			ok = tdb_chainwalk_check(tdb, &chainwalk, rec_ptr);
+			if (!ok) {
+				return 0;
+			}
+		}
+
 		/* if we've found a record that is big enough, then
 		   stop searching if its also not too big. The
 		   definition of 'too big' changes as we scan
@@ -597,6 +555,17 @@ static bool tdb_alloc_dead(
 	return (tdb_ofs_write(tdb, last_ptr, &rec->next) == 0);
 }
 
+static void tdb_purge_dead(struct tdb_context *tdb, uint32_t hash)
+{
+	int max_dead_records = tdb->max_dead_records;
+
+	tdb->max_dead_records = 0;
+
+	tdb_trim_dead(tdb, hash);
+
+	tdb->max_dead_records = max_dead_records;
+}
+
 /*
  * Chain "hash" is assumed to be locked
  */
@@ -605,7 +574,7 @@ tdb_off_t tdb_allocate(struct tdb_context *tdb, int hash, tdb_len_t length,
 		       struct tdb_record *rec)
 {
 	tdb_off_t ret;
-	int i;
+	uint32_t i;
 
 	if (tdb->max_dead_records == 0) {
 		/*
@@ -661,6 +630,12 @@ blocking_freelist_allocate:
 	if (tdb_lock(tdb, -1, F_WRLCK) == -1) {
 		return 0;
 	}
+	/*
+	 * Dead records can happen even if max_dead_records==0, they
+	 * are older than the max_dead_records concept: They happen if
+	 * tdb_delete happens concurrently with a traverse.
+	 */
+	tdb_purge_dead(tdb, hash);
 	ret = tdb_allocate_from_freelist(tdb, length, rec);
 	tdb_unlock(tdb, -1, F_WRLCK);
 	return ret;
diff --git a/common/io.c b/common/io.c
index 94b3163..df46017 100644
--- a/common/io.c
+++ b/common/io.c
@@ -96,7 +96,7 @@ static int tdb_ftruncate(struct tdb_context *tdb, off_t length)
 	return ret;
 }
 
-#if HAVE_POSIX_FALLOCATE
+#ifdef HAVE_POSIX_FALLOCATE
 static int tdb_posix_fallocate(struct tdb_context *tdb, off_t offset,
 			       off_t len)
 {
@@ -413,7 +413,7 @@ static int tdb_expand_file(struct tdb_context *tdb, tdb_off_t size, tdb_off_t ad
 		return -1;
 	}
 
-#if HAVE_POSIX_FALLOCATE
+#ifdef HAVE_POSIX_FALLOCATE
 	ret = tdb_posix_fallocate(tdb, size, addition);
 	if (ret == 0) {
 		return 0;
diff --git a/common/lock.c b/common/lock.c
index 9f30c7a..f55184d 100644
--- a/common/lock.c
+++ b/common/lock.c
@@ -149,8 +149,8 @@ static int fcntl_unlock(struct tdb_context *tdb, int rw, off_t off, off_t len)
  * This is the memory layout of the hashchain array:
  *
  * FREELIST_TOP + 0 = freelist
- * FREELIST_TOP + 4 = hashtbale list 0
- * FREELIST_TOP + 8 = hashtbale list 1
+ * FREELIST_TOP + 4 = hashtable list 0
+ * FREELIST_TOP + 8 = hashtable list 1
  * ...
  *
  * Otoh lock_offset computes:
diff --git a/common/open.c b/common/open.c
index 8baa7e4..dd5783e 100644
--- a/common/open.c
+++ b/common/open.c
@@ -230,8 +230,6 @@ static bool check_header_hash(struct tdb_context *tdb,
 static bool tdb_mutex_open_ok(struct tdb_context *tdb,
 			      const struct tdb_header *header)
 {
-	int locked;
-
 	if (tdb->flags & TDB_NOLOCK) {
 		/*
 		 * We don't look at locks, so it does not matter to have a
@@ -240,37 +238,6 @@ static bool tdb_mutex_open_ok(struct tdb_context *tdb,
 		return true;
 	}
 
-	locked = tdb_nest_lock(tdb, ACTIVE_LOCK, F_WRLCK,
-			       TDB_LOCK_NOWAIT|TDB_LOCK_PROBE);
-
-	if ((locked == -1) && (tdb->ecode == TDB_ERR_LOCK)) {
-		/*
-		 * CLEAR_IF_FIRST still active. The tdb was created on this
-		 * host, so we can assume the mutex implementation is
-		 * compatible. Important for tools like tdbdump on a still
-		 * open locking.tdb.
-		 */
-		goto check_local_settings;
-	}
-
-	/*
-	 * We got the CLEAR_IF_FIRST lock. That means the database was
-	 * potentially copied from somewhere else. The mutex implementation
-	 * might be incompatible.
-	 */
-
-	if (tdb_nest_unlock(tdb, ACTIVE_LOCK, F_WRLCK, false) == -1) {
-		/*
-		 * Should not happen
-		 */
-		TDB_LOG((tdb, TDB_DEBUG_ERROR, "tdb_mutex_open_ok: "
-			 "failed to release ACTIVE_LOCK on %s: %s\n",
-			 tdb->name, strerror(errno)));
-		return false;
-	}
-
-check_local_settings:
-
 	if (!(tdb->flags & TDB_MUTEX_LOCKING)) {
 		TDB_LOG((tdb, TDB_DEBUG_ERROR, "tdb_mutex_open_ok[%s]: "
 			 "Can use mutexes only with "
@@ -281,10 +248,10 @@ check_local_settings:
 
 	if (tdb_mutex_size(tdb) != header->mutex_size) {
 		TDB_LOG((tdb, TDB_DEBUG_ERROR, "tdb_mutex_open_ok[%s]: "
-			 "Mutex size changed from %u to %u\n.",
+			 "Mutex size changed from %"PRIu32" to %zu\n.",
 			 tdb->name,
-			 (unsigned int)header->mutex_size,
-			 (unsigned int)tdb_mutex_size(tdb)));
+			 header->mutex_size,
+			 tdb_mutex_size(tdb)));
 		return false;
 	}
 
@@ -297,7 +264,9 @@ _PUBLIC_ struct tdb_context *tdb_open_ex(const char *name, int hash_size, int td
 				tdb_hash_func hash_fn)
 {
 	int orig_errno = errno;
-	struct tdb_header header;
+	struct tdb_header header = {
+		.version = 0,
+	};
 	struct tdb_context *tdb;
 	struct stat st;
 	int rev = 0;
@@ -309,8 +278,6 @@ _PUBLIC_ struct tdb_context *tdb_open_ex(const char *name, int hash_size, int td
 	uint32_t magic1, magic2;
 	int ret;
 
-	ZERO_STRUCT(header);
-
 	if (!(tdb = (struct tdb_context *)calloc(1, sizeof *tdb))) {
 		/* Can't log this */
 		errno = ENOMEM;
@@ -418,14 +385,6 @@ _PUBLIC_ struct tdb_context *tdb_open_ex(const char *name, int hash_size, int td
 		 * the runtime check for existing tdb's comes later.
 		 */
 
-		if (!(tdb->flags & TDB_CLEAR_IF_FIRST)) {
-			TDB_LOG((tdb, TDB_DEBUG_ERROR, "tdb_open_ex: "
-				"invalid flags for %s - TDB_MUTEX_LOCKING "
-				"requires TDB_CLEAR_IF_FIRST\n", name));
-			errno = EINVAL;
-			goto fail;
-		}
-
 		if (tdb->flags & TDB_INTERNAL) {
 			TDB_LOG((tdb, TDB_DEBUG_ERROR, "tdb_open_ex: "
 				"invalid flags for %s - TDB_MUTEX_LOCKING and "
@@ -634,6 +593,30 @@ _PUBLIC_ struct tdb_context *tdb_open_ex(const char *name, int hash_size, int td
 		 * mutex locking.
 		 */
 		tdb->hdr_ofs = header.mutex_size;
+
+		if ((!(tdb_flags & TDB_CLEAR_IF_FIRST)) && (!tdb->read_only)) {
+			/*
+			 * Open an existing mutexed tdb, but without
+			 * CLEAR_IF_FIRST. We need to initialize the
+			 * mutex array and keep the CLEAR_IF_FIRST
+			 * lock locked.
+			 */
+			ret = tdb_nest_lock(tdb, ACTIVE_LOCK, F_WRLCK,
+					    TDB_LOCK_NOWAIT|TDB_LOCK_PROBE);
+			locked = (ret == 0);
+
+			if (locked) {
+				ret = tdb_mutex_init(tdb);
+				if (ret == -1) {
+					TDB_LOG((tdb,
+						 TDB_DEBUG_FATAL,
+						 "tdb_open_ex: tdb_mutex_init "
+						 "failed for ""%s: %s\n",
+						 name, strerror(errno)));
+					goto fail;
+				}
+			}