Merge lp:~stub/charms/trusty/cassandra/ensure-thrift into lp:charms/trusty/cassandra

[Stuart Bishop]

Fix Bug #1523546 by supporting Cassandra 2.2.

Note that Cassandra 2.1 is still the release series recommended for production use.

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/1719/

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/1697/

review: Needs Fixing (automated testing)

[Stuart Bishop (stub)]

Cassandra 2.2 is now the default, as it is now what upstream recommends.

Also added 3.0 support for the brave (it is not recommended for production yet).

Also rolled up all the outstanding feature and bug fixes still up for review, as the fixes are needed here and conflicts need to be resolved somewhere. Up to reviewers if they want to digest this in small chunks, or just deal with this rollup branch.

[Review Queue (review-queue)]

The results (PASS) are in and available here: http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/1750/

review: Approve (automated testing)

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/1727/

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/1755/

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

The results (PASS) are in and available here: http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/1756/

review: Approve (automated testing)

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/1732/

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

The results (PASS) are in and available here: http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/1747/

review: Approve (automated testing)

[Review Queue (review-queue)]

The results (PASS) are in and available here: http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/1816/

review: Approve (automated testing)

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/1838/

review: Needs Fixing (automated testing)

[Stuart Bishop (stub)]

review: Abstain

[Stuart Bishop (stub)]

Successful run at http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/1864/console

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/1884/

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

The results (PASS) are in and available here: http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/1864/

review: Approve (automated testing)

[Adam Israel (aisrael)]

I see we've had recent tests pass and fail. I attempted to run the tests myself, using both the local and amazon provider. In both cases, I ran into fairly consistent timeouts causing the tests to fail.

In one test run, I saw the test fail while the units were in the state of either "Waiting for permission to bootstrap" or "Waiting for agent initialization to finish". In another, the testrunner sat for hours while all units were deployed and idle. I wonder if the current timeout logic is prone to error or race conditions?

Have you seen similar timeouts in your own testing? The consistency is something I'd really like to see fixed, especially as we're preparing to roll out the cloud weather report.

Here's the log from bundletester:

http://pastebin.ubuntu.com/14569933/

Here's the output of juju status immediately after the failure:

http://pastebin.ubuntu.com/14569948/

I also have the full (~5M) output of debug-log, if that'd be useful.

review: Needs Fixing

[Stuart Bishop (stub)]

I'm mainly interested in a code review for this branch. Tests are passing locally and at http://review.juju.solutions/review/2412. Going over the diff, there is nothing in this branch that will affect reliability of the test runs apart from there being more tests and any issues will be shared with trunk.

Looking at your bundletester log, the test suite ran juju-deployer to deploy a single unit and gave up waiting an hour and a half later. The environment was not in an error state, or we wouldn't have gotten as far as timing out (the 124 return code is from the timeout(1) command that wraps juju-wait). My assumption is that one of the 2 machines needed for these tests never completed provisioning, which is an issue I see regularly with juju and the local provider (lxc sets up a unit fine, but it gets stuck in juju in 'provisioning' state either without an agent state or with an agent state but no IP address). If provisioning doesn't complete locally within a couple of minutes, it is never going to complete. The timeout is only set to 1.5 hours to handle other clouds, and I'd appreciate input on whether this timeout should be reduced (what is the worst case you see to provision 4 VMs?). The alternative explanation is somehow one of the hooks ended up in an infinite loop, but I haven't seen that happen on my recent branches as loops like 'wait for Cassandra to start responding' have timeouts. This would be obvious from the debug-log, as you would see a hook repeatedly spamming messages in a loop for most of the hour and a half before Jan 18 18:56:32.

The output of juju status is for the next set of tests that deploys three units, and shows the units happily executing their hooks and settings things up. 'waiting for agent initialization to finish' is a Juju message, where the subordinates are waiting for hooks on their primaries to complete so they can be setup. 'waiting for permission to bootstrap' is a message from the charm and is normal, where it is waiting for permission from the leader to continue Cassandra setup (we need to coordinate adding units to the Cassandra cluster, one node at a time with a two minute delay between each one).

About the only thing I can think of to do here is catch provisioning failures and loudly blame others if appropriate (or perhaps skipping tests is better?). I normally Ctrl-C, rebootstrap and try again but I think it makes sense for CI runs.

[Stuart Bishop (stub)]

https://code.launchpad.net/~stub/charms/trusty/cassandra/blamejuju/+merge/283311 for test suite improvements.

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/2202/

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

The results (PASS) are in and available here: http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/2206/

review: Approve (automated testing)

[Review Queue (review-queue)]

The results (PASS) are in and available here: http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/2183/

review: Approve (automated testing)

[Cory Johns (johnsca)]

I merged this and was then notified that it broke someone's bundle. It turns out that the two new PPAs in the install_sources option are absolutely required, so anyone who was using that option previously ends up with a broken install if they don't add them: https://paste.ubuntu.com/15198649/

I think the mandatory PPAs should not be part of that config option, and instead should be added by the code.

This has been reverted for the time being.

review: Needs Fixing

[Cory Johns (johnsca)]

Note, a work-around is to ensure that the PPAs are always included in the config value (e.g. http://pastebin.ubuntu.com/15199178/) but I think that is onerous for the user.

[Stuart Bishop (stub)]

This is works as intended, per Bug #1517803. The PPA can't be hard coded, as it means the charm requires network access to ppa.launchpad.net to install.

[Stuart Bishop (stub)]

To clarify, the PPAs are not mandatory. In our internal and customer deployments the necessary packages are placed on internal archives.

[Antoni Segura Puimedon (celebdor)]

I understand the need for the change, I just wish that charms had major and minor releases and I could pin to major instead of to minor in my bundles, to still get non-breaking changes.

@johnsca: I'll add the options to the bundle tomorrow and test it. I would appreciate if you can delay the re-merge until then.

[Cory Johns (johnsca)]

This has been merged again.

=== modified file 'Makefile'
--- Makefile	2015-11-12 05:40:51 +0000
+++ Makefile	2016-01-15 08:41:58 +0000
@@ -77,12 +77,24 @@
 	AMULET_TIMEOUT=5400 \
 	$(NOSETESTS) tests.test_integration:Test1UnitDeployment 2>&1 | ts
 	
-20test: unittest Test21Deployment
+20test: unittest Test20Deployment
 Test20Deployment: deps
 	date
 	AMULET_TIMEOUT=5400 \
 	$(NOSETESTS) tests.test_integration:Test20Deployment 2>&1 | ts
-	
+
+21test: unittest Test21Deployment
+Test21Deployment: deps
+	date
+	AMULET_TIMEOUT=5400 \
+	$(NOSETESTS) tests.test_integration:Test21Deployment 2>&1 | ts
+
+30test: unittest Test30Deployment
+Test30Deployment: deps
+	date
+	AMULET_TIMEOUT=5400 \
+	$(NOSETESTS) tests.test_integration:Test30Deployment 2>&1 | ts
+			
 3test: unittest Test3UnitDeployment
 Test3UnitDeployment: deps
 	date

=== modified file 'README.md'
--- README.md	2015-12-08 20:16:53 +0000
+++ README.md	2016-01-15 08:41:58 +0000
@@ -13,8 +13,8 @@
 
 # Editions
 
-This charm supports Apache Cassandra 2.0, Apache Cassandra 2.1, and
-Datastax Enterprise 4.8. The default is Apache Cassandra 2.1.
+This charm supports Apache Cassandra 2.0, 2.1, 2.2 & 3.0, and
+Datastax Enterprise 4.7 & 4.8. The default is Apache Cassandra 2.2.
 
 To use Apache Cassandra 2.0, specify the Apache Cassandra 2.0 archive source
 in the `install_sources` config setting when deploying.
@@ -60,14 +60,15 @@
 ## Planning
 
 - Do not attempt to store too much data per node. If you need more space,
-  add more nodes. Most workloads work best with a capacity under 500GB
+  add more nodes. Most workloads work best with a capacity under 1TB
   per node.
 
 - You need to keep 50% of your disk space free for Cassandra maintenance
   operations. If you expect your nodes to hold 500GB of data each, you
-  will need a 1TB partition.
+  will need a 1TB partition. Using non-default compaction such as
+  LeveledCompactionStrategy can lower this waste.
 
-- Much more information can be found in the [Cassandra 2.1 documentation](http://www.datastax.com/documentation/cassandra/2.1/cassandra/planning/architecturePlanningAbout_c.html)
+- Much more information can be found in the [Cassandra 2.2 documentation](http://docs.datastax.com/en/cassandra/2.2/cassandra/planning/planPlanningAbout.html)
 
 
 ## Network Access

=== modified file 'config.yaml'
--- config.yaml	2015-10-27 13:39:42 +0000
+++ config.yaml	2016-01-15 08:41:58 +0000
@@ -35,7 +35,9 @@
             If you are using Datastax Enterprise, you will need to
             override one defaults with your own username and password.
         default: |
-            - deb http://www.apache.org/dist/cassandra/debian 21x main
+            - deb http://www.apache.org/dist/cassandra/debian 22x main
+            - ppa:openjdk-r/ppa   # For OpenJDK 8
+            - ppa:stub/cassandra  # For Python driver
         #   - deb http://debian.datastax.com/community stable main
         #   DSE requires you to register and add your username/password here.
         #   - deb http://un:[e-mail]/enterprise stable main
@@ -45,7 +47,9 @@
             charm-helpers standard listing of package install source
             signing keys, corresponding to install_sources.
         default: |
+            - null  # ppa:stub/cassandra signing key added automatically.
             - null  # Apache package signing key added automatically.
+            - null  # PPA package signing key added automatically.
             # - null  # DataStack package signing key added automatically.
     http_proxy:
         type: string
@@ -65,7 +69,7 @@
         default: ""
         description: >
             URL for the private jre tar file. DSE requires
-            Oracle Java SE 7 Server JRE (eg. server-jre-7u76-linux-x64.tar.gz).
+            Oracle Java SE 8 Server JRE (eg. server-jre-8u60-linux-x64.tar.gz).
     edition:
         type: string
         default: community

=== modified file 'hooks/actions.py'
--- hooks/actions.py	2015-10-27 15:21:44 +0000
+++ hooks/actions.py	2016-01-15 08:41:58 +0000
@@ -280,7 +280,7 @@
     if helpers.get_jre() != 'oracle':
         subprocess.check_call(['update-java-alternatives',
                                '--jre-headless',
-                               '--set', 'java-1.7.0-openjdk-amd64'])
+                               '--set', 'java-1.8.0-openjdk-amd64'])
 
 
 @action
@@ -739,6 +739,7 @@
     # to control access.
     ufw.service('ssh', 'open')
     ufw.service('nrpe', 'open')  # Also NRPE for nagios checks.
+    ufw.service('rsync', 'open')  # Also rsync for data transfer and backups.
 
     # Clients need client access. These protocols are configured to
     # require authentication.

=== modified file 'hooks/helpers.py'
--- hooks/helpers.py	2015-11-20 14:21:56 +0000
+++ hooks/helpers.py	2016-01-15 08:41:58 +0000
@@ -16,6 +16,7 @@
 import configparser
 from contextlib import contextmanager
 from datetime import timedelta
+from distutils.version import LooseVersion
 import errno
 from functools import wraps
 import io
@@ -177,7 +178,7 @@
 
 def get_all_database_directories():
     config = hookenv.config()
-    return dict(
+    dirs = dict(
         data_file_directories=[get_database_directory(d)
                                for d in (config['data_file_directories'] or
                                          'data').split()],
@@ -185,6 +186,10 @@
             config['commitlog_directory'] or 'commitlog'),
         saved_caches_directory=get_database_directory(
             config['saved_caches_directory'] or 'saved_caches'))
+    if has_cassandra_version('3.0'):
+        # Not yet configurable. Make configurable with Juju native storage.
+        dirs['hints_directory'] = get_database_directory('hints')
+    return dirs
 
 
 # FOR CHARMHELPERS
@@ -303,10 +308,22 @@
 
 def get_cassandra_version():
     if get_cassandra_edition() == 'dse':
-        return '2.1' if get_package_version('dse-full') else None
+        dse_ver = get_package_version('dse-full')
+        if not dse_ver:
+            return None
+        elif LooseVersion(dse_ver) >= LooseVersion('4.7'):
+            return '2.1'
+        else:
+            return '2.0'
     return get_package_version('cassandra')
 
 
+def has_cassandra_version(minimum_ver):
+    cassandra_version = get_cassandra_version()
+    assert cassandra_version is not None, 'Cassandra package not yet installed'
+    return LooseVersion(cassandra_version) >= LooseVersion(minimum_ver)
+
+
 def get_cassandra_config_dir():
     if get_cassandra_edition() == 'dse':
         return '/etc/dse/cassandra'
@@ -354,8 +371,9 @@
         # agreement.
         pass
     else:
-        # NB. OpenJDK 8 not available in trusty.
-        packages.add('openjdk-7-jre-headless')
+        # NB. OpenJDK 8 not available in trusty. This needs to come
+        # from a PPA or some other configured source.
+        packages.add('openjdk-8-jre-headless')
 
     return packages
 
@@ -434,10 +452,11 @@
     # harmless, it causes shutil.chown() to fail.
     assert not is_cassandra_running()
     db_dirs = get_all_database_directories()
-    unpacked_db_dirs = (db_dirs['data_file_directories'] +
-                        [db_dirs['commitlog_directory']] +
-                        [db_dirs['saved_caches_directory']])
-    for db_dir in unpacked_db_dirs:
+    ensure_database_directory(db_dirs['commitlog_directory'])
+    ensure_database_directory(db_dirs['saved_caches_directory'])
+    if 'hints_directory' in db_dirs:
+        ensure_database_directory(db_dirs['hints_directory'])
+    for db_dir in db_dirs['data_file_directories']:
         ensure_database_directory(db_dir)
 
 
@@ -531,13 +550,21 @@
         hookenv.log('Creating SUPERUSER {}'.format(username))
     else:
         hookenv.log('Creating user {}'.format(username))
-    query(session,
-          'INSERT INTO system_auth.users (name, super) VALUES (%s, %s)',
-          ConsistencyLevel.ALL, (username, superuser))
-    query(session,
-          'INSERT INTO system_auth.credentials (username, salted_hash) '
-          'VALUES (%s, %s)',
-          ConsistencyLevel.ALL, (username, encrypted_password))
+    if has_cassandra_version('2.2'):
+        query(session,
+              'INSERT INTO system_auth.roles '
+              '(role, can_login, is_superuser, salted_hash) '
+              'VALUES (%s, TRUE, %s, %s)',
+              ConsistencyLevel.ALL,
+              (username, superuser, encrypted_password))
+    else:
+        query(session,
+              'INSERT INTO system_auth.users (name, super) VALUES (%s, %s)',
+              ConsistencyLevel.ALL, (username, superuser))
+        query(session,
+              'INSERT INTO system_auth.credentials (username, salted_hash) '
+              'VALUES (%s, %s)',
+              ConsistencyLevel.ALL, (username, encrypted_password))
 
 
 @logged
@@ -721,6 +748,11 @@
     # with the system_auth keyspace replication settings.
     cassandra_yaml['endpoint_snitch'] = 'GossipingPropertyFileSnitch'
 
+    # Per Bug #1523546 and CASSANDRA-9319, Thrift is disabled by default in
+    # Cassandra 2.2. Ensure it is enabled if rpc_port is non-zero.
+    if int(config['rpc_port']) > 0:
+        cassandra_yaml['start_rpc'] = True
+
     cassandra_yaml.update(overrides)
 
     write_cassandra_yaml(cassandra_yaml)
@@ -773,12 +805,20 @@
 
 
 def get_auth_keyspace_replication(session):
-    statement = dedent('''\
-        SELECT strategy_options FROM system.schema_keyspaces
-        WHERE keyspace_name='system_auth'
-        ''')
-    r = query(session, statement, ConsistencyLevel.QUORUM)
-    return json.loads(r[0][0])
+    if has_cassandra_version('3.0'):
+        statement = dedent('''\
+            SELECT replication FROM system_schema.keyspaces
+            WHERE keyspace_name='system_auth'
+            ''')
+        r = query(session, statement, ConsistencyLevel.QUORUM)
+        return dict(r[0][0])
+    else:
+        statement = dedent('''\
+            SELECT strategy_options FROM system.schema_keyspaces
+            WHERE keyspace_name='system_auth'
+            ''')
+        r = query(session, statement, ConsistencyLevel.QUORUM)
+        return json.loads(r[0][0])
 
 
 @logged

=== modified file 'hooks/hooks.py'
--- hooks/hooks.py	2015-06-18 15:47:26 +0000
+++ hooks/hooks.py	2016-01-15 08:41:58 +0000
@@ -24,8 +24,7 @@
         import cassandra  # NOQA: flake8
     except ImportError:
         packages = ['python3-bcrypt', 'python3-cassandra']
-        fetch.add_source('ppa:stub/cassandra')
-        fetch.apt_update(fatal=True)
+        fetch.configure_sources(update=True)
         fetch.apt_install(packages, fatal=True)
         import bcrypt     # NOQA: flake8
         import cassandra  # NOQA: flake8
@@ -43,8 +42,8 @@
 
     # Only useful for debugging, or perhaps have this enabled with a config
     # option?
-    ## from loglog import loglog
-    ## loglog('/var/log/cassandra/system.log', prefix='C*: ')
+    # from loglog import loglog
+    # loglog('/var/log/cassandra/system.log', prefix='C*: ')
 
     hookenv.log('*** {} Hook Start'.format(hookenv.hook_name()))
     sm = definitions.get_service_manager()

=== modified file 'tests/test_actions.py'
--- tests/test_actions.py	2015-10-27 15:21:44 +0000
+++ tests/test_actions.py	2016-01-15 08:41:58 +0000
@@ -294,7 +294,7 @@
         install_packages.assert_called_once_with(sentinel.cassandra_packages)
         check_call.assert_called_once_with(['update-java-alternatives',
                                             '--jre-headless', '--set',
-                                            'java-1.7.0-openjdk-amd64'])
+                                            'java-1.8.0-openjdk-amd64'])
 
     @patch('subprocess.check_call')
     @patch('helpers.get_jre')
@@ -857,6 +857,7 @@
         # SSH and the client protocol ports are always fully open.
         ufw.service.assert_has_calls([call('ssh', 'open'),
                                       call('nrpe', 'open'),
+                                      call('rsync', 'open'),
                                       call(9042, 'open'),
                                       call(9160, 'open')])
 
@@ -905,11 +906,13 @@
                                            call('1.1.0.2', 'any', 7002)],
                                           any_order=True)
 
+    @patch('helpers.get_cassandra_version')
     @patch('charmhelpers.core.host.write_file')
     @patch('charmhelpers.contrib.charmsupport.nrpe.NRPE')
     @patch('helpers.local_plugins_dir')
     def test_nrpe_external_master_relation(self, local_plugins_dir, nrpe,
-                                           write_file):
+                                           write_file, cassandra_version):
+        cassandra_version.return_value = '2.2'
         # The fake charm_dir() needs populating.
         plugin_src_dir = os.path.join(os.path.dirname(__file__),
                                       os.pardir, 'files')
@@ -947,11 +950,13 @@
 
             nrpe().write.assert_called_once_with()
 
+    @patch('helpers.get_cassandra_version')
     @patch('charmhelpers.core.host.write_file')
     @patch('os.path.exists')
     @patch('charmhelpers.contrib.charmsupport.nrpe.NRPE')
-    def test_nrpe_external_master_relation_no_local(self, nrpe,
-                                                    exists, write_file):
+    def test_nrpe_external_master_relation_no_local(self, nrpe, exists,
+                                                    write_file, ver):
+        ver.return_value = '2.2'
         # If the local plugins directory doesn't exist, we don't attempt
         # to write files to it. Wait until the subordinate has set it
         # up.
@@ -959,9 +964,12 @@
         actions.nrpe_external_master_relation('')
         self.assertFalse(write_file.called)
 
+    @patch('helpers.get_cassandra_version')
     @patch('os.path.exists')
     @patch('charmhelpers.contrib.charmsupport.nrpe.NRPE')
-    def test_nrpe_external_master_relation_disable_heapchk(self, nrpe, exists):
+    def test_nrpe_external_master_relation_disable_heapchk(self, nrpe,
+                                                           exists, ver):
+        ver.return_value = '2.2'
         exists.return_value = False
 
         # Disable our checks
@@ -980,9 +988,12 @@
             call(shortname='cassandra_disk_var_lib_cassandra_commitlog',
                  description=ANY, check_cmd=ANY)], any_order=True)
 
+    @patch('helpers.get_cassandra_version')
     @patch('os.path.exists')
     @patch('charmhelpers.contrib.charmsupport.nrpe.NRPE')
-    def test_nrpe_external_master_relation_disable_diskchk(self, nrpe, exists):
+    def test_nrpe_external_master_relation_disable_diskchk(self, nrpe,
+                                                           exists, ver):
+        ver.return_value = '2.2'
         exists.return_value = False
 
         # Disable our checks

=== modified file 'tests/test_helpers.py'
--- tests/test_helpers.py	2015-11-20 14:21:56 +0000
+++ tests/test_helpers.py	2016-01-15 08:41:58 +0000
@@ -220,8 +220,10 @@
         # Absolute paths are absolute and passed through unmolested.
         self.assertEqual(helpers.get_database_directory('/bar'), '/bar')
 
+    @patch('helpers.get_cassandra_version')
     @patch('relations.StorageRelation')
-    def test_get_all_database_directories(self, storage_relation):
+    def test_get_all_database_directories(self, storage_relation, ver):
+        ver.return_value = '2.2'
         storage_relation().mountpoint = '/s'
         self.assertDictEqual(
             helpers.get_all_database_directories(),
@@ -229,6 +231,18 @@
                  commitlog_directory='/s/cassandra/commitlog',
                  saved_caches_directory='/s/cassandra/saved_caches'))
 
+    @patch('helpers.get_cassandra_version')
+    @patch('relations.StorageRelation')
+    def test_get_all_database_directories_30(self, storage_relation, ver):
+        ver.return_value = '3.0'
+        storage_relation().mountpoint = '/s'
+        self.assertDictEqual(
+            helpers.get_all_database_directories(),
+            dict(data_file_directories=['/s/cassandra/data'],
+                 commitlog_directory='/s/cassandra/commitlog',
+                 saved_caches_directory='/s/cassandra/saved_caches',
+                 hints_directory='/s/cassandra/hints'))
+
     @patch('helpers.recursive_chown')
     @patch('charmhelpers.core.host.mkdir')
     @patch('helpers.get_database_directory')
@@ -458,7 +472,7 @@
         # Default
         self.assertSetEqual(helpers.get_cassandra_packages(),
                             set(['cassandra', 'ntp', 'run-one',
-                                 'netcat', 'openjdk-7-jre-headless']))
+                                 'netcat', 'openjdk-8-jre-headless']))
 
     def test_get_cassandra_packages_oracle_jre(self):
         # Oracle JRE
@@ -731,8 +745,10 @@
                           sentinel.consistency, sentinel.args)
         self.assertEqual(session.execute.call_count, 4)
 
+    @patch('helpers.get_cassandra_version')
     @patch('helpers.query')
-    def test_ensure_user(self, query):
+    def test_ensure_user(self, query, ver):
+        ver.return_value = '2.1'
         helpers.ensure_user(sentinel.session,
                             sentinel.username, sentinel.pwhash,
                             superuser=sentinel.supflag)
@@ -746,6 +762,21 @@
                  ConsistencyLevel.ALL,
                  (sentinel.username, sentinel.pwhash))])
 
+    @patch('helpers.get_cassandra_version')
+    @patch('helpers.query')
+    def test_ensure_user_22(self, query, ver):
+        ver.return_value = '2.2'
+        helpers.ensure_user(sentinel.session,
+                            sentinel.username, sentinel.pwhash,
+                            superuser=sentinel.supflag)
+        query.assert_called_once_with(sentinel.session,
+                                      'INSERT INTO system_auth.roles (role, '
+                                      'can_login, is_superuser, salted_hash) '
+                                      'VALUES (%s, TRUE, %s, %s)',
+                                      ConsistencyLevel.ALL,
+                                      (sentinel.username, sentinel.supflag,
+                                       sentinel.pwhash))
+
     @patch('helpers.ensure_user')
     @patch('helpers.encrypt_password')
     @patch('helpers.nodetool')
@@ -991,6 +1022,79 @@
                 stream_throughput_outbound_megabits_per_sec: 200
                 tombstone_warn_threshold: 1000
                 tombstone_failure_threshold: 100000
+                start_rpc: true
+                ''')
+            self.maxDiff = None
+            self.assertEqual(yaml.safe_load(new_config),
+                             yaml.safe_load(expected_config))
+
+            # Confirm we can use an explicit cluster_name too.
+            write_file.reset_mock()
+            hookenv.config()['cluster_name'] = 'fubar'
+            helpers.configure_cassandra_yaml()
+            new_config = write_file.call_args[0][1]
+            self.assertEqual(yaml.safe_load(new_config)['cluster_name'],
+                             'fubar')
+
+    @patch('helpers.get_cassandra_version')
+    @patch('helpers.get_cassandra_yaml_file')
+    @patch('helpers.get_seed_ips')
+    @patch('charmhelpers.core.host.write_file')
+    def test_configure_cassandra_yaml_22(self, write_file, seed_ips, yaml_file,
+                                         get_cassandra_version):
+        get_cassandra_version.return_value = '2.0'
+        hookenv.config().update(dict(num_tokens=128,
+                                     cluster_name='test_cluster_name',
+                                     partitioner='test_partitioner'))
+
+        seed_ips.return_value = ['10.20.0.1', '10.20.0.2', '10.20.0.3']
+
+        existing_config = '''
+            seed_provider:
+                - class_name: blah.SimpleSeedProvider
+                  parameters:
+                      - seeds: 127.0.0.1  # Comma separated list.
+            start_rpc: false  # Defaults to False starting 2.2
+            '''
+
+        with tempfile.TemporaryDirectory() as tmpdir:
+            yaml_config = os.path.join(tmpdir, 'c.yaml')
+            yaml_file.return_value = yaml_config
+            with open(yaml_config, 'w', encoding='UTF-8') as f:
+                f.write(existing_config)
+
+            helpers.configure_cassandra_yaml()
+
+            self.assertEqual(write_file.call_count, 2)
+            new_config = write_file.call_args[0][1]
+
+            expected_config = dedent('''\
+                start_rpc: true
+                cluster_name: test_cluster_name
+                authenticator: PasswordAuthenticator
+                num_tokens: 128
+                partitioner: test_partitioner
+                listen_address: 10.20.0.1
+                rpc_address: 0.0.0.0
+                rpc_port: 9160
+                native_transport_port: 9042
+                storage_port: 7000
+                ssl_storage_port: 7001
+                authorizer: AllowAllAuthorizer
+                seed_provider:
+                    - class_name: blah.SimpleSeedProvider
+                      parameters:
+                        # No whitespace in seeds is important.
+                        - seeds: '10.20.0.1,10.20.0.2,10.20.0.3'
+                endpoint_snitch: GossipingPropertyFileSnitch
+                data_file_directories:
+                    - /var/lib/cassandra/data
+                commitlog_directory: /var/lib/cassandra/commitlog
+                saved_caches_directory: /var/lib/cassandra/saved_caches
+                compaction_throughput_mb_per_sec: 16
+                stream_throughput_outbound_megabits_per_sec: 200
+                tombstone_warn_threshold: 1000
+                tombstone_failure_threshold: 100000
                 ''')
             self.maxDiff = None
             self.assertEqual(yaml.safe_load(new_config),
@@ -1043,6 +1147,7 @@
                 listen_address: 10.20.0.1
                 rpc_address: 0.0.0.0
                 broadcast_rpc_address: 10.30.0.1
+                start_rpc: true
                 rpc_port: 9160
                 native_transport_port: 9042
                 storage_port: 7000
@@ -1203,8 +1308,10 @@
         # Weird errors are reraised.
         self.assertRaises(RuntimeError, helpers.is_cassandra_running)
 
+    @patch('helpers.get_cassandra_version')
     @patch('helpers.query')
-    def test_get_auth_keyspace_replication(self, query):
+    def test_get_auth_keyspace_replication(self, query, ver):
+        ver.return_value = '2.2'
         query.return_value = [('{"json": true}',)]
         settings = helpers.get_auth_keyspace_replication(sentinel.session)
         self.assertDictEqual(settings, dict(json=True))
@@ -1214,6 +1321,19 @@
                 WHERE keyspace_name='system_auth'
                 '''), ConsistencyLevel.QUORUM)
 
+    @patch('helpers.get_cassandra_version')
+    @patch('helpers.query')
+    def test_get_auth_keyspace_replication_30(self, query, ver):
+        ver.return_value = '3.0'
+        query.return_value = [({"json": True},)]  # Decoded under 3.0
+        settings = helpers.get_auth_keyspace_replication(sentinel.session)
+        self.assertDictEqual(settings, dict(json=True))
+        query.assert_called_once_with(
+            sentinel.session, dedent('''\
+                SELECT replication FROM system_schema.keyspaces
+                WHERE keyspace_name='system_auth'
+                '''), ConsistencyLevel.QUORUM)
+
     @patch('helpers.status_set')
     @patch('charmhelpers.core.hookenv.status_get')
     @patch('helpers.query')

=== modified file 'tests/test_integration.py'
--- tests/test_integration.py	2015-11-13 11:40:06 +0000
+++ tests/test_integration.py	2016-01-15 08:41:58 +0000
@@ -532,8 +532,9 @@
     rf = 1
     test_config = dict(
         edition='DSE',  # Forces Oracle JRE
-        install_sources=yaml.safe_dump([os.environ.get('DSE_SOURCE')]),
-        install_keys=yaml.safe_dump([None]),
+        install_sources=yaml.safe_dump([os.environ.get('DSE_SOURCE'),
+                                        'ppa:stub/cassandra']),
+        install_keys=yaml.safe_dump([None, None]),
         private_jre_url=get_jre_url())
 
     @classmethod
@@ -573,8 +574,36 @@
     test_config = dict(
         edition='community',
         install_sources=yaml.safe_dump([
+            'ppa:stub/cassandra',
+            'ppa:openjdk-r/ppa',
             'deb http://www.apache.org/dist/cassandra/debian 20x main']),
-        install_keys=yaml.safe_dump([None]))
+        install_keys=yaml.safe_dump([None, None, None]))
+
+
+class Test21Deployment(Test1UnitDeployment):
+    """Tests run on a single node Apache Cassandra 2.1 cluster.
+    """
+    rf = 1
+    test_config = dict(
+        edition='community',
+        install_sources=yaml.safe_dump([
+            'ppa:stub/cassandra',
+            'ppa:openjdk-r/ppa',
+            'deb http://www.apache.org/dist/cassandra/debian 21x main']),
+        install_keys=yaml.safe_dump([None, None, None]))
+
+
+class Test30Deployment(Test1UnitDeployment):
+    """Tests run on a single node Apache Cassandra 3.0 cluster.
+    """
+    rf = 1
+    test_config = dict(
+        edition='community',
+        install_sources=yaml.safe_dump([
+            'ppa:stub/cassandra',
+            'ppa:openjdk-r/ppa',
+            'deb http://www.apache.org/dist/cassandra/debian 30x main']),
+        install_keys=yaml.safe_dump([None, None, None]))
 
 
 # Bug #1417097 means we need to monkey patch Amulet for now.

=== modified file 'tests/tests.yaml'
--- tests/tests.yaml	2015-10-27 15:21:44 +0000
+++ tests/tests.yaml	2016-01-15 08:41:58 +0000
@@ -10,4 +10,6 @@
     - Test1UnitDeployment
     - Test3UnitDeployment
     - Test20Deployment
+    - Test21Deployment
+    - Test30Deployment
     - TestAllowAllAuthenticatorDeployment