Merge lp:~akuzminsky/twindb-agent/bug-1410509 into lp:twindb-agent

[Aleksandr Kuzminsky]

Fixes Bug #1410509

[lp:~akuzminsky/twindb-agent/bug-1410509]

5. By root <root@ubuntu-lucid> on 2015-01-18

Make the packager's full name and email match to ones from the packager GPG key

6. By root <root@ubuntu-lucid> on 2015-01-18

fix arithmetic expansion error in shell

7. By root <root@ubuntu-lucid> on 2015-01-18

Use a template @@CODENAME@@ in changelog. It is replaced by a build script

8. By root <root@ubuntu-lucid> on 2015-01-19

sh-compatible increment is fixed again

9. By root <root@ubuntu-lucid> on 2015-01-19

show registration hint only if the agent isn't registred

10. By root <root@ubuntu-lucid> on 2015-01-19

On lucid sys.path doen't include path where mysql-connector-python installs its modules

11. By root <root@ubuntu-lucid> on 2015-01-19

Include OS codename into debian package version

12. By root <root@ubuntu-lucid> on 2015-01-19

package version ++

13. By root <root@ubuntu-lucid> on 2015-01-19

Use UNRELEASED as target name, otherwise dch throws and error

14. By root <root@ubuntu-lucid> on 2015-01-19

Version ++

15. By root <root@ubuntu-lucid> on 2015-01-19

updated changelog

[Ovais Tariq (ovais-tariq)]

The Makefile target "build-deb" builds the package in the .build directory. But the "build-rpm" package builds the rpm in the directory /root/rpmbuild/ Should we use identical directories for both targets?

[lp:~akuzminsky/twindb-agent/bug-1410509]

16. By Aleksandr Kuzminsky on 2015-01-20

Build rpm packages in local .build directory instead of /root/rpmbuild

[Ovais Tariq (ovais-tariq)]

Have the dependencies for the deb package not been specified?

I am able to install the package using:
dpkg -i /tmp/twindb_src/.build/twindb_0.1.32-1.wheezy_all.deb

Although the dependencies are not installed on the system. It should have failed.
root@server55-dev-debian-74:~# dpkg -l | grep twindb
ii twindb 0.1.32-1.wheezy all This is a TwinDB agent - a tool that takes backups from MySQL server

When I try to start twindb agent it fails which is expected:
root@server55-dev-debian-74:~# twindb --help
Traceback (most recent call last):
  File "/usr/bin/twindb", line 27, in <module>
    import mysql.connector
ImportError: No module named mysql.connector

And then uninstalling and purging fails because pre-removal script is failing:
root@server55-dev-debian-74:~# apt-get remove twindb
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  mysql-server percona-xtrabackup
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
  twindb
0 upgraded, 0 newly installed, 1 to remove and 68 not upgraded.
After this operation, 129 kB disk space will be freed.
Do you want to continue [Y/n]? y
(Reading database ... 72372 files and directories currently installed.)
Removing twindb ...
Pid file(/var/run/twindb.pid) doesn't exist.
invoke-rc.d: initscript twindb, action "stop" failed.
dpkg: error processing twindb (--remove):
 subprocess installed pre-removal script returned error exit status 1
Traceback (most recent call last):
  File "/usr/bin/twindb", line 27, in <module>
    import mysql.connector
ImportError: No module named mysql.connector
================================================================================
TwinDB client is installed.
Now you need to register this server. To do so run:

twindb --register <registration code>

To obtain the registration code follow link https://console.twindb.com/?get_code
================================================================================
Starting TwinDB agent ... OK
Errors were encountered while processing:
 twindb
E: Sub-process /usr/bin/dpkg returned an error code (1)

root@server55-dev-debian-74:~# dpkg -r twindb
(Reading database ... 72372 files and directories currently installed.)
Removing twindb ...
Pid file(/var/run/twindb.pid) doesn't exist.
invoke-rc.d: initscript twindb, action "stop" failed.
dpkg: error processing twindb (--remove):
 subprocess installed pre-removal script returned error exit status 1
Traceback (most recent call last):
  File "/usr/bin/twindb", line 27, in <module>
    import mysql.connector
ImportError: No module named mysql.connector
================================================================================
TwinDB client is installed.
Now you need to register this server. To do so run:

twindb --register <registration code>

To obtain the registration code follow link https://console.twindb.com/?get_code
================================================================================
Starting TwinDB agent ... OK
Errors were encountered while processing:
 twindb

...

review: Needs Fixing

[lp:~akuzminsky/twindb-agent/bug-1410509]

17. By root <root@ubuntu-lucid> on 2015-01-19

* add mysql-connector-python as dependency in Debian packages
* Number of small fixes in twindb init script for Debian

[Ovais Tariq (ovais-tariq)]

The deb package gets installed but following errors are reported:

Setting up twindb (0.1.32-1.wheezy) ...
       2015-01-21 04:05:02,893: twindb: ERROR: is_registered():760: Failed to decode
       {"success":false,"errors":{"msg":"Ignoring request without type","msg_debug":null}}

The complete installation log is as follows:
       [2015-01-21T04:04:47+00:00] INFO: bash[build_twindb_deb] sending run action to bash[install-twindb-agent-deb] (immediate)
         * bash[install-twindb-agent-deb] action run[2015-01-21T04:04:47+00:00] INFO: Processing bash[install-twindb-agent-deb] action run (twindb-agent::dev line 100)
Reading package lists... Done
Building dependency tree... 0%
Building dependency tree
Reading state information... 0%
Reading state information... Done
   lding data structures... 0%
Building data structures... Done
   lding data structures... 0%
Building data structures... Done
0% [Working]
Get:1 http://deb.twindb.com/ wheezy/main mysql-connector-python all 2.0.2-1debian7.6 [111 kB]
Fetched 111 kB in 0s (0 B/s)
       Selecting previously unselected package mysql-server.
(Reading database ... 72352 files and directories currently installed.)
       Unpacking mysql-server (from .../mysql-server_5.5.40-0+wheezy1_all.deb) ...
       Selecting previously unselected package mysql-connector-python.
       Unpacking mysql-connector-python (from .../mysql-connector-python_2.0.2-1debian7.6_all.deb) ...
       Selecting previously unselected package percona-xtrabackup.
       Unpacking percona-xtrabackup (from .../percona-xtrabackup_2.2.8-5059-1.wheezy_amd64.deb) ...
       Setting up mysql-server (5.5.40-0+wheezy1) ...
       Setting up mysql-connector-python (2.0.2-1debian7.6) ...
       Setting up percona-xtrabackup (2.2.8-5059-1.wheezy) ...
       Selecting previously unselected package twindb.
(Reading database ... 72492 files and directories currently installed.)
       Unpacking twindb (from .../twindb_0.1.32-1.wheezy_all.deb) ...
       Setting up twindb (0.1.32-1.wheezy) ...
       2015-01-21 04:05:02,893: twindb: ERROR: is_registered():760: Failed to decode
       {"success":false,"errors":{"msg":"Ignoring request without type","msg_debug":null}}
       ================================================================================
       TwinDB client is installed.
       Now you need to register this server. To do so run:

       twindb --register <registration code>

       To obtain the registration code follow link https://console.twindb.com/?get_code
       ================================================================================
Starting TwinDB agent ... OK
       Processing triggers for man-db ...
Fetched 111 kB in 0s (0 B/s)

[Ovais Tariq (ovais-tariq)]

The twindb agent fails to register though. Following is the complete log of the test run:

  1) twindb agent is able to register Command "/usr/bin/twindb --register 0f6714f735a06a488ec9279f59d0a3f2" exit_status should eq 0
     Failure/Error: its(:exit_status) { should eq 0 }

       expected: 0
            got: 1

              (compared using ==)
              /bin/sh -c /usr/bin/twindb\ --register\ 0f6714f735a06a488ec9279f59d0a3f2
              2015-01-21 04:06:49,112: twindb: INFO: check_gpg():241: Looks like GPG never ran. Initializing GPG configuration
       2015-01-21 04:06:54,341: twindb: INFO: install_api_pub_key():205: Installing twindb public key
       2015-01-21 04:06:54,537: twindb: INFO: install_api_pub_key():214: Twindb public key successfully installed
       gpg: key A8AD9889: public key "TwinDB (TwinDB API Portal) <email address hidden>" imported
       gpg: Total number processed: 1
       gpg: imported: 1 (RSA: 1)
       2015-01-21 04:06:54,743: twindb: INFO: gen_gpg_keypair():336: Generating GPG keys pair for <email address hidden>
       2015-01-21 04:06:54,949: twindb: INFO: gen_gpg_keypair():337: It may take really, really long time. Please be patient.
       gpg: Generating a standard key
       ...+++++
       ....+++++
       .............+++++
       .+++++
       gpg: key 3C43C384 marked as ultimately trusted
       gpg: done
       2015-01-21 04:06:56,481: twindb: INFO: action_handler_register():1012: Registering TwinDB agent with code 0f6714f735a06a488ec9279f59d0a3f2
       2015-01-21 04:06:56,801: twindb: INFO: action_handler_register():1013: The agent needs to generate cryptographically strong keys.
       2015-01-21 04:06:57,119: twindb: INFO: action_handler_register():1014: It may take really, really long time. Please be patient.
       2015-01-21 04:06:57,431: twindb: INFO: action_handler_register():1019: Reading GPG public key of <email address hidden>.
       2015-01-21 04:06:57,746: twindb: INFO: action_handler_register():1031: Generating SSH keys pair.
       Generating public/private rsa key pair.
       Created directory '/root/.ssh'.
       Your identification has been saved in /root/.ssh/twindb.key.
       Your public key has been saved in /root/.ssh/twindb.key.pub.
       The key fingerprint is:
       cc:b8:87:b7:06:fd:f3:8b:78:e7:7d:6a:12:6d:2e:a4 root@server55-dev-debian-74
       The key's randomart image is:
       +--[ RSA 2048]----+
       | |
       | |
       | |
       | + |
       | ..S . |
       | .o. o o |
       | o.o. o + |
       | o.oE.+.o .|
       | .o..=+=oo |
       +-----------------+
       2015-01-21 04:06:58,170: twindb: INFO: action_handler_register():1037: Reading SSH public key from /root/.ssh/twindb.key.pub.
       Traceback (most recent call last):
         File "/usr/bin/twindb", line 2283, in <module>
           main()
         File "/usr/bin/twindb", line 2257, in main
           action_handler_register(regcode)
         File "/usr/bin/twindb", line 104...

[Ovais Tariq (ovais-tariq)]

twindb --is-registered also appears to be broken:

         2) twindb agent is able to register Command "/usr/bin/twindb --is-registered" stdout should match /YES/
            Failure/Error: its(:stdout) { should match /YES/ }
              expected "2015-01-21 04:07:04,075: twindb: ERROR: is_registered():760: Failed to decode \n{\"success\":false,\"errors\":{\"msg\":\"Ignoring request without type\",\"msg_debug\":null}}\n" to match /YES/

[Ovais Tariq (ovais-tariq)]

review: Needs Fixing

[lp:~akuzminsky/twindb-agent/bug-1410509]

18. By root <email address hidden> on 2015-01-21

Don't send debug logs to dispatcher, otherwise it's impossible to read debug output

19. By root <root@ubuntu-lucid> on 2015-01-20

pretty debug messages in get_response()

20. By Aleksandr Kuzminsky on 2015-01-23

* Debian standard bump
* Removed mysql-server and mysql-client dependencies. I'm not sure all MySQL vendors provide mysql-server. Besides, if mysql-server-5.5 is installed it will try to install mysql-server.
* Use get_mysql_connection() to manage connection to MySQL
* Pretty print JSON response from the dispatcher and ignore empty responses
* Make debug output nicer and more readable

[Ovais Tariq (ovais-tariq)]

It fails when agent interacts with the production dispatcher as well.

/usr/bin/twindb -g --dispatcher dispatcher.twindb.com --is-registered
       2015-01-23 18:07:07,432: twindb: DEBUG: get_response():485: Enter get_response(uri=api.php)
       2015-01-23 18:07:07,514: twindb: DEBUG: get_response():495: Sending to dispatcher.twindb.com: {
           "params": {
        "server_id": "1f84dfce-41b0-4a84-a661-add8b4067ee5"

           },
           "type": "is_registered"
       }
       2015-01-23 18:07:07,521: twindb: DEBUG: encrypt():386: Public key of <email address hidden> is not installed.
       2015-01-23 18:07:07,522: twindb: DEBUG: encrypt():387: Will not encrypt message
       2015-01-23 18:07:07,616: twindb: DEBUG: get_response():521: Response from http://dispatcher.twindb.com//api.php : {
           "errors": {
        "msg": "Ignoring request without type",
        "msg_debug": null
           },
           "success": false
       }
       2015-01-23 18:07:07,616: twindb: ERROR: is_registered():759: Failed to decode
       {"success":false,"errors":{"msg":"Ignoring request without type","msg_debug":null}}
       2015-01-23 18:07:07,618: twindb: DEBUG: exit_on_error():2005: Traceback (most recent call last):
         File "/usr/bin/twindb", line 750, in is_registered
           msg_enc = response_body_decoded["response"]
       KeyError: u'response'

/usr/bin/twindb -g --dispatcher dispatcher.twindb.com --register 0f6714f735a06a488ec9279f59d0a3f2
       2015-01-23 18:07:07,775: twindb: INFO: check_gpg():243: Looks like GPG never ran. Initializing GPG configuration
       2015-01-23 18:07:07,776: twindb: INFO: install_api_pub_key():207: Installing twindb public key
       2015-01-23 18:07:07,789: twindb: INFO: install_api_pub_key():216: Twindb public key successfully installed
       gpg: keyring `/root/.gnupg//secring.gpg' created
       gpg: keyring `/root/.gnupg//pubring.gpg' created
       gpg: /root/.gnupg//trustdb.gpg: trustdb created
       gpg: key A8AD9889: public key "TwinDB (TwinDB API Portal) <email address hidden>" imported
       2015-01-23 18:07:07,797: twindb: INFO: gen_gpg_keypair():338: Generating GPG keys pair for <email address hidden>
       2015-01-23 18:07:07,798: twindb: INFO: gen_gpg_keypair():339: It may take really, really long time. Please be patient.
       gpg: Total number processed: 1
       gpg: imported: 1 (RSA: 1)
       gpg: Generating a standard key
       gpg: key 94B30215 marked as ultimately trusted
       gpg: done
       2015-01-23 18:07:08,334: twindb: DEBUG: check_env():271: Enviroment is OK
       2015-01-23 18:07:08,335: twindb: INFO: action_handler_register():1011: Registering TwinDB agent with code 0f6714f735a06a488ec9279f59d0a3f2

       2015-01-23 18:07:08,335: twindb: INFO: action_handler_register():1012: The agent needs to generate cryptographically strong keys.
       2015-01-23 18:07:08,336: twindb: INFO: action_handler_register():1013: It may take really, really long time. Please be patient.
       2015-01-23 18:07:08,336: twindb: INFO: action_handler_register():1018: Reading GPG public key of 1f84dfce-41b0-4a84-a661-add8b4067ee5@twin...

[Ovais Tariq (ovais-tariq)]

Above test was done on CentOS 6.5

[lp:~akuzminsky/twindb-agent/bug-1410509]

21. By Aleksandr Kuzminsky on 2015-01-23

add lsof as dependency

22. By Aleksandr Kuzminsky on 2015-01-23

add lsof as dependency in rpm spec file too

23. By Aleksandr Kuzminsky on 2015-01-23

Connect to MySQL with option_files only if at least one options file was found

24. By Aleksandr Kuzminsky on 2015-01-23

Specify socket for innobackupex

25. By Aleksandr Kuzminsky on 2015-01-24

Select only processes that fully match mysqld

26. By Aleksandr Kuzminsky on 2015-01-24

Parse all command line arguments before calling is_registered()

[Ovais Tariq (ovais-tariq)]

All tests pass.

review: Approve

=== modified file 'Makefile'
--- Makefile	2015-01-16 07:41:06 +0000
+++ Makefile	2015-01-24 21:00:42 +0000
@@ -1,12 +1,16 @@
 # TwinDB agent Makefile
 
 spec = support/rpm/twindb.spec
-client_version = "`grep Version: ${spec} | awk '{ print $$2}' | head -1`"
-client_release = "`grep Release: ${spec} | awk '{ print $$2}' | head -1`"
+client_version = 0.1.32
+client_release = 1
 rh_release = "$$(r=`lsb_release -sr | cut -d . -f1`; if [ \"$$r\" = \"n/a\" ] ; then echo latest; else echo $$r; fi)"
+rpmmacros = /usr/lib/rpm/macros:/usr/lib/rpm/redhat/macros:/etc/rpm/macros:support/rpm/rpmmacros
+deb_release = "$$(lsb_release -sc)"
 build_dir = .build
 src_dir = twindb-${client_version}
-top_dir = /root/rpmbuild
+pwd := $(shell pwd)
+top_dir = ${pwd}/${build_dir}/rpmbuild
+deb_packages = build-essential devscripts debhelper
 
 .PHONY: help rpm sign rpmmacros
 
@@ -40,13 +44,21 @@
 
 clean:
 	rm -f twindb twindb.8.gz
+	rm -f support/rpm/twindb.spec
+	rm -rf .build
 
-build-rpm: rpmmacros
+archive:
 	rm -rf "${build_dir}"
 	mkdir -p "${build_dir}/${src_dir}"
 	cp -R * "${build_dir}/${src_dir}"
-	tar zcf "${top_dir}/SOURCES/${src_dir}.tar.gz" -C "${build_dir}" "${src_dir}"
-	rpmbuild -ba ${spec}
+	tar zcf "${build_dir}/${src_dir}.tar.gz" -C "${build_dir}" "${src_dir}"
+
+# RPM stuff
+build-rpm: rpmmacros spec archive
+	mkdir -p "${top_dir}/SOURCES/"
+	mkdir -p "${top_dir}/BUILD/"
+	mv "${build_dir}/${src_dir}.tar.gz" "${top_dir}/SOURCES/${src_dir}.tar.gz"
+	rpmbuild --macros=${rpmmacros} -D '%_topdir ${top_dir}' -ba ${spec}
 
 sign-rpm:
 	rpm --addsign ${top_dir}/RPMS/noarch/twindb-${client_version}-${client_release}.noarch.rpm
@@ -59,3 +71,79 @@
 
 rpmmacros:
 	if ! test -f ~/.rpmmacros ; then cp support/rpm/rpmmacros ~/.rpmmacros; fi
+
+spec:
+	sed -e 's/@@VERSION@@/${client_version}/' -e 's/@@RELEASE@@/${client_release}/' \
+		support/rpm/twindb.spec.template > support/rpm/twindb.spec
+
+# Debian stuff
+deb-dependencies:
+	@echo "Checking dependencies"
+	@for p in ${deb_packages}; \
+    do echo -n "$$p ... " ; \
+        if test -z "`dpkg -l | grep $$p`"; \
+        then \
+            echo "$$p ... NOT installed"; \
+            apt-get -y install $$p; \
+        else \
+            echo "installed"; \
+        fi ; \
+    done
+
+build-deb: deb-dependencies archive
+	mv "${build_dir}/${src_dir}.tar.gz" "${build_dir}/twindb_${client_version}.orig.tar.gz"
+	cp -R support/deb/debian/ "${build_dir}/${src_dir}"
+	rm -f "${build_dir}/${src_dir}/debian/changelog"
+	@echo "Generating changelog"
+	@export DEBEMAIL="TwinDB Packager (TwinDB packager key) <[e-mail]>" ; \
+	export version=${client_version}-${client_release} ; \
+	export tags="`bzr tags | sort -nk2 | cut -d" " -f1`" ; \
+	export last_tag=`bzr tags | sort -nrk2 | head -1 | cut -d" " -f1` ; \
+	export prev_tag="" ; \
+	if [ "$$last_tag" != "$$version" ] ; \
+	then \
+		echo "Error: Last tag $$last_tag doesn't match current version $$version in Makefile" ; \
+		echo "Please create tag $$version:" ; \
+		echo "bzr tag $$version" ; \
+		exit 1 ; \
+	fi ; \
+	cd support/deb/ ; \
+	for tag in $$tags ; \
+	do \
+        echo -n "Version: $$tag. Revisions " ; \
+		if test -z "$$prev_tag" ; \
+		then \
+			start_revno=1 ; \
+		else \
+			export start_revno=`bzr tags | grep $$prev_tag | awk '{ print $$2 }'` ; \
+			export start_revno=$$(( $$start_revno + 1 )); \
+		fi; \
+        echo -n "$$start_revno .. " ; \
+		export end_revno=`bzr tags | grep $$tag | awk '{ print $$2 }'` ; \
+        echo "$$end_revno" ; \
+		if ! test -f debian/changelog || test -z "`grep "twindb .*$$tag.*UNRELEASED; urgency=" debian/changelog`"; \
+		then \
+            echo "Adding new version $$tag.UNRELEASED to the changelog" ; \
+			if ! test -f debian/changelog; then export opt_create='--create' ; else export opt_create='' ; fi ; \
+			echo "dch $$opt_create -v $$tag.UNRELEASED --package twindb --distribution UNRELEASED "New version $$tag"" ; \
+			dch $$opt_create -v $$tag.UNRELEASED --package twindb --distribution UNRELEASED "New version $$tag" ; \
+			for revno in `seq $$end_revno -1 $$start_revno`; \
+			do \
+				export log_msg="`bzr log -r $$revno --line`" ; \
+				echo "Adding log entry to version $$tag: $$log_msg" ; \
+				dch --append "$$log_msg" ; \
+			done ; \
+		fi ; \
+		export prev_tag=$$tag ; \
+	done ; \
+	cd - > /dev/null;
+	export distr=`lsb_release -sc` ; \
+	sed s/UNRELEASED/$$distr/g support/deb/debian/changelog > "${build_dir}/${src_dir}/debian/changelog"
+	cd "${build_dir}/${src_dir}" && debuild -us -uc
+
+sign-deb: 
+	cd "${build_dir}"; debsign [e-mail] *.changes
+
+upload-deb:
+	ssh [e-mail] "mkdir -p ~/deb/`lsb_release -sc`"
+	scp ${build_dir}/*.changes ${build_dir}/*.dsc ${build_dir}/*.tar.gz ${build_dir}/*.deb [e-mail]:~/deb/`lsb_release -sc`

=== added file 'support/deb/debian/changelog'
--- support/deb/debian/changelog	1970-01-01 00:00:00 +0000
+++ support/deb/debian/changelog	2015-01-24 21:00:42 +0000
@@ -0,0 +1,36 @@
+twindb (0.1.32-1.UNRELEASED) UNRELEASED; urgency=low
+
+  * New version 0.1.32-1
+  * 14: root 2015-01-19 {0.1.32-1} Version ++
+  * 13: root 2015-01-19 Use UNRELEASED as target name, otherwise dch
+    throws and error
+
+ -- TwinDB Packager (TwinDB packager key) <[e-mail]>  Mon, 19 Jan 2015 05:08:48 -0800
+
+twindb (0.1.31-2.UNRELEASED) UNRELEASED; urgency=low
+
+  * New version 0.1.31-2
+  * 12: root 2015-01-19 {0.1.31-2} package version ++
+  * 11: root 2015-01-19 Include OS codename into debian package version
+  * 10: root 2015-01-18 On lucid sys.path doen't include path where
+    mysql-connector-python installs its modules
+  * 9: root 2015-01-18 show registration hint only if the agent isn't
+    registred
+  * 8: root 2015-01-18 sh-compatible increment is fixed again
+  * 7: root 2015-01-18 Use a template @@CODENAME@@ in changelog. It is
+    replaced by a build script
+  * 6: root 2015-01-18 fix arithmetic expansion error in shell
+  * 5: root 2015-01-18 Make the packager's full name and email match to
+    ones from the packager GPG key
+  * 4: root 2015-01-17 Makefile can build Debian packages
+  * 3: root 2015-01-15 Moved Makefile from twindb project
+  * 2: root 2015-01-15 Fixed Bug #1403183
+
+ -- TwinDB Packager (TwinDB packager key) <[e-mail]>  Mon, 19 Jan 2015 04:47:04 -0800
+
+twindb (0.1.31-1.UNRELEASED) UNRELEASED; urgency=low
+
+  * New version 0.1.31-1
+  * 1: Aleksandr Kuzminsky 2015-01-15 {0.1.31-1} Initial import
+
+ -- TwinDB Packager (TwinDB packager key) <[e-mail]>  Thu, 15 Jan 2015 20:25:52 -0800

=== removed file 'support/deb/debian/changelog'
--- support/deb/debian/changelog	2015-01-16 07:41:06 +0000
+++ support/deb/debian/changelog	1970-01-01 00:00:00 +0000
@@ -1,15 +0,0 @@
-twindb (0.0.80-1)  testing; urgency=low
-
-  * Fixes https://bugs.launchpad.net/twindb-client/+bug/1273461
-  * Package numbering is synced with RPM
-
- -- Aleksandr Kuzminsky <[e-mail]>  Mon, 27 Jan 2014 17:14:01 -0500
-
-
-twindb (0.0-0.1) testing; urgency=low
-
-  * Non-maintainer upload.
-  * Initial release. (Closes: #1234)
-
- -- Aleksandr Kuzminsky <[e-mail]>  Fri, 15 Nov 2013 17:26:15 +0000
-

=== modified file 'support/deb/debian/control'
--- support/deb/debian/control	2015-01-16 07:41:06 +0000
+++ support/deb/debian/control	2015-01-24 21:00:42 +0000
@@ -1,13 +1,20 @@
 Source: twindb
-Maintainer: Aleksandr Kuzminsky <[e-mail]>
+Homepage: https://twindb.com
+Maintainer: TwinDB Packager (TwinDB packager key) <[e-mail]>
 Section: database
 Priority: optional
-Standards-Version: 3.8.4
+Standards-Version: 3.9.3
 Build-Depends: debhelper (>= 7)
+Vcs-Browser: https://code.launchpad.net/twindb-agent
+Vcs-Bzr: bzr+ssh://bazaar.launchpad.net/%2Bbranch/twindb-agent/
 
 Package: twindb
 Architecture: all
-Depends: ${misc:Depends}, percona-xtrabackup, php5-json | php5-common, php5-cli, gnupg, php5-mcrypt, mysql-server, mysql-client, php5-mysql
-Description: tool to manage backups of MySQL server
- This is a TwinDB agent - a tool that takes backups from MySQL server 
-
+Section: database
+Priority: optional
+Depends: ${misc:Depends}, lsof, percona-xtrabackup, gnupg, python (>= 2.6.0), mysql-connector-python (>= 2.0.0)
+Description: backup and recovery tool for MySQL
+ TwinDB is online backup service for MySQL.
+ It includes scheduler, retention policy manager, storage.
+ Visit us on https://twindb.com to get more details
+Homepage: https://twindb.com

=== modified file 'support/deb/debian/twindb.init'
--- support/deb/debian/twindb.init	2015-01-16 07:41:06 +0000
+++ support/deb/debian/twindb.init	2015-01-24 21:00:42 +0000
@@ -23,19 +23,45 @@
 	'start')
         if test -f $pid_file
         then
-            $0 stop
+            #$0 stop
+            echo "Pid file $pid_file exists."
+            echo "Either another instance of twindb is running or $pid_file is abandoned file."
+            echo "Please stop twindb agent first."
+            echo "$0 stop"
+            exit 1
         fi
+		echo -n "Starting TwinDB agent ... "
 		/usr/bin/twindb --start > /dev/null 2>&1 &
+		echo "OK"
 	;;
 	'stop')
 	if test -f $pid_file
 	then
+		echo -n "Stopping TwinDB agent ... "
+        pid=`cat "$pid_file"`
+        wait_timeout=300
 		/usr/bin/twindb --stop
+        # Wait till agent exits
+        while kill -0 $pid 2>/dev/null
+        do
+            if [ $wait_timeout -lt 1 ]; then break; fi
+            sleep 1
+            wait_timeout=$(( $wait_timeout - 1 ))
+        done
+		echo "OK"
+    else
+        echo "Pid file($pid_file) doesn't exist."
 	fi
-	;;
-	'restart'|'reload'|'force-reload')
-	$0 stop
-	$0 start
+    # Kill all abandoned and hung agents
+    pkill -9 -f /usr/bin/twindb || true
+	;;
+	'restart')
+	$0 stop
+    $0 start
+	;;
+	'force-reload')
+	$0 stop
+    $0 start
 	;;
 	'status')
 	if test -f $pid_file && ! test -z "$(ps -p `cat $pid_file` -o pid=)"

=== modified file 'support/deb/debian/twindb.manpages'
--- support/deb/debian/twindb.manpages	2015-01-16 07:41:06 +0000
+++ support/deb/debian/twindb.manpages	2015-01-24 21:00:42 +0000
@@ -1,1 +1,1 @@
-twindb.man
+support/twindb.man

=== modified file 'support/deb/debian/twindb.postinst'
--- support/deb/debian/twindb.postinst	2015-01-16 07:41:06 +0000
+++ support/deb/debian/twindb.postinst	2015-01-24 21:00:42 +0000
@@ -2,18 +2,16 @@
 
 chmod 755 /usr/bin/twindb
 
-echo "Please run "
-echo ""
-echo "twindb --register <registration code>"
-echo ""
-echo "to register the server"
-echo "The registration code you can find on https://twindb.com"
-
-# on saucy mcrypt is not loaded in cli php
-if test -z "`php -m | grep mcrypt`"
+if [ "`twindb --is-registered`" != "YES" ]
 then
-    ln -s /etc/php5/conf.d/mcrypt.ini /etc/php5/cli/conf.d/30-mcrypt.ini
+    echo "================================================================================"
+    echo "TwinDB client is installed."
+    echo "Now you need to register this server. To do so run:"
+    echo ""
+    echo "twindb --register <registration code>"
+    echo ""
+    echo "To obtain the registration code follow link https://console.twindb.com/?get_code"
+    echo "================================================================================"
 fi
-
 #DEBHELPER#
 

=== modified file 'support/rpm/rpmmacros'
--- support/rpm/rpmmacros	2015-01-16 07:41:06 +0000
+++ support/rpm/rpmmacros	2015-01-24 21:00:42 +0000
@@ -2,5 +2,3 @@
 %_gpg_name          [e-mail]
 %vendor             TwinDB LLC
 %packager           [e-mail]
-%_topdir            /root/rpmbuild/
-%_tmppath           /root/rpmbuild/tmp

=== renamed file 'support/rpm/twindb.spec' => 'support/rpm/twindb.spec.template'
--- support/rpm/twindb.spec	2015-01-16 07:41:06 +0000
+++ support/rpm/twindb.spec.template	2015-01-24 21:00:42 +0000
@@ -1,6 +1,6 @@
 Name:		twindb
-Version:	0.1.31
-Release:	1
+Version:	@@VERSION@@
+Release:	@@RELEASE@@
 Summary:	TwinDB agent for online MySQL backups
 
 Group:		Applications/Databases
@@ -25,6 +25,7 @@
 
 Requires:	gpg
 Requires:	tar 
+Requires:	lsof
 Requires:	chkconfig 
 Requires:	procps
 Requires:	mysql 

=== modified file 'support/twindb'
--- support/twindb	2015-01-16 07:41:06 +0000
+++ support/twindb	2015-01-24 21:00:42 +0000
@@ -26,7 +26,7 @@
             echo "Either another instance of twindb is running or $pid_file is abandoned file."
             echo "Please stop twindb agent first."
             echo "$0 stop"
-            exit -1
+            exit 1
         fi
 		echo -n "Starting TwinDB agent ... "
 		/usr/bin/twindb --start > /dev/null 2>&1 &
@@ -44,14 +44,14 @@
         do
             if [ $wait_timeout -lt 1 ]; then break; fi
             sleep 1
-            let wait_timeout=$wait_timeout-1
+            wait_timeout=$(( $wait_timeout - 1 ))
         done
 		echo "OK"
     else
         echo "Pid file($pid_file) doesn't exist."
 	fi
     # Kill all abandoned and hung agents
-    pkill -9 -f /usr/bin/twindb
+    pkill -9 -f /usr/bin/twindb || true
 	;;
 	'restart')
 	$0 stop

=== modified file 'twindb.py'
--- twindb.py	2015-01-15 21:02:14 +0000
+++ twindb.py	2015-01-24 21:00:42 +0000
@@ -19,7 +19,11 @@
 import json
 import urllib
 import re
-import mysql.connector
+try:
+    import mysql.connector
+except ImportError:
+    sys.path.insert(0, '/usr/lib/python2.6/site-packages')
+    import mysql.connector
 import tempfile
 import uuid
 
@@ -109,13 +113,15 @@
     def emit(self, record):
         global server_id
         global job_id
+        global debug
+
         request = {}
         request["type"] = "log"
         request["params"] = {}
         if job_id != 0:
             request["params"]["job_id"] = job_id
         request["params"]["msg"] = record.getMessage()
-        if self.log_flag:
+        if self.log_flag and not debug:
             self.log_flag = False
             get_response(request)
             self.log_flag = True
@@ -145,6 +151,38 @@
     -p | --password password        MySQL password")
     return
 
+
+def get_mysql_connection(user = None, passwd = None):
+    global mysql_user
+    global mysql_password
+
+    try:
+        if user == None:
+            user = mysql_user
+        if passwd == None:
+            passwd = mysql_password
+        mysql_option_files = []
+        unix_socket = get_unix_socket()
+        conn = mysql.connector.connect()
+        if user != None:
+            for f in ['/etc/mysql/my.cnf', '/etc/my.cnf', '/root/.my.cnf']:
+                if os.path.exists(f):
+                    mysql_option_files.append(f)
+            if mysql_option_files:
+                conn = mysql.connector.connect(option_files = mysql_option_files, unix_socket = unix_socket)
+                logger.debug("Connected to MySQL as '%s'@'localhost' with options files %r" % (conn.user, mysql_option_files) )
+        if not conn.is_connected():
+            conn = mysql.connector.connect(user = user, passwd = passwd, unix_socket = unix_socket)
+            logger.debug("Connected to MySQL as %s@localhost " % conn.user)
+    except mysql.connector.Error as err:
+        logger.error("Can not connect to local MySQL server")
+        logger.error("MySQL said: %s" % err.msg)
+        return None
+    except:
+        logger.error("Can not connect to local MySQL server")
+        logger.error("Unexpected error: %s,%s" % sys.exc_info()[:2]);
+        return None
+    return conn
 # Checks if TwinDB API public key is installed
 # Returns
 #   True    - if the key is installed
@@ -475,8 +513,7 @@
     
     uri = "api.php"
     response_body = None
-    logger.debug("Enter get_response(uri=" + uri + ") with request:")
-    logger.debug(request)
+    logger.debug("Enter get_response(uri=" + uri + ")")
     if proto == "http":
         conn = httplib.HTTPConnection(host)
     elif proto == "https":
@@ -486,8 +523,7 @@
     try:
         url = proto + "://" + host + "/" + api_dir + "/" + uri
         conn.connect()
-        logger.debug("Sending to " + host + ":")
-        logger.debug(request)
+        logger.debug("Sending to " + host + ": %s" % json.dumps(request, indent=4, sort_keys=True))
         data_json = json.dumps(request)
         data_json_enc = encrypt(data_json)
         data_json_enc_urlenc = urllib.urlencode({'data': data_json_enc})
@@ -502,7 +538,6 @@
         http_response = conn.getresponse()
         if(http_response.status == 200):
             response_body = http_response.read()
-            logger.debug("Response from server: %s" % response_body);
             if(len(response_body) == 0):
                 return None
             url = "%(proto)s://%(host)s/%(api_dir)s/%(uri)s" % {
@@ -511,9 +546,16 @@
                 "api_dir": api_dir, 
                 "uri": uri
             }
+            d = json.JSONDecoder()
+            try:
+                json_object = json.loads(response_body)
+            except ValueError as e:
+                msg = response_body
+            else:
+                msg = json.dumps(d.decode(response_body), indent=4, sort_keys=True)
             logger.debug("Response from %(url)s : %(resp)s" % {
                 "url": url, 
-                "resp": response_body
+                "resp": msg
             })
         else:
             logger.error("HTTP error %d %s" % (http_response.status, http_response.reason))
@@ -575,7 +617,6 @@
         response_body_decoded = d.decode(response_body)
         if response_body_decoded:
             msg_enc = response_body_decoded["response"]
-            logger.debug("Reply from server %s" % msg_enc)
             msg_pt = d.decode(decrypt(msg_enc))
             config = msg_pt["data"]
             logger.info("Got config:\n%s" % json.dumps(sanitize_config(config), indent=4, sort_keys=True))
@@ -634,10 +675,7 @@
         logger.error("Unexpected error: %s,%s" % sys.exc_info()[:2])
         return
     try:
-        unix_socket = get_unix_socket()
-        logger.debug("Connecting to MySQL as '%s'@'localhost' with password '%s' via socket %s" % 
-                (mysql_user, mysql_password, unix_socket))
-        con = mysql.connector.connect(user=mysql_user, passwd=mysql_password, unix_socket = unix_socket)
+        con = get_mysql_connection(user=mysql_user, passwd=mysql_password)
         cursor = con.cursor()
         query = "SELECT PRIVILEGE_TYPE FROM information_schema.USER_PRIVILEGES"
         logger.debug("Sending query : %s" % query)
@@ -742,15 +780,18 @@
         d = json.JSONDecoder()
         response_body_decoded = d.decode(response_body)
         if response_body_decoded:
-            msg_enc = response_body_decoded["response"]
-            logger.debug("Reply from server %s" % msg_enc)
-            msg_pt = d.decode(decrypt(msg_enc))
-            registration_status = msg_pt["data"]
-            logger.debug("Got registration status:\n%s" % json.dumps(registration_status, indent=4, sort_keys=True))
-            if msg_pt["error"]:
-                logger.error(msg_pt["error"])
-                exit_on_error(msg_pt["error"])
-            return registration_status["registered"]
+            if "response" in response_body_decoded:
+                msg_enc = response_body_decoded["response"]
+                msg_pt = d.decode(decrypt(msg_enc))
+                registration_status = msg_pt["data"]
+                logger.debug("Got registration status:\n%s" % json.dumps(registration_status, indent=4, sort_keys=True))
+                if msg_pt["error"]:
+                    logger.error(msg_pt["error"])
+                    exit_on_error(msg_pt["error"])
+                return registration_status["registered"]
+            else:
+                logger.debug("No valid response from dispatcher. Conseder agent unregistered")
+                return False
     except exceptions.KeyError as err:
         logger.error("Failed to decode %s" % response_body);
         exit_on_error(traceback.format_exc())
@@ -939,25 +980,15 @@
     return
 
 def get_slave_status():
-    global mysql_user
-    global mysql_password
 
     result = dict()
     # Read master host
     try:
-        mysql_option_files = []
-        for f in ['/etc/mysql/my.cnf', '/etc/my.cnf', '/root/.my.cnf']:
-            if os.path.exists(f):
-                mysql_option_files.append(f)
-        conn = mysql.connector.connect(option_files=mysql_option_files)
-        if not conn.is_connected():
-            conn = mysql.connector.connect(user=mysql_user, passwd=mysql_password)
-    except mysql.connector.Error as err:
-        logger.error("Can not connect to local MySQL server.")
-        logger.error("MySQL said: %s" % err.msg)
-        exit_on_error(traceback.format_exc())
-    try:
-        cursor = conn.cursor(dictionary=True)
+        conn = get_mysql_connection() 
+        if conn:
+            cursor = conn.cursor(dictionary=True)
+        else:
+            return None
         
         result["mysql_server_id"] = None
         result["mysql_master_server_id"] = None
@@ -981,9 +1012,9 @@
         cursor.close()
         conn.close()
     except:
-        logger.error("Can not read Master_host from MySQL")
+        logger.error("Could not read Master_host from MySQL")
         logger.error("Unexpected error: %s,%s" % sys.exc_info()[:2]);
-        exit_on_error(traceback.format_exc())
+        return None
     return result
 
 # Registers this server in TwinDB dispatcher
@@ -1042,6 +1073,9 @@
     cmd += "| awk '{ print $2}'"
     cmd += "| awk -F/ '{ print $1}'"
     ss = get_slave_status()
+    if ss is None:
+        logger.error("Could not get slave status on this server")
+        exit_on_error()
     p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
     local_ip = list()
     for row in p.stdout:
@@ -1065,8 +1099,6 @@
             "local_ip": local_ip
             }
     }
-    logger.debug("Registration request:")
-    logger.debug(data)
     response = get_response(data)
     if response <> None:
         jd = json.JSONDecoder()
@@ -1091,18 +1123,7 @@
 
     config = get_config()
     try:
-        mysql_option_files = []
-        for f in ['/etc/mysql/my.cnf', '/etc/my.cnf', '/root/.my.cnf']:
-            if os.path.exists(f):
-                mysql_option_files.append(f)
-        conn = mysql.connector.connect(option_files=mysql_option_files)
-        if not conn.is_connected():
-            conn = mysql.connector.connect(user=mysql_user, passwd=mysql_password)
-    except mysql.connector.Error as err:
-        logger.error("Can not connect to local MySQL server.")
-        logger.error("MySQL said: %s" % err.msg)
-        exit_on_error(traceback.format_exc())
-    try:
+        conn = get_mysql_connection()
         q = "GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT, SUPER, CREATE TABLESPACE"
         q += " ON *.* TO %s@'localhost' IDENTIFIED BY %s"
         cursor = conn.cursor()
@@ -1218,16 +1239,13 @@
 #   True    - if all necessary privileges are granted
 #   False   - if not all necessary privilegers are granted
 
-def check_mysql_permissions(config, job):
+def check_mysql_permissions(config):
     global logger
 
     try:
         logger.info("Checking if '%s'@'localhost' has enough rights to backup MySQL" % config["mysql_user"])
         result = False
-        unix_socket = get_unix_socket()
-        logger.debug("Connecting to MySQL as '%s'@'localhost' with password '%s' via socket %s" % 
-                (config["mysql_user"], config["mysql_password"], unix_socket))
-        con = mysql.connector.connect(user=config["mysql_user"], passwd=config["mysql_password"], unix_socket = unix_socket)
+        con = get_mysql_connection(user=config["mysql_user"], passwd=config["mysql_password"])
         required_priv = ["RELOAD", "LOCK TABLES", "REPLICATION CLIENT", "SUPER", "CREATE TABLESPACE"]
         for priv in required_priv:
             cursor = con.cursor()
@@ -1363,6 +1381,7 @@
         xtrabackup_cmd.append("--stream=xbstream")
         xtrabackup_cmd.append("--user=%s" % mysql_user)
         xtrabackup_cmd.append("--password=%s" % mysql_password)
+        xtrabackup_cmd.append("--socket=%s" % get_unix_socket())
         xtrabackup_cmd.append("--slave-info")
         xtrabackup_cmd.append("--safe-slave-backup")
         xtrabackup_cmd.append("--safe-slave-backup-timeout=3600")
@@ -1457,10 +1476,9 @@
     try:
         f, extra_config = tempfile.mkstemp()
         os.write(f, "[mysqld]\n")
-        mysql_user = config["mysql_user"]
-        mysql_password = config["mysql_password"]
-        unix_socket = get_unix_socket()
-        con = mysql.connector.connect(user = mysql_user, passwd = mysql_password, unix_socket = unix_socket)
+        con = get_mysql_connection(
+            user = config["mysql_user"], 
+            passwd = config["mysql_password"])
         cur = con.cursor()
         cur.execute("SELECT @@datadir")
         row = cur.fetchone()
@@ -1880,7 +1898,7 @@
     socket = ""
     try:
         cmd = [
-            "lsof", "-U", "-c", "mysqld", "-a", "-F", "n"
+            "lsof", "-U", "-c", "/^mysqld$/", "-a", "-F", "n"
         ]
         p = subprocess.Popen(cmd, stdout = subprocess.PIPE, stderr = subprocess.PIPE)
         cout, cerr = p.communicate()
@@ -1997,8 +2015,9 @@
 
 # Reports error removes pid and exits
 
-def exit_on_error(tb):
-    logger.debug(tb)
+def exit_on_error(tb = None):
+    if tb is not None:
+        logger.debug(tb)
     remove_pid()
     sys.exit(2)
 
@@ -2020,7 +2039,7 @@
         os.kill(pid, signal.SIGTERM)
         wait_pid(pid, 300)
         f.close()
-    except OSError:
+    except OSError as err:
         logger.error("Couldn't kill process %d" % pid)
         logger.error(err)
         exit_on_error(traceback.format_exc())
@@ -2090,7 +2109,7 @@
     global job_id
 
     try:
-        if not check_mysql_permissions(config, job):
+        if not check_mysql_permissions(config):
             return False
         if job["start_scheduled"] == None:
             logger.error("Job start time isn't set")
@@ -2150,6 +2169,8 @@
     global logger
     global mysql_user
     global mysql_password
+    global debug
+
     # Read or generate server id
     if os.path.exists(init_config):
         try:
@@ -2205,6 +2226,7 @@
         elif opt in ['-p', '--password']:
             mysql_password = arg
         elif opt == '-g':
+            debug = True
             logger.setLevel(logging.DEBUG)
         elif opt == '--start':
             action = "start"
@@ -2220,11 +2242,7 @@
         elif opt == '--backup':
             action = "backup"
         elif opt == '--is-registered':
-            if is_registered():
-                print("YES")
-            else:
-                print("NO")
-            sys.exit(0)
+            action = "is-registered"
         elif opt in ["-h", "--help"]:
             usage()
             sys.exit(0)
@@ -2265,6 +2283,12 @@
         if not action_handler_backup():
             logger.error("Failed to take backup")
             sys.exit(2)
+    elif action == "is-registered":
+        if is_registered():
+            print("YES")
+        else:
+            print("NO")
+        sys.exit(0)
     else:
         # If we got there print usage() and exit
         logger.error("Neither --start nor --stop nor --register is specified")