Merge ~paelzer/ubuntu/+source/ruby2.5:merge-disco-2.5.3-3ubuntu1 into ubuntu/+source/ruby2.5:debian/sid

[Christian Ehrhardt]

[Christian Ehrhardt (paelzer)]

Extra tags to ease review:
 * [new tag] lp1806694/deconstruct/2.5.1-6ubuntu3 -> lp1806694/deconstruct/2.5.1-6ubuntu3
 * [new tag] lp1806694/logical/2.5.1-6ubuntu3 -> lp1806694/logical/2.5.1-6ubuntu3
 * [new tag] lp1806694/new/debian -> lp1806694/new/debian
 * [new tag] lp1806694/old/debian -> lp1806694/old/debian
 * [new tag] lp1806694/old/ubuntu -> lp1806694/old/ubuntu
 * [new tag] lp1806694/reconstruct/2.5.1-6ubuntu3 -> lp1806694/reconstruct/2.5.1-6ubuntu3

Test builds in PPA https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3552

Tests to be scheduled from Bileto once build is completed

[Christian Ehrhardt (paelzer)]

Note: there was a bunch of old Delta that was formerly undocumented, but I split it out in the deconstruct&logical tags.
When rebasing some of them (of course not all or none) were already upstream - so I modified them accordingly to drop them post merge. That way it was much more transparent what was removed (and why).

[Dimitri John Ledkov (xnox)]

LGTM.

review: Approve

[Christian Ehrhardt (paelzer)]

Test results: https://bileto.ubuntu.com/excuses/3552/disco.html

Still many running, will look into test failures (puma seems broken) tomorrow.

[Christian Ehrhardt (paelzer)]

Test issues with:
- puma (all)
  Has a bad test history, but also is only in -proposed.
  Needs trigger with proposed enabled, then retries and if it continues to fail like in the past
  it will need a force-badtest if three is no new regression)

- ruby-hamlit (all)
  covered ubuntu-release:35:force-badtest ruby-hamlit/2.8.8-1

- ruby-prof (amd64)
  "rake failed to run ..."
  Has a history of failing that way in 2/10 recent tests, rerun the test to be good

- ruby-rjb (x86)
  But here broke on "W: Unable to locate package ruby-rjb"
  => exists only in proposed, needs trigger with proposed enabled and then retries for the 50%
     flaky test

TL;DR: Tests look mostly good and those which failed seem handle-able.

@xnox - thanks for the review!

[Christian Ehrhardt (paelzer)]

Tag pushed so the next one doing a merge can use the split history.
Uploaded to disco and awaiting the expected test fallout in -proposed.

Uploading to ubuntu (via ftp to upload.ubuntu.com):
  Uploading ruby2.5_2.5.3-3ubuntu1.dsc: done.
  Uploading ruby2.5_2.5.3.orig.tar.xz: done.
  Uploading ruby2.5_2.5.3-3ubuntu1.debian.tar.xz: done.
  Uploading ruby2.5_2.5.3-3ubuntu1_source.buildinfo: done.
  Uploading ruby2.5_2.5.3-3ubuntu1_source.changes: done.
Successfully uploaded packages.

diff --git a/debian/changelog b/debian/changelog
index 7b20f12..cab2b90 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,29 @@
+ruby2.5 (2.5.3-3ubuntu1) disco; urgency=medium
+
+  * Merge with Debian unstable (LP: #1806694). Remaining changes:
+    - d/p/1dfc377ae3b174b043d3f0ed36de57b0296b34d0.patch: Cherrypick
+      upstream commit to fix session resumption with TLS 1.3.
+    - d/p/rubygems-2388.patch: Allow either Fetcher or OpenSSL exceptions
+      when using invalid cert in rubygems testcase.
+    - various backports for better openssl support (formerly undocumented in
+      changelog)
+      + d/p/0001-openssl-buffering.rb-no-RS-when-output.patch
+      + d/p/0006-Workaround-for-old-LibreSSL.patch
+  * Dropped changes
+    - various backports for better openssl support (formerly undocumented in
+      changelog, but upstream now)
+      + d/p/0002-no-ID-cache-in-Init-functions.patch
+      + d/p/0003-search-winsock-libraries-explicitly.patch
+      + d/p/0004-openssl-search-winsock.patch
+      + d/p/0007-openssl_missing.h-constified.patch
+      + d/p/0008-reduce-LibreSSL-warnings.patch
+      + d/p/0009-openssl-sync-with-upstream-repository.patch
+    - SECURITY UPDATE: Name equality check CVE-2018-16395 (in upstream)
+    - SECURITY UPDATE: Tainted flags not propagted CVE-2018-16396 (in upstream)
+    - 0012-test-time-tzdata-2018f.patch: Adjust tz tests for new tzdata.
+
+ -- Christian Ehrhardt <[e-mail]>  Tue, 04 Dec 2018 15:40:55 +0100
+
 ruby2.5 (2.5.3-3) unstable; urgency=medium
 
   * arm64: also skip TestBugReporter#test_bug_reporter_add, which also fails~
@@ -44,6 +70,35 @@ ruby2.5 (2.5.3-1) unstable; urgency=medium
 
  -- Antonio Terceiro <[e-mail]>  Sat, 24 Nov 2018 12:38:59 -0200
 
+ruby2.5 (2.5.1-6ubuntu3) disco; urgency=medium
+
+  * SECURITY UPDATE: Name equality check
+    - debian/patches/CVE-2018-16395.patch: fix in
+      ext/openssl/ossl_x509name.c.
+    - CVE-2018-16395
+  * SECURITY UPDATE: Tainted flags not propagted
+    - debian/patches/CVE-2018-16396.patch: fix in
+      pack.c, test/ruby/test_pack.rb.
+    - CVE-2018-16396
+
+ -- Leonidas S. Barbosa <[e-mail]>  Tue, 20 Nov 2018 10:30:19 -0300
+
+ruby2.5 (2.5.1-6ubuntu2) disco; urgency=medium
+
+  * 0012-test-time-tzdata-2018f.patch: Adjust tz tests for new tzdata.
+
+ -- Adam Conrad <[e-mail]>  Sun, 04 Nov 2018 15:38:57 -0700
+
+ruby2.5 (2.5.1-6ubuntu1) disco; urgency=medium
+
+  * Merge with Debian; remaining changes:
+    - Allow either Fetcher or OpenSSL exceptions when using invalid cert in
+      rubygems testcase.
+    - Cherrypick upstream commit to fix session resumption with TLS 1.3.
+    - Cherrypick upstream commit to bump the test key size for OpenSSL 1.1.1.
+
+ -- Matthias Klose <[e-mail]>  Sun, 04 Nov 2018 09:39:53 +0100
+
 ruby2.5 (2.5.1-6) unstable; urgency=medium
 
   * Fix build with openssl 1.1.1 (Closes: #907790)
@@ -63,6 +118,32 @@ ruby2.5 (2.5.1-6) unstable; urgency=medium
 
  -- Antonio Terceiro <[e-mail]>  Sat, 06 Oct 2018 14:15:02 -0300
 
+ruby2.5 (2.5.1-5ubuntu4) cosmic; urgency=high
+
+  * No change rebuild against openssl 1.1.1 with TLS1.3 support.
+
+ -- Dimitri John Ledkov <[e-mail]>  Tue, 25 Sep 2018 11:36:36 +0100
+
+ruby2.5 (2.5.1-5ubuntu3) cosmic; urgency=medium
+
+  * Allow either Fetcher or OpenSSL exceptions when using invalid cert in
+    rubygems testcase.
+
+ -- Dimitri John Ledkov <[e-mail]>  Mon, 24 Sep 2018 12:10:21 +0100
+
+ruby2.5 (2.5.1-5ubuntu2) cosmic; urgency=medium
+
+  * Cherrypick upstream commits to sync ruby-openssl extension to 2.1.1.
+
+ -- Dimitri John Ledkov <[e-mail]>  Mon, 24 Sep 2018 10:49:54 +0100
+
+ruby2.5 (2.5.1-5ubuntu1) cosmic; urgency=medium
+
+  * Cherrypick upstream commit to fix session resumption with TLS 1.3.
+  * Cherrypick upstream commit to bump the test key size for OpenSSL 1.1.1.
+
+ -- Dimitri John Ledkov <[e-mail]>  Sun, 23 Sep 2018 19:42:19 +0100
+
 ruby2.5 (2.5.1-5) unstable; urgency=medium
 
   * Fix spelling error in patch description
diff --git a/debian/control b/debian/control
index 9398c85..2d1e35a 100644
--- a/debian/control
+++ b/debian/control
@@ -1,7 +1,8 @@
 Source: ruby2.5
 Section: ruby
 Priority: optional
-Maintainer: Debian Ruby Team <[e-mail]>
+Maintainer: Ubuntu Developers <[e-mail]>
+XSBC-Original-Maintainer: Debian Ruby Team <[e-mail]>
 Uploaders: Antonio Terceiro <[e-mail]>,
            Chris Hofstaedtler <[e-mail]>
 Build-Depends: bison,
diff --git a/debian/patches/0001-openssl-buffering.rb-no-RS-when-output.patch b/debian/patches/0001-openssl-buffering.rb-no-RS-when-output.patch
new file mode 100644
index 0000000..f7b0d8e
--- /dev/null
+++ b/debian/patches/0001-openssl-buffering.rb-no-RS-when-output.patch
@@ -0,0 +1,42 @@
+From 07decad25f969908f7b2cc97208548605e35c5db Mon Sep 17 00:00:00 2001
+From: nobu <nobu@b2dd03c8-39d4-4d8f-98ff-823fe69b080e>
+Date: Thu, 25 Jan 2018 11:21:47 +0000
+Subject: [PATCH 1/9] openssl/buffering.rb: no RS when output
+
+* ext/openssl/lib/openssl/buffering.rb (do_write, puts): output
+  methods should not be affected by the input record separator.
+
+git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62038 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
+---
+ ext/openssl/lib/openssl/buffering.rb | 8 +++-----
+ 1 file changed, 3 insertions(+), 5 deletions(-)
+
+diff --git a/ext/openssl/lib/openssl/buffering.rb b/ext/openssl/lib/openssl/buffering.rb
+index 935f61f0ef..1f2b2a7e44 100644
+--- a/ext/openssl/lib/openssl/buffering.rb
++++ b/ext/openssl/lib/openssl/buffering.rb
+@@ -316,8 +316,8 @@ def do_write(s)
+     @wbuffer << s
+     @wbuffer.force_encoding(Encoding::BINARY)
+     @sync ||= false
+-    if @sync or @wbuffer.size > BLOCK_SIZE or idx = @wbuffer.rindex($/)
+-      remain = idx ? idx + $/.size : @wbuffer.length
++    if @sync or @wbuffer.size > BLOCK_SIZE or idx = @wbuffer.rindex("\n")
++      remain = idx ? idx + 1 : @wbuffer.size
+       nwritten = 0
+       while remain > 0
+         str = @wbuffer[nwritten,remain]
+@@ -409,9 +409,7 @@ def puts(*args)
+     end
+     args.each{|arg|
+       s << arg.to_s
+-      if $/ && /\n\z/ !~ s
+-        s << "\n"
+-      end
++      s.sub!(/(?<!\n)\z/, "\n")
+     }
+     do_write(s)
+     nil
+-- 
+2.17.1
+
diff --git a/debian/patches/0006-Workaround-for-old-LibreSSL.patch b/debian/patches/0006-Workaround-for-old-LibreSSL.patch
new file mode 100644
index 0000000..c160ebb
--- /dev/null
+++ b/debian/patches/0006-Workaround-for-old-LibreSSL.patch
@@ -0,0 +1,27 @@
+From 162adb82d1fc1abb3ca0ba86d450709b3bd7e321 Mon Sep 17 00:00:00 2001
+From: nobu <nobu@b2dd03c8-39d4-4d8f-98ff-823fe69b080e>
+Date: Wed, 30 May 2018 09:13:21 +0000
+Subject: [PATCH 6/9] Workaround for old LibreSSL
+
+git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63526 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
+---
+ ext/openssl/openssl_missing.h | 3 +++
+ 1 file changed, 3 insertions(+)
+
+diff --git a/ext/openssl/openssl_missing.h b/ext/openssl/openssl_missing.h
+index debd25adea..da3248a7c8 100644
+--- a/ext/openssl/openssl_missing.h
++++ b/ext/openssl/openssl_missing.h
+@@ -27,6 +27,9 @@ int ossl_EC_curve_nist2nid(const char *);
+ 
+ #if !defined(HAVE_X509_STORE_CTX_GET0_STORE)
+ #  define X509_STORE_CTX_get0_store(x) ((x)->ctx)
++#elif defined(LIBRESSL_VERSION_NUMBER) && LIBRESSL_VERSION_NUMBER < 0x2070000fL
++/* old LibreSSL provides this function but lacks the declaration */
++X509_STORE *X509_STORE_CTX_get0_store(X509_STORE_CTX *xs);
+ #endif
+ 
+ #if !defined(HAVE_SSL_IS_SERVER)
+-- 
+2.17.1
+
diff --git a/debian/patches/1dfc377ae3b174b043d3f0ed36de57b0296b34d0.patch b/debian/patches/1dfc377ae3b174b043d3f0ed36de57b0296b34d0.patch
new file mode 100644
index 0000000..707dbd7
--- /dev/null
+++ b/debian/patches/1dfc377ae3b174b043d3f0ed36de57b0296b34d0.patch
@@ -0,0 +1,157 @@
+From 1dfc377ae3b174b043d3f0ed36de57b0296b34d0 Mon Sep 17 00:00:00 2001
+From: rhe <rhe@b2dd03c8-39d4-4d8f-98ff-823fe69b080e>
+Date: Wed, 8 Aug 2018 14:13:55 +0000
+Subject: [PATCH] net/http, net/ftp: fix session resumption with TLS 1.3
+
+When TLS 1.3 is in use, the session ticket may not have been sent yet
+even though a handshake has finished. Also, the ticket could change if
+multiple session ticket messages are sent by the server. Use
+SSLContext#session_new_cb instead of calling SSLSocket#session
+immediately after a handshake. This way also works with earlier protocol
+versions.
+
+git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64234 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
+---
+ lib/net/ftp.rb              |  5 ++++-
+ lib/net/http.rb             |  7 +++++--
+ test/net/http/test_https.rb | 35 ++++++++++-------------------------
+ 3 files changed, 19 insertions(+), 28 deletions(-)
+
+diff --git a/lib/net/ftp.rb b/lib/net/ftp.rb
+index c3ee47ef4d36..9902f9dc657a 100644
+--- a/lib/net/ftp.rb
++++ b/lib/net/ftp.rb
+@@ -230,6 +230,10 @@ def initialize(host = nil, user_or_options = {}, passwd = nil, acct = nil)
+         if defined?(VerifyCallbackProc)
+           @ssl_context.verify_callback = VerifyCallbackProc
+         end
++        @ssl_context.session_cache_mode =
++          OpenSSL::SSL::SSLContext::SESSION_CACHE_CLIENT |
++          OpenSSL::SSL::SSLContext::SESSION_CACHE_NO_INTERNAL_STORE
++        @ssl_context.session_new_cb = proc {|sock, sess| @ssl_session = sess }
+         @ssl_session = nil
+         if options[:private_data_connection].nil?
+           @private_data_connection = true
+@@ -349,7 +353,6 @@ def start_tls_session(sock)
+       if @ssl_context.verify_mode != VERIFY_NONE
+         ssl_sock.post_connection_check(@host)
+       end
+-      @ssl_session = ssl_sock.session
+       return ssl_sock
+     end
+     private :start_tls_session
+diff --git a/lib/net/http.rb b/lib/net/http.rb
+index 281b15cedff0..683a884f5dbe 100644
+--- a/lib/net/http.rb
++++ b/lib/net/http.rb
+@@ -983,6 +983,10 @@ def connect
+         end
+         @ssl_context = OpenSSL::SSL::SSLContext.new
+         @ssl_context.set_params(ssl_parameters)
++        @ssl_context.session_cache_mode =
++          OpenSSL::SSL::SSLContext::SESSION_CACHE_CLIENT |
++          OpenSSL::SSL::SSLContext::SESSION_CACHE_NO_INTERNAL_STORE
++        @ssl_context.session_new_cb = proc {|sock, sess| @ssl_session = sess }
+         D "starting SSL for #{conn_address}:#{conn_port}..."
+         s = OpenSSL::SSL::SSLSocket.new(s, @ssl_context)
+         s.sync_close = true
+@@ -990,13 +994,12 @@ def connect
+         s.hostname = @address if s.respond_to? :hostname=
+         if @ssl_session and
+            Process.clock_gettime(Process::CLOCK_REALTIME) < @ssl_session.time.to_f + @ssl_session.timeout
+-          s.session = @ssl_session if @ssl_session
++          s.session = @ssl_session
+         end
+         ssl_socket_connect(s, @open_timeout)
+         if @ssl_context.verify_mode != OpenSSL::SSL::VERIFY_NONE
+           s.post_connection_check(@address)
+         end
+-        @ssl_session = s.session
+         D "SSL established"
+       end
+       @socket = BufferedIO.new(s, read_timeout: @read_timeout,
+diff --git a/test/net/http/test_https.rb b/test/net/http/test_https.rb
+index 8004d5c5f29f..a5182a1fe9db 100644
+--- a/test/net/http/test_https.rb
++++ b/test/net/http/test_https.rb
+@@ -71,20 +71,11 @@ def test_session_reuse
+     http.get("/")
+     http.finish
+ 
+-    http.start
+-    http.get("/")
+-    http.finish # three times due to possible bug in OpenSSL 0.9.8
+-
+-    sid = http.instance_variable_get(:@ssl_session).id
+-
+     http.start
+     http.get("/")
+ 
+     socket = http.instance_variable_get(:@socket).io
+-
+-    assert socket.session_reused?
+-
+-    assert_equal sid, http.instance_variable_get(:@ssl_session).id
++    assert_equal true, socket.session_reused?
+ 
+     http.finish
+   rescue SystemCallError
+@@ -101,16 +92,12 @@ def test_session_reuse_but_expire
+     http.get("/")
+     http.finish
+ 
+-    sid = http.instance_variable_get(:@ssl_session).id
+-
+     http.start
+     http.get("/")
+ 
+     socket = http.instance_variable_get(:@socket).io
+     assert_equal false, socket.session_reused?
+ 
+-    assert_not_equal sid, http.instance_variable_get(:@ssl_session).id
+-
+     http.finish
+   rescue SystemCallError
+     skip $!
+@@ -160,15 +147,16 @@ def test_certificate_verify_failure
+   end
+ 
+   def test_identity_verify_failure
++    # the certificate's subject has CN=localhost
+     http = Net::HTTP.new("127.0.0.1", config("port"))
+     http.use_ssl = true
+-    http.verify_callback = Proc.new do |preverify_ok, store_ctx|
+-      true
+-    end
++    http.cert_store = TEST_STORE
++    @log_tester = lambda {|_| }
+     ex = assert_raise(OpenSSL::SSL::SSLError){
+       http.request_get("/") {|res| }
+     }
+-    assert_match(/hostname \"127.0.0.1\" does not match/, ex.message)
++    re_msg = /certificate verify failed|hostname \"127.0.0.1\" does not match/
++    assert_match(re_msg, ex.message)
+   end
+ 
+   def test_timeout_during_SSL_handshake
+@@ -193,16 +181,13 @@ def test_timeout_during_SSL_handshake
+   end
+ 
+   def test_min_version
+-    http = Net::HTTP.new("127.0.0.1", config("port"))
++    http = Net::HTTP.new("localhost", config("port"))
+     http.use_ssl = true
+     http.min_version = :TLS1
+-    http.verify_callback = Proc.new do |preverify_ok, store_ctx|
+-      true
+-    end
+-    ex = assert_raise(OpenSSL::SSL::SSLError){
+-      http.request_get("/") {|res| }
++    http.cert_store = TEST_STORE
++    http.request_get("/") {|res|
++      assert_equal($test_net_http_data, res.body)
+     }
+-    assert_match(/hostname \"127.0.0.1\" does not match/, ex.message)
+   end
+ 
+   def test_max_version
diff --git a/debian/patches/rubygems-2388.patch b/debian/patches/rubygems-2388.patch
new file mode 100644
index 0000000..26bf929
--- /dev/null
+++ b/debian/patches/rubygems-2388.patch
@@ -0,0 +1,15 @@
+Description: Allow either Fetcher or OpenSSL exceptions when using invalid cert in rubygems testcase.
+Author: Dimitri John Ledkov <[e-mail]>
+Bug-Upstream: https://github.com/rubygems/rubygems/issues/2388
+
+--- ruby2.5-2.5.1.orig/test/rubygems/test_gem_remote_fetcher.rb
++++ ruby2.5-2.5.1/test/rubygems/test_gem_remote_fetcher.rb
+@@ -842,7 +842,7 @@ PeIQQkFng2VVot/WAQbv3ePqWq07g1BBcwIBAg==
+     with_configured_fetcher(
+       ":ssl_ca_cert: #{temp_ca_cert}\n" +
+       ":ssl_client_cert: #{temp_client_cert}\n") do |fetcher|
+-        assert_raises Gem::RemoteFetcher::FetchError do
++        assert_raises Gem::RemoteFetcher::FetchError, OpenSSL::SSL::SSLError do
+           fetcher.fetch_path("https://localhost:#{ssl_server.config[:Port]}/yaml")
+         end
+     end
diff --git a/debian/patches/series b/debian/patches/series
index 7ca7914..7ef25fc 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -9,3 +9,7 @@
 0009-test-test_pair-fix-deadlock-in-test_connect_accept_n.patch
 0010-test-use-larger-keys-for-SSL-tests.patch
 0011-Update-for-tzdata-2018f.patch
+1dfc377ae3b174b043d3f0ed36de57b0296b34d0.patch
+0001-openssl-buffering.rb-no-RS-when-output.patch
+0006-Workaround-for-old-LibreSSL.patch
+rubygems-2388.patch