radosgw: S3/Keystone integration should be disabled by default

I guess we should revert https://review.openstack.org/#/c/257492/

Enabling or disabling S3/Keystone integration in radosgw
is a decision associated with a trade-off: load on Keystone
(performance) vs easiness of configuration and management.
For technical details, please take a look at the comment's
bottom.

IMO the decision should be taken directly by user and
leaving Ceph's defaults for "rgw_s3_auth_use_keystone"
we followed for ages seems to be a reasonable option for
MOS 8. For MOS 9 and above we should have a checkbox
in Fuel with warning/message explaining the trade-off.

=== root cause of performance problem ===
An application employing S3 API authenticates a message,
assures its integrity and prohibits replay attacks through
appending result of HMAC function calculated against his
signing key and some parts of the message (including
current time):

  signature := HMAC(<signing key>, <parts of message>)

Then, signature and user ID are appended to the message.

RadosGW cannot know user credentials (its solely Keystone's
responsibility), so it has to ask Keystone each time when it
comes to verification whether the signature is correct or not.
As the signature is specific for a given message, caching which
is normally employed in case of Swift API, won't be effective
here.

Fixed in https://review.openstack.org/#/c/274783/

Patch for 8.0 is merged - https://review.openstack.org/#/c/275334/

Verified on MOS 8.0 build 529

cat /etc/ceph/ceph.conf
[global]
fsid = 69b9cdea-9611-4130-be94-6a8b2d12f12f
mon_initial_members = node-1 node-2
mon_host = 192.168.0.5 192.168.0.6
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
log_to_syslog_level = info
log_to_syslog = True
osd_pool_default_size = 3
osd_pool_default_min_size = 1
osd_pool_default_pg_num = 64
public_network = 192.168.0.0/24
log_to_syslog_facility = LOG_LOCAL0
osd_journal_size = 2048
auth_supported = cephx
osd_pool_default_pgp_num = 64
osd_mkfs_type = xfs
cluster_network = 192.168.1.0/24
osd_recovery_max_active = 1
osd_max_backfills = 1
max_open_files = 131072
debug_default = True

[client]
rbd_cache_writethrough_until_flush = True
rbd_cache = True

[client.radosgw.gateway]
rgw_keystone_accepted_roles = _member_, Member, admin, swiftoperator
keyring = /etc/ceph/keyring.radosgw.gateway
rgw_frontends = fastcgi socket_port=9000 socket_host=127.0.0.1
rgw_socket_path = /tmp/radosgw.sock
rgw_keystone_revocation_interval = 1000000
rgw_keystone_url = 192.168.0.2:35357
rgw_keystone_admin_token = JV0iKwNrvkgBKkkvEB6Vrv8L
host = node-1
rgw_dns_name = *.test.domain.local
rgw_print_continue = True
rgw_keystone_token_cache_size = 10
rgw_data = /var/lib/ceph/radosgw
user = www-data

[osd]
journal_queue_max_ops = 3000
objecter_inflight_ops = 10240
journal_queue_max_bytes = 1048576000
filestore_queue_max_ops = 500
osd_mkfs_type = xfs
osd_mount_options_xfs = rw,relatime,inode64,logbsize=256k,delaylog,allocsize=4M
osd_op_threads = 20
filestore_queue_committing_max_ops = 5000
journal_max_write_entries = 1000
objecter_infilght_op_bytes = 1048576000
filestore_queue_max_bytes = 1048576000
filestore_max_sync_interval = 10
journal_max_write_bytes = 1048576000
filestore_queue_committing_max_bytes = 1048576000
ms_dispatch_throttle_bytes = 1048576000

verified on MOS 8.0 RC1:

root@node-1:~# egrep -e "rgw_s3_auth_use_keystone" -r /etc/ceph/
root@node-1:~#

>> S3 authentication, significant load on Keystone may be generated
if we suspect that keystone load might be generated - please create proper bug against keystone. Just turning of functionality is not a suitable solution for customer.

>> For MOS 9 and above we should have a checkbox in Fuel with warning/message explaining the trade-off.
where is this work tracked? Please reference Fuel bug/BP where this will be addressed.

Verified on MOS 9.0 build 106

root@node-1:~# egrep -e "rgw_s3_auth_use_keystone" -r /etc/ceph/
root@node-1:~#

I set status of bug to fix released for 8.0 (as it was before). I don't see any comment which describes why it was set to confirmed.

If someone need additional changes it should be covered in separate bug.