Compression in Ceph

How to enable Compression in Ceph and Allowed Compression Types

Photo by Oleg Ivanov on Unsplash

From https://docs.ceph.com/docs/mimic/radosgw/compression/ and https://www.bookstack.cn/read/ceph-en/184864:

“The compression type refers to the name of the compression plugin to use when writing new object data. Each compressed object remembers which plugin was used, so changing this setting does not hinder the ability to decompress existing objects, not does it force existing objects to be recompressed.

This compression setting applies to all new objects uploaded to buckets using this placement target. Compression can be disabled by setting the type to an empty string or none."

As described in http://manpages.ubuntu.com/manpages/bionic/man8/radosgw-admin.8.html, with “radosgw-admin zone get” command, we get placement_pools info:

sh-4.2# radosgw-admin zone get
{
"id": "0287b371-cd27-4f79-a7dd-ff230cf7ce4e",
"name": "default",
"domain_root": "default.rgw.meta:root",
"control_pool": "default.rgw.control",
"gc_pool": "default.rgw.log:gc",
"lc_pool": "default.rgw.log:lc",
"log_pool": "default.rgw.log",
"intent_log_pool": "default.rgw.log:intent",
"usage_log_pool": "default.rgw.log:usage",
"reshard_pool": "default.rgw.log:reshard",
"user_keys_pool": "default.rgw.meta:users.keys",
"user_email_pool": "default.rgw.meta:users.email",
"user_swift_pool": "default.rgw.meta:users.swift",
"user_uid_pool": "default.rgw.meta:users.uid",
"otp_pool": "default.rgw.otp",
"system_key": {
"access_key": "",
"secret_key": ""
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": "default.rgw.buckets.index",
"storage_classes": {
"STANDARD": {
"data_pool": "default.rgw.buckets.data"
}
},
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_type": 0
}
}
],
"metadata_heap": "",
"realm_id": ""
}

We run the command below (compression zlib):

radosgw-admin zone placement modify \
--rgw-zone default \
--placement-id default-placement \
--storage-class STANDARD \
--compression zlib

For more information, you can visit https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/object_gateway_guide_for_ubuntu/rgw-administration-rgw. Under 3.5. Enabling Compression title, types of compression are listed as below:

• zlib: Supported.
• snappy: Technology Preview. (not fully supported, as Red Hat has not completed quality assurance testing on it yet)
• zstd: Technology Preview. (not fully supported, as Red Hat has not completed quality assurance testing on it yet)

To get more info about zlib, you can visit https://www.euccas.me/zlib/

After we run Modify command, we run the query again and the field “compression_type” : “zlib” appears with its value.

[root@ymtcos02 ~]# radosgw-admin zone placement list
[
{
"key": "default-placement",
"val": {
"index_pool": "default.rgw.buckets.index",
"storage_classes": {
"STANDARD": {
"data_pool": "default.rgw.buckets.data",
"compression_type": "zlib"
}
},
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_type": 0
}
}
]

“After enabling or disabling compression, restart the Ceph Object Gateway instance so the change will take effect.”

We run the command “systemctl restart ceph-radosgw” as in https://github.com/rook/rook/issues/5574

For statistics, we can compare size_actual and size_utilized values (normal and compressed values) with the command below.

“The size_utilized and size_kb_utilized fields represent the total size of compressed data in bytes and kilobytes respectively."

[root@ymtcos02 ~]# radosgw-admin bucket stats --bucket=yaanibucket
{
"bucket": "yaanibucket",
"num_shards": 1,
"tenant": "",
"zonegroup": "16a4636b-f707-4435-9d2a-c63d191f1be2",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "ce5fe910-c9a4-4031-8472-bba9035fd1c9.6047.2",
"marker": "ce5fe910-c9a4-4031-8472-bba9035fd1c9.6047.2",
"index_type": "Normal",
"owner": "yaaniuser",
"ver": "0#3",
"master_ver": "0#0",
"mtime": "2020-06-23 07:43:30.661214Z",
"max_marker": "0#",
"usage": {
"rgw.main": {
"size": 57444442,
"size_actual": 57446400,
"size_utilized": 57444442,
"size_kb": 56099,
"size_kb_actual": 56100,
"size_kb_utilized": 56099,
"num_objects": 2
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}

Happy Coding!