Kafka: kafka-cli cannot connect to broker with different encryption type
Issue
We have old kafka cluster setup with RC4-HMAC. Kafka v2.5.0
We build new kafka with AES128/256. Kafka v3.4.1
Both configured with
SASL_SSL
protocol andGSSAPI
mechanismWhen new node trying to connect to old kafka, it failed with errors
23/12/05 10:12:22 INFO network.Selector: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Failed authentication with kafka01.example.com/10.10.0.2 (channelId=-1) (Authentication failed during authentication due to invalid credentials with SASL mechanism GSSAPI) 23/12/05 10:12:22 INFO clients.NetworkClient: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Node -1 disconnected. 23/12/05 10:12:22 ERROR clients.NetworkClient: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Connection to node -1 (kafka01.example.com/10.10.0.2:9093) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism GSSAPI 23/12/05 10:12:22 WARN internals.AdminMetadataManager: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed during authentication due to invalid credentials with SASL mechanism GSSAPI Error while executing topic command : Authentication failed during authentication due to invalid credentials with SASL mechanism GSSAPI 23/12/05 10:12:22 ERROR admin.TopicCommand$: [main]: org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed during authentication due to invalid credentials with SASL mechanism GSSAPI 23/12/05 10:12:22 INFO utils.AppInfoParser: [kafka-admin-client-thread | adminclient-1]: App info kafka.admin.client for adminclient-1 unregistered 23/12/05 10:12:22 INFO internals.AdminMetadataManager: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Metadata update failed org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited. Call: fetchMetadata 23/12/05 10:12:22 INFO internals.AdminMetadataManager: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Metadata update failed org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited. Call: fetchMetadata 23/12/05 10:12:22 INFO admin.KafkaAdminClient: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Timed out 2 remaining operation(s) during close. 23/12/05 10:12:22 WARN kerberos.KerberosLogin: [kafka-kerberos-refresh-thread-null]: [Principal=null]: TGT renewal thread has been interrupted and will exit. 23/12/05 10:12:22 INFO metrics.Metrics: [kafka-admin-client-thread | adminclient-1]: Metrics scheduler closed 23/12/05 10:12:22 INFO metrics.Metrics: [kafka-admin-client-thread | adminclient-1]: Closing reporter org.apache.kafka.common.metrics.JmxReporter 23/12/05 10:12:22 INFO metrics.Metrics: [kafka-admin-client-thread | adminclient-1]: Metrics reporters closed
Cause
Kafka Broker unable to identifying credential used in client due to different encryption types.
Resolution
Add RC4-HMAC to new cluster node
Make sure
rc4-hmac
is listed in first order inkrb5.conf
default_tgs_enctypes = rc4-hmac aes128-cts aes256-cts default_tkt_enctypes = rc4-hmac aes128-cts aes256-cts permitted_enctypes = rc4-hmac aes128-cts aes256-cts