Kafka: kafka-cli cannot connect to broker with different encryption type

Issue

  • We have old kafka cluster setup with RC4-HMAC. Kafka v2.5.0

  • We build new kafka with AES128/256. Kafka v3.4.1

  • Both configured with SASL_SSL protocol and GSSAPI mechanism

  • When new node trying to connect to old kafka, it failed with errors

      23/12/05 10:12:22 INFO  network.Selector: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Failed authentication with kafka01.example.com/10.10.0.2 (channelId=-1) (Authentication failed during authentication due to invalid credentials with SASL mechanism GSSAPI)
      23/12/05 10:12:22 INFO  clients.NetworkClient: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Node -1 disconnected.
      23/12/05 10:12:22 ERROR clients.NetworkClient: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Connection to node -1 (kafka01.example.com/10.10.0.2:9093) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism GSSAPI
      23/12/05 10:12:22 WARN  internals.AdminMetadataManager: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error
      org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed during authentication due to invalid credentials with SASL mechanism GSSAPI
      Error while executing topic command : Authentication failed during authentication due to invalid credentials with SASL mechanism GSSAPI
      23/12/05 10:12:22 ERROR admin.TopicCommand$: [main]: org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed during authentication due to invalid credentials with SASL mechanism GSSAPI
    
      23/12/05 10:12:22 INFO  utils.AppInfoParser: [kafka-admin-client-thread | adminclient-1]: App info kafka.admin.client for adminclient-1 unregistered
      23/12/05 10:12:22 INFO  internals.AdminMetadataManager: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Metadata update failed
      org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited. Call: fetchMetadata
      23/12/05 10:12:22 INFO  internals.AdminMetadataManager: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Metadata update failed
      org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited. Call: fetchMetadata
      23/12/05 10:12:22 INFO  admin.KafkaAdminClient: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Timed out 2 remaining operation(s) during close.
      23/12/05 10:12:22 WARN  kerberos.KerberosLogin: [kafka-kerberos-refresh-thread-null]: [Principal=null]: TGT renewal thread has been interrupted and will exit.
      23/12/05 10:12:22 INFO  metrics.Metrics: [kafka-admin-client-thread | adminclient-1]: Metrics scheduler closed
      23/12/05 10:12:22 INFO  metrics.Metrics: [kafka-admin-client-thread | adminclient-1]: Closing reporter org.apache.kafka.common.metrics.JmxReporter
      23/12/05 10:12:22 INFO  metrics.Metrics: [kafka-admin-client-thread | adminclient-1]: Metrics reporters closed
    

Cause

Kafka Broker unable to identifying credential used in client due to different encryption types.

Resolution

  • Add RC4-HMAC to new cluster node

  • Make sure rc4-hmac is listed in first order in krb5.conf

      default_tgs_enctypes = rc4-hmac aes128-cts aes256-cts
      default_tkt_enctypes = rc4-hmac aes128-cts aes256-cts
      permitted_enctypes = rc4-hmac aes128-cts aes256-cts