Issue

• We have old kafka cluster setup with RC4-HMAC. Kafka v2.5.0

• We build new kafka with AES128/256. Kafka v3.4.1

• Both configured with SASL_SSL protocol and GSSAPI mechanism

• When new node trying to connect to old kafka, it failed with errors

    23/12/05 10:12:22 INFO  network.Selector: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Failed authentication with kafka01.example.com/10.10.0.2 (channelId=-1) (Authentication failed during authentication due to invalid credentials with SASL mechanism GSSAPI)
    23/12/05 10:12:22 INFO  clients.NetworkClient: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Node -1 disconnected.
    23/12/05 10:12:22 ERROR clients.NetworkClient: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Connection to node -1 (kafka01.example.com/10.10.0.2:9093) failed authentication due to: Authentication failed during authentication due to invalid credentials with SASL mechanism GSSAPI
    23/12/05 10:12:22 WARN  internals.AdminMetadataManager: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error
    org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed during authentication due to invalid credentials with SASL mechanism GSSAPI
    Error while executing topic command : Authentication failed during authentication due to invalid credentials with SASL mechanism GSSAPI
    23/12/05 10:12:22 ERROR admin.TopicCommand$: [main]: org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed during authentication due to invalid credentials with SASL mechanism GSSAPI
  
    23/12/05 10:12:22 INFO  utils.AppInfoParser: [kafka-admin-client-thread | adminclient-1]: App info kafka.admin.client for adminclient-1 unregistered
    23/12/05 10:12:22 INFO  internals.AdminMetadataManager: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Metadata update failed
    org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited. Call: fetchMetadata
    23/12/05 10:12:22 INFO  internals.AdminMetadataManager: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Metadata update failed
    org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited. Call: fetchMetadata
    23/12/05 10:12:22 INFO  admin.KafkaAdminClient: [kafka-admin-client-thread | adminclient-1]: [AdminClient clientId=adminclient-1] Timed out 2 remaining operation(s) during close.
    23/12/05 10:12:22 WARN  kerberos.KerberosLogin: [kafka-kerberos-refresh-thread-null]: [Principal=null]: TGT renewal thread has been interrupted and will exit.
    23/12/05 10:12:22 INFO  metrics.Metrics: [kafka-admin-client-thread | adminclient-1]: Metrics scheduler closed
    23/12/05 10:12:22 INFO  metrics.Metrics: [kafka-admin-client-thread | adminclient-1]: Closing reporter org.apache.kafka.common.metrics.JmxReporter
    23/12/05 10:12:22 INFO  metrics.Metrics: [kafka-admin-client-thread | adminclient-1]: Metrics reporters closed
  

Cause

Kafka Broker unable to identifying credential used in client due to different encryption types.

Resolution

• Add RC4-HMAC to new cluster node

• Make sure rc4-hmac is listed in first order in krb5.conf

    default_tgs_enctypes = rc4-hmac aes128-cts aes256-cts
    default_tkt_enctypes = rc4-hmac aes128-cts aes256-cts
    permitted_enctypes = rc4-hmac aes128-cts aes256-cts