Release Notes - Kafka - Version 3.9.1

Below is a summary of the JIRA issues addressed in the 3.9.1 release of Kafka. For full documentation of the release, a guide to get started, and information about the project, see the Kafka project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Kafka.

The documentation for the most recent release can be found at https://kafka.apache.org/documentation.html.

Improvement

  • [KAFKA-806] - Index may not always observe log.index.interval.bytes
  • [KAFKA-17520] - align the low bound of ducktape version
  • [KAFKA-18046] - High CPU usage when using Log4j2
  • [KAFKA-18237] - add 3.7.2 to code base

    • Bug

  • [KAFKA-8862] - Misleading exception message for non-existant partition
  • [KAFKA-12679] - Rebalancing a restoring or running task may cause directory livelocking with newly created task
  • [KAFKA-15931] - Cached transaction index gets closed if tiered storage read is interrupted
  • [KAFKA-16407] - ForeignKey INNER join ignores FK change when its previous value is null
  • [KAFKA-16434] - ForeignKey INNER join does not unset join result when FK becomes null
  • [KAFKA-17299] - Kafka Streams consumer stops consumption
  • [KAFKA-17402] - Test failure: DefaultStateUpdaterTest.shouldGetTasksFromRestoredActiveTasks expected: <2> but was: <3>
  • [KAFKA-17455] - `TaskCorruptedException` After Client Quota Throttling
  • [KAFKA-17507] - WriteTxnMarkers API must not return until markers are written and materialized in group coordinator's cache
  • [KAFKA-17635] - Lost events on internal repartition topic when excatly_once_v2 is set and producer is fenced
  • [KAFKA-17792] - header parsing ends up timing out and using large quantities of memory if the string looks like a number
  • [KAFKA-17801] - RemoteLogManager may compute inaccurate upperBoundOffset for aborted txns
  • [KAFKA-17872] - Kafka stream job does not commit offset when packets are getting ignored intentionally
  • [KAFKA-17954] - Error getting oldest-iterator-open-since-ms from JMX
  • [KAFKA-17969] - Fix StorageToolTest for 3.9
  • [KAFKA-17994] - Checked exceptions are not handled when deserializing kafka stream record
  • [KAFKA-17995] - Large value for `retention.ms` could prevent remote data cleanup in Tiered Storage
  • [KAFKA-18001] - KafkaNetworkChannel missing UpdateRaftVoterRequestData logic
  • [KAFKA-18029] - Fix quorum_reconfiguration_test.py
  • [KAFKA-18051] - Disallow creating ACLs with principals that do not contain a colon
  • [KAFKA-18063] - SnapshotRegistry should not leak memory
  • [KAFKA-18070] - Update kafka-metadata-quorum.sh output in docs to match post-KIP-853 appearance
  • [KAFKA-18100] - `Using` block suppresses all errors
  • [KAFKA-18171] - Unexpected Change in bootstrap.servers Behavior After Upgrade to 3.8.0
  • [KAFKA-18199] - Incorrect size calculation for ConsumerGroupMemberMetadataValue.classicMemberMetadata
  • [KAFKA-18263] - Group lock must be acquired when reverting static membership rejoin
  • [KAFKA-18281] - Kafka (3.9.0) is improperly validating non-advertised listeners for routable controller addresses
  • [KAFKA-18326] - Cached stores may return deleted values
  • [KAFKA-18342] - Vagrantfile can't work with ruby 3.2
  • [KAFKA-18478] - RocksDBTimeOrderedKeyValueBuffer does only use serdes from config
  • [KAFKA-18547] - Failing test ClientUtilsTest#testParseAndValidateAddressesWithReverseLookup in trunk
  • [KAFKA-18597] - max-buffer-utilization-percent is always 0
  • [KAFKA-18713] - Kafka Streams Left-Join not always emitting the last value
  • [KAFKA-18723] - KRaft must handle corrupted records in the fetch response
  • [KAFKA-18758] - NullPointerException in shutdown following InvalidConfigurationException
  • [KAFKA-18803] - The acls would appear at the wrong level of the metadata shell "tree"
  • [KAFKA-18844] - Stale features information in QuorumController#registerBroker
  • [KAFKA-18920] - The kcontrollers do not set kraft.version in ApiVersionsResponse
  • [KAFKA-18943] - Kafka Streams incorrectly commits TX during task revokation
  • [KAFKA-18962] - StateRestoreListener onBatchRestored method is called with the totalRestored on GlobalStateStore reprocess
  • [KAFKA-18991] - AbstractFetcherThread should verify the current leader epoch in the fetch request matches the leader epoch in the fetch state
  • [KAFKA-19071] - Incorrect documentation for remote.storage.enable
  • [KAFKA-19166] - Release tags are orphaned

    • Task

  • [KAFKA-17078] - Add SecurityManager reflective shim
  • [KAFKA-18047] - Add org.apache.kafka.automatic.config.providers to System Properties section
  • [KAFKA-18583] - Improve/fix KRaftMetadataCache.getPartitionReplicaEndpoints

    • Test

  • [KAFKA-18546] - Change ClientUtilsTest#testParseAndValidateAddressesWithReverseLookup to mock test

    • Sub-task

  • [KAFKA-17030] - Voters should not assume that the leader will send them BeginQuorumEpoch requests
  • [KAFKA-17722] - Fix "this-escape" compiler warnings (BrokerTopicMetrics) for JDK 23
  • [KAFKA-17723] - Fix "this-escape" compiler warnings (MultiThreadedEventProcessor and DistributedHerder) for JDK 23
  • [KAFKA-17973] - Relax Restriction for Voters Set Change
  • [KAFKA-18270] - SaslHandshake v0 and FindCoordinator v0 incorrectly tagged as deprecated
  • [KAFKA-18272] - Deprecated protocol api usage should be logged at info level
  • [KAFKA-18330] - Update documentation to remove controller deployment limitations
  • [KAFKA-18334] - Produce v4-v6 should be undeprecated
  • [KAFKA-18352] - DeleteGroups v0 incorrectly tagged as deprecated
  • [KAFKA-18663] - Metadata v0-v3 should be undeprecated
  • [KAFKA-18675] - add unit tests for KAFKA-18171 to verify the parser of bootstrap.servers
  • [KAFKA-18984] - failed to reset interval.ms by using kafka-client-metrics.sh