Skip to content

Bug: Encrypted batch messages fail to deserialize when compression is disabled #1527

Open
Asthenia0412/pulsar-client-go
#1

Description

@n2tp9

Bug: Encrypted batch messages fail to deserialize when compression is disabled

Expected behavior

Encrypted batch messages with no compression should be consumed successfully, just like encrypted batch messages with compression (LZ4, ZSTD, etc).

Actual behavior

Consumer fails to deserialize encrypted batch messages when compression is not applied. This can occur when CompressionType: NoCompression is explicitly set, or when compression is conditionally skipped (e.g., for small batches where compression would not provide benefit).

The root cause is in pulsar/consumer_partition.go lines 1331-1340: the reader.ResetBuffer() call is only executed when msgMeta.UncompressedSize != nil && *msgMeta.UncompressedSize > 0.

Steps to reproduce

Failing test
func TestBatchProducerConsumerRSAEncryptionWithoutCompression(t *testing.T) {
	client, err := NewClient(ClientOptions{
		URL: lookupURL,
	})
	assert.Nil(t, err)
	defer client.Close()

	topic := fmt.Sprintf("my-topic-enc-no-comp-%v", time.Now().Nanosecond())

	cryptoConsumer, err := client.Subscribe(ConsumerOptions{
		Topic: topic,
		Decryption: &MessageDecryptionInfo{
			KeyReader: crypto.NewFileKeyReader("crypto/testdata/pub_key_rsa.pem",
				"crypto/testdata/pri_key_rsa.pem"),
		},
		SubscriptionName: "crypto-subscription",
		Schema:           NewStringSchema(nil),
	})

	assert.Nil(t, err)

	normalConsumer, err := client.Subscribe(ConsumerOptions{
		Topic:            topic,
		SubscriptionName: "normal-subscription",
		Schema:           NewStringSchema(nil),
	})

	assert.Nil(t, err)
	batchSize := 2
	cryptoProducer, err := client.CreateProducer(ProducerOptions{
		Topic: topic,
		Encryption: &ProducerEncryptionInfo{
			KeyReader: crypto.NewFileKeyReader("crypto/testdata/pub_key_rsa.pem",
				"crypto/testdata/pri_key_rsa.pem"),
			Keys: []string{"client-rsa.pem"},
		},
		Schema:              NewStringSchema(nil),
		CompressionType:     NoCompression,
		DisableBatching:     false,
		BatchingMaxMessages: uint(batchSize),
	})

	assert.Nil(t, err)

	msgFormat := "my-message-%v"

	totalMessages := 10

	ctx := context.Background()

	for i := 0; i < totalMessages; i++ {
		_, err := cryptoProducer.Send(ctx, &ProducerMessage{
			Value: fmt.Sprintf(msgFormat, i),
		})

		assert.Nil(t, err)
	}

	// try to consume with normal consumer
	normalConsumerCtx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond)
	defer cancel()

	msg, err := normalConsumer.Receive(normalConsumerCtx)
	// msg should be null as the consumer will not be able to decrypt
	assert.NotNil(t, err)
	assert.Nil(t, msg)

	// try to consume the message by crypto consumer
	// consumer should be able to read all the messages
	var actualMessage *string
	for i := 0; i < totalMessages; i++ {
		msg, err := cryptoConsumer.Receive(ctx)
		assert.Nil(t, err)
		expectedMsg := fmt.Sprintf(msgFormat, i)
		err = msg.GetSchemaValue(&actualMessage)
		assert.Nil(t, err)
		assert.Equal(t, expectedMsg, *actualMessage)
		cryptoConsumer.Ack(msg)
	}
}

System configuration

  • Pulsar client: v0.19.0, v0.20.0, v0.21.0, master
  • Go/OS: any

Activity

  1. EdwardShen125 commented on Nov 15, 2026

    @EdwardShen125

    Doesn't reproduce with a Go producer on master: Go and Java always set uncompressed_size (batch_builder.go, ProducerImpl.java). The trigger is the C++ client (and the Python client built on it), which sets it only when compression is enabled (ProducerImpl.cc) — an encrypted batch with CompressionNone arrives without the field, and after decryption the reader stays over the ciphertext.

    The fork PR above (@Asthenia0412) has the same fix, but its base is his own fork, so it can't be merged here and it's been idle since September. #1569 submits the fix to this repo, adding a broker-free test with a real RSA round-trip (fails on master without the fix) and the e2e test from this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions