Apache Pulsar Cheat Sheet

Overview

Apache Pulsar is a cloud-native, distributed messaging and streaming platform originally created at Yahoo and now an Apache top-level project. It provides a unified messaging model supporting both queueing and streaming use cases with multi-tenancy, geo-replication, and tiered storage built in.

Pulsar separates serving (brokers) from storage (Apache BookKeeper), enabling independent scaling. It supports multiple subscription types (exclusive, shared, failover, key-shared), schema enforcement, message deduplication, and delayed message delivery. Pulsar Functions provide lightweight serverless compute directly on the messaging layer.

Installation

Standalone Mode (Development)

# Download Pulsar
wget https://archive.apache.org/dist/pulsar/pulsar-3.3.0/apache-pulsar-3.3.0-bin.tar.gz
tar xvfz apache-pulsar-3.3.0-bin.tar.gz
cd apache-pulsar-3.3.0

# Start standalone (broker + bookie + zookeeper)
bin/pulsar standalone

# Verify
bin/pulsar-admin brokers list use

Docker

docker run -d --name pulsar \
  -p 6650:6650 \
  -p 8080:8080 \
  apachepulsar/pulsar:3.3.0 \
  bin/pulsar standalone

Docker Compose (Full Cluster)

version: '3'
services:
  zookeeper:
    image: apachepulsar/pulsar:3.3.0
    command: bin/pulsar zookeeper
    ports:
      - "2181:2181"
  bookie:
    image: apachepulsar/pulsar:3.3.0
    command: bin/pulsar bookie
    depends_on:
      - zookeeper
  broker:
    image: apachepulsar/pulsar:3.3.0
    command: bin/pulsar broker
    ports:
      - "6650:6650"
      - "8080:8080"
    depends_on:
      - bookie

Core CLI Commands

Tenant and Namespace Management

Command	Description
`pulsar-admin tenants list`	List all tenants
`pulsar-admin tenants create my-tenant`	Create a tenant
`pulsar-admin namespaces list my-tenant`	List namespaces in a tenant
`pulsar-admin namespaces create my-tenant/my-ns`	Create a namespace
`pulsar-admin namespaces delete my-tenant/my-ns`	Delete a namespace
`pulsar-admin namespaces policies my-tenant/my-ns`	Show namespace policies

Topic Management

# Create a partitioned topic
bin/pulsar-admin topics create-partitioned-topic \
  persistent://my-tenant/my-ns/my-topic -p 4

# List topics
bin/pulsar-admin topics list my-tenant/my-ns

# Get topic stats
bin/pulsar-admin topics stats persistent://my-tenant/my-ns/my-topic

# Peek at messages
bin/pulsar-admin topics peek-messages \
  persistent://my-tenant/my-ns/my-topic -s my-sub -n 10

# Skip messages on a subscription
bin/pulsar-admin topics skip \
  persistent://my-tenant/my-ns/my-topic -s my-sub -n 100

# Delete a topic
bin/pulsar-admin topics delete persistent://my-tenant/my-ns/my-topic

Producing and Consuming

# Produce messages
bin/pulsar-client produce persistent://my-tenant/my-ns/my-topic \
  -m "Hello Pulsar" -n 10

# Consume messages
bin/pulsar-client consume persistent://my-tenant/my-ns/my-topic \
  -s my-subscription -n 10

# Consume with specific subscription type
bin/pulsar-client consume persistent://my-tenant/my-ns/my-topic \
  -s my-shared-sub -t Shared -n 0

Configuration

Broker Configuration (`conf/broker.conf`)

# Cluster name
clusterName=my-cluster

# Zookeeper connection
zookeeperServers=zk1:2181,zk2:2181,zk3:2181
configurationStoreServers=zk1:2181,zk2:2181,zk3:2181

# Broker settings
brokerServicePort=6650
webServicePort=8080

# Message retention
defaultRetentionTimeInMinutes=4320
defaultRetentionSizeInMB=10240

# Backlog quota
backlogQuotaDefaultLimitGB=10
backlogQuotaDefaultRetentionPolicy=producer_request_hold

# Deduplication
brokerDeduplicationEnabled=true

# Max message size (5MB default)
maxMessageSize=5242880

# Managed ledger settings
managedLedgerDefaultEnsembleSize=2
managedLedgerDefaultWriteQuorum=2
managedLedgerDefaultAckQuorum=2

Namespace Policies

# Set retention policy
bin/pulsar-admin namespaces set-retention my-tenant/my-ns \
  --size 10G --time 7d

# Set backlog quota
bin/pulsar-admin namespaces set-backlog-quota my-tenant/my-ns \
  --limit 10G --policy producer_request_hold

# Set message TTL
bin/pulsar-admin namespaces set-message-ttl my-tenant/my-ns --messageTTL 3600

# Set replication clusters
bin/pulsar-admin namespaces set-clusters my-tenant/my-ns \
  --clusters us-east,us-west,eu-central

# Enable deduplication
bin/pulsar-admin namespaces set-deduplication my-tenant/my-ns --enable

# Set schema validation
bin/pulsar-admin namespaces set-schema-validation-enforce \
  my-tenant/my-ns --enable

Subscription Types

Type	Description	Use Case
`Exclusive`	Single consumer per subscription	Ordered processing
`Shared`	Round-robin across consumers	Parallel processing
`Failover`	Active-standby consumers	High availability
`Key_Shared`	Partition by message key	Ordered per-key processing

Advanced Usage

Pulsar Functions

# Deploy a function
bin/pulsar-admin functions create \
  --function-name my-func \
  --inputs persistent://my-tenant/my-ns/input-topic \
  --output persistent://my-tenant/my-ns/output-topic \
  --jar my-function.jar \
  --classname com.example.MyFunction

# List functions
bin/pulsar-admin functions list --tenant my-tenant --namespace my-ns

# Get function status
bin/pulsar-admin functions status \
  --tenant my-tenant --namespace my-ns --name my-func

# Delete a function
bin/pulsar-admin functions delete \
  --tenant my-tenant --namespace my-ns --name my-func

Tiered Storage (Offloading)

# Configure S3 offloader in broker.conf
# managedLedgerOffloadDriver=aws-s3
# s3ManagedLedgerOffloadBucket=pulsar-offload
# s3ManagedLedgerOffloadRegion=us-east-1

# Set offload threshold on namespace
bin/pulsar-admin namespaces set-offload-threshold \
  my-tenant/my-ns --size 10G

# Trigger manual offload
bin/pulsar-admin topics offload \
  persistent://my-tenant/my-ns/my-topic -s 10G

Geo-Replication

# Enable replication on namespace
bin/pulsar-admin namespaces set-clusters my-tenant/my-ns \
  --clusters us-east,eu-west

# Check replication status
bin/pulsar-admin topics stats persistent://my-tenant/my-ns/my-topic \
  | jq '.replication'

Monitoring

# Broker metrics endpoint
curl http://localhost:8080/metrics

# Key metrics to watch
# pulsar_broker_topics_count
# pulsar_subscription_back_log
# pulsar_throughput_in / pulsar_throughput_out
# pulsar_storage_size
# pulsar_msg_backlog
# bookkeeper_server_ADD_ENTRY_REQUEST (bookie write latency)

Troubleshooting

Issue	Solution
Broker fails to start	Check ZooKeeper connectivity; verify `clusterName` matches metadata
Messages not consumed	Verify subscription exists; check consumer subscription type
Backlog growing	Scale consumers; check consumer errors in logs; increase parallelism
Topic creation fails	Verify tenant/namespace exists; check authorization
High publish latency	Check BookKeeper health; ensure sufficient bookie nodes
Out of disk	Configure tiered storage offloading; adjust retention policies
Schema compatibility error	Check schema compatibility strategy; use `BACKWARD` for safe evolution
Geo-replication lag	Monitor replication throughput; check cross-region network latency