Pubsub

Pub/Sub is a messaging service for exchanging event data among applications and services. A producer of data publishes a messages to a Pub/Sub topic. A consumer creates a subscription to that topic. From this point on, Pub/Sub guarantees that the message will be delivered to every consumer of the message at least once. Consumers either pull messages from a subscription or are configured as webhooks for push subscriptions. Every subscriber must acknowledge each message within a configurable time window. Unacknowledged messages are redelivered.

Pub/Sub is geographically global and does not require sharding or additional configuration to scale with demand.

Google Cloud Pub/Sub brings the scalability, flexibility, and reliability of enterprise message-oriented middleware to the cloud. By providing many-to-many, asynchronous messaging that decouples senders and receivers, it allows for secure and highly available communication between independently written applications. Google Cloud Pub/Sub delivers low-latency, durable messaging that helps developers quickly integrate systems hosted on the Google Cloud Platform and externally.

Basic concepts

Benefits and features

• Unified messaging: Durability and low-latency delivery in a single product
• Global presence: Connect services located anywhere in the world
• Flexible delivery options: Both push- and pull-style subscriptions supported
• Data reliability: Replicated storage and guaranteed at-least-once message delivery
• End-to-end reliability: Explicit application-level acknowledgement
• Data security and protection: Encryption of data on the wire and at rest
• Flow control: Dynamic rate limiting implemented by the Pub/Sub system
• Simplicity: Easy-to-use REST/JSON API

Here are some classic use cases for Google Cloud Pub/Sub:

• Balancing workloads in network clusters. For example, a large queue of tasks can be efficiently distributed among multiple workers, such as Google Compute Engine instances.
• Implementing asynchronous workflows. For example, an order processing application can place an order on a topic, from which it can be processed by one or more workers.
• Distributing event notifications. For example, a service that accepts user signups can send notifications whenever a new user registers, and downstream services can subscribe to receive notifications of the event.
• Refreshing distributed caches. For example, an application can publish invalidation events to update the IDs of objects that have changed.
• Logging to multiple systems. For example, a Google Compute Engine instance can write logs to the monitoring system, to a database for later querying, and so on.
• Data streaming from various processes or devices. For example, a residential sensor can stream data to backend servers hosted in the cloud.
• Reliability improvement. For example, a single-zone Compute Engine service can operate in additional zones by subscribing to a common topic, to recover from failures in a zone or region.

The Basics of a Publish/Subscribe Service

Google Cloud Pub/Sub is a publish/subscribe(Pub/Sub)service: a messaging service where the senders of messages are decoupled from the receivers of messages. There are several key concepts in a Pub/Sub service:

• Message: the data that moves through the service.

• Topic: a named entity that represents a feed of messages.

• Subscription: a named entity that represents an interest in receiving messages on a particular topic.

• Publisher(also called a producer): creates messages and sends (publishes) them to the messaging service on a specified topic.

• Subscriber(also called a consumer): receives messages on a specified subscription.

A publisher application creates and sends messages to a topic. Subscriber applications create a _subscription _to a topic to receive messages from it. Communication can be one-to-many (fan-out), many-to-one (fan-in), and many-to-many.

Example:

The basic flow of messages through Google Cloud Pub/Sub can be summarized in the following diagram:

In this scenario, there are two publishers publishing messages on a single topic. There are two subscriptions to the topic, where the first subscription has two subscribers, and the second subscription has one subscriber. The bold letters represent messages. Message A comes from Publisher 1 and is sent to Subscriber 2 via Subscription 1, and to Subscriber 3 via Subscription 2. Message B comes from Publisher 2 and is sent to Subscriber 1 via Subscription 1 and to Subscriber 3 via Subscription 2.

Components of Pubsub

Here is an overview of the components in the Pub/Sub system and how messages flow between them:

1. A _publisher _application creates a _topic _in the Google Cloud Pub/Sub service and sends _messages _to the topic. A message contains a payload and optional _attributes _that describe the payload content.
2. Messages are persisted in a _message store _until they are delivered and acknowledged by subscribers.
3. The Pub/Sub service forwards messages from a topic to all of its subscriptions, individually. Each subscription receives messages either by Pub/Sub _pushing _them to the subscriber's chosen endpoint, or by the subscriber _pulling _them from the service.
4. The subscriber receives pending messages from its subscription and acknowledges each one to the Pub/Sub service.
5. When a message is acknowledged by the subscriber, it is removed from the subscription's queue of messages.

Publisher and subscriber endpoints

Publishers can be any application that can make HTTPS requests to googleapis.com: an App Engine app, a web service hosted on Google Compute Engine or any other third-party network, an installed app for desktop or mobile device, or even a browser.

Pull subscribers can also be any application that can make HTTPS requests to googleapis.com. Currently, push subscribers must be Webhook endpoints that can accept POST requests over HTTPS.

Pubsub resources and data models

The following are the types of objects used by the Pub/Sub API. Topics and Subscriptions are the only resource types exposed as REST collections.

Topic

A named resource to which messages are sent by publishers.

Subscription

A named resource representing the stream of messages from a single, specific topic, to be delivered to the subscribing application. For more details about subscriptions and message delivery semantics, see the Subscriber Guide.

Message

The combination of data and (optional) attributes that a publisher sends to a topic and is eventually delivered to subscribers.

Message attribute

A key-value pair that a publisher can define for a message. For example, key iana.org/language_tagand value encould be added to messages to mark them as readable by an English-speaking subscriber.

Pub/Sub topic and subscription names must be scoped to a project:

projects/project-identifier/collection/resource-name

• The collection must be one of subscriptions or topics.
• The project-identifier must be the project ID, available from the Google Cloud Platform Console. For example, projects/myproject/topics/mytopic.
• The resource-name must start with a letter, and contain only letters ([A-Za-z]), numbers ([0-9]), dashes (-), underscores (_), periods (.), tildes (~), plus (+) or percent signs (%). It must be between 3 and 255 characters in length, and cannot begin with the stringgoog.

This format lets you use URL encoding to include special characters in the resource name. For example, a topic name mi-tópico, which is an invalid resource name for Cloud Pub/Sub, could be encoded as mi-t%C3%B3pico, which is valid.

Service architecture and internals

Performance of a Messaging Service

A messaging service like Google Cloud Pub/Sub can be judged on its performance in three aspects: scalability, availability, and latency. These three factors are often at odds with each other, requiring compromises on one to improve the other two.

The terms "scalability," “availability,” and “latency” can refer to different properties of a system, so the following sections describe how they are defined in Google Cloud Pub/Sub.

Scalability

A scalable service should be able to handle increases in load without noticeable degradation of latency or availability. "Load" can refer to various dimensions of usage in Google Cloud Pub/Sub:

• Number of topics

• Number of publishers

• Number of subscriptions

• Number of subscribers

• Number of messages

• Size of messages

• Rate of messages (throughput) published or consumed

• Size of backlog on any given subscription

Availability

In a distributed system, the types and severity of problems can vary greatly. A system’s availability is measured on how well it deals with different types of issues, gracefully failing over in a way that is unnoticeable to end users. Failures can occur in hardware (e.g., disk drives not working or network connectivity problems), in software, and due to load. Failure due to load could happen when a sudden increase in traffic in the service (or in other software components running on the same hardware or in software dependencies) results in resource scarcity. Availability can also degrade due to human error, where one makes mistakes in building or deploying software or configurations.

Latency

Latency is a time-based measure of the performance of a system. A service generally wants to minimize latency wherever possible. For Google Cloud Pub/Sub, the two most important latency metrics are:

1. The amount of time it takes to acknowledge a published message.

2. The amount of time it takes to deliver a published message to a subscriber.

Pubsub architecture

This section explains the design of Google Cloud Pub/Sub to show how the service attains its scalability and low latency while retaining availability. The system is designed to behorizontally scalable, where an increase in the number of topics, subscriptions, or messages can be handled by increasing the number of instances of running servers.

• Google Cloud Pub/Sub’s servers run in multiple Google data centers, which are distributed around the world. Each data center contains one or more instances of a cluster, a logical grouping of machines that generally share the same failure domain (e.g., shared local network and shared power).
• Google Cloud Pub/Sub is a global service: The clients are unaware of the physical location (or data center location) of any servers or data and can publish and subscribe from anywhere in the world to anywhere in the world.
• Google Cloud Pub/Sub is divided into two primary parts: the data plane, which handles moving messages between publishers and subscribers, and the control plane, which handles the assignment of publishers and subscribers to servers on the data plane.
• The servers in the data plane are called forwarders, and the servers in the control plane are called routers. When publishers and subscribers are connected to their assigned forwarders, they do not need any information from the routers (as long as those forwarders remain accessible). Therefore, it is possible to upgrade the control plane of Google Cloud Pub/Sub without affecting any clients that are already connected and sending or receiving messages.

Control Plane

The Google Cloud Pub/Sub control plane distributes clients to forwarders in a way that provides scalability, availability, and low latency for all clients.

• Any forwarder is capable of serving clients for any topic or subscription. When a client connects to Google Cloud Pub/Sub, the router decides the data centers the client should connect to based on shortest network distance, a measure of the latency on the connection between two points.
• Within any given data center the router tries to distribute overall load across the set of available forwarders. The router must balance two different goals when performing this assignment: (a) uniformity of load (i.e., ideally every forwarder is equally loaded); and (b) stability of assignments (i.e., ideally a change in load or a change in the set of available forwarders changes the smallest number of existing assignments). The router uses a variant of consistent hashing developed by Google Research to achieve a tunable balance between consistency and uniformity.
• The router provides the client with an ordered list of forwarders it can consider connecting to. This ordered list may change based on forwarder availability and the shape of the load from the client. A client takes this list of forwarders and connects to one or more of them. The client prefers connecting to the forwarders most recommended by the router, but also takes into consideration any failures that have occurred, e.g., it may decide to try forwarders in a different data center if several attempts to the nearest ones have failed.

In order to abstract Google Cloud Pub/Sub clients away from these implementation details, there is a service proxy between the clients and forwarders that performs this connection optimization on behalf of clients.

Data Plane - The Life of a Message

The data plane receives messages from and sends them to clients. Perhaps the best way of understanding Google Cloud Pub/Sub’s data plane is by looking at the life of a message, from the moment it is received by the service to the moment it is no longer present in the service. Let us trace the steps of processing a message. For the purposes of this section, we assume that the topic on which the message is published has at least one subscription attached to it. In general, a message goes through these steps:

1. A publisher sends a message.

2. The message is written to storage.

3. Google Cloud Pub/Sub sends an acknowledgement to the publisher that it has received the message and guarantees its delivery to all attached subscriptions.

4. At the same time as writing the message to storage, Google Cloud Pub/Sub delivers it to subscribers.

5. Subscribers send an acknowledgement to Google Cloud Pub/Sub that they have processed the message.

6. Once at least one subscriber for each subscription has acknowledged the message, Google Cloud Pub/Sub deletes the message from storage.

The life of a message is fairly complex with several connections involved to send messages from publishers to subscribers. The flow of messages through connections among publishers, subscribers, and forwarders is as follows:

Getting started with Cloud Pubsub

Using the gcloud command-line tool

You can use the gcloud command-line tool to perform operations in Google Cloud Pub/Sub. See the gcloud pubsub reference for a complete list of Google Cloud Pub/Sub gcloud commands.

This example shows the steps required to create a topic, subscribe to it, publish a message to it, and receive the message:

gcloud init
gcloud components install beta
gcloud beta pubsub topics create myTopic
gcloud beta pubsub subscriptions create --topic myTopic mySubscription
gcloud beta pubsub topics publish myTopic "hello"
gcloud beta pubsub subscriptions pull --auto-ack mySubscription

This example uses a pull subscription. Pub/Sub also supports push subscriptions, as described in the Subscriber guide.

You can perform all administrative operations through the API, and most through the gcloud tool and Cloud Platform Console UI.

Using the API

For data operations, like publishing and consuming messages, you will probably write your own code, as described below. This section shows you how to programmatically publish and consume messages.

In Python

Publish a message to the topic:

def publish_message(topic_name, data):
    """Publishes a message to a Pub/Sub topic with the given data."""
    pubsub_client = pubsub.Client()
    topic = pubsub_client.topic(topic_name)

    # Data must be a bytestring
    data = data.encode('utf-8')

    message_id = topic.publish(data)

    print('Message {} published.'.format(message_id))

Pull a message from a topic:

def receive_message(topic_name, subscription_name):
    """Receives a message from a pull subscription."""
    pubsub_client = pubsub.Client()
    topic = pubsub_client.topic(topic_name)
    subscription = topic.subscription(subscription_name)

    # Change return_immediately=False to block until messages are
    # received.
    results = subscription.pull(return_immediately=True)

    print('Received {} messages.'.format(len(results)))

    for ack_id, message in results:
        print('* {}: {}, {}'.format(
            message.message_id, message.data, message.attributes))

    # Acknowledge received messages. If you do not acknowledge, Pub/Sub will
    # redeliver the message.
    if results:
        subscription.acknowledge([ack_id for ack_id, message in results])

In Java

Publish a message to the topic:

TopicName topicName = TopicName.create("my-project-id", "my-topic-id");
Publisher publisher = null;
List<ApiFuture<String>> messageIdFutures = new ArrayList<>();

try {
  // Create a publisher instance with default settings bound to the topic
  publisher = Publisher.defaultBuilder(topicName).build();

  List<String> messages = Arrays.asList("first message", "second message");

  // schedule publishing one message at a time : messages get automatically batched
  for (String message : messages) {
    ByteString data = ByteString.copyFromUtf8(message);
    PubsubMessage pubsubMessage = PubsubMessage.newBuilder().setData(data).build();

    // Once published, returns a server-assigned message id (unique within the topic)
    ApiFuture<String> messageIdFuture = publisher.publish(pubsubMessage);
    messageIdFutures.add(messageIdFuture);
  }
} finally {
  // wait on any pending publish requests.
  List<String> messageIds = ApiFutures.allAsList(messageIdFutures).get();

  for (String messageId : messageIds) {
    System.out.println("published with message ID: " + messageId);
  }

  if (publisher != null) {
    // When finished with the publisher, shutdown to free up resources.
    publisher.shutdown();
  }
}

Pull a message from a topic:

String projectId = "my-project-id";
String subscriptionId = "my-subscription-id";

SubscriptionName subscriptionName = SubscriptionName.create(projectId, subscriptionId);
// Instantiate an asynchronous message receiver
MessageReceiver receiver = new MessageReceiver() {
      @Override
      public void receiveMessage(PubsubMessage message, AckReplyConsumer consumer) {
        // handle incoming message, then ack/nack the received message
        System.out.println("Id : " + message.getMessageId());
        System.out.println("Data : " + message.getData().toStringUtf8());
        consumer.ack();
      }
    };

Subscriber subscriber = null;
try {
  // Create a subscriber for "my-subscription-id" bound to the message receiver
  subscriber = Subscriber.defaultBuilder(subscriptionName, receiver).build();
  subscriber.startAsync();
  // ...
} finally {
  // stop receiving messages
  if (subscriber != null) {
    subscriber.stopAsync();
  }
}