Event streaming

Streaming is a feature where client code can subscribe to event notifications. A subscription is, essentially, a connection to Fauna that is held open. Change notification messages are sent over the connection as they occur.

There are two kinds of event streaming:

  1. Document streaming

    Client code subscribes to a document reference. Whenever the referenced document is created, updated, or deleted, an event notification is immediately sent to the subscriber code.

  2. Set streaming

    Client code subscribes to a set reference, such as:

    • Documents(Collection("spells"))
      the set of documents in the spells collection.

    • Match(Index("elements_of_spells"))
      the set of documents in the term-less index elements_of_spells.

    • Match(Index("spells_by_element"), "fire")
      the set of spells documents whose element field matches fire.

    Whenever one or more documents enter or exit the set, via creation, deletion, or field value change (when the set is defined by an index’s terms or values fields), an event notification is immediately sent to the subscriber code.

Streaming is a much better alternative to the standard approach of polling. Polling occurs when client code repeatedly issues queries to the database at regular intervals to discover document updates. With pay-as-you-go pricing, polling is the much more expensive alternative, and your code is only aware of changes when query results are returned.

Polling

A sequence diagram demonstrating the communications during database polling

Streaming

A sequence diagram demonstrating the communications during database streaming

The polling diagram demonstrates that the client has to execute many more queries in order to discover when a document has been updated. For a streaming client, the subscription happens once and events are automatically broadcast to the client whenever the subscribed document is updated.

There is a cost in compute operations to hold a stream open, or to repeatedly start a stream that fails.

See Billing for details.
Streaming works with HTTP/1.x, but HTTP/2 is much more efficient so use that if your client environment supports it.

Events

Fauna streaming uses a protocol inspired by HTTP Server-Sent Events (SSE). Fauna streams events for a subscribed document to the client, keeping the connection open (where possible) to minimize transmission delays. Client event handling (in supported drivers) is similar to WebSockets, however streams are unidirectional: the client cannot send events to the server via the stream.

Similar to the SSE protocol, events are communicated over a text-based channel. Each event is formatted as a single-line JSON object. Unlike SSE, Fauna adds an additional line ending, \r\n, to delimit payloads, which helps with JSON parsing when network middleware splits the event payload into multiple packets.

Events are communicated one at a time regardless of volume.

Document streaming events

Here is an example document streaming event, with the JSON formatted for easy identification of the structure:

{
  "action": "update", (1)
  "document": { (2)
    "ref": {
      "@ref": {
        "id": "1",
        "collection": {
          "@ref": {
            "id": "Scores",
            "collection": {
              "@ref": {
                "id": "collections"
              }
            }
          }
        }
      }
    },
    "ts": 1643910577140000,
    "data": {
      "scores": [
        1,
        11,
        17
      ],
      "foo": "bar",
      "final": true
    }
  },
  "diff": { (3)
    "ref": {
      "@ref": {
        "id": "1",
        "collection": {
          "@ref": {
            "id": "Scores",
            "collection": {
              "@ref": {
                "id": "collections"
              }
            }
          }
        }
      }
    },
    "ts": 1643910577140000,
    "data": {
      "foo": "bar",
      "final": true
    }
  },
  "prev": { (4)
    "ref": {
      "@ref": {
        "id": "1",
        "collection": {
          "@ref": {
            "id": "Scores",
            "collection": {
              "@ref": {
                "id": "collections"
              }
            }
          }
        }
      }
    },
    "ts": 1643910545600000,
    "data": {
      "scores": [
        1,
        11,
        17
      ]
    }
  }
}
1 The action field, showing that this event is an update.
2 The document field, showing the document definition for the event.
3 The diff field, showing the difference between the document and prev fields. In this case, we see that the final field has been added with the value of true.
4 The prev field, showing the previous document definition prior to the current event.

The outermost structure is a "metadata" wrapper for the event, which contains the fields:

Document streaming event fields

  • type: the type of event payload. One of:

    • start: An event marking the start of the stream.

      Use the txn field as the stream’s starting timestamp. txn values for other events in the same stream are guaranteed to come after the start event’s txn.

      Two streams for the same document are not guaranteed to have the same txn timestamp.

    • version: An event containing information about a given document.

    • error: An event in response to an error with the stream.

    • history_rewrite: An event containing information about a historical change, such as when the subscribed document’s history is revised.

  • txn: the timestamp of the transaction emitting the event.

  • event: a value describing the particular event.

    For the start event type, the value is a timestamp.

    For other event types, the value is an object that contains the fields:

    • action: the type of event. One of:

      • create: Occurs when a document is created.

      • update: Occurs when an existing document is updated.

      • delete: Occurs when an existing document is deleted.

    • document: An object containing the subscribed document’s details. For update events, only the modified fields are included.

      The document's ts field is the document’s timestamp expressed as a Long. It is often the same as the event wrapper’s txn field, but it is not guaranteed to be identical.

When you establish a stream, you can opt-in to receive additional fields in the event object:

  • prev: Provides the event’s previous data.

  • diff: Provides the difference between the prev field and the document field.

The methods to respond to events differ in each driver:

Set streaming events

The notification events you receive from a set stream are generated by changes in the set to which you have subscribed. Events coming from a set stream differ from events coming from a document stream in that set streaming events include references to the document which has changed, rather than what has changed within a collection document itself. You may use the document references returned in the set streaming event to query for the changed document if needed.

Here is an example set streaming event, with the JSON formatted for easy identification of the structure:

{
  type: "set",
  txn: 1646743802330000,
  event: {
    action: "add", (1)
    document: { (2)
     ref: ref(id = "325567069374382153", collection = ref( id = "Scores", collection = ref(id = "collections"))),
     ts: 1646743802330000
    },
    index: { (3)
      terms: [
        ref(id = "Scores", collection = ref(id = "collections"))
      ],
      values: []
    }
  }
}
1 The action field, showing that this event was triggered by the addition of a new document to the set.
2 The document field, showing the document definition for the event.
3 The index field, showing the set whose change caused this notification to be sent.

Set streaming event fields

  • type: the type of event payload. One of:

    • start: An event marking the start of the stream.

      Use the txn field as the stream’s starting timestamp. txn values for other events in the same stream are guaranteed to come after the start event’s txn.

      Two streams for the same document are not guaranteed to have the same txn timestamp.

    • error: An event in response to an error with the stream.

    • set: An event containing information about a change to a set’s membership.

  • txn: the timestamp of the transaction emitting the event.

  • event: a value describing the particular event.

    For the start event type, the value is a timestamp.

    For other event types, the value is an object that contains the fields:

    • action: the type of event. One of:

      • add: Occurs when a document is added to a set.

      • remove: Occurs when a document is removed from a set.

    • document: An object containing the details of a member document in the set.

      The document's ts field is the document’s timestamp expressed as a Long. It is often the same as the event wrapper’s txn field, but it is not guaranteed to be identical.

When you establish a stream, you can opt-in to receive additional fields in the event object:

  • index: Provides the definition of the involved index, if any (not all sets are index-based).

The methods to respond to events differ in each driver:

Tips

  • Avoid running a query to fetch a document and then establishing a stream. Multiple events may have modified the document prior to stream startup, which can lead to inaccurate representation of the document data in your application.

    For the JavaScript driver, you can use the document helper, which takes care of this problem for you.

Limitations

For the initial release of streaming, the following limitation exist:

  • Active stream count:

    • Only 100 simultaneous streams per browser. Browsers manage the number of concurrent HTTP2 streams using a hard-coded limit. No matter how many windows or tabs are open, you cannot exceed 100 streams simultaneously.

    • Using a driver, you can have more than 100 streams active at once by creating additional connection objects: each connection supports up to 100 streams.

    • There may be other limits based on each host language’s HTTP2 implementation but we have not encountered those yet.

  • A document stream only reports events for the fields and values within the document’s data field.

  • No support for GraphQL subscriptions is available.

Is this article helpful? 

Tell Fauna how the article can be improved:
 Visit Fauna's forums or email docs@fauna.com

Thank you for your feedback!