Commit Scopes v5

Commit Scopes give applications granular control about durability and consistency of EDB Postgres Distributed.

A Commit Scope is a named rule that describes behavior of COMMIT replication. The actual behavior depends on whether a Commit Scope uses Group Commit, Commit At Most Once, Lag Control or combination of these.

Configuration

To use Group Commit, first define a commit scope. This determines the BDR nodes involved in the commit of a transaction. Once a scope is established, you can configure a transaction to use Group Commit as follows:

BEGIN;
SET LOCAL bdr.commit_scope = 'example_scope';
...
COMMIT;

The commit scope must be set before the transaction has written any data.

For this example, you might previously have defined the commit scope as:

SELECT bdr.add_commit_scope(
    commit_scope_name := 'example_scope',
    origin_node_group := 'example_bdr_group',
    rule := 'ANY 2 (example_bdr_group)' GROUP COMMIT,
    wait_for_ready := true
);

This assumes a node group named example_bdr_group exists and includes at least two BDR nodes as members, either directly or in subgroups. Any transaction committed in the example_scope requires one extra confirmation from a BDR node in the group. Together with the origin node, this accounts for "ANY 2" nodes out of the group, on which the transaction is guaranteed to be visible and durable after the commit.

Origin groups

Rules for commit scopes can depend on the node the transaction is committed on, that is, the node that acts as the origin for the transaction. To make this transparent for the application, BDR allows a commit scope to define different rules depending on where the transaction originates from.

For example, consider a EDB Postgres Distributed cluster with nodes spread across two data centers: a left and a right one. Assume the top-level BDR node group is called top_group. You can use the following commands to set up subgroups and create a commit scope requiring all nodes in the local data center to confirm the transaction but only one node from the remote one.

-- create sub-groups
SELECT bdr.create_node_group(
    node_group_name := 'left_dc',
    parent_group_name := 'top_group',
    join_node_group := false
);
SELECT bdr.create_node_group(
    node_group_name := 'right_dc',
    parent_group_name := 'top_group',
    join_node_group := false
);

-- create a commit scope with individual rules
-- for each sub-group
SELECT bdr.add_commit_scope(
    commit_scope_name := 'example_scope',
    origin_node_group := 'left_dc',
    rule := 'ALL (left_dc) GROUP COMMIT AND ANY 1 (right_dc) GROUP COMMIT',
    wait_for_ready := true
);
SELECT bdr.add_commit_scope(
    commit_scope_name := 'example_scope',
    origin_node_group := 'right_dc',
    rule := 'ANY 1 (left_dc) GROUP COMMIT AND ALL (right_dc) GROUP COMMIT',
    wait_for_ready := true
);

Now using the example_scope on any node that's part of left_dc will use the first scope, while using same scope on node that's part of right_dc will use the second scope. This is effective way of creating inverted scope without having to juggle scope names in application.

In addition to this, each group can also have default commit scope specified using bdr.alter_node_group_option admin interface.

So making the above scopes the default ones for all transactions originatin on nodes withing those groups would look like this.

SELECT bdr.alter_node_group_option(
  node_group_name := 'left_dc',
  config_key := 'default_commit_scope',
  config_value := 'example_scope'
);
SELECT bdr.alter_node_group_option(
  node_group_name := 'right_dc',
  config_key := 'default_commit_scope',
  config_value := 'example_scope'
);

Confirmation levels

BDR nodes can send confirmations for a transaction at different points in time. In increasing levels of protection, from the perspective of the confirming node, these are:

• received — A remote BDR node confirms the transaction immediately after receiving it, prior to starting the local application.
• replicated — Confirm after applying changes of the transaction but before flushing them to disk.
• durable — Confirm the transaction after all of its changes are flushed to disk.
• visible (default) — Confirm the transaction after all of its changes are flushed to disk and it's visible to concurrent transactions.

In rules for commit scopes, you can append these confirmation levels to the node group definition in parenthesis with ON as follows:

• ANY 2 (right_dc) ON replicated
• ALL (left_dc) ON visible (default and may as well be omitted)
• ALL (left_dc) ON received AND ANY 1 (right_dc) ON durable

Reference

Commit scope grammar

For reference, the grammar for commit scopes is composed as follows:

commit_scope:
    confirmation [AND ...]

commit_scope_operation:
    commit_scope_group [ ON { received|replicated|durable|visible } ] commit_scope_kind

commit_scope_group:
{ ANY num (node_group [, ...])
  | MAJORITY (node_group [, ...])
  | ALL (node_group [, ...]) }

commit_scope_kind:
{ GROUP COMMIT [ ( group_commit_parameter = value ) ] [ ABORT ON ( abort_on_parameter = value ) ]
  | CAMO [ DEGRADE ON ( degrade_on_parameter = value ) TO ASYNC ]
  | LAG CONTROL [ ( lag_control_parameter = value ) ] }

Parameters

• node_group - name of a node group
• ( group_commit_parameter = value ) - options for Group Commit
  • transaction_tracking (boolean) - specifies whether status of transaction should be tracked
  • conflict_resolution (enum) - how to handle conflicts, possible values are either async meaning conflicts should be resolved asynchronously after during replication using the conflict resolution policy or eager meaning that conflicts are resolved eagerly during COMMIT by aborting one of the conflicting transactions
  • commit_decision (enum) - how is COMMIT decision made, it can be either group meaning the commit_scope_group specification also affects the COMMIT decision, not just durability, it can also be partner which means partner node decides whether transaction can be committed (this is only allowed on 2 data node groups) or it can be raft which means COMMIT decision is done using Raft consensus independently of commit_scope_group consensus.
• ABORT ON ( abort_on_parameter = value ) - allows automatic transaction abort on timeout
  • timeout (interval) - timeout in milliseconds (accepts other units)
• DEGRADE ON ( degrade_on_parameter = value ) - allows degrading to asynchronous operation on timeout
  • timeout (interval) - timeout in milliseconds (accepts other units) after which operation becomes asynchronous
  • require_write_lead (boolean) - whether the node has to be a write lead to to be able to switch to asynchronous mode
• ( lag_control_parameter = value ) - options for Lag Control
  • max_lag_size (int) - maximum allowed lag based on WAL bytes
  • max_lag_time (interval) - maximum allowed lag based on wall clock sampling
  • max_commit_delay (interval) - maximum delay that can be injected to commit in order to try to keep within the lag limits

Adding a commit scope rule

The function bdr.add_commit_scope creates a rule for the given commit scope name and origin node group. If the rule is the same for all nodes in the EDB Postgres Distributed cluster, invoking this function once for the top-level node group is enough to fully define the commit scope.

Alternatively, you can invoke it multiple times with the same commit_scope_name but different origin node groups and rules for commit scopes that vary depending on the origin of the transaction.

Synopsis

bdr.add_commit_scope(
    commit_scope_name NAME,
    origin_node_group NAME,
    rule TEXT
    wait_for_ready boolean DEFAULT true)

Changing a commit scope rule

To change a specific rule for a single origin node group in a commit scope, you can use the function bdr.alter_commit_scope.

Synopsis

bdr.alter_commit_scope(
    commit_scope_name NAME,
    origin_node_group NAME,
    rule TEXT)

Removing a commit scope rule

You can use bdr.remove_commit_scope to drop a single rule in a commit scope. If you define multiple rules for the commit scope, you must invoke this function once per rule to fully remove the entire commit scope.

Synopsis

bdr.remove_commit_scope(
    commit_scope_name NAME,
    origin_node_group NAME)

• On this page
• Configuration
• Reference