<sect1 id="example-cluster">
 <title><productname>Pgpool-II</productname> + Watchdog Setup Example</title>
 <para>
  This section shows an example of streaming replication configuration using
  <productname>Pgpool-II</productname>. In this example, we use 3
  <productname>Pgpool-II</productname> servers to manage <productname>PostgreSQL</productname>
  servers to create a robust cluster system and avoid the single point of failure or split brain.
 </para>
 <para>
  <emphasis><productname>PostgreSQL</productname> 18</emphasis> is used in this configuration example.
 </para>
 <sect2 id="example-cluster-requirement">
  <title>Requirements</title>
  <para>
   We assume that all the Pgpool-II servers and the <productname>PostgreSQL</productname> servers are in the same subnet.
  </para>
 </sect2>

 <sect2 id="example-cluster-structure">
  <title>Cluster System Configuration</title>
  <para>
   We use three servers with <emphasis>Rocky Linux 10</emphasis> installed and
   the hostnames of the three servers are <literal>server1</literal>
   <literal>server2</literal> and <literal>server3</literal> respectively.
   We install <productname>PostgreSQL</productname> and <productname>Pgpool-II</productname> on each server.
  </para>
  <para>
   <figure>
    <title>Cluster System Configuration</title>
    <mediaobject>
     <imageobject>
      <imagedata fileref="cluster_40.gif">
     </imageobject>
    </mediaobject>
   </figure>
  </para>
  <note>
   <para>
    The roles of <literal>Leader</literal>, <literal>Standby</literal>, <literal>Primary</literal>,
    <literal>Standby</literal> are not fixed and may be changed by further operations.
   </para>
  </note>
  <table id="example-cluster-table-ip">
   <title>Hostname and IP address</title>
   <tgroup cols="3">
    <thead>
     <row>
      <entry>Hostname</entry>
      <entry>IP Address</entry>
      <entry>Virtual IP</entry>
     </row>
    </thead>
    <tbody>
     <row>
      <entry>server1</entry>
      <entry>192.168.100.51</entry>
      <entry morerows="2">192.168.100.50</entry>
     </row>
     <row>
      <entry>server2</entry>
      <entry>192.168.100.52</entry>
     </row>
     <row>
      <entry>server3</entry>
      <entry>192.168.100.53</entry>
     </row>
    </tbody>
   </tgroup>
  </table>

  <table id="example-cluster-table-postgresql-config">
   <title>PostgreSQL version and Configuration</title>
   <tgroup cols="3">
    <thead>
     <row>
      <entry>Item</entry>
      <entry>Value</entry>
      <entry>Detail</entry>
     </row>
    </thead>
    <tbody>
     <row>
      <entry>PostgreSQL Version</entry>
      <entry>18.0</entry>
      <entry>-</entry>
     </row>
     <row>
      <entry>port</entry>
      <entry>5432</entry>
      <entry>-</entry>
     </row>
     <row>
      <entry>$PGDATA</entry>
      <entry>/var/lib/pgsql/18/data</entry>
      <entry>-</entry>
     </row>
     <row>
      <entry>Archive mode</entry>
      <entry>off</entry>
      <entry>-</entry>
     </row>
     <row>
      <entry>Replication Slots</entry>
      <entry>Enabled</entry>
      <entry>In this configuration example, replication slots are
       automatically created or deleted in the scripts which are executed
       during failover or online recovery. 
       These scripts use the hostname specified in backend_hostnameX as
       the replication slot name.
       See <xref linkend="example-cluster-table-sample-scripts"> for
       more information about the scripts.</entry>
     </row>
     <row>
      <entry>Async/Sync Replication</entry>
      <entry>Async</entry>
      <entry>-</entry>
     </row>
    </tbody>
   </tgroup>
  </table>

  <table id="example-cluster-table-pgpool-config">
   <title>Pgpool-II version and Configuration</title>
   <tgroup cols="3">
    <thead>
     <row>
      <entry>Item</entry>
      <entry>Value</entry>
      <entry>Detail</entry>
     </row>
    </thead>
    <tbody>
     <row>
      <entry>Pgpool-II Version</entry>
      <entry>4.7.0</entry>
      <entry>-</entry>
     </row>
     <row>
      <entry morerows='3'>port</entry>
      <entry>9999</entry>
      <entry>Pgpool-II accepts connections</entry>
     </row>
     <row>
      <entry>9898</entry>
      <entry>PCP process accepts connections</entry>
     </row>
     <row>
      <entry>9000</entry>
      <entry>watchdog accepts connections</entry>
     </row>
     <row>
      <entry>9694</entry>
      <entry>UDP port for receiving Watchdog's heartbeat signal</entry>
     </row>
     <row>
      <entry>Config file</entry>
      <entry>/etc/pgpool-II/pgpool.conf</entry>
      <entry>Pgpool-II config file</entry>
     </row>
     <row>
      <entry>User running Pgpool-II</entry>
      <entry>postgres (Pgpool-II 4.1 or later)</entry>
      <entry>Pgpool-II 4.0 or before, the default user running Pgpool-II is root</entry>
     </row>
     <row>
      <entry>Running mode</entry>
      <entry>streaming replication mode</entry>
      <entry>-</entry>
     </row>
     <row>
      <entry>Watchdog</entry>
      <entry>on</entry>
      <entry>Life check method: heartbeat</entry>
     </row>
    </tbody>
   </tgroup>
  </table>

  <table id="example-cluster-table-sample-scripts">
   <title>Various sample scripts included in rpm package</title>
   <tgroup cols="3">
    <thead>
     <row>
      <entry>Feature</entry>
      <entry>Script</entry>
      <entry>Detail</entry>
     </row>
    </thead>
    <tbody>
     <row>
      <entry morerows='1'>Failover</entry>
      <entry><ulink url="https://raw.githubusercontent.com/pgpool/pgpool2/refs/heads/master/src/sample/scripts/failover.sh.sample">/etc/pgpool-II/sample_scripts/failover.sh.sample</ulink></entry>
      <entry>Run by <xref linkend="GUC-FAILOVER-COMMAND"> to perform failover</entry>
     </row>
     <row>
      <entry><ulink url="https://raw.githubusercontent.com/pgpool/pgpool2/refs/heads/master/src/sample/scripts/follow_primary.sh.sample">/etc/pgpool-II/sample_scripts/follow_primary.sh.sample</ulink></entry>
      <entry>Run by <xref linkend="GUC-FOLLOW-PRIMARY-COMMAND"> to synchronize the Standby with the new Primary after failover.</entry>
     </row>
     <row>
      <entry morerows='1'>Online recovery</entry>
      <entry><ulink url="https://raw.githubusercontent.com/pgpool/pgpool2/refs/heads/master/src/sample/scripts/recovery_1st_stage.sample">/etc/pgpool-II/sample_scripts/recovery_1st_stage.sample</ulink></entry>
      <entry>Run by <xref linkend="GUC-RECOVERY-1ST-STAGE-COMMAND"> to recovery a Standby node</entry>
     </row>
     <row>
      <entry><ulink url="https://raw.githubusercontent.com/pgpool/pgpool2/refs/heads/master/src/sample/scripts/pgpool_remote_start.sample">/etc/pgpool-II/sample_scripts/pgpool_remote_start.sample</ulink></entry>
      <entry>Run after <xref linkend="GUC-RECOVERY-1ST-STAGE-COMMAND"> to start the Standby node</entry>
     </row>
     <row>
      <entry morerows='1'>Watchdog</entry>
      <entry><ulink url="https://raw.githubusercontent.com/pgpool/pgpool2/refs/heads/master/src/sample/scripts/escalation.sh.sample">/etc/pgpool-II/sample_scripts/escalation.sh.sample</ulink></entry>
      <entry>Optional Configuration. Run by <xref linkend="guc-wd-escalation-command"> to switch the Leader/Standby Pgpool-II safely</entry>
     </row>
    </tbody>
   </tgroup>
  </table>
  <para>
   The above scripts are included in the RPM package and can be customized as needed.
  </para>
 </sect2>

 <sect2 id="example-cluster-installation">
  <title>Installation</title>
  <para>
   In this example, we install <productname>Pgpool-II</productname> and <productname>PostgreSQL</productname> RPM packages with YUM.
  </para>

  <para>
   Install <productname>PostgreSQL</productname> from <productname>PostgreSQL</productname> YUM repository.
  </para>
  <programlisting>
[all servers]# dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-10-x86_64/pgdg-redhat-repo-latest.noarch.rpm
[all servers]# dnf -qy module disable postgresql
[all servers]# dnf install -y postgresql18-server
  </programlisting>
  <para>
   Since <productname>Pgpool-II</productname> related packages are also included in <productname>PostgreSQL</productname> YUM repository,
   add the "exclude" settings to <filename>/etc/yum.repos.d/pgdg-redhat-all.repo</filename>
   so that <productname>Pgpool-II</productname> is not installed from <productname>PostgreSQL</productname> YUM repository. 
  </para>
  <programlisting>
[all servers]# vi /etc/yum.repos.d/pgdg-redhat-all.repo
  </programlisting>
  <para>
   The following is a setting example of <filename>/etc/yum.repos.d/pgdg-redhat-all.repo</filename>. 
  </para>
  <programlisting>
[pgdg-common]
...
exclude=pgpool*

[pgdg18]
...
exclude=pgpool*

[pgdg17]
...
exclude=pgpool*

[pgdg16]
...
exclude=pgpool*

[pgdg15]
...
exclude=pgpool*

[pgdg14]
...
exclude=pgpool*

[pgdg13]
...
exclude=pgpool*
  </programlisting>

  <para>
   Install <productname>Pgpool-II</productname> from
   <productname>Pgpool-II</productname> YUM repository.
   To install <productname>Pgpool-II</productname> on Rocky Linux 10,
   you need to enable <literal>epel</literal> repository and install libmemcached-awesome-devel to satisfy dependencies.
  </para>
  <programlisting>
[all servers]# dnf install -y https://www.pgpool.net/yum/rpms/4.7/redhat/rhel-10-x86_64/pgpool-II-release-4.7-1.noarch.rpm
[all servers]# dnf install -y epel-release
[all servers]# dnf install -y libmemcached-awesome-devel
[all servers]# dnf install -y pgpool-II-pg18-*
  </programlisting>
  <para>
   If you're using <literal>Rocky Linux 9</literal>,
   use <literal>crb</literal> repository insted of <literal>epel</literal>.
   For <literal>Rocky Linux 8</literal>,
   use <literal>powertools</literal> repository instead.
  </para>
  <programlisting>
[all servers]# dnf install -y --enablerepo=crb pgpool-II-pg18-*
  </programlisting>
 </sect2>

 <sect2 id="example-cluster-pre-setup">
  <title>Before Starting</title>
  <para>
   Before you start the configuration process, please check the following prerequisites.
  </para>

  <sect3 id="example-cluster-before-starting-ssh">
   <title>Setting up SSH public key authentication</title>
   <para>
    To use the automated failover and online recovery of <productname>Pgpool-II</productname>, 
    it is required to configure <emphasis>SSH public key authentication
    (passwordless SSH login)</emphasis> to all backend servers using
    <literal>postgres</literal> user (the default user Pgpool-II is running as.
    Pgpool-II 4.0 or before, the default user is <literal>root</literal>).
   </para>
   <para>
    Execute the following command on all servers to generate a key pair using
    the RSA algorithm. In this example, we assume that the generated key file
    name is <literal>id_rsa_pgpool</literal>.
   </para>
   <programlisting>
[all servers]# su - postgres
[all servers]$ mkdir ~/.ssh && chmod 700 ~/.ssh
[all servers]$ ssh-keygen -t rsa -f ~/.ssh/id_rsa_pgpool
   </programlisting>
   <para>
    Then add the public key <filename>id_rsa_pgpool.pub</filename> to
    <filename>/var/lib/pgsql/.ssh/authorized_keys</filename> file
    on each server.
   </para>
   <para>
    After setting SSH, make sure that you can run
    <command>ssh postgres@serverX -i ~/.ssh/id_rsa_pgpool</command> command
    as <literal>postgres</literal> user to login to each server
    without entering a password. 
   </para>

   <note>
    <para>
     If you failed to login using SSH public key authentication, please check the following:
     <itemizedlist>
      <listitem>
       <para>
        Ensure the public key authentication option <literal>PubkeyAuthentication</literal> are allowed in <filename>/etc/ssh/sshd_config</filename>:
       </para>
      </listitem>
     </itemizedlist>
     <programlisting>
PubkeyAuthentication yes
     </programlisting>
     <itemizedlist>
      <listitem>
       <para>
        If SELinux is enabled, SSH public key authentication (passwordless SSH) may fail.
        You need to run the following command on all servers.
       </para>
      </listitem>
     </itemizedlist>
    <programlisting>
[all servers]# su - postgres
[all servers]$ restorecon -Rv ~/.ssh
    </programlisting>
    </para>
   </note>
  </sect3>

  <sect3 id="example-cluster-before-starting-firewall">
   <title>Setting up firewall</title>
   <para>
    When connect to <productname>Pgpool-II</productname> and <productname>PostgreSQL</productname> servers, the target port must be accessible by enabling firewall management softwares. Following is an example for <systemitem>Rocky Linux 10/RHEL 10</systemitem>.
   </para>
   <programlisting>
[all servers]# firewall-cmd --permanent --zone=public --add-service=postgresql
[all servers]# firewall-cmd --permanent --zone=public --add-port=9999/tcp --add-port=9898/tcp --add-port=9000/tcp  --add-port=9694/udp
[all servers]# firewall-cmd --reload
   </programlisting>
  </sect3>

  <sect3 id="example-cluster-before-starting-primary">
   <title>Setting up PostgreSQL primary</title>
   <para>
    Set up the PostgreSQL primary server on <literal>server1</literal>.
   </para>

   <para>
    First, initialize the database cluster. 
    In <productname>PostgreSQL</productname> 18, the default directory
    for creating the database cluster is <filename>/var/lib/pgsql/18/data</filename>,
    and this is specified in the environment variable <varname>PGDATA</varname>
    in the <filename>~/.bash_profile</filename> of the <literal>postgres</literal>
    user. Modify it if you want to change the default path.
   </para>
   <para>
    Run <command>initdb</command> to initialize the database cluster.
    Specify <ulink url="https://www.postgresql.org/docs/18/app-initdb.html">options</ulink>
    such as <literal>--encoding</literal> and <literal>--locale</literal>
    if you need to configure the database encoding and locale settings.
   </para>
   <programlisting>
[root@server1 ~]# su - postgres
[postgres@server1 ~]$ /usr/pgsql-18/bin/initdb
    </programlisting>

    <para>
     Then edit the configuration file <filename>$PGDATA/postgresql.conf</filename>
     on <literal>server1</literal> as follows. Enable <literal>wal_log_hints</literal>
     to use <literal>pg_rewind</literal>. 
    </para>
    <programlisting>
[postgres@server1 ~]$ vi $PGDATA/postgresql.conf
listen_addresses = '*'
wal_log_hints = on
    </programlisting>
    <para>
     Start <productname>PostgreSQL</productname> on <literal>server1</literal>.
    </para>
   <programlisting>
[postgres@server1 ~]$ /usr/pgsql-18/bin/pg_ctl start
    </programlisting>
  </sect3>

  <sect3 id="example-cluster-before-starting-standby">
   <title>Setting up PostgreSQL standby</title>
   <para>
    There are multiple methods to setup a standby server, such as:
    <itemizedlist>
     <listitem>
      <para>
       use <productname>Pgpool-II</productname>'s online recovery feature
       (<xref linkend="runtime-online-recovery">) to automatically
       setup a standby server.
      </para>
     </listitem>
     <listitem>
      <para>
       use <command>pg_basebackup</command> to backup the data directory of the primary from the standby.
      </para>
     </listitem>
    </itemizedlist>
   </para>

   <note>
    <para>
     If you manually set up standby without using online recovery, please note the following:
     <itemizedlist>
      <listitem>
       <para>
        You must manually create the replication slots.
       </para>
      </listitem>
     </itemizedlist>
     <itemizedlist>
      <listitem>
       <para>
        Do not specify the -R option when running pg_basebackup.
        Write <varname>primary_conninfo</varname>, <varname>recovery_target_timeline</varname>, and <varname>primary_slot_name</varname> in <filename>$PGDATA/myrecovery.conf</filename>, not in <filename>postgresql.auto.conf</filename>, and add the following to <filename>postgresql.conf</filename>.
       </para>
      </listitem>
     </itemizedlist>
     <programlisting>
include_if_exists = 'myrecovery.conf'
     </programlisting>
    </para>
   </note>

   <para>
    In this example, we use <productname>Pgpool-II</productname>'s
    online recovery to setup the standby server 
    in section <xref linkend="example-cluster-verify-standby">
    after the configuration of <productname>Pgpool-II</productname>
    is completed.
   </para>
  </sect3>

  <sect3 id="example-cluster-before-starting-users">
   <title>Setting up PostgreSQL users</title>
   <para>
    A PostgreSQL user is required to use <productname>Pgpool-II</productname>'s
    health check and replication delay check features.
    Because of the security reasons, we create a dedicated user named
    <literal>pgpool</literal> for streaming replication delay check and
    health check.
    And create a dedicated user named <literal>repl</literal> for replication.
    Because online recovery feature requires superuser privilege,
    we use <literal>postgres</literal> user here.
    </para>
    <para>
     Since <productname>Pgpool-II</productname> 4.0,
     <acronym>scram-sha-256</acronym> authentication is supported. 
     This configuration example uses <acronym>scram-sha-256</acronym>
     authentication method. 
     First, set <literal>password_encryption = 'scram-sha-256'</literal>
     and then created the users.
    </para>

    <table id="example-cluster-user">
     <title>Users</title>
     <tgroup cols="3">
      <thead>
       <row>
	    <entry>User Name</entry>
	    <entry>Password</entry>
	    <entry>Detail</entry>
       </row>
      </thead>
      <tbody>
       <row>
	    <entry>repl</entry>
	    <entry>repl</entry>
	    <entry>PostgreSQL replication user</entry>
       </row>
       <row>
	    <entry>pgpool</entry>
	    <entry>pgpool</entry>
	    <entry>Pgpool-II health check (<xref linkend="GUC-HEALTH-CHECK-USER">) and replication delay check (<xref linkend="GUC-SR-CHECK-USER">) user</entry>
       </row>
       <row>
	<entry>postgres</entry>
	<entry>postgres</entry>
	<entry>User running online recovery</entry>
       </row>
      </tbody>
     </tgroup>
    </table>

    <programlisting>
[postgres@server1 ~]$ psql
postgres=# SET password_encryption = 'scram-sha-256';
postgres=# CREATE ROLE pgpool WITH LOGIN;
postgres=# CREATE ROLE repl WITH REPLICATION LOGIN;
postgres=# \password pgpool
postgres=# \password repl
postgres=# \password postgres
    </programlisting>

    <para>
     To show <literal>replication_state</literal> and
     <literal>replication_sync_state</literal> column in
     <xref linkend="SQL-SHOW-POOL-NODES"> command result, role
     <literal>pgpool</literal> needs to be PostgreSQL superuser or in
     <literal>pg_monitor</literal> group (<productname>Pgpool-II</productname> 4.1 or later).
     Grant <literal>pg_monitor</literal>
     to <literal>pgpool</literal>:
    </para>
    <programlisting>
postgres=# GRANT pg_monitor TO pgpool;
postgres=# \q
    </programlisting>
    <note>
     <para>
      If you plan to use <xref linkend="guc-detach-false-primary">(<productname>Pgpool-II</productname> 4.0 or later),
      role "pgpool" needs to be <productname>PostgreSQL</productname> superuser
      or in <literal>pg_monitor</literal> group to use this feature.
     </para>
    </note>
    <para>
     In this example, assuming that all the <productname>Pgpool-II</productname> servers and the 
     <productname>PostgreSQL</productname> servers are in the same subnet and edit <filename>pg_hba.conf</filename> to 
     enable <literal>scram-sha-256</literal> authentication method.
    </para>
    <programlisting>
[postgres@server1 ~]$ vi $PGDATA/pg_hba.conf
(Add the following entries)
host    all             pgpool             samenet                 scram-sha-256
host    all             postgres           samenet                 scram-sha-256
host    replication     repl               samenet                 scram-sha-256
    </programlisting>
  </sect3>

  <sect3 id="example-cluster-before-starting-pgpass">
   <title>Creating .pgpass</title>
   <para>
    To allow <literal>repl</literal> user without specifying password for streaming 
    replication and online recovery, and execute <application>pg_rewind</application>
    using <literal>postgres</literal>, we
    Create the <filename>.pgpass</filename> file in <literal>postgres</literal>
    user's home directory and change the permission to <literal>600</literal>
    on each <productname>PostgreSQL</productname> server.
    This file allows <literal>repl</literal> user and <literal>postgres</literal>
    user without providing a password for streaming replication and failover.
   </para>
   <programlisting>
[postgres@server1 ~]$ vi ~/.pgpass
server1:5432:replication:repl:&lt;repl user password&gt;
server2:5432:replication:repl:&lt;repl user password&gt;
server3:5432:replication:repl:&lt;repl user password&gt;
server1:5432:postgres:postgres:&lt;postgres user password&gt;
server2:5432:postgres:postgres:&lt;postgres user password&gt;
server3:5432:postgres:postgres:&lt;postgres user password&gt;

[postgres@server1 ~]$ chmod 600 ~/.pgpass
   </programlisting>
   <para>
    Copy it to the home directory of <literal>postgres</literal> user
    on <literal>server2</literal> and <literal>server3</literal>.
   </para>
   <programlisting>
[postgres@server1 ~]$ scp -i ~/.ssh/id_rsa_pgpool -p ~/.pgpass postgres@server2:
[postgres@server1 ~]$ scp -i ~/.ssh/id_rsa_pgpool -p ~/.pgpass postgres@server3:
   </programlisting>
  </sect3>
 </sect2>

 <sect2 id="pcp-authentication">
  <title>PCP connection authentication</title>
  <para>
   To use PCP commands PCP user names and md5 encrypted passwords must be
   declared in <filename>pcp.conf</filename> in format
   "<literal>username:encrypted password</literal>".
  </para>
  <para>
   In this example, we set PCP username to "<literal>pgpool</literal>"
   and password to "<literal>pgpool_password</literal>".
   Use <xref linkend="PG-MD5"> to create the encrypted password entry for
   <literal>pgpool</literal> user as below:
  </para>
  <programlisting>
[postgres@server1 ~]$ echo 'pgpool:'`pg_md5 pgpool_password` &gt;&gt; /etc/pgpool-II/pcp.conf

[postgres@server1 ~]$ cat /etc/pgpool-II/pcp.conf
# USERID:MD5PASSWD
pgpool:4aa0cb9673e84b06d4c8a848c80eb5d0
  </programlisting>
  <para>
   Since <filename>follow_primary.sh</filename> script mentioned
   later must execute PCP command without entering a password,
   create <filename>.pcppass</filename> in
   <literal>postgres</literal> user's home directory
   (the home directory of the user Pgpool-II is running as).
   The format of <filename>.pcppass</filename> is
   "<literal>hostname:port:username:password</literal>".
  </para>
  <para>
   In this example, we assume that the PCP user is <literal>pgpool</literal>
   and the password is <literal>pgpool_password</literal>.
  </para>
  <programlisting>
[postgres@server1 ~]$ echo 'localhost:9898:pgpool:pgpool_password' &gt; ~/.pcppass
[postgres@server1 ~]$ chmod 600 ~/.pcppass
  </programlisting>
  <para>
   Copy to <literal>server2</literal> and <literal>server3</literal>.
  </para>
  <programlisting>
[postgres@server1 ~]$ scp -i ~/.ssh/id_rsa_pgpool -p ~/.pcppass postgres@server2:
[postgres@server1 ~]$ scp -i ~/.ssh/id_rsa_pgpool -p ~/.pcppass postgres@server3:
[postgres@server1 ~]$ exit
  </programlisting>
 </sect2>

 <sect2 id="example-cluster-pgpool-node-id">
  <title>Create pgpool_node_id</title>
  <para>
    From <productname>Pgpool-II</productname> 4.2, now all configuration parameters are identical on all hosts.
    If <literal>watchdog</literal> feature is enabled, to distinguish which host is which,
    a <filename>pgpool_node_id</filename> file is required.
    You need to create a <filename>pgpool_node_id</filename> file and specify the pgpool (watchdog) node number
    (e.g. 0, 1, 2 ...) to identify pgpool (watchdog) host.
  </para>
  <itemizedlist>
   <listitem>
    <para>
     <literal>server1</literal>
    </para>
    <programlisting>
[root@server1 ~]# echo 0 &gt; /etc/pgpool-II/pgpool_node_id
[root@server1 ~]# cat /etc/pgpool-II/pgpool_node_id
0
    </programlisting>
   </listitem>
   <listitem>
    <para>
     <literal>server2</literal>
    </para>
    <programlisting>
[root@server2 ~]# echo 1 &gt; /etc/pgpool-II/pgpool_node_id
[root@server2 ~]# cat /etc/pgpool-II/pgpool_node_id
1
    </programlisting>
   </listitem>
   <listitem>
    <para>
     <literal>server3</literal>
    </para>
    <programlisting>
[root@server3 ~]# echo 2 &gt; /etc/pgpool-II/pgpool_node_id
[root@server3 ~]# cat /etc/pgpool-II/pgpool_node_id
2
    </programlisting>
   </listitem>
  </itemizedlist>
 </sect2>

 <sect2 id="example-cluster-pgpool-config">
  <title><productname>Pgpool-II</productname> Configuration</title>
  <para>
   When installing <productname>Pgpool-II</productname> using YUM, the
   <productname>Pgpool-II</productname> configuration file <filename>pgpool.conf</filename>
   is installed in <filename>/etc/pgpool-II</filename>.
  </para>
  <para>
   Since from <productname>Pgpool-II</productname> 4.2, all configuration parameters are
   identical on all hosts, you can edit <filename>pgpool.conf</filename> on any pgpool node
   and copy the edited <filename>pgpool.conf</filename> file to the other pgpool nodes.
  </para>

  <sect3 id="example-cluster-pgpool-config-config-file">
   <title>Clustering mode</title>
   <para>
    <productname>Pgpool-II</productname> has several clustering modes. To set the clustering
    mode, <xref linkend="GUC-BACKEND-CLUSTERING-MODE"> can be used. In this configuration
    example, streaming replication mode is used.
   </para>
   <programlisting>
[root@server1 ~]# vi /etc/pgpool-II/pgpool.conf 
backend_clustering_mode = streaming_replication
   </programlisting>
  </sect3>

  <sect3 id="example-cluster-pgpool-config-listen-addresses">
   <title>listen_addresses</title>
   <para>
    To allow Pgpool-II and PCP to accept all incoming connections, set the following
    parameters to <literal>'*'</literal>.
   </para>
   <programlisting>
listen_addresses = '*'
pcp_listen_addresses = '*'
   </programlisting>
  </sect3>

  <sect3 id="example-cluster-pgpool-config-port">
   <title>port</title>
   <para>
    Specify the port number Pgpool-II listen on.
   </para>
   <programlisting>
port = 9999
   </programlisting>
  </sect3>

  <sect3 id="example-cluster-pgpool-config-sr-check">
   <title>Streaming Replication Check</title>
   <para>
    Specify replication delay check user and password in <xref linkend="GUC-SR-CHECK-USER">
    and <xref linkend="GUC-SR-CHECK-PASSWORD">. In this example, we leave
    <xref linkend="GUC-SR-CHECK-PASSWORD"> empty, and create the entry in
    <xref linkend="GUC-POOL-PASSWD">. See <xref linkend="example-cluster-pgpool-config-auth">
    for how to create the entry in <xref linkend="GUC-POOL-PASSWD">.
    From <productname>Pgpool-II</productname> 4.0, if these parameters are left blank,
    <productname>Pgpool-II</productname> will first try to get the password for that
    specific user from <xref linkend="GUC-POOL-PASSWD"> file before using the empty password.
   </para>
   <programlisting>
sr_check_user = 'pgpool'
sr_check_password = ''
   </programlisting>
  </sect3>

  <sect3 id="example-cluster-pgpool-config-health-check">
   <title>Health Check</title>
   <para>
    Enable health check so that <productname>Pgpool-II</> performs failover. Also, if the network is unstable,
    the health check fails even though the backend is running properly, failover or degenerate operation may occur.
    In order to prevent such incorrect detection of health check, we set <varname>health_check_max_retries = 3</varname>.
    Specify <xref linkend="GUC-HEALTH-CHECK-USER"> and <xref linkend="GUC-HEALTH-CHECK-PASSWORD"> in
      the same way like <xref linkend="GUC-SR-CHECK-USER"> and <xref linkend="GUC-SR-CHECK-PASSWORD">.
   </para>
   <programlisting>
health_check_period = 5
health_check_timeout = 30
health_check_user = 'pgpool'
health_check_password = ''
health_check_max_retries = 3
   </programlisting>
  </sect3>

  <sect3 id="example-cluster-pgpool-config-backend-settings">
   <title>Backend Settings</title>
   <para>
    Specify the <productname>PostgreSQL</productname> backend information.
    Multiple backends can be specified by adding a number at the end of the parameter name.
   </para>
   <programlisting>
backend_hostname0 = 'server1'
backend_port0 = 5432
backend_weight0 = 1
backend_data_directory0 = '/var/lib/pgsql/18/data'
backend_flag0 = 'ALLOW_TO_FAILOVER'

backend_hostname1 = 'server2'
backend_port1 = 5432
backend_weight1 = 1
backend_data_directory1 = '/var/lib/pgsql/18/data'
backend_flag1 = 'ALLOW_TO_FAILOVER'

backend_hostname2 = 'server3'
backend_port2 = 5432
backend_weight2 = 1
backend_data_directory2 = '/var/lib/pgsql/18/data'
backend_flag2 = 'ALLOW_TO_FAILOVER'
   </programlisting>
   <para>
    To show <literal>replication_state</literal> and
    <literal>replication_sync_state</literal> column in
    <xref linkend="SQL-SHOW-POOL-NODES"> command result,
    <xref linkend="GUC-BACKEND-APPLICATION-NAME"> parameter is required.
    Here we specify each backend's hostname in these parameters
    (<productname>Pgpool-II</productname> 4.1 or later).
    Make sure that the value set in <varname>backend_application_nameX</varname>
    matches the value set in <varname>application_name</varname>
    of <varname>primary_conninfo</varname>.
   </para>
   <programlisting>
backend_application_name0 = 'server1'
backend_application_name1 = 'server2'
backend_application_name2 = 'server3'
   </programlisting>
  </sect3>

  <sect3 id="example-cluster-pgpool-config-failover">
   <title>Failover configuration</title>
   <para>
    Specify the script that will be executed when failover occurs in
    <xref linkend="GUC-FAILOVER-COMMAND">. When using three or more
    PostgreSQL servers, it's required to specify
    <xref linkend="GUC-FOLLOW-PRIMARY-COMMAND"> to synchronize the standby
    with the new primary. In case of two PostgreSQL servers, the setting of
    <xref linkend="GUC-FOLLOW-PRIMARY-COMMAND"> is not required.
   </para>
   <note>
    <para>
     When performing a switchover using <xref linkend="PCP-PROMOTE-NODE">
     with <option>switchover</option> option added in <productname>Pgpool-II</productname> 4.3,
     if you want to turn the former primary into standby automatically,
     <xref linkend="GUC-FOLLOW-PRIMARY-COMMAND"> is required to be set
     even if there are two PostgreSQL servers.
    </para>
   </note>
   <para>
    <productname>Pgpool-II</productname> replaces the following special characters with the backend specific
    information while executing the scripts. 
    See <xref linkend="GUC-FAILOVER-COMMAND"> and <xref linkend="GUC-FOLLOW-PRIMARY-COMMAND"> for more details about each character.
   </para>
   <programlisting>
failover_command = '/etc/pgpool-II/failover.sh %d %h %p %D %m %H %M %P %r %R %N %S'
follow_primary_command = '/etc/pgpool-II/follow_primary.sh %d %h %p %D %m %H %M %P %r %R'
   </programlisting>
   <note>
    <para>
     <emphasis>%N</emphasis> and <emphasis>%S</emphasis> are added in <productname>Pgpool-II</productname> 4.1.
     Please note that these characters cannot be specified if using Pgpool-II 4.0 or earlier.
    </para>
   </note>
   <para>
    Sample scripts <ulink url="https://raw.githubusercontent.com/pgpool/pgpool2/refs/heads/master/src/sample/scripts/failover.sh.sample">failover.sh</ulink>
    and <ulink url="https://raw.githubusercontent.com/pgpool/pgpool2/refs/heads/master/src/sample/scripts/follow_primary.sh.sample">follow_primary.sh</ulink>
    are installed in <filename>/etc/pgpool-II/</filename>. Create failover scripts using these sample files.
   </para>
   <programlisting>
[root@server1 ~]# cp -p /etc/pgpool-II/sample_scripts/failover.sh.sample /etc/pgpool-II/failover.sh
[root@server1 ~]# cp -p /etc/pgpool-II/sample_scripts/follow_primary.sh.sample /etc/pgpool-II/follow_primary.sh
[root@server1 ~]# chown postgres:postgres /etc/pgpool-II/{failover.sh,follow_primary.sh}
   </programlisting>
   <para>
    Basically, it should work if you change <emphasis>PGHOME</emphasis> according to
    PostgreSQL installation directory.
   </para>
   <programlisting>
[root@server1 ~]# vi /etc/pgpool-II/failover.sh
...
PGHOME=/usr/pgsql-18
...

[root@server1 ~]# vi /etc/pgpool-II/follow_primary.sh
...
PGHOME=/usr/pgsql-18
...
   </programlisting>

   <para>
    Make sure the entry of the PCP user specified in <varname>PCP_USER</varname> in
    <filename>follow_primary.sh</filename> is created in <filename>pcp.conf</filename>.
    In this example, we have created in <xref linkend="PCP-AUTHENTICATION">
   </para>
   <programlisting>
[root@server1 ~]# cat /etc/pgpool-II/follow_primary.sh
...
PCP_USER=pgpool
...
   </programlisting>
   <note>
    <para>
     The <filename>follow_primary.sh</filename> script does not support tablespaces.
     If you are using tablespaces, you need to modify the script to support tablespaces.
    </para>
   </note>
  </sect3>

  <sect3 id="example-cluster-pgpool-config-online-recovery">
   <title>Pgpool-II Online Recovery Configurations</title>
   <para>
    Next, configure the required parameters to perform online recovery
    (<xref linkend="runtime-online-recovery">).
    Because <emphasis>Superuser</emphasis> privilege in <productname>PostgreSQL</productname>
    is required for performing online recovery, we specify <literal>postgres</literal>
    user in <xref linkend="GUC-RECOVERY-USER">. In this example, we leave
    <xref linkend="GUC-RECOVERY-PASSWORD"> empty, and create the entry in
    <xref linkend="GUC-POOL-PASSWD">. See <xref linkend="example-cluster-pgpool-config-auth">
    for how to create the entry in <xref linkend="GUC-POOL-PASSWD">.
   </para>
   <programlisting>
recovery_user = 'postgres'
recovery_password = ''
recovery_1st_stage_command = 'recovery_1st_stage'
   </programlisting>
   <para>
    Then, we create <filename>recovery_1st_stage</filename> and <filename>pgpool_remote_start</filename>
    in database cluster directory of <productname>PostgreSQL</productname>
    primary server (server1).
   </para>
   <para>
    The sample scripts of online recovery <ulink url="https://raw.githubusercontent.com/pgpool/pgpool2/refs/heads/master/src/sample/scripts/pgpool_remote_start.sample">recovery_1st_stage</ulink>
    and <ulink url="https://raw.githubusercontent.com/pgpool/pgpool2/refs/heads/master/src/sample/scripts/pgpool_remote_start.sample">pgpool_remote_start</ulink>
    are installed in <filename>/etc/pgpool-II/</filename>. Copy these files to the data directory of the primary server (server1).
   </para>
   <programlisting>
[root@server1 ~]# su - postgres -c "cp -p /etc/pgpool-II/sample_scripts/recovery_1st_stage.sample \$PGDATA/recovery_1st_stage"
[root@server1 ~]# su - postgres -c "cp -p /etc/pgpool-II/sample_scripts/pgpool_remote_start.sample \$PGDATA/pgpool_remote_start"
   </programlisting>
   <para>
    Basically, it should work if you change <emphasis>PGHOME</emphasis> according to PostgreSQL installation directory.
   </para>
   <programlisting>
[root@server1 ~]# vi /var/lib/pgsql/18/data/recovery_1st_stage
...
PGHOME=/usr/pgsql-18
...

[root@server1 ~]# vi /var/lib/pgsql/18/data/pgpool_remote_start
...
PGHOME=/usr/pgsql-18
...
   </programlisting>

   <para>
    In addition, to perform online recovery, the functions
    <function>pgpool_recovery</function>, <function>pgpool_remote_start</function>
    and <function>pgpool_switch_xlog</function> must be created in advance
    in the database specified by <xref linkend="guc-recovery-database">
    (the default is <literal>'postgres'</literal>).
    Execute the following command to create these functions in
    <literal>postgres</literal> on <literal>server1</literal>.
    If a value other than the default is configured for
    <xref linkend="guc-recovery-database">, replace <literal>postgres</literal>
    with the configured value.
   </para>
   <programlisting>
[root@server1 ~]# psql -U postgres postgres -c "CREATE EXTENSION pgpool_recovery"
   </programlisting>
   <note>
    <para>
     The <filename>recovery_1st_stage</filename> script does not support tablespaces.
     If you are using tablespaces, you need to modify the script to support tablespaces.
    </para>
   </note>
  </sect3>

  <sect3 id="example-cluster-pgpool-config-auth">
   <title>Client Authentication Configuration</title>
   <para>
    Because in the section <link linkend="EXAMPLE-CLUSTER-PRE-SETUP">Before Starting</link>,
    we already set <productname>PostgreSQL</productname> authentication method to
    <acronym>scram-sha-256</acronym>, it is necessary to set a client authentication by
    <productname>Pgpool-II</productname> to connect to backend nodes.
    When installing with RPM, the <productname>Pgpool-II</productname> configuration file
    <filename>pool_hba.conf</filename> is in <filename>/etc/pgpool-II</filename>.
    By default, pool_hba authentication is disabled, set <varname>enable_pool_hba = on</varname>
    to enable it.
   </para>
   <programlisting>
enable_pool_hba = on
   </programlisting>
   <para>
    The format of <filename>pool_hba.conf</filename> file follows
    PostgreSQL's <filename>pg_hba.conf</filename> format very closely.
    Set <literal>pgpool</literal> and <literal>postgres</literal> user's
    authentication method to <literal>scram-sha-256</literal>. In this example,
    it is assumed that the application connecting to <productname>Pgpool-II</productname>
    is in the same subnet.
   </para>
   <programlisting>
[root@server1 ~]# vi /etc/pgpool-II/pool_hba.conf
(Add the following entries)
host    all         pgpool           samenet          scram-sha-256
host    all         postgres         samenet          scram-sha-256
   </programlisting>
   <note>
    <para>
     Please note that in <productname>Pgpool-II</productname> 4.0 only AES encrypted password or clear text password
     can be specified in <xref linkend="guc-health-check-password">, <xref linkend="guc-sr-check-password">, 
       <xref linkend="guc-wd-lifecheck-password">, <xref linkend="guc-recovery-password"> in <filename>pgpool.conf</filename>.
    </para>
   </note>
   <para>
    The default password file name for authentication is <xref linkend="GUC-POOL-PASSWD">.
     To use <literal>scram-sha-256</literal> authentication, the decryption key to
     decrypt the passwords is required. We create the <literal>.pgpoolkey</literal>
     file in <literal>postgres</literal> user's home directory
     (the user Pgpool-II is running as. <productname>Pgpool-II</productname> 4.0 or before,
     <productname>Pgpool-II</productname> is running as <literal>root</literal> by default)
     <programlisting>
[root@server1 ~]# su - postgres -c "echo 'some string' > ~/.pgpoolkey"
[root@server1 ~]# su - postgres -c "chmod 600 ~/.pgpoolkey"
[root@server1 ~]# su - postgres -c "scp -i ~/.ssh/id_rsa_pgpool -p ~/.pgpoolkey postgres@server2:"
[root@server1 ~]# su - postgres -c "scp -i ~/.ssh/id_rsa_pgpool -p ~/.pgpoolkey postgres@server3:"
     </programlisting>
   </para>
   <para>
    Execute command <command>pg_enc -m -k /path/to/.pgpoolkey -u username -p</command> to register user
    name and <literal>AES</literal> encrypted password in file <filename>pool_passwd</filename>.
    If <filename>pool_passwd</filename> doesn't exist yet, it will be created in the same directory as
    <filename>pgpool.conf</filename>.
   </para>
   <programlisting>
[root@server1 ~]# pg_enc -m -k /var/lib/pgsql/.pgpoolkey -u pgpool -p
db password: (Enter the password for pgpool user)
trying to read key from file /var/lib/pgsql/.pgpoolkey

[root@server1 ~]# pg_enc -m -k /var/lib/pgsql/.pgpoolkey -u postgres -p
db password: (Enter the password for postgres user)
trying to read key from file /var/lib/pgsql/.pgpoolkey

[root@server1 ~]# cat /etc/pgpool-II/pool_passwd 
pgpool:AESheq2ZMZjynddMWk5sKP/Rw==
postgres:AESHs/pWL5rtXy2IwuzroHfqg==
   </programlisting>
  </sect3>

  <sect3 id="example-cluster-pgpool-config-watchdog">
   <title>Watchdog Configuration</title>
   <para>
    Enable watchdog functionality on <literal>server1</literal>, <literal>server2</literal>, <literal>server3</literal>.
   </para>
   <programlisting>
use_watchdog = on
   </programlisting>
   <para>
    Set virtual IP address to <xref linkend="GUC-DELEGATE-IP">.
    Ensure that the IP address isn't used yet.
   </para>
   <programlisting>
delegate_ip = '192.168.100.50'
   </programlisting>

   <para>
    To bring up/down the virtual IP and send the ARP requests,
    we set <xref linkend="GUC-IF-UP-CMD">, <xref linkend="GUC-IF-DOWN-CMD"> and <xref linkend="GUC-ARPING-CMD">.  
    Set the netmask and network interface name according to your network environment.
    The network interface used in this example is <literal>enp0s8</literal>.
    Since root privilege is required to execute <varname>if_up/down_cmd</varname> or
    <varname>arping_cmd</varname> command, use setuid on these command or allow
    <literal>postgres</literal> user (the user Pgpool-II is running as) to run
    <command>sudo</command> command without a password.
   </para>
   <note>
    <para>
    If <productname>Pgpool-II</productname> is installed using RPM, the <literal>postgres</literal>
    user has been configured to run <command>ip/arping</command> via <command>sudo</command> without
    a password.
    <programlisting>
postgres ALL=NOPASSWD: /sbin/ip
postgres ALL=NOPASSWD: /usr/sbin/arping
    </programlisting>
    </para>
   </note>
   <para>
    Here we configure the following parameters to run <varname>if_up/down_cmd</varname> or <varname>arping_cmd</varname> with sudo.
   </para>
   <programlisting>
if_up_cmd = '/usr/bin/sudo /sbin/ip addr add $_IP_$/24 dev enp0s8 label enp0s8:0'
if_down_cmd = '/usr/bin/sudo /sbin/ip addr del $_IP_$/24 dev enp0s8'
arping_cmd = '/usr/bin/sudo /usr/sbin/arping -U $_IP_$ -w 1 -I enp0s8'
   </programlisting>
   <note>
    <para>
     If "Defaults requiretty" is set in the <filename>/etc/sudoers</filename>,
     please ensure that the user that <productname>Pgpool-II</productname> is running as can execute the <command>if_up_cmd</command>, <command>if_down_cmd</command> and <command>arping_cmd</command> command without a tty.
    </para>
   </note>
   <para>
    Set <xref linkend="GUC-IF-CMD-PATH"> and <xref linkend="GUC-ARPING-PATH"> according to the
    command path.
    If <varname>if_up/down_cmd</varname> or <varname>arping_cmd</varname> starts with "/", these parameters will be ignored. 
   </para>
   <programlisting>
if_cmd_path = '/sbin'
arping_path = '/usr/sbin'
   </programlisting>
   <para>
    Specify all <productname>Pgpool-II</productname> nodes information for configuring watchdog.
    Specify <varname>pgpool_portX</varname> using the port number specified in <varname>port</varname> in
    <xref linkend="example-cluster-pgpool-config-port">.
   </para>
   <programlisting>
hostname0 = 'server1'
wd_port0 = 9000
pgpool_port0 = 9999

hostname1 = 'server2'
wd_port1 = 9000
pgpool_port1 = 9999

hostname2 = 'server3'
wd_port2 = 9000
pgpool_port2 = 9999
   </programlisting>
   <para>
    Configure the method of lifecheck <xref linkend="guc-wd-lifecheck-method">
    and the lifecheck interval <xref linkend="guc-wd-interval">.
    Here, we use <literal>heartbeat</literal> method to perform watchdog lifecheck.
   </para>
   <programlisting>
wd_lifecheck_method = 'heartbeat'
wd_interval = 10
   </programlisting>
   <para>
    Specify all <productname>Pgpool-II</productname> nodes information for sending and receiving heartbeat signal.
   </para>
   <programlisting>
heartbeat_hostname0 = 'server1'
heartbeat_port0 = 9694
heartbeat_device0 = ''
heartbeat_hostname1 = 'server2'
heartbeat_port1 = 9694
heartbeat_device1 = ''
heartbeat_hostname2 = 'server3'
heartbeat_port2 = 9694
heartbeat_device2 = ''
   </programlisting>
   <para>
    If <xref linkend="guc-wd-lifecheck-method"> is set to <literal>heartbeat</literal>,
    specify the time to detect a fault <xref linkend="guc-wd-heartbeat-deadtime"> and
    the interval to send heartbeat signals <xref linkend="guc-wd-heartbeat-deadtime">.
   </para>
   <programlisting>
wd_heartbeat_keepalive = 2
wd_heartbeat_deadtime = 30
   </programlisting>

   <para>
     This setting is optional.
     When <literal>Watchdog</literal> process is abnormally terminated,
     the virtual IP may be "up" on both of the old and new active pgpool nodes.
     To prevent this, configure <xref linkend="guc-wd-escalation-command">
     to bring down the virtual IP on other Pgpool-II nodes before
     bringing up the virtual IP on the new leader Pgpool-II node.
   </para>
    <programlisting>
wd_escalation_command = '/etc/pgpool-II/escalation.sh'
    </programlisting>
   <para>
    The sample script <ulink url="https://raw.githubusercontent.com/pgpool/pgpool2/refs/heads/master/src/sample/scripts/escalation.sh.sample">escalation.sh</ulink> is installed in <filename>/etc/pgpool-II/</filename>.
   </para>
    <programlisting>
[root@server1 ~]# cp -p /etc/pgpool-II/sample_scripts/escalation.sh.sample /etc/pgpool-II/escalation.sh
[root@server1 ~]# chown postgres:postgres /etc/pgpool-II/escalation.sh
    </programlisting>

   <para>
    Basically, it should work if you change the following variables according to your environment.
    PGPOOLS is a list of hostnames where Pgpool-II is running.
    VIP is the virtual IP address that is set to <xref linkend="guc-delegate-ip">.
    DEVICE is the network interface for the virtual IP.
   </para>
    <programlisting>
[root@server1 ~]# vi /etc/pgpool-II/escalation.sh
...
PGPOOLS=(server1 server2 server3)
VIP=192.168.100.50
DEVICE=enp0s8
CIDR_NETMASK=24
...
    </programlisting>

   <note>
    <para>
     If you have even number of watchdog nodes, you need to turn on <xref linkend="guc-enable-consensus-with-half-votes"> parameter.
    </para>
   </note>
   <note>
    <para>
     If <varname>use_watchdog = on</varname>, please make sure the pgpool
     node number is specified in <filename>pgpool_node_id</filename> file.
     See <xref linkend="example-cluster-pgpool-node-id"> for details.
    </para>
   </note>
  </sect3>

  <sect3 id="example-cluster-pgpool-config-log">
   <title>Logging</title>
   <para>
    Since <productname>Pgpool-II</productname> 4.2,
    the logging collector process has been implemented.
    When installed using the RPM packages, the logging
    collector process (<xref linkend="guc-logging-collector">)
    is enabled by default. Log files are output to
    <filename>/var/log/pgpool_log</filename> by default.
    Configure the logging related configuration parameters
    as needed to meet your requirements.
   </para>
   <programlisting>
log_destination = 'stderr'
logging_collector = on
log_directory = '/var/log/pgpool'
log_filename = 'pgpool-%a.log'
log_truncate_on_rotation = on
log_rotation_age = 1d
   </programlisting>

  <para>
   The configurations of Pgpool-II on <literal>server1</literal> are completed.
   Finally, copy the configuration files to <literal>server2</literal>
   and <literal>server3</literal>.
  </para>
  <programlisting>
[root@server1 ~]# scp -p /etc/pgpool-II/{*.conf,*.sh,pool_passwd} server2:/etc/pgpool-II/
[root@server1 ~]# ssh server2 "chown postgres:postgres /etc/pgpool-II/{*.conf,*.sh,pool_passwd}"
[root@server1 ~]# scp -p /etc/pgpool-II/{*.conf,*.sh,pool_passwd} server3:/etc/pgpool-II/
[root@server1 ~]# ssh server3 "chown postgres:postgres /etc/pgpool-II/{*.conf,*.sh,pool_passwd}"
  </programlisting>
  </sect3>
 </sect2>

 <sect2 id="example-cluster-verify">
  <title>How to use</title>
  <para>
   Let's start to use <productname>Pgpool-II</productname>.
  </para>
  <sect3 id="example-cluster-verify-starting-stopping">
   <title>Starting/Stopping Pgpool-II</title>
   <itemizedlist>
    <listitem>
     <para>
      Starting <productname>Pgpool-II</productname>
     </para>
     <para>
      First, let's start <productname>Pgpool-II</productname>.
     </para>
     <para>
      Before starting <productname>Pgpool-II</productname>,
      the PostgreSQL primary server must be already running.
      If PostgreSQL primary server is not running, start it first
      using the following command.
     </para>
     <programlisting>
[root@server1 ~]# su - postgres -c "/usr/pgsql-18/bin/pg_ctl start"
     </programlisting>
     <para>
      Start <productname>Pgpool-II</productname> on <literal>server1</literal>,
      <literal>server2</literal>, <literal>server3</literal> by using the following command.
     </para>
     <programlisting>
[all servers]# systemctl start pgpool.service
     </programlisting>
    </listitem>
    <listitem>
     <para>
      Stopping <productname>Pgpool-II</productname>
     </para>
     <para>
      When stopping <productname>PostgreSQL</productname>,
      <productname>Pgpool-II</productname> must be stopped first.
     </para>
     <programlisting>
[all servers]# systemctl stop pgpool.service
     </programlisting>
    </listitem>
   </itemizedlist>
  </sect3>

  <sect3 id="example-cluster-verify-standby">
   <title>Setting up PostgreSQL standby server</title>
   <para>
    First, we should set up <productname>PostgreSQL</productname> standby server by
    using <productname>Pgpool-II</productname> online recovery functionality.
   </para>
   <para>
    Connect to Pgpool-II via virtual IP to check the status of backend nodes.
    As shown in the result, primary server is running on <literal>server1</literal>,
    standby servers on <literal>server2</literal> and <literal>server3</literal>
    are in "down" status.
   </para>
   <programlisting>
[any server]# psql -h 192.168.100.50 -p 9999 -U pgpool postgres -c "show pool_nodes"
Password for user pgpool: 
 node_id | hostname | port | status | pg_status | lb_weight |  role   | pg_role | select_cnt | load_balance_node | replication_delay | replication_state | replication_sync_state | last_status_change  
---------+----------+------+--------+-----------+-----------+---------+---------+------------+-------------------+-------------------+-------------------+------------------------+---------------------
 0       | server1  | 5432 | up     | up        | 0.333333  | primary | primary | 0          | true              | 0                 |                   |                        | 2023-11-10 15:30:14
 1       | server2  | 5432 | down   | down      | 0.333333  | standby | unknown | 0          | false             | 0                 |                   |                        | 2023-11-10 15:30:14
 2       | server3  | 5432 | down   | down      | 0.333333  | standby | unknown | 0          | false             | 0                 |                   |                        | 2023-11-10 15:30:14
(3 rows)
   </programlisting>
   <para>
    Before running <xref linkend="pcp-recovery-node"> command,
    ensure that <filename>recovery_1st_stage</filename> and
    <filename>pgpool_remote_start</filename> scripts exist in the
    data directory of <productname>PostgreSQL</productname>
    primary server (<literal>server1</literal>).
   </para>
   <programlisting>
[any server]# pcp_recovery_node -h 192.168.100.50 -p 9898 -U pgpool -n 1
Password:
pcp_recovery_node -- Command Successful

[any server]# pcp_recovery_node -h 192.168.100.50 -p 9898 -U pgpool -n 2
Password:
pcp_recovery_node -- Command Successful
   </programlisting>
   <para>
    After executing <command>pcp_recovery_node</command> command,
    verify that <productname>PostgreSQL</productname> standby servers
    are running on <literal>server2</literal> and <literal>server3</literal>.
   </para>
   <programlisting>
[any server]# psql -h 192.168.100.50 -p 9999 -U pgpool postgres -c "show pool_nodes"
Password for user pgpool
 node_id | hostname | port | status | pg_status | lb_weight |  role   | pg_role | select_cnt | load_balance_node | replication_delay | replication_state | replication_sync_state | last_status_change
---------+----------+------+--------+-----------+-----------+---------+---------+------------+-------------------+-------------------+-------------------+------------------------+---------------------
 0       | server1  | 5432 | up     | up        | 0.333333  | primary | primary | 0          | false             | 0                 |                   |                        | 2023-11-10 15:30:14
 1       | server2  | 5432 | up     | up        | 0.333333  | standby | standby | 0          | true              | 0                 | streaming         | async                  | 2023-11-10 16:32:33
 2       | server3  | 5432 | up     | up        | 0.333333  | standby | standby | 0          | false             | 0                 | streaming         | async                  | 2023-11-10 16:33:08
(3 rows)
   </programlisting>
  </sect3>

  <sect3 id="example-cluster-verify-watchdog">
   <title>Switching leader/standby watchdog</title>
   <para>
    Confirm the watchdog status by using <command>pcp_watchdog_info</command>. The <command>Pgpool-II</command> server which is started first runs as <literal>LEADER</literal>.
   </para>
   <programlisting>
[any server]# pcp_watchdog_info -h 192.168.100.50 -p 9898 -U pgpool
Password:
3 3 YES server1:9999 Linux server1 server1

server1:9999 Linux server1 server1 9999 9000 4 LEADER 0 MEMBER  # The Pgpool-II server started first becomes "LEADER".
server2:9999 Linux server2 server2 9999 9000 7 STANDBY 0 MEMBER # running as STANDBY
server3:9999 Linux server3 server3 9999 9000 7 STANDBY 0 MEMBER # running as STANDBY
   </programlisting>
   <para>
    If the <literal>LEADER</literal> <productname>Pgpool-II</productname>
    on <literal>server1</literal> goes down, standby
    <productname>Pgpool-II</productname> on <literal>server2</literal> or 
    <literal>server3</literal> will become the new <literal>LEADER</literal>.
   </para>
   <para>
    To verify this behavior, you may stop <productname>Pgpool-II</productname> 
    service or shutdown the whole system. Here, we stop
    <productname>Pgpool-II</productname> service.
   </para>
   <programlisting>
[root@server1 ~]# systemctl stop pgpool.service

[root@server1 ~]# pcp_watchdog_info -p 9898 -h 192.168.100.50 -U pgpool
Password:
3 3 YES server2:9999 Linux server2 server2

server2:9999 Linux server2 server2 9999 9000 4 LEADER 0 MEMBER    # server2 becomes LEADER
server1:9999 Linux server1 server1 9999 9000 10 SHUTDOWN 0 MEMBER # server1 is stopped
server3:9999 Linux server3 server3 9999 9000 7 STANDBY 0 MEMBER   # server3 is running as a STANDBY
   </programlisting>
   <para>
    Restart the stopped <productname>Pgpool-II</productname>
    on <literal>server1</literal> and verify that it is running
    as a <literal>STANDBY</literal>.
   </para>
   <programlisting>
[root@server1 ~]# systemctl start pgpool.service

[root@server1 ~]# pcp_watchdog_info -p 9898 -h 192.168.100.50 -U pgpool
Password: 
3 3 YES server2:9999 Linux server2 server2

server2:9999 Linux server2 server2 9999 9000 4 LEADER 0 MEMBER
server1:9999 Linux server1 server1 9999 9000 7 STANDBY 0 MEMBER
server3:9999 Linux server3 server3 9999 9000 7 STANDBY 0 MEMBER
   </programlisting>
  </sect3>

  <sect3 id="example-cluster-verify-failover">
   <title>Failover</title>
   <para>
    First, use <command>psql</command> to connect to
    <productname>PostgreSQL</productname> via virtual IP,
    and verify the backend information.
   </para>
   <programlisting>
[any server]# psql -h 192.168.100.50 -p 9999 -U pgpool postgres -c "show pool_nodes"
Password for user pgpool:
 node_id | hostname | port | status | pg_status | lb_weight |  role   | pg_role | select_cnt | load_balance_node | replication_delay | replication_state | replication_sync_state | last_status_change
---------+----------+------+--------+-----------+-----------+---------+---------+------------+-------------------+-------------------+-------------------+------------------------+---------------------
 0       | server1  | 5432 | up     | up        | 0.333333  | primary | primary | 0          | false             | 0                 |                   |                        | 2023-11-10 15:30:14
 1       | server2  | 5432 | up     | up        | 0.333333  | standby | standby | 0          | false             | 0                 | streaming         | async                  | 2023-11-10 16:32:33
 2       | server3  | 5432 | up     | up        | 0.333333  | standby | standby | 0          | true              | 0                 | streaming         | async                  | 2023-11-10 16:33:08
(3 rows)
   </programlisting>
   <para>
    Next, stop the primary <productname>PostgreSQL</productname> server 
    on <literal>server1</literal> and verify that failover is performed
    automatically.
   </para>
   <programlisting>
[root@server1 ~]# su - postgres -c "/usr/pgsql-18/bin/pg_ctl -m immediate stop"
   </programlisting>
   <para>
    After stopping <productname>PostgreSQL</productname> on
    <literal>server1</literal>, failover occurs.
    <productname>PostgreSQL</productname> on 
    <literal>server2</literal> becomes the new primary and
    the standby server on <literal>server3</literal>
    is configured as a standby of the new primary.
   </para>
   <programlisting>
[any server]# psql -h 192.168.100.50 -p 9999 -U pgpool postgres -c "show pool_nodes"
Password for user pgpool:
 node_id | hostname | port | status | pg_status | lb_weight |  role   | pg_role | select_cnt | load_balance_node | replication_delay | replication_state | replication_sync_state | last_status_change
---------+----------+------+--------+-----------+-----------+---------+---------+------------+-------------------+-------------------+-------------------+------------------------+---------------------
 0       | server1  | 5432 | down   | down      | 0.333333  | standby | unknown | 0          | false             | 0                 |                   |                        | 2023-11-10 17:05:40
 1       | server2  | 5432 | up     | up        | 0.333333  | primary | primary | 0          | false             | 0                 |                   |                        | 2023-11-10 17:05:40
 2       | server3  | 5432 | up     | up        | 0.333333  | standby | standby | 0          | true              | 0                 | streaming         | async                  | 2023-11-10 17:05:51
(3 rows)
   </programlisting>
   <para>
    <literal>server3</literal> is running as a standby of new primary <literal>server2</literal>.
   </para>

   <programlisting>
[any server]# psql -h server3 -p 5432 -U pgpool postgres -c "select pg_is_in_recovery()"
pg_is_in_recovery 
-------------------
t

[any server]# psql -h server2 -p 5432 -U pgpool postgres -c "select pg_is_in_recovery()"
pg_is_in_recovery 
-------------------
f

[any server]# psql -h server2 -p 5432 -U pgpool postgres -c "select * from pg_stat_replication" -x
-[ RECORD 1 ]----+------------------------------
pid              | 7198
usesysid         | 16385
usename          | repl
application_name | server3
client_addr      | 192.168.100.53
client_hostname  |
client_port      | 40916
backend_start    | 2023-11-10 17:10:03.067241+00
backend_xmin     |
state            | streaming
sent_lsn         | 0/12000260
write_lsn        | 0/12000260
flush_lsn        | 0/12000260
replay_lsn       | 0/12000260
write_lag        |
flush_lag        |
replay_lag       |
sync_priority    | 0
sync_state       | async
reply_time       | 2023-11-10 17:17:23.886477+00
   </programlisting>
  </sect3>

  <sect3 id="example-cluster-verify-online-recovery">
   <title>Online Recovery</title>
   <para>
    Here, we use <productname>Pgpool-II</productname> online recovery
    feature to restore the former primary on <literal>server1</literal>
    as a standby.
   </para>
   <programlisting>
[any server]# pcp_recovery_node -h 192.168.100.50 -p 9898 -U pgpool -n 0
Password: 
pcp_recovery_node -- Command Successful
   </programlisting>
   <para>
    Then verify that PostgreSQL on <literal>server1</literal> is
    running as a standby.
   </para>
   <programlisting>
[any server]# psql -h 192.168.100.50 -p 9999 -U pgpool postgres -c "show pool_nodes"
Password for user pgpool:
 node_id | hostname | port | status | pg_status | lb_weight |  role   | pg_role | select_cnt | load_balance_node | replication_delay | replication_state | replication_sync_state | last_status_change
---------+----------+------+--------+-----------+-----------+---------+---------+------------+-------------------+-------------------+-------------------+------------------------+---------------------
 0       | server1  | 5432 | up     | up        | 0.333333  | standby | standby | 0          | true              | 0                 | streaming         | async                  | 2023-11-10 17:22:03
 1       | server2  | 5432 | up     | up        | 0.333333  | primary | primary | 0          | false             | 0                 |                   |                        | 2023-11-10 17:05:40
 2       | server3  | 5432 | up     | up        | 0.333333  | standby | standby | 0          | falas             | 0                 | streaming         | async                  | 2023-11-10 17:05:51
(3 rows)
   </programlisting>
  </sect3>
 </sect2>
</sect1>
