diff --git a/doc/source/shared/abbrevs.txt b/doc/source/shared/abbrevs.txt index 54dd05135..29e2c1d64 100755 --- a/doc/source/shared/abbrevs.txt +++ b/doc/source/shared/abbrevs.txt @@ -190,6 +190,7 @@ .. |TPM| replace:: :abbr:`TPM (Trusted Platform Module)` .. |TPMs| replace:: :abbr:`TPMs (Trusted Platform Modules)` .. |UDP| replace:: :abbr:`UDP (User Datagram Protocol)` +.. |UDS| replace:: :abbr:`UDS (Unix Domain Socket)` .. |UEFI| replace:: :abbr:`UEFI (Unified Extensible Firmware Interface)` .. |UFT| replace:: :abbr:`UFT (Unified Flow Tool)` .. |UUID| replace:: :abbr:`UUID (Universally Unique Identifier)` diff --git a/doc/source/system_configuration/kubernetes/index-sysconf-kub-78f0e1e9ca5a.rst b/doc/source/system_configuration/kubernetes/index-sysconf-kub-78f0e1e9ca5a.rst index 2285ce172..f321e4a16 100644 --- a/doc/source/system_configuration/kubernetes/index-sysconf-kub-78f0e1e9ca5a.rst +++ b/doc/source/system_configuration/kubernetes/index-sysconf-kub-78f0e1e9ca5a.rst @@ -46,7 +46,7 @@ PTP Server Configuration ------------------------ .. toctree:: - :maxdepth: 2 + :maxdepth: 3 ptp-introduction-d981dd710bda configuring-ptp-service-using-the-cli @@ -54,6 +54,7 @@ PTP Server Configuration instance-specific-considerations-d9d9509c79dd remove-ptp-configurations-4885c027dfa5 advanced-ptp-configuration-334a08dc50fb + redundant-system-clock-synchronization-89ee23f54fbb configure-silicom-sts-ptp-application-1bc4a8d07aad ptp-instance-examples-517dce312f56 ptp-instance-troubleshooting-7a7c576ee57a diff --git a/doc/source/system_configuration/kubernetes/redundant-system-clock-synchronization-89ee23f54fbb.rst b/doc/source/system_configuration/kubernetes/redundant-system-clock-synchronization-89ee23f54fbb.rst new file mode 100644 index 000000000..716b9270d --- /dev/null +++ b/doc/source/system_configuration/kubernetes/redundant-system-clock-synchronization-89ee23f54fbb.rst @@ -0,0 +1,512 @@ +.. _redundant-system-clock-synchronization-89ee23f54fbb: + +====================================== +Redundant System Clock Synchronization +====================================== + +.. rubric:: |context| + +The ``phc2sys`` application can be configured to accept multiple source clock +inputs. The quality of these sources are compared to user-defined priority +values and the best available source is selected to set the system time. + +The quality of the configured sources is continuously monitored by ``phc2sys`` +application and will select a new best source if the current source degrades +or if another source becomes higher quality. + +The following section describes how to configure ``phc2sys`` with multiple clock +sources, and manage and monitor the ``phc2sys`` application. + + +|HA| Phc2sys configuration +-------------------------- + +.. rubric:: |proc| + +#. Create a ``phc2sys`` instance with **multiple source selection** enabled. + + .. note:: + See the :ref:`ha-phc2sys-cong-param` section for additional information + on parameters required for configuration. + + .. code-block:: none + + # Set bash variables to simplify commands + INSTANCE=phc2sys1 + TYPE=phc2sys + IFNAME1=phc1-iface1 + IFNAME2=phc1-iface2 + HOST=controller-0 + + .. code-block:: none + + # Create instance + system ptp-instance-add $INSTANCE $TYPE + + # Create interface(s) + system ptp-interface-add $IFNAME1 $INSTANCE + system ptp-interface-add $IFNAME2 $INSTANCE + + # Assign host interfaces + system host-if-ptp-assign $HOST data0 $IFNAME1 + system host-if-ptp-assign $HOST data1 $IFNAME2 + +#. Assign global instance parameters. The parameters described in + :ref:`ha-phc2sys-cong-param` are required for |HA| ``Phc2sys`` + functionality. The ``domainNumber`` value can be included if all interfaces + use the same ``ptp4l`` domain. If ``ptp4l`` instances use different domains, + see the interface configuration options for each interface domain numbers. + + .. code-block:: none + + # Assign required global instance parameters + system ptp-instance-parameter-add $INSTANCE \ + ha_enabled=1 \ + ha_phc2sys_com_socket=/var/run/$INSTANCE \ + domainNumber=24 + +#. Assign optional global instance parameters. These parameters determine what + clock quality values are required by ``phc2sys`` to consider a source valid + for selection. See :ref:`ha-phc2sys-cong-param` for more details on the + behaviour of these parameters. + + .. code-block:: none + + system ptp-instance-parameter-add $INSTANCE \ + ha_max_gm_clockClass=6 + +#. Assign CLI options. This parameter is required for |HA| ``Phc2sys`` + operation. + + .. code-block:: none + + system ptp-instance-parameter-add $INSTANCE \ + cmdline_opts='-w' + +#. Assign required interface parameters. The ``ha_domainNumber`` parameter can + be ommited if a shared domain number is used and is configured under the + global instance parameters. + + .. code-block:: none + + system ptp-interface-parameter-add $IFNAME1 \ + ha_priority=100 \ + ha_uds_address=/var/run/ptp4l-ptp1 \ + ha_domainNumber=24 + + system ptp-interface-parameter-add $IFNAME2 \ + ha_priority=90 \ + ha_uds_address=/var/run/ptp4l-ptp2 \ + ha_domainNumber=25 + +#. Assign the instance to a host and apply the |PTP| configuration. + + .. code-block:: none + + system host-ptp-instance-assign $HOST $INSTANCE + system ptp-instance-apply + +.. _ha-phc2sys-cong-param: + +|HA| Phc2sys Configuration Parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. note:: + The parameters listed here can be used in combination with the default + ``phc2sys`` parameters. Use the ``phc2sys`` help page for additional + parameters. + + +Global Instance Parameters +************************** + +``ha_enabled`` + The ``ha_enabled`` is a global setting to enable or disable the |HA| multiple + ``ptp4l`` inputs support. The default is 0 (disabled). Set 1 to enable |HA| + multiple input support. + +``ha_stability_timer`` + The ``ha_stability_timer`` option is a global setting used to configure the + stability timer. The value is expressed in seconds, and the value 0 disables + the timer. The default value is 0 (disabled). + + When the timer is enabled, ``phc2sys`` waits a few seconds after a source + becomes a candidate for selection before allowing it to be selected as the + synchronization source. This can be useful to prevent frequent source switching + if a given source is frequently changing quality. A value of 300 would cause + ``phc2sys`` to ensure that a source has had stable quality for 300 seconds + before it is eligible to be used as a synchronization source. + + The stability timer only applies to sources that were in a degraded state + and have recovered to meet the selection threshold. If the active source + degrades and ``phc2sys`` determines that a new source must be selected, this + happens without delay. + + The ``ha_stability_timer`` is important when the clock has different priorities + (ha_priority) to prevent multiple and consecutive clock switches. When the + service starts with both clocks in a valid state, the higher priority source + is selected, and when it degrades the lower priority is selected quickly. Then + when the higher priority recovers it will only become active after the + stability timer expires. + +``ha_phc2sys_com_socket`` + Provides a path for the communication socket used for monitoring and + maintenance activities. This value is required for system alarming to check + the ``phc2sys`` state. It is recommended to create the socket in the ``/var/run`` + directory, for example, ``ha_phc2sys_com_socket=/var/run/`` + See :ref: ``manage-phc2sys`` for managing interaction with ``phc2sys`` via this + socket. + +Source Selection Parameters +*************************** + +The parameters in this section are applied at the instance level. For example, +``system ptp-instance-parameter-add ha_max_gm_clockClass=7``. + +These parameters set the threshold for a source to be considered eligible for +use in setting the system time. A source that does not meet one or more of the +requirements will not be selected. ``Phc2sys`` then selects the best available +source that meets the threshold. The interface parameter ``ha_priority`` is +used as a tiebreaker for sources of equal quality, with the higher priority +being selected. + +The default value for each parameter is in effect even if the parameter is not +explicitly defined in the configuration file. + +.. note:: + + In most cases, users may use the "|GM| clock values" set of parameters to + configure the selection threshold. The local clock values are available for + additional fine tuning if required, but are generally not a good indicator + of the quality of a clock source compared to the |GM| values. + +.. Cole we need values for GM + ++---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Parameter | Default Value | Range | Description | ++=================================+===============+===============+=====================================================================================================================================================================================================================+ +| **GM clock values** | | | | ++---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| ha_max_gm_clockClass | 6 | 6-255 | The ha_max_gm_clockClass option is a global setting used to configure the maximal GM clock class requirement. It ranges from 6 to 255 and its default is 6. A source with a higher clockClass will not be selected. | ++---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| ha_max_gm_clockAccuracy | 0xfe | 0x00-0xff | A source must have a clockAccuracy value equal or lower than this parameter to be selected. | ++---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| ha_max_gm_offsetScaledLogVar | 0xffff | 0x0000-0xffff | A source must have an offsetScaledLogVariance value equal or lower than this parameter to be selected. | ++---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| ha_gm_timeTraceable | 0 | 0/1 | If this parameter is set to 1, then a source must have the timeTraceable flag set in order to be selected. | ++---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| ha_gm_frequencyTraceable | 0 | 0/1 | If this parameter is set to 1, then a source must have the frequencyTraceable flag set in order to be selected. | ++---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| **Local clock values** | | | | ++---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| ha_max_local_clockClass | 255 | 6-255 | Same as ha_max_gm_clockClass, but only considers the local clock class, not the class received from the parent time source. | ++---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| ha_max_local_clockAccuracy | 0xfe | 0x00-0xff | Only considers the local clockAccuracy. | ++---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| ha_max_local_offsetScaledLogVar | 0xffff | 0x0000-0xffff | Only considers the local offsetScaledLogVariance | ++---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + +Interface Parameters +******************** + +``ha_uds_address`` + The ``ha_uds_address`` is a **required** parameter for each source configured + in ``phc2sys``. This provides a path to the associated ``ptp4l`` instance on + that source which allows ``phc2sys`` to determine the quality of the source. + + This value should match the ``uds_address`` parameter of the associated ``ptp4l`` + instance. For example, ``ha_uds_address=/var/run/ptp4l-ptp1``. + +``ha_priority`` + The ``ha_priority`` option is an interface setting used to configure the clock + priority. The lowest priority is 0, highest is 254, and the default value is 0. + + If all clock quality parameters are equal between multiple sources, then the + highest priority source is selected. + + If multiple sources have the same clock quality and the same priority, then + ``phc2sys`` will select the first interface in the configuration file. + + An interface configured with ``ha_priority`` 254 will force ``phc2sys`` to + select that source, regardless of its status. + + If more then one interface is configured with ``ha_priority`` 254 the first + interface in the configuration file with priority 254 is selected as clock + source. + +``ha_domainNumber`` + The ``ha_domainNumber`` option is an interface setting to configure the domain + number for a |UDS| socket. It ranges from 0 to 127. The ``ha_domainNumber`` + has no default value, if the ``ha_domainNumber`` is not configured for a given + interface, the global ``domainNumber`` setting is used. + + This allows for configurations where multiple ``ptp4l`` instances providing + clock source quality information may be in different domains. + + If all ``ptp4l`` instances on the system are in the same domain, it is + recommended to use the domainNumber parameter in the global section. + +Example Configuration File +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The following is an example configuration file for |HA| ``Phc2sys`` which +shows how services can be configured with multiple interfaces tracking separate +``ptp4l`` instances for quality information. + +.. code-block:: none + + [global] + ## + ## Default Data Set + ## + domainNumber 24 + ha_enabled 1 + ha_gm_frequencyTraceable 1 + ha_gm_timeTraceable 1 + ha_max_gm_clockAccuracy 0x20 + ha_max_gm_clockClass 6 + ha_max_gm_offsetScaledLogVar 0x4e5d + ha_max_local_clockAccuracy 0x20 + ha_max_local_clockClass 255 + ha_max_local_offsetScaledLogVar 0xffff + ha_phc2sys_com_socket /var/run/phc1 + ha_stability_timer 60 + logging_level 7 + message_tag phc1 + + [enp81s0f1] + ## + ## Associated interface: enp81s0f1 + ## + ha_priority 100 + ha_uds_address /var/run/ptp4l-ptp11 + + [enp138s0f0] + ## + ## Associated interface: enp138s0f0 + ## + ha_priority 101 + ha_uds_address /var/run/ptp4l-ptp12 + +.. _manage-phc2sys: + +Manage Phc2sys +-------------- + +The ``ha_phc2sys_com_socket`` displays information about the state of the +source selection algorithm, as well as performs test and troubleshooting +operations. + +To perform these operations, locate the socket file for the ``phc2sys`` instance. +This should be configured in the global instance parameters for the +``phc2sys`` instance. For example, ``/var/run/``. + +These commands must be run on the specific host being managed, as access to the +socket file is required. For example: + +.. code-block:: none + + [sysadmin@controller-0 ~(keystone_admin)]$ echo -n "" | socat -t 1 - UNIX-CONNECT: + +Informational Commands +~~~~~~~~~~~~~~~~~~~~~~ + +``status`` + The ``status`` command prints a list of configured sources along with the + quality characteristics that have been read from the associated ``ptp4l`` + instances. + + The ``status`` command also indicates which source has been selected. + +.. code-block:: none + + [sysadmin@controller-0 ~(keystone_admin)]$ echo -n "status" | socat -t 1 - UNIX-CONNECT:/var/run/phc0 + + act interface priority clockClass clockAcc offset time freq gm.clockClass gm.clockAcc gm.offset + + * enp138s0f0 100 248 0xfe 0xffff yes yes 6 0x20 0x4e5d + enp81s0f1 100 248 0xfe 0xffff yes yes 6 0x20 0x4e5d + +``forced lock`` + The ``forced lock`` command prints "False" if ``phc2sys`` has not been forced to + select a specific interface via a management operation. It prints "True" if + a force lock has been set via a management operation. + +.. code-block:: none + + [sysadmin@controller-0 ~(keystone_admin)]$ echo -n "forced lock" | socat -t 1 - UNIX-CONNECT: + + +``valid sources`` + The ``valid sources`` command prints the interfaces that meet the quality + threshold for selection. If none of the configured interfaces meet the + quality threshold then the string "None" is printed. + +.. code-block:: none + + [sysadmin@controller-0 ~(keystone_admin)]$ echo -n "valid sources" | socat -t 1 - UNIX-CONNECT: + +``clock source`` + The ``clock source`` command returns the current source that is selected by + ``phc2sys``. + +.. code-block:: none + + [sysadmin@controller-0 ~(keystone_admin)]$ echo -n "clock source" | socat -t 1 - UNIX-CONNECT: + + +Management Operations +~~~~~~~~~~~~~~~~~~~~~ + +``enable lock`` + The ``enable lock `` command forces ``phc2sys`` to use the given + interface as the time source. This operation can be helpful for debugging + operations that require the clock source to stay fixed. + + .. note:: + ``phc2sys`` can also be forced to select an interface by configuring it + with ``ha_priority=254`` which is preferred if the lock needs to + persist between reboots and restarts of the ``phc2sys`` service. + + ``Phc2sys`` can only be locked to a single interface at a time. + +.. code-block:: none + + [sysadmin@controller-0 ~(keystone_admin)]$ echo -n "enable lock enp81s0f0" | socat -t 1 - UNIX-CONNECT: + +``disable lock`` + The ``disable lock`` command returns ``phc2sys`` to normal operations where + it selects the best quality clock source based on the configured thresholds. + +.. code-block:: none + + [sysadmin@controller-0 ~(keystone_admin)]$ echo -n "disable lock" | socat -t 1 - UNIX-CONNECT: + +``disable source `` + The ``disable source `` command removes a given interface from the + list of sources considered by ``phc2sys`` for selection. This can be used for + troubleshooting and testing by allowing sources to be removed and re-enabled + without restarting the ``phc2sys`` service. + + ``Phc2sys`` will not allow a source to be disabled if it is the only source + currently configured and enabled. + +.. code-block:: none + + [sysadmin@controller-0 ~(keystone_admin)]$ echo -n "disable source enp81s0f0" | socat -t 1 - UNIX-CONNECT: + +``enable source `` + The ``enable source `` command returns a previously disabled + interface to the selection pool, allowing ``phc2sys`` to consider it for + selection based on its quality. + +.. code-block:: none + + [sysadmin@controller-0 ~(keystone_admin)]$ echo -n "enable source enp81s0f0" | socat -t 1 - UNIX-CONNECT: + +|HA| Phc2sys Alarms +------------------- + +New alarms have been added to report the state of |HA| ``Phc2sys``. To view +system alarms on |prod|, use the :command:`fm alarm-list` command. + +**Phc2sys HA has no source clock** + +Example EID: +``host=controller-0.phc2sys=phc0.phc2sys=no-source-clock`` + +The "no source clock" alarm is raised when ``phc2sys`` does not have any sources +that meet the specified threshold in the user configuration. This is due to the +quality of the available sources not meeting specified thresholds. The alarm +clears when an available source meets the specified quality requirements. + +The ``ha_phc2sys_com_socket`` "status" and "valid sources" commands can be used +to assist with troubleshooting this alarm. + +**phc2sys HA source clock is not locked to a PRC** + +Example EID: +``host=controller-0.phc2sys=ptp01.interface=enp81s0f1.phc2sys=source-clock-no-prc-lock`` + +The "source clock is not locked to a PRC" alarm is raised when one of the |HA| +``phc2sys`` interfaces is indicating a clockClass that is of lesser quality than +the specified maximum clockClass in the user config. The alarm indicates which +interface is degraded. The presence of this alarm should coincide with ``phc2sys`` +selecting a different source if a better one is available. The alarm will clear +when the clockClass for the source meets the selection threshold. + +The ``ha_phc2sys_com_socket`` "status" and "valid sources" commands can be used to +assist with troubleshooting this alarm. + +**phc2sys HA has selected a lower priority clock source** + +Example EID: +``host=controller-0.phc2sys=phc1. phc2sys=source-clock-low-priority`` + +The "selected lower priority clock source" alarm is raised when ``phc2sys`` is +configured with multiple sources and the selected one that is not the highest +priority. This alarm indicates that there is degraded quality for the highest +priority source and that ``phc2sys`` has taken action to switch. The alarm will be +cleared when the highest priority source becomes available for selection. + +**phc2sys HA automatic source selection has been disabled** + +Example EID: +``host=controller-0.phc2sys=phc0.phc2sys=forced-clock-selection`` + +The "automatic source selection disabled" alarm is raised when an operator has +used the ``ha_phc2sys_com_socket`` :command:`enable lock `command +to force ``phc2sys`` to use a specific source. This alarm is informational and +can be cleared by running the :command:`disable lock` command via the +``ha_phc2sys_com_socket``. + +**phc2sys HA source selection algorithm selected new active source** + +Example EID: +``host=controller-0.phc2sys=ptp02.interface=enp138s0f0.phc2sys=source-failover`` + +The "selected new active source" event is raised when ``phc2sys`` switches between +sources for any reason. This is not raised as a persistent alarm, but rather +as an event, viewable via :command:`fm event-list`. This event enables users to +see when source selection events have occurred. There is no action required to +clear this event. + +Phc2sys Source Selection Behaviour +---------------------------------- + +When ``phc2sys`` is configured with multiple sources and "ha_enabled", a selection +algorithm runs repeatedly to determine which source to sync the system time to. +The source selection algorithm operates by determining a list of valid sources +(i.e. sources that meet the configured selection requirements), and then +choosing the source from that list with the highest priority. + +When ``phc2sys`` detects that a source's quality has changed, the list is +re-calculated and a new source is selected if a better one has become available. + +The source selection operates in the following way: + +.. code-block:: none + + For each configured clock source: + if local.clockClass > ha_max_local_clockClass -> disqualify source + if local.clockAccuracy > ha_max_local_clockAccuracy -> disqualify source + if local.offsetScaledLogVariance > ha_max_gm_offsetScaledLogVar -> disqualify source + if source.timeTraceable = 0 AND ha_gm_timeTraceable = 1 -> disqualify source + if source.frequencyTraceable = 0 and ha_gm_frequencyTraceable = 1 -> disqualify source + if gm.clockClass > ha_max_gm_clockClass -> disqualify source + if gm.clockAccuracy > ha_max_gm_clockAccuracy -> disqualify source + if gm.offsetScaledLogVariance > ha_max_gm_offsetScaledLogVar -> disqualify source + +From list of qualified sources, select the source with highest "ha_priority". If +qualified sources have the same "ha_priority", select the first one as it +appears in the ``phc2sys`` config file. + +If there is no valid source, select the source with lowest "local.clockClass". +If "local.clockClass" is tied, select the first source in ``phc2sys`` config file +from among the tied sources. + + + + + +