Add system peer management API support

Add dcmanager system-peer management api.

Test Plan:
1. PASS - Verify that cloud manage system-peer
          through api successfully.
2. PASS - Add system peer with invalid UUID, manager_endpoint,
	  systemcontroller_gateway_address, administrative_state,
	  heartbeat_interval
3. PASS - Update system peer with invalid administrative_state,
	  heartbeat_interval
4. PASS - Get system peer with UUID, name
5. PASS - Delete system peer with UUID, name

CLI example:
dcmanager system-peer add --peer_uuid $(uuidgen) --peer_name dc-0
--manager_endpoint http://128.128.128.1:5000/v3
(The peer_uuid get from the peer site with command `system show`)

dcmanager system-peer list

dcmanager system-peer update --administrative_state enabled 1

dcmanager system-peer show 1

dcmanager system-peer delete 1

Story: 2010852
Task: 48482

Change-Id: I349cd24bccc732eb8ed56df9346185cfce7b2570
Signed-off-by: Zhang Rong(Jon) <rong.zhang@windriver.com>

api-ref/source/api-ref-dcmanager-v1.rst

@@ -2262,4 +2262,284 @@ Response Example
 ----------------
 
 .. literalinclude:: samples/phased-subcloud-deploy/phased-subcloud-deploy-patch-resume-response.json
-      :language: json
+      :language: json
+
+------------
+System Peers
+------------
+
+System Peers are logical entities which are managed by a central System Controller.
+Each System Peer maintains the information which is used for health check
+and data synchronization in the protection group in Geo-Redundancy deployment.
+
+**********************
+Lists all system peers
+**********************
+
+.. rest_method:: GET /v1.0/system-peers
+
+This operation does not accept a request body.
+
+**Normal response codes**
+
+200
+
+**Error response codes**
+
+badRequest (400), unauthorized (401), forbidden (403),
+itemNotFound (404), badMethod (405), HTTPUnprocessableEntity (422),
+internalServerError (500), serviceUnavailable (503)
+
+Response
+--------
+
+.. rest_parameters:: parameters.yaml
+
+  - system_peers: system_peers
+  - id: system_peer_id
+  - peer-uuid: peer_uuid
+  - peer-name: peer_name
+  - manager-endpoint: manager_endpoint
+  - manager-username: manager_username
+  - peer-controller-gateway-address: peer_controller_gateway_address
+  - administrative-state: administrative_state
+  - heartbeat-interval: heartbeat_interval
+  - heartbeat-failure-threshold: heartbeat_failure_threshold 
+  - heartbeat-failure-policy: heartbeat_failure_policy
+  - heartbeat-maintenance-timeout: heartbeat_maintenance_timeout
+  - created-at: created_at
+  - updated-at: updated_at
+
+Response Example
+----------------
+
+.. literalinclude:: samples/system-peers/system-peers-get-response.json
+      :language: json
+
+
+*********************
+Creates a system peer
+*********************
+
+.. rest_method:: POST /v1.0/system-peers
+
+Accepts Content-Type multipart/form-data.
+
+
+**Normal response codes**
+
+200
+
+**Error response codes**
+
+badRequest (400), unauthorized (401), forbidden (403), badMethod (405),
+HTTPUnprocessableEntity (422), internalServerError (500),
+serviceUnavailable (503)
+
+**Request parameters**
+
+.. rest_parameters:: parameters.yaml
+
+  - peer_uuid: peer_uuid
+  - peer_name: peer_name
+  - manager_endpoint: manager_endpoint
+  - manager_username: manager_username
+  - manager_password: manager_password
+  - peer_controller_gateway_address: peer_controller_gateway_address
+  - administrative_state: administrative_state
+  - heartbeat_interval: heartbeat_interval
+  - heartbeat_failure_threshold: heartbeat_failure_threshold 
+  - heartbeat_failure_policy: heartbeat_failure_policy
+  - heartbeat_maintenance_timeout: heartbeat_maintenance_timeout
+
+Request Example
+----------------
+
+.. literalinclude:: samples/system-peers/system-peers-post-request.json
+      :language: json
+
+
+**Response parameters**
+
+.. rest_parameters:: parameters.yaml
+
+  - id: system_peer_id
+  - peer-uuid: peer_uuid
+  - peer-name: peer_name
+  - manager-endpoint: manager_endpoint
+  - manager-username: manager_username
+  - peer-controller-gateway-address: peer_controller_gateway_address
+  - administrative-state: administrative_state
+  - heartbeat-interval: heartbeat_interval
+  - heartbeat-failure-threshold: heartbeat_failure_threshold 
+  - heartbeat-failure-policy: heartbeat_failure_policy
+  - heartbeat-maintenance-timeout: heartbeat_maintenance_timeout
+  - created-at: created_at
+  - updated-at: updated_at
+
+Response Example
+----------------
+
+.. literalinclude:: samples/system-peers/system-peers-post-response.json
+      :language: json
+
+
+**********************************************
+Shows information about a specific system peer
+**********************************************
+
+.. rest_method:: GET /v1.0/system-peers/​{system-peer}​
+
+**Normal response codes**
+
+200
+
+**Error response codes**
+
+badRequest (400), unauthorized (401), forbidden (403),
+itemNotFound (404), badMethod (405), HTTPUnprocessableEntity (422),
+internalServerError (500), serviceUnavailable (503)
+
+**Request parameters**
+
+.. rest_parameters:: parameters.yaml
+
+  - system-peer: system_peer_uri
+
+This operation does not accept a request body.
+
+**Response parameters**
+
+.. rest_parameters:: parameters.yaml
+
+  - id: system_peer_id
+  - peer-uuid: peer_uuid
+  - peer-name: peer_name
+  - manager-endpoint: manager_endpoint
+  - manager-username: manager_username
+  - peer-controller-gateway-address: peer_controller_gateway_address
+  - administrative-state: administrative_state
+  - heartbeat-interval: heartbeat_interval
+  - heartbeat-failure-threshold: heartbeat_failure_threshold 
+  - heartbeat-failure-policy: heartbeat_failure_policy
+  - heartbeat-maintenance-timeout: heartbeat_maintenance_timeout
+  - created-at: created_at
+  - updated-at: updated_at
+
+Response Example
+----------------
+
+.. literalinclude:: samples/system-peers/system-peer-get-response.json
+         :language: json
+
+
+*******************************
+Modifies a specific system peer
+*******************************
+
+.. rest_method:: PATCH /v1.0/system-peers/​{system-peer}​
+
+The attributes of a subcloud group which are modifiable:
+
+-  peer-uuid
+
+-  peer-name
+
+-  manager-endpoint
+
+-  manager-username
+
+-  manager-password
+
+-  peer-controller-gateway-address
+
+-  administrative-state
+
+-  heartbeat-interval
+
+-  heartbeat-failure-threshold
+
+-  heartbeat-failure-policy
+
+-  heartbeat-maintenance-timeout
+
+**Normal response codes**
+
+200
+
+**Error response codes**
+
+badRequest (400), unauthorized (401), forbidden (403), badMethod (405),
+HTTPUnprocessableEntity (422), internalServerError (500),
+serviceUnavailable (503)
+
+**Request parameters**
+
+.. rest_parameters:: parameters.yaml
+
+  - system-peer: system_peer_uri
+  - peer_uuid: peer_uuid
+  - peer_name: peer_name
+  - manager_endpoint: manager_endpoint
+  - manager_username: manager_username
+  - manager_password: manager_password
+  - peer_controller_gateway_address: peer_controller_gateway_address
+  - administrative_state: administrative_state
+  - heartbeat_interval: heartbeat_interval
+  - heartbeat_failure_threshold: heartbeat_failure_threshold 
+  - heartbeat_failure_policy: heartbeat_failure_policy
+  - heartbeat_maintenance_timeout: heartbeat_maintenance_timeout
+
+Request Example
+----------------
+.. literalinclude:: samples/system-peers/system-peer-patch-request.json
+         :language: json
+
+**Response parameters**
+
+.. rest_parameters:: parameters.yaml
+
+  - id: system_peer_id
+  - peer-uuid: peer_uuid
+  - peer-name: peer_name
+  - manager-endpoint: manager_endpoint
+  - manager-username: manager_username
+  - peer-controller-gateway-address: peer_controller_gateway_address
+  - administrative-state: administrative_state
+  - heartbeat-interval: heartbeat_interval
+  - heartbeat-failure-threshold: heartbeat_failure_threshold 
+  - heartbeat-failure-policy: heartbeat_failure_policy
+  - heartbeat-maintenance-timeout: heartbeat_maintenance_timeout
+  - created-at: created_at
+  - updated-at: updated_at
+
+Response Example
+----------------
+
+.. literalinclude:: samples/system-peers/system-peer-patch-response.json
+         :language: json
+
+
+******************************
+Deletes a specific system peer
+******************************
+
+.. rest_method:: DELETE /v1.0/system-peers/​{system-peer}​
+
+**Normal response codes**
+
+200
+
+**Error response codes**
+
+badRequest (400), unauthorized (401), forbidden (403),
+itemNotFound (404), badMethod (405), HTTPUnprocessableEntity (422),
+internalServerError (500), serviceUnavailable (503)
+
+**Request parameters**
+
+.. rest_parameters:: parameters.yaml
+
+  - system-peer: system_peer_uri
+
+This operation does not accept a request body.

api-ref/source/parameters.yaml

@@ -39,7 +39,19 @@ sw_update_strategy_type:
   in: path
   required: false
   type: string
+system_peer_uri:
+  description: |
+    The system peer reference, name or id or UUID.
+  in: path
+  required: true
+  type: string
 # variables in body
+administrative_state:
+  description: |
+    The administrative state of the system peer site. (enabled, disabled)
+  in: body
+  required: true
+  type: string
 alarm_restriction_type:
   description: |
     Whether to allow update if subcloud alarms are present or not.
@@ -245,6 +257,30 @@ group_id:
   in: body
   required: true
   type: integer
+heartbeat_failure_policy:
+  description: |
+    The failure policy of the peer site heartbeats. (alarm, rehome, delegate)
+  in: body
+  required: true
+  type: string
+heartbeat_failure_threshold:
+  description: |
+    The failure threshold of the peer site heartbeats.
+  in: body
+  required: true
+  type: integer
+heartbeat_interval:
+  description: |
+    The interval of the message between the peer site heartbeats. (in seconds)
+  in: body
+  required: true
+  type: integer
+heartbeat_maintenance_timeout:
+  description: |
+    The maintenance timeout of the peer site heartbeats. (in seconds)
+  in: body
+  required: true
+  type: integer
 install_values:
   description: |
     The content of a file containing install variables such as subcloud
@@ -288,6 +324,24 @@ management_subnet:
   in: body
   required: true
   type: string
+manager_endpoint:
+  description: |
+    The endpoint of the system peer site manager.
+  in: body
+  required: true
+  type: string
+manager_password:
+  description: |
+    The password of the system peer site manager.
+  in: body
+  required: true
+  type: string
+manager_username:
+  description: |
+    The username of the system peer site manager.
+  in: body
+  required: true
+  type: string
 max_parallel_subclouds:
   description: |
     The maximum number of subclouds to update in parallel.
@@ -331,6 +385,24 @@ patch_strategy_upload_only:
   in: body
   required: false
   type: boolean
+peer_controller_gateway_address:
+  description: |
+    The gateway IP address of the system peer site system controller.
+  in: body
+  required: true
+  type: string
+peer_name:
+  description: |
+    The name of a peer as a string.
+  in: body
+  required: true
+  type: string
+peer_uuid:
+  description: |
+    The UUID of a peer as a string.
+  in: body
+  required: true
+  type: string
 prestage_software_version:
   description: |
     The prestage software version for the subcloud.
@@ -678,6 +750,18 @@ system_mode:
   in: body
   required: true
   type: string
+system_peer_id:
+  description: |
+    The ID of a system peer as an integer.
+  in: body
+  required: true
+  type: integer
+system_peers:
+  description: |
+    The list of ``system-peer`` objects.
+  in: body
+  required: true
+  type: array
 systemcontroller_gateway_ip:
   description: |
     The gateway IP address of the system controller of the subcloud.

api-ref/source/samples/system-peers/system-peer-get-response.json

@@ -0,0 +1,15 @@
+{
+  "id": 1,
+  "peer-uuid": "b00d0863-c54e-4340-af4d-3e2093764276",
+  "peer-name": "PeerDistributedCloud1",
+  "manager-endpoint": "http://128.128.128.1:5000/v3",
+  "manager-username": "admin",
+  "peer-controller-gateway-address": "192.168.204.1",
+  "administrative-state": "enabled",
+  "heartbeat-interval": 60,
+  "heartbeat-failure-threshold": 3,
+  "heartbeat-failure-policy": "alarm",
+  "heartbeat-maintenance-timeout": 600,
+  "created-at": "2023-08-14 05:47:35.587528",
+  "updated-at": "2023-08-14 05:47:35.587528"
+}

api-ref/source/samples/system-peers/system-peer-patch-request.json

@@ -0,0 +1,13 @@
+{
+  "peer_uuid": "b00d0863-c54e-4340-af4d-3e2093764276",
+  "peer_name": "PeerDistributedCloud1",
+  "manager_endpoint": "http://128.128.128.1:5000/v3",
+  "manager_username": "admin",
+  "manager-password": "V2luZDEyMyQ=",
+  "peer_controller_gateway-address": "192.168.204.1",
+  "administrative_state": "enabled",
+  "heartbeat_interval": 60,
+  "heartbeat_failure_threshold": 3,
+  "heartbeat_failure_policy": "alarm",
+  "heartbeat_maintenance_timeout": 600
+}

api-ref/source/samples/system-peers/system-peer-patch-response.json

@@ -0,0 +1,15 @@
+{
+  "id": 1,
+  "peer-uuid": "b00d0863-c54e-4340-af4d-3e2093764276",
+  "peer-name": "PeerDistributedCloud1",
+  "manager-endpoint": "http://128.128.128.1:5000/v3",
+  "manager-username": "admin",
+  "peer-controller-gateway-address": "192.168.204.1",
+  "administrative-state": "enabled",
+  "heartbeat-interval": 60,
+  "heartbeat-failure-threshold": 3,
+  "heartbeat-failure-policy": "alarm",
+  "heartbeat-maintenance-timeout": 600,
+  "created-at": "2023-08-14 05:47:35.587528",
+  "updated-at": "2023-08-14 06:47:35.587528"
+}

api-ref/source/samples/system-peers/system-peers-get-response.json

@@ -0,0 +1,19 @@
+{
+  "system_peers": [
+    {
+      "id": 1,
+      "peer-uuid": "b00d0863-c54e-4340-af4d-3e2093764276",
+      "peer-name": "PeerDistributedCloud1",
+      "manager-endpoint": "http://128.128.128.1:5000/v3",
+      "manager-username": "admin",
+      "peer-controller-gateway-address": "192.168.204.1",
+      "administrative-state": "enabled",
+      "heartbeat-interval": 60,
+      "heartbeat-failure-threshold": 3,
+      "heartbeat-failure-policy": "alarm",
+      "heartbeat-maintenance-timeout": 600,
+      "created-at": "2023-08-14 05:47:35.587528",
+      "updated-at": "2023-08-14 05:47:35.587528"
+    }
+  ]
+}

api-ref/source/samples/system-peers/system-peers-post-request.json

@@ -0,0 +1,13 @@
+{
+  "peer_uuid": "b00d0863-c54e-4340-af4d-3e2093764276",
+  "peer_name": "PeerDistributedCloud1",
+  "manager_endpoint": "http://128.128.128.1:5000/v3",
+  "manager_username": "admin",
+  "manager-password": "V2luZDEyMyQ=",
+  "peer_controller_gateway-address": "192.168.204.1",
+  "administrative_state": "enabled",
+  "heartbeat_interval": 60,
+  "heartbeat_failure_threshold": 3,
+  "heartbeat_failure_policy": "alarm",
+  "heartbeat_maintenance_timeout": 600
+}

api-ref/source/samples/system-peers/system-peers-post-response.json

@@ -0,0 +1,15 @@
+{
+  "id": 1,
+  "peer-uuid": "b00d0863-c54e-4340-af4d-3e2093764276",
+  "peer-name": "PeerDistributedCloud1",
+  "manager-endpoint": "http://128.128.128.1:5000/v3",
+  "manager-username": "admin",
+  "peer-controller-gateway-address": "192.168.204.1",
+  "administrative-state": "enabled",
+  "heartbeat-interval": 60,
+  "heartbeat-failure-threshold": 3,
+  "heartbeat-failure-policy": "alarm",
+  "heartbeat-maintenance-timeout": 600,
+  "created-at": "2023-08-14 05:47:35.587528",
+  "updated-at": null
+}

distributedcloud/dcmanager/api/controllers/v1/root.py

@@ -25,6 +25,7 @@ from dcmanager.api.controllers.v1 import subcloud_group
 from dcmanager.api.controllers.v1 import subclouds
 from dcmanager.api.controllers.v1 import sw_update_options
 from dcmanager.api.controllers.v1 import sw_update_strategy
+from dcmanager.api.controllers.v1 import system_peers
 
 
 class Controller(object):
@@ -54,6 +55,8 @@ class Controller(object):
                 SubcloudBackupController
             sub_controllers["phased-subcloud-deploy"] = phased_subcloud_deploy.\
                 PhasedSubcloudDeployController
+            sub_controllers["system-peers"] = system_peers.\
+                SystemPeersController
 
         for name, ctrl in sub_controllers.items():
             setattr(self, name, ctrl)

distributedcloud/dcmanager/api/controllers/v1/system_peers.py

@@ -0,0 +1,485 @@
+# Copyright (c) 2023 Wind River Systems, Inc.
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+
+import http.client as httpclient
+import json
+import uuid
+
+import ipaddress
+from oslo_config import cfg
+from oslo_db import exception as db_exc
+from oslo_log import log as logging
+from oslo_messaging import RemoteError
+import pecan
+from pecan import expose
+from pecan import request
+
+from dcmanager.api.controllers import restcomm
+from dcmanager.api.policies import system_peers as system_peer_policy
+from dcmanager.api import policy
+from dcmanager.common.i18n import _
+from dcmanager.common import utils
+from dcmanager.db import api as db_api
+
+CONF = cfg.CONF
+LOG = logging.getLogger(__name__)
+
+# validation constants for System Peer
+MAX_SYSTEM_PEER_NAME_LEN = 255
+MAX_SYSTEM_PEER_MANAGER_ENDPOINT_LEN = 255
+MAX_SYSTEM_PEER_MANAGER_USERNAME_LEN = 255
+MAX_SYSTEM_PEER_MANAGER_PASSWORD_LEN = 255
+MAX_SYSTEM_PEER_STRING_DEFAULT_LEN = 255
+# validation constants for System Peer Administrative State
+# Set to disabled this function will be disabled
+#
+# We will not support this function in the first release
+SYSTEM_PEER_ADMINISTRATIVE_STATE_LIST = ["enabled", "disabled"]
+MIN_SYSTEM_PEER_HEARTBEAT_INTERVAL = 10
+MAX_SYSTEM_PEER_HEARTBEAT_INTERVAL = 600
+MIN_SYSTEM_PEER_HEARTBEAT_FAILURE_THRESHOLD = 1
+MAX_SYSTEM_PEER_HEARTBEAT_FAILURE_THRESHOLD = 30
+# validation constants for System Peer Heartbeat Failure Policy
+# Set to alarm this function will be triggered alarm when the
+# heartbeat failure threshold is reached
+# Set to rehome this function will be automatically rehome the
+# subcloud when the heartbeat failure threshold is reached
+# Set to delegate this function will be delegate the system when
+# the heartbeat failure threshold is reached
+#
+# We will only support alarm in the first release
+SYSTEM_PEER_HEARTBEAT_FAILURE_POLICY_LIST = \
+    ["alarm", "rehome", "delegate"]
+MIN_SYSTEM_PEER_HEARTBEAT_MAINTENACE_TIMEOUT = 300
+MAX_SYSTEM_PEER_HEARTBEAT_MAINTENACE_TIMEOUT = 36000
+
+
+class SystemPeersController(restcomm.GenericPathController):
+
+    def __init__(self):
+        super(SystemPeersController, self).__init__()
+
+    @expose(generic=True, template='json')
+    def index(self):
+        # Route the request to specific methods with parameters
+        pass
+
+    @staticmethod
+    def _get_payload(request):
+        try:
+            payload = json.loads(request.body)
+        except Exception:
+            error_msg = 'Request body is malformed.'
+            LOG.exception(error_msg)
+            pecan.abort(400, _(error_msg))
+
+        if not isinstance(payload, dict):
+            pecan.abort(400, _('Invalid request body format'))
+        return payload
+
+    def _get_system_peer_list(self, context):
+        peers = db_api.system_peer_get_all(context)
+
+        system_peer_list = list()
+        for peer in peers:
+            peer_dict = db_api.system_peer_db_model_to_dict(peer)
+            system_peer_list.append(peer_dict)
+
+        result = dict()
+        result['system_peers'] = system_peer_list
+        return result
+
+    @index.when(method='GET', template='json')
+    def get(self, peer_ref=None):
+        """Get details about system peer.
+
+        :param peer_ref: ID or UUID or Name of system peer
+        """
+        policy.authorize(system_peer_policy.POLICY_ROOT % "get", {},
+                         restcomm.extract_credentials_for_policy())
+        context = restcomm.extract_context_from_environ()
+
+        if peer_ref is None:
+            # List of system peers requested
+            return self._get_system_peer_list(context)
+
+        peer = utils.system_peer_get_by_ref(context, peer_ref)
+        if peer is None:
+            pecan.abort(httpclient.NOT_FOUND, _('System Peer not found'))
+        system_peer_dict = db_api.system_peer_db_model_to_dict(peer)
+        return system_peer_dict
+
+    def _validate_uuid(self, _uuid):
+        try:
+            uuid.UUID(str(_uuid))
+            return True
+        except ValueError:
+            LOG.exception("Invalid UUID: %s" % _uuid)
+            return False
+
+    def _validate_name(self, name):
+        if not name or name.isdigit() or len(name) >= MAX_SYSTEM_PEER_NAME_LEN:
+            LOG.debug("Invalid name: %s" % name)
+            return False
+        return True
+
+    def _validate_manager_endpoint(self, endpoint):
+        if not endpoint or len(endpoint) >= MAX_SYSTEM_PEER_MANAGER_ENDPOINT_LEN or \
+            not endpoint.startswith(("http", "https")):
+            LOG.debug("Invalid manager_endpoint: %s" % endpoint)
+            return False
+        return True
+
+    def _validate_manager_username(self, username):
+        if not username or len(username) >= MAX_SYSTEM_PEER_MANAGER_USERNAME_LEN:
+            LOG.debug("Invalid manager_username: %s" % username)
+            return False
+        return True
+
+    def _validate_manager_password(self, password):
+        if not password or len(password) >= MAX_SYSTEM_PEER_MANAGER_PASSWORD_LEN:
+            LOG.debug("Invalid manager_password: %s" % password)
+            return False
+        return True
+
+    def _validate_peer_controller_gateway_ip(self, ip):
+        if not ip or len(ip) >= MAX_SYSTEM_PEER_STRING_DEFAULT_LEN:
+            LOG.debug("Invalid peer_manager_gateway_address: %s" % ip)
+            return False
+        try:
+            ipaddress.ip_address(ip)
+            return True
+        except Exception:
+            LOG.warning("Invalid IP address: %s" % ip)
+            return False
+
+    def _validate_administrative_state(self, administrative_state):
+        if administrative_state not in SYSTEM_PEER_ADMINISTRATIVE_STATE_LIST:
+            LOG.debug("Invalid administrative_state: %s" % administrative_state)
+            return False
+        return True
+
+    def _validate_heartbeat_interval(self, heartbeat_interval):
+        try:
+            # Check the value is an integer
+            val = int(heartbeat_interval)
+        except ValueError:
+            LOG.warning("Invalid heartbeat_interval: %s" % heartbeat_interval)
+            return False
+
+        # We do not support less than min or greater than max
+        if val < MIN_SYSTEM_PEER_HEARTBEAT_INTERVAL or \
+            val > MAX_SYSTEM_PEER_HEARTBEAT_INTERVAL:
+            LOG.debug("Invalid heartbeat_interval: %s" % heartbeat_interval)
+            return False
+        return True
+
+    def _validate_heartbeat_failure_threshold(self,
+                                              heartbeat_failure_threshold):
+        try:
+            # Check the value is an integer
+            val = int(heartbeat_failure_threshold)
+        except ValueError:
+            LOG.warning("Invalid heartbeat_failure_threshold: %s" %
+                        heartbeat_failure_threshold)
+            return False
+
+        # We do not support less than min or greater than max
+        if val < MIN_SYSTEM_PEER_HEARTBEAT_FAILURE_THRESHOLD or \
+            val > MAX_SYSTEM_PEER_HEARTBEAT_FAILURE_THRESHOLD:
+            LOG.debug("Invalid heartbeat_failure_threshold: %s" %
+                      heartbeat_failure_threshold)
+            return False
+        return True
+
+    def _validate_heartbeat_failure_policy(self, heartbeat_failure_policy):
+        if heartbeat_failure_policy not in \
+            SYSTEM_PEER_HEARTBEAT_FAILURE_POLICY_LIST:
+            LOG.debug("Invalid heartbeat_failure_policy: %s" %
+                      heartbeat_failure_policy)
+            return False
+        return True
+
+    def _validate_heartbeat_maintenance_timeout(self,
+                                                heartbeat_maintenance_timeout):
+        try:
+            # Check the value is an integer
+            val = int(heartbeat_maintenance_timeout)
+        except ValueError:
+            LOG.warning("Invalid heartbeat_maintenance_timeout: %s" %
+                        heartbeat_maintenance_timeout)
+            return False
+
+        # We do not support less than min or greater than max
+        if val < MIN_SYSTEM_PEER_HEARTBEAT_MAINTENACE_TIMEOUT or \
+            val > MAX_SYSTEM_PEER_HEARTBEAT_MAINTENACE_TIMEOUT:
+            LOG.debug("Invalid heartbeat_maintenance_timeout: %s" %
+                      heartbeat_maintenance_timeout)
+            return False
+        return True
+
+    @index.when(method='POST', template='json')
+    def post(self):
+        """Create a new system peer."""
+
+        policy.authorize(system_peer_policy.POLICY_ROOT % "create", {},
+                         restcomm.extract_credentials_for_policy())
+        context = restcomm.extract_context_from_environ()
+        LOG.info("Creating a new system peer: %s" % context)
+
+        payload = self._get_payload(request)
+        if not payload:
+            pecan.abort(httpclient.BAD_REQUEST, _('Body required'))
+
+        # Validate payload
+        peer_uuid = payload.get('peer_uuid')
+        if not self._validate_uuid(peer_uuid):
+            pecan.abort(httpclient.BAD_REQUEST, _('Invalid peer uuid'))
+
+        peer_name = payload.get('peer_name')
+        if not self._validate_name(peer_name):
+            pecan.abort(httpclient.BAD_REQUEST, _('Invalid peer name'))
+
+        endpoint = payload.get('manager_endpoint')
+        if not self._validate_manager_endpoint(endpoint):
+            pecan.abort(httpclient.BAD_REQUEST,
+                        _('Invalid peer manager_endpoint'))
+
+        username = payload.get('manager_username')
+        if not self._validate_manager_username(username):
+            pecan.abort(httpclient.BAD_REQUEST,
+                        _('Invalid peer manager_username'))
+
+        password = payload.get('manager_password')
+        if not self._validate_manager_password(password):
+            pecan.abort(httpclient.BAD_REQUEST,
+                        _('Invalid peer manager_password'))
+
+        gateway_ip = payload.get('peer_controller_gateway_address')
+        if not self._validate_peer_controller_gateway_ip(gateway_ip):
+            pecan.abort(httpclient.BAD_REQUEST,
+                        _('Invalid peer peer_controller_gateway_address'))
+
+        # Optional request parameters
+        kwargs = {}
+        administrative_state = payload.get('administrative_state')
+        if administrative_state:
+            if not self._validate_administrative_state(administrative_state):
+                pecan.abort(httpclient.BAD_REQUEST,
+                            _('Invalid peer administrative_state'))
+            kwargs['administrative_state'] = administrative_state
+
+        heartbeat_interval = payload.get('heartbeat_interval')
+        if heartbeat_interval is not None:
+            if not self._validate_heartbeat_interval(heartbeat_interval):
+                pecan.abort(httpclient.BAD_REQUEST,
+                            _('Invalid peer heartbeat_interval'))
+            kwargs['heartbeat_interval'] = heartbeat_interval
+
+        heartbeat_failure_threshold = \
+            payload.get('heartbeat_failure_threshold')
+        if heartbeat_failure_threshold is not None:
+            if not self._validate_heartbeat_failure_threshold(
+                heartbeat_failure_threshold):
+                pecan.abort(httpclient.BAD_REQUEST,
+                            _('Invalid peer heartbeat_failure_threshold'))
+            kwargs['heartbeat_failure_threshold'] = heartbeat_failure_threshold
+
+        heartbeat_failure_policy = payload.get('heartbeat_failure_policy')
+        if heartbeat_failure_policy:
+            if not self._validate_heartbeat_failure_policy(
+                heartbeat_failure_policy):
+                pecan.abort(httpclient.BAD_REQUEST,
+                            _('Invalid peer heartbeat_failure_policy'))
+            kwargs['heartbeat_failure_policy'] = heartbeat_failure_policy
+
+        heartbeat_maintenance_timeout = \
+            payload.get('heartbeat_maintenance_timeout')
+        if heartbeat_maintenance_timeout is not None:
+            if not self._validate_heartbeat_maintenance_timeout(
+                heartbeat_maintenance_timeout):
+                pecan.abort(httpclient.BAD_REQUEST,
+                            _('Invalid peer heartbeat_maintenance_timeout'))
+            kwargs['heartbeat_maintenance_timeout'] = \
+                heartbeat_maintenance_timeout
+
+        try:
+            peer_ref = db_api.system_peer_create(context,
+                                                 peer_uuid,
+                                                 peer_name,
+                                                 endpoint,
+                                                 username,
+                                                 password,
+                                                 gateway_ip, **kwargs)
+            return db_api.system_peer_db_model_to_dict(peer_ref)
+        except db_exc.DBDuplicateEntry:
+            LOG.info("Peer create failed. Peer UUID %s already exists"
+                     % peer_uuid)
+            pecan.abort(httpclient.CONFLICT,
+                        _('A system peer with this UUID already exists'))
+        except RemoteError as e:
+            pecan.abort(httpclient.UNPROCESSABLE_ENTITY, e.value)
+        except Exception as e:
+            LOG.exception(e)
+            pecan.abort(httpclient.INTERNAL_SERVER_ERROR,
+                        _('Unable to create system peer'))
+
+    @index.when(method='PATCH', template='json')
+    def patch(self, peer_ref):
+        """Update a system peer.
+
+        :param peer_ref: ID or UUID of system peer to update
+        """
+
+        policy.authorize(system_peer_policy.POLICY_ROOT % "modify", {},
+                         restcomm.extract_credentials_for_policy())
+        context = restcomm.extract_context_from_environ()
+        LOG.info("Updating system peer: %s" % context)
+
+        if peer_ref is None:
+            pecan.abort(httpclient.BAD_REQUEST,
+                        _('System Peer UUID or ID required'))
+
+        payload = self._get_payload(request)
+        if not payload:
+            pecan.abort(httpclient.BAD_REQUEST, _('Body required'))
+
+        peer = utils.system_peer_get_by_ref(context, peer_ref)
+        if peer is None:
+            pecan.abort(httpclient.NOT_FOUND, _('System Peer not found'))
+
+        peer_uuid, peer_name, endpoint, username, password, gateway_ip, \
+            administrative_state, heartbeat_interval, \
+            heartbeat_failure_threshold, heartbeat_failure_policy, \
+            heartbeat_maintenance_timeout = (
+                payload.get('peer_uuid'),
+                payload.get('peer_name'),
+                payload.get('manager_endpoint'),
+                payload.get('manager_username'),
+                payload.get('manager_password'),
+                payload.get('peer_controller_gateway_address'),
+                payload.get('administrative_state'),
+                payload.get('heartbeat_interval'),
+                payload.get('heartbeat_failure_threshold'),
+                payload.get('heartbeat_failure_policy'),
+                payload.get('heartbeat_maintenance_timeout')
+            )
+
+        if not (peer_uuid or peer_name or endpoint or username or password
+                or administrative_state or heartbeat_interval
+                or heartbeat_failure_threshold or heartbeat_failure_policy
+                or heartbeat_maintenance_timeout or gateway_ip):
+            pecan.abort(httpclient.BAD_REQUEST, _('nothing to update'))
+
+        # Check value is not None or empty before calling validate
+        if peer_uuid:
+            if not self._validate_uuid(peer_uuid):
+                pecan.abort(httpclient.BAD_REQUEST, _('Invalid peer uuid'))
+
+        if peer_name:
+            if not self._validate_name(peer_name):
+                pecan.abort(httpclient.BAD_REQUEST, _('Invalid peer name'))
+
+        if endpoint:
+            if not self._validate_manager_endpoint(endpoint):
+                pecan.abort(httpclient.BAD_REQUEST,
+                            _('Invalid peer manager_endpoint'))
+
+        if username:
+            if not self._validate_manager_username(username):
+                pecan.abort(httpclient.BAD_REQUEST,
+                            _('Invalid peer manager_username'))
+
+        if password:
+            if not self._validate_manager_password(password):
+                pecan.abort(httpclient.BAD_REQUEST,
+                            _('Invalid peer manager_password'))
+
+        if gateway_ip:
+            if not self._validate_peer_controller_gateway_ip(gateway_ip):
+                pecan.abort(httpclient.BAD_REQUEST,
+                            _('Invalid peer peer_controller_gateway_address'))
+
+        if administrative_state:
+            if not self._validate_administrative_state(administrative_state):
+                pecan.abort(httpclient.BAD_REQUEST,
+                            _('Invalid peer administrative_state'))
+
+        if heartbeat_interval:
+            if not self._validate_heartbeat_interval(heartbeat_interval):
+                pecan.abort(httpclient.BAD_REQUEST,
+                            _('Invalid peer heartbeat_interval'))
+
+        if heartbeat_failure_threshold:
+            if not self._validate_heartbeat_failure_threshold(
+                heartbeat_failure_threshold):
+                pecan.abort(httpclient.BAD_REQUEST,
+                            _('Invalid peer heartbeat_failure_threshold'))
+
+        if heartbeat_failure_policy:
+            if not self._validate_heartbeat_failure_policy(
+                heartbeat_failure_policy):
+                pecan.abort(httpclient.BAD_REQUEST,
+                            _('Invalid peer heartbeat_failure_policy'))
+
+        if heartbeat_maintenance_timeout:
+            if not self._validate_heartbeat_maintenance_timeout(
+                heartbeat_maintenance_timeout):
+                pecan.abort(httpclient.BAD_REQUEST,
+                            _('Invalid peer heartbeat_maintenance_timeout'))
+
+        try:
+            updated_peer = db_api.system_peer_update(
+                context,
+                peer.id,
+                peer_uuid, peer_name,
+                endpoint, username, password,
+                gateway_ip,
+                administrative_state,
+                heartbeat_interval,
+                heartbeat_failure_threshold,
+                heartbeat_failure_policy,
+                heartbeat_maintenance_timeout)
+            return db_api.system_peer_db_model_to_dict(updated_peer)
+        except RemoteError as e:
+            pecan.abort(httpclient.UNPROCESSABLE_ENTITY, e.value)
+        except Exception as e:
+            # additional exceptions.
+            LOG.exception(e)
+            pecan.abort(httpclient.INTERNAL_SERVER_ERROR,
+                        _('Unable to update system peer'))
+
+    @index.when(method='delete', template='json')
+    def delete(self, peer_ref):
+        """Delete the system peer."""
+
+        policy.authorize(system_peer_policy.POLICY_ROOT % "delete", {},
+                         restcomm.extract_credentials_for_policy())
+        context = restcomm.extract_context_from_environ()
+        LOG.info("Deleting system peer: %s" % context)
+
+        if peer_ref is None:
+            pecan.abort(httpclient.BAD_REQUEST,
+                        _('System Peer UUID or ID required'))
+        peer = utils.system_peer_get_by_ref(context, peer_ref)
+        if peer is None:
+            pecan.abort(httpclient.NOT_FOUND, _('System Peer not found'))
+
+        # TODO(jon): Add this back in when we have peer group associations
+        # a system peer may not be deleted if it is use by any associations
+        # association = db_api.peer_group_association_get_by_system_peer_id(context,
+        #                                                                   str(peer.id))
+        # if len(association) > 0:
+        #     pecan.abort(httpclient.BAD_REQUEST,
+        #                 _('System peer associated with peer group'))
+
+        try:
+            db_api.system_peer_destroy(context, peer.id)
+        except RemoteError as e:
+            pecan.abort(httpclient.UNPROCESSABLE_ENTITY, e.value)
+        except Exception as e:
+            LOG.exception(e)
+            pecan.abort(httpclient.INTERNAL_SERVER_ERROR,
+                        _('Unable to delete system peer'))

distributedcloud/dcmanager/api/policies/__init__.py

@@ -15,6 +15,7 @@ from dcmanager.api.policies import subcloud_group
 from dcmanager.api.policies import subclouds
 from dcmanager.api.policies import sw_update_options
 from dcmanager.api.policies import sw_update_strategy
+from dcmanager.api.policies import system_peers
 
 
 def list_rules():
@@ -27,5 +28,6 @@ def list_rules():
         sw_update_options.list_rules(),
         subcloud_group.list_rules(),
         subcloud_backup.list_rules(),
-        phased_subcloud_deploy.list_rules()
+        phased_subcloud_deploy.list_rules(),
+        system_peers.list_rules()
     )

distributedcloud/dcmanager/api/policies/system_peers.py

@@ -0,0 +1,65 @@
+# Copyright (c) 2023 Wind River Systems, Inc.
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+
+from dcmanager.api.policies import base
+from oslo_policy import policy
+
+POLICY_ROOT = 'dc_api:system_peers:%s'
+
+
+system_peers_rules = [
+    policy.DocumentedRuleDefault(
+        name=POLICY_ROOT % 'create',
+        check_str='rule:' + base.ADMIN_IN_SYSTEM_PROJECTS,
+        description="Create system peer.",
+        operations=[
+            {
+                'method': 'POST',
+                'path': '/v1.0/system-peers'
+            }
+        ]
+    ),
+    policy.DocumentedRuleDefault(
+        name=POLICY_ROOT % 'delete',
+        check_str='rule:' + base.ADMIN_IN_SYSTEM_PROJECTS,
+        description="Delete system peer.",
+        operations=[
+            {
+                'method': 'DELETE',
+                'path': '/v1.0/system-peers/{system_peer}'
+            }
+        ]
+    ),
+    policy.DocumentedRuleDefault(
+        name=POLICY_ROOT % 'get',
+        check_str='rule:' + base.READER_IN_SYSTEM_PROJECTS,
+        description="Get system peers.",
+        operations=[
+            {
+                'method': 'GET',
+                'path': '/v1.0/system-peers'
+            },
+            {
+                'method': 'GET',
+                'path': '/v1.0/system-peers/{system_peer}'
+            }
+        ]
+    ),
+    policy.DocumentedRuleDefault(
+        name=POLICY_ROOT % 'modify',
+        check_str='rule:' + base.ADMIN_IN_SYSTEM_PROJECTS,
+        description="Modify system peer.",
+        operations=[
+            {
+                'method': 'PATCH',
+                'path': '/v1.0/system-peers/{system_peer}'
+            }
+        ]
+    )
+]
+
+
+def list_rules():
+    return system_peers_rules

distributedcloud/dcmanager/common/exceptions.py

@@ -137,6 +137,18 @@ class SubcloudPatchOptsNotFound(NotFound):
                 "defaults will be used.")
 
 
+class SystemPeerNotFound(NotFound):
+    message = _("System Peer with id %(peer_id)s doesn't exist.")
+
+
+class SystemPeerNameNotFound(NotFound):
+    message = _("System Peer with peer_name %(name)s doesn't exist.")
+
+
+class SystemPeerUUIDNotFound(NotFound):
+    message = _("System Peer with peer_uuid %(uuid)s doesn't exist.")
+
+
 class SubcloudGroupNotFound(NotFound):
     message = _("Subcloud Group with id %(group_id)s doesn't exist.")
 

distributedcloud/dcmanager/common/utils.py

@@ -28,6 +28,7 @@ import six.moves
 import string
 import subprocess
 import tsconfig.tsconfig as tsc
+import uuid
 import xml.etree.ElementTree as ElementTree
 import yaml
 
@@ -495,6 +496,26 @@ def get_loads_for_prestage(loads):
     return [load.software_version for load in loads if load.state in valid_states]
 
 
+def system_peer_get_by_ref(context, peer_ref):
+    """Handle getting a system peer by either UUID, or ID, or Name
+
+    :param context: The request context
+    :param peer_ref: Reference to the system peer, either an UUID or an ID or
+                     a Name
+    """
+    try:
+        if peer_ref.isdigit():
+            return db_api.system_peer_get(context, peer_ref)
+        try:
+            uuid.UUID(peer_ref)
+            return db_api.system_peer_get_by_uuid(context, peer_ref)
+        except ValueError:
+            return db_api.system_peer_get_by_name(context, peer_ref)
+    except (exceptions.SystemPeerNotFound, exceptions.SystemPeerUUIDNotFound,
+            exceptions.SystemPeerNameNotFound):
+        return None
+
+
 def subcloud_get_by_ref(context, subcloud_ref):
     """Handle getting a subcloud by either name, or ID
 

distributedcloud/dcmanager/db/api.py

@@ -351,6 +351,96 @@ def subcloud_group_destroy(context, group_id):
     return IMPL.subcloud_group_destroy(context, group_id)
 
 
+###################
+# system_peer
+def system_peer_db_model_to_dict(system_peer):
+    """Convert system_peer db model to dictionary."""
+    result = {"id": system_peer.id,
+              "peer-uuid": system_peer.peer_uuid,
+              "peer-name": system_peer.peer_name,
+              "manager-endpoint": system_peer.manager_endpoint,
+              "manager-username": system_peer.manager_username,
+              "peer-controller-gateway-address": system_peer.
+              peer_controller_gateway_ip,
+              "administrative-state": system_peer.administrative_state,
+              "heartbeat-interval": system_peer.heartbeat_interval,
+              "heartbeat-failure-threshold": system_peer.
+              heartbeat_failure_threshold,
+              "heartbeat-failure-policy": system_peer.heartbeat_failure_policy,
+              "heartbeat-maintenance-timeout": system_peer.
+              heartbeat_maintenance_timeout,
+              "created-at": system_peer.created_at,
+              "updated-at": system_peer.updated_at}
+    return result
+
+
+def system_peer_create(context,
+                       peer_uuid, peer_name,
+                       endpoint, username, password,
+                       gateway_ip,
+                       administrative_state,
+                       heartbeat_interval,
+                       heartbeat_failure_threshold,
+                       heartbeat_failure_policy,
+                       heartbeat_maintenance_timeout):
+    """Create a system_peer."""
+    return IMPL.system_peer_create(context,
+                                   peer_uuid, peer_name,
+                                   endpoint, username, password,
+                                   gateway_ip,
+                                   administrative_state,
+                                   heartbeat_interval,
+                                   heartbeat_failure_threshold,
+                                   heartbeat_failure_policy,
+                                   heartbeat_maintenance_timeout)
+
+
+def system_peer_get(context, peer_id):
+    """Retrieve a system_peer or raise if it does not exist."""
+    return IMPL.system_peer_get(context, peer_id)
+
+
+def system_peer_get_by_uuid(context, uuid):
+    """Retrieve a system_peer by uuid or raise if it does not exist."""
+    return IMPL.system_peer_get_by_uuid(context, uuid)
+
+
+def system_peer_get_by_name(context, uuid):
+    """Retrieve a system_peer by name or raise if it does not exist."""
+    return IMPL.system_peer_get_by_name(context, uuid)
+
+
+def system_peer_get_all(context):
+    """Retrieve all system peers."""
+    return IMPL.system_peer_get_all(context)
+
+
+def system_peer_update(context, peer_id,
+                       peer_uuid, peer_name,
+                       endpoint, username, password,
+                       gateway_ip,
+                       administrative_state,
+                       heartbeat_interval,
+                       heartbeat_failure_threshold,
+                       heartbeat_failure_policy,
+                       heartbeat_maintenance_timeout):
+    """Update the system peer or raise if it does not exist."""
+    return IMPL.system_peer_update(context, peer_id,
+                                   peer_uuid, peer_name,
+                                   endpoint, username, password,
+                                   gateway_ip,
+                                   administrative_state,
+                                   heartbeat_interval,
+                                   heartbeat_failure_threshold,
+                                   heartbeat_failure_policy,
+                                   heartbeat_maintenance_timeout)
+
+
+def system_peer_destroy(context, peer_id):
+    """Destroy the system peer or raise if it does not exist."""
+    return IMPL.system_peer_destroy(context, peer_id)
+
+
 ###################
 
 def sw_update_strategy_db_model_to_dict(sw_update_strategy):

distributedcloud/dcmanager/db/sqlalchemy/api.py

@@ -764,6 +764,145 @@ def sw_update_opts_default_destroy(context):
         session.delete(sw_update_opts_default_ref)
 
 
+##########################
+# system peer
+##########################
+@require_context
+def system_peer_get(context, peer_id):
+    try:
+        result = model_query(context, models.SystemPeer). \
+            filter_by(deleted=0). \
+            filter_by(id=peer_id). \
+            one()
+    except NoResultFound:
+        raise exception.SystemPeerNotFound(peer_id=peer_id)
+    except MultipleResultsFound:
+        raise exception.InvalidParameterValue(
+            err="Multiple entries found for system peer %s" % peer_id)
+
+    return result
+
+
+@require_context
+def system_peer_get_by_name(context, name):
+    try:
+        result = model_query(context, models.SystemPeer). \
+            filter_by(deleted=0). \
+            filter_by(peer_name=name). \
+            one()
+    except NoResultFound:
+        raise exception.SystemPeerNameNotFound(name=name)
+    except MultipleResultsFound:
+        # This exception should never happen due to the UNIQUE setting for name
+        raise exception.InvalidParameterValue(
+            err="Multiple entries found for system peer %s" % name)
+
+    return result
+
+
+@require_context
+def system_peer_get_by_uuid(context, uuid):
+    try:
+        result = model_query(context, models.SystemPeer). \
+            filter_by(deleted=0). \
+            filter_by(peer_uuid=uuid). \
+            one()
+    except NoResultFound:
+        raise exception.SystemPeerUUIDNotFound(uuid=uuid)
+    except MultipleResultsFound:
+        # This exception should never happen due to the UNIQUE setting for uuid
+        raise exception.InvalidParameterValue(
+            err="Multiple entries found for system peer %s" % uuid)
+
+    return result
+
+
+@require_context
+def system_peer_get_all(context):
+    result = model_query(context, models.SystemPeer). \
+        filter_by(deleted=0). \
+        order_by(models.SystemPeer.id). \
+        all()
+
+    return result
+
+
+@require_admin_context
+def system_peer_create(context,
+                       peer_uuid, peer_name,
+                       endpoint, username, password,
+                       gateway_ip,
+                       administrative_state="enabled",
+                       heartbeat_interval=60,
+                       heartbeat_failure_threshold=3,
+                       heartbeat_failure_policy="alarm",
+                       heartbeat_maintenance_timeout=600):
+    with write_session() as session:
+        system_peer_ref = models.SystemPeer()
+        system_peer_ref.peer_uuid = peer_uuid
+        system_peer_ref.peer_name = peer_name
+        system_peer_ref.manager_endpoint = endpoint
+        system_peer_ref.manager_username = username
+        system_peer_ref.manager_password = password
+        system_peer_ref.peer_controller_gateway_ip = gateway_ip
+        system_peer_ref.administrative_state = administrative_state
+        system_peer_ref.heartbeat_interval = heartbeat_interval
+        system_peer_ref.heartbeat_failure_threshold = \
+            heartbeat_failure_threshold
+        system_peer_ref.heartbeat_failure_policy = heartbeat_failure_policy
+        system_peer_ref.heartbeat_maintenance_timeout = \
+            heartbeat_maintenance_timeout
+        session.add(system_peer_ref)
+        return system_peer_ref
+
+
+@require_admin_context
+def system_peer_update(context, peer_id,
+                       peer_uuid=None, peer_name=None,
+                       endpoint=None, username=None, password=None,
+                       gateway_ip=None,
+                       administrative_state=None,
+                       heartbeat_interval=None,
+                       heartbeat_failure_threshold=None,
+                       heartbeat_failure_policy=None,
+                       heartbeat_maintenance_timeout=None):
+    with write_session() as session:
+        system_peer_ref = system_peer_get(context, peer_id)
+        if peer_uuid is not None:
+            system_peer_ref.peer_uuid = peer_uuid
+        if peer_name is not None:
+            system_peer_ref.peer_name = peer_name
+        if endpoint is not None:
+            system_peer_ref.manager_endpoint = endpoint
+        if username is not None:
+            system_peer_ref.manager_username = username
+        if password is not None:
+            system_peer_ref.manager_password = password
+        if gateway_ip is not None:
+            system_peer_ref.peer_controller_gateway_ip = gateway_ip
+        if administrative_state is not None:
+            system_peer_ref.administrative_state = administrative_state
+        if heartbeat_interval is not None:
+            system_peer_ref.heartbeat_interval = heartbeat_interval
+        if heartbeat_failure_threshold is not None:
+            system_peer_ref.heartbeat_failure_threshold = \
+                heartbeat_failure_threshold
+        if heartbeat_failure_policy is not None:
+            system_peer_ref.heartbeat_failure_policy = heartbeat_failure_policy
+        if heartbeat_maintenance_timeout is not None:
+            system_peer_ref.heartbeat_maintenance_timeout = \
+                heartbeat_maintenance_timeout
+        system_peer_ref.save(session)
+        return system_peer_ref
+
+
+@require_admin_context
+def system_peer_destroy(context, peer_id):
+    with write_session() as session:
+        system_peer_ref = system_peer_get(context, peer_id)
+        session.delete(system_peer_ref)
+
+
 ##########################
 # subcloud group
 ##########################
@@ -897,7 +1036,6 @@ def initialize_subcloud_group_default(engine):
             pass
     except Exception as ex:
         LOG.error("Exception occurred setting up default subcloud group", ex)
-
 ##########################
 
 

distributedcloud/dcmanager/db/sqlalchemy/migrate_repo/versions/014_add_subcloud_peer_group_and_association.py

@@ -16,6 +16,36 @@ def upgrade(migrate_engine):
     # Add the 'rehome_data' column to the subclouds table.
     subclouds.create_column(sqlalchemy.Column('rehome_data', sqlalchemy.Text))
 
+    # Declare the new system_peer table
+    system_peer = sqlalchemy.Table(
+        'system_peer', meta,
+        sqlalchemy.Column('id', sqlalchemy.Integer,
+                          primary_key=True,
+                          autoincrement=True,
+                          nullable=False),
+        sqlalchemy.Column('peer_uuid', sqlalchemy.String(36), unique=True),
+        sqlalchemy.Column('peer_name', sqlalchemy.String(255), unique=True),
+        sqlalchemy.Column('manager_endpoint', sqlalchemy.String(255)),
+        sqlalchemy.Column('manager_username', sqlalchemy.String(255)),
+        sqlalchemy.Column('manager_password', sqlalchemy.String(255)),
+        sqlalchemy.Column('peer_controller_gateway_ip', sqlalchemy.String(255)),
+        sqlalchemy.Column('administrative_state', sqlalchemy.String(255)),
+        sqlalchemy.Column('heartbeat_interval', sqlalchemy.Integer),
+        sqlalchemy.Column('heartbeat_failure_threshold', sqlalchemy.Integer),
+        sqlalchemy.Column('heartbeat_failure_policy', sqlalchemy.String(255)),
+        sqlalchemy.Column('heartbeat_maintenance_timeout', sqlalchemy.Integer),
+        sqlalchemy.Column('heartbeat_status', sqlalchemy.String(255)),
+        sqlalchemy.Column('reserved_1', sqlalchemy.Text),
+        sqlalchemy.Column('reserved_2', sqlalchemy.Text),
+        sqlalchemy.Column('created_at', sqlalchemy.DateTime),
+        sqlalchemy.Column('updated_at', sqlalchemy.DateTime),
+        sqlalchemy.Column('deleted_at', sqlalchemy.DateTime),
+        sqlalchemy.Column('deleted', sqlalchemy.Integer, default=0),
+        mysql_engine=ENGINE,
+        mysql_charset=CHARSET
+    )
+    system_peer.create()
+
 
 def downgrade(migrate_engine):
     raise NotImplementedError('Database downgrade is unsupported.')

distributedcloud/dcmanager/db/sqlalchemy/models.py

@@ -100,6 +100,25 @@ class DCManagerBase(models.ModelBase,
         session.commit()
 
 
+class SystemPeer(BASE, DCManagerBase):
+    """Represents a system peer"""
+
+    __tablename__ = 'system_peer'
+
+    id = Column(Integer, primary_key=True, autoincrement=True, nullable=False)
+    peer_uuid = Column(String(36), unique=True)
+    peer_name = Column(String(255), unique=True)
+    manager_endpoint = Column(String(255))
+    manager_username = Column(String(255))
+    manager_password = Column(String(255))
+    peer_controller_gateway_ip = Column(String(255))
+    administrative_state = Column(String(255))
+    heartbeat_interval = Column(Integer)
+    heartbeat_failure_threshold = Column(Integer)
+    heartbeat_failure_policy = Column(String(255))
+    heartbeat_maintenance_timeout = Column(Integer)
+
+
 class SubcloudGroup(BASE, DCManagerBase):
     """Represents a subcloud group"""
 

distributedcloud/dcmanager/tests/unit/api/v1/controllers/test_system_peer.py

@@ -0,0 +1,316 @@
+# Copyright (c) 2023 Wind River Systems, Inc.
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+
+import mock
+from six.moves import http_client
+import uuid
+
+from dcmanager.db.sqlalchemy import api as db_api
+from dcmanager.rpc import client as rpc_client
+
+from dcmanager.tests.unit.api import test_root_controller as testroot
+from dcmanager.tests.unit.api.v1.controllers.mixins import APIMixin
+from dcmanager.tests.unit.api.v1.controllers.mixins import DeleteMixin
+from dcmanager.tests.unit.api.v1.controllers.mixins import GetMixin
+from dcmanager.tests.unit.api.v1.controllers.mixins import PostJSONMixin
+from dcmanager.tests.unit.api.v1.controllers.mixins import UpdateMixin
+from dcmanager.tests import utils
+
+SAMPLE_SYSTEM_PEER_UUID = str(uuid.uuid4())
+SAMPLE_SYSTEM_PEER_NAME = 'SystemPeer1'
+SAMPLE_MANAGER_ENDPOINT = 'http://127.0.0.1:5000'
+SAMPLE_MANAGER_USERNAME = 'admin'
+SAMPLE_MANAGER_PASSWORD = 'password'
+SAMPLE_ADMINISTRATIVE_STATE = 'enabled'
+SAMPLE_HEARTBEAT_INTERVAL = 10
+SAMPLE_HEARTBEAT_FAILURE_THRESHOLD = 3
+SAMPLE_HEARTBEAT_FAILURES_POLICY = 'alarm'
+SAMPLE_HEARTBEAT_MAINTENANCE_TIMEOUT = 600
+SAMPLE_PEER_CONTROLLER_GATEWAY_IP = '128.128.128.1'
+
+
+class SystemPeerAPIMixin(APIMixin):
+
+    API_PREFIX = '/v1.0/system-peers'
+    RESULT_KEY = 'system_peers'
+    EXPECTED_FIELDS = ['id',
+                       'peer-uuid',
+                       'peer-name',
+                       'manager-endpoint',
+                       'manager-username',
+                       'peer-controller-gateway-address',
+                       'administrative-state',
+                       'heartbeat-interval',
+                       'heartbeat-failure-threshold',
+                       'heartbeat-failure-policy',
+                       'heartbeat-maintenance-timeout',
+                       'created-at',
+                       'updated-at']
+
+    def setUp(self):
+        super(SystemPeerAPIMixin, self).setUp()
+        self.fake_rpc_client.some_method = mock.MagicMock()
+
+    def _get_test_system_peer_dict(self, data_type, **kw):
+        # id should not be part of the structure
+        system_peer = {
+            'peer_uuid': kw.get('peer_uuid', SAMPLE_SYSTEM_PEER_UUID),
+            'peer_name': kw.get('peer_name', SAMPLE_SYSTEM_PEER_NAME),
+            'administrative_state': kw.get('administrative_state',
+                                           SAMPLE_ADMINISTRATIVE_STATE),
+            'heartbeat_interval': kw.get('heartbeat_interval',
+                                         SAMPLE_HEARTBEAT_INTERVAL),
+            'heartbeat_failure_threshold': kw.get(
+                'heartbeat_failure_threshold', SAMPLE_HEARTBEAT_FAILURE_THRESHOLD),
+            'heartbeat_failure_policy': kw.get(
+                'heartbeat_failure_policy', SAMPLE_HEARTBEAT_FAILURES_POLICY),
+            'heartbeat_maintenance_timeout': kw.get(
+                'heartbeat_maintenance_timeout',
+                SAMPLE_HEARTBEAT_MAINTENANCE_TIMEOUT)
+        }
+
+        if data_type == 'db':
+            system_peer['endpoint'] = kw.get('manager_endpoint',
+                                             SAMPLE_MANAGER_ENDPOINT)
+            system_peer['username'] = kw.get('manager_username',
+                                             SAMPLE_MANAGER_USERNAME)
+            system_peer['password'] = kw.get('manager_password',
+                                             SAMPLE_MANAGER_PASSWORD)
+            system_peer['gateway_ip'] = kw.get(
+                'peer_controller_gateway_ip', SAMPLE_PEER_CONTROLLER_GATEWAY_IP)
+        else:
+            system_peer['manager_endpoint'] = kw.get('manager_endpoint',
+                                                     SAMPLE_MANAGER_ENDPOINT)
+            system_peer['manager_username'] = kw.get('manager_username',
+                                                     SAMPLE_MANAGER_USERNAME)
+            system_peer['manager_password'] = kw.get('manager_password',
+                                                     SAMPLE_MANAGER_PASSWORD)
+            system_peer['peer_controller_gateway_address'] = kw.get(
+                'peer_controller_gateway_ip', SAMPLE_PEER_CONTROLLER_GATEWAY_IP)
+        return system_peer
+
+    def _post_get_test_system_peer(self, **kw):
+        post_body = self._get_test_system_peer_dict('dict', **kw)
+        return post_body
+
+    # The following methods are required for subclasses of APIMixin
+
+    def get_api_prefix(self):
+        return self.API_PREFIX
+
+    def get_result_key(self):
+        return self.RESULT_KEY
+
+    def get_expected_api_fields(self):
+        return self.EXPECTED_FIELDS
+
+    def get_omitted_api_fields(self):
+        return []
+
+    def _create_db_object(self, context, **kw):
+        creation_fields = self._get_test_system_peer_dict('db', **kw)
+        return db_api.system_peer_create(context, **creation_fields)
+
+    def get_post_object(self):
+        return self._post_get_test_system_peer()
+
+    def get_update_object(self):
+        update_object = {
+            'peer_controller_gateway_address': '192.168.205.1'
+        }
+        return update_object
+
+
+# Combine System Peer API with mixins to test post, get, update and delete
+class TestSystemPeerPost(testroot.DCManagerApiTest,
+                         SystemPeerAPIMixin, PostJSONMixin):
+    def setUp(self):
+        super(TestSystemPeerPost, self).setUp()
+
+    def verify_post_failure(self, response):
+        # Failures will return text rather than JSON
+        self.assertEqual(response.content_type, 'text/plain')
+        self.assertEqual(response.status_code, http_client.BAD_REQUEST)
+
+    @mock.patch.object(rpc_client, 'ManagerClient')
+    def test_create_with_numerical_uuid_fails(self, mock_client):
+        # A numerical uuid is not permitted. otherwise the 'get' operations
+        # which support getting by either name or ID could become confused
+        # if a name for one peer was the same as an ID for another.
+        ndict = self.get_post_object()
+        ndict['peer_uuid'] = '123'
+        response = self.app.post_json(self.get_api_prefix(),
+                                      ndict,
+                                      headers=self.get_api_headers(),
+                                      expect_errors=True)
+        self.verify_post_failure(response)
+
+    @mock.patch.object(rpc_client, 'ManagerClient')
+    def test_create_with_blank_uuid_fails(self, mock_client):
+        # An empty name is not permitted
+        ndict = self.get_post_object()
+        ndict['peer_uuid'] = ''
+        response = self.app.post_json(self.get_api_prefix(),
+                                      ndict,
+                                      headers=self.get_api_headers(),
+                                      expect_errors=True)
+        self.verify_post_failure(response)
+
+    @mock.patch.object(rpc_client, 'ManagerClient')
+    def test_create_with_empty_manager_endpoint_fails(self, mock_client):
+        # An empty description is considered invalid
+        ndict = self.get_post_object()
+        ndict['manager_endpoint'] = ''
+        response = self.app.post_json(self.get_api_prefix(),
+                                      ndict,
+                                      headers=self.get_api_headers(),
+                                      expect_errors=True)
+        self.verify_post_failure(response)
+
+    @mock.patch.object(rpc_client, 'ManagerClient')
+    def test_create_with_wrong_manager_endpoint_fails(self, mock_client):
+        # An empty description is considered invalid
+        ndict = self.get_post_object()
+        ndict['manager_endpoint'] = 'ftp://somepath'
+        response = self.app.post_json(self.get_api_prefix(),
+                                      ndict,
+                                      headers=self.get_api_headers(),
+                                      expect_errors=True)
+        self.verify_post_failure(response)
+
+    @mock.patch.object(rpc_client, 'ManagerClient')
+    def test_create_with_wrong_peergw_ip_fails(self, mock_client):
+        # An empty description is considered invalid
+        ndict = self.get_post_object()
+        ndict['peer_controller_gateway_address'] = '123'
+        response = self.app.post_json(self.get_api_prefix(),
+                                      ndict,
+                                      headers=self.get_api_headers(),
+                                      expect_errors=True)
+        self.verify_post_failure(response)
+
+    @mock.patch.object(rpc_client, 'ManagerClient')
+    def test_create_with_bad_administrative_state(self, mock_client):
+        # update_apply_type must be either 'enabled' or 'disabled'
+        ndict = self.get_post_object()
+        ndict['administrative_state'] = 'something_invalid'
+        response = self.app.post_json(self.get_api_prefix(),
+                                      ndict,
+                                      headers=self.get_api_headers(),
+                                      expect_errors=True)
+        self.verify_post_failure(response)
+
+    @mock.patch.object(rpc_client, 'ManagerClient')
+    def test_create_with_bad_heartbeat_interval(self, mock_client):
+        # heartbeat_interval must be an integer between 1 and 600
+        ndict = self.get_post_object()
+        # All the entries in bad_values should be considered invalid
+        bad_values = [0, 601, -1, 'abc']
+        for bad_value in bad_values:
+            ndict['heartbeat_interval'] = bad_value
+            response = self.app.post_json(self.get_api_prefix(),
+                                          ndict,
+                                          headers=self.get_api_headers(),
+                                          expect_errors=True)
+            self.verify_post_failure(response)
+
+
+class TestSystemPeerGet(testroot.DCManagerApiTest,
+                        SystemPeerAPIMixin, GetMixin):
+    def setUp(self):
+        super(TestSystemPeerGet, self).setUp()
+
+    @mock.patch.object(rpc_client, 'ManagerClient')
+    def test_get_single_by_uuid(self, mock_client):
+        # create a system peer
+        context = utils.dummy_context()
+        peer_uuid = str(uuid.uuid4())
+        self._create_db_object(context, peer_uuid=peer_uuid)
+
+        # Test that a GET operation for a valid ID works
+        response = self.app.get(self.get_single_url(peer_uuid),
+                                headers=self.get_api_headers())
+        self.assertEqual(response.content_type, 'application/json')
+        self.assertEqual(response.status_code, http_client.OK)
+        self.validate_entry(response.json)
+
+    @mock.patch.object(rpc_client, 'ManagerClient')
+    def test_get_single_by_name(self, mock_client):
+        # create a system peer
+        context = utils.dummy_context()
+        peer_name = 'TestPeer'
+        self._create_db_object(context, peer_name=peer_name)
+
+        # Test that a GET operation for a valid ID works
+        response = self.app.get(self.get_single_url(peer_name),
+                                headers=self.get_api_headers())
+        self.assertEqual(response.content_type, 'application/json')
+        self.assertEqual(response.status_code, http_client.OK)
+        self.validate_entry(response.json)
+
+
+class TestSystemPeerUpdate(testroot.DCManagerApiTest,
+                           SystemPeerAPIMixin, UpdateMixin):
+    def setUp(self):
+        super(TestSystemPeerUpdate, self).setUp()
+
+    def validate_updated_fields(self, sub_dict, full_obj):
+        for key, value in sub_dict.items():
+            key = key.replace('_', '-')
+            self.assertEqual(value, full_obj.get(key))
+
+    @mock.patch.object(rpc_client, 'ManagerClient')
+    def test_update_invalid_administrative_state(self, mock_client):
+        context = utils.dummy_context()
+        single_obj = self._create_db_object(context)
+        update_data = {
+            'administrative_state': 'something_bad'
+        }
+        response = self.app.patch_json(self.get_single_url(single_obj.id),
+                                       headers=self.get_api_headers(),
+                                       params=update_data,
+                                       expect_errors=True)
+        # Failures will return text rather than json
+        self.assertEqual(response.content_type, 'text/plain')
+        self.assertEqual(response.status_code, http_client.BAD_REQUEST)
+
+    @mock.patch.object(rpc_client, 'ManagerClient')
+    def test_update_invalid_heartbeat_interval(self, mock_client):
+        context = utils.dummy_context()
+        single_obj = self._create_db_object(context)
+        update_data = {
+            'heartbeat_interval': -1
+        }
+        response = self.app.patch_json(self.get_single_url(single_obj.id),
+                                       headers=self.get_api_headers(),
+                                       params=update_data,
+                                       expect_errors=True)
+        # Failures will return text rather than json
+        self.assertEqual(response.content_type, 'text/plain')
+        self.assertEqual(response.status_code, http_client.BAD_REQUEST)
+
+
+class TestSystemPeerDelete(testroot.DCManagerApiTest,
+                           SystemPeerAPIMixin, DeleteMixin):
+    def setUp(self):
+        super(TestSystemPeerDelete, self).setUp()
+
+    @mock.patch.object(rpc_client, 'ManagerClient')
+    def test_delete_by_uuid(self, mock_client):
+        context = utils.dummy_context()
+        peer_uuid = str(uuid.uuid4())
+        self._create_db_object(context, peer_uuid=peer_uuid)
+        response = self.app.delete_json(self.get_single_url(peer_uuid),
+                                        headers=self.get_api_headers())
+        self.assertEqual(response.status_int, 200)
+
+    @mock.patch.object(rpc_client, 'ManagerClient')
+    def test_delete_by_name(self, mock_client):
+        context = utils.dummy_context()
+        peer_name = 'TestPeer'
+        self._create_db_object(context, peer_name=peer_name)
+        response = self.app.delete_json(self.get_single_url(peer_name),
+                                        headers=self.get_api_headers())
+        self.assertEqual(response.status_int, 200)