summaryrefslogtreecommitdiff
path: root/specs/2019.03/approved/deployment-improvements-2004695-ansible-bootstrap-deployment.rst
blob: a1e20cdad61e74a9773330ec96f2ecd760d5b049 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
..
 This work is licensed under a Creative Commons Attribution 3.0 Unported
 License.

 http://creativecommons.org/licenses/by/3.0/legalcode


============================
Ansible Bootstrap Deployment
============================

Storyboard: https://storyboard.openstack.org/#!/story/2004695.

This spec describes the initial phase of StarlingX deployment improvement
effort.

Problem description
===================

The primary controller is currently configured using the ``config_controller``
Python script which can only be executed on the controller console. The script
requires input for many networking aspects upfront in order to run both
bootstrap operations and host configuration to completion. Over time, the
script logic has grown overly complex to accommodate a plethora of host
configuration scenarios and so has increased the configuration time.

Furthermore, once all required input configuration parameters have been
successfully validated, the script will run all its steps. If the script fails
due to a software issue or a configuration mistake, a re-install will be
required. It is not possible for the user to apply a software patch and/or
rerun the script to apply updated configurations.

Use Cases
=========

* As a developer/tester/operator, I need the ability to configure the
  controller remotely.
* As a developer/tester/operator, I need to the ability to modify and
  reapply configurations during initial host config.
* As a developer/tester/operator, I need the ability to automate the
  initial host deployment and build out my system from there.
* As a developer of StarlingX community, I would like to streamline
  the initial host config using an industry adopted tool to enable
  automation and to promote process/code visibility and customization.

Proposed change
===============

Existing workflow with config_controller (high level)
-----------------------------------------------------
**Config_controller:**

1. Create bootstrap hiera config
2. Apply bootstrap puppet manifest
3. Persist local configuration
4. Populate initial system inventory
5. Create system hiera config
6. Apply controller puppet manifest
7. Finalize controller configuration
8. Activate all services

**Host-configuration:**

   Manual or scripted configurations required for unlock.

**Host-unlock:**

1. Apply controller puppet manifest (and worker, storage puppet manifests
   for All-in-one)
2. Activate all services

Proposed workflow with Ansible Playbook (high level)
----------------------------------------------------
The bootstrap and configuration of the initial host will be orchestrated
by an Ansible Playbook [1]_.

**Playbook:**

1. Apply bootstrap puppet manifest
2. Populate system configuration (with defaults and user-supplied config)
3. Bring up Kubernetes master node and essential services

**Host-configuration:**

   Manual or scripted configurations required for unlock.

**Host-unlock**

1. Apply controller puppet manifest (and worker, storage puppet manifests
   for All-in-one)
2. Activate all services

After phase #2 of the Playbook, the host configuration will resemble
All-in-one simplex (i.e. defaulting to the loopback interface) until it
is unlocked for the first time. Interface configuration is being deferred
to ensure the network connection is not interrupted while the playbook is
being *played*. Interface reconfiguration will only take effect on unlock
operations. Previously, this would occur as part of the controller
manifest apply which has been eliminated.

Scope of the new workflow
-------------------------
The new workflow will cover the **initial config** for all supported system
configurations in a containerized platform.

Bootstrap playbook roles and tasks (high level)
-----------------------------------------------
Below is a list of major roles and tasks. The names are deliberately long
to make them self-explanatory for review purpose. They can be renamed to
be more terse as role variables should be prefixed with role names.
During implementation, some roles and tasks will likely be decomposed or
combined.

Role: validate-config-input
   * Task: validate-config
Role: prepare-environment-for-execution
   * Task: validate-environment
   * Task: set-environment-variables
Role: cleanup-environment-after-execution
   * Task: unset-environment-variables
   * Task: remove-temp-files
Role: store-admin-password
   * Task: validate-password
   * Task: store-password
Role: apply-bootstrap-manifest
   * Task: generate-bootstrap-data
   * Task: apply-manifest
Role: populate-initial-config
   * Task: persist-keyring
   * Task: set-permanent-puppet-workdir
   * Task: set-permanent-pxe-configdir
   * Task: set-postgres-config-for-mate
   * Task: process-branding-and-banner
   * Task: populate-system-config
   * Task: populate-load-config
   * Task: populate-network-config
   * Task: populate-controller-config
   * Task: create-loopback-interface
   * Task: update-local-dns
   * Task: update-platform-config-file
   * Task: add-dns-server
Role: bring-up-kubernetes-master-and-dependent-services
   * Task: bring-up-kubernetes-master
   * Task: bring-up-tiller
   * Task: bring-up-fault-management
   * Task: bring-up-maintenance
   * Task: bring-up-vim

Playbook directory layout
-------------------------
The directory layout of the playbook initially could be as follows:

bootstrap.yml

roles/
  validate-config-input/
    tasks/
      main.yml
    handlers/
      main.yml
    files/
      <scripts, files>
    vars/
      main.yml
    defaults/
      main.yml
    meta/
      main.yml

  prepare-environment-for-execution/

  cleanup-environment-after-execution/

  store-admin-password/

  apply-bootstrap-manifest/

  popupate-initial-config/

  bring-up-Kubernetes-master-and-dependent-services/

Playbook pre_tasks and post_tasks
---------------------------------
The pre_tasks and post_tasks can be as simple as marking the start and end
of the playbook execution.

Running ``bootstrap playbook``
------------------------------
ansible-playbook bootstrap.yml -u <named-account-with-sudo-privileges>
[-K -i <config-input-file> -e <list-of-variable-value-pairs-to-overwrite>
--ask-vault-password]

The playbook should be run using wrsroot account. However, it can be run using
another account with sudo privileges if desired provided that the account has
already been setup beforehand. Many playbook tasks must be run as root.
The option -K will prompt for privilege escalation password.

Overwriting playbook defaults
-----------------------------
The ``bootstrap playbook`` will come with default variables and Ansible
hosts file /etc/ansible/hosts.yml. These defaults and content of the hosts
file are meant for running the playbook locally and bootstrapping the initial
controller for All-in-one simplex in virtual box. In practice, some of these
defaults will need to be overwritten with user supplied values.

Variables that usually require overwriting are:

* host IP (for running the playbook remotely)
* system properties
* Management, OAM, PXE, cluster subnets
* Default DNS server

There are various ways to overwrite variables in Ansible Playbook.

**Overwrite with configuration input file**

One simple and clean option is to overwrite with -i command line parameter.
The content of the provided configuration input file must be in YAML format.

The default hosts (Ansible inventory) file will have the following entries:

bootstrap:
  hosts:
    local:
      ansible_connection: local

  vars:
    ansible_user: wrsroot
    ansible_become: true

To overwrite the bootstrap host for remote execution and/or user in the custom
configuration input file:

bootstrap:
  hosts:
    remote:
      ansible_host: '128.224.150.83'
      ansible_connection: ssh

  vars:
    ansible_user: wrsroot
    ansible_become: true

To overwrite the role default variables, one option is to add the list of of
overwritten variables under ``vars`` section of the configuration input file:

  vars:
    system_mode: duplex-direct
    dns_server: 8.8.8.8

**Overwrite with role vars**

Another option to overwrite role defaults is to replace main.yml file under
``vars`` directory of the corresponding role(s) with custom one(s) before
running the playbook. This takes precedence over the overwriting method above.

**Overwrite with extra vars**

Command line -e option which has the highest precedence can also be used
to overwrite defaults. However, this method can be cumbersome if many
defaults need overwriting and the playbook is run manually.

The list of role defaults as well as the preferred method to overwrite
these defaults will be documented after the playbook has been developed.

Overwriting sensitive variables
-------------------------------
The admin password is a sensitive variable that usually needs to be
overwritten. To ensure sensitive information is encrypted, sensitive
variables and values are copied to a vault file and secure using
ansible-vault encrypt command. The corresponding defaults will need to be
mapped to the variables in vaulted file using jinja2 syntax.

The command line argument --ask-vault-pass or --vault-password-file will need
to be supplied when running the playbook with encrypted vault file.

For development/test purposes, these variables can simply be overwritten
using the command line -e option.

Validating configuration parameters
-----------------------------------
The config_controller script has extensive logic to validate config
parameters in user input file which could be leveraged in
validate-config-input role of the ``bootstrap playbook``.

Config_controller script changes
--------------------------------
Currently this complex script has multiple uses: a) perform initial
configuration required mainly to bring up the controller services,
b) backup system configuration, c) restore system configuration from
backup file, d) clone the image, and e) restore the system from a clone.

The proposed Ansible bootstrap deployment will replace the initial system
configuration aspect of the script. The script will continue to be used for
other operations. Relevant code will be removed from the script once the
implementation of the playbook is complete.

Puppet changes
--------------
The initial ``bootstrap playbook`` will leverage the existing Puppet
bootstrap.pp manifest to bring up the following services that will be
used by the playbook for the remaining tasks:

**Required services to bring up Kubernetes master:**

* docker
* etcd

**Required services for host unlock:**

* fm
* mtcAgent
* nfv-vim

The puppet .pp and in some cases .py files related to these services and
Kubernetes will require update.

Sysinv changes
--------------
Traditionally, the ``config_controller`` script is provided with all
required parameters either interactively or via a config file to perform
both bootstrap operations and host configuration. Networking and storage
provisioning using system commands beyond this point have certain
restrictions as the controller manifest has been applied.

With Ansible bootstrap deployment method, some system commands will
require changes to support manual configuration adjustments and replays of
the ``bootstrap playbook``. The ``cgtsclient`` will also need minor
modification to avoid requesting for smapi endpoint which is not yet
available in this early stage.

Maintenance changes
-------------------
Some minor tweaks to maintenance code will be required for maintenance
Client and Agent to operate properly during the bootstrap phase.

Packaging of ``bootstrap playbook`` in the ISO and SDK
------------------------------------------------------
The playbook will be packaged in the ISO as well as SDK to allow
both local and remote execution.

Alternatives
============

Additional host configuration roles to support the initial host-unlock
were considered. However, this would add much of the complex modeling of
input configuration (i.e. more upfront planning) to the intial deployment step.

Data model impact
=================

No impact to existing system inventory data model.

REST API impact
===============

At this time, no REST API impact is anticipated.

Security impact
===============

The proposal is to make use of Ansible Playbook which is a well adopted
multi-node configuration and deployment orchestration tool partly due to
Ansible secure architecture and design.

The scope of the proposed ``bootstrap playbook`` is limited to bringing the
initial controller to the state where it can be unlocked and allow other
Kubernetes nodes on an internal cluster network if configured to join.

The Playbook can only be executed remotely over SSH using a named account
with sudo privileges. Ansible vault will be used to store secrets/private
information where applicable. As such, no additional security impact is
introduced.

Other end user impact
=====================

The user will be expected to interact with the feature using
ansible-playbook [2]_ and ansible-vault [3]_ commands. The bootstrap deployment
method will give the user more flexibility to customize and automate
the deployment.

Once the initial controller is ready to accept system commands and
Kubernetes master is up, the user can:
* perform minimum host configurations and unlock the host
* join other Kubernetes nodes and perform more extensive custom
configurations before the unlock

The playbook can be replayed to update system properties and general
networking information. It will not be playable after the host is unlocked.

Performance Impact
==================

Ansible execution overhead is unknown at this time. However, as the
controller manifest application and services activation steps are deferred
till host-unlock, the time to bring the controller to unlock-ready state
should be significantly faster than with the traditional method.

Other deployer impact
=====================

None

Developer impact
================

See end user impact.

The developers can extend the ``bootstrap playbook`` with custom host
configuration role(s) or another playbook to suit their specific needs.

Upgrade impact
==============

None as this is the initial release of Bootstrap Deployment using
Ansible Playbook.

Implementation
==============

Assignee(s)
===========

Primary assignee:

* Tee Ngo (teewrs)

Other contributors:

* Eric McDonald (emacdona)

Repos Impacted
==============

* stx-config
* stx-metal
* stx-root
* stx-docs

Work Items
==========

* Modify maintenance to enable maintenance operations during bootstrap
  phase.
* Modify sysinv and cgtsclient to be more flexible with configuration
  updates during bootstrap deployment using either system commands or APIs.
* Modify puppet classes and python scripts to allow launching a limited
  number of services required for bootstrap operations and initial host
  unlock.
* Create a ``bootstrap`` Playbook to bring up Kubernetes master node and
  configure the primary controller based on default and user-supplied config
  parameters.
* Package the Playbook as part of the ISO & SDK to allow both on premise
  and remote execution.
* Make other necessary changes to support primary controller configuration
  using either the playbook or traditional config_controller until the
  transition is complete. This includes lab setup tool changes.


Dependencies
============

* config_controller script
* Ansible [4]_
* Containerized OpenStack based deployment

Testing
=======

This story changes the way StarlingX system is deployed, specifically
how the primary controller is configured, which will require changes in
existing automated installation and lab setup tools.

The system deployment tests will be limited to All-in-one simplex,
All-in-one duplex, and Standard configurations. Deployment tests for
Region and Distributed Cloud configurations are deferred until the support
for these configurations in a containerized OpenStack based platform is
available. At which point, either the ``bootstrap playbook`` will be
extended with additional roles or with new playbook(s) to process steps in
``config_region`` and ``config_subcloud``. This will be documented either
in a later version of this spec or in a separate spec.

Documentation Impact
====================

This story affects the StarlingX installation and configuration
documentation. Specific details of the documentation changes will be
addressed once the implementation is complete.

References
==========

.. [1]  https://docs.ansible.com/ansible/2.7/user_guide/playbooks.html
.. [2]  https://docs.ansible.com/ansible/2.7/cli/ansible-playbook.html
.. [3]  https://docs.ansible.com/ansible/2.7/cli/ansible-vault.html
.. [4]  https://docs.ansible.com/ansible/2.7/index.html

History
=======

.. list-table:: Revisions
   :header-rows: 1

   * - Release Name
     - Description
   * - TBD
     - Introduced