Ansible Development Lifecycle for Epic on Azure

Overview

This guide covers the complete lifecycle of Ansible development for Epic on Azure. MUST understand that everything you develop locally eventually runs in AWX, where playbooks pull roles via requirements.yml.

Key Concepts

Concept	Description	Critical Rule
requirements.yml	Single source of truth for role versions	MUST update for every role change
Git tags	Production-stable versions	ALWAYS use tags in production
Feature branches	Testing unreleased changes	NEVER use branches in production
AWX execution	Final destination for all code	MUST test in AWX before production

Repository Structure

Playbooks Repository: ohemr-ansible-playbooks

MUST follow this structure:

ohemr-ansible-playbooks/
├── playbooks/
│   ├── epic-on-azure/
│   │   ├── pb_hyperspace.yml
│   │   ├── pb_citrix_vda.yml
│   │   ├── pb_dynatrace_oneagent_linux_install.yml
│   │   └── vars/
│   └── shared/
├── roles/
│   ├── requirements.yml  # ⭐ CRITICAL: Role version management
│   └── my_chart/         # Optional local roles
├── inventories/
├── collections/
│   └── requirements.yml
└── ansible.cfg

Role Repositories: ohemr-ansible-role-*

MUST include these directories:

ohemr-ansible-role-citrix-vda/
├── defaults/
│   └── main.yml          # User-overridable variables
├── tasks/
│   └── main.yml          # Main task file
├── handlers/
│   └── main.yml          # Handler definitions
├── templates/            # Jinja2 templates
├── files/               # Static files
├── vars/                # Internal variables
├── meta/
│   └── main.yml         # Role metadata and dependencies
├── molecule/            # Testing (REQUIRED)
├── tests/
├── README.md            # Documentation (REQUIRED)
├── CHANGELOG.md         # Version history (REQUIRED)
└── vitals.yaml

Critical: requirements.yml Role Versioning

The roles/requirements.yml file is the single source of truth for role versions.

Sample requirements.yml

---
# roles/requirements.yml in ohemr-ansible-playbooks

# Base OS configuration
- name: os_base_configuration
  src: git+https://github.com/optum-tech-compute/ohemr-ansible-role-base-os-config.git
  scm: git
  version: v1.31.1 # Git tag for stable release

# Citrix VDA role
- name: citrix_vda
  src: https://github.com/optum-tech-compute/ohemr-ansible-role-citrix-vda
  version: v1.0.1 # Git tag

# Development/testing: feature branch
- name: hyperspace
  src: git+https://github.com/optum-tech-compute/ohemr-ansible-role-hyperspace.git
  scm: git
  version: feature/memory-optimization # Feature branch for testing ONLY

Version Types

Type	Example	Use Case	Environment
Git Tags	v1.2.3	Stable releases	MUST use in production
Branches	feature/x	Testing changes	dev/qa ONLY
Commit SHA	abc123	Pinning to commit	Rare use case

Development Workflow

Phase 1: Local Development

Creating a New Playbook

MUST follow this structure:

---
- name: Deploy My Application
  hosts: my_app_servers
  gather_facts: true
  become: true

  pre_tasks:
    - name: Get secrets from HCP Vault
      ansible.builtin.import_role:
        name: utilities
        tasks_from: get_vault_secrets.yml
      tags: always
      vars:
        vault_secrets:
          - vault_path: 'data/my-app/{{ env }}/credentials'
            fact_name: app_password
            vault_property: password

  roles:
    - role: os_base_configuration
      tags: base

    - role: my_application
      tags: app

  post_tasks:
    - name: Verify application health
      ansible.builtin.uri:
        url: 'http://localhost:8080/health'
        status_code: 200
      tags: verify

Creating a New Role

MUST include these files:

defaults/main.yml (user-overridable):

---
my_app_version: '2.0.0'
my_app_port: 8080
my_app_install_dir: '/opt/my-app'
my_app_service_name: 'my-application'
my_app_enable_monitoring: true

tasks/main.yml:

---
- name: Include OS-specific variables
  ansible.builtin.include_vars: '{{ ansible_os_family }}.yml'
  tags: always

- name: Install prerequisites
  ansible.builtin.package:
    name: '{{ my_app_prerequisites }}'
    state: present
  tags: install

- name: Create application directory
  ansible.builtin.file:
    path: '{{ my_app_install_dir }}'
    state: directory
    owner: '{{ my_app_user }}'
    group: '{{ my_app_group }}'
    mode: '0755'
  tags: install

- name: Configure application
  ansible.builtin.template:
    src: app.conf.j2
    dest: '{{ my_app_install_dir }}/app.conf'
    owner: '{{ my_app_user }}'
    group: '{{ my_app_group }}'
    mode: '0644'
  notify: Restart my application
  tags: config

handlers/main.yml:

---
- name: Restart my application
  ansible.builtin.systemd:
    name: '{{ my_app_service_name }}'
    state: restarted
  become: true

Phase 2: Local Testing

MUST test locally before pushing:

# 1. Install role dependencies
cd ~/scm/optum-tech-compute/ohemr-ansible-playbooks
ansible-galaxy install -r roles/requirements.yml -p ./roles --force

# 2. ALWAYS run check mode first (dry run)
ansible-playbook playbooks/epic-on-azure/pb_my_application.yml \
  -i inventories/dev/hosts \
  --limit dev-app-server-01 \
  --check \
  --diff

# 3. Execute only if dry run looks good
ansible-playbook playbooks/epic-on-azure/pb_my_application.yml \
  -i inventories/dev/hosts \
  --limit dev-app-server-01 \
  -e "app_version=2.1.0"

Test Role with Molecule

MUST use Molecule for role testing:

cd ~/scm/optum-tech-compute/ohemr-ansible-role-my-application
molecule init scenario --driver-name docker
molecule test

Phase 3: Feature Branch Testing

Scenario A: Playbook-Only Changes

1. Create feature branch in playbooks repo
2. Make changes, commit, push
3. Test in AWX with branch override:

awx_job_launch_list:
  - name: 'deploy-my-app-dev'
    scm_branch: 'feature/app-deployment-improvements'
    limit: 'dev-app-server-01'
    wait: true

Scenario B: Role-Only Changes

1. Create role feature branch
2. MUST create playbooks feature branch with updated requirements.yml:

# In playbooks feature branch, update roles/requirements.yml
- name: my_application
  src: git+https://github.com/optum-tech-compute/ohemr-ansible-role-my-application.git
  version: feature/add-monitoring-support # Feature branch for testing

1. Test coordinated branches in AWX

Scenario C: Coordinated Playbook + Role Changes

MUST follow this order:

1. Create role feature branch → make changes → push
2. Create playbooks feature branch
3. Update requirements.yml to use role feature branch
4. Test together in AWX

Phase 4: Role Release

MUST follow semantic versioning:

# 1. Merge role feature branch
git checkout main
git merge feature/add-monitoring-support
git push origin main

# 2. Create semantic version tag
git tag -a v1.1.0 -m "feat: add Dynatrace monitoring support"
git push origin v1.1.0

# 3. Update playbooks requirements.yml to use new version
# In ohemr-ansible-playbooks:
- name: my_application
  src: git+https://github.com/optum-tech-compute/ohemr-ansible-role-my-application.git
  version: v1.1.0  # New stable version

Version Type	When to Use	Example
Major (v2.0.0)	Breaking changes	API changes
Minor (v1.1.0)	New features	New capabilities
Patch (v1.0.1)	Bug fixes	Fixes only

AWX Execution

How AWX Executes Your Code

1. Project Sync: AWX pulls playbooks repo (specific branch)
2. Role Installation: AWX runs ansible-galaxy install -r roles/requirements.yml
3. Playbook Execution: Runs playbook with roles available

AWX Job Template Configuration

awx_job_template_list:
  - name: 'deploy-my-app-dev'
    description: 'Deploy My Application to dev'
    organization: 'Epic Platform'
    inventory: 'epic-dev-inventory'
    project: 'ohemr-ansible-playbooks'
    playbook: 'playbooks/epic-on-azure/pb_my_application.yml'
    execution_environment: 'Epic EE'
    credentials:
      - 'epic-ssh-credential-dev'
      - 'azure-sp-dev'

    # MUST enable these for testing flexibility
    ask_scm_branch_on_launch: true
    ask_variables_on_launch: true
    ask_limit_on_launch: true

Best Practices

Role Development

Practice	Rule
Semantic versioning	MUST tag releases properly
Idempotent tasks	MUST be safe to run multiple times
defaults/main.yml	MUST make everything configurable
Documentation	MUST include README.md with all vars
Molecule tests	MUST verify role works
Tags	ALWAYS use for selective execution
CHANGELOG	MUST document changes

Playbook Development

• MUST keep playbooks simple - orchestration, not implementation
• MUST use roles for reusable logic
• ALWAYS use pre_tasks for secrets retrieval
• ALWAYS use post_tasks for verification
• MUST tag everything for selective execution

requirements.yml Management

• MUST pin production versions with git tags
• NEVER use branches in production requirements.yml
• ALWAYS test with feature branches before tagging
• MUST comment why versions are chosen
• ALWAYS test updates before production

Troubleshooting

Issue	Cause	Solution
Role not updating	AWX cached old version	Set ansible_galaxy_force: true
Wrong role version	Wrong requirements.yml	Check branch being pulled
Role fails in AWX	Missing dependencies	Check execution environment
Can't test role branch	No playbooks branch	Create playbooks feature branch

Summary

Key Takeaways:

1. MUST understand: Playbooks orchestrate, roles implement
2. MUST use: requirements.yml as single source of truth
3. MUST follow: Git tags for stable releases, branches for testing only
4. ALWAYS test: locally first, then feature branches in AWX
5. NEVER deploy: untested code to production
6. MUST document: README, CHANGELOG, variable docs

Remember: Your local development is the starting point, but AWX execution is the destination. ALWAYS consider how changes will behave in AWX's execution environment.