Setting up Ansible control node for managing Linux and Windows hosts

Background

In this post, I will walk you through the complete end-to-end procedure for setting up an Ansible controller node and setting up the connection for managing RHEL and Windows Server nodes.

Azure setup

I have created 3 servers and placed them in a simple network configuration.

Infra:

    10.0.3.4 ansiblecontrol (RHEL 8.7)

App:

    10.0.5.4 linuxnode1 (RHEL 8.7)

    10.0.5.5 windowsnode1 (Windows Server 2019)

OS preparation

Ensure all hosts are configured with the correct FQDN and IP configurations. This is more so important when dealing with domain-joined nodes.

ex:

• run hostnamectl set-hostname ansiblecontrol.local.vanguard.com to set the hostname
• add DNS suffix to computer name on System Settings

Make sure the control node can reach the managed nodes in the absence of DNS by adding entries in /etc/hosts

Setting up Ansible on Control node

1. Install ansible sudo yum install ansible

     2. Generate SSH key pair for ansible user ssh-keygen

     3. Copy public key to linuxnode1 ssh-copy-id linuxnode1.local.vanguard.com

     4. Install pywinrm to manage Windows nodes pip3 install "pywinrm>=0.3.0"

     5. Import windows module collection ansible-galaxy collection install ansible.windows

Setting up Linux managed node

1. Add ansible user on linuxnode1 sudo useradd ansible
2. Add ansible user to the sudoers file for password-less authentication for all elevated commands using sudo visudo

Setting up Windows managed node

     1. Ensure PowerShell version is later than 5.0 using $PSVersionTable command. Also make sure .NET version is later than 4.0. However, with newer Windows Servers, this shouldnt be an issue.

    2. Set up WinRM listener on port 5986 for HTTPS transport. This requires the creation of a self-signed certificate. Ansible has provided a script for this task which is available to download on GitHub

    2. Additionally, make sure Windows Firewall is switched off

Sample Ansible directory setup

1. Create inventory file by grouping the 2 managed nodes by operating system

    2. Define group_vars.

        Create a new file called windows.yml in group_vars. The contents of the file defines the connection parameters that will be used by ansible to connect with hosts under the windows group of the inventory file. This will also work for any dynamic inventory as long as those hosts are also added to the windows group during playbook runtime.

         

        For domain-joined windows servers, it is advised to use the more secure kerberos instead of ntlm. In such cases, kerberos packages must be installed and krb5.conf configured on the ansible control node to obtain kerberos tickets from the AD domain.

    3. Encrypt windows group_vars using ansible-vault to protect sensitive information

        ansible-vault encrypt windows.yml

    

    4. Perform a ping to verify connectivity using ad-hoc ping command. Make sure to use --ask-vault-pass for windows nodes

Sample playbook

 1. The following playbook will install webserver on both nodes depending on their OS

2. Execute the playbook first by doing a dry-run

ansible-playbook installwebserver.yml -i testservers --check --ask-vault-pass

3. Run the playbook

ansible-playbook installwebserver.yml -i testservers --ask-vault-pass

Password sync issues between AD and Entra ID - ‘User must change password at next logon’

Background

You want to reset a user's password in an Entra ID-connected Active Directory environment.

Issue

In Active Directory Users and Computers, you reset a user's password by making sure 'User must change password at next login' is selected.

The password was reset successfully. However, after allowing enough time for the Entra Connect sync, the user cannot authenticate with the new password on M365 and/or the company portal. The error message indicates an incorrect password.

Troubleshooting steps

In the Entra ID admin portal, the password for the same user is reset, enabling successful authentication into web-based services, with the updated password also reflected in Active Directory.

It appeared that the password hash sync only works from Entra to AD, not the other way around.

If the ‘User must change password at next logon’ flag was deselected when the password was reset, it would work across both Entra and AD.

Conclusion

The ‘User must change password at next logon’ flag will not sync with Entra unless the ForcePasswordChangeOnLogOn feature is enabled on the Entra ID tenant (see below).

Implement password hash synchronization with Microsoft Entra Connect Sync - Microsoft Entra ID

If the option ‘User must change password at next logon’ is selected in Active Directory, but that feature is not enabled in Entra ID, password changes will not be synced (see below).

azure-content/articles/active-directory/active-directory-aadconnectsync-implement-password-synchronization.md at master · toddkitta/azure-content

As a workaround, 'User must change password at next logon’ was deselected during password reset until changes were made to the Entra ID tenant.

How to change existing Azure VM from security type Trusted to Standard in order to enable nested virtualisation capabilities

Background

It is not possible to change the security type of an existing Azure VM back to standard. The only known way to achieve this is to setup a new standard VM and attaching the old OS disk to it.

This process appears to be suitable for domain joined production environments as well, however, it may depend on the complexities of your own environment.

Steps

1. Stop the target VM.


2. Export the current OS disk as VHD. Run the following script in cloud shell bash.

#Provide the subscription Id where managed disk is created

subscriptionId="your sub ID"

#Provide the name of your resource group where managed disk is created

resourceGroupName="rg name"

#Provide the managed disk name

diskName="current OS disk"

#Provide Shared Access Signature (SAS) expiry duration in seconds e.g. 3600.

#Know more about SAS here: https://docs.microsoft.com/en-us/azure/storage/storage-dotnet-shared-access-signature-part-1

sasExpiryDuration=3600

#Provide storage account name where you want to copy the underlying VHD file of the managed disk.

storageAccountName="sa name"

#Name of the storage container where the downloaded VHD will be stored

storageContainerName="blob container for storing disks"

#Provide the key of the storage account where you want to copy the VHD

storageAccountKey="sa key"

#Provide the name of the destination VHD file to which the VHD of the managed disk will be copied.

destinationVHDFileName="name of VHD"

az account set --subscription $subscriptionId

sas=$(az disk grant-access --resource-group $resourceGroupName --name $diskName --duration-in-seconds $sasExpiryDuration --query [accessSas] -o tsv)

az storage blob copy start --destination-blob $destinationVHDFileName --destination-container $storageContainerName --account-name $storageAccountName --account-key $storageAccountKey --source-uri $sas

3. Create managed disk from the exported VHD by running the following script in PowerShell Cloud Shell.

    Make sure HyperVGeneration and Zone match the new VM that will be created in the next step.

#Provide the subscription Id

$subscriptionId = 'your sub ID'

#Provide the name of your resource group

$resourceGroupName ='rg name'

#Provide the name of the Managed Disk

$diskName = 'name of VHD'

#Provide the size of the disks in GB. It should be greater than the VHD file size.

$diskSize = '127'

#Provide the URI of the VHD file that will be used to create Managed Disk.

# VHD file can be deleted as soon as Managed Disk is created.

# e.g. https://contosostorageaccount1.blob.core.windows.net/vhds/contoso-um-vm120170302230408.vhd

$vhdUri = 'https://<your sa>.blob.core.windows.net/<container name>/VHDfilename.vhd'

#Provide the resource Id of the storage account where VHD file is stored.

#e.g. /subscriptions/6472s1g8-h217-446b-b509-314e17e1efb0/resourceGroups/MDDemo/providers/Microsoft.Storage/storageAccounts/contosostorageaccount

$storageAccountId = '/subscriptions/<your sub id>/resourceGroups/<rg name>/providers/Microsoft.Storage/storageAccounts/<sa name>'

#Provide the storage type for the Managed Disk. PremiumLRS or StandardLRS.

$sku = 'StandardSSD_LRS'

#Provide the Azure location (e.g. westus) where Managed Disk will be located.

#The location should be same as the location of the storage account where VHD file is stored.

#Get all the Azure location using command below:

#Get-AzureRmLocation

$location = 'your location'

#Set the context to the subscription Id where Managed Disk will be created

Set-AzContext -Subscription $subscriptionId

#If you're creating an OS disk, add the following lines

#Acceptable values are either Windows or Linux

#$OSType = 'yourOSType'

#Acceptable values are either V1 or V2

#$HyperVGeneration = 'yourHyperVGen'

#Specify Zone

#Zone = 1

#If you're creating an OS disk, add -HyperVGeneration and -OSType parameters

$diskConfig = New-AzDiskConfig -SkuName $sku -Location $location -DiskSizeGB $diskSize -SourceUri $vhdUri -StorageAccountId $storageAccountId -Zone 2 -OsType Windows -HyperVGeneration "v2" -CreateOption Import

#Create Managed disk

New-AzDisk -DiskName $diskName -Disk $diskConfig -ResourceGroupName $resourceGroupName

4. Delete current VM while retaining OS Disk (just in case). 

    Make sure new VM uses dsv4 series of Azure VMs that support nested virtualisation.

    https://learn.microsoft.com/en-us/azure/virtual-machines/ddv4-ddsv4-series#ddsv4-series

5. Go to the new VM -> Disks, and choose Swap OS Disk. Choose the managed disk created in step 3.

    

6. Re-assign IP on the new NIC on Azure

7. Start new VM and connect using local administrator account

    After this step, OS disk provisioned along with new VM can be deleted.

8. Remove old and hidden NICs in Device Manager (Select Show hidden devices under View menu)

9. Edit Ethernet adapter by re-assigning IP along with default gateway and DNS server

You should now be able to install Hyper-V role on your Azure VM

Adding domain CA trusted certificates to workgroup servers for SCCM connection

Background

In certain enterprise scenarios, it is possible to have isolated servers, a.k.a workgroup servers, that are not joined to the main Active Directory domain due reasons such as security requirements (ex: internet facing servers).

Generally, Windows Server Clients should be part of the same domain as SCCM servers to be able to retrieve packages and security updates from SCCM Distribution Points (DPs). That is because SCCM needs to trust the client before it could push/pull packages and updates.

However, there is a workround to this problem that involves installing domain certificates in workgroup servers that are not part of the domain. To add on, this could be automated using PowerShell.

Steps

1. From a domain connected client server, copy the following types of certs over to the target workgroup client server:

    a. Root

    b. CertificateAuthority

    c. TrustedPublisher

2.  Run the following PowerShell script on the workgroup server with admin privileges. It uses .NET classes x509certificate2 and x509certificate. Make sure .NET 2.0 or above is installed on the target server.

#define script log file

$logfile = 'C:\Windows\Temp\Set-Certs.log'

#function to write into the log file

function writetolog([string] $txt) {

$Stamp = (Get-Date).toString("yyyy/MM/dd HH:mm:ss")

Add-content $logfile -value $Stamp':'$txt

}

#define list of certs

$certs = @(@("<corp root>.cer","Root"),@("<CA authority 1>.cer","CertificateAuthority"),@("<CA authority 2, if any>.cer","CertificateAuthority"),@("<Trusted publisher>.cer","TrustedPublisher"))

#define path to folder containing certs

$certFolder = "C:\Users\admin\Desktop\WorkgroupCerts"

#go through each cert

foreach ($certName in $certs) {

#start of the try/catch

try {

$cert = $certName[0]

Write-Host $cert

$storename = $certName[1]

write-host $storename

$certPath = "$certFolder\$cert"

Write-Host $certPath

#create a new certificate object using the x509certificate2 .NET class

$newCert = new-object system.security.cryptography.x509certificates.x509certificate2

#import the pfx file and use the x509KeyStorageFlag 'PersistKeySet' to set the certificate as persistent in our certificate store

$newCert.Import($certPath)

#get the thumbprint of the certificate going to be installed

$newCertThumbprint = $newCert.Thumbprint

#create a new object using the x509Store .NET class to manage the certificate catalog

$store = New-Object System.Security.Cryptography.X509Certificates.x509Store("\\localhost\$storename", "LocalMachine")

#open the certificate store with the 'ReadWrite' flag to be able to modify it

$store.Open([System.Security.Cryptography.X509Certificates.OpenFlags].ReadWrite)

#check if there is a certificate which match the new one's thumbprint present in the store

$certPresent = Get-ChildItem -Path CERT:\\LocalMachine -Recurse | Where-object {$_.Thumbprint -eq $newCertThumbprint}

#verify if the certificate is present. if not, install it

if(!$certPresent) {

Write-Host "Installing certificate on machine $machine"

$store.add($newCert)

}else{

Write-Host "Certificate already present on machine $machine"

}

#close the store

$store.Close()

}

catch {

throw

#update log

Write-Host "The installation has Failed on the machine $machine"

}

}

Checking EMC Networker configuration on new Windows Server builds

Background

You have been tasked to find out if EMC Networker is configured properly on your new Windows Server builds. This could be achieved by running an Ansible Playbook coupled with a PowerShell script.

Script

Playbook

#Define list of servers and their respective login details, this could be achieved differently depending on your infrastructure design

- name: capture servers list

hosts: localhost

vars:

- servers: "{{ myhost.split(',') }}"

gather_facts: false

tasks:

- import_tasks: automation-add-group.yml

- name: Check build status for windows

hosts: temp_group

gather_facts: false

tasks:

- name: Check for networker agent

win_shell: if(Get-WmiObject -Class Win32_Product | where Name -eq 'Networker') { write-host "Found" } else { write-host "Not Found" }

register: backupagent

failed_when: "'Not Found' in backupagent.stdout"

ignore_errors: true

- name: make sure networker service is running

win_service:

name: nsrexecd

state: started

register: srtbckup

ignore_errors: true

#If networker service is not running

- block:

- name: rename nsr folder

win_file:

path: C:\Program Files\EMC NetWorker\nsr\res\nsrladb

state: absent

- name: make sure network service is running

win_service:

name: nsrexecd

state: started

when: srtbckup is failed

ignore_errors: true

- pause:

seconds: 30

    #Calls the PowerShell script

- name: Check backup configuration

script: backservercheck.ps1

register: backupserver

failed_when: "'Backup client configured properly' not in backupserver.stdout"

ignore_errors: true

PowerShell

In this example, we wanted to check the backup configuration on 2 sites where each site had their own Backup servers. Hence, this script taps on the naming convention of each Windows Server to determine which Backup server it should connect to.

$hostname = $env:COMPUTERNAME

if ($hostname -like '*naming convention 1*') {

$backserverlist = @('backup server 1 - site 1','backup server 2 - site 1')

foreach ($bckupserver in $backserverlist) {

 #Define nsr command

        $command = "echo print type: nsr client; name: $hostname | nsradmin -s $bckupserver -p nsrd"

 #Run nsr command in cmd

        $output = cmd.exe /c $command

if($output -like '*scheduled backup: Enabled*') {

Write-Host "Backup client configured properly"

break

}

}

} elseif ($hostname -like '*naming convention 2*') {

$backserverlist = @('backup server 1 - site 2','backup server 2 - site 2')

foreach ($bckupserver in $backserverlist) {

$command = "echo print type: nsr client; name: $hostname | nsradmin -s $bckupserver -p nsrd"

$output = cmd.exe /c $command

if($output -like '*scheduled backup: Enabled*') {

Write-Host "Backup client configured properly"

break

}

}

}

Powershell script for checking Windows Server cluster health

Background

You have been tasked to write a script to check the health of your Windows Server clusters. Additionally, you are also required to email the cluster health report to your team.

Script

In order for the following script to work, FailoverClusters module must be imported to the sever that this script will run on. This can be achived by running the following command and it is a one-time task.

Import-Module FailoverClusters

param(

#parameter 1, cluster server list

[String]$clustersfile,

#parameter 2, email the report or not. Default is Yes.

[String]$emailtoteam="Yes"

)

#start of summary text

$summary = "############## Summary ##############`r`n"

#define log file

$logfile = "path to log file_$(get-date -f yyyyMMddhhmmss).txt"

$clusterlist = get-content $clustersfile

foreach ($cluster in $clusterlist) {

$errorflag = 0

Write-Host $cluster

$summary += $cluster

$cluster | Out-File $logfile -Append

if(Get-Cluster -Name $cluster -ErrorAction Continue) {

### cluster node

$clusternodes = Get-ClusterNode -Cluster $cluster

$clusternodes | ft | Out-File $logfile -Append

foreach ($clusternode in $clusternodes) {

if(($clusternode | select -ExpandProperty State) -ne "Up") {

$errorflag += 1 #if the status is not UP, then report error

}

}

### cluster resource

$clusterresources = Get-ClusterResource -Cluster $cluster

$clusterresources | ft | Out-File $logfile -Append

foreach ($clusterresource in $clusterresources) {

if(($clusterresource | select -ExpandProperty State) -ne "Online") {

$errorflag += 1 #if the status is not ONLINE, then report error

}

}

if($errorflag -ne 0) {

"ERROR: Check cluster`r`n" | Out-File $logfile -Append

Write-Host "Check cluster" -BackgroundColor Red -ForegroundColor White

Write-Host ""

$summary += " : Check cluster`r`n"

} else {

"OK`r`n" | Out-File $logfile -Append

Write-Host "OK" -BackgroundColor Green -ForegroundColor Black

Write-Host ""

$summary += " : OK`r`n"

}

"----------------------------------------------`r`n" | Out-File $logfile -Append

} else {

"`r`nERROR: Cannot connect to cluster, Please check manually`r`n" | Out-File $logfile -Append

"----------------------------------------------`r`n" | Out-File $logfile -Append

Write-Host "Cannot connect to cluster, Please check manually" -BackgroundColor Red -ForegroundColor White

Write-Host ""

$summary += " : Cannot connect to cluster, Please check manually`r`n"

Continue

}

}

$summary | Out-File $logfile -Append

function sendemail{

if($emailtoteam -eq "Yes") {

$smtpserver = "smtp server hostname"

$dateformat = Get-Date -Format M

$emailsubject = "Daily cluster check - $dateformat"

$to = "report recipient mailbox"

Send-MailMessage -To $to -From "sender address" -SmtpServer $smtpserver -Subject $emailsubject -Body $summary -Attachments $logfile -Cc "copied recipient mailbox if any"

}

}

sendemail

Install Splunk Universal Forwarder on Linux using Ansible

Before automating the installation, there are some things that needs to be taken care of. 

Universal Forwarder requires you to create a splunk administrator username and password during the installation. This can either be entered at prompt during the installation or specified in the installation command line. To get around this, we will be using a user-seed.conf that contains a preconfigured username and password as follows that can be called during the installation. More on that here https://docs.splunk.com/Documentation/Splunk/8.0.5/Security/Secureyouradminaccount

[user_info]
USERNAME = admin
PASSWORD = <your password>

Playbook is as follows.

- name: Install Universal Forwarder(UF) agent

hosts: ufservers

gather_facts: no

become: yes

tasks:

- name: Get previous versions of UF if installed

shell: rpm -qa | grep splunk

register: oldUFnamerpm

ignore_errors: yes

- debug:

var: oldUFnamerpm

- name: Remove previous version of UF

block:

- name: Remove old splunk from boot script

shell: ./splunk disable boot-start

args:

chdir: /opt/splunkforwarder/bin

- name: Stop old splunk

shell: ./splunk stop

args:

chdir: /opt/splunkforwarder/bin

- name: Uninstall previous version of UF

shell: rpm -e {{ oldUFnamerpm.stdout }}

when: oldUFnamerpm.stdout != ""

ignore_errors: yes

- name: Get old splunk process

shell: netstat -tulpn | grep -i splunkd | awk '{print $7}' | awk -F/ '{print $1}' | head -1

register: oldUFproc

- name: Kill old splunk process

shell: kill -9 {{ oldUFproc.stdout }}

when: oldUFproc.stdout != ""

- name: Remove old splunk dir

file:

path: /opt/splunkforwarder

state: absent

- name: Copy tgz from control server to remote node

copy:

src: /home/yinidu/Linux_UF/splunkforwarder-8.0.0-1357bef0a7f6-Linux-x86_64.tgz

dest: /home/yinidu

- name: Untar tgz

shell: tar xvfz splunkforwarder-8.0.0-1357bef0a7f6-Linux-x86_64.tgz

args:

chdir: /home/yinidu

- name: Move untar file to /opt

shell: mv /home/yinidu/splunkforwarder /opt/

- name: Copy user seed from control server to remote node

copy:

src: /home/yinidu/Linux_UF/user-seed.conf

dest: /opt/splunkforwarder/etc/system/local/user-seed.conf

- name: Install splunk

shell: ./splunk start --accept-license --answer-yes --no-prompt

args:

chdir: /opt/splunkforwarder/bin

- pause:

seconds: 10

- name: Stop splunk

shell: ./splunk stop

args:

chdir: /opt/splunkforwarder/bin

- name: Add splunk user

user:

name: splunk

state: present

- name: Enable boot-start

shell: ./splunk enable boot-start -user splunk

args:

chdir: /opt/splunkforwarder/bin

- name: Change folder permission

shell: chown -R splunk:splunk /opt/splunkforwarder

- name: Start splunk

service:

name: splunk

state: started

- name: Copy org_all_forwarder_outputs which contains heavy forwarder configuration

copy:

src: /home/yinidu/Linux_UF/org_all_forwarder_outputs

dest: /opt/splunkforwarder/etc/apps/

- name: Copy Splunk_TA_nix which contains input parameters

copy:

src: /home/yinidu/Linux_UF/Splunk_TA_nix

dest: /opt/splunkforwarder/etc/apps/

- name: Change folder permission

shell: chown -R splunk:splunk /opt/splunkforwarder

- name: Restart splunk

service:

name: splunk

state: restarted

- name: Set permission on /var/log

shell: setfacl --recursive -m u:splunk:r-x,d:u:splunk:r-x /var/log

- name: Check netstat to make sure UF is connected to the heavy forwarder

shell: netstat -an | grep 9997

register: netstatop

- debug:

var: netstatop.stdout