Zero Downtime Upgrades – An upgrade approach to avoid late nights

Let’s see, it has been a while since the last time you heard from me, but this time I will spare you the excuse!

What is a Zero Downtime Upgrade?

This is an upgrade where no matter from which version of VOS (CUCM, UNITY, and UCCX) you are coming from can be done with very little or zero downtime

The Reasoning behind

Look I have always worked with companies that like their stuff done quickly, efficiently and with minimal downtime. I’m also very lazy, or very practical, so if I don’t have to spend all night doing an upgrade, I simply don’t do it. Now, like everything else, there are a lot of exceptions to the scenario I will be going over on this Post, so keep reading!

The technology is there, why not use it?

There are various technologies out there that help you have an easier life. In our example we are going to make use of the following technologies:

– VMWare (vSwitches)
– VLANs
– Static Routing
– Dynamic Routing
– GRE Tunnels
– Transport VRFs (Front Door VRFs)

The Scenario

We are looking to upgrade a CUCM, UNITY Connection and UCCX Servers from version 9.x to Version 11.x – This client relies heavily on UCCX to provide close to 24/7 support to its clients, there is only a small Maintenance Window allowed for this upgrade to happen.
The client also has 2 Sites where the UC/Collaboration Servers are located.

The Physical Environment

This image is just a quick representation of the Physical environment
NewImage

The logical Infrastructure

This is how the UC/Collaboration servers are distributed
NewImage

Specifically what we are doing?

Time to go to the whiteboard and draw this one out

– Create a Parallel Network where we can Install Version 9.x (Same IP, same Network, same Name and I think same NTP server)
– Load DRS Restore to the newly installed 9.x Version
– Upgrade server to 11.x
– Build Subscribers (My Cisco friends will debate on this one and ask why I did not restore the SUBs with the DRS restore, there are good reasons to do it, but for brevity, we will rebuild the SUB like if it was the first time, with no restore)
– Build the other VOS applications just like we did the CUCM server
– There should be a point where you have all the current infrastructure completely up and running like the Production but upgraded to the latest version.

What are things to really consider with this approach?

The fact that you are doing this upgrade in a “vacuum” at some point you have to declare a Changes Freeze on the Collaboration Environment you are working with. You have to build a Windows VM, I always carry my WIN7 ISO, or you can do it with a Windows Server, or even Linux, you pick your poison on this one 🙂 | Make sure this VM has 2 NICs, so 1 NIC can be used on the Production Network and the other NIC could be used with the Upgrade Network, or “BlackHole” as I like to call it **This network should not be routable by any of your production Network devices. Need to leverage a Cisco CSR1000v Router at both locations to create a parallel network with the same IP address as your Production environment, I like to use GRE Tunnels to reach both places

Let’s look at how to create the Parallel Network

The following helps represent how to create such environment, this is a high level, but I will dive deep into the process in a second post
NewImage

Now you understand the craziness, and how everything is laid from the beginning, now let’s jump to the configuration

The CLI, and the configuration

I created this quick image to represent how this is supposed to look as well as the code snippets
NewImage

Site A Configuration


!!!!!!!!!!!!!!!!! Make sure you create a Transport VRF
vrf definition transport
 address-family ipv4
 exit-address-family


!!!!!!!!!!!!!!!!! Create your Network reachable Interface **This is a usable IP address
inter GigabitEthernet 1
 vrf forwarding transport
 ip address 10.1.100.100 255.255.255.0


!!!!!!!!!!!!!!!!! This one comes out of the "BlackHole Network"
inter GigabitEthernet 2
 ip address 10.1.120.1 255.255.255.0
 ip address 10.1.110.10 255.255.255.0 secondary


!!!!!!!!!!!!!!!!! Create your Tunnel
interface tunnel 100
 ip address 1.1.1.1 255.255.255.252
 tunnel source GigabitEthernet 1
 tunnel destination 10.2.100.100
 tunnel vrf transport

!!!!!!!!!!!!!!!!! Create your EIGRP configuration 
!!!!!!!!!!!!!!!!! there will be no effect if you 
!!!!!!!!!!!!!!!!! add Gig1 to the Netowrk List, 
!!!!!!!!!!!!!!!!! so don't add it
router eigrp 111
 network 10.1.120.0 0.0.0.255
 network 11.11.11.0 0.0.0.255

!!!!!!!!!!!!!!!!! This one will make the routers 
!!!!!!!!!!!!!!!!! reachable, but only to the /32 
!!!!!!!!!!!!!!!!! IP address, no need to add the whole network
ip route vrf transport 10.2.100.100 255.255.255.255 10.1.100.1

 

Site B Configuration


!!!!!!!!!!!!!!!!! Make sure you create a Transport VRF
vrf definition transport
 address-family ipv4
 exit-address-family

!!!!!!!!!!!!!!!!! Create your Network reachable Interface 
!!!!!!!!!!!!!!!!! **This is a usable IP address
inter GigabitEthernet 1
 vrf forwarding transport
 ip address 10.2.100.100 255.255.255.0

!!!!!!!!!!!!!!!!! This one comes out of the 
!!!!!!!!!!!!!!!!! "BlackHole Network" you created
inter GigabitEthernet 2
 ip address 10.2.120.1 255.255.255.0

!!!!!!!!!!!!!!!!! Create your Tunnel
interface tunnel 100
 ip address 1.1.1.2 255.255.255.252
 tunnel source GigabitEthernet 1
 tunnel destination 10.1.100.100
 tunnel vrf transport

!!!!!!!!!!!!!!!!! Create your EIGRP configuration 
!!!!!!!!!!!!!!!!! there will be no effect if you 
!!!!!!!!!!!!!!!!! add Gig1 to the Netowrk List, 
!!!!!!!!!!!!!!!!! so don't add it
router eigrp 111
 network 10.1.120.0 0.0.0.255
 network 11.11.11.0 0.0.0.255

!!!!!!!!!!!!!!!!! This one will make the routers 
!!!!!!!!!!!!!!!!! reachable, but only to the /32 
!!!!!!!!!!!!!!!!! IP address, no need to add the whole network
ip route vrf transport 10.2.100.100 255.255.255.255 10.1.100.1

 

What to look forward to?

Hoping to get back to the blog a bit more and as possible in the next few days, so stay alert for new posts

About the Author:

Andres Sarmiento, CCIE # 53520 (Collaboration)
With more than 13 years of experience, Andres is specialized in the Unified Communications and Collaboration technologies. Consulted for several companies in South Florida, also Financial Institutions on behalf of Cisco Systems. Andres has been involved in high-profile implementations including Cisco technologies; such as Data Center, UC & Collaboration, Contact Center Express, Routing & Switching, Security and Hosted IPT Service provider infrastructures.

You can follow Andres using Twitter, LinkedIn or Facebook

ccna collab

4 thoughts on “Zero Downtime Upgrades – An upgrade approach to avoid late nights

  1. Quick question. We were looking at a similar approach for a hardware refresh, not upgrade. Essentially the arguments are the same. We were debating the same DRS/rebuild for subs. We have multi-SAN certs on 11.x. Thoughts on lowest touch approach, specifically considering certificates?

    Liked by 1 person

    1. Jonathan, Thank you for the message – I was stunned a few weeks ago when I was playing with UCC certs myself and on how easy it was to share the same certificate between servers. However, I think it may be slightly different using VOS applications, which is one thing that I wish to test and document in a separate post, since I have not played with it lately.
      The conversation with the SUBs, is an interesting one, TFTP files and Music On Hold files will not carry over to the new installation if you re-build them – and in my mind this seems not too terrifying for a small base config. – Sorry for the late reply, I hope this helps 🙂

      Like

  2. I do upgrades all the time. This seems overly complicated. I use a Cisco 1811 and I can simulate the entire network (or pretty close) including DNS and NTP with 1 router. I am not sure the needs for all of this. Maybe if your remote and cannot touch the servers. This document also talks about a perfect senecio. What if hostnames or IPs have to change and the ITL/CTL files? What about secure voice? None of this talks about potential CAD to Finesse. What about voicemails during the change feeeze. Are they unified messaging or does COBRAS need to used? What if there are 10,000 phones and how to handle phone firmware upgrades.

    I could go on and on. Not a bad document but not great either. Ensure you read the entire upgrade documents before attempting the upgrade.

    Liked by 1 person

    1. Hi Anthony,
      Changing hostnames and IP address along with ITL and CTL, is very possible, one of my posts this year contains a scenario that deals with this, and it also uses the same no late night’s approach
      Secure Voice is an interesting one that I have not looked at, and deals also with certificates, which I think will also be possible to deal with this approach ** I’m glad certs can be combined 🙂
      CAD to Finesse, in this case, I always make sure I can train my Supervisors and Agents on Finesse before I even pull the trigger, using DCLOUD.COM or any other environment that I can build for them
      When you talk about the Voicemails after the change freeze started, there are many ways you can go about this one, hopefully, at this point, all your Voicemails are going to the clients Outlook/Email mailbox. if not, COBRAS is your friend… I personally think COBRAS is way too slow for this, but there is a new tool called Connection Message Shuttle –> http://www.ciscounitytools.com/Applications/CxN/MessageShuttle/MessageShuttle.html

      Last but not least… the best way to deal with any upgrade is to upgrade the phones firmware of the phones while the phones are still registered to the old version.

      I love the comment, and this is the intention for it all. In my mind I’d always like to put in a balance, complicated on one side, or an all-nighter on the other side… to me the difference is huge and will wake up a very interesting debate, in addition to the complicated piece, my engineering answer will always be “it depends on how comfortable I feel with the technology”.

      Thank you very much for the message, I would like to hear more in case you find flaws in the process

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s