At the beginning of December, I was planning to update the software on the Cisco wireless controller – model AIR-CT5520-K9. Everything was ready, but 3 days before the change, Twitter started buzzing about a bug that “activated” after December 4th, 2022.
Cancel or postpone the change? Many questions swirled in my head, especially since Cisco classified the problem as “Severity 1 Catastrophic”. Was there cause for concern? Let’s take a look.
What’s going on?
Some access points with the IOS system, after changing the software version on the wireless controller, are unable to download updates. The result: they hang in this process. This happens because the image signing certificates contained in the images of these access points were issued on December 4, 2012, for 10 years. As it’s easy to notice, they expired on December 4, 2022.
So, after December 4, 2022, when an access point downloads a code different from what it currently has installed, it will not verify the correctness of the image and will remain in a loop of downloading the image for an indefinite period.
Which devices are vulnerable to this issue?
This issue affects access points running IOS such as:
- Access points 802.11ac Wave 1 (IW3702/3700/2700/1700/1570)
- Access points 700/1530/1550/3600/2600/1600/3500/AP802 /AP803
Who should stay calm?
If you have access points from the 802.11ac Wave 2 series, Wi-Fi 6, Wi-Fi 6E in your infrastructure, you can take a breath and calmly read the rest of the entry. Unfortunately, I wasn’t that lucky.
What now?
The solution is simple. We travel in time. All you need to do is change the time on the wireless controller before December 4, but no earlier than November 1, 2022.
And now a bit of practice, my experiences.
As you might guess, I didn’t abandon my plans and on December 11th, I got to work. Here are some details about my software update.
- I updated the software from version 8.5.161.0 to 8.10.171.0 in order to connect Cisco Catalyst 9120AX access points.
- In addition to the access points from the Cisco Catalyst 9120AX series, the controller also manages the following access points – AIR-CAP2702I-E-K9.
After updating the software, these logs appeared to my eyes on the wireless controller.
After issuing the command:
show ap image all
We see that the access points cannot download the software.
When we log in to the access point, we will see more details that I described earlier.
Now it’s time to take action by disabling the NTP protocol (if configured) and manually setting the time between November 1st and December 4th, 2022, by issuing the following commands:
config time ntp delete <index>
config time manual 12/01/22 11:10:00
The magic begins now. The access points reconnect to the wireless controller after successfully downloading and verifying the software.
TIP: If the access point does not connect for an extended period (about 30 minutes), a restart may be required.
The only thing left for us to do is to reconfigure the NTP server or manually set the time and date to match the actual state.
Summary
The issue indeed has a significant impact on the infrastructure, especially on very large deployments. Being aware of this, one can easily safeguard against it, saving oneself a lot of stress. Additionally, Cisco is working on further software versions and patches that address this problem. I encourage you to follow the official thread located at the provided link.