Top convenience store chain triumphs over complete network overhaul – what they learn saves up to$930k per day in lost sales.
Last year alone, total global convenience store sales topped off at $199 billion, not including fuel. Besides all the soda pop, pizza, and aspirin, today’s chains provide a host of other products and services, including ATM, money order, and wire transfers. Behind this 24/7 commitment is a lot of infrastructure, not to mention IT teams working to ensure reliable network and application delivery to power these quick and easy retail options.
Recently a U.S. chain with its headquarters in the Midwest region embarked on a super-sized, multi-year network overhaul that threatened to offline their business. Besides an upgrade to 10 Gb in their data center and disaster recovery sites, they set about virtualizing nearly 90 percent of their servers and implemented two new business-critical applications. Plus, since they had more bandwidth with 10 Gb, they added VoIP as well.
During the 10 Gb upgrade, it was crucial to guarantee visibility into the new applications and evolving technologies, while continuing to provide their end users with the same level of quality service they needed to do their jobs. “Monitoring data is all the same whether it’s on a gigabit or 10 Gb network,” said the LAN/WAN administrator for the convenience store chain. “You need to see it to troubleshoot it.”
To improve data center efficiencies and reduce costs, they first virtualized large portions of their infrastructure. “We wanted to eliminate our physical boxes,” the administrator said. “In addition to obvious infrastructure cost savings, it’s easier to operate in a virtual environment. This is certainly the case with disaster recovery where virtualized infrastructure is much easier to restore.”
Despite the numerous benefits of implementing the virtualized network however, the loss of visibility became immediately apparent. The network team couldn’t get comprehensive views into the communications between all of their virtual servers. This impacted their ability to provide answers to application designers, leaving them dependent upon the server team for information and troubleshooting. Adding this unnecessary step slowed down problem resolution and meant that the team could no longer rely on their existing monitoring solutions.
SOLUTIONS AT THE SOURCE
When you’re running a business with sales of over $3.4 billion per year, every day of downtime has the potential to impact nearly a million dollars in sales. Already familiar with Network Instruments Observer for network analysis, the IT team purchased GigaStor because of its award-winning forensics capabilities and precision-troubleshooting technology.
The appliance allowed network engineers to rewind network activity to the exact date and time that performance problems occurred, revealing the source with clarity. “Once our team saw how effectively Observer monitored current network activity, we knew we could benefit from the retrospective analysis features of GigaStor,” said the administrator. “It quickly became the key asset for resolving any problems we had. It reduced the number of times the network was blamed, and shaved hours to days off the problem resolution process. We could now show other IT teams everything that occurred, and prove the network was functioning properly.”
To resolve the visibility issue, the network administrator created a SPAN off his vSwitch to mirror virtual communications and push these packets to GigaStor for in-depth analysis. The network team gained full visibility into all virtual networks – and regained network control because they no longer needed to rely on the server team to resolve network problems.
Next on the upgrade agenda was the deployment of new applications. IBM Maximo®, an internally-hosted app for asset and inventory management tracks all in-store, IT, and engineering inventory. With the company’s retail locations relying on Maximo to ensure that store shelves stay stocked, it’s one of the organization’s most essential applications. However, as it was first deployed across the enterprise, the program inexplicably shut down.
The network team immediately encountered issues with the software locking up and what appeared to be a loss of network connectivity. Without accurate inventory management, shelves go empty, sales drop, and so do share prices.
Working with IBM Maximo support, the IT team had to prove that the network functioned properly and pinpoint the actual application error.
“We spent a lot of time monitoring users and servers, while simultaneously setting up tests to verify the network was solid and to diagnose the actual problem,” the network administrator said. “While troubleshooting with GigaStor, we figured out it was a Java® error within the software, but there wasn’t an actual error code for the program to relay the error message back to the end user or the developers. Instead, the process would stop and shut down after it timed out. Once we located this, we turned over the GigaStor capture data to the Maximo team. They were then able to confirm and address the application issue.”
In addition to troubleshooting issues with internally hosted applications like Maximo, the IT team continuously monitors cloud applications. Even though these apps are managed outside of the company’s IT department, user complaints and performance problems are first-reported to the network administrator.
This was the case when the new cloud-hosted self-service payroll application began locking up and freezing. Human Resources had automated its payroll functions by shifting to an online HR self-service application. Designated as a business-critical site, the payroll web service was a new resource that the network team needed to monitor vigilantly, because nobody can afford to miss a paycheck.
“Using pings and synthetic transactions, we were unable to get back the requested data from the site,” the administrator explained. “With GigaStor we verified that while our data was going out, we weren’t seeing the expected data coming back. We shared this information with the provider, and they went back and detected an IP misconfiguration issue on their side.”
MAKING UC BETTER
Finally, it was time for the unified communications (UC) portion of the upgrade. The company had implemented a new hybrid Nortel Networks™ PBX system as a part of a new VoIP installation. But as soon as the system was up and running, it began dropping calls. This resulted in an unacceptable disruption in communications.
They used GigaStor to monitor all voice communications for call quality, consistency, and issue resolution in collaboration with the vendor’s support team. GigaStor quickly proved its worth once again.
“We operate a predominantly Cisco® network and run a pure Nortel VoIP system,” the administrator said. “VoIP problems are often blamed on the network, since Cisco doesn’t support Nortel Systems.”
To troubleshoot, the administrator placed a TAP in the VoIP environment where GigaStor collected data – proving that while the network stayed up, the primary PBX switch wasn’t responding during the issue timeframe. Once the network itself was excluded as the problem source, Nortel used the information provided by GigaStor to resolve the simple problem of a faulty switch.
Network Instruments’ GigaStor appliance played a pivotal role for the network team in the ongoing VoIP rollout support and fine-tuning. “We now rely on GigaStor to supply us with the information needed to get Nortel back on track,” said the administrator.
When it comes to super-sized upgrades, using the right network performance monitoring solutions is essential for effective troubleshooting, and it can save time and money. “Since proving the information using GigaStor, everything’s been running problem free. It’s great,” added the administrator.
As network traffic, volume, and demands increase, the convenience store company is looking to expand the amount of data its GigaStor appliances can retain. “As we deploy new applications and technologies, GigaStor will be central to ensuring the successful performance of our network and company,” said the administrator.
Thanks to Network Instruments for the article.