Ever play Whac-a-Mole?
You’ve got a hammer. Maybe five holes that moles could pop up from, give or take a few depending on your version. You put the moles back in their hole.
This is probably the point where you expect me to make a clever turn from an arcade game to the work of our managed IT services team. And you’d be wrong! Because it’s not a perfect analogy.
Supporting our clients might feel like Whac-a-Mole sometimes. But when we look at an incident or a simpler issue, the first step isn’t to swing our metaphorical hammers and call it a day. There’s a deeper level of analysis, establishing the context of where the issue is taking place, assessing whether this is a one-off issue or something affecting multiple users — you get where this is going.
Then once the issue is solved and our “hammer” has been swung, there’s more work to be done to ensure the “mole” doesn’t come back again. Documentation of the issue, system or configuration updates, a project to be scheduled, or maybe just training for an employee to watch for some warning signs of a common issue — nobody likes to see the same issue come back over and over again.
All that said: for the newsletter this week, we’re featuring the members of Lighthouse’s Managed IT Services team, who are here to share some of their most memorable issues resolved. Hope you enjoy learning a bit more about what we do!
JASON GUNNELLS - Director of IT Services
Early on a Friday morning, a client reported their internet was down and employees couldn't access company resources. Computers were on and functioning — but nobody could do any work.
As we began our investigation, I logged into one of the servers and immediately found what every IT professional hopes never to see: a ransom note.
The attackers had encrypted the client's virtual machines at the Linux hypervisor level. The individual workstations were untouched, but the servers powering the business were completely inaccessible.
Fortunately, another part of the story was already unfolding behind the scenes. Our 24/7 SentinelOne MDR solution and SOC had detected the malicious activity and automatically isolated the network — containing the threat before it could spread further.
Our investigation revealed the attackers had gained entry through an old VPN account that should have been disabled. We immediately shut down all external access points and reset every administrative account.
Because the client had immutable backups in place, recovery was clean. Our team worked through the weekend rebuilding and restoring every server. By Sunday, the environment was confirmed clean and normal operations resumed.
The outcome was exactly what every organization hopes for: operations back online Monday morning, no data lost, no ransom paid.
This incident is a reminder that preparation — managed detection, tested backups, disciplined access management — is what determines whether a cyberattack becomes a disruption or a disaster.
MARK RIZZO - End User Support Manager
One project that stands out to me recently was helping a client replace a Legacy listserv server with Google Groups.
The original server had become increasingly unreliable, and the client was spending too much time dealing with and trying to troubleshoot recurring issues with one of their most important communication tools.
They already used Microsoft 365 for email, so the challenge was to move the lists — and to help Google Groups and Microsoft 365 work together.
It took quite a bit of trial and error to resolve this, looking at DNS records and 365/Google domain aliases. There were a few points where everything looked correct on paper, but messages still weren’t flowing the way we expected, so it took some additional time to get everything communicating as envisioned.
What I enjoyed most about the project was learning more about how different mail systems can be configured to trust and route messages between each other. It’s one of those areas that most people never think about until a necessary project like this comes up.
In the end, the client now has a much easier way to independently manage one of their most heavily used services in a more modern capacity.
EVAN HOLME - Systems Engineer
A client at a previous job experienced an overnight ceiling collapse that caused significant water damage to their server infrastructure. Maintenance staff redirected some of the water using a tarp, but the primary Virtual Machine Host had already sustained water damage to its internal components. With this critical host offline, the client was unable to maintain normal business operations.
I transported the affected server to my office for a detailed assessment. After inspecting the damage, I carefully cleaned and dried the affected components. The server's motherboard qualified for a warranty replacement, but even with that replaced, the server still failed to boot. Further troubleshooting revealed that three hard drives, including the operating system drive, had also been damaged and required replacement before the system could successfully start.
I coordinated the motherboard warranty replacement, replaced the failed hard drives, rebuilt the operating system drive, and restored the Virtual Machine Host to a fully operational state.
The client's server infrastructure was successfully restored, and business operations resumed within two business days. The customer incurred no additional motherboard replacement costs due to the approved warranty claim and was extremely satisfied with the rapid recovery and overall outcome.
JAMISON POMROY - Service Desk Technician
One story that comes to mind started with a response to a site-wide internet outage. One of our clients reported no one could get online and at about 7 p.m. I went over to check on the issue. As this was a critical issue, I was ready for a long night of troubleshooting and knew I couldn't leave until the problem was solved.
Once I got there, an initial look at their network gear indicated everything was working as it should be. Their ISP confirmed there were no known outages in the area.
After some further investigation, I was able to find the culprit, and a power strip that was mounted to the wall had been flipped off. This particular power strip supported a couple of pieces of networking gear and happened to be located in a room that doubled as janitorial storage.
I noticed a broom was leaned on it right against the power switch and it became obvious what had happened: someone who was storing the broom set it down in a way that turned off this power strip!
Once I removed the broom and powered it on, internet came back online.
Sometimes the largest problems have the simplest solutions. A case has since been added to this power strip to prevent another late night accident in the future. This story always reminds me to start simple when troubleshooting, check the basics before overcomplicating a solution.
NIKOLAS GUTIERREZ - Service Desk Technician
During an internship, there was a user reporting that their docking station and monitors shut off. This was known as a normal request, so I went ahead and replaced the hardware and confirmed it was working.
After about a week goes by, the user reports that there docking station has shut off again, so I thought that the replacement I gave them just failed because it was used before and was fairly old. I then replaced the docking station with a brand new one.
Roughly another week goes by, and the user reaches out again stating that their monitor and docking stations have stopped working again. At this point I reached out to another member of our team to see where the issue was.
It turned out that the room that the user was in, had extreme power surges and has been causing the issue of frying the power to the dock and monitors. From here, we worked with maintenance/electrical to take care of the issue before we replaced the equipment again.
What I enjoyed about this issue was that it taught me to look at issues and tickets in a different way, and to always explore all avenues as to what is causing the issue. This showed me the importance of cross-communication between teams, and why we always look for the root cause of the issue.
The common thread in every one of these stories?
Our team learns to look under the surface. We’ll stay with the problem until it is truly solved. Technology is core to how every business operates today, and we’re here to make sure everything works for you.
This is what looks like. Your business doesn’t have time for Whac-a-Mole. You need structure, systems, and a team who knows how to deliver service with a human touch.
EXPLORE OUR MANAGED IT SERVICES and see what it’s like when technology works with your business.
—
Do you know a business that dealt with something like Jason saw in his story? Ransomware, business email compromise, there are many threats out there and it’s easy to feel overwhelmed.
We’re hoping to help more business owners feel less fear and more confidence when it comes to protecting their business.
Jason is presenting “Hacked: What Happens Next? Understanding the Cyber Response Playbook” alongside Thomas Barillari of the United States Secret Service at Walsh Insurance’s Benefits in Bloom event next Tuesday, June 9.
What We’re Hiring For This Week
Our Technology Talent Managers are always tuned in to the market, creating opportunities for technology professionals and innovative businesses to connect!
Here’s a few roles we’re recruiting for this week:
Senior Innovation Analyst: A very unique direct hire opportunity with a Buffalo-area manufacturing company. Onsite, $105,000-135,000/year.
Senior Functional Systems Analyst: 12+ month contract-to-hire opportunity, hybrid in Charlotte, NC, $55-60/hr.
Senior Salesforce Developer: 18-month contract opportunity, preferred hybrid in Buffalo but open to remote, $80-88/hr.