On the busiest shopping day of the year, our entire production system failed. Three hours later, we were still dead in the water with our machines non-functioning. To make a bad situation worse, the Black Friday - Cyber Monday weekend promotion we were running, was free laser engraved personalization.
The story began a few weeks prior. We had moved into a larger space as the company had grown significantly, and we needed to increase our production capabilities with the Christmas season swell. Our product is the ideal gift for friends, family, or employees, so we knew our customization was going to be popular. Thus I had ordered a second laser engraver.
It arrived a week before Thanksgiving, plenty of time for setup, but the carriage head was broken in shipping. We rushed a replacement carriage head, which arrived Wednesday, and I was able to get it installed in the machine. I wasn't able to do the laser alignment at that time as we were headed off to the holiday with family.
Friday morning, orders were coming fast, the primary machine was working away, and I was aligning the laser. To make a longer story a little shorter, aligning a laser is incredibly time consuming and frustrating. Every tiny adjustment on the first mirror ripples through the whole system.
Four hours into the process, it still wasn't perfect, but I wanted to run a live test to see how much the alignment was off. I unplugged the primary engraver, plugged in the second engraver, and the machine whirred up. It ran the first test seemingly fine, despite the test showing the laser was indeed far away from alignment. I plugged the primary engraver back into the computer.
Something happened at that point, I'm still not sure what, but the system failed. Perhaps the computer software began sending the wrong execution code to the primary engraver, or the hardware wouldn't respond to that specific code, or maybe something completely different. Rather than the rapid back and forth, finely tuned motion of a correct engraving, the carriage head would only move the full length of the gantry arm and return over and over.
Three weeks ago, the idea had been that we would have the new engraver system, which would double our production output capability, but wouldn't need to be busy yet. The free personalization had seemed like the best way to make use of the new machine while showcasing an important part of our company's value proposition, mainly our ability to customize binders with text and logos.
With the secondary engraver not aligned, and the primary engraver unable to get instructions this idea suddenly seemed much worse. I quickly replugged the new machine back into the computer to see if it would get proper instructions. It began to operate in the same way as the primary had.
In my favorite movie, Apollo 13, Gene Kranz has a famous line. "Failure is not an option." So I began to run through a troubleshooting checklist. Uninstall and reinstall the software. Failed. Reinitialize the laser driver. Failed. Run alternative engraver files. Failed. And so on for two hours. Every few minutes, my phone dinged with another order.
It was at that point I made the decision to scrap the old software. I was fairly confident the hardware was functioning, and the only real way to test the hardware functionality was with proper software. I found open source freeware that was compatible with the machines, and began to download and install the proper components. 20 minutes later, I was able to run the first test of the system. Success.
Now we had a new problem. While the system was able to communicate accurately with the machine, and the calibration adjustments made the engraving look the same as before, none of the old files were compatible with the new software.
When you are triaging a bleeding patient, you need to find the most serious wounds and treat them first. Usually our production system goes in order of rush orders first, and then when the order was placed. Desperate times call for a different measures. I was able to recreate the basic engravings, logo on back, name on spine, name on front, and we were able to get the primary engraver working again on the simpler orders.
There were still sporadic failures with the new system, which we have been able to isolate and fix as they arose. Now, I am working on converting the old laser files to our new software so we can deal with more complex engravings, and then I will return to working on the secondary laser alignment.
I am confident we will be able to get most of our weekend orders out before Wednesday this week. That would be slightly delayed from our usual same day shipping or next day shipping, but all things considered we are moving briskly back to our usual track.
Sometimes in life we feel trapped by crisis. Things go from bad to worse in the blink of an eye. We need to remember that "Failure is not an option" and that we need to "Work the problem" (Another great Gene Kranz line). Solve each problem as they arise, and only as they arise. Finally to all our customers from this weekend, please feel free to reach out to me, and I will be happy to let you know when your order will likely ship!
Happy Thanksgiving, and Have a Blessed Christmas Season.