Cimpress reduces load testing time by 80% with Speedscale

Company

Cimpress is a global eCommerce company that produces custom marketing materials for small businesses. Millions of customers worldwide use their design and printing services to create business cards, signage, promotional items, stationary, gifts, and more.

Challenge

As Principal Software Engineer, Jeff Kwan leads a team of API engineers that manages the software and product data behind Cimpress' expansive catalog. "The fidelity of this product data is very important because if it's wrong, then a lot of downstream activities are impacted—product design, fulfillment, prices. There's a lot that could go wrong if the software isn't working the right way," Jeff says.

Jeff's team had inherited legacy software from a different team and was looking to migrate off of it. He realized they needed a solution to test that the new software worked the same way as the old one.

Today, I know my endpoint for this ID produces this result. Tomorrow, when I make a change in the code, I need to know, 'Did I get the same result?' It sounds simple, but there's actually a lot of complexity behind answering this question confidently.

Jeff Kwan, Principal Software Engineer, Cimpress

Evaluating Solutions

Before the team found Speedscale, they had tried a couple open source tools, Diffy and GoReplay. While the tools initially met their needs, they soon felt that certain key elements were missing, like clear documentation, a thoughtful developer experience, and customer support.

As an engineering leader, it was hard for me to tell my team to use a tool that only partially solved our problem, leaving them with the challenge of figuring out how to make it work. What we needed was a more complete API testing solution long term.

Jeff Kwan, Principal Software Engineer, Cimpress

Jeff came across Speedscale through a Google search while looking for automated API testing tools, and recalls it was the pure focus on APIs and Kubernetes, as well as good documentation that initially attracted him to the offering.

Using Speedscale

Jeff recalls a very smooth implementation, guided mostly by documentation, a few calls with the Speedscale team, and engagement with other developers in Speedscale's Slack community. Jeff's team uses Speedscale today for 3 main use cases:

• Regression Testing
• Load Testing
• Observability

Regression testing

Despite having talented, senior engineers on the team, catching regression bugs was still difficult without a solution like Speedscale. Data setup and scripting was time-consuming, and the overall testing process was inconsistent, involving light spot-checking instead of a more comprehensive approach to testing. Now, they can record and apply real customer traffic to validate that things like platform or node version updates aren't changing the outputs.

Load testing

Prior to Speedscale, Jeff and his team had attempted to test application performance by scripting tests in K6. But this had 2 major drawbacks: 1) it was a manual process, and 2) scripting tests with the wrong product data could stress parts of the product catalog that weren't realistic.

Then, Jeff discovered Speedscale's K6 converter and was able to record real customer traffic and run it through K6, taking a 3-4 day testing process for one service down to 40 minutes. Today, Jeff uses Speedscale's own pod-based load generators to bypass K6 altogether, saving even more time.

Speedscale gave us clear insight on the upper load limit of our software. Whereas with K6, it was more binary, like a PASS/FAIL grade. Speedscale took it one step further to help us understand, based on transactions per second, exactly where our breakpoints were. We felt more confident in where our service could be.

Jeff Kwan, Principal Software Engineer, Cimpress

As Jeff and his team prepared for Black Friday, they reduced a typical 4-week testing process down to 3-4 days with Speedscale.

API observability

Since implementing Speedscale, Jeff and his team have realized additional benefits beyond their initial expectations. Speedscale's capabilities around API observability have helped them pinpoint and fix specific issues that they wouldn't have discovered otherwise.

"In one instance, I noticed that there was a small amount of traffic that had been causing some failures in our system. It was only happening 3% of the time, but it was still somehow critical to how it all worked. So, I went in and fixed it, but I wouldn't have found that issue had I not seen it in Speedscale."

Speedscale's offering is really, really good. The products, especially Traffic Viewer and Diff Viewer, and the overall developer-centric experience are all very polished.

Jeff Kwan, Principal Software Engineer, Cimpress

Results & Future Outlook

One year later, Jeff and his team report that they are incredibly pleased with their experience and the results they've achieved with Speedscale. "Speedscale's offering is really, really good. The products, especially Traffic Viewer and Diff Viewer, and the overall developer-centric experience are all very polished," he says.

He appreciates how open the Speedscale team is to receiving product feedback, and is excited to have his team trial some new features, like local desktop testing and auto-updating of web tokens.