RUN-WithIT and Drivewyze – a Case Study
Drivewyze needed to know they were ready
for more than 1000% growth.
"Everyone building a client-server application should be testing in this way, to understand the limitations of their system and to force the test-fail-fix cycle well before paying customers experience that pain."
Wayne Malkin, Drivewyze CTO
Drivewyze called on RUN-WithIT to help ensure the continued performance of their system during rapid growth. Large trucking fleets were coming online and Drivewyze CTO, Wayne Malkin, wanted to know they were ready.
Drivewyze’s smart mobility services are pioneering the future of highway safety and transportation management. Their main app approves truck drivers to bypass over 600 weigh stations in 36 states from their smartphone, tablet, or electronic logging devices (including Omnitracs, Rand McNally, PeopleNet, Zonar).
Their customer growth in 2015 was over 1100% and 2016’s even higher projections include the tiered rollout to over 360,000 Omnitracs equipped trucks. Waiting and responding to field failures or downtime does not fit with their value-driven, world class brand. They needed unparalleled insight into every aspect of the performance of their complex technology stack, before scale happened.
America's largest weigh station bypass service with plans to revolutionize the delivery of highway safety and transportation
IoT and SaaS - mobile and invehicle device app
Knowing they could continue to innovate and perform as large trucking fleets came online
Automate real devices - testing on the bench instead of driving
Send their realistic client code activity to their "ghosted" production simulation system at scale, continuously
Instrument their entire stack in simulation and push millions of live metrics to a Grafana dashboard
Inspire the team to break, fix, hypothesize, experiment and repeat until it is right
Improved performance, prevented field failures, reduced downtime, all during 1000% growth in server load
Reduced deployment risks
Uplifted and engaged dev team
Created an environment to achieve the 5x capacity limit increase aligned with business goals
"RUN-WithIT has helped us overcome 4-5 hurdles that would have taken our system down for weeks. Without them our customer confidence would be very low, but because of the emphasis on load/performance/automated testing we were able to see these issues in other environments outside of production and fix them before they affected clients."
Sean Feehan, Drivewyze Director of Product Development and Business Intelligence
The only way to reach the confidence level Drivewyze was looking for was through realism - real activity at real sites on real devices at scale. Drivewyze knew the production numbers and performance targets they needed to reach and wanted to maintain at least a 3 month capacity buffer. “There are some pretty decent tools out there to do protocol level simulation for load testing, but they have a steepish learning curve and mostly produce fragile scripts to drive load,” Malkin found. For this type of application, other approaches, such as testing in production, present unacceptable business risk.
RUN-WithIT specializes in wrapping actual client/device code to run completely real edge activity through entire, well-instrumented technology stacks in continuous simulation. Malkin states, “Testing of our systems for performance, load limits, and longevity was on the roadmap, but we had not made any real progress until we met RUN-WithIT.” With RUN-WithIT, Drivewyze could bring “production” into development. “We quickly arrived at a shared vision of where we wanted to go,” notes Malkin.
For the emulator, RUN-WithIT wrapped actual Drivewyze mobile and invehicle device code and stubbed in all of the inputs (everything from GPS emulation for truck position, Google Earth travel paths, VMC screen controls, and scripted truck speed and behaviour) to automate these devices, generate real activity, and observe performance from the actual client perspective.
While some of the functional testers eagerly put to use the emulated devices (on the bench testing versus actually driving the roads), the development team ghosted their production system in the cloud to run these simulated devices at scale with ramps, spikes or sustained numbers over time.
As a combination of teams observed and solutioned, RUN-WithIT evolved the fidelity of the simulation, including daily activity cycles. RUN-WithIT also tapped into their live production data, mirroring and setting load level multiples in real time. RUN-WithIT developed service virtualizations for various servers in the Drivewyze stack to make issues easier to enable tuning. The team accessed unparalleled insight through dashboard visualizations of their entire stack’s live metrics using Graphite and Grafana.
Some of the largest hurdles the simulation helped Drivewyze overcome were the interactions between different pieces of their system. “We ended up redesigning our layered micro service approach. It turns out that a single layer was much more stable then several layers based on micro services,” states Sean Feehan, Drivewyze Director of Product Development and Business Intelligence.
Improvements rolled smoothly into production along the way. Feehan notes, “RUN-WithIT has helped us overcome 4-5 hurdles that would have taken our system down for weeks. Without them our customer confidence would be very low, but because of the emphasis on load/performance/automated testing we were able to see these issues in other environments outside of production and fix them before they affected clients.” They could get ahead of issues and evolve their technologies, test different environments and deploy updates quickly with confidence at least 3 months ahead of production - minimizing innovation risk .
An exciting part of the Drivewyze project involved watching their team engage. The continuous simulation provided a focus for exploration and mastery. Component teams collaborated closely everyday, QA became involved, and feedback loops were intense and short. They truly ran with the opportunity to tackle performance, solutioning with fact-based data at a rapid pace.
Getting IT right makes a difference. “Our own journey has been very successful. In the last year our system load has increased more than 10-fold…We have experienced no interruptions to our production services due to capacity or concurrency limitations, although our load is now more than 5 times the capacity limit of our system from a year ago,” Malkin concludes.