RUN-WithIT and Drivewyze – a Case Study

Drivewyze needed to know they were ready
for more than 1000% growth.


"Everyone building a client-server application should be testing in this way, to understand the limitations of their system and to force the test-fail-fix cycle well before paying customers experience that pain."
Wayne Malkin, Drivewyze CTO
Drivewyze called on RUN-WithIT to help ensure the continued performance of their system during rapid growth. Large trucking fleets were coming online and Drivewyze CTO, Wayne Malkin, wanted to know they were ready.


Drivewyze’s smart mobility services are pioneering the future of highway safety and transportation management. Their main app approves truck drivers to bypass over 600 weigh stations in 36 states from their smartphone, tablet, or electronic logging devices (including Omnitracs, Rand McNally, PeopleNet, Zonar).

Their customer growth in 2015 was over 1100% and 2016’s even higher projections include the tiered rollout to over 360,000 Omnitracs equipped trucks. Waiting and responding to field failures or downtime does not fit with their value-driven, world class brand. They needed unparalleled insight into every aspect of the performance of their complex technology stack, before scale happened.

Company

America's largest weigh station bypass service with plans to revolutionize the delivery of highway safety and transportation

Product

IoT and SaaS - mobile and invehicle device app

Challenge

Knowing they could continue to innovate and perform as large trucking fleets came online

Solutions

Automate real devices - testing on the bench instead of driving

Send their realistic client code activity to their "ghosted" production simulation system at scale, continuously

Instrument their entire stack in simulation and push millions of live metrics to a Grafana dashboard

Inspire the team to break, fix, hypothesize, experiment and repeat until it is right

Successes

Improved performance, prevented field failures, reduced downtime, all during 1000% growth in server load

Reduced deployment risks

Uplifted and engaged dev team

Created an environment to achieve the 5x capacity limit increase aligned with business goals
"RUN-WithIT has helped us overcome 4-5 hurdles that would have taken our system down for weeks. Without them our customer confidence would be very low, but because of the emphasis on load/performance/automated testing we were able to see these issues in other environments outside of production and fix them before they affected clients."
Sean Feehan, Drivewyze Director of Product Development and Business Intelligence
 

The Challenge
 

    System performance for Drivewyze was always a big deal. Better customer experience helped establish their market leadership. For the truck driver, the value of using Drivewyze is very real every time they approach a weigh station – they anticipate their savings in time, money, stress, hassle and fuel when they receive a “bypass” message. A live safety-rating score enables bypasses up to 98% of the time, and this decision is communicated back to the driver’s chosen device, the weigh station and law enforcement, all before the driver arrives. The drivers expect the application to work perfectly, on time, and every time. If it doesn’t, the consequences are very real for both the trucker - and Drivewyze.

    The only way to reach the confidence level Drivewyze was looking for was through realism - real activity at real sites on real devices at scale. Drivewyze knew the production numbers and performance targets they needed to reach and wanted to maintain at least a 3 month capacity buffer. “There are some pretty decent tools out there to do protocol level simulation for load testing, but they have a steepish learning curve and mostly produce fragile scripts to drive load,” Malkin found. For this type of application, other approaches, such as testing in production, present unacceptable business risk.

    RUN-WithIT specializes in wrapping actual client/device code to run completely real edge activity through entire, well-instrumented technology stacks in continuous simulation. Malkin states, “Testing of our systems for performance, load limits, and longevity was on the roadmap, but we had not made any real progress until we met RUN-WithIT.” With RUN-WithIT, Drivewyze could bring “production” into development. “We quickly arrived at a shared vision of where we wanted to go,” notes Malkin.


    The Solution
     

      RUN-WithIT leveraged their framework and collaborated with Drivewyze’s team to create a realistic end-to-end continuous system simulation starting with a custom built emulator.

      Drivewyze bypass message on array of supported devices

      For the emulator, RUN-WithIT wrapped actual Drivewyze mobile and invehicle device code and stubbed in all of the inputs (everything from GPS emulation for truck position, Google Earth travel paths, VMC screen controls, and scripted truck speed and behaviour) to automate these devices, generate real activity, and observe performance from the actual client perspective.

      While teams experiment with design and configuration - realistic results are immediate and apparent
      While some of the functional testers eagerly put to use the emulated devices (on the bench testing versus actually driving the roads), the development team ghosted their production system in the cloud to run these simulated devices at scale with ramps, spikes or sustained numbers over time.

      As a combination of teams observed and solutioned, RUN-WithIT evolved the fidelity of the simulation, including daily activity cycles. RUN-WithIT also tapped into their live production data, mirroring and setting load level multiples in real time. RUN-WithIT developed service virtualizations for various servers in the Drivewyze stack to make issues easier to enable tuning. The team accessed unparalleled insight through dashboard visualizations of their entire stack’s live metrics using Graphite and Grafana.


      The Success
       

        With the RUN-WithIT simulation, the Drivewyze team could recreate any scenario of realistic “in the wild” activity. They could break, fix, hypothesize, experiment, and repeat. They had the visibility into their system to pursue issues - from SQL bottlenecks, query optimizations, Django timings, MongoDB reconfigurations, NodeIIS threads, to client, device and server code architecture - and know when they had it right.

        Demonstrated actual performance at capacity well ahead of production - the assurance they were looking for
        Some of the largest hurdles the simulation helped Drivewyze overcome were the interactions between different pieces of their system. “We ended up redesigning our layered micro service approach. It turns out that a single layer was much more stable then several layers based on micro services,” states Sean Feehan, Drivewyze Director of Product Development and Business Intelligence.

        Improvements rolled smoothly into production along the way. Feehan notes, “RUN-WithIT has helped us overcome 4-5 hurdles that would have taken our system down for weeks. Without them our customer confidence would be very low, but because of the emphasis on load/performance/automated testing we were able to see these issues in other environments outside of production and fix them before they affected clients.” They could get ahead of issues and evolve their technologies, test different environments and deploy updates quickly with confidence at least 3 months ahead of production - minimizing innovation risk .

        An exciting part of the Drivewyze project involved watching their team engage. The continuous simulation provided a focus for exploration and mastery. Component teams collaborated closely everyday, QA became involved, and feedback loops were intense and short. They truly ran with the opportunity to tackle performance, solutioning with fact-based data at a rapid pace.

        Getting IT right makes a difference. “Our own journey has been very successful. In the last year our system load has increased more than 10-fold…We have experienced no interruptions to our production services due to capacity or concurrency limitations, although our load is now more than 5 times the capacity limit of our system from a year ago,” Malkin concludes.





         

          RUN-WithIT helps teams bring real world performance into development.
          Find out how our tools and approach could help you get ahead.
          Book a quick ideas session.


          Contact Us