INFORMATION NOW

Editor’s Note: Guest author Steve Ziegler is a senior architect at BetterCloud—Arun Nagarajan

FlashPanel, a leading enterprise-grade security and management application for Google Apps, makes use of many Google Apps APIs. Each API comes with its own set of rate-limiting quotas, network traffic behavior and response conditions. The challenge we face is to provide our customers with a stable, consistent experience despite these API behavior differences. Fortunately, Google has provided two simple, yet powerful tools to greatly increase our success rate and reliability.

Controlling Throttle Rates with Guava’s RateLimiter

Google Apps APIs operate with quota rates per second or per day. As such, the FlashPanel development team’s objective is to minimize total operation time of a parallelized set of API-intensive processes while avoiding running into Google’s API quota limits. To accomplish this, we can very easily set up a thread-safe RateLimiter for a specific rate per second for each rate restricted API:

final RateLimiter driveApiRateLimiter = RateLimiter.create(QUOTA_RATE_PER_SECOND);

The Guava RateLimiter allows us to attempt to acquire a permit based on the configured rate per second, block until available, and then take it when available, allowing execution.

To use it within the context of a method, we simply reference the rate limiter before making the API call:

public Result performDriveApiCall(driveApiRateLimiter, otherParams){
        driveApiRateLimiter.acquire(); // blocks according to rate
        // make API call...

}
This provides us with a very easy solution for rate limiting parallelized calls to the same APIs.

Reducing Impact of Network Traffic Issues Through Configured Backoffs

Occasionally, we experience additional network traffic issues despite enforcing rate limits. Google suggests that API clients use an exponential backoff strategy to handle this variety of error response. Typically this category of network issue resolves itself on its own after fractions of a second, but occasionally more time is needed. All we need to do to receive a successful API response is to simply retry the call some number of times, with the interval of time between retries growing exponentially.

Again, Google has made this easy through the Google HTTP Client for Java library’s ExponentialBackOff builder. This class allows us to configure the initial retry time interval, the exponential multiplier, the maximum total time to attempt to retry, and the maximum total time between retries. For example, we could configure a retry to span five minutes, growing with a factor of two, starting at a one second interval, and growing up to a maximum of one minute between retries. An API call with this configuration would retry with the following pattern in terms of seconds between retries:

1, 2, 4, 8, 16, 32, 60, 60, 60

If after this last retry, the API call still was not successful, the failed http response is returned. The code to configure such a strategy using the ExponentialBackOff.Builder reads as follows:

ExponentialBackOff backoff = new ExponentialBackOff.Builder()
    .setInitialIntervalMillis(ONE_SECOND)
    .setMultiplier(2.0)
    .setMaxIntervalMillis(ONE_MINUTE)
    .setMaxElapsedTimeMillis(FIVE_MINUTES)
    .build();

One potential “gotcha” that we’ve seen is if we accidentally slam a particular API with many simultaneous API calls. In this event, not only would each of these API calls fail, but they would also schedule their retry strategy to occur simultaneously. All subsequent retries would end up firing simultaneously, causing the API calls to continue to fail due to excess per second volumes. The ExponentialBackoff class accounts for this by including a randomization factor within our retry logic that allows us to have each simultaneous API call stagger at different intervals.

For example, using our previous backoff but now with randomization, one API call may retry with these intervals:

1.04, 1.9, 4.23, 7.8, etc.

While a second API call would retry with something like these intervals:

.98, 2.04, 4.1, 8.15, etc.

With this randomization, we avoid API call sequencing collision, mitigating our chances of encountering quota related errors. To simply add this type of randomization, we append to our builder:

builder.setRandomizationFactor(RANDOMIZATION_FACTOR);

Once we have our exponential backoff strategy configured, the Google HTTP Client for Java library allows us to instantiate and assign an HttpBackOffUnsuccessfulResponseHandler to an HttpRequest as part of the request’s initialization:

private HttpBackOffUnsuccessfulResponseHandler handler = new HttpBackOffUnsuccessfulResponseHandler(backoff);

public void initialize(HttpRequest request){ 
    request.setUnsuccessfulResponseHandler(handler);
}

Final Thoughts

By restricting our API calls using Google Guava’s easy-to-use RateLimiter, and by employing an exponential backoff strategy using the Google HTTP Client for Java library’s ExponentialBackOff, we are able to significantly reduce the amount of network traffic errors received by our Google Apps API calls, improving FlashPanel’s reliability and the overall user experience.

Steve is Senior Architect at BetterCloud, the makers of FlashPanel, the number one security and management application for Google Apps. Follow BetterCloud on Google+ at plus.google.com/+Bettercloud.

Posted by Greg Knoke, Googler

Review:
The Prince of Persia: Sands of Time trilogy has always been known for its tight controls and satisfying environmental puzzles set in beautiful levels. The series control scheme has served as a model for countless other games that also feature an acrobatic protagonist like the prince. The same great interface remains in the third game of the series, The Two Thrones. Like the other two games, youll find yourself wall running, jumping, flipping, and vaulting your way up fiendishly designed environmental puzzles, as well as fighting vicious enemies with the improved combat system introduced in the last game, Warrior Within. The prince has a few new tricks up his sleeve as well, but longtime fans of the series should still feel right at home.
The story of the Sands of Time trilogy began with the princes army attacking and conquering the capital of the Indian empire. In the midst of the battle, however, an evil vizier caused the release of the sands of time, wreaking havoc over the city and the surrounding countryside. The prince and an Indian princess named Farah battled together to defeat the vizier, bottled up the sands, and restored the world to a normal state. The story continued in Warrior Within, where an embittered prince set sail for the Island of Time, seeking sanctuary from an unstoppable time creature called the Dahaka, who pursued him endlessly because of his part in unleashing the sands of time. There, the prince battled not only the Dahaka, but also Kaileena, the Empress of Time. Fans of the first game felt the series lost a lot of its soul in Warrior Within, as the suddenly cynical prince was much more arrogant and less likable.

The Two Thrones begins with the prince returning to Babylon from the Island of Time with the mortal Kaileena as his new lover (this part of the plot may be confusing to those who didnt see the alternate ending of Warrior Within). As they pull in to the harbor at Babylon, the duo finds the city under siege. Their ship is wrecked by the invaders, and Kaileena is captured by the enemies. You quickly find that the vizier is back and responsible for the uprising. He murders Kaileena and unleashes the sands of time upon Babylon, and the prince must battle to regain his kingdom and avenge Kaileenas death.

But its not just sand creatures and other bad guys from the viziers army that the prince must fight. With the sands of time infecting his soul, the prince must also battle the whims of his darker, more arrogant side. Throughout the games long, 12- to 15-hour campaign, the prince will morph back and forth between his normal self and the dark prince. Hell also have internal dialogues between his split personalities, with the original actor from Sands of Time voicing the normal prince, and a new voice actor representing the sneering, more sarcastic dark prince. Here, the game almost becomes self-referential in addressing the popular criticism levied against the arrogant prince from Warrior Within. The internal strife in the schizophrenic princes mind forms a compelling part of the storyline in The Two Thrones, especially because both the writing and voice acting are pretty good.

System requirements:
Windows XP/2000
AMD Athlon or Pentium III 1GHz Processor
256MB RAM
16X CD-ROM Drive
1.5GB Hard Disk Space
DirectX 9.0c Compatible Video Card
DirectX 9.0c Compatible Dolby Digital Live Sound Card

DirectX 9.0c

Screen Shots: Click to view large