INFORMATION NOW

Editor’s Note: Guest author Steve Ziegler is a senior architect at BetterCloud—Arun Nagarajan

FlashPanel, a leading enterprise-grade security and management application for Google Apps, makes use of many Google Apps APIs. Each API comes with its own set of rate-limiting quotas, network traffic behavior and response conditions. The challenge we face is to provide our customers with a stable, consistent experience despite these API behavior differences. Fortunately, Google has provided two simple, yet powerful tools to greatly increase our success rate and reliability.

Controlling Throttle Rates with Guava’s RateLimiter

Google Apps APIs operate with quota rates per second or per day. As such, the FlashPanel development team’s objective is to minimize total operation time of a parallelized set of API-intensive processes while avoiding running into Google’s API quota limits. To accomplish this, we can very easily set up a thread-safe RateLimiter for a specific rate per second for each rate restricted API:

final RateLimiter driveApiRateLimiter = RateLimiter.create(QUOTA_RATE_PER_SECOND);

The Guava RateLimiter allows us to attempt to acquire a permit based on the configured rate per second, block until available, and then take it when available, allowing execution.

To use it within the context of a method, we simply reference the rate limiter before making the API call:

public Result performDriveApiCall(driveApiRateLimiter, otherParams){
        driveApiRateLimiter.acquire(); // blocks according to rate
        // make API call...

}
This provides us with a very easy solution for rate limiting parallelized calls to the same APIs.

Reducing Impact of Network Traffic Issues Through Configured Backoffs

Occasionally, we experience additional network traffic issues despite enforcing rate limits. Google suggests that API clients use an exponential backoff strategy to handle this variety of error response. Typically this category of network issue resolves itself on its own after fractions of a second, but occasionally more time is needed. All we need to do to receive a successful API response is to simply retry the call some number of times, with the interval of time between retries growing exponentially.

Again, Google has made this easy through the Google HTTP Client for Java library’s ExponentialBackOff builder. This class allows us to configure the initial retry time interval, the exponential multiplier, the maximum total time to attempt to retry, and the maximum total time between retries. For example, we could configure a retry to span five minutes, growing with a factor of two, starting at a one second interval, and growing up to a maximum of one minute between retries. An API call with this configuration would retry with the following pattern in terms of seconds between retries:

1, 2, 4, 8, 16, 32, 60, 60, 60

If after this last retry, the API call still was not successful, the failed http response is returned. The code to configure such a strategy using the ExponentialBackOff.Builder reads as follows:

ExponentialBackOff backoff = new ExponentialBackOff.Builder()
    .setInitialIntervalMillis(ONE_SECOND)
    .setMultiplier(2.0)
    .setMaxIntervalMillis(ONE_MINUTE)
    .setMaxElapsedTimeMillis(FIVE_MINUTES)
    .build();

One potential “gotcha” that we’ve seen is if we accidentally slam a particular API with many simultaneous API calls. In this event, not only would each of these API calls fail, but they would also schedule their retry strategy to occur simultaneously. All subsequent retries would end up firing simultaneously, causing the API calls to continue to fail due to excess per second volumes. The ExponentialBackoff class accounts for this by including a randomization factor within our retry logic that allows us to have each simultaneous API call stagger at different intervals.

For example, using our previous backoff but now with randomization, one API call may retry with these intervals:

1.04, 1.9, 4.23, 7.8, etc.

While a second API call would retry with something like these intervals:

.98, 2.04, 4.1, 8.15, etc.

With this randomization, we avoid API call sequencing collision, mitigating our chances of encountering quota related errors. To simply add this type of randomization, we append to our builder:

builder.setRandomizationFactor(RANDOMIZATION_FACTOR);

Once we have our exponential backoff strategy configured, the Google HTTP Client for Java library allows us to instantiate and assign an HttpBackOffUnsuccessfulResponseHandler to an HttpRequest as part of the request’s initialization:

private HttpBackOffUnsuccessfulResponseHandler handler = new HttpBackOffUnsuccessfulResponseHandler(backoff);

public void initialize(HttpRequest request){ 
    request.setUnsuccessfulResponseHandler(handler);
}

Final Thoughts

By restricting our API calls using Google Guava’s easy-to-use RateLimiter, and by employing an exponential backoff strategy using the Google HTTP Client for Java library’s ExponentialBackOff, we are able to significantly reduce the amount of network traffic errors received by our Google Apps API calls, improving FlashPanel’s reliability and the overall user experience.

Steve is Senior Architect at BetterCloud, the makers of FlashPanel, the number one security and management application for Google Apps. Follow BetterCloud on Google+ at plus.google.com/+Bettercloud.

Posted by Greg Knoke, Googler

Schoolr: http://www.schoolr.com/
Schoolrs Blog: http://schoolr.wordpress.com/

This All-In-One search tool enables you to access several I-always-use sites (to find stuff) from one site, which currently include Reference.com family, Google, Wikipedia, Acrnonym Finder, Urban Dictionary, SparkNotes, NCSU, and unitconversion (Hopefully Answers.com and other juicy search sites could be added to the list soon). Since the search fields on Schoolr are linked directly to the sites they represent, it should not be difficult to add more (Though, information overload might kick-in).

Better yet, why not create a tool that enables us to add whatever tools we want to use for online researching (By the way the r in Schoolr stands for research), providing us the flexibility to access all our personal favorite juicy research tools from one site, saving us precious time. To add spice to it, we are able to see statistics on which research tools are mostly used, and who are using what, and the different combinations (on one site) used by anyone (according to the users preferences), including gurus like Joseph Hart, Jay Cross, and Jane Knight (who linked me to this tool). For information overload geeks like me, I would like to see all these research tools search at once, meaning that we only need to enter once (one entry field only, saving us more precious time) with checkboxes to the research tools (and the ability to add new checkboxes as we discover), and then the window will split into 2,4,6, 8 or more smaller windows (depending upon your selected checkboxes), enabling us to multi-task between all the juice (or summarize the findings using some Artificial Intelligence and/or Fuzzy logic). Enough!!!!

In addition, there are surely many other things we could do to spice up and make this great initiative more crispy and delicious.

Actually (I am no programmer, so please ignore the rest of this post if it does not make sense), the technology behind this tool is not spectacular, but the idea is creative and innovative (Respect!). As far as I understand (after watching the Channel News video. Yeah, I liked the Chicken Little promotion method, too!), it all came about when one whizz kid (founder) simply got fed up going to all these sites to do his online researching and solved his problem by creating Schoolr (all-in-one), and by doing so he has actually made our research life easier (if we use it, too).

So, if you are fed up, got a problem, or frustrated with something?

Please, remember that problems and frustrations are great opportunities to discover, invent, create, innovate,change the world, become famous, and make big bucks :)