Cloud Spin, Part 1 and Part 2 introduced the Google Cloud Spin project, an exciting demo built for Google Cloud Platform Next, and how we built the mobile applications that orchestrated 19 Android phones to record simultaneous video. 
And now the last step is to retrieve the videos from each phone, find the frame corresponding to an audio cue in each video, and compile those images into a 180-degree animated GIF. This post explains the design decisions we made for the Cloud Spin back-end processing and how we built it.
The following figure shows a high level view of the back-end design:
1. A mobile app running on each phone uploads the raw video to a Google Cloud Storage bucket.

Video taken by one of the cameras

2. An extractor process running on a Google Compute Engine instance finds and extracts the single frame corresponding to the audio cue.

3. A stitcher process running on an App Engine Managed VM combines the individual frames into a video that pans across an 180-degree view of an instant in time, and then generates a corresponding animated GIF.

How we built the Cloud Spin back-end services
After we’d designed what the back end services would do, there were several challenges to solve as we built them. We had to figure how to:
  • Store large amounts of video, frame, and GIF data
  • Extract the frame in each video that corresponds to the audio cue
  • Merge frames into an animated GIF
  • Make the video processing run quickly
Storing video, frame, and GIF data
We decided the best place to store incoming raw videos, extracted frames, and the resulting animated GIFs was Google Cloud Storage. It’s easy to use, integrates well with mobile devices, provides strong consistency, and automatically scales to handle large amounts of traffic, should our demo become popular. 
We also configured the Cloud Storage buckets with Object Change Notifications that kicked off the back-end video processing when the Recording app uploaded new video from the phones.
Extracting a frame that corresponds to an audio cue
Finding the frame corresponding to the audio cue or beep poses challenges. Audio and video are recorded with different qualities and at different sample rates, so it takes some work to match them up.. We needed to find the frame that matched the noisiest section of the audio.To do so, we grouped the audio frames into frame intervals, each interval containing the audio that roughly corresponded to a single video frame. We computed the average noise of each interval by calculating the average of the squared amplitude of the samples. Once we identified the interval with the largest average noise, we extracted the corresponding video frame as a PNG file.
We wrote the extractor process in Python and used MoviePy, a module for video editing that uses the FFmpeg framework to handle video encoding and decoding.
Merging frames into an animated GIF
The process of generating an animated GIF from a set of video frames can be done with only four FFmpeg commands, run by a Bash script. First we generate a video by stitching all the frames together in order, extract a color palette, and then use it to generate a lower-resolution GIF to upload to Twitter.
Making the video processing run quickly
Processing the videos one-by-one on a single machine would take longer than we wanted. Next participants to have to wait to see their animated GIF. Cloud Spin takes 19 videos for each demo, one from each phone in the 180-degree arc.If extracting the synchronized frame from each video takes 5 seconds, and merging the frames takes 10 seconds, with serial processing the time between taking the demo shot and the final animated video would be (19 * 5s + 10s = 110s), almost two minutes!
We can make the process faster by parallelizing the frame extraction. If we use 19 virtual machines, one to process each video, the time between the demo shot and the animated GIF is only 15 seconds. To make this improvement work, we had to modify our design to handle synchronization of multiple machines.
Parallelizing the workload
We developed the extraction and stitching process as independent applications. This made it easy to parallelize the frame extraction. We can run 19 extractors and one stitcher, each  as a Docker container on Google Compute Engine.
But how do we make sure that each video is processed by one, and only one, extractor? Google Cloud Pub/Sub is a messaging system that solves this problem in a performant and scalable way. Using Cloud Pub/Sub, we can create a communication channel that is loosely coupled across subscribers.This means that the extractor and stitcher applications  interact through Cloud Pub/Sub, with no assumptions about the underlying implementation of either application. This makes future evolutions of the infrastructure easier to implement.
The preceding diagram  shows two Cloud Pub/Sub topics that act as processing queues for the extractor and the stitcher applications. Each time a mobile app uploads a new video, Cloud Pub/Sub publishes a message on the videos topic. The extractors subscribe to the videos topic. When a new message is published, the first extractor to pull it down has a lease on the message during which it processes  the video in order to extract the frame corresponding to the audio cue. If the processing completes successfully, the extractor  acknowledges the videos message to Cloud Pub\Sub, which causes Cloud Pub\Sub to publish a new message to the frames topic. If the extractor process fails, the lease on the videos message expires and Cloud Pub\Sub republishes the message, where it can be handled by another extractor.
When a message is published on the frames topic the stitcher pulls it down and waits until all of the frames of a single session are ready to be stitched together into an animated GIF. In order for the stitcher application to detect when it has all the frames, it needs a way to check the real-time status of all of the frames in a session.
Managing frame status with Firebase
Part 2 discussed how we managed orchestrating the phones to take simultaneous video using a Firebase database that provides real-time synchronization.
We also used Firebase to track the process of extracting the frame from each camera that corresponds to the audio cue. To do so, we added a status field to each extracted frame in the session as shown in the following screenshot.
When the Android phone takes the video it sets this status to RECORDING and then to UPLOADING when it uploads the video to Cloud Storage. The extractor process sets the status to READY when the frame matching the audio cue has been extracted. When all of the frames in a session are set to READY the stitcher process combines the extracted frames into an animated GIF and stores the GIF in Cloud Storage and its path on Firebase.
Having  the status stored in Firebase made it possible for us to create a dashboard that showed, in real time, each step of the processing of the video taken by the Android phones and the resulting animated  GIF.
We finished development of the Cloud Spin backend in time for the Google Cloud Platform Next events, and together with the mobile apps that captured the video, ran a successful demo. 
You can find more Cloud Spin animations on our Twitter feed: @googlecloudspin. We plan to release the complete demo code. Watch Cloudspin on GitHub for updates.
This is the final post in the three-part series on Google Cloud Spin. I hope you had as much fun discovering this demo as we did building it. Our goal was to demonstrate the possibilities of Google Cloud Platform in a fun way, and inspire you build something awesome!
- Posted by Francesc Campoy Flores, Google Cloud Platform

Google Cloud Launcher was created to provide customers an easy way to discover and deploy a variety of third-party solutions onto Google Cloud Platform. Today, we’re happy to announce new collaborations with Zend, NGINX, and Expert System as our first partners to contribute commercial solutions to our growing inventory.

You can now deploy commercially supported solutions with technical support included such as Zend Server, NGINX Plus, and Cogito API Core onto your choice of VM in just a few clicks. And to keep things simple, you will receive a unified bill from us for these services along with your other Google Cloud Platform usage costs at the end of the month.

Zend Server is an application server with a supported PHP runtime that can scale apps seamlessly across cloud resources, from the company most associated with PHP. Zend Server gives PHP developers and DevOps engineers amazing dev tools like Z-Ray, app deployment automation, performance monitoring, request analysis, and configuration management so apps run faster, scale better, and stay up longer.

"Zend Server on Google Cloud Platform will allow PHP developers to produce enterprise-grade applications in the very epicenter of disruption -- the cloud.," said Andi Gutmans, CEO and co-founder, Zend. "Our work with Google Cloud Platform gives developers the ammunition they need to create the quality, forward-looking applications businesses require to thrive."
NGINX Plus is a high performance, flexible, scalable, secure web accelerator and a web server. NGINX features include: Layer 4 / Layer 7 load balancing; application request routing; support for FastCGI, uwsgi, SCGI and memcached protocols; HTTP streaming media; SSL termination; bandwidth and request control; reverse proxy for HTTP, SMTP, IMAP and POP3; content caching; static content offload.

"We're excited to make NGINX Plus available on Google Cloud Platform as a fully supported solution," said Paul Oh, head of business development at NGINX, Inc. "Together, we're making it easier than ever before for organizations to deliver applications from the cloud with the speed and performance their customers have come to expect."
Through an understanding of the meaning of words in context, Cogito API Core identifies the entities present in documents, relations between the entities, concepts, and the semantically relevant information contained in a text.

“We are excited to offer a text analytics, semantic API through Google Cloud Platform, the most scalable and reliable infrastructure in the world”, said Marcello Pellacani, VP Strategic Partnerships, Expert System. “We’re looking forward to expanding the features and services available via Cloud Launcher, as APIs are a key element in developing and delivering world-class products.”

Moreover, Cloud Launcher can now be easily accessed directly in the Google Developers Console, creating a more unified experience that doesn’t require you to leave the management console to find and deploy a solution. This marks a next step toward our goal of surfacing the right solution at the right time for customers and making 3rd party solutions feel as native as possible.

Figure 1 - Cloud Launcher is a prominent menu item in Developers Console

Since our last update, we’ve also added Windows Active Directory, ASP.NET framework, and several popular open-source solutions that our users have requested, including Open edX and Redmine.

Expect more partners and solutions to be added as we expand our inventory of useful open-source and commercial options, and continue to give us suggestions about products and services you love using or would like to see us include in Cloud Launcher!

Posted by - Leslie Lee, Product Manager

Since our inception in 1998, Google has lived on the leading edge of technology as we built out an incredible set of data centers around the world to power our many services. The numbers simply boggle the mind — many exabytes of storage, petabit networking throughput, millions of vCPU capacity and so on.

These really big data centers are very efficient, have great networking capabilities, take advantage of natural elements such as the cooling effects of snow and seawater, and use renewable energy. They are what lets us give you incredible performance at amazing prices. Yes, we are proud of them … we even think they are pretty!

Today, we’re bringing Google Cloud Platform closer to even more people and businesses.
Google Cloud Network consists o
New Compute Regions
We’re opening our new us-east1 region in South Carolina for Google Compute Engine, Google Cloud Storage, and Google Cloud SQL.  Google App Engine will be coming soon, we’ll have more on this shortly! This will open up our services to customers that were waiting on a US East Coast presence. Besides lowering latency to those on the US East Coast, the addition of the South Carolina location gives customers across North America the capability to build multi-region disaster recovery plans for their applications running on Google Cloud Platform.

Regional Storage GA
In line with giving customer choices to locate their resources where it best suits, today four Google Cloud Storage regional buckets are moving into general availability so that you can put storage buckets closer to your Compute Engine instances. You can learn more about regional buckets here. The following regions are moving to GA and are now open for use for Standard, Durable Reduced Availability, and Nearline Storage:
    • Taiwan (asia-east1)
    • Belgium (europe-west1)
    • South Carolina (us-east1)
    • Iowa (us-central1)

To learn more about service locations, please see the Cloud Datacenter Locations overview page.  To learn how make the best use of our locations for your application needs, please see the Geography and Regions details page.

If you’re ready to run your applications in the US East Coast, please try out the new South Carolina location.

-Posted by Jay Judkowitz, Sr. Product Manager

We’ve answered and raised a lot of questions (keep ‘em coming) with our cloud pricing posts (Compute, Local SSDs, Data Warehouses, NoSQL) and have seen growing interest in our TCO tools (Compute Engine and Nearline so far…) but today we want to share some news about the TCO tool we used to make our pricing comparisons: it passed the accuracy test!

It’s one thing to have Google engineers review in painstaking detail the minutiae of our TCO comparison versus AWS to ensure that our comparison is fair and unbiased, founded on correct mathematics and built on reasonable assumptions about the behavior of these types of systems. But to help validate our approach and assess our tool with the most objectivity, we retained an independent third party, with a different background and context, to review this same body of work.

Nucleus Research a technology research firm that has published numerous ROI technology case studies and is registered with the National Association of State Boards of Accountancy (NASBA) took a comprehensive pass through our application, modelling and code base to ensure that our tool correctly computes the TCO of both Google Cloud Platform (GCP) and AWS systems. Nucleus’ depth of experience in TCO and ROI research provided significant benefit in this analysis, which evaluated not just our math, but the foundational assumptions that underpin our model.


Here’s how Ian Campbell, CEO at Nucleus, summed it up:

“Nucleus Research reviewed Google’s internally developed total cost of ownership (TCO) modeling tool for accuracy and completeness and found that the Google Cloud Platform TCO tool is an accurate calculation of the comparative costs of Google Cloud Platform versus Amazon Web Services.”

So our model is legit? “Yup, legit.” he said.

We know customers have lots of questions about this model, which in many cases shows a 40% or greater cost advantage for systems running on GCP versus AWS, and that’s a number that’s hard to ignore.  Try out our TCO tool with your own workload values, with your analysis of the market dynamics; we think you’ll find it a useful way to think about how the cloud can add value to your business.

- Posted by Miles Ward, Global Head of Solutions, Google Cloud Platform

From Google’s pioneering data centers to its network edge, we’re continuing to push the boundaries of our global network, enabling our customers to run workloads that reach a global audience from the very start. Google continues to lead with its global coverage of 70+ points of presence across 33 countries, allowing our enterprise customers to connect to Google privately for latency and reliability gains compared to going over the open Internet.
In April of this year we announced the expansion of our load balancing solutions to 12 additional locations globally. This expansion minimizes the distance users' traffic must travel over the public Internet, maximizing the distance traveled on Google's private network backbone instead critically important to achieving low latency in today’s media-rich web and mobile applications. Today we’re pleased to announce the extension of our load balancing locations to 20 additional cities, reaching a total of 32 distinct points globally. Our broad coverage ensures users get the lightning-fast, responsive experience they expect from Google Cloud Platform. Here is our full list of cities:
Berlin, Budapest, Chennai, Chicago, Dallas, Denver, Dublin, Hong Kong, Johannesburg, Kuala Lumpur, Lisbon, London,
Los Angeles, Madrid, Marseille, Miami, Milan, Mumbai, Munich, New York City, Osaka, Paris, São Paulo, San Francisco, Seattle, Singapore, Sofia, Stockholm, Sydney, Tokyo, Warsaw, Washington D.C.
In addition to announcing our new global load balancing locations, we’re also continuing to expand our Carrier Interconnect service provider ecosystem. Carrier Interconnect offers Cloud Platform customers a wide range of providers to satisfy their enterprise workloads wherever they may run around the world. Today we’re extending our partnership to include global communications operator and integrator Orange Business Services. For many Orange Business Services customers, which include leading multinational corporations, this collaboration with Google allows them to extend their on-premises operations via a private link to Google Cloud Platform. Existing customers using Business VPN can take advantage of a direct link to Google’s network to run mission critical workloads that blend on-premises and cloud seamlessly based on business demands. We are pleased to announce immediate availability in North America, Europe, and Asia.
To get started, contact Orange Business Services or get in touch with Google Cloud Platform directly. We’d love to learn about your most important business processes and how cloud might help you better serve your users.

- Posted by Morgan Dollard, Cloud Networking Product Management Lead

I’ll come right out and say it: Google Cloud Platform is a better cloud. Cloud Platform has clear, specific, technical differentiators, core advantages at the network, storage, compute and distributed software tiers which deliver incredible features and capabilities to our customers. This is important for two reasons:

  1. Cloud Platform offers features that are very valuable for customers, and very difficult for competitors to emulate.
  2. The underlying technologies, created and honed by Google over the last 15 years, enable us to offer our services at a much lower price point.

First, a few examples of these differentiated, valuable features:
1. Live Migration
Google Compute Engine’s instances can be moved to nearby hosts while active, even while under extreme load, complete with up to 1.5TB of Local SSD. As a result, since your instances don’t need to be rebooted for host software updates or other standard operational tasks, and even some classes of detectable hardware failure, average instance up times are superb. This improves the performance of our customer’s applications by increasing the consistency of performance for distributed systems and dramatically improving availability for applications that depend on an instance as a single point of failure (we all can’t have perfect software).

2. 0-1m+ scaling load balancers
Our load balancer service, a builtin feature of Compute Engine, Container Engine and App Engine, is itself a core component of the Google Frontend, an enormous, worldwide distributed system for delivering customers to infrastructure. This system powers the vast majority of Google applications, like Maps, Apps, Gmail, and Search. This system is designed to tolerate extreme spikes in traffic, easily scaling from no traffic to millions of requests per second, in seconds. This improves the performance of our customer’s applications; every user, no matter how many of them show up at once, will make it through to your stack.

3. 45 second instance boot times
What happens once they make it through the load balancer to your instances? Even with automation in place like our Autoscaler, as traffic mounts, you need to be able to scale up quickly. Google Compute Engine instances boot very consistently in the range of 40-50 seconds, roughly 1/5 of the time required by competing clouds.  This means you can grow your application’s hosting framework very rapidly in response to incoming traffic, just like Google does for it’s own applications.

4. 680,000 IOPS sustained Local SSD read rate
Each instance within Google Compute Engine, excepting micro and small shared-core instances, can mount up to 1.5TB of Local SSD capable of 680k of sustained read performance. This is radically faster than competing systems, which max out at less than half of that, at nearly four times the cost, all while tying SSD size/performance to specific instance sizes, meaning that in many cases you pay for compute and memory you don’t need.  This means caches, databases, NoSql systems, file systems and more operate at crazy fast speed to respond to user requests quickly, and to handle more users per instance.

5. 3 seconds to archive restore
Our “archival” data storage service, Google Cloud Storage Nearline, delivers data availability within 3 seconds, and provides high throughput for prompt restoration of data. In fact, it’s fast enough that many customers simply use it as their only storage tier. Competing systems take 4-5 hours to do the same task, and offer substantially lower throughput, not to mention a confusing, potentially extremely expensive restore fee structure.  Ours is simple: 1 penny per GB/month to store, 1 penny per GB to restore. New competitive offers like Standard-IA storage cost more, add weird minimum size restrictions, and fail to deliver a global service. Close, but no cigar!

How does Google Cloud Platform deliver on these features, and why are they so difficult for our competitors to match?  Google has built some of the most amazing technology, custom to our specific needs and design, which fills some of the most sophisticated data centers in the world.  We have powerful software-defined networks, faster storage components and better storage software, a more nimble hypervisor and some of the finest operations engineers and distributed software developers in the world.

You can’t fake these features, or simply buy the most expensive parts; you have to invest, heavily, in core componentry and skills development to deliver best in class capabilities.

The advantage doesn’t stop there; as it turns out, in most cases and for most products, Google Cloud Platform has a substantial cost advantage, typically on the order of 40% cheaper than competing clouds.

How can a cloud platform be both Better AND Cheaper?

Easy: it turns out that the same technology that you need to create incredible services for your customers, is the same as what we need to deliver incredible services to you. We consume these same resources to deliver not only the basic components of our offering, but importantly the managed services like App Engine, BigQuery, Cloud Bigtable, Cloud Dataflow, Cloud Pub/Sub and more.  How do we build a faster cheaper data warehouse like BigQuery? For starters; have container technology which boots quicker, SSD that serves data faster, and a load balancer designed to more efficiently distribute traffic. How do we build an efficient ETL engine like Cloud Dataflow? Well; have an long history in developing distributed processing software like MapReduce, Dremel, and Spanner, deliver it with the most powerful software defined network in the world and back it with rock solid storage.

Similarly, our internal operational tools, the monitoring, logging, metering, billing, auditing and forensics infrastructure that allows us to deliver a scaled cloud infrastructure to hundreds of thousands of customers and billions of users all operate dramatically more efficiently because of this foundation. Remember, it’s only a tiny fraction of the cloud which you have access to directly as products and services; the real measure of a cloud is its capacity for efficient scale, and Google has built efficiently at planet scale.

So, it works out that across the board, from instance prices to warehouses, from storage tools to automation engines, we’re able to deliver a really substantial price advantage to all of our customers, even while giving you the best tools in the world to deliver for yours.

But don’t rely on me, the English language makes it easy to say “it’s cheaper!” but math is what proves it.  We’ve built several tools to help you make your own analysis of the cost comparison between different clouds, public and private, as well as running a static infrastructure in colocation facilities or your own data centers.

Total Cost of Ownership and Google Cloud Platform

First of those tools is the GCP vs. AWS TCO Tool, a simple web UI for observing how some factors which many customers don’t anticipate in their modeling can have a big impact in real TCO over time.  Cost of capital, the consistent downwards trend in infrastructure costs over time, the likelihood of system design change, as well as the value of per-minute versus per-hour billing can often deliver a huge difference in what you’d expect a system to cost. Our model correctly captures these factors (as verified by Independent Analyst firm ESG) and provides an easy to understand comparison.

We’ve even included some pre-configured examples which capture some of the patterns we see play out every day on our infrastructure, which might look similar to the types of systems you’d design.  The first, which we call a “mature app”, is designed to reflect a production system, still under development, but already in the hands of customers. It has some development resources, systems which run dev and test workloads which demand bursty, short-lived infrastructure.  It also has a production system with a 6:1 day to night diurnal utilization swing (so, if your system needs to run 2 computers at night to serve traffic, in this example you’d run 12 during the day to handle peak load), which is typical of many applications, and has relatively conservative measures for the likelihood of system change, cost of capital, and expected downwards price trajectory. Given these settings, even when using the most efficient combination of AWS Reserved instances, yields a Google Cloud Platform price advantage of 40%.

Some customers are looking to build the next Snapchat, so we’ve included an even more flexible, nimble example of a smaller system called “startup app”.  Using this example, the advantages of per-minute billing, and ability to tolerate huge day/night swings drive a Google Cloud Platform price advantage of nearly 60%.

We talk to many customers in enterprise who argue that their systems don’t work like this; that they don’t develop software, they buy it. That they run static infrastructure systems to minimize operational overhead, and that they license software in a fixed way which demands that they avoid this kind of variability in implementation. Surely paying up front to achieve a fixed discount, like AWS Reserved Instances, must save customers following this usage pattern quite a bit over Cloud Platform?  We’ve captured this workload as “Static enterprise app” in our TCO tool, and if you take a look, it turns out it doesn’t matter: our lower basic rates, combined with automatic sustained usage discounts erase the price advantage of Reserved Instances.  Even in this example, Google Cloud Platform still enjoys a 36% price advantage.

These views are a bit of a summary, but we know folks are eager to dive into additional detail. Linked to the TCO tool is our Google Cloud Platform Pricing Calculator, a simple UI to help you accurately estimate the price you should expect to pay for running applications on our cloud.

If you already know what you’re running somewhere else, and have a good idea of what your fully loaded costs are, try entering those same infrastructure requirements into our Pricing Calculator, I suspect that you’ll come away quite surprised about what your monthly costs would be. (And if you don’t, we sure want to know about it - leave us a comment!)

So, how can you optimize?

Customers often ask me how they can best optimize their systems for cost, how they can save using Google Cloud Platform. In reality, simply implementing applications following basic best practices on Cloud Platform can deliver really substantial cost advantages over more static data center deployments, but often folks expect that there are special tricks to getting a good deal on cloud.

It turns out, most of the time, the real trick is “un-learning” the inefficient behaviors that data centers require to deliver on the needs of business.  These inefficient behaviors typically go along the lines of …
  • Planning on growth? Buy infrastructure months in advance and pre-test it.  
  • Planning on software changes? Radically overprovision hardware and memory “just in case”.
  • Planning on robust testing? Duplicate the production infrastructure for your test setup.
  • Planning, at all? Spend cycles in complex estimation rather than improving your product.

For most cloud customers, all of this planning simply gets eliminated; you pay only for what you use, only when you need it, so the work effort is re-oriented around ensuring that your systems accommodate scalability easily enough to nimbly adjust according to real demand. A few examples:
  • Put work into queues, and autoscale processors against queue depth = never have a machine on that’s not doing productive work!
  • If your software can’t run in an autoscaler group, it’s a bug!
  • For internal tool systems, consider a static “holding” page which calls a deployment manager script to start the dynamic hosting system for an app when users visit; if nobody is online, it turns off!  
  • Don’t over-provision instances for projected future demand, pick exactly the size you need now, and scale up or out as your demands grow.
  • Most DB systems are deeply IO bound; don’t get huge computers when what you really need is huge storage.

While the above might take a bit of tweaking for your software if it’s running in Google Compute Engine, it turns out that lots of Google Cloud Platform services work this way by default.  App Engine, BigQuery, Cloud Bigtable, Cloud Dataflow, CloudSQL Part-time instances, and more all do their best to minimize unnecessary resource consumption automatically.

We’re excited about the advantages we’re sharing with customers, the way that they’re building amazing products on top of us, and how those products are challenging and disrupting the status quo for the better.  What a great time to build!  We can’t wait to see what you build next on Google Cloud Platform.

- Posted by Miles Ward, Global Head of Solutions