Migrating an App to High-Replication Datastore on Appengine

Google's Appengine now gives you the choice between the original master-slave datastore, and the new high-replication datastore. In a nutshell, high-replication provides more reliable storage (in terms of data security and latency) but has slower write times and costs considerably more.

The owners of almost all serious applications will choose the high-replication datastore. The extra cost is a small price to pay (literally) to escape the maintenance outages, datastore timeouts and high-latency requests that plague the master-slave datastore.

The following article explains how to switch a production application from master-slave to high-replication with the minimum impact on your users.

Lets assume the app is called myapp, and it is currently available at the URL www.mydomain.com through Google Apps for your Domain. If you use myapp.appspot.com instead, you become reliant on Google to create an alias from the old app to the new one, and so have less control over the timing and length of the cut-over when your app isn't available to users. I'm also assuming you have billing enabled, because transferring data is likely to exceed the free quotas (particularly CPU time).


Many of the steps below involve switching between different Google accounts, in Appengine and Google Apps. Ideally you would log in to each account/service in a different tab before you started, but you can only have one account open in each browser. This means you either have to log in and out several times, or you can try running different browsers (Firefox, Chrome, IE) or different virtual machines for each account you need. In particular, the Datastore Admin page in Appengine admin may have a blank content section - signing out and in again usually cures this.

The biggest challenge in the migration is transferring all the data from myapp to myapp-hr. You probably don't want to create new entities with the high-level datastore API on myapp-hr because they will have different keys, and if you use keys or ids to link entities you'll need to map between the myapp keys and the myapp-hr keys. There are several possible approaches to this:

1. Use the appcfg.py utility to download data from myapp, and then upload it again to myapp-hr. This preserves keys, but is very slow because it is a single thread and all data has to be transferred across the internet to your machine and then back again - a two-stage serial transfer across many networks.
2. Write your own transfer utility with the Remote API so you can select which data to move at which time. This might allow you to limit the cut-over data transfer to active data only, and you can transfer the rest at your leisure during the preparation stage. However, this entails some fairly complex development and testing - you need to preserve keys (or map between old and new) and ideally have several threads running.
3. Write your own transfer utility using the Urlfetch API on myapp calling a handler on myapp-hr. As well as limiting the cut-over data transfer to active data only, it happens across Google's local network and both read and write occur simultaneously. This will be the fastest option, but again requires substantial development and testing.
4. Use the little-known Datastore Admin tool. This has all the advantages - across Google's local network, multiple threads and simultaneous read/write - except it transfers all data (you can select by entity kind, but usually each kind will have active and non-active entities so this isn't useful).
5. [UPDATE] There is now a built-in migration tool in the admin console. I haven't used this, but this would now be my first choice.

My attempts to use appcfg.py to do the transfer ended when it took over an hour to transfer a fairly small dataset. I then looked at developing my own transfer process, before a kind soul on the Appengine group pointed me to the datastore admin documentation. This transferred 140,000 entities (110MB) in 20 minutes - and chewed through 5 hours of CPU time!

If you have massive amounts of data, you may choose option 3 to minimise the cut-over transfer time (and the cost of the associated CPU time), but otherwise I recommend the datastore admin tool - it's simple, fast and safe. [UPDATE]The built-in migration tool in the admin console would now be my first choice.

OK, let's get started! Each instruction is colour-coded for Appengine admin, Google Apps admin, Google Apps mail, and shell/terminal.

The migration process is broken into three stages:

• Preparation, which can be carried out at any time before the cut-over.
• Cut-over, when your app is read-only for users.
• Tidying up, done after the cut-over.

Preparation

First of all, we set up the new application, enable the datastore admin tool on the old and new apps, and test the data transfer process.

• Create a new app called myapp-hr. Edit the storage options and select High Replication.
• If your app sends emails, invite the sending address(es) to become developers for myapp-hr. Accept the invitation(s) in Google Apps for mydomain.com.
• Enable billing for myapp-hr, and allocate resources as required. A large allocation for CPU time is a good idea - the transfer may require more than you expect.
• Create myapp-hr source directory, and copy everything from myapp source directory.
• Edit the new myapp-hr app.yaml:
• Change application: from myapp to myapp-hr
  
• Add builtins section:
    builtins:
    - remote_api: on
    - datastore_admin: on
• Create file appengine_config.py containing (as a single line):
    remoteapi_CUSTOM_ENVIRONMENT_AUTHENTICATION
    ('HTTP_X_APPENGINE_INBOUND_APPID',['myapp'])
• Upload the new myapp-hr application:
•   appcfg.py upload myapp-hr
• Enable datastore admin in myapp. Edit the original myapp app.yaml to add builtins section:
    builtins:
    - remote_api: on
• Upload the myapp application:
    appcfg.py upload myapp
• Test the application is available by going to http://myapp-hr.appspot.com.
• Test the data transfer process to find out how much real and CPU time it takes.Switch to myapp (not myapp-hr) application.
• On the Datastore Admin page, select all entity kinds, and click "Copy to Other App".
• Change {TARGET_APPID} to myapp-hr in target URL, leave extra header as is.
• Note start time, go to the Task Queues page, refresh until default task queue is empty, and note end time.
• Switch to myapp-hr application, and go to the Dashboard page to see how much CPU time the data transfer required.
• Still in myapp-hr (double-check it's not myapp!), go to the Datastore Admin page, select all entity kinds, and click "Delete Entities" to clear all the data you just transferred.
• Change your local copy of myapp so it shows a maintenance message.

Cut-Over

Now you know how long the data transfer takes, you can warn your users when and for how long the system will be under maintenance. I usually announce a pessimistic figure of at least twice what I think it will take, to cover any problems that might arise. If data transfer takes 10 minutes, you should be able to complete the cut-over in 15 - but I told users maintenance would take an hour. If everything goes well, users will be happy the outage was shorter than announced. Otherwise you have more time to resolve the issue or revert to the old application.

Here we go!

• Upload the myapp application to show the maintenance message:
    appcfg.py upload myapp
  
• Switch to myapp application.
• Set myapp to read-only mode by clicking "Disable writes..." on the Application Settings page.
• On the Datastore Admin page, select all entity kinds, and click "Copy to Other App".
• Change {TARGET_APPID} to myapp-hr in target URL, leave extra header as is.
• Go to Task Queues page and wait until default task queue is empty.
• Log in as admin, click on myapp in the Google App Engine section, and then delete www.mydomain.com under Web address.
• Switch to myapp-hr application, go to the Application Settings page and click "Add Domain".
• Accept agreement, click "Activate this service", add new URL "www" and click "I've completed these steps".
• Go to www.mydomain.com and wait for the maintenance message to disappear, which means the domain is now pointing to myapp-hr. This should happen within a minute.
• Test the application. On one occasion I found issues that disappeared in a few minutes (before I could properly diagnose them), so don't panic if odd things happen at first.

Tidy Up

All that's left to do is remove the datastore admin capability from your application, clear all the data from the old application, and remove it so it doesn't count against your application limit.

• Delete file appengine_config.py from myapp-hr.
• Remove datastore_admin and remote_api from builtins section of myapp-hr app.yaml.
• Change the CPU time allocation to it's normal level on the myapp-hr Billing Settings page.
• Switch to myapp application, and disable billing on the Billing Settings page. Also cancel recurring charge authorization. This will take several days to complete.
• Go to the Application Settings page, and disable the application. You can then request permanent deletion, which will take 72 hours to occur.

Read our migration post-mortem.

Finally, please leave a comment below about how your migration went, or if you have any questions about the process. If you'd like dedicated support with your migration, please contact me (greg at vig dot co dot nz).

If this article has helped you, you can show your appreciation by telling any schools you are involved with about www.SchoolConferences.com. This simple online app (built on Appengine of course!) has already revolutionised parent-teacher conference evenings for over a thousand schools. See how easy it is to book conferences with the demonstration event code HIGH4 for high schools, and ELEM4 for elementary schools. Thanks!