This week at Google I/O we are launching support for MongoLab‘s fifth cloud provider – Google Cloud Platform. You can now use MongoLab to provision and manage MongoDB deployments on Google Compute Engine (GCE)!

So far we are very impressed with the capabilities of the GCE infrastructure.  In particular:

• The network is fast. I mean really fast. Some of the bandwidth and latency benchmark scores are astounding. Since I/O is king for databases this will be great for connecting your GCE-hosted application to a MongoDB instance hosted by MongoLab.

• GCE has a global private network connecting GCE regions across the world. This will be great for global multi-region clusters. We don’t support this quite yet, but when we do GCE will provide a high-speed private backbone upon which to build a great solution.

• The API is clean, and VMs spin-up fast. This is key for automation, and we like to automate.

For now we are in an early access beta, supporting only our free Sandbox database plans in GCE’s us-central1 region. We will be launching support for the rest of our product line in subsequent releases.

We will have a Developer Sandbox (a.k.a “booth”) at the conference on Friday May 17th. If you are at Google I/O and into MongoDB come visit us!

Palace Hotel, c. 1930

You know, I attend a fair number of MongoDB events, and frankly I keep expecting them to get stale. But after being at MongoSF this past Friday, I’m happy to say it hasn’t happened yet. The growth and vigor of the Mongo ecosystem was everywhere apparent, and it has never been more encouraging. Our sincere thanks go out to the 10gen team for putting together another fabulous and informative event.

If you were there and managed to stop by MongoLab’s table in the exhibit hall of the super-elegant Palace Hotel, then thanks! It was nice to meet you and/or see you again! Hope you got as much out of the day as we did. If you didn’t — or if you’d just like my personal take on the whole thing — well, please read on.

Ecosystem predicts viability

Setting aside any of the relative merits of MongoDB as a database for just a moment, I have to say my top takeaway continues to be amazement at the size and enthusiasm of the community around MongoDB.

[I]f an organism or aggregate of organisms sets to work with a focus on its own survival and thinks that that is the way to select its adaptive moves, its “progress” ends up with a destroyed environment. If the organism ends up destroying its environment, it has in fact destroyed itself. … The unit of survival is a flexible organism-in-its-environment.” [1]

History is littered with the scarcely recognizable fossils of good ideas, clever inventions, and even superior products that might have flourished save for one thing: adoption. The modern proving ground for technological species looks less and less like the traditional “marketplace” with pockets of asymmetric information and discrete “deals.”  Today, the landscape has evolved to include open-source transparency, synergies of technologies and ideas, and a globally interconnected (and often, informed) fabric of opinion. A vigorous ongoing conversation (and overlap!) among diversified populations of users and developers is now the surest predictor, I believe, of long-term survival.

So, more than the database technology (which is impressive) or the well-capitalized company devoted to developing it (which is formidable), it is the people and the strength of this community that inspire my confidence that MongoDB will continue to thrive, improving and growing in popularity as a viable or even preëminent database for an ever increasing number of applications.

MongoDB: the Next Generation

Eliot Horowitz, 10gen CTO & Co-founder, kicked things off on a strong note, clearly articulating his focus for the immediate future of MongoDB. In my opinion, these are exactly the right priorities for taking the platform to the next level:

• Maturity
• Innovation
• Operations

If you peer into its internals today, you’ll see the evolutionary legacy of MongoDB: steadily improving and expanding functionality, accreted around a core of pragmatic and sometimes downright scrappy engineering — just what you might expect from a small, clever team with a product rapidly establishing itself in the marketplace. But many of the expedients that accelerate a large piece of software in the short term can eventually bog down development and become obstacles to its further progress. You want a larger team to be able to add and maintain a growing number of features, without commensurate increases in code complexity. At some point, once experience has shown where the grain boundaries lie, there comes a time to refactor (not reinvent!) the core, teasing out clear and minimal abstraction contracts that the new implementations of existing and future features can target.

This engineering story arc is not lost on Eliot. Cleaner factoring, he explains, will be a a key enabler to efficiently deliver capabilities that MongoDB has needed for a long time, to make it a more “mature,” fully-featured general purpose database. It will also form the groundwork for innovating and building on the strengths of MongoDB as a data substrate for modern applications. Specific examples Eliot mentioned included:

• non-constant query constraints — e.g., find all documents where the values of fields “a” and “b” are equal.
• inline aggregation operations — e.g., update each document to set its “total” field to the sum of the “dollarAmt” field of each element of its “lineItems” array.
• index intersections — e.g., optimize a query like {a: 3, b: 6} by dynamically combining an index on “a” with an index on “b” to yield performance comparable to what today would require an explicit compound index comprising both fields.

So that’s the broad story around Maturity and Innovation — right on. What about the third item: Operations? This of course refers to the realities of keeping a database running and available behind a production system of any kind. Happily, there is another three-item list here:

• Monitoring
• Backups
• Management

Eliot spoke to 10gen’s efforts on each of these facets: MMS, which became available some 18 months ago; the remote backup service, which is in Limited Release now; and a suite of management tools to be announced later this year.

Of course, the topic of production-class operations is near to our hearts: seamlessly handling these three facets for our customers is what MongoLab is all about!

You got your lagerstätten in my Burgess Shale!

Opabinia, c. 505,000,000 BC

Max Schireson, the 10gen CEO who claims to have been born the same year as the relational database, followed up with a pointedly evolutionary perspective on database technologies. He compared today’s landscape to the early part of the Cambrian Explosion, in which biodiversity increased by orders of magnitude in a small fraction of the total history of life on earth up to that point. Of course, the unstated implication was that hitherto more “established” databases (Oracle, MySQL) were the long-dominant single-celled organisms in this analogy, whereas MongoDB would be perhaps more like a sighted predator of some kind.

Schireson quoted some consumption figures from the top of the food chain (e.g., 3 of the top 10 global investment bank use MongoDB) and noted some recent shifts in environmental pressures (e.g., developer-driven decision making).  He also cited an amusing factoid: prior to this year’s report, the last Gartner Research update on databse technology came out in 2003. That’s right: a full ten years ago. Something new must be going on. (Can you guess what?)

In short, Schireson made it sound like a pretty exciting time to be in databases, with MongoDB figuring prominently on the changing landscape.

Okay, now back to your niche…

After this inspiring keynote, of course, there followed a full day of stimulating talks and sessions at all levels of the mongo-guru ladder — oceans of fresh, insightful, useful stuff.

My personal favorite was probably the session led by Charity Majors, who is responsible for the MongoDB servers at the heart of Parse.com. If you were lucky enough to catch her outstanding talk on the care and feeding of a grown-up mongo deployment, you’ll know that there’s a whole host of operational issues that you’d just rather not worry about — or at the very least, you’d very much like an experienced hand at the helm when you do. Why do I say her talk was outstanding? Because that stuff is our bread and butter. It’s what we do all day every day here at MongoLab: hook you up with the database of tomorrow, so you can use more of your energy to dominate YOUR product’s ecological niche today, and still get a good night’s sleep (assuming your species isn’t nocturnal).

There’s never been a richer ecosystem, or a better time to be a database consumer. And there are more reasons than ever today for your consumption preferences to be of the MongoDB phyla. Yummy! Why not try one right now?

T. Dampier, 2013-05-11

Notes

[1] Source: Gregory Bateson, “Form, Substance and Difference”, 19th Annual Alfred Korzybski Memorial Lecture, 9 Januiary 1970, Oceanic Institute, Hawaii. From the book Ecology and Consciousness, edited by Richard Grossinger, North Atlantic Books, 1978. p. 32.

A lot of our users upgrade from single-node databases to replica set clusters without fully understanding how their driver, and therefore their application, will react to failover. In fact, we get so many questions about best practices with MongoDB replica sets that we thought it could be cool to host a replica set that anyone can connect to using their MongoDB driver of choice.

Today we invite you to check out flip-flop, a MongoDB Replica Set demonstration and experimentation service.  The flip-flop service consists of:

• A live replica set that fails-over (i.e. “flips” and “flops”) every 60 seconds.  This cluster is always running and available to all at the following address:

  mongodb://testdbuser:testdbpass@flip.mongolab.com:53117,flop.mongolab.com:54117/testdb

• A real-time visualization of the cluster flippin’ and floppin’ with streaming database server logs

• A set of example client scripts (currently just in Python) that simulate client interactions with the cluster that you can use as a starting point for your own experimentation

The flip-flop service is also great for those of you working on third-party drivers. Gustavo Niemeyer, author of mgo, a MongoDB driver for the Go language, told us flip-flop helped him find and quickly fix a small bug in the driver: “This is brilliant. I actually managed to find an edge case coding a trivial example against it due to the timing of the server re-election.” Pretty cool!

How to get started with flip-flop:

1. Open http://mongolab.org/flip-flop in one window
2. Download and run our sample Python script (requires pymongo 2.4+) in a second window

   sudo pip install pymongo
   curl https://gist.github.com/mongolab-org/5347810/raw/766f8266db377e462f90e5c1389be6211fbd2db6/watch-flip-flop.py > watch-flip-flop.py
   python watch-flip-flop.py
   

3. Watch this script gracefully recover from failover as the cluster flips and flops every 60 seconds.

We hope you all find this service informative and useful. As you play around with flip-flop, please do let us know what you think!

Sincerely,

Your friends at MongoLab

Today… Remote Connections got even better! You can now use exactly the same backup tools on remote databases that our users that host with MongoLab know, love, and trust.

MongoLab’s backup system makes it extremely easy to schedule and manage backups. You can use the system to perform one-time backups or create recurring Backup Plans with custom schedules and retention policies. Backups can be stored in MongoLab’s own secure cloud containers or in a container at the cloud storage provider of your choice (e.g. Amazon S3).

How it works

Step 1. Create a Remote Connection

After you have signed up for a MongoLab account, create a Remote Connection to your database by providing us with the MongoDB connection URI for your database, server, or Replica Set cluster.

 Step 2. Kick-off a one-time backup or create a recurring Backup Plan

Step 3. Sit back and relax

Why implement and manage backups yourself when you can ask our trusty robots to do it for you?

Not only do we take the headache out of taking backups, but we make sure backups happen as they should be. Our comprehensive backup auditing system is continuously monitoring all Backup Plans to ensure that every backup is happening when it is supposed to, and without error.  You will be alerted when any backups do not occur as planned.

Free during beta

For now, this feature is in beta. During this beta period, we are offering this functionality for free with a database data size limit of 5 GB. When we come out of beta, Remote Connection backups will support larger database sizes and be priced on a per-run basis.

If you run into any trouble and/or have any feedback, please don’t hesitate to send us an email at support@mongolab.com.

Thank you in advance for trying out backup tools for Remote Connections; we look forward to hearing what you think!

The growing popularity of MongoDB means more and more people are thinking about data in ways divergent from traditional relational models. For this reason alone, it’s exciting to experiment with new ways of modelling data. However, with additional flexibility comes the need to properly analyze the performance impact of data model decisions.

Embedding arrays in documents is a great example of this. MongoDB’s versatile array operators ($push/$pull, $addToSet, $elemMatch, etc.) offer the ability to manage data sets within documents. However, one must be careful. Data models that call for very large arrays, or arrays with high rates of modification, can often lead to performance problems.

That’s because at high levels of scale, large arrays require relatively high CPU-overhead that leads to longer insert/update/query times than desired.

Let’s use car dealerships as an example and discuss why an array of cars in a dealer document isn’t necessarily the ideal data model.

Here’s our dealer document:

{ "_id" : 1234,
  “dealershipName”: “Eric’s Mongo Cars”,
  “cars”: [
           {“year”: 2013,
            “make”: “10gen”,
            “model”: “MongoCar”,
            “vin”: 3928056,
            “mechanicNotes”: “Runs great!”},
           {“year”: 1985,
            “make”: “DeLorean”,
            “model”: “DMC-12”,
            “vin”: 8056309,
            “mechanicNotes”: “Great Scott!”}
  ]
}

Now for some concerns with this model.

Querying Cars

One of the advantages of MongoDB is its rich query language, which supports accessing documents by the contents of an array. If we want to locate cars by make and model using a query like {“make”: MAKE, “model”: MODEL}, we need a specific feature of MongoDB’s query language, $elemMatch. This operator, while optimizable by indexes, still needs to traverse the entire “cars” array of every eligible dealer document in order to execute the query. If we query documents with “cars” arrays that contain thousands of car entries, we are basically doing mini collection-scans. As a result, we’ll notice high CPU utilization and slow query execution.

What about more complex computation? The Aggregation framework — with the $unwind operator — could support many of the queries we’d like to perform, but indexes may not be used to their full effectiveness.

Finally, dealer documents can get very large with this data model. Although projections and atomic update operators cut down the size of the documents transmitted over the wire, there may still be scenarios where the system is hauling more baggage around than it should.

Adding and updating Cars

Adding and modifying car entries can require a scan of much or all of each array being updated, resulting in slow operations. For example, $addToSet, is a way of adding new elements to arrays that requires the database to scan through every array item to make sure the new element does not already exist. $pull can be similarly inefficient.

Furthermore, sometimes modifications to the “cars” array can grow a dealer document in size such that it must be moved in memory. These moves can be very expensive, particularly when the collection is heavily indexed as each index bucket that points to the document being relocated must also be updated to point to its new memory location.

Can we fix it?

Sometimes a data model that entails very large arrays can be reformulated into a data model that is much more efficient. In our case we could alternatively model dealers and cars like this:

Dealers

{ "_id": 3423, “dealershipName”: “Eric’s Mongo Cars” }

Cars

{ "_id" : 1234,
  “dealership”: 3423,
  “year”: 2013,
  “make”: “10gen”,
  “model”: “Mongos”,
  “vin”: 3928056,
  “mechanicNotes”: “Runs great!”},
{ "_id" : 54321,
  “dealership”: 3423,
  “year”: 1985,
  “make”: “DeLorean”,
  “model”: “DMC-12”,
  “vin”: 8056309,
  “mechanicNotes”: “Great Scott!"}

In this data model we avoid the excessively large arrays and, with the right indexes, perform efficient queries on dealerships with even the largest of inventories. By keeping cars in their own collection, Eric’s Mongo Cars is ready to move inventory with crazy low prices, without fear that our volume is going to bring down the system for Eddie’s Junker Shack down the road. We love those guys.

Conclusion

Storing information in document arrays is an exciting capability available in MongoDB, but we want to avoid it under the following conditions:

• The arrays can get very large, even if only in some documents
• Individual elements must be regularly queried and computed on
• Elements are added and removed often

None of the pitfalls described above are deal-breakers in and of themselves. It’s just that when summed together, the total overhead can become noticeable. So, be wary. In these high-volume cases, it is appropriate for us to use collections, not arrays, to store data. When we store such data using collections,

• Regular computation is performed using simple, efficient methods
• Adding and removing elements are simple insert/remove operations
• Each element is accessible using simpler queries that can be effectively indexed to scale well

There’s a lot more detail under the hood, but if you’d like to discuss it, we’ll have to get there in the comments section below.

Thanks for reading, and good luck out there!

Sincerely,
Eric@MongoLab

In today’s post we’ll be presenting a quick-start guide on how to connect OpenShift, the free RedHat auto-scaling Platform-as-a-Service (PaaS), with our popular MongoDB Database-as-a-Service (DBaaS), MongoLab.

For demonstration purposes, we’ll be using a Node.js application that we’ve written (available for download here). All it takes to connect your OpenShift application is five easy steps!

Step 1. Create an OpenShift account and application

Create an account at http://openshift.redhat.com and install the rhc command-line tool on your development machine. For more info about rhc, see https://openshift.redhat.com/community/developers/rhc-client-tools-install.

Once rhc is installed, create a nodejs-0.6 application using the path to this repository as the --from-code argument and by replacing with your desired application name:

% rhc app create  nodejs-0.6
       --from-code https://github.com/mongolab/mongolab-openshift-quickstart
% cd

rhc initializes your application using this repository as a baseline.

Step 2. Create a MongoLab account and database

1. Sign up for an account at http://www.mongolab.com. After you’ve successfully created your MongoLab account, you’ll see “Databases” header and a “Create new” button.
2. Click on the “Create new” button to create a database. Be sure to specify a database user name and password. These credentials are not the same as your MongoLab account credentials.
3. Click on your database. The database landing page provides a mongodb URI connection string of the form:
   mongodb://<db user>:<db password>@<host>:<port>/<db name>

4. Copy this value somewhere helpful and replace placeholders with your database user credentials.

Step 3. Commit and deploy the app

When you’ve created your app, the rhc command line client automatically initialized a git repo with a remote link to OpenShift. The code is also already deployed to your app gear.

If you don’t make any changes, you can skip this step. However, if you make any modifications (now or later), perform the following to update the code on the gear:

% git add .
% git commit -m "my first commmit" -a
% git push

Step 4. Configure environment variables on the app gear

The example code uses mongodb://localhost:27017/test when the MONGOLAB_URI environment variable is not available. This is sufficient for testing with a MongoDB database running on your local machine, but not for production.

Note: We find that configuring this value outside of the code (and not storing it in a repository) allows for maximum security and flexibility. However, there are repository-driven alternatives for configuring this environment variable that may meet your requirements. See how to create and use environment variables on the server for more information.

To configure your environment variable without placing credentials in a repository:

1. Login to http://openshift.redhat.com
2. Click My Apps.
3. Click the > next to your app name to reach your application page.
4. Click Want to log in to your application?
5. Copy the provided ssh shell command to a shell window and press enter to ssh to your app gear.
6. Open your gear’s .bash_profile in your text editor of choice. It is located at ~/app-root/data/.bash_profile.
7. Add the line export MONGOLAB_URI=<db uri> where db uri is the mongodb URI you obtained in Step 2, with your database user credentials added.
8. After editing the file, run source ~/app-root/data/.bash_profile
9. Use echo $MONGOLAB_URI to confirm success. The value you added should be displayed at the console.
10. Restart your app by running ctl_all stop then ctl_all start.

Step 5. View the app

Visit your deployed app at:

    http://<app name>-<app namespace>.rhcloud.com

And there you have it – just five steps to get your OpenShift deployed application connected to your MongoLab database!

If you ever have any questions, don’t hesitate to get in touch with us at support@mongolab.com.

Arduino is an open-source electronics platform that can acknowledge and interact with its environment through a variety of sensor types.  It’s great for hardware prototyping and one-off projects.

I just got an Arduino Board from our friends at SendGrid, who also gave me a little tutorial in the art of Arduino hacking. Inspired by the tutorial and armed with this new  board, I bought a passive infared (PIR) motion sensor from my local Radio Shack. Now I was ready to play; in particular, I wanted to be able to collect that continuous stream of hardware sensor data into a MongoDB database for logging, trend analysis, system event correlation, etc.

To this end, I created the demo project “mongodb-motion”, which I’ve made public on Github. In the “mongodb-motion” Github repo, you will find an Arudino project that writes motion sensor data to a cloud MongoDB database at MongoLab and sends alerts via email based on certain criteria. I built this demo using Node.js and the MongoLab REST API.

Below, I’ll go through exactly what hardware you need to make your own “mongodb-motion” project a success, and how the code actually works.

What You Need

The hardware used in this demo includes: an Arduino UNO R3 and a Parallax PIR motion sensor.

How the Code Works

You can use a variety of motion sensors with the Arduino. In this particular experiment, I used a PIR motion sensor. The PIR motion sensor behaves like a switch, with ‘down’ events emitted on motion detection and ‘up’ events a few seconds after motion ceases to be detected.

On the receiving side, I used JohnnyFive, an appropriately named Node.js package that accepts sensor events and sends messages to the Arduino board.

With the two ends set, I’ll move on to the project’s configuration file. In this demo, I’ve included a configuration file, config-sample.js, where credentials for the MongoLab REST API and for the email SMTP server can be added. In my case, I used the SendGrid SMTP service.

The configuration file also has two callbacks that determine when an email is emitted, one for each type of event – “detect” and “ceased”. I’ve used this feature to automatically send an email alert if an event timestamp is between 7:00pm and 8:00am, ostensibly when my office should be motionless… I’m out there watching you, office!

Once you’ve customized this config-sample.js file, be sure to rename it to config.js in order for it to be usable.

If you inspect the project code, you’ll notice that the MongoLab REST API is called in the logMsg() function, using an https.request.

Building this little demo has given me some new ideas for hardware hacking the cloud. I hope you give it a try too.

Thanks to the Arduino, Node.js and Javascript communities, and special thanks to Rick Waldon for Johnny Five, SendGrid for the UNO board, and a big shout out to @swiftalphaone for the Waza tutorial.

We’re excited to partner with Overdriver, a unique Platform-as-a-Service for gaming that is showcasing their beta service for the first time at this year’s Game Developers Conference (GDC)!

This exciting partnership means that MongoLab will serve as the core data storage provider for Overdriver’s online game developer community. You can find out more right here.

By marrying dynamic cloud scaling with gaming-specific tooling for features like virtual goods and social functionality, Overdriver addresses the biggest demand issue that game studios face: bringing titles to market quickly but with little operational risk.

MongoDB naturally supports the object-oriented information model in games, capturing concepts such as user characters, possessions, game pieces, and state of play. MongoLab’s robust and performant MongoDB-as-a-Service lets Overdriver game developers give their full attention to designing compelling games.

We are fascinated by Overdriver and honored to be an inaugural partner in their ground-breaking specialized PaaS approach.

If you’re in San Francisco this week stop by their booth, #1838, at GDC and meet the Overdriver team. Also, be sure to sign up for an account at Overdriver.com. FYI, creating a new environment at Overdriver automatically creates a new MongoLab account and database.

We can’t wait to play with what you build!

Don’t worry… we’ll get to the bottom of this! In doing so, we’ll examine a source of risk that’s easy to overlook in a MongoDB application: replication lag — what it means, why it happens, and what you can do about it.

Here’s what we’re going to cover:

Continuing this cautionary tale… Seriously, wtf?! You were doing everything right!

Using MongoDB with a well-designed schema and lovingly-tuned indexes, your application back-end has been handling thousands of transactions per second without breaking a sweat. You’ve got multiple nodes arranged in a replica set with no single point of failure. Your application tier’s Mongo driver connections are aware of the replica set and can follow changes in the PRIMARY node during failover. All critical writes are “safe” writes. Your app has been up without interruption for almost six months now! How could this have happened?

This unsettling situation has the hallmarks of an insidious foe in realm of high-availability data stewardship: unchecked replication lag.

Closely monitoring a MongoDB replica set for replication lag is critical.

What is replication lag?

As you probably know, like many data stores MongoDB relies on replication — making redundant copies of data — to meet design goals around availability.

In a perfect world, data replication would be instantaneous; but in reality, thanks to pesky laws of physics, some delay is inevitable — it’s a fact of life. We need to be able to reason about how it affects us so as to manage around the phenomenon appropriately. Let’s start with definitions…

For a given secondary node, replication lag is the delay between the time an operation occurs on the primary and the time that same operation gets applied on the secondary.

For the replica set as a whole, replication lag is (for most purposes) the smallest replication lag found among all its secondary nodes.

In a smoothly running replica set, all secondaries closely follow changes on the primary, fetching each group of operations from its oplog and replaying them approximately as fast as they occur. That is, replication lag remains as close to zero as possible. Reads from any node are then reasonably consistent; and, should the current primary become unavailable, the secondary that assumes the PRIMARY role will be able to serve to clients a dataset that is almost identical to the original.

For a variety of reasons, however, secondaries may fall behind. Sometimes elevated replication lag is transient and will remedy itself without intervention. Other times, replication lag remains high or continues to rise, indicating a systemic problem that needs to be addressed. In either case, the larger the replication lag grows and the longer it remains that way, the more exposure your database has to the associated risks.

Why is lag problematic?

Significant replication lag creates failure modes that can be problematic for a MongoDB database deployment that is meant to be highly available.  Here’s why:

• If your replica set fails over to a secondary that is significantly behind the primary, a lot of un-replicated data may be on the original primary that will need to be manually reconciled. This will be painful or impossible if the original primary is unrecoverable.

• If the failed primary cannot be recovered quickly, you may be forced to run on a node whose data is not up-to-date, or forced to take down your database altogether until the primary can be recovered.

• If you have only one secondary, and it falls farther behind than the earliest history retained in the primary’s oplog, your secondary will require a full resynchronization from the primary.
  • During the resync, your cluster will lack the redundancy of a valid secondary; the cluster will not return to high availability until the entire data set is copied.
  • If you only take backups from your secondary (which we highly recommend), backups must be suspended for the duration of the resync.

• Replication lag makes it more likely that results of any read operations distributed across secondaries will be inconsistent.

• A “safe” write with ‘w’ > 1 — i.e., requiring multiple nodes acknowledge the write before it returns — will incur latency proportional to the current replication lag, and/or may time out.

Strictly speaking, the problem of replication lag is distinct from the problem of data durability. But as the last point above regarding multi-node write concern illustrates, the two concepts are most certainly linked. Data that has not yet been replicated is not completely protected from single-node failure; and client writes specified to be safe from single-node failure must block until replication catches up to them.

What causes a secondary to fall behind?

In general, a secondary falls behind on replication any time it cannot keep up with the rate at which the primary is writing data. Some common causes:

Secondary is weak

To have the best chance of keeping up, a secondary host should match the primary host’s specs for CPU, disk IOPS, and network I/O. If it’s outmatched by the primary on any of these specs, a secondary may fall behind during periods of sustained write activity. Depending on load this will, at best, create brief excursions in replication lag and, at worst, cause the secondary to fall irretrievably behind.

Bursty writes

In the wake of a burst of write activity on the primary, a secondary may not be able to fetch and apply the ops quickly enough. If the secondary is underpowered, this effect can be quite dramatic. But even when the nodes have evenly matched specs, such a situation is possible. For example, a command like:

 db.coll.update({x: 7}, {$set: {y: 42}}, {multi: true}}

can place an untold number of separate “update” ops in the primary’s oplog. To keep up, a secondary must fetch those ops (max 4MB at a time for each getMore command!), read into RAM any index and data pages necessary to satisfy each _id lookup (remember: each oplog entry references a single target document by _id; the original query about “x” is never directly reflected the oplog), and finally perform the update op, altering the document and placing the corresponding entry into its oplog; and it must do all this in the same amount of time that the primary does merely the last step. Multiplied by a large enough number of ops, that disparity can amount to a noticeable lag.

Map/reduce output

A specific type of the extreme write burst scenario might be a command like:

 db.coll.mapReduce( ... { out: other_coll ... })

From the point of view of the oplog, the entire output collection basically materializes at once, from which point the replication to the secondary plays out as above.

Index build

It may surprise you to learn that, even if you build an index in the background on the primary, it will be built in the foreground on each secondary. There is currently no way to build indexes in the background on secondary nodes (cf. SERVER-2771). Therefore, whenever a secondary builds an index, it will block all other operations, including replication, for the duration. If the index builds quickly, this may not be a problem; but long-running index builds can swiftly manifest as significant replication lag.

Secondary is locked for backup

One of the suggested methods for backing up data in a replica set involves explicitly locking a secondary against changes while the backup is taken. Assuming the primary is still conducting business as usual, of course replication lag will climb until the backup is complete and the lock is released.

Secondary is offline

Similarly, if the secondary is not running or cannot reach the primary for whatever reason, it cannot make progress against the replication backlog. When it rejoins the replica set, the replication lag will naturally reflect the time spent away.

How do I measure lag?

Run the db.printSlaveReplicationInfo() command

To determine the current replication lag of your replica set, you can use the mongo shell and run the db.printSlaveReplicationInfo() command.

rs-ds046297:PRIMARY db.printSlaveReplicationInfo()

source: ds046297-a1.mongolab.com:46297
syncedTo: Tue Mar 05 2013 07:48:19 GMT-0800 (PST)
      = 7475 secs ago (2.08hrs)
source: ds046297-a2.mongolab.com:46297
syncedTo: Tue Mar 05 2013 07:48:19 GMT-0800 (PST)
      = 7475 secs ago (2.08hrs)

More than 2 hours — whoa, isn’t that a lot? Maybe!

See, those “syncedTo” times don’t have much to do with the clock on the wall; they’re just the timestamp on the last operation that the replica has copied over from the PRIMARY. If the last write operation on the PRIMARY happened 5 minutes ago, then yes: 2 hours is a lot. On the other hand, if the last op was 2.08 hours ago, then this is golden!

To fill in that missing piece of the story, we can use the db.printReplicationInfo() command.

rs-ds046297:PRIMARY db.printReplicationInfo()

configured oplog size:   1024MB
log length start to end: 5589secs (1.55hrs)
oplog first event time:  Tue Mar 05 2013 06:15:19 GMT-0800 (PST)
oplog last event time:   Tue Mar 05 2013 07:48:19 GMT-0800 (PST)
now:                     Tue Mar 05 2013 09:53:07 GMT-0800 (PST)

Let’s see … PRIMARY’s “oplog last event time” – SECONDARY’s “syncedTo” = 0.0. Yay.

As fun as that subtraction may be, it’s seldom called for. If there is a steady flow of write operations, the last op on the PRIMARY will usually have been quite recent. Thus, a figure like “2.08 hours” should probably raise eyebrows; you would expect to see a nice low number there instead — perhaps as high as a few seconds. And, having seen a low number, there would be no need to qualify its context with the second command.

Examine the “repl lag” graph in MMS

You can also view recent and historical replication lag using the MongoDB Monitoring Service (MMS) from 10gen. On the Status tab of each SECONDARY node, you’ll find the repl lag graph:

How do I monitor for lag?

It is critical that the replication lag of your replica set(s) be monitored continuously.   Since you have to sleep occasionally, this is a job best done by robots.  It is essential that these robots be reliable, and that they notify you promptly whenever a replica set is lagging too far behind.

Here are a couple ways you can make sure this is taken care of:

• If MongoLab is hosting your replica set, relax! For any multi-node, highly-available replica set we host for you, you can monitor replication lag in our UI and by default you will receive automated alerts whenever the replication lag exceeds 10 minutes.

• You can also set up an alert using the MMS system. Its exciting new features allow you to configure a replication lag alert:

What can I do to minimize lag?

Out of courtesy (for them or for ourselves), we would like to make those lag-monitoring automata’s lives as boring as possible. Here are some tips:

Tip #1: Make sure your secondary has enough horsepower

It’s not uncommon for people to run under-powered secondaries to save money — this can be fine if the write load is light. But in scenarios where the write load is heavy, the secondary might not be able to keep up with the primary. To avoid this, you should beef up your secondary so that it’s as powerful as your primary.

Specifically, a SECONDARY node should have enough network bandwidth that it can retrieve ops from the PRIMARY’s oplog at roughly the rate they’re created and also enough storage throughput that it can apply the ops — i.e., read any affected documents and their index entries into RAM, and commit the altered documents back to disk — at that same rate. CPU rarely becomes a bottleneck, but it may need to be considered if there are many index keys to compute and insert for the documents that are being added or changed.

Tip #2: Consider adjusting your write concern

Your secondary may be lagging simply because your primary’s oplog is filling up faster than it can be replicated. Even with an equally-brawny SECONDARY node, the PRIMARY will always be capable of depositing 4MB in its memory-mapped oplog in a fraction of the time those same 4MB will need to make it across a TCP/IP connection.

One viable way to apply some back-pressure to the primary might be to adjust your write concern.

If you are currently using a write concern that does not acknowledge writes (aka “fire-and-forget” mode), you can change your write concern to require an acknowledgement from the primary (w:1) and/or a write to the primary’s journal (j:true). Doing so will slow down the rate at which the concerned connection can generate new ops needing replication.

Other times it may be appropriate to use a ‘w’ > 1 or a ‘w’ set to “majority” to ensure that each write to the cluster is replicated to more than one node before the command returns. Requiring confirmation that a write has replicated to secondaries will effectively guarantee that those secondaries have caught up (at least up to the timestamp of this write) before the next command on the same connection can produce more ops in the backlog.

As previously alluded to, choosing the most appropriate write concern for the data durability requirements of your application — or for particular critical write operations within the application — is something you must give thought to irrespective of the replication lag issue we’re focusing on here. But you should be aware of the interrelationship: just as the durability guarantee of w>1 can be used as a means of forcing a periodic “checkpoint” on replication, excessive replication lag can show up as a surprisingly high latency (or timeout) for that very occasional critical write operation where you’ve used “w: majority” to make sure it’s truly committed.

Adjust to taste

Having servers acknowledge every write can be a big hit to system throughput. If it makes sense for your application, you can amortize that penalty by doing inserts in batches, requiring acknowledgement only at the end of each batch. The smaller the batch, the greater the back-pressure on PRIMARY data creation rate, and correspondingly greater potential adverse impact to overall throughput.

Don’t overdo it

Using a large value for ‘w’ can itself be problematic. It represents a demand that w nodes finish working through their existing backlog before the command returns. So, if replication lag is high (in the sense of there being a large volume of data waiting to copy over) when the write command is issued, the command execution time will suffer a proportionally high latency. Also, if enough nodes go offline such that ‘w’ cannot be satisfied, you have effectively locked up your database. This is basically the opposite of “high availability.”

Tip #3: Plan for index builds

As mentioned earlier, an index build on a secondary is a foreground, blocking operation. If you’re going to create an index that is sizeable, perhaps you can arrange to do it during a period of low write activity on the primary. Alternately, if you have more than one secondary, you can follow the steps here to minimize the impact of building large indexes.

Tip #4: Take backups without blocking

Earlier we discussed the technique of locking the secondary to do a backup. There are other alternatives to consider here, including filesystem snapshots and “point-in-time” backups using the “--oplog” option of mongodump without locking. These are preferable to locking the secondary during a period of active writes if there’s any chance you’ll use the secondary for anything other than backups.

Tip #5: Be sure capped collections have an _id field & a unique index

Reliable replication is not possible unless there is a unique index on the _id field. Before MongoDB version 2.2, capped collections did not have an _id field or index by default. If you have a collection like this, you should create an index on the _id field, specifying unique: true. Failing to do this can, in certain situations, cause replication to halt entirely. So … this should not be regarded as optional.

Tip #6: Check for replication errors

If you see that replication lag is only increasing (and never falling), your replica set could be experiencing replication errors. To check for errors, run rs.status() and look at the errmsg field in the result. Additionally, check the log file of your secondary and look for error messages there.

One specific example: if you see “RS102 too stale to catch up” in the secondary’s mongodb.log or in the errmsg field when running rs.status(), it means that secondary has fallen so far behind that there is not enough history retained by the primary (its “oplog size”) to bring it up to date. In this case, your secondary will require a full resynchronization from the primary.

In general, though, what you do in response to an error depends on the error. Sometimes you can simply restart the mongod process for your secondary; but the majority of the time you will need to understand the root cause of the error before you can fix the problem.

Don’t let replication lag take you by surprise.

At the end of the day, replication lag is just one more source of risk in any high-availability system that we need to understand and design around. Striking the right balance between performance and “safety” of write operations is an exercise in risk management — the “right” balance will be different in different situations. For an application on a tight budget with occasional spikes in write volume, for example, you might decide that a large replication lag in the wake of those spikes is acceptable given the goals of the application, and so an underpowered secondary makes sense. At the opposite extreme, for an application where every write is precious and sacred, the required “majority” write concern will mean you have essentially no tolerance for replication lag above the very minimum possible.  The good news is that MongoDB makes this all very configurable, even on an operation by operation basis.

We hope this article has given you some insight into the phenomenon of replication lag that will enable you to reason about the risk it poses for a high-availability MongoDB application, and armed you with some tools for managing it. As always, let us know if we can help!

Learn more and download the sample Node.js app right here at the Heroku Dev Center.