About eric

Author Archive | eric

About eric

Here are my most recent posts

Telemetry Series: Queues and Effective Lock Percent

A key component of optimizing application performance is tuning the performance of the database that supports it. Each post in our Telemetry series discusses an important metric used by developers and database administrators to tune the database and describes how MongoLab users can leverage Telemetry, MongoLab’s monitoring interface, to effectively review and take action on these metrics.

Queues and Effective Lock Percent

Any time an operation can’t acquire a lock it needs, it becomes queued and must wait for the lock to be released. Because operations that are queued on the database side often imply that operations are queued on the application side, Queues is an important Telemetry metric for assessing how well your MongoLab deployment is responding to the demands of your app.

In MongoDB 2.6 and earlier, you will find that Queues tend to rise and fall with Effective Lock %. This is because Queues refers specifically to the operations that are waiting on another operation (or series of operations) to release MongoDB’s database-level and global-level locks.

With MongoDB 3.0 (using the MMAP storage engine), locking is enforced at the collection level, and Effective Lock % is not reported as a server-level metric. This makes the Queues metric even more important. While it may not be clear from the Telemetry interface exactly which collection(s) is/are heavily locked, elevated queueing is usually a consequence of locking.

The focus on Queues is also preferable because, by design, locking is going to happen on any MongoDB that is receiving writes. As long as that locking isn’t resulting in queueing, it is usually not a concern.

Image of Telemetry charts

High locks leading to high queues

What is Effective Lock Percent?

MongoDB uses multi-granular reader-writer locking. Reads prevent a write from acquiring the lock, and a write prevents reads or other writes from acquiring the lock. But, reads do not block other reads. As well, each operation holds the lock at a granularity level appropriate for the operation itself.

In MongoDB 2.6 there are two granularity levels: a Global lock and Database lock for each database. In this scheme, operations performed on separate databases do not lock each other unless those operations also require the Global lock.

Effective Lock Percent in MongoDB 2.6 is a calculated metric that adds together the Global Lock % and the Lock % of the most-locked database at the time. Because of this computation, and because of the way operations are sampled, values greater than 100% may occur.    

In MongoDB 3.0 with the MMAP storage engine, MongoDB locks at the Global, Database, and Collection-level. A normal write operation holds the Database lock in MongoDB 2.6, but only holds a specific collection’s Collection lock in MongoDB 3.0. This improvement means separate collections can be concurrently read from or written to.

MongoDB 3.0 with the WiredTiger storage engine uses document-level locking for even greater parallelization. Writes to a single collection won’t block each other unless they are to the same document.

Note that locking operations can and do yield periodically, so incoming operations may still progress on a heavily locked server. For more detail, read MongoDB’s concurrency documentation.

What do I do if I see locking and queueing?

Locking is a normal part of databases so some level of locking and queueing is expected. First, consider if the locking and queueing is a problem. You should typically not be concerned with Effective Lock Percent values of less than 15%, but each app is different. Likewise, queueing can be fine as long as the app is not blocked on queued requests.

If you see a rise in Queues and Effective Lock % in Telemetry that corresponds to problems with your application, try the following steps:

  1. If queues and locks coincide with Page Faults, check out Telemetry Series: Page Faults–the previous blog in this series–for potential resolutions, such as optimizing indexes or ultimately increasing RAM.
  2. If locking and queueing don’t coincide with Page Faults, there are two potential causes:
    1. You may have an inefficient index. While poor indexing typically leads to page faulting, this is not the case if all of your data and indexes already fit into available RAM. Yet the CPU cost of collection scans can still cause a lock to be held for longer than necessary. In this case, reduce collection scanning using the index optimization steps in Telemetry Series: Page Faults.
    2. If operations are well-indexed, check your write operations and consider reducing the need for frequent incidents of:
      • updates to large documents
      • updates that require document moves
      • full-document updates (i.e., those that don’t use update operators)
      • updates using array update operators like $push, $pull, etc.
  3. If queuing and locking cannot be reduced by improving indexes, write strategies, or the data model, it is time to consider heavier hardware, and potentially sharding.

Importantly, queueing can occur because of a small number of long-running operations. If those operations haven’t finished yet, they won’t appear in the mongod logs.  Viewing and potentially killing the offending current operations can be a short-term fix until those operations can be examined for efficiency. To learn more about viewing and killing operations, refer to our documentation on Operation Management.

Have questions or feedback?

We’d love to hear from you as this Telemetry blog series continues. What topics would be most interesting to you? What types of performance problems have you struggled to diagnose?

Email us at support@mongolab.com to let us know your thoughts, or to get our help tuning your MongoLab deployment.


Telemetry Series: Page Faults

A key component of optimizing application performance is tuning the performance of the database that supports it. Each post in our Telemetry series discusses an important metric used by developers and database administrators to tune the database and describes how MongoLab users can leverage Telemetry, MongoLab’s monitoring interface, to effectively review and take action on these metrics.

Page Faults

Databases are optimized for working with data that is stored on disk, but usually cache as much data as possible in RAM in order to access disk as infrequently as possible. However, as it is cost-prohibitive to store in RAM all the data accessed by the application, the database must eventually go to disk. Because disks are slower than RAM, this incurs a significant time cost.

Effectively tuning a database deployment commonly involves assessing how often the database accesses disk with an eye towards reducing the need to do so. To that end, one of the best ways to analyze the RAM and disk needs of a MongoDB deployment is to focus on what are called Page Faults.

What is a Page Fault?

MongoDB manages documents and indexes in memory by using an OS facility called MMAP, which translates data files on disk to addresses in virtual memory. The database then accesses disk blocks as though it is accessing memory directly. Meanwhile, the operating system transparently keeps as much of the mapped data cached in RAM as possible, only going to disk to retrieve data when necessary.

When MMAP receives a request for a page that is not cached, a Page Fault occurs, indicating that the OS had to read the page from disk into memory.

What do Page Faults mean for my cluster?

The frequency of Page Faults indicates how often the OS goes to disk to read data. Operations that cause Page Faults are slower because they necessarily incur disk latency.

Page Faults are one of the most important metrics to look at when diagnosing poor database performance because they suggest the cluster does not have enough RAM for what you’re trying to do. Analyzing Page Faults will help you determine if you need more RAM, or need to use RAM more efficiently.

How does Telemetry help me interpret Page Faults?

Select a deployment and then look back through Telemetry over months or even years to determine the normal level of Page Faults. In instances where Page Faults deviate from that norm, check application and database logs for operations that could be responsible. If these deviations are transient and infrequent they may not pose a practical problem. However, if they are regular or otherwise impact application performance you may need to take action.

A burst in Page Faults corresponding to an increase in database activity.

A burst in Page Faults corresponding to an increase in database activity.

If Page Faults are steady but you suspect they are too high, consider the ratio of Page Faults to Operations. If this ratio is high it could indicate unindexed queries or insufficient RAM. The definition of “high” varies across deployments and requires knowledge of the history of the deployment, but consider taking action if any of the following are true:

  • The ratio of Page Faults to Operations is greater than or equal to 1.
  • Effective Lock % is regularly above 15%.
  • Queues are regularly above 0.
  • The app seems sluggish.

Note: Future Telemetry blog posts will cover additional metrics, such as Effective Lock % and Queues. See MongoDB’s serverStatus documentation for more information.

How do I reduce Page Faults?

How you reduce Page Faults depends on their source. There are three main reasons for excessive Page Faults.

  1. Not having enough RAM for the dataset. In this case, the solution is to add more RAM to the deployment by scaling either vertically to machines with more RAM, or horizontally by adding more shards to a sharded cluster.
  2. Inefficient use of RAM due to lack of appropriate indexes. The most inefficient queries are those that cause collection scans. When a collection scan occurs, the database is iterating over every document in a collection to identify the result set for a query. During the scan, the whole collection is read into RAM, where it is inspected by the query engine. Page Faults are generally acceptable when obtaining the actual results of a query, but collection scans cause Page Faults for documents that won’t be returned to the app. Worse, these unnecessary Page Faults are likely to evict “hot” data, resulting in even more Page Faults for subsequent queries.
  3. Inefficient use of RAM due to excess indexes. When the indexed fields of a document are updated, the indexes that include those fields must be updated. When a document is moved on disk, all indexes that contain the document must be updated. These affected indexes must enter RAM to be updated. As above, this can lead to thrashing memory.

Note: For assistance determining what indexes your deployment needs, MongoLab offers a Slow Query Analyzer that provides index recommendations to Shared and Dedicated plan users.

Have questions or feedback?

We’d love to hear from you as this Telemetry blog series continues. What topics would be most interesting to you? What types of performance problems have you struggled to diagnose?

Email us at support@mongolab.com to let us know your thoughts, or to get our help tuning your MongoLab deployment.

{ "comments": 1 }

[“Thinking”, “About”, “Arrays”, “In”, “MongoDB”]

Greetings adventurers!

The growing popularity of MongoDB means more and more people are thinking about data in ways divergent from traditional relational models. For this reason alone, it’s exciting to experiment with new ways of modelling data. However, with additional flexibility comes the need to properly analyze the performance impact of data model decisions.

Embedding arrays in documents is a great example of this. MongoDB’s versatile array operators ($push/$pull, $addToSet, $elemMatch, etc.) offer the ability to manage data sets within documents. However, one must be careful. Data models that call for very large arrays, or arrays with high rates of modification, can often lead to performance problems.

Continue Reading →

{ "comments": 33 }

How to use MongoDB on RedHat OpenShift with MongoLab

Hey RedHat fans – we’ve got your MongoDB hosting needs covered!

In today’s post we’ll be presenting a quick-start guide on how to connect OpenShift, the free RedHat auto-scaling Platform-as-a-Service (PaaS), with our popular MongoDB Database-as-a-Service (DBaaS), MongoLab.

For demonstration purposes, we’ll be using a Node.js application that we’ve written (available for download here). All it takes to connect your OpenShift application is five easy steps!

Continue Reading →

{ "comments": 7 }

Node.js and MongoLab on Windows Azure

(This tutorial was originally published on the Windows Azure documentation portal in January 2013)

Greetings, adventurers! Welcome to MongoDB-as-a-Service. Are you looking to create a Node.js Application on Windows Azure with MongoDB using the MongoLab Azure Store add-on?

In this tutorial you will:

  1. Provision the database – The Windows Azure Store MongoLab add-on will provide you with a MongoDB database hosted in the Windows Azure cloud and managed by MongoLab‘s cloud database platform.
  2. Create the app – It’ll be a simple Node.js app for maintaining a list of tasks.
  3. Deploy the app – By tying a few configuration hooks together, we’ll make pushing our code a breeze.
  4. Manage the database – Finally, we’ll show you MongoLab’s web-based database management portal where you can search, visualize, and modify data with ease.

At any time throughout this tutorial, feel free to kick off an email to support@mongolab.com if you have any questions. To complete this tutorial, you need a Windows Azure account that has the Windows Azure Web Sites feature enabled. You can create a free trial account and enable preview features in just a couple of minutes. For details, see the Create a Windows Azure account and enable preview features tutorial.

In addition, ensure that you have the following installed:
Continue Reading →

{ "comments": 2 }

Dex 0.5: Weighted Index Recommendations for MongoDB

Greetings, adventurers!

I am happy to announce the latest version of Dex, MongoLab’s Index Bot. With version 0.5, we’re declaring Dex halfway-there in terms of major features. The most important and convenient improvements in this version are Weighted Index Recommendations and support for MongoDB 2.2 log files.

If you haven’t given Dex a try yet, check out the README and sudo pip install dex to get started. Even if you are already a Dex user, I highly recommend you revisit the readme, as Dex’s output has changed.

Now when you run Dex, Dex compiles statistics for its recommendations, and provides those in its output. In contrast to prior versions, this gives a weight to each suggested index, clearly identifying the worst offenders.

Here’s a rundown of major changes:

  • Weighted Recommendations – For each recommendation, Dex tallies the number of affected queries, the total time consumed by those queries, and the average time for a single query. Note that Dex keeps subtotal statistics for each unique query pattern that prompted a given recommendation. Subtotals are available in –verbose/-v mode only.
  • Output changes – We’ve modified Dex’s output for the purposes of readability and convenience, in the following ways:
    • By default (i.e., not in –watch/-w mode), Dex no longer provides runtime output. Dex reads the entire profile collection or log file and then outputs one set of full results.
    • In –watch/-w mode, Dex still provides runtime output, periodically printing all recommendations with up-to-date statistics.
    • The shell command suggestion is removed from default output in favor of concise, weighted index recommendations. The shell command is still available in –verbose/-v output, but is no longer included by default.
  • Support for MongoDB 2.2 log files – While Dex has supported MongoDB 2.2 in –profile/-p mode, updates to Dex’s log file regexes now support recent updates to the MongoDB log file line format.

For more information about Dex, check out Introducing Dex. As always, good luck out there!


{ "comments": 2 }

Remote Dex: Index Analysis Using the Profile Collection

(also posted to the 10gen blog: here)

Greetings Adventurers!

I’m excited to report that Dex (github) is now equipped with his first planned upgrade. For those of you who haven’t met him, Dex is an open-source python tool that suggests indexes for your MongoDB database. The initial release of Dex supported logfile-based analysis only. Now, Dex can be run remotely by leveraging MongoDB’s built-in database profiler. This new feature is invoked with the -p or --profile option. When you run Dex with -p, it connects to the specified MongoDB database and analyzes the queries already logged to your system.profile collection.

As the diagram below shows, this is particularly good news for MongoLab’s shared plan customers and anyone who does not have direct terminal access to their database machine.

In case you missed my introduction of Dex at the San Francisco MongoDB User Group Meetup, I’ll be presenting Dex at the Silicon Valley MongoDB Users Group on July 17 at 10gen’s offices in Palo Alto!

Here’s a quick set of steps to get you started.

  1. If you haven’t already, get Dex:
    sudo pip install dex
  2. Or, if you already have Dex:
    sudo pip install dex --upgrade
  3. Log into your database through the mongo shell and run db.setProfilingLevel(1). If your MongoDB is hosted with us at MongoLab, you can also enable profiling through our UI.
  4. Let your app run for a while. With profiling enabled, MongoDB deposits documents into system.profile. By default, each of these documents represents a database operation that took more than 100ms to complete. For apps that perform specific operation at specific times, you will need to profile during those times. Once you feel that your profile collection is populated with a representative set of data, you’re ready to run Dex!
  5. Run Dex with --profile or -p (instead of -f):
    dex <mongodb-uri> -p

    Where <mongodb-uri> is your database connection string (ex: mongodb://me:mypassword@myserver.mongolab.com:27227/mydb)

    Note: If you use a Sandbox plan on MongoLab (or do not have an admin URI for other reasons) you must provide a -n/–namespace filter to narrow your request, or your Dex attempt will fail for authentication reasons. (ex: -n "mydb.*")

    > dex mongodb://me:mypassword@myserver.mongolab.com:27227/mydb -p -n "mydb.*"

  6. Dex outputs index recommendations and corresponding creation syntax.  Because Dex relies on heuristics that don’t take your data into account, you’ll want to validate and sanity check Dex’s suggestions before implementing.
  7. Run db.setProfilingLevel(0) to disable profiling when you’re done. Profiling requires a small bit of overhead and is entirely diagnostic, so you don’t need to leave it running. If you like, you can also drop the system.profile collection afterwards.
  8. Enjoy!

As always: if you have any questions, bug reports, or feature requests, please email us at support@mongolab.com.

Until next time, good luck out there!


(updates 2012-07-16: fixed missing URI; 2012-07-17: added 10gen cross-post URL)

{ "comments": 2 }

Introducing Dex: the Index Bot

(update 2012-07-19: A new remote feature detailed here.)
(update 2012-10-09: Version 0.5 detailed here.)

Greetings adventurers! MongoLab is pleased to introduce Dex! We hope he assists you on your quests.

Dex is a MongoDB performance tuning tool, open-sourced under the MIT license, that compares logged queries to available indexes in the queried collection(s) and generates index suggestions based on a simple rule of thumb. To use Dex, you provide a path to your MongoDB log file and a connection URI for your database. Dex is also registered with PyPi, so you can install it easily with pip.

Quick Start

pip install dex


dex -f mongodb.log mongodb://localhost

Dex provides runtime output to STDERR as it finds recommendations:

"index": "{'simpleIndexedField': 1, 'simpleUnindexedFieldThree': 1}",
"namespace": "dex_test.test_collection",
"shellCommand": "db.test_collection.ensureIndex(
  {'simpleIndexedField': 1, 'simpleUnindexedFieldThree': 1},
  {'background': true})"

As well as summary statistics:

Total lines read: 7
Understood query lines: 7
Unique recommendations: 5
Lines impacted by recommendations: 5

Just copy and paste each shellCommand value into your MongoDB shell to create the suggested indexes.

Dex also provides the complete analysis on STDOUT when it’s done, so you will see this information repeated before Dex exits. The output to STDOUT is an entirely JSON version of the above, so Dex can be part of an automated toolchain.

For more information check out the README.md and tour the source code at https://github.com/mongolab/dex. Or if you’re feeling extra adventurous, fiddle with the source yourself!

git clone https://github.com/mongolab/dex.git

The motivation behind Dex

MongoLab manages tens of thousands of MongoDB databases, heightening our sensitivity to slow queries and their impact on CPU.  What started as a set of Continue Reading →

{ "comments": 18 }