Tag Archives | index

How big is your MongoDB?

Update 11/10/14: The next blog post in this series is on managing disk space in MongoDB.

As your MongoDB grows in size, information from the db.stats() diagnostic command (or the database “Stats” tab in our management portal) becomes increasingly helpful for evaluating hardware requirements.

We frequently get questions about the dataSize, storageSize and fileSize metrics, so we want to help developers better understand how MongoDB storage works and what these particular metrics mean.

Continue Reading →

{ "comments": 6 }

Are you ready for production?

Transitioning your application from development to production is a monumental step. Over the years, we at MongoLab have acquired a lot of valuable experience helping our customers successfully manage their production-grade MongoDB deployments.

What follows is a checklist of action items that we’ve found imperative for successfully taking your application to production. For the experienced, we hope you can use this guide as a refresher on best practices. For the newer folks, this is a must-read on how to ready your database for the big move.

Continue Reading →

Remote Dex: Index Analysis Using the Profile Collection

(also posted to the 10gen blog: here)

Greetings Adventurers!

I’m excited to report that Dex (github) is now equipped with his first planned upgrade. For those of you who haven’t met him, Dex is an open-source python tool that suggests indexes for your MongoDB database. The initial release of Dex supported logfile-based analysis only. Now, Dex can be run remotely by leveraging MongoDB’s built-in database profiler. This new feature is invoked with the -p or --profile option. When you run Dex with -p, it connects to the specified MongoDB database and analyzes the queries already logged to your system.profile collection.

As the diagram below shows, this is particularly good news for MongoLab’s shared plan customers and anyone who does not have direct terminal access to their database machine.

In case you missed my introduction of Dex at the San Francisco MongoDB User Group Meetup, I’ll be presenting Dex at the Silicon Valley MongoDB Users Group on July 17 at 10gen’s offices in Palo Alto!

Here’s a quick set of steps to get you started.

  1. If you haven’t already, get Dex:
    sudo pip install dex
  2. Or, if you already have Dex:
    sudo pip install dex --upgrade
  3. Log into your database through the mongo shell and run db.setProfilingLevel(1). If your MongoDB is hosted with us at MongoLab, you can also enable profiling through our UI.
  4. Let your app run for a while. With profiling enabled, MongoDB deposits documents into system.profile. By default, each of these documents represents a database operation that took more than 100ms to complete. For apps that perform specific operation at specific times, you will need to profile during those times. Once you feel that your profile collection is populated with a representative set of data, you’re ready to run Dex!
  5. Run Dex with --profile or -p (instead of -f):
    dex <mongodb-uri> -p

    Where <mongodb-uri> is your database connection string (ex: mongodb://me:mypassword@myserver.mongolab.com:27227/mydb)

    Note: If you use a Sandbox plan on MongoLab (or do not have an admin URI for other reasons) you must provide a -n/–namespace filter to narrow your request, or your Dex attempt will fail for authentication reasons. (ex: -n "mydb.*")

    > dex mongodb://me:mypassword@myserver.mongolab.com:27227/mydb -p -n "mydb.*"

  6. Dex outputs index recommendations and corresponding creation syntax.  Because Dex relies on heuristics that don’t take your data into account, you’ll want to validate and sanity check Dex’s suggestions before implementing.
  7. Run db.setProfilingLevel(0) to disable profiling when you’re done. Profiling requires a small bit of overhead and is entirely diagnostic, so you don’t need to leave it running. If you like, you can also drop the system.profile collection afterwards.
  8. Enjoy!

As always: if you have any questions, bug reports, or feature requests, please email us at support@mongolab.com.

Until next time, good luck out there!


(updates 2012-07-16: fixed missing URI; 2012-07-17: added 10gen cross-post URL)

{ "comments": 2 }

Introducing Dex: the Index Bot

(update 2012-07-19: A new remote feature detailed here.)
(update 2012-10-09: Version 0.5 detailed here.)

Greetings adventurers! MongoLab is pleased to introduce Dex! We hope he assists you on your quests.

Dex is a MongoDB performance tuning tool, open-sourced under the MIT license, that compares logged queries to available indexes in the queried collection(s) and generates index suggestions based on a simple rule of thumb. To use Dex, you provide a path to your MongoDB log file and a connection URI for your database. Dex is also registered with PyPi, so you can install it easily with pip.

Quick Start

pip install dex


dex -f mongodb.log mongodb://localhost

Dex provides runtime output to STDERR as it finds recommendations:

"index": "{'simpleIndexedField': 1, 'simpleUnindexedFieldThree': 1}",
"namespace": "dex_test.test_collection",
"shellCommand": "db.test_collection.ensureIndex(
  {'simpleIndexedField': 1, 'simpleUnindexedFieldThree': 1},
  {'background': true})"

As well as summary statistics:

Total lines read: 7
Understood query lines: 7
Unique recommendations: 5
Lines impacted by recommendations: 5

Just copy and paste each shellCommand value into your MongoDB shell to create the suggested indexes.

Dex also provides the complete analysis on STDOUT when it’s done, so you will see this information repeated before Dex exits. The output to STDOUT is an entirely JSON version of the above, so Dex can be part of an automated toolchain.

For more information check out the README.md and tour the source code at https://github.com/mongolab/dex. Or if you’re feeling extra adventurous, fiddle with the source yourself!

git clone https://github.com/mongolab/dex.git

The motivation behind Dex

MongoLab manages tens of thousands of MongoDB databases, heightening our sensitivity to slow queries and their impact on CPU.  What started as a set of Continue Reading →

{ "comments": 17 }