Archive | general RSS feed for this section

Automated Slow Query Analysis: Dex recorded presentation from MongoSV 2012

Automated Slow Query Analysis: Dex the Index Robot

On December 4th, MongoLab engineer Eric Sedor presented about “Automated Slow Query Analysis: Dex the Index Robot” at MongoSV 2012, an annual one-day conference in Silicon Valley, CA, dedicated to the open source, non-relational database MongoDB.

A well-indexed query improves performance by several orders of magnitude. The trick is to identify an ideal set of indexes for a particular use case. Even for experts, hand-crawling MongoDB log for slow queries is a laborious process. Introducing Dex: an open-source automated tool for analyzing the slow query log or system.profile collection. Dex’s primary author Eric Sedor demonstrates Dex’s usage and elaborates on indexing topics from the basic to the advanced. Includes how to pick indexes in an elegant, practical way. You learn how Dex categorizes slow queries and recommends indexes to help keep your application running smoothly. Eric is an engineer at MongoLab, cloud-hoster of MongoDB, where Dex is used daily to optimize customer indexes.

Check out the recording below and please be sure to check out 10gen’s page for the presentation which has links to Speaker Deck slides and an alternate video link.

Here’s Dex’s github page, and the latest version (0.5) announcement

MongoLab sponsors Node Knockout hackathon 2012


We’re fans of Node.js and so we’re happy to be sponsoring the Node Knockout 48-hour hackathon and to be providing two prizes!

We host MongoDB at JoyentCloud with 500MB free plans.  JoyentCloud is where Nodejitsu hosts Node Knockout apps.  If you select JoyentCloud’s us-east-1 (Virgina) datacenter, you’ll have a low latency connection between your Node Knockout app and your database.

Instructions on how to provision a MongoLab server on the JoyentCloud (UPDATED):

  1. “npm update -g jitsu” OR “npm install -g jitsu”
  2. “jitsu databases create mongo”

We’ll have a person on-site at Joyent’s San Francisco headquarters for the kickoff on Nov 9.  As always, we have engineers at support@mongolab.com at the ready to answer questions.

We are offering the winners of the Team category and the Innovation category each a dedicated replica-set cluster with 2 GB RAM for 6 months.  May the best apps win!

updates: 2012-11-09 clarification on prizes, official Node Knockout URL added.  Old draft URL here: https://gist.github.com/4042395.  Updated instructions.

Dex 0.5: Weighted Index Recommendations for MongoDB

Greetings, adventurers!

I am happy to announce the latest version of Dex, MongoLab’s Index Bot. With version 0.5, we’re declaring Dex halfway-there in terms of major features. The most important and convenient improvements in this version are Weighted Index Recommendations and support for MongoDB 2.2 log files.

If you haven’t given Dex a try yet, check out the README and sudo pip install dex to get started. Even if you are already a Dex user, I highly recommend you revisit the readme, as Dex’s output has changed.

Now when you run Dex, Dex compiles statistics for its recommendations, and provides those in its output. In contrast to prior versions, this gives a weight to each suggested index, clearly identifying the worst offenders.

Here’s a rundown of major changes:

  • Weighted Recommendations – For each recommendation, Dex tallies the number of affected queries, the total time consumed by those queries, and the average time for a single query. Note that Dex keeps subtotal statistics for each unique query pattern that prompted a given recommendation. Subtotals are available in –verbose/-v mode only.
  • Output changes – We’ve modified Dex’s output for the purposes of readability and convenience, in the following ways:
    • By default (i.e., not in –watch/-w mode), Dex no longer provides runtime output. Dex reads the entire profile collection or log file and then outputs one set of full results.
    • In –watch/-w mode, Dex still provides runtime output, periodically printing all recommendations with up-to-date statistics.
    • The shell command suggestion is removed from default output in favor of concise, weighted index recommendations. The shell command is still available in –verbose/-v output, but is no longer included by default.
  • Support for MongoDB 2.2 log files – While Dex has supported MongoDB 2.2 in –profile/-p mode, updates to Dex’s log file regexes now support recent updates to the MongoDB log file line format.

For more information about Dex, check out Introducing Dex. As always, good luck out there!

Sincerely,
Eric@MongoLab

{ "comments": 2 }

MongoLab Discount for JS.everywhere() 2012

MongoLab is happy to be sponsoring the JS.everywhere() conference in Silicon Valley at the end of October. If you’re interested in joining us, please use this discount code on the registration page: “mongolabJS”.  You’ll get 50% discount on attendance. We are looking forward to seeing you there.

MongoDB’s native support for JSON of course makes it a natural fit to work with Javascript.  Javascript’s growing popularity beyond browser clients is driving the need for a scalable JSON persistence layer.  Having that cloud persistence layer at MongoLab, we get to see many interesting new projects in enterprises large and small.  So we’re excited to be reaching out to meet new users.

Details on our events page:

URL: http://www.jseverywhere.org/
Registration URL: http://jse2012.eventbrite.com/
Discount Code: mongolabJS
Dates: October 26-27, 2012
Location: San Jose, CA

Nodestack


We’re excited be part of the Oct 17 Nodestack.org online conference with Joyent, Clock Ltd, 10gen, and Nodejitsu.

What is Nodestack?

If you’re a Web developer you may have felt the same thing in the last year or so: Javascript is winning. More precisely, Joyent’s Node.js, supported by 10gen’s JSON-centric MongoDB database for persistence is winning. And by using SmartOS as the host for Node.js, Joyent offers the inspectability, performance, and debuggability capabilities of DTrace and ZFS to Nodestack.

Parochially, I’ve included a Google Trends widget above comparing “node”, “ruby”, and “java” when searched with “mongodb”. As of this writing, “node” had just crossed “ruby”s trend line and was headed up to challenge “java”.

Why Nodestack?

There are many reasons why Nodestack is emerging as a leading developer choice, including:

  • developer familiarity with Javascript from front-end browser domains
  • the battle-tested underlying Google V8 Javascript engine for high performance
  • a harmonious non-blocking asynchronous IO environment resulting in efficient CPU utilization
  • good fitness for demanding near real-time dynamic web and mobile applications
  • effortless JSON-awareness across the stack means fewer developer cycles wasted on data translation
  • a well-supported package management system with growing library of components for basic and advanced needs
  • a deep-bench ecosystem of infrastructure, platform and consulting services from vendors like Joyent Cloud, Nodejitsu and Clock Ltd. for even easier design, development, and production.
  • mdb_v8, DTrace and flame graphs (visual temporal call graphs) on SmartOS for fast root-cause analysis / debugging.

Nodestack Conference

At the Oct 17 online conference, you’ll talk with:

  • Nodejitsu’s Nuno Job on “Crazy, Cool Things You can do with Node.js”
  • 10gen’s Aaron Heckmann on “Node.js + MongoDB = Love” and why these technologies fit so well together.
  • Joyent’s Bryan Cantrill on “Stack Foundation = SmartOS” on SmartOS’ hypervisor benefits for Nodestack including flexibility for KVM virtualization
  • A panel including 10gen’s Jared Rosoff, Joyent’s Jason Hoffman, Clock Ltd’s Paul Serby and yours truly on the economic benefits of Nodestack.

So please sign up here to join us. The webcast is scheduled to start at 9am PT on Oct 17, 2012.

*If you are local in San Francisco, CA, we’re also inviting a few folks to join us as part of the studio audience. Email ben at mongolab dot com if you’re interested. See our Events page for other events.

Updated: 2012-09-28 with exact start time. Grammar fix ^less^fewer. Added link to Aaron’s preview post; mdb_v8, David Pacheo deck.

2012-10-01 fixed broken SmartOS link.

MongoSeattle 2012 with talk on Dex

(update: slide deck here https://speakerdeck.com/u/ericsedor/p/dex-the-index-bot)

We’re attending and sponsoring MongoSeattle 2012 on September 14!  Eric Sedor will be presenting his recent work on Automated Slow Query Analysis: Dex the Index Robot our open source tool at 11:30am in the “Sound” hall.  Source code and instructions available here on github.

We’ll have our booth to talk to folks about our MongoDB offering in the cloud and with PaaS providers.

Location: Bell Harbor International Conference Centerq 2211 Alaskan Way, Pier 66, Seattle, WA 98121 206.441.6666

Registration: here.

We have a few community discount passes left.  Email ben at mongolab.com if you’re interested.

PaaS and DBaaS as tools

At MongoLab we’re obsessed with software tools to get stuff done.  DBaaS and PaaS represent new tools in a developer’s toolbox.  To borrow a 1997 Steve Jobs metaphor below, writing an app is like constructing a building.  And a good building tool (in the then newly acquired NeXT case, OpenStep) “lets you start developing your app on the 20th floor [instead of the 7th].” Likewise a good PaaS lets you develop your online service by starting with a pre-assembled deployment, provisioning, and scaling infrastructure.

Reformatted from Macworld Expo Jan 1997 video frame at about 14:00 http://www.youtube.com/watch?v=QhhFQ-3w5tE

A PaaS also comes with a pre-hired, trained, and conditioned team to keep it updated and running.  “Conditioned” means a team that will likely have resolved an issue prior to you running into it.  It also means that operational maneuvers that are rare for any one user are practiced nearly daily across a large population.  For example, we are constantly restoring databases, upgrading servers, finding helpful indexes, and have it down to a science.

Finally, in a developed PaaS, add-ons provide components that raise the base functionality even higher.  We’re in several PaaS add-on marketplaces, and to end users add-ons represent a menu of capabilities from internet telephony to new databases to analytics.  To add-on providers, PaaS represents a route to a market that is comfortable with as-a-Service offerings.

Off the rack or artisanal?

In the early days of UI application APIs, ignoring the API and writing directly to bare metal was a way to eke out every last bit of performance and control.  That hack-around code was fragile, needing to be maintained across API revisions, and more likely to exhibit inter-application conflicts that made for user headaches.  (PC Interrupt hell anyone?)

Will there always be performance and control advantages to building out your own server or rack?  Yes, and if your business demands it for raw competitive or cash flow reasons (performance ~= money), then it makes sense.  But for a growing fraction of businesses, focusing precious developer energy into infrastructure is less important than building a killer feature.

PaaS is a disruptive change: replacing existing processes, enabling new businesses, and needing market education.  New tools are always challenging practitioners to learn and adapt.  To an expert lathe operator, the language of CNC milling: 3D solid models and computerized tool paths can be unwieldy and foreign.  But to an industrial designer versed in modern manufacturing, CNC milling becomes a new way to express oneself, maybe even for an aluminum chassis previously unattainable at scale. That designer can still create objects with chisel and wood, and he can refine manufactured models with hand files and drills.

At the end of the day, any new tool must prove its worth and find its place in the toolkit.  As practitioners of software, we are always comparing the “known good” tools in our kit with the “new potentials”.  PaaS and DBaaS are tools that should evaluate for their capabilities and understand how they fit into your toolbelt.


Join fellow PaaS and DBaaS providers Nodejitsu, Xeround, Cloudant, and us for a discussion on PaaS and DBaaS hosted by Joyent.  Starts at August 22, 2012 at 11am US Pacific. Register here.

MongoLab Add-On at AppFog polyglot PaaS

MongoLab is now available as an Add-On at AppFog!

MongoLab was the second Add-On to be included in PHPFog, AppFog’s predecessor (the first was NewRelic). So we’ve been fortunate to work with the team at AppFog and witness their transformation from a solid PHP point PaaS to a solid multi-cloud polyglot PaaS vendor, adopting the open CloudFoundry framework.

Now that the AppFog Add-On program has been launched, we’re especially happy to serve an expanded audience including now NodeJS, Ruby, Python, Java, and .NET programmers.

Click Add-Ons in the left nav and click Install to provision and configure a database at MongoLab. Documentation is available here: http://docs.phpfog.com/addons/mongolab.  MongoLab support is available at support@mongolab.com. Happy Mongo-ing!

{ "comments": 1 }