Build Custom Dashboards with MongoDB, Azure & Serverless Functions

Original Source: https://www.sitepoint.com/build-custom-dashboards-with-mongodb-azure-serverless-functions/

This article was originally published on Ahmad Awais. Thank you for supporting the partners who make SitePoint possible.

TL;DR: I’m building a custom WordPress dashboard for an enterprise client which is powered by React.js on top of Node.js, with MongoDB Atlas as the database.

This dashboard uses several Microsoft Azure services, e.g., Cognitive Services, Azure App Services, and especially serverless Azure Functions. In this post, you’ll learn how to build a small module from it and the reason behind my choice of stack, apps, and products.

One of my enterprise clients who owns a huge networking and media company has a large-scale WordPress site set up. He recently consulted me about the possibility of building a custom WordPress dashboard (based on the WordPress REST API) — to help him make intelligent business decisions via Machine Learning and Artificial Intelligence.

With JavaScript eating up the world and WordPress adapting to the move by creating the Gutenberg project, I thought of an architecture/stack where WordPress would be our content layer, a familiar battle-tested environment that does its job well with a custom dashboard that’s built with JavaScript.

When you’re tasked to build a modern JavaScript application, you find yourself in a mix of different frameworks, tools, and dev workflows. The JavaScript ecosystem has grown a lot over the last couple of years. We have many, many good options available today.

So, after researching my options for a bit, I opted to use React.js on top of Node.js to start building the custom WordPress dashboard. While the project is in its ideation phase at the moment, I think it’s important that I share some of our goals here to define context behind my choice of the stack.

Custom WordPress Dashboard Goals

Imagine you own a large networking company where over 500 hotels (in three different countries) use your services to power their conference halls, IT meetings, and online property management like the sites and blogs. That’s what my client does.

Most of this is powered by a huge multi-site WordPress instance that manages everything for the hotels, websites, online booking, registrations, events, tickets, reviews, and comments. There’re also other systems running different software which are able to produce content via REST API.

We’ve set out to create a custom WordPress dashboard with many goals in mind, but I’m listing a few of them which are related to this particular article. Take a look at what I have built so far, it’s all based on serverless Azure functions — which are pretty awesome.

? High-level Data Reporting

The custom dashboard will report all the high-level data, e.g. things like live sales happening throughout my client’s portfolio (500+ hotels), entity/time-based and date-based breakdowns.

And how each of his franchise performing on a daily, weekly, monthly basis. All of this data is being fed to MongoDB Atlas. More on that later.

⚡ Serverless Automation

Most of the modules are built upon serverless architecture — which in this case provides huge benefits. All the automation is always running and the cost is paid as you go i.e. pay for what you use.

An initial rough estimate puts this solution 34% more economical than having a server VM running all the time. We are using Azure Functions for this serverless automation.

? IoT (Internet of Things) Hub

There are about ~200 IT managers working for my client who have IoT enabled devices that feed data into several online projects. This custom dashboard also includes that data for making better decisions and connecting the whole registration, management, maintenance team’s hub into a single place.

As you might have already guessed, this project makes use of IoT Hub from Microsoft Azure to connect, monitor, and manage all of the IoT assets.

? Machine Learning and Artificial Intelligence

We’re using a lot of different services from Microsoft Azure for the sake of making this dashboard artificially intelligent by Machine Learning.

There’s a huge dataset that is fed to the ML Studio which later helps us predict different decisions like space management, low registrations trends for IT events, and questions like why and when these things happen.

While the Machine Learning part is beyond the scope of this article, I still plan to touch base with some of the awesome Artificial Intelligence I’ve been able to cook in via Azure’s Cognitive Services.

? Live & Real-time

One of the most important aspects of this custom dashboard is that it’s live and real-time. Which means I need a managed database that can cope with this amount of data and still stay highly available.

But at the same time, it’s for the management purposes and doesn’t need to have any impact on the WordPress sites. That is a crucial system design decision for this dashboard.

By that what I mean is we can do all sorts of experiments with this custom dashboard but it shouldn’t have any impact on the database/servers which are running the multi-site WordPress instance.

MongoDB & MongoDB Atlas

For this custom WordPress dashboard, I am using MongoDB Atlas as a DBaaS (Database as a Service). And I couldn’t be happier. When I first shared that I’d be using MongoDB, many developers had concerns.

Most of the questions asked why I’d add another layer of complexity by adding yet another database to the mix. Why not use the WordPress database as it is? To answer these questions and more I have prepared a list of reasons as to why I am using MongoDB Atlas.

♨ Dislike for RDBMS

I personally dislike relational databases. Most of the time, for me they get in the way of building applications. I have to completely get out of the app I am building, think about my database in the future and design a good schema which always ends up a bad exercise for my dev workflow. It’s counter-intuitive at best — at least for me, it is.

? HDD Is Cheap — CPU/RAM Is Not

Old databases were mostly designed to save disk space, among other things. This led to a plethora of problems like normalization, indexing, and made sharding, auto-scaling, and replication harder.

Nowadays, disk space is dirt cheap. On the other hand, CPU/RAM is not, and your sysadmin costs can skyrocket very quickly if you end up with a bad choice here.

Like you wanted to create a custom dashboard but your system design architect cost you two sysadmins with how they chose to design your system. Similarly, my client wanted a managed solution without having to hire a team of IT/DevOps folks — at least for an experimental custom dashboard.

? MongoDB’s Pros

Schema-less. Flexible schema for the win. I don’t have to change anything, my regular app development workflow, creating a Node.js-based app that I am manipulating with JSON type data, I can just feed that into MongoDB and it just works.
Workflow-consistency. Creates documents the way my custom dashboard is represented. Sales, Videos, Talks, Comments, Reviews, Registrations, etc. all of that have similar data representation on the frontend and the backend — and even in the database. I manage 3rd party data via middleware. This consistency translates to clean code.
Ease of scale-out. It scales reads by using replica sets. Scales writes by using sharding (auto-balancing). Just fire up another machine and away you go. Most importantly, instead of vertical scaling via RDBMS, MongoDB lets you scale horizontally with different levels of consistency. That’s a big plus. ➕
Cost. Depends on which RDBMS of course, but MongoDB is free and can run on Linux, ideal for running on cheaper commodity kits.

? Why MongoDB Atlas?

Well, now that I know MongoDB is the right database choice, there are so many different options to host your database. I can self-host on my Linux machine via DigitalOcean, use a cloud provider like AWS/Azure or a choose a DBaaS service specific to MongoDB.

But I want a fast, secure, and managed MongoDB solution that I can easily scale with the growth of the number of modules we attach in this custom WordPress dashboard. That’s MongoDB Atlas.

MongoDB Atlas is a cloud-hosted MongoDB service engineered and run by the same team that builds the database. And guess what, I trust that they follow the best operational practices since they are the ones who’re building MongoDB in the first place.

I want this custom dashboard to be self-managed, serverless, and using MongoDB Atlas saves me from worrying about software patching, backups, and reliable configuration setup for new DB updates. Again a big plus. ➕

Also, the fact that MongoDB Atlas is supported cross-platform as well as cross-region and across different cloud providers makes it a much better choice. I think each Cluster comes with two replica sets, ready to scale.

? MongoDB Compass

Now that we are going to work with MongoDB, it’d be great to have a tool through which we can explore our database, view the changes, debug and so on. For this purpose, MongoDB again takes the lead with a product called MongoDB Compass. Take a look.

I suggest that you go ahead and download MongoDB Compass. It’s literally the best tool to visualize your MongoDB database. Here’s a set of features:

Visualize and explore: Take a look at your database, find out how things are looking, and even visualize stuff like maps/coordinates.
Insert, modify, and delete: You can also perform CRUD operations for your DB right from MongoDB compass. Makes testing easier.
Debug and optimize: Finally, analyze your data, debug it and even find out about performance issues right inside a great GUI for your database. This tool is a must-have if you work with MongoDB.
Extensible: And the best part is you can build your own plugins to extend MongoDB Compass. Here’s the documentation on building your own Compass plugins.
Enterprise Flavor: MongoDB Compass comes in a few flavors: Community (Free), and Enterprise (Licensed) — the Enterprise version is the one that lets you visualize DB schema.

✅ Getting Started with MongoDB Atlas

Let’s get started and build a simple module that’s part of the custom WordPress dashboard I am building. For this module, we are collecting all the sales related data. For that, we need a MongoDB instance, and of course we’re using MongoDB Atlas here.

Step #1: Go to MongoDB Atlas →

Go to the MongoDB Atlas site and register a completely free MongoDB instance hosted on AWS, with shared RAM and 512 MB storage. Click the Get Started Free button.

Step #2: Sign up at MongoDB Atlas →

Now go ahead and sign up with your email ID and fill up the details. It’s amazing that you can sign up and use a free MongoDB Atlas hosted DB instance, and they don’t even require you to add a credit card for that.

Step #3: Create the Cluster

Now you’ll be redirected to a page with a bunch of information about the new MongoDB Cluster you’re about to create. I suggest that you review this information, and move ahead by clicking the Create Cluster button at the bottom just like in the screenshot below.

Step #4: Create DB Username & Password

It’ll take a minute and your DB will be created. Once that happens, head over to the Security > MongoDB Users and click on the + ADD NEW USER button on the right, to create a new user for your database. Let’s keep all the other settings set to default for the sake of this intro-article.

I’m setting the user/pass as usermongo but you know better.

Step #5: Add IP to Whitelist for Access

To be able to access your MongoDB Atlas database, you need to setup the IP Whitelist with the IP of your server where your app is hosted. Authentication is beyond what I am discussing here so for the purpose of this demo let’s just allow everyone (obviously a bad practice in production).

So, again, head over to the Security > IP Whitelist and click on the + ADD IP ADDRESS button on the right, and finally ALLOW ACCESS FROM ANYWHERE button to allow the anonymous access.

Step #6: Connect via MongoDB Compass

Now that our DB’s IP access and a user has been created, we can pick up the connection string and use it to connect to our database with our MongoDB Compass application.

Go to Connect then choose Connect with MongoDB Compass and download Compass if you haven’t. Copy the URI Connection String. Finally, open Compass and it should be able to detect the connection string in your clipboard, allow it to connect to your database.

And you are set to visualize your database, analyze its performance, and even run complete CRUD operations. Awesome! ?

Now that we have created a MongoDB Atlas and connected it with MongoDB Compass, we can move forward and start building our Node.js application.

WordPress REST API — FTW!

This WordPress based Node.js custom dashboard interacts with the WordPress instance via the WordPress REST API. Since this is a Node.js app, I am using an awesome library called wpapi written by K Adam White. He has also built a demo Express based WordPress app. That’s what I got inspired by while building this custom dashboard, so you’ll see a lot of it here.

? WordPress Custom Router Based on Express

The router is set up with Express. Here’s a basic error handler and router template for using WordPress with express.

‘use strict’;

var express = require(‘express’);
var router = express.Router();
var siteInfoMiddleware = require(‘../middleware/site-info’);

// Set global site info on all routes
router.use(siteInfoMiddleware);

// Public Routes
// =============

router.get(‘/’, require(‘./index’));
router.get(‘/page/:page’, require(‘./index’));
router.get(‘/:slug’, require(‘./single’));
router.use(‘/tags/:tag’, require(‘./tag’));
router.use(‘/categories/:category’, require(‘./category’));

// Catch 404 and forward to error handler.
router.use(function (req, res, next) {
var err = new Error(‘Not Found’);
err.status = 404;
next(err);
});

// Error Handling
// ==============

// Development error handler will print stacktrace.
function developmentErrorRoute(err, req, res, next) {
res.status(err.status || 500);
res.render(‘error’, {
message: err.message,
error: err
});
}

// Production error handler. No stacktraces leaked to user.
function friendlyErrorRoute(err, req, res, next) {
res.status(err.status || 500);
res.render(‘error’, {
message: err.message,
error: {}
});
}

// Configure error-handling behavior
if (router.get(‘env’) === ‘development’) {
router.use(developmentErrorRoute);
} else {
router.use(friendlyErrorRoute);
}

module.exports = router;

View the code on Gist.

? Basic Express Based Implementation

I am not hosting this entire thing on WordPress, but the initial plan was to do just that. If you want to go do that, you’d wanna build the index by querying all the info using the RSVP.hash utility for convenience and parallelism. For that here’s what you should do.

‘use strict’;

var wp = require( ‘../services/wp’ );
var contentService = require( ‘../services/content-service’ );
var pageNumbers = require( ‘../services/page-numbers’ );
var pageTitle = require( ‘../services/page-title’ );
var RSVP = require( ‘rsvp’ );

function getHomepage( req, res, next ) {
var pages = pageNumbers( req.params.page );

RSVP.hash({
archiveBase: ”,
pages: pages,
title: pageTitle(),
// Primary page content
posts: wp.posts().page( pages.current ),
sidebar: contentService.getSidebarContent()
}).then(function( context ) {
if ( req.params.page && ! context.posts.length ) {
// Invalid pagination: 404
return next();
}

res.render( ‘index’, context );
}).catch( next );
}

module.exports = getHomepage;

View the code on Gist.

? Authentication Cooked In

For this setup, you’ll also need to authenticate your Node.js app by giving it the authentication data, which along with wpapi can be processed like this. Beware this is not always a best practice if you don’t use correct permissions and environment variables settings.

var WP = require( ‘wordpress-rest-api’ );
var _ = require( ‘lodash’ );

var config = _.pick( require( ‘./config’ ).wordpress, [
// Whitelist valid config keys
‘username’,
‘password’,
‘endpoint’
]);

var wp = new WP( config );

module.exports = wp;

View the code on Gist.

? Site Content Accumulation

And finally, you are able to consume all the content by creating a content service which handles recursively fetching:

All the pages of a paged collection.
Your WordPress site’s info.
An alphabetized list of categories.
A specific category (specified by slug) from the content cache.
An alphabetized list of tags.
A specific tag (specified by slug) from the content cache
Other content required to have some feature parity with WP.

The code for this looks somewhat like this.

‘use strict’;

var wp = require( ‘./wp’ );
var cache = require( ‘./content-cache’ );
var _ = require( ‘lodash’ );
var RSVP = require( ‘rsvp’ );

/**
* Recursively fetch all pages of a paged collection
*
* @param {Promise} request A promise to a WP API request’s response
* @returns {Array} A promise to an array of all matching records
*/
function all( request ) {
return request.then(function( response ) {
if ( ! response._paging || ! response._paging.next ) {
return response;
}
// Request the next page and return both responses as one collection
return RSVP.all([
response,
all( response._paging.next )
]).then(function( responses ) {
return _.flatten( responses );
});
});
}

function siteInfo( prop ) {
var siteInfoPromise = cache.get( ‘site-info’ );

if ( ! siteInfoPromise ) {
// Instantiate, request and cache the promise
siteInfoPromise = wp.root( ‘/’ ).then(function( info ) {
return info;
});
cache.set( ‘site-info’, siteInfoPromise );
}

// Return the requested property
return siteInfoPromise.then(function( info ) {
return prop ? info[ prop ] : info;
});
}

/**
* Get an alphabetized list of categories
*
* All archive routes display a sorted list of categories in their sidebar.
* We generate that list here to ensure the sorting logic isn’t duplicated
* across routes.
*
* @method sortedCategories
* @return {Array} An array of category objects
*/
function sortedCategories() {
return all( wp.categories() ).then(function( categories ) {
return _.chain( categories )
.sortBy( ‘slug’ )
.value();
});
}

function sortedCategoriesCached() {
var categoriesPromise = cache.get( ‘sorted-categories’ );

if ( ! categoriesPromise ) {
categoriesPromise = sortedCategories();
cache.set( ‘sorted-categories’, categoriesPromise );
}

return categoriesPromise;
}

/**
* Get a specific category (specified by slug) from the content cache
*
* The WP API doesn’t currently support filtering taxonomy term collections,
* so we have to request all categories and filter them down if we want to get
* an individual term.
*
* To make this request more efficient, it uses sortedCategoriesCached.
*
* @method categoryCached
* @param {String} slug The slug of a category
* @return {Promise} A promise to the category with the provided slug
*/
function categoryCached( slug ) {
return sortedCategoriesCached().then(function( categories ) {
return _.findWhere( categories, {
slug: slug
});
});
}

/**
* Get a specific tag (specified by slug) from the content cache
*
* The WP API doesn’t currently support filtering taxonomy term collections,
* so we have to request all tags and filter them down if we want to get an
* individual term.
*
* To make this request more efficient, it uses the cached sortedTags promise.
*
* @method tagCached
* @param {String} slug The slug of a tag
* @return {Promise} A promise to the tag with the provided slug
*/
function tagCached( slug ) {
return sortedTagsCached().then(function( tags ) {
return _.findWhere( tags, {
slug: slug
});
});
}

/**
* Get an alphabetized list of tags
*
* @method sortedTags
* @return {Array} An array of tag objects
*/
function sortedTags() {
return all( wp.tags() ).then(function( tags ) {
return _.chain( tags )
.sortBy( ‘slug’ )
.value();
});
}

function sortedTagsCached() {
var tagsPromise = cache.get( ‘sorted-tags’ );

if ( ! tagsPromise ) {
tagsPromise = sortedTags();
cache.set( ‘sorted-tags’, tagsPromise );
}

return tagsPromise;
}

function getSidebarContent() {
return RSVP.hash({
categories: sortedCategoriesCached(),
tags: sortedTagsCached()
});
}

module.exports = {
// Recursively page through a collection to retrieve all matching items
all: all,
// Get (and cache) the top-level information about a site, returning the
// value corresponding to the provided key
siteInfo: siteInfo,
sortedCategories: sortedCategories,
sortedCategoriesCached: sortedCategoriesCached,
categoryCached: categoryCached,
tagCached: tagCached,
sortedTags: sortedTags,
sortedTagsCached: sortedTagsCached,
getSidebarContent: getSidebarContent
};

View the code on Gist.

? Custom Routes & Sales Data

Finally, I have cooked in quite a few custom routes from where I can attain any kind of sales related data. For the particular architecture I have in place, I’m again using the RSVP.hash utility for convenience and parallelism. It works like a charm.

The post Build Custom Dashboards with MongoDB, Azure & Serverless Functions appeared first on SitePoint.