DeriveJS - a Reactive ODM (Object Data Mapper) framework for MongoDB and NodeJS.

About 3 years ago I started working on a personal project of mine - which involved handling thousands of "Bot" entities, with different, rapidly continuously changing properties. I wanted a way to completely separate the data logic, with the actual DB persistence itself - in a way that will be completely transparent to the logic layer, and automatic. I couldn't find a framework that does it the way I wanted and Imagined, so I developed one myself - and thus DeriveJS was born (eventually I haven't finished the original project that spawned the development of the framework - but Derive itself became a project that I keep updating, maintaining, and using for several projects.

"Modifier" characters in the data model

In Derive - you define data models using object literals, allowing you to use special "modifier" characters to set certain characteristics of data properties, for example, this data model, representing a "Spaceship" entity:

{
    _name$: "",
    _TYPE: "",
    captain: null,
    crew: [],
    addCrew: function(crewMember) {
        this.crew.push(crewMember);
    },
    $callback: null,
}

The underscore (_) before name sets it as an index, and the dollar sign ($) at its end sets it as a unique index. The uppercase of the TYPE property sets it as read-only (settable only when initializing). The crewMember argument of the addCrew function will reference another data entity (CrewMember). The dollar sign at the beginning of callback - sets it as a "meta" (or "secret") property - one that is not part of the data schema - its value won't be saved to the db, and it will be used only "locally".

Reactive, transparent, and automatic

Once you pass the data model to Derive - you get a special class, that is already connected to the DB, and associated with an equivalent collection (you specify its name as another argument to the function). That class uses Javascript Proxies - a Javascript mechanism that allows "overriding" certain "native" operations - which, in Derive's case are, the new operator, for creating new object instances, and the assignment operator (=) for setting values of properties. And so, once you create a new instance of your data class: let ship = new Spaceship(); An equivalent document will already be automatically created on the DB. You can also assign values for the indexes from the class constructor (in the order they are defined in the model), so: let ship = new Spaceship("The Final Frontier", "Galaxy"); Will create a new document in the Spaceships collection in the DB, with The Final Frontier as the value of its _name property, and Galaxy as the value for its _TYPE. Once you set a value for a property, e.g.: ship.captain = "Ricard";

The same value will be assigned and saved to the equivalent property in the DB document.

So, data entities defined and initialized using Derive, are completely reactive - you don't need to explicitly call and .save(), or .update() methods.

Bulk requests in timed intervals

The connection and interface with the MongoDB are always done, under the hood, using Mongo's bulk capabilities, by passing queues (arrays) of operations (the last operations queued since the previous request). This is done in set intervals (that can be set when defining the model, with passing an additional argument to Derive's function, and defaults to 0), and run within setImmediate to be run when the Event Loop is relatively "free".

The engine (residing in a classed called SyncManager - each collection has its own SyncManager that mediates between it and the data entity) is smart enough to arrange the pending data operations in an nonconflicting way - for example, updates for a document that was not inserted to the DB yet, will only be sent after the document they belong to is persisted. The mechanism also facilitate Lock mechanisms to avoid race conditions.

Callbacks and hooks

The data classes returned from initializing a model with Derive - has several built-in methods - amongst which are callback functions that run when a certain DB related actions occurs. For example, any time a new document is persisted on the DB, the _inserted method will be called. Other examples are the _isDuplicate and _error methods, and more. You can override this methods and put your own logic in them, either by extending the class, or defining functions with an identical name, in the model object itself. Each object will also have a $_dbEvents EventEmitter, that triggers inserted and updated events. In addition, to listen for specific DB updates, you can use the special assignment with $callback syntax, where you pass an object with a value and a callback function, for example:

myDataObject.someProp = {
    $value: someValue,
    $callback() {
        // This function will be called as soon as someProp will be updated to someValue on the DB
    }
}

Deriving a data model

DeriveJS model classes exposes a function called derive (which the framework is named after) - allowing you to "extend" data models -- this is not the same as extending classes, it is a way to "inherit" a certain data model and add additional (usually more "specific) properties.

For example, if we'd like to define a "sub-type" of our Spaceship model, a Fighter type spaceship that has additional properties such as weapons, and functions such as attack, we can derive from the previously defined Spaceship model:

const FighterShip = Spaceship.derive({
   _TYPE_: "Fighter",
   weapons: [],
   attack: function(target, weapon) {
      console.log (this._name + 'is attacking ' + target + ' using ' + weapon + '.');
   }
});

The FighterShip model includes both the properties of FigtherShip and all of the properties defined on its "super-model" (the Spaceship model). Instances will also always have the value Fighter set as their _TYPE, as we set it as the default value for FighterShip objects. In addition, we have used the "Default Criteria" modifier character (the underscore at the end of _TYPE), to have the {_TYPE:"Fighter"} query be added to all data retrieving methods of FighterShip

Data restoring

Derive data classes have several built-in methods for retrieving data from the equivalent collections: get, getAll, map, and has. There are also methods for performing data "joins" (join, and joinAll).

More...

This was just a review of the framework - for the full documentation and features, see the Github page.

If you find the framework useful and use it in one of your projects - please write me, and I'll also give you a mention - in the readme.