Merge pull request #1 from limit-zero/elastic

Elasticsearch Pagination

README.md

@@ -1,13 +1,15 @@
-# GraphQL Mongoose Cursor Pagination
-Adds support for Relay-like cursor pagination with Mongoose models/documents. In addition, this library also provides type-ahead (autocomplete) functionality (with pagination) using MongoDB regular expression queries.
+# GraphQL Mongoose Cursor Pagination with Elasticsearch Support
+Adds support for Relay-like cursor pagination with Mongoose models/documents. This library also provides type-ahead (autocomplete) functionality (with pagination) using MongoDB regular expression queries. In addition, you can optionally paginate hydrated Mongoose models from Elasticsearch.
 
 ## Install
 `yarn add @limit0/mongoose-graphql-pagination`
 
 ## Usage
-Pagination and type-ahead support are available via the `Pagination` and `TypeAhead` classes, respectively. Both classes should be considered, "single use," and should be instantiated once per query or request.
+Pagination, type-ahead, and Elastic+Mongoose pagination support are available via the `Pagination`, `TypeAhead` and `SearchPagination` classes, respectively. **All classes should be considered, "single use," and should be instantiated once per query or request.**
 
 ### Pagination
+Returns paginated results from MongoDB for the provided Mongoose model.
+
  To begin using, require the class
 ```js
 const { Pagination } = require('@limit0/mongoose-graphql-pagination');
@@ -48,7 +50,7 @@ Returns the total number of documents that match the query criteria, regardless
 Returns the edges (Mongoose document nodes) that match the query criteria. The results will be limited by the `first` value.
 
 #### getEndCursor()
-Returns the cursor value (non-obfuscated) of the last edge that matches the current criteria. This value will resolve to the Mongoose document ID, and can be converted into an obfuscated ID when returned by the GraphQL server. Will return `null` if no additional documents remain.
+Returns the cursor value (non-obfuscated) of the last edge that matches the current criteria. This value will resolve to the Mongoose document ID, and can be converted into an obfuscated ID when returned by the GraphQL server. Will return `null` if no results were found.
 
 #### hasNextPage()
 Determines if another page of edges is available, based on the current criteria.
@@ -104,6 +106,58 @@ const paginated = typeahead.paginate(YourModel, { first: 25 });
 const edges = paginated.getEdges();
 ```
 
+### Search Pagination
+Returns paginated and hydrated Mongoose/MongoDB results from Elasticsearch. This assumes that you are, in some fashion, saving _some_ data for a specific Mongoose model within Elasticsearch, as this will attempt to convert the Elastic results back into Mongoose documents from MongoDB. This also assumes that the `_id` value of the indexed document in Elasticsearch matches the `_id` value that is stored in MongoDB.
+
+ To begin using, require the class
+```js
+const { SearchPagination } = require('@limit0/mongoose-graphql-pagination');
+```
+Use the class constructor to configure the settings for the paginated query.
+
+#### constructor(Model, client, { params = {}, pagination = {} })
+`Model`: The Mongoose model instance to use for re-querying MongoDB to hydrate the Elasticsearch results. _Required._
+
+`client`: The Elasticsearch client instance created from the `elasticsearch` Node/JS API library. _Required._
+
+`params`: The elastic [search parameters](https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/api-reference.html#api-search) that will be sent to `client.search`. For example: `{ index: 'some-index', type: 'some-type', body: { query: { match_all: {} } } }` _Required_.
+
+`pagination`: The pagination parameters object. Can accept a `first` and/or `after` property. The `first` value specifies the limit/page size. The `after` value specifies the cursor to start at when paginating. For example: `{ first: 50, after: 'some-cursor-value' }` would return the first 50 edges after the provided cursor. By default the results will be limited to 10 edges. Optional.
+
+Complete example:
+```js
+const { SearchPagination } = require('@limit0/mongoose-graphql-pagination');
+const elasticClient = require('./your-elasticsearch-client');
+const YourModel = require('./your-model');
+
+const params = {
+  index: 'your-index',
+  type: 'your-type',
+  body: {
+    query: { match_all: {} },
+  },
+};
+const pagination = { first: 25 };
+
+const paginated = new SearchPagination(YourModel, elasticClient, { params, pagination });
+// Retrieve the edges...
+const edges = paginated.getEdges();
+```
+
+Once the instance is created, use the methods listed below to access the paginated info.
+
+#### getTotalCount()
+Returns the total number of documents that match the search criteria, regardless of page size (limit).
+
+#### getEdges()
+Returns the edges (hydrated Mongoose document nodes from Elasticsearch hits) that match the search criteria. The results will be limited by the `first` value.
+
+#### getEndCursor()
+Returns the cursor value (non-obfuscated) of the last edge that matches the current criteria. This value will resolve to a JSON stringified version of the Elasticsearch `sort` value, and can be converted into an obfuscated ID when returned by the GraphQL server. Will return `null` if no results were found.
+
+#### hasNextPage()
+Determines if another page of edges is available, based on the current criteria.
+
 ### Pagination Resolvers
 Finally, the `paginationResolvers` provide helper functions for using Pagination with a GraphQL server such as Apollo.
 
@@ -113,6 +167,7 @@ scalar Cursor
 
 type Query {
   allContacts(pagination: PaginationInput = {}, sort: ContactSortInput = {}): ContactConnection!
+  searchContacts(pagination: PaginationInput = {}, phrase: String!): ContactConnection!
 }
 
 type ContactConnection {
@@ -157,24 +212,45 @@ enum ContactSortField {
 The following resolvers could be applied:
 ```js
 const { CursorType } = require('@limit0/graphql-custom-types');
-const { Pagination, paginationResolvers } = require('@limit0/mongoose-graphql-pagination');
+const { Pagination, paginationResolvers, SearchPagination } = require('@limit0/mongoose-graphql-pagination');
+
+const elasticClient = require('./path-to/elasticsearch-client');
 const Contact = require('./path-to/contact-model');
 
 module.exports = {
-  // The cursor type. Will obfuscate the MongoID.
+  // The cursor type.
+  // Will obfuscate the MongoID for `Pagination` and the Elastic sort value for `SearchPagination`
   Cursor: CursorType,
 
-  // Apply the pagination resolvers for the connection and edge.
+  // Apply the pagination resolvers for the connection.
   ContactConnection: paginationResolvers.connection,
-  ContactEdge: paginationResolvers.edge,
 
   Query: {
-    // Use pagination on the query.
+    /**
+     * Use pagination on the query.
+     * Will query MongoDB via Mongoose for the provided model.
+     */
     allContacts: (root, { pagination, sort }) => new Pagination(Contact, { pagination, sort }, {
       // Informs the sort that the `createdAt` field specifies a created date.
       // Will instead use the document ID when sorting.
       sort: { createdField: 'createdAt' },
     }),
+
+    /**
+     * Use search pagination on the query.
+     * Will query Elasticsearch and then hydrate the results from
+     * MongoDB (via Mongoose) for the provided model.
+     */
+    searchContacts: (root, { pagination, phrase }) => {
+      // Set the parameters for the elastic client `search` method.
+      // @see https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/api-reference.html#api-search
+      const params = {
+        index: 'index-where-contacts-exist',
+        type: 'type-where-contacts-exist',
+        body: { query: { match: { name: phrase } } },
+      };
+      return new SearchPagination(Contact, elasticClient, { params, pagination });
+    }),
   },
 };
 ```

package.json

@@ -35,22 +35,25 @@
     "escape-string-regexp": "^1.0.5"
   },
   "optionalDependencies": {
-    "@limit0/graphql-custom-types": "^1.0.1"
+    "@limit0/graphql-custom-types": "^1.0.1",
+    "elasticsearch": "^15.0.0"
   },
   "peerDependencies": {
-    "graphql": "^0.13.2"
+    "mongoose": "^5.0.17"
   },
   "devDependencies": {
     "bluebird": "^3.5.1",
     "chai": "^4.1.2",
     "chai-as-promised": "^7.1.1",
+    "elasticsearch": "^15.0.0",
     "eslint": "^4.19.1",
     "eslint-config-airbnb-base": "^12.1.0",
     "eslint-plugin-import": "^2.11.0",
     "graphql": "^0.13.2",
     "mocha": "^5.1.1",
     "mongoose": "^5.0.17",
-    "nyc": "^11.7.1"
+    "nyc": "^11.7.1",
+    "sinon": "^5.0.7"
   },
   "keywords": [
     "graphql",

src/index.js

@@ -1,5 +1,11 @@
 const Pagination = require('./pagination');
+const ElasticPagination = require('./search-pagination');
 const TypeAhead = require('./type-ahead');
 const paginationResolvers = require('./resolvers');
 
-module.exports = { Pagination, TypeAhead, paginationResolvers };
+module.exports = {
+  Pagination,
+  TypeAhead,
+  ElasticPagination,
+  paginationResolvers,
+};

src/pagination.js

@@ -20,6 +20,8 @@ class Pagination {
    * @param {object} options Additional sort and limit options. See the corresponding classes.
    */
   constructor(Model, { criteria = {}, pagination = {}, sort = {} } = {}, options = {}) {
+    this.promises = {};
+
     // Set the Model to use for querying.
     this.Model = Model;
 
@@ -43,21 +45,35 @@ class Pagination {
    * @return {Promise}
    */
   getTotalCount() {
-    return this.Model.find(this.criteria).comment(this.createComment('getTotalCount')).count();
+    const run = () => this.Model
+      .find(this.criteria)
+      .comment(this.createComment('getTotalCount')).count();
+
+    if (!this.promises.count) {
+      this.promises.count = run();
+    }
+    return this.promises.count;
   }
 
   /**
    * Gets the document edges for the current limit and sort.
    *
    * @return {Promise}
    */
-  async getEdges() {
-    const criteria = await this.getQueryCriteria();
-    return this.Model.find(criteria)
-      .sort(this.sort.value)
-      .limit(this.first.value)
-      .collation(this.sort.collation)
-      .comment(this.createComment('getEdges'));
+  getEdges() {
+    const run = async () => {
+      const criteria = await this.getQueryCriteria();
+      const docs = await this.Model.find(criteria)
+        .sort(this.sort.value)
+        .limit(this.first.value)
+        .collation(this.sort.collation)
+        .comment(this.createComment('getEdges'));
+      return docs.map(doc => ({ node: doc, cursor: doc.id }));
+    };
+    if (!this.promises.edge) {
+      this.promises.edge = run();
+    }
+    return this.promises.edge;
   }
 
   /**
@@ -66,16 +82,16 @@ class Pagination {
    *
    * @return {Promise}
    */
-  async getEndCursor() {
-    const criteria = await this.getQueryCriteria();
-    const doc = await this.Model.findOne(criteria)
-      .sort(this.sort.value)
-      .limit(this.first.value)
-      .skip(this.first.value - 1)
-      .select({ _id: 1 })
-      .collation(this.sort.collation)
-      .comment(this.createComment('getEndCursor'));
-    return doc ? doc.get('id') : null;
+  getEndCursor() {
+    const run = async () => {
+      const edges = await this.getEdges();
+      if (!edges.length) return null;
+      return edges[edges.length - 1].cursor;
+    };
+    if (!this.promises.cursor) {
+      this.promises.cursor = run();
+    }
+    return this.promises.cursor;
   }
 
   /**
@@ -84,14 +100,22 @@ class Pagination {
    * @return {Promise}
    */
   async hasNextPage() {
-    const criteria = await this.getQueryCriteria();
-    const count = await this.Model.find(criteria)
-      .select({ _id: 1 })
-      .sort(this.sort.value)
-      .collation(this.sort.collation)
-      .comment(this.createComment('hasNextPage'))
-      .count();
-    return Boolean(count > this.first.value);
+    const run = async () => {
+      const criteria = await this.getQueryCriteria();
+      const count = await this.Model.find(criteria)
+        .select({ _id: 1 })
+        .skip(this.first.value)
+        .limit(1)
+        .sort(this.sort.value)
+        .collation(this.sort.collation)
+        .comment(this.createComment('hasNextPage'))
+        .count();
+      return Boolean(count);
+    };
+    if (!this.promises.nextPage) {
+      this.promises.nextPage = run();
+    }
+    return this.promises.nextPage;
   }
 
   /**
@@ -100,46 +124,56 @@ class Pagination {
    * @param {object} fields
    * @return {Promise}
    */
-  async findCursorModel(id, fields) {
-    const doc = await this.Model.findOne({ _id: id })
-      .select(fields)
-      .comment(this.createComment('findCursorModel'));
-    if (!doc) throw new Error(`No record found for ID '${id}'`);
-    return doc;
+  findCursorModel(id, fields) {
+    const run = async () => {
+      const doc = await this.Model.findOne({ _id: id })
+        .select(fields)
+        .comment(this.createComment('findCursorModel'));
+      if (!doc) throw new Error(`No record found for ID '${id}'`);
+      return doc;
+    };
+    if (!this.promises.model) {
+      this.promises.model = run();
+    }
+    return this.promises.model;
   }
 
   /**
    * @private
    * @return {Promise}
    */
-  async getQueryCriteria() {
-    if (this.filter) return this.filter;
-
-    const { field, order } = this.sort;
-
-    const filter = deepMerge({}, this.criteria);
-    const limits = {};
-    const ors = [];
-
-    if (this.after) {
-      let doc;
-      const op = order === 1 ? '$gt' : '$lt';
-      if (field === '_id') {
-        // Sort by ID only.
-        doc = await this.findCursorModel(this.after, { _id: 1 });
-        filter._id = { [op]: doc.id };
-      } else {
-        doc = await this.findCursorModel(this.after, { [field]: 1 });
-        limits[op] = doc[field];
-        ors.push({
-          [field]: doc[field],
-          _id: { [op]: doc.id },
-        });
-        filter.$or = [{ [field]: limits }, ...ors];
+  getQueryCriteria() {
+    const run = async () => {
+      const { field, order } = this.sort;
+
+      const filter = deepMerge({}, this.criteria);
+      const limits = {};
+      const ors = [];
+
+      if (this.after) {
+        let doc;
+        const op = order === 1 ? '$gt' : '$lt';
+        if (field === '_id') {
+          // Sort by ID only.
+          doc = await this.findCursorModel(this.after, { _id: 1 });
+          filter._id = { [op]: doc.id };
+        } else {
+          doc = await this.findCursorModel(this.after, { [field]: 1 });
+          limits[op] = doc[field];
+          ors.push({
+            [field]: doc[field],
+            _id: { [op]: doc.id },
+          });
+          filter.$or = [{ [field]: limits }, ...ors];
+        }
       }
+      return filter;
+    };
+
+    if (!this.promises.criteria) {
+      this.promises.criteria = run();
     }
-    this.filter = filter;
-    return this.filter;
+    return this.promises.criteria;
   }
 
   /**