Getting Started: Viewing the Cluster

Thu, 11 Feb 2021 00:00:00 +0000

View Cluster

Once you have your cluster configured, you can view the cluster using the cluster endpoint.

curl --location --request GET 'http://localhost:8080/nuix-restful-service/svc/v1/cluster' \
--header 'nuix-auth-token: 9729a460-eda7-48dc-ba70-d12b3aae3c8d' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \

{
    "nodes": [
        {
            "hostAddress": "192.168.1.160",
            "hostName": "localhost",
            "port": 5601,
            "details": {
                "serverId": "nuix-restful-server1",
                "publicURL": "http://192.168.1.160:8080",
                "name": "Coordinator Node",
                "tags": [
                    "coordinator"
                ],
                "guid": "353a72fc-43ed-394a-b015-aa2caa2d048b"
            }
        },
        {
            "hostAddress": "192.168.1.161",
            "hostName": "localhost",
            "port": 5601,
            "details": {
                "serverId": "nuix-restful-server-2",
                "publicURL": "http://192.168.1.161:8080",
                "name": "Worker Node",
                "tags": [
                    "worker"
                ],
                "guid": "63f2fc6b-b6b9-32be-93db-51f1f01d602e"
            }
        }
    ]
}

Getting Started: Node Steering

Thu, 11 Feb 2021 00:00:00 +0000

Steering Tags

While no action is required to perform tasks in the cluster, it is possible to steer requests to a specific server or a group of servers. This is accomplished by providing an array of server tags specified in the server.tags field for the server in the application.properties file in combination with nodeTags provided on the request itself.

This example will assume we have two servers. The first server is configured on the server side as a coordinator node. All clustered HTTP requests should be sent through a coordinator node. The second server is configured on the server side as a consumer with a server.tags field equal to worker.

Ingestion Steering

As you can see this request differs slightly from a typical file ingestion request. We have added the restClusterSettings object to the request and have specified that a server tagged as a worker should process this request through the use of nodeTags on the request.

Note

We do not need to submit this request directly to the worker server. Instead, we will submit the request to the coordinator and let Nuix REST handle the task distribution internally.

curl --location --request POST 'http://localhost:8080/nuix-restful-service/svc/v1/cases/776a45b03d1841379b1dea524098a33b/evidence/file' \
--header 'nuix-auth-token: 9729a460-eda7-48dc-ba70-d12b3aae3c8d' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data-raw '{
   "processorSettings": {
   	"storeBinary": true
   },
   "target": {
    "path": "/Engines/9.0.0.171/doc"
  },
  "restClusterSettings": {
      "nodeTags": {
          "defaultTags": [ "worker" ]
      }
   }
}
'

{
    "functionKey": "62a120ce-bebe-492c-bbb5-fd68c99bb0db",
    "location": "http://localhost:8080/nuix-restful-service/svc/v1/asyncFunctions/62a120ce-bebe-492c-bbb5-fd68c99bb0db"
}

Ingestion Status

The functionKey field returned from the ingestion endpoint above can be polled for status.

Note

It is important to note that we are polling status from the coordinator node. Nuix REST will handle retrieving status from node that is actually performing the work.

curl --location --request GET 'http://localhost:8080/nuix-restful-service/svc/v1/asyncFunctions/62a120ce-bebe-492c-bbb5-fd68c99bb0db' \
--header 'nuix-auth-token: 9729a460-eda7-48dc-ba70-d12b3aae3c8d'

{
    "done": true,
    "cancelled": false,
    "result": true,
    "token": "eb4afd86-913f-4b52-8781-07d0523a5fd1",
    "functionKey": "62a120ce-bebe-492c-bbb5-fd68c99bb0db",
    "progress": 7581225,
    "total": 0,
    "percentComplete": null,
    "updatedOn": 1613072891973,
    "status": null,
    "statusId": null,
    "requestTime": 1613072849860,
    "startTime": 1613072850818,
    "finishTime": 1613072895964,
    "caseId": "776a45b03d1841379b1dea524098a33b",
    "caseName": "HelloElasticCase",
    "hasSuccessfullyCompleted": true,
    "friendlyName": "Evidence Ingestion Function",
    "caseLocation": "/Cases/HelloElasticCase",
    "requestor": "username",
    "action": "AsyncBulkIngestionFunction",
    "options": {
        "reloadQuery": null,
        "processorSettings": {
            "processText": null,
            "processLooseFileContents": null,
            "processForensicImages": null,
            "analysisLanguage": null,
            "stopWords": null,
            "stemming": null,
            "enableExactQueries": null,
            "extractNamedEntities": null,
            "extractNamedEntitiesFromText": null,
            "extractNamedEntitiesFromProperties": null,
            "extractNamedEntitiesFromTextStripped": null,
            "extractShingles": null,
            "processTextSummaries": null,
            "calculateSSDeepFuzzyHash": null,
            "detectFaces": null,
            "extractFromSlackSpace": null,
            "carveFileSystemUnallocatedSpace": null,
            "carveUnidentifiedData": null,
            "carvingBlockSize": null,
            "recoverDeletedFiles": null,
            "extractEndOfFileSlackSpace": null,
            "smartProcessRegistry": null,
            "identifyPhysicalFiles": null,
            "createThumbnails": null,
            "skinToneAnalysis": null,
            "calculateAuditedSize": null,
            "storeBinary": true,
            "maxStoredBinarySize": null,
            "maxDigestSize": null,
            "digests": [],
            "addBccToEmailDigests": null,
            "addCommunicationDateToEmailDigests": null,
            "reuseEvidenceStores": null,
            "processFamilyFields": null,
            "hideEmbeddedImmaterialData": null,
            "reportProcessingStatus": null,
            "workerItemCallback": null,
            "workerItemCallbacks": null
        },
        "evidence": [
            {
                "guid": null,
                "name": null,
                "customMetadata": null,
                "encoding": null,
                "custodian": null,
                "timeZone": null,
                "description": null,
                "locale": null,
                "files": [
                    {
                        "path": "/Engines/9.0.0.171/doc"
                    }
                ],
                "exchangeMailboxes": null,
                "s3Buckets": null,
                "sqlServers": null,
                "enterpriseVaults": null,
                "sharepointSites": null,
                "mailStores": null,
                "loadFiles": null,
                "centeraClusters": null,
                "splitFiles": null,
                "dropboxes": null,
                "sshServers": null
            }
        ],
        "localWorkerCount": 1,
        "repositories": [],
        "parallelProcessingSettings": {
            "workerCount": null,
            "workerMemory": null,
            "workerTemp": null,
            "brokerMemory": null,
            "workerBrokerAddress": null,
            "useRemoteWorkers": false,
            "embedBroker": true
        },
        "rescanEvidenceRepositories": false,
        "loadProcessingJob": {
            "casePath": "/Cases/HelloElasticCase",
            "jobGuid": "143cae10-1825-433b-8de4-71aa5ac0a722",
            "processingMode": "Load",
            "startDate": 1613072863682,
            "workerCount": 1,
            "finished": true,
            "paused": false,
            "masterAddress": "192.168.86.53",
            "bytesProcessed": 7581225,
            "itemsProcessed": 192,
            "jobSizeTotalBytes": 0
        }
    },
    "participatingInCaseFunctionQueue": true,
    "processedBy": "nuix-restful-server-2",
    "errorMsg": null
}

As you can see from polling the status, the response contains a processedBy field and this task was processed by nuix-restful-server-2.

Congratulations! You have successfully performed your first clustered ingestion and have distributed your workload.

Nuix SDK – Part 15: Clustering

Getting Started: Viewing the Cluster

View Cluster

Getting Started: Node Steering

Steering Tags

Ingestion Steering

Note

Ingestion Status

Note