Nuix SDK – Ingesting from Microsoft Office 365

Getting Started: Microsoft Azure Tenant Configuration

Wed, 21 Jul 2021 00:00:00 +0000

The Nuix connector for Microsoft Office 365 utilizes the Microsoft Graph API to authenticate and interact with the Office 365 suite of applications. Authentication is facilitated by creating an Azure application with appropriate permissions on the Azure Active Directory (AD) tenant where Microsoft Office 365 is hosted.

Important

The Azure application described in the following sections must be registered and approved by Microsoft before the Nuix connector endpoint can be used. Details about receiving this protected API access approval is detailed later within this tutorial.

Microsoft Graph Authentication Methods

The following authentication methods are implemented within this topic:

OAuth 2.0 client credentials grant flow.
- Used for certificate and client secret-based authentication.
- Learn more: https://docs.microsoft.com/en-us/graph/auth-v2-service#4-get-an-access-token
OAuth 2.0 Resource Owner Password Credentials (ROPC) grant flow.
- Used to retrieve Teams calendar data.
- Learn more: https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth-ropc

Use the steps in the following sections to create and register an Azure application with the appropriate permissions to provide a method of authentication for your Azure tenant.

Registering a New Application with Azure

Use the following procedure to register and configure an Azure application that can be used by Nuix to authenticate with your Microsoft Office 365 Instance.

Important

This procedure must be performed by a user within your Azure tenant that has Global Administrator privileges.

Access the Microsoft Azure portal (https://portal.azure.com).
Log into the portal using an account that has been assigned the role of Global Administrator for your tenant.
If your account is associated with multiple tenants, switch to the Active Directory tenant where Microsoft Office 365 is configured. To do this:
1. Click on your account name from the upper right side of the Azure portal.
2. Click Switch directory from the pane.
3. Choose an available directory from the menu.
From the Azure portal Home page, select Azure Active Directory from the Azure services menu.
From the left navigation panel, select App registrations.
Click New registration from the top navigation bar within the view.
Provide the following details to register the application.
1. A descriptive name for the application. For example: Nuix Authentication
2. Under Supported account types, select Accounts in this organizational directory only (Single tenant).
3. Click Register to register the new app.
After registering the application, the Overview page displays. Copy the following displayed values for later use:
- Application (client) ID
- Directory (tenant) ID
Proceed to the following section to set up authentication.

Configuring an Authentication Method

After successfully registering your Azure application, an authentication method must be configured. The method that is chosen will determine how the Nuix connector authenticates with Microsoft Office 365.

Use the following procedure to configure an authentication method:

From the Overview page of your Azure application, select Certificates & secrets from the left navigation panel.

Tip
To access the Overview page from the Azure portal Home page, select Azure Active Directory from the Azure services list, then App registrations from the left navigation panel. Finally, click on the name of the new application to access its Overview page.

Two types of authentication are available for Azure Active Directory applications:
- Certificate-based authentication
- Password-based authentication (application secret)
Both methods are supported by the Nuix Connector for Microsoft Office 365 and provide the same level of access to the Azure tenant.
Implement one of the methods using the procedures in the following sections.

Certificate-Based Authentication

To configure certificate-based authentication, the selected certificate must meet the following requirements:

PKCS#12 based certificate store.
- Public key (.CER, .PEM, or .CRT) Applied within Azure.
- Private key (.PFX or .P12) Applied within the Nuix Connector endpoint.
RSA key algorithm
2048 key size

Note

The certificate store being used must not include more than one key value pair.

Use the following procedure to configure certificate-based authentication within your Azure tenant:

From the Certificates & secrets panel, locate the Certificates section and click Upload certificate.

Use the file explorer to select a valid certificate (public key) with either a .CER, .PEM, or .CRT file extension.

Note
Certificates should be obtained and signed by a trusted Certificate Authority (CA). If necessary, self-signed certificates can be used for testing purposes only.
Click Add to upload and apply the certificate.
Proceed to the next topic to apply API permissions to the Azure application.

Password-Based Authentication

Use the following procedure to configure password-based authentication within your Azure tenant.

From the Certificates & secrets panel, locate the Client secrets section and click New client secret.
Provide a description for the new client secret.
Select an expiration period for the client secret.

Note
When the client secret expires, this process must be repeated to create a new client secret for your application.
Click Add to generate the new client secret.
Copy the generated value to a secure location. The client secret is required when making requests to the Office 365 connector endpoint.

Warning
The client secret value is only visible at the time of creation. If lost, a new secret key must be generated for the application.
Proceed to the next topic to apply API permissions to the Azure application.

Getting Started: Applying Permissions to the Application

Wed, 21 Jul 2021 00:00:00 +0000

Once you have registered and configured an authentication method for your Azure application, API permissions must be assigned to the application to allow access to specific data types within Microsoft Office 365.

Perform the following steps to apply the required API permissions to the application.

Access the Overview page of your newly created application.

Tip
To access this page from the Azure portal Home page, select Azure Active Directory from the home page then App registrations from the left navigation panel. Finally, click on the name of the new application to access its Overview page.
Select API permissions from the left navigation panel and then click Add a permission.
From the list of Commonly used Microsoft APIs, select Microsoft Graph.
Click Delegated permissions and then select the following permission:

API/Permission Name	Type	Description
`Chat.Read`	Delegated	Read user chat messages
`Group.Read.All`	Delegated	Read all groups
`Mail.Read`	Delegated	Read user mail
`User.Read.All`	Delegated	Read all users' full profiles

Return to the top of the page and click Application permissions to change the permission type, then select the following permissions:

API/Permission name	Type	Description
`Calendars.Read`	Application	Read Calendars in all mailboxes
`Channel.ReadBasic.All`	Application	Read the names and descriptions of all channels
`ChannelMember.Read.All`	Application	Read the members of all channels
`ChannelMessage.Read.All`	Application	Read the chat messages from all channels
`ChannelSettings.Read.All`	Application	Read the names, descriptions, and settings of all channels
`Chat.Read.All`	Application	Read all chat messages
`Contacts.Read`	Application	Read contacts in all mailboxes
`Directory.Read.All`	Application	Read directory data
`Files.Read.All`	Application	Read files in all site collections
`Group.Read.All`	Application	Read all groups
`Mail.Read`	Application	Read mail in all mailboxes
`OrgContact.Read.All`	Application	Read organizational contacts
`Sites.Read.All`	Application	Read items in all site collections (preview)
`User.Read.All`	Application	Read all users' full profiles

After all the specified API permissions have been selected, click Add permissions from the bottom of the view to apply the selections.
The newly added API permissions are initially displayed with a Not Granted status. Click Grant admin consent for <AD Tenant name> to update the status.

Note
By granting admin consent, the Service Principal account for the selected AD tenant is granted access to the selected API permissions. The defined application permissions provide access to specific data contained within Microsoft Teams, Exchange, and SharePoint.
Proceed to the next topic to request access to the protected APIs.

Getting Started: Requesting Access to Protected APIs

Wed, 21 Jul 2021 00:00:00 +0000

Some of the selected API permissions required by the newly registered Azure application provide access to sensitive data within the tenant. Microsoft considers these protected APIs. To use these APIs with the registered Azure application, additional validation and approval is required by Microsoft.

To request access to protected Microsoft Graph APIs, visit the following site and complete the form provided by Microsoft.

https://aka.ms/teamsgraph/requestaccess

The form includes several questions concerning the intended use of the protected APIs. The following table includes quick answers to the required questions included on the form.

Question	Answer
Publisher name	Enter the name of your company or organization.
App name	Enter the name of the Azure application that you registered. For example: Nuix Authentication
App ID	Provide the Client ID of the registered Azure application.
What does your app do?	The application provides a method of authentication that allows chat data to be retrieved from Microsoft Teams.
Why does your app need access to all messages?	The application is being used to retrieve data for eDiscovery.
Data retention	Select: “It is obvious to any admin installing this app that it will make a copy of Microsoft Teams messages.”
What are the tenant ID’s that this app needs to run in?	Provide the AD tenant ID where the Azure application was registered.
Does your organization own all of the specified tenants?	Provide an answer based on the response to the previous question.

After completing all fields, click Next to submit the form.

Microsoft reviews access requests every Wednesday and deploys approvals every Friday, except during major holiday weeks in the U.S. Submissions during those weeks will be processed the following non-holiday week. For additional information, see Protected APIs in Microsoft Teams.

Proceed to the next topic to learn how to ingest data from Office 365 using Nuix RESTful Service.

Getting Started: Using the Microsoft Office 365 Endpoint

Mon, 26 Jul 2021 00:00:00 +0000

Before you begin, ensure that your Azure tenant has been configured as described in the previous Microsoft Office 365 topics.

When processing data into a Nuix case, there are many configurable parameters that can be used to tailor what data is retrieved, how it is processed, where it is stored, and what resources should be used to perform the work. The details provided within this topic cover only the endpoint parameters related to retrieving data from Office 365.

For details about all other configurable parameters for the /cases/{caseId}/evidence/ms365 endpoint, view the reference documentation.

Important

To avoid reduced performance and potential data loss resulting from connection throttling, it is recommended to not use more than four workers when ingesting data from Office 365 with this endpoint.

Use the following details to ingest data from the following Microsoft Office 365 applications:

Microsoft Teams
Microsoft Exchange
Microsoft SharePoint

To get started, first create a case then submit a POST request to the /cases/{caseId}/evidence/ms365 endpoint.

The following example performs a basic ingestion from Office 365, retrieving Teams chats and the Exchange mailbox of a single user from a time period of one year.

curl -L -X POST 'http://127.0.0.1:8080/nuix-restful-service/svc/cases/36c83aabe9d74808ac1d47300909b016/evidence/ms365' \
-H 'nuix-auth-token: 974cd40d-a5b8-45b1-936f-9f3745a783be' \
-H 'Content-Type: application/json' \
--data-raw '{
    "processingProfile": "Default",
    "target": {
        "tenantId": "<tenantId>",
        "clientId": "<clientId>",
        "clientSecret": "<clientSecret>",
        "from": "2019-01-01",
        "to": "2021-01-01",
        "username": "user@email.com",
        "password": "pAsSwOrD",
        "userPrincipalNames": [
            "user@email.com"
        ],
        "mailboxRetrievals": [ "MAILBOX"],
        "retrievals": [
            "USERS_CHATS", "USERS_EMAILS"
        ]
    }
}'

{
    "done": true,
    "cancelled": false,
    "result": true,
    "token": "974cd40d-a5b8-45b1-936f-9f3745a783be",
    "functionKey": "58abc476-c7d0-44c5-b71c-4abd2982212f",
    "progress": 0,
    "total": 0,
    "percentComplete": null,
    "updatedOn": 1627314002180,
    "status": null,
    "statusId": null,
    "requestTime": 1627313971596,
    "startTime": 1627313971596,
    "finishTime": 1627314004551,
    "caseId": "36c83aabe9d74808ac1d47300909b016",
    "caseName": "MSOffice365",
    "hasSuccessfullyCompleted": true,
    "friendlyName": "Evidence Ingestion Function",
    "caseLocation": "C:\\ProgramData\\Nuix\\NuixCases\\MSOffice365",
    "requestor": "nuix",
    "licenseShortName": "enterprise-workstation",
    "action": "AsyncBulkIngestionFunction",
    "options": {
        "reloadQuery": null,
        "processorSettings": {
            "processText": null,
            "processLooseFileContents": null,
            "processForensicImages": null,
            "analysisLanguage": null,
            "stopWords": null,
            "stemming": null,
            "enableExactQueries": null,
            "extractNamedEntities": null,
            "extractNamedEntitiesFromText": null,
            "extractNamedEntitiesFromProperties": null,
            "extractNamedEntitiesFromTextStripped": null,
            "extractNamedEntitiesFromTextCommunications": null,
            "extractShingles": null,
            "processTextSummaries": null,
            "calculateSSDeepFuzzyHash": null,
            "calculatePhotoDNARobustHash": null,
            "detectFaces": null,
            "classifyImagesWithDeepLearning": null,
            "imageClassificationModelUrl": null,
            "extractFromSlackSpace": null,
            "carveFileSystemUnallocatedSpace": null,
            "carveUnidentifiedData": null,
            "carvingBlockSize": null,
            "recoverDeletedFiles": null,
            "extractEndOfFileSlackSpace": null,
            "smartProcessRegistry": null,
            "identifyPhysicalFiles": null,
            "createThumbnails": null,
            "skinToneAnalysis": null,
            "calculateAuditedSize": null,
            "storeBinary": null,
            "maxStoredBinarySize": null,
            "maxDigestSize": null,
            "digests": [],
            "addBccToEmailDigests": null,
            "addCommunicationDateToEmailDigests": null,
            "reuseEvidenceStores": null,
            "processFamilyFields": null,
            "hideEmbeddedImmaterialData": null,
            "reportProcessingStatus": null,
            "enableCustomProcessing": [],
            "performOcr": null,
            "ocrProfileName": null,
            "createPrintedImage": null,
            "imagingProfileName": null,
            "exportMetadata": null,
            "metadataExportProfileName": null,
            "workerItemCallback": null,
            "workerItemCallbacks": null,
            "traversalScope": null,
            "namedEntities": null
        },
        "evidence": [
            {
                "guid": null,
                "name": null,
                "customMetadata": null,
                "encoding": null,
                "custodian": null,
                "timeZone": null,
                "description": null,
                "locale": null,
                "files": null,
                "exchangeMailboxes": null,
                "s3Buckets": null,
                "sqlServers": null,
                "oracleServers": null,
                "enterpriseVaults": null,
                "sharepointSites": null,
                "mailStores": null,
                "loadFiles": null,
                "centeraClusters": null,
                "splitFiles": null,
                "dropboxes": null,
                "sshServers": null,
                "twitterLocations": null,
                "documentumServers": null,
                "ms365Locations": [
                    {
                        "type": "microsoft_graph",
                        "username": "************",
                        "password": "************",
                        "tenantId": "<tenantId>",
                        "clientId": "<clientId>",
                        "clientSecret": "************",
                        "certificateStorePath": null,
                        "certificateStorePassword": null,
                        "from": 1546315200000,
                        "to": 1625630400000,
                        "teamNames": null,
                        "userPrincipalNames": [
                            "user@email.com"
                        ],
                        "retrievals": [
                            "USERS_CHATS",
                            "USERS_EMAILS"
                        ],
                        "mailboxRetrievals": [
                            "MAILBOX"
                        ],
                        "versionFilters": null
                    }
                ]
            }
        ],
        "localWorkerCount": 2,
        "repositories": [],
        "parallelProcessingSettings": {
            "workerCount": null,
            "workerMemory": null,
            "workerTemp": null,
            "brokerMemory": null,
            "workerBrokerAddress": null,
            "useRemoteWorkers": false,
            "embedBroker": true
        },
        "rescanEvidenceRepositories": false,
        "loadProcessingJob": {
            "casePath": "C:\\ProgramData\\Nuix\\NuixCases\\MSOffice365",
            "jobGuid": "b9f4933c-96b5-4bc1-9d48-23f1046da79e",
            "processingMode": "Load",
            "startDate": 1627313976482,
            "workerCount": 2,
            "finished": true,
            "paused": false,
            "masterAddress": "192.168.56.1",
            "bytesProcessed": 0,
            "itemsProcessed": 2,
            "jobSizeTotalBytes": 0
        }
    },
    "participatingInCaseFunctionQueue": true,
    "processedBy": "72670a5e-f0e1-4046-8b2e-22a42fe2ac9a",
    "errorMsg": null
}

A complete list of target parameters for this endpoint are described below.

Credential Parameters

The following parameters are used to authenticate with the Microsoft Azure tenant.

Parameter	Required	Description
`tenantId`	Yes	The ID of the Azure Active Directory (AAD) tenant where the Azure authentication application is registered.
`clientId`	Yes	The application ID of the registered authentication application.
`clientSecret`	Yes	The authentication key string associated with the Application (Client) ID. Note: Use this parameter if not authenticating with a private key certificate.
`certificateStorePath`	Yes	The location of a PKCS#12 based private key certificate. Note: Use this parameter if not authenticating with a client secret.
`certificateStorePassword`	Yes	The password associated with the PKCS#12 based private key certificate.
`username`	No	The username of a Microsoft Teams user to retrieve calendar data from. Note: Required only if the `TEAMS_CALENDARS` retrieval parameter has been defined.
`password`	No	The password associated with a Microsoft Teams user.

Scoping Parameters

The following parameters can be used to narrow the scope of data ingested into your case.

parameter	Required	Description
`from`	Yes	Retrieve only the items with a Last Modified date on or after the specified date. If a Last Modified date does not exist, the item’s Creation Date will be used.
`to`	Yes	Retrieve only the items with a Last Modified date on or before the specified date.
`teamNames`	No	Scope the ingestion to specific Microsoft Teams by specifying a semicolon separated list of Team names.
`userPrincipalNames`	No	Scope the ingestion to specific users by specifying a semicolon separated list of Office 365 User Principal Names. Note: A User Principal Name (UPN) is the name of a Windows Active Directory system user in the format of an email address. For example: john.doe@domain.com.

The from and to date filters apply only to the following Office 365 item types:

Exchange mail messages
Teams chat messages
Calendar events (filtered by scheduled event start time)
Resources from SharePoint and OneDrive for Business

Note

If retrieving calendar data (USERS_CALENDARS or TEAMS_CALENDARS), the date rage must not exceed five years (1825 days).

Retrieval Parameters

The following retrieval parameter options are available to specify which Office 365 data is collected.

Retrievals	Service	Retrieves…
`TEAMS_CHANNELS`	Teams	Channel data such as chat messages and attachments from all public channels within a team. Note: User emoji reactions to chat messages are included as child items of the parent message.
`TEAMS_CALENDARS`	Teams	Teams calendar data from all teams that a specific user is a member of. Note: If selected, the `username` and `password` of a Microsoft Teams user must be provided in order for their calendar data to be retrieved.
`USERS_CHATS`	Teams	Chat messages from one-on-one or group conversations that take place outside of a public channel. Note: See Private Chat Limitations for additional information on how user chats are retrieved.
`USERS_CALENDARS`	Teams	Individual user calendar data from the members of a team.
`USERS_CONTACTS`	Exchange	Personal contacts that have been saved by individual users.
`USERS_EMAILS`	Exchange	Individual user mailboxes. Tip: Select Extract from mailbox slack space when configuring your data processing settings to also retrieve deleted and other lower-level slack space items from the mailbox.
`ORG_CONTACTS`	Exchange	Contacts created by an administrator that are shared to all users in an organization. Also known as Mail Contacts within Exchange.
`SHAREPOINT`	SharePoint	Data from SharePoint sites, subsites, lists, and users. Note: The retrieval of SharePoint list attachments is not supported.

Private Chat Limitations

Due to restrictions in how private chat data is retrieved using the Microsoft Graph API, certain limitations exist when determining the recipients of specific messages within a chat. Limitations include:

Chat/conversation IDs are only generated when a new chat is initiated. The generated ID is persisted for the entire life of the chat, regardless of whether participants are later added or removed.
The To communication metadata property, which identifies the participants of a chat, represents only the list of participants that are included in the chat at the time of ingestion. Due to this limitation, the following must be considered:
- New participants who are added to an existing conversation are identified within the metadata as a chat recipient. However, based on the Microsoft Teams chat history settings selected when adding the new participant, they may or may not have been granted access to view messages that were sent prior to them joining the chat.
- Participants who left a chat after receiving messages will not be identified within the metadata as a recipient at all.

Workaround:

To establish a complete view of recipients and verify who received messages from a private chat conversation, the specified private conversation must be examined from the /Users/<userName>/Chats directory of each suspected participant. For example:

To verify if John saw a specific private chat message, the messages located within /Users/John/Chats must be examined. Looking at the same chat but from a different participant, /Users/Jane/Chats, may not properly confirm if John saw a specific message because Jane’s chat history may be incomplete and reflect only a portion of the conversation due to the limitations previously noted.

Mailbox Retrieval Parameters

The following mailbox retrieval parameter options are available to specify which Exchange mailbox data collected.

Mailbox Retrievals	Required	Retrieves…
`MAILBOX`	No	All current mailbox data, including recoverables such as deletions and purges.
`DELETIONS`	No	Items that have been deleted from any folder and placed in the Deleted items default folder.
`RECOVERABLE_ITEMS`	No	Items that have been deleted from the Deleted items default folder.
`PURGES`	No	Deleted items that have been marked to be removed from the mailbox database.
`ARCHIVE`	No	All archived mailbox data, including archived recoverables such as deletions and purges.
`ARCHIVE_DELETIONS`	No	Archived items that have been deleted from any folder and placed in the Deleted items default folder.
`ARCHIVE_RECOVERABLE_ITEMS`	No	Archived items that have been deleted from the Deleted items default folder.
`ARCHIVE_PURGES`	No	Archived deleted items that have been marked to be removed from the mailbox database.
`PUBLIC_FOLDERS`	No	Shared folders within an organization.

Version Filter Parameters

The following version filter parameters can be configured to also include item versions during ingestion. With these parameters, you can choose to ingest only the latest version of a item, a selection of recent versions, or all available versions.

Note

Only item versions with a Last Modified date within the specified from / to date range will be retrieved.

Version filters apply to all files and attachments retrieved from Microsoft Office 365 in which versions are maintained, including data from SharePoint, OneDrive for Business, and Teams.

If the following parameters are left undefined only the latest version of a file or attachment will be retrieved.

Version Filters	Required	Description
`versionRetrievalEnabled`	No	Set to `true` to retrieve all versions of an item. Set to `false` (default) to retrieve only the latest version of an item.
`versionRetirevalLimit`	No	Specify an integer value to limit the number of retrieved versions to a specific number of recent versions. Requires `versionRetrievalEnabled` be set to `true`. A value of `-1` will retrieve all available versions.

Version Metadata

When item versions are included in a case, the following version related metadata is applied to all applicable items:

Metadata	Value	Description
`Flag: versioned`	boolean	Indicates that multiple versions of the selected item exist and have been ingested.
`Flag: current_version`	boolean	Indicates if an item is the most recent version. Determined at the time of ingestion.
`Property: Version Age`	integer	Identifies the version of the selected item in relation to the latest known version. A value of 0 represents the latest version. Each increment of 1 indicates a subsequent older version. Warning: If the data within a case is updated by reloading the source data or scanning for new child items, the Version Age of an item will change if newer versions of that item are found.
`Property: Version Group ID`	string	A unique identifier that is used to link all versioned items that originate from the same item. The Version Group ID value is derived from the original item’s ID value.
`Property: Version Value`	integer	Identifies the exact version of the selected item based on source data. Note: Microsoft 365 uses a basic sequential numbering method (1.0, 2.0, 3.0) when assigning version values. Other data sources may assign non-readable values, such as a hash, when defining this value.

Querying Items Based on Version Metadata

The following table contains example queries that can be used after ingestion to locate versioned items based on their metadata:

Query	Returns…
`Flag: versioned`	All items that have multiple versions.
`flag:(versioned AND current_version)`	The latest version of each item that has multiple versions.
`flag:(versioned AND NOT current_version)`	All versions of an item except for the latest version.
`integer-properties:"Version Age":(0 to 10)`	The most recent version (indicated by 0) and the previous 10 versions.
`properties:"Version Value:2.0"`	All items that have a version value that matches the value specified in the query (2.0).
`properties:"Version Group ID:<GROUPIDGUID>"`	All versions of an item, including the original, that match the group ID specified in the query.

Getting Started: Microsoft Office 365 Data Overview

Mon, 26 Jul 2021 00:00:00 +0000

Once ingested into a Nuix case, the data from Microsoft Office 365 is organized in a logical hierarchical structure that provides a clear view of the information.

Within the user-defined evidence container, a Microsoft Graph directory is generated for each Microsoft Office 365 connection. This directory contains all data ingested from the Azure tenant organized into the following folders:

Teams
Users
Organizational Contacts
SharePoint

The following diagram illustrates the typical data structure that is created when performing an ingestion using the Nuix Connector for Microsoft Office 365 endpoint.

+Microsoft Graph
|  +--Teams
|  |  +--TeamX
|  |  |  +--Channels
|  |  |  |  +--ChannelA
|  |  |  |  |  +--Chat Conversation
|  |  |  |  |  |  +--Chat message
|  |  |  |  |  |  |  |--Chat message attachment
|  |  |  |  |  |  +--Chat message
|  |  |  |  +--ChannelB
|  |  |  +--Calendars
|  |  |  |  +--Calendar event
|  |  |  |  |  |--Calendar event attachment
|  |  +--TeamY
|  |
|  +--Users
|  |  +--User1
|  |  |  +--Calendars
|  |  |  |  +--Calendar event
|  |  |  |  |  |--Calendar event attachment
|  |  |  +--Chats
|  |  |  |  +--Chat conversation
|  |  |  |  |  +--Chat message
|  |  |  |  |  |  |--Chat message attachment
|  |  |  +--Contacts
|  |  |  |  +--Contact or folder
|  |  |  |  |  |--Contact
|  |  |  +--Exchange
|  |  |  |  |--Recoverable items
|  |  |  |  +--Top of information store
|  |  |  |  |  +--Mail folder
|  |  |  |  |  |  +--Sub mail folder
|  |  |  |  |  |  |  |--Recursive containing more sub-folders and/or emails
|  |  |  |  |  |--Email
|  |  +--User2
|  |
|  +--Organizational Contacts
|  |  |--Org contact
|  |
|  +SharePoint
|  |  +--Site1
|  |  |  |--Drive1
|  |  |  +--List1
|  |  |  |  |--List item1
|  |  |  +--SubSite1
|  |  |  |  +--Drive2
|  |  |  |  |  +--Folder1
|  |  |  |  |  |  |--Item1
|  |  |  |  |  |--Item2
|  |  |  |  |--List2
|  |  |  |  |--SubSite2
|  |  +--Site2

As with all data that is ingested using the Nuix Engine, the contents of the evidence set is analyzed to ensure that all content as well as available metadata is extracted into the case.

The following table identifies the Mime-types that are used to classify Microsoft Office 365 data.

Mime Type Kind	Mime Type	Description
container	application/vnd.microsoft.m365.account	Microsoft Office 365 Tenant/Account
container	application/vnd.microsoft.m365.contact-folder	Contact folder
container	application/vnd.microsoft.m365.exchange.folder	Exchange mail folder
container	application/vnd.microsoft.m365.folder	Nuix generated folder used to organize data retrieved from Office 365
container	application/vnd.microsoft.m365.sharepoint.drive	SharePoint drive
container	application/vnd.microsoft.m365.sharepoint.drive.folder	SharePoint drive folder
container	application/vnd.microsoft.m365.sharepoint.list	SharePoint list
container	application/vnd.microsoft.m365.sharepoint.site	SharePoint site or subsite
container	application/vnd.microsoft.m365.team	Team within Teams
container	application/vnd.microsoft.m365.teams.channel	Channel within a team
container	application/vnd.microsoft.m365.user	Office 365 user
chat-conversation	application/vnd.microsoft.m365.teams.channel-conversation	Conversation or thread within a channel
chat-conversation	application/vnd.microsoft.m365.teams.chat-conversation	Conversation within private chats
chat-message	application/vnd.microsoft.m365.message.reaction	Reaction to a message
chat-message	application/vnd.microsoft.m365.teams.message	Deprecated in v9.0. Message within a chat or channel
chat-message	application/vnd.microsoft.m365.teams.channel-message	Message within a channel
chat-message	application/vnd.microsoft.m365.teams.chat-message	Message within a chat
calendar	application/vnd.microsoft.m365.calendar-event	Team calendar event
contact	application/vnd.microsoft.m365.contact	Contact
other-document	application/vnd.microsoft.m365.sharepoint.item	SharePoint item (such as a list)
other-document	application/vnd.microsoft.m365.teams.unknown-attachment	Unknown message attachment type
other-document	application/vnd.microsoft.teams.card.o365connector	An Office 365 card added to a channel (RSS feed as an example)

Getting Started: Troubleshooting Microsoft Office 365 Errors

Mon, 26 Jul 2021 00:00:00 +0000

This topic provides details for troubleshooting known errors that result in inaccessible items (mime-type:"filesystem/inaccessible“) that may be encountered when processing data.

ERROR: com.nuix.data.util.RetryCallable - Final retry attempt number 10 failed

Description: The request to retrieve data from Office 365 has failed. This error is likely to occur after a connection has been throttled and the maximum number of retries has been reached.
Impact: Data will be missing from the evidence set.
Resolution: Retry the ingestion once the connection is no longer being throttled.
Important: To reduce the potential for data loss resulting from connection throttling, it is recommended to not use more than four workers when ingesting data from Office 365.

To try and prevent future failures of this type, the following system properties can also be configured at startup:

Increase the default (10) maximum retry count:
nuix.data.msgraph.retries=10
Increase the default (1 second) retry delay:
nuix.data.msgraph.retryDelay=1
Increase the default (5 seconds) retry delay increment:
nuix.data.msgraph.retryDelayIncrement=5

Errors: com.nuix.data.network.microsoftgraph.client.exception.MsGraphMailboxItemMissingException: The user mailbox does not contain the requested item. Missing id: ArchiveMsgFolderRoot; com.nuix.data.network.microsoftgraph.client.exception.MsGraphMailboxItemMissingException: The user mailbox does not contain the requested item. Missing id: ArchiveRecoverableItemsRoot

Description: A request was made to retrieve a mailbox that does not exist for a user. For example: Requesting to ingest archived mailboxes from a user with no archives.
Impact: Result set includes Inaccessible Items.

Error: MsGraphApiException: Failed to retrieve data from Microsoft Graph API: Access is denied. Check credentials and try again.

Description: Indicates that an error with code “ErrorAccessDenied” was returned from the Microsoft Graph API for this request.
Impact: Data is inaccessible, potentially due to a missing application permission or invalid credentials.
Resolution: Verify that the credentials being used are correct and review Applying Permissions to the Application to ensure that all required permissions are applied within your Azure tenant.

Error: RuntimeException: REST API is not yet supported for this mailbox.

Description: Indicates that the user associated with a particular mailbox has not been issued a license.

Error: java.net.SocketException: Connection reset by peer: socket write error.

Description: Indicates that long-lived connections are being closed, potentially due to firewall configurations.
Impact: Attachments are missing when retrieving Teams messages.
Resolution: Set the following system property at startup to allow the embedded Apache HTTPClient to accept additional Java system properties, such as proxy settings.

nuix.data.msgraph.http.use-system-props=true

See HttpClientConfiguration for details on configuring the Apache HTTPClient.

Nuix SDK – Ingesting from Microsoft Office 365

Getting Started: Microsoft Azure Tenant Configuration

Important

Microsoft Graph Authentication Methods

Registering a New Application with Azure

Important

Configuring an Authentication Method

Tip

Certificate-Based Authentication

Note

Note

Password-Based Authentication

Note

Warning

Getting Started: Applying Permissions to the Application

Tip

Note

Getting Started: Requesting Access to Protected APIs

Getting Started: Using the Microsoft Office 365 Endpoint

Important

Credential Parameters

Scoping Parameters

Note

Retrieval Parameters

Private Chat Limitations

Mailbox Retrieval Parameters

Version Filter Parameters

Note

Version Metadata

Querying Items Based on Version Metadata

Getting Started: Microsoft Office 365 Data Overview

Getting Started: Troubleshooting Microsoft Office 365 Errors