Which set of tokenizers should you enable?

adminJanuary 13, 2018

Background:
You are a developer for ProseWare Inc., a software-as-a-service (SaaS) company that provides a comment system that websites use to allow for end users to post comments associated with a webpage or topic on a customer’s website.
Business requirements
Moderation:
The moderation of comments is a feature of the software, and usually involves the editing of a comment.
Only users who have accounts in a group in Azure Active Directory (Azure AD) have the ability moderate.
External users can also become moderators, but only by explicit invitation.
Any moderation action must include the name of the moderator.
Comment navigation:
Each comment is identified by a unique string consisting of a random string of characters.
Within the body of a comment, internal links to other comment threads can be specified using the link format: “/
<parent comment id> / <child comment id>”
Comment search:
Comments can be searched using Azure Search. Searches must do the following:
Searching for email addresses must match email addresses in comments.
Searching must work for the client’s language.
Internal links to other comments using the link format should be searched.
Content screening:
Comment content is screened for inappropriate language, length, and topic using content analysis. Content must be screened, but can appear prior to be screened.
Mobile App:
The moderation functionality can be accessed using a Universal Windows Platform (UWP) app named
ProsewareApp. The app includes functionality that notifies moderators when changes are made to a comment they modified.
Export:
Customers can perform an export of all comments to a customer supplied Microsoft OneDrive folder on demand. The export functionality is implemented as an Azure Logic App, and it must be able to be triggered by the customer from their local network.
Interaction agents:
Interaction agents are parts of the system that interact with comment threads. The main purpose is to modify a comment’s body based on the contents of the comment. For example, one of the agents is WikiAgent, which adds links to Wikipedia articles when it sees text in the comment body that exactly matches a Wikipedia article title. Interaction Agents are implemented in Service Fabric.
Interaction agents must meet the following requirements:
Only successfully process each comment once
Any errors encountered during the processing of a comment should be retried
Must run on systems that allow for custom applications to be installed
Must run in a VNet or private network space
Must be run on a system that can scale up and down based on demand
A single user’s usage of Interaction Agents must not impact other users’ usage of Interaction Agents
Technical requirements
Authentication:
ProseWare Inc. allows for user authentication through Azure AD and Twitter.
Storage:
The application runs as a Web App on Azure. Comments are stored in an Azure DocumentDB databasenamed “Proseware”.
Performance:
The product includes a service level agreement (SLA) for individual method performance. All data retrieval methods must return within 100ms 99% of the time.
API:
The ProseWare Inc. API is made available to public callers using an Azure API App. Azure AD and Twitter are the Authentication Providers.
Application structure
CommentController.cs: