Package com.tnsai.quality.security.moderation

Interface ContentModerator

All Known Implementing Classes:
PatternBasedModerator
public interface ContentModerator
Interface for content moderation.

Content moderators check user input, model output, and tool results for policy violations before processing or returning to users.

Moderation Points

 User Input → [checkInput] → LLM → [checkOutput] → User
                    ↓
                  Tools → [checkToolResult] → LLM

Implementation Types

  • PatternBasedModerator - Regex pattern matching
  • OpenAIModerator - OpenAI Moderation API (external)
  • PerspectiveAPIModerator - Google Perspective API (external)

Usage Example


 ContentModerator moderator = PatternBasedModerator.withDefaults();

 // Check user input
 ModerationResult inputResult = moderator.checkInput(userMessage);
 if (inputResult.isBlocked()) {
     return inputResult.getSafeResponse();
 }

 // Process with LLM
 String response = llm.chat(userMessage);

 // Check output
 ModerationResult outputResult = moderator.checkOutput(response);
 if (outputResult.isBlocked()) {
     return outputResult.getSafeResponse();
 }

 return response;

Agent Middleware Integration


 Agent agent = Agent.builder("moderated-agent")
     .llm(llmClient)
     .addMiddleware(new ModerationMiddleware(moderator))
     .build();
See Also:

  • Method Details

    • checkInput

      ModerationResult checkInput(String userInput)
      Checks user input for policy violations.

      Called before processing user messages. Should detect:

      • Harmful or offensive content
      • Prompt injection attempts
      • PII that shouldn't be processed
      • Spam or off-topic content
      Parameters:
      userInput - the user's input message
      Returns:
      moderation result

    • checkOutput

      ModerationResult checkOutput(String modelOutput)
      Checks model output for policy violations.

      Called before returning responses to users. Should detect:

      • Harmful or offensive content generated by the model
      • Hallucinated harmful instructions
      • PII leakage from training data
      • Content that bypassed input filtering
      Parameters:
      modelOutput - the model's response
      Returns:
      moderation result

    • checkToolResult

      ModerationResult checkToolResult(String toolName, String result)
      Checks tool execution results for policy violations.

      Called after tool execution, before using results. Should detect:

      • Sensitive data in tool results
      • Harmful content from external sources
      • Unexpected or dangerous outputs
      Parameters:
      toolName - the name of the tool
      result - the tool's output
      Returns:
      moderation result

    • isEnabled

      default boolean isEnabled()
      Checks if this moderator is enabled.
      Returns:
      true if moderation is active

    • getName

      default String getName()
      Gets the name of this moderator.
      Returns:
      moderator name