Artificial intelligence security is no longer limited to protecting servers, APIs, or databases. Organizations must also protect prompts, model outputs, training data, vector databases, agent tools, user identities, and the decisions made with AI-generated information.
This creates an important challenge for businesses adopting large language models (LLMs): how do you choose the right model and architecture while considering security, privacy, safety, reliability, cost, and long-term maintainability?
The answer is not simply choosing the model with the highest benchmark score. A secure AI implementation requires a combination of risk frameworks, cloud-native controls, application guardrails, adversarial testing, and continuous evaluation.
This article reviews important AI security frameworks and tools available from AWS, Google Cloud, Microsoft Azure, and the open-source ecosystem. It also explains how businesses can use these resources to evaluate LLMs more responsibly and make better technology decisions.
AI safety and AI security are related, but different
From one side, AI safety and AI security are often used interchangeably, but they address different areas of risk.
Complementary, AI safety generally focuses on the behavior and impact of an AI system. This includes:
- Harmful or toxic content
- Bias and discriminatory outputs
- Hallucinations and unreliable information
- Unsafe recommendations
- Inappropriate content generation
- Excessive reliance on automated decisions
AI security focuses on protecting the system, its data, and its surrounding infrastructure. This includes:
- Prompt injection and jailbreaks
- Sensitive information disclosure
- Data poisoning
- Model and software supply-chain risks
- Unauthorized access
- Excessive agent permissions
- Insecure tool calls
- Data exfiltration
- Denial-of-service and excessive consumption
In practice, the two areas overlap. For example, a prompt injection attack is a security problem, but it may cause the system to produce unsafe content or perform a harmful action.
The NIST AI Risk Management Framework Generative AI Profile provides a useful foundation for managing these risks across the AI lifecycle. It organizes AI risk management around four functions: Govern, Map, Measure, and Manage.
For technical application security, the OWASP Top 10 for LLM Applications provides a practical list of common vulnerabilities, including prompt injection, sensitive information disclosure, supply-chain vulnerabilities, improper output handling, excessive agency, and unbounded consumption.
Organizations can also use MITRE ATLAS, a living knowledge base of adversary tactics and techniques targeting artificial intelligence systems. Together, these resources help businesses move from general concern to structured risk assessment.
Cloud providers are adding AI-specific security controls
Cloud providers increasingly offer security frameworks and tools designed specifically for AI workloads. These services can reduce implementation effort, but they do not eliminate the customer’s responsibility.
The shared-responsibility model still applies. The provider protects the underlying cloud infrastructure and managed services, while the customer remains responsible for data, identity and access management, application configuration, prompts, permissions, and the way AI outputs are used.
AWS
AWS provides several relevant services for organizations building generative AI applications.
Amazon Bedrock Guardrails provides configurable safeguards for model inputs and outputs. Depending on the configuration, organizations can apply content filters, denied topics, sensitive information detection, contextual grounding checks, and prompt-attack detection.
Guardrails can be applied to model inference, agents, knowledge bases, and flows. This is especially relevant for businesses building RAG applications or agents that can interact with internal systems.
AWS also provides Amazon Bedrock model evaluations. Organizations can use built-in metrics or define custom metrics to compare models against business-specific requirements. Evaluation jobs can also use custom prompt datasets, which is important because generic benchmarks rarely represent the actual risks and workflows of a particular organization.
AWS further documents security through its Shared Responsibility Model, which clarifies that customers remain responsible for security “in” the cloud, including data, applications, access controls, encryption choices, and network configurations.
For businesses already operating on AWS, a practical security architecture may combine:
- IAM roles and least-privilege permissions
- Amazon Bedrock Guardrails
- Private networking and controlled egress
- Encryption and secrets management
- CloudTrail and centralized logging
- Data classification and PII detection
- Model and application evaluations
- Human approval for high-risk actions
Google Cloud
Google’s Secure AI Framework, or SAIF, is a conceptual framework for securing AI systems throughout their lifecycle. It addresses model risk management, data protection, infrastructure security, and defense in depth.
Google Cloud’s current generative AI security best practices reference several services that can support AI security programs.
Model Armor can inspect prompts and responses to help mitigate prompt injection, data leakage, and harmful content. Current Google Cloud guidance also describes Model Armor as supporting AI agents, MCP connections, and agentic communications, although availability can vary by product, region, and service tier.
Security Command Center AI Protection provides visibility and threat detection capabilities for AI assets and agent workloads. It can help organizations inventory AI resources, investigate findings, and detect suspicious activity involving AI agents.
Google Cloud also provides safety settings for Gemini models and workflows for evaluating model performance on Vertex AI.
For organizations using Google Cloud, the relevant question is not only whether Gemini produces high-quality responses. It is also whether the surrounding architecture provides appropriate identity controls, data protection, monitoring, content filtering, and auditability.
Microsoft Azure
Microsoft Azure provides AI security controls through Azure AI Content Safety, Microsoft Foundry, and the Azure Well-Architected Framework.
Azure AI Content Safety can detect harmful user-generated and AI-generated content across text and image applications.
Prompt Shields are designed to detect user prompt attacks and document attacks. This is particularly important for RAG systems, where untrusted instructions may be hidden inside uploaded files, emails, websites, or knowledge-base documents.
Microsoft Foundry also provides a broad catalog of models from Microsoft, OpenAI, Anthropic, Meta, Mistral, Cohere, Hugging Face, and other providers. The Microsoft Foundry Models documentation describes capabilities for comparing and evaluating models using real-world tasks and an organization’s own data.
However, model availability does not mean that every model has the same data-handling characteristics. Microsoft documents different privacy and security conditions for models sold by Azure and for partner or community models. Organizations should review the data, privacy, and security documentation for Foundry models before using sensitive information.
Microsoft also provides guidance for responsible AI in Azure workloads, emphasizing the relationship between responsible AI, privacy, confidentiality, integrity, and secure architecture.
Open-source tools for AI security and evaluation
Cloud-provider services are valuable, but they are not the only option. Open-source projects can provide flexibility, portability, and greater control over testing and deployment.
They also introduce additional responsibilities. Businesses must evaluate project maintenance, licensing, dependency risk, telemetry, documentation, community activity, and the ability to support the tool in production.
Several open-source projects are particularly useful for AI security and evaluation.
NVIDIA NeMo Guardrails
NVIDIA NeMo Guardrails is an open-source toolkit for adding programmable guardrails to LLM applications.
It supports different control points, including:
- User input
- Dialogue flow
- Retrieved documents
- Tool execution
- Model output
This makes it useful for conversational systems, RAG applications, and agents that need explicit policies around what they can discuss, retrieve, or execute.
NeMo Guardrails should be treated as one layer of an application architecture. It does not replace IAM, network security, audit logging, secure software development, or human review.
NVIDIA garak
NVIDIA garak is an open-source LLM vulnerability scanner. It uses different probes to explore how a model or dialogue system can fail.
It can be useful during model selection and pre-production testing, especially for identifying weaknesses related to harmful content, jailbreaks, prompt injection, data leakage, and other undesirable behaviors.
Because garak focuses primarily on model and dialogue behavior, organizations should combine it with application-level testing when the AI system has access to company data, tools, APIs, or business workflows.
Microsoft PyRIT
PyRIT is Microsoft’s open-source Python Risk Identification Tool for generative AI. It is designed to help security professionals and engineers proactively identify risks in generative AI systems.
PyRIT is especially relevant for structured adversarial testing and red-team campaigns. It can help organizations test how an application behaves when exposed to manipulated prompts, multi-turn attacks, unsafe requests, and other adversarial scenarios.
Promptfoo
Promptfoo is an open-source CLI and library for evaluating and red-teaming LLM applications.
It can be used to compare prompts, models, RAG pipelines, and AI applications using custom test cases and metrics. Its red-teaming workflows can generate adversarial inputs for applications rather than testing only isolated model behavior.
This distinction matters. A model may appear safe in a basic test but still expose information when connected to a poorly configured database or an over-permissioned tool.
DeepEval and DeepTeam
DeepEval is an open-source LLM evaluation framework that supports metrics for relevance, hallucination, task completion, tool correctness, faithfulness, bias, toxicity, and other criteria.
DeepTeam focuses on red-teaming LLMs and AI agents. It can simulate prompt injection, jailbreaks, multi-turn attacks, PII leakage, broken authorization, tool misuse, and other vulnerabilities.
These tools are useful when a business needs repeatable evaluations that can run during development and CI/CD rather than relying on informal manual testing.
Meta PurpleLlama
Meta PurpleLlama is an umbrella project that includes tools and evaluations for improving the security of open generative AI systems.
Its components include Llama Guard, Prompt Guard, Code Shield, and CyberSec Eval. These tools address areas such as harmful content moderation, prompt injection, jailbreaks, insecure code generation, and cybersecurity-related risks.
As with any open-source model or safeguard, organizations should review licensing, deployment requirements, compute needs, and the suitability of the tool for their specific industry and risk profile.
Which metrics should businesses use to compare LLMs?
A model evaluation should reflect the actual business use case. Generic leaderboards can provide useful context, but they should not be the primary decision mechanism.
A practical evaluation should include several categories.
Security and privacy
Measure whether the system:
- Leaks PII, credentials, or confidential business information
- Reveals system prompts or internal instructions
- Resists direct and indirect prompt injection
- Prevents unauthorized tool use
- Respects role-based access controls
- Handles sensitive documents correctly
- Maintains the expected data-retention and processing model
For high-risk workflows, some failures should be treated as hard stops rather than averaged into a general score. For example, a model that performs well but exposes confidential information should not remain a candidate simply because its average score is high.
Safety and responsible behavior
Evaluate:
- Harmful or toxic content
- Discriminatory or biased responses
- Unsafe recommendations
- Inappropriate refusals
- Jailbreak resistance
- Responses involving regulated or high-impact decisions
- Whether the model escalates appropriately to a human
Safety is contextual. The correct behavior for a healthcare assistant, legal research tool, customer-service bot, and internal coding assistant will not be identical.
Quality and reliability
Measure:
- Accuracy on representative tasks
- Groundedness in approved sources
- Hallucination rate
- Relevance
- Structured-output compliance
- Tool-call correctness
- Consistency across repeated requests
- Performance with ambiguous or incomplete information
For RAG systems, evaluate the retrieval process as well as the model. A strong model cannot compensate for poor document permissions, inaccurate retrieval, or contaminated knowledge bases.
Operational and financial performance
Also consider:
- Latency and throughput
- Availability and regional deployment
- Cost per request or workflow
- Context-window requirements
- Integration effort
- Monitoring and logging capabilities
- Vendor support
- Portability and lock-in
- Maintenance requirements
- Environmental impact where relevant
The best model is usually the one that meets the business requirements with the lowest acceptable risk and operational complexity—not necessarily the most powerful or most expensive model.
A practical AI security approach for businesses
A manageable AI security program can begin with five steps.
1. Classify the use case and data
Document what the AI system will do, what information it will process, who will use it, and what actions it can take.
Separate low-risk use cases, such as summarizing public content, from higher-risk use cases involving healthcare data, legal documents, financial information, employee records, or automated decisions.
2. Define security gates before selecting a model
Establish non-negotiable requirements for privacy, data residency, access control, harmful content, tool permissions, auditability, and human oversight.
This prevents the organization from selecting a model first and trying to justify the risks later.
3. Build a representative evaluation set
Use real business scenarios, sanitized data, difficult edge cases, multilingual inputs, adversarial prompts, and examples of expected refusals.
The evaluation set should be versioned and reused whenever a model, prompt, retrieval system, or guardrail changes.
4. Test the complete application
Evaluate the model together with the prompt, retrieval layer, databases, APIs, tools, identity system, and user interface.
This is essential because many serious AI security vulnerabilities are created by the application architecture rather than by the model itself.
5. Monitor after deployment
AI security is not a one-time certification. Models, APIs, dependencies, prompts, data sources, and attacks change over time.
Organizations should monitor failures, unexpected tool calls, sensitive-data detections, refusal behavior, user feedback, latency, costs, and changes in model performance.
How Origo helps organizations make better AI decisions
At Origo, we believe businesses should not adopt AI based on hype, isolated benchmarks, or vendor pressure.
Our role is to help organizations connect business objectives with practical technology decisions. That may include evaluating commercial and open models, reviewing cloud architecture, designing RAG or agent workflows, defining evaluation criteria, implementing guardrails, and establishing governance that teams can actually maintain.
The process usually begins with the workflow and the risk profile—not with a specific model.
From there, we help organizations compare options using evidence across:
- Security and privacy
- Safety and responsible behavior
- Quality and groundedness
- Cost and latency
- Cloud compatibility
- Integration complexity
- Operational maturity
- Long-term maintainability
The result is not simply a model recommendation. It is a clearer decision about what should be built, how it should be protected, how it should be evaluated, and when human oversight is required.
AI security is becoming a core part of technology strategy. Businesses that approach it systematically can adopt AI with greater confidence while reducing avoidable risks, unnecessary spending, and dependence on tools that do not fit their operating environment.
If your organization is evaluating LLMs, AI agents, RAG systems, or cloud-based AI services, Origo can help you assess the options and design a practical path toward secure AI adoption.
This article provides general technical guidance. Specific security, privacy, regulatory, and contractual requirements should be reviewed with the appropriate legal, compliance, and security professionals.
Ready to explore what AI can do for your team?