IBM watsonx

What is IBM watsonx?

IBM watsonx is an AI platform for enterprises - an alternative to ChatGPT/Azure OpenAI, but with full control over data and models. You can train models on your own data, deploy them on-premises, and meet AI Act requirements.

Three main components:

1. watsonx.ai - studio for working with AI models (prompts, fine-tuning, RAG)
2. watsonx.data - lakehouse for AI data preparation
3. watsonx.governance - tools for AI management and compliance

Difference vs ChatGPT/Azure OpenAI:

• Your data doesn’t leave your servers (on-premises option)
• You can fine-tune models on your own documents
• You have complete documentation of what the model knows and from where (AI Act compliance)
• Apache 2.0 license - you can modify and deploy without restrictions

Worth anticipating something that surfaces after a few months: models end up running in several processes and nobody keeps a register of them. We return to this further down.

What is watsonx for?

Typical Use Cases

Process Automation:

• Customer support chatbot trained on company FAQ
• Contract analysis and key information extraction
• Report generation from structured data

Employee Support:

• Assistant for sales reps (product database search)
• Internal knowledge search engine (documentation, procedures)
• Automatic email responses

Data Analysis:

• Document classification (invoices, complaints, orders)
• Data extraction from unstructured sources
• Customer opinion sentiment analysis

What are Granite models?

Granite is IBM’s family of AI models - an alternative to GPT-4, Claude, Llama:

Model	Parameters	Use Case
Granite 3.0 8B	8 billion	General use, chatbots
Granite 3.0 2B	2 billion	Fast responses, edge devices
Granite Code	8B/20B	Code generation and analysis
Granite Guardian	8B	Content moderation

Why Granite instead of GPT-4?

• Open source (Apache 2.0) - you know what’s inside
• Training data documentation - IBM discloses sources
• Smaller and faster - you don’t need a $10K GPU
• You can modify - fine-tuning, distillation

Benchmark: Granite 3.0 8B achieves comparable quality to Llama 3.1 8B with better energy efficiency.

How does RAG work?

RAG (Retrieval-Augmented Generation) is a technique for connecting LLM with company knowledge base:

User question
       ↓
[Search in document database]
       ↓
Found fragments + question → LLM → Answer

In practice:

1. You load company documents into watsonx (PDF, Word, web pages)
2. watsonx indexes them and creates embeddings
3. User asks “What is the return procedure?”
4. System finds relevant fragments from documentation
5. LLM generates answer based on those fragments

Result: Model answers questions about your company, doesn’t hallucinate (has sources).

watsonx.governance - AI Act Compliance

AI Act requires documentation and audit of AI systems. watsonx.governance provides:

• Model origin tracking - who, when, from what data trained
• Bias detection - does the model discriminate against any groups
• Explainability - why the model made that decision
• Audit trail - complete history of changes and usage
• Risk assessment - risk evaluation according to AI Act categorization

For regulated industries: Banks, insurance, healthcare need to prove to auditors how their AI works. watsonx.governance generates reports automatically.

Where can you deploy watsonx?

Option	For whom
IBM Cloud	Quick start, pay-as-you-go
On-premises	Banks, sensitive data, air-gap
Hybrid	Part in cloud, part on-premises
Multi-cloud	AWS, Azure, GCP

On-premises: watsonx runs on OpenShift. You need a cluster with GPU (NVIDIA) or you can use CPU for smaller models.

How much does it cost?

watsonx has a token and resource-based model:

Element	Approximate price
watsonx.ai (cloud)	From $0.01 per 1000 tokens
Granite 8B on-prem	License + infrastructure
watsonx.governance	Per user/year
watsonx.data	Per TB of data

Free tier: 50,000 tokens monthly for testing.

Enterprise: Custom pricing depending on scale.

Specifications

Models	Granite, Llama, Mistral, custom
Deployment	IBM Cloud, on-prem, hybrid
GPU support	NVIDIA A100, H100, L40S
Integrations	SAP, Salesforce, ServiceNow
API	REST, Python SDK
Governance	AI Act ready, SOC 2, ISO 27001

Oversight of what the model does in the organization

Deploying a language model rarely stops at one use case. After a few months it turns out models run in several processes and nobody keeps a register of them.

• A register of use cases. A list of where a model is used, what data it processes and who owns it is the starting point for both risk analysis and regulatory requirements.
• Approving deployments. Putting a model into a process touching customers or personal data is a decision with consequences comparable to deploying a system — it should go through the same approval path.
• Providers as part of the chain. Using models supplied by an external party brings them into the supply chain that NIS2 requires you to oversee.
• Decision trail. When a model participates in a decision, reconstructing it requires recording the input, the model version and the output.

We address supplier and process oversight requirements as part of NIS2 compliance and ISO 27001.

FAQ

How is watsonx different from ChatGPT? watsonx is an enterprise platform - you can train your own models, deploy on-premises, have full control over data. ChatGPT is a ready-made model in Microsoft cloud.

Can I use GPT-4 in watsonx? Not directly - but you can use open-source models (Llama, Mistral) or Granite.

What is fine-tuning? Additional training of a model on your own data. E.g., you teach Granite on thousands of customer service emails - the model learns your company’s style and specifics.

Do I need GPU? For 8B+ models in production - yes. For smaller models (2B) or testing - CPU is enough, but slower.

How long does deployment take? PoC with RAG - 2-4 weeks. Full production deployment - 2-6 months depending on scale.

Does watsonx integrate with SAP? Yes. IBM has ready integrations with SAP, Salesforce, ServiceNow, Microsoft 365.

Is training data secure? In on-premises version - data never leaves your infrastructure. In cloud - data in dedicated tenant, encrypted.

What if the model hallucinates? RAG minimizes hallucinations - model answers based on your documents. watsonx.governance allows monitoring answer quality.

Does nFlo deploy watsonx? Yes. We do PoC, fine-tuning, RAG solution building, integration with client systems.

How to start? Free trial on IBM Cloud or workshop with nFlo - we define use case and build PoC.