6 Local AI Setups for Businesses That Cannot Share Data With the Cloud

Cloud AI is convenient until it is not. Healthcare providers cannot send patient data to a third-party API. Law firms cannot route privileged client communications through servers they do not control. Finance teams handling proprietary trading data face similar constraints. And even outside regulated industries, some businesses simply do not want their data training someone else's model or sitting in a foreign data center.

Local AI—running language models on your own hardware—has become genuinely practical in 2026. The models are better, the tooling is easier, and the hardware requirements are lower than they were even 18 months ago. This post covers six setups that real businesses are using today, from the simplest possible local install to production-grade self-hosted infrastructure.

Honest framing: Local AI is not free. The compute cost is paid upfront in hardware. The privacy benefit is real, but so is the operational complexity compared to a managed cloud API. These setups make sense when the privacy requirement or the volume economics justify the investment.

Who Needs Local AI

Before the setups: a quick check on whether local AI is actually the right answer for your situation.

Strong cases for local AI:

• Regulated industries with specific data residency requirements (healthcare, legal, finance)
• Businesses processing proprietary competitive information that cannot leave internal systems
• High-volume workloads where fixed hardware cost beats per-token pricing over time
• Teams that need guaranteed uptime independent of external API availability
• Jurisdictions with strict data sovereignty rules

Cases where cloud AI is likely better:

• You need frontier-level model quality for complex reasoning or nuanced writing
• Your data is not sensitive and cloud pricing is manageable
• You cannot invest in hardware or IT infrastructure maintenance
• Your volume is low enough that cloud costs are not a budget concern

With that context set, here are the six setups.

The 6 Local AI Setups

1. Ollama + Open WebUI — The Easiest Start

What it is: Ollama is an open-source tool that makes downloading and running local LLMs as simple as a terminal command. Open WebUI adds a browser-based chat interface that looks and works like ChatGPT—no command line required for end users.

How to set it up:

1. Install Ollama from ollama.ai (Mac, Linux, or Windows)
2. Run ollama pull llama4 or ollama pull mistral to download a model
3. Run ollama serve to start the local API
4. Install Open WebUI via Docker: docker run -d -p 3000:80 --add-host=host.docker.internal:host-gateway ghcr.io/open-webui/open-webui:main
5. Open your browser to localhost:3000—done

API integration: Ollama exposes an OpenAI-compatible API at localhost:11434. In n8n, point your OpenAI node's base URL to this address. Your existing workflows switch from cloud to local with a single configuration change.

Best for: Teams that want to test local AI quickly, small teams needing a private ChatGPT-style interface, and developers building and testing prompts locally before deploying to production.

Hardware minimum: 8GB RAM (CPU only, slower) or a GPU with 8GB VRAM for 7B models. For 13B models, 16GB VRAM is recommended.

Limitations: Inference speed depends entirely on your hardware. On CPU, smaller models are usable but slow. GPU acceleration is strongly recommended for anything resembling production use. Models larger than 13B require significant GPU investment.

2. LM Studio — Best for Non-Technical Teams

What it is: LM Studio is a desktop application for Mac, Windows, and Linux that provides a graphical interface for downloading, managing, and running local models. It is designed for users who are not comfortable with the terminal and want a point-and-click experience for local AI.

How it works: Open LM Studio, search for models in its built-in model browser (connected to HuggingFace), download the one you want, load it, and start chatting. The application also exposes a local server with an OpenAI-compatible API for workflow integration.

Best for: Small businesses without dedicated IT staff where the people using AI are not developers. Non-technical team members who need a private AI assistant for document work, drafting, and analysis without data leaving their laptop.

Hardware requirements: Same as Ollama. LM Studio will show you which models are compatible with your detected hardware and flag ones that may be too large.

Limitations: LM Studio is excellent for individual use but is not designed for multi-user or server deployment. For team-wide local AI access, Ollama with Open WebUI is more appropriate. LM Studio is a client application, not a server.

Pro Tip: LM Studio's model browser includes quantized models—compressed versions of larger models that run faster on consumer hardware with a modest quality trade-off. Quantized 13B models often outperform full 7B models and run on the same hardware.

3. AnythingLLM — Best for Document RAG on Private Files

What it is: AnythingLLM is an open-source application that adds Retrieval-Augmented Generation (RAG) on top of a local (or cloud) LLM. You upload documents, and the model answers questions about their content without those documents leaving your system.

How it works: AnythingLLM creates a local vector store from your uploaded documents. When you ask a question, it retrieves the relevant document sections and passes them to the LLM as context. The model sees your internal documents as reference material, not as training data.

Best for: Legal teams querying contracts, finance teams running analysis over internal reports, knowledge management across internal wikis and documentation, and any use case where the value is in asking questions of private documents.

Integration options: AnythingLLM supports Ollama as its local model backend, cloud models (Claude, GPT, Gemini) with your own API keys, and several vector database options. You can run it entirely locally or use it as a privacy-preserving RAG layer in front of a cloud model.

Setup: Available as a Docker image, desktop app, or cloud-hosted with your infrastructure. The desktop version is the simplest starting point.

Limitations: RAG quality depends on document preprocessing and chunking strategy. Large, complex documents with unusual formatting may produce less accurate retrieval. This is a limitation of RAG in general, not specific to AnythingLLM.

For context on vector databases in general, see What Is a Vector Database: Pinecone and Weaviate.

4. LocalAI — Best for API-Compatible Server Deployment

What it is: LocalAI is an open-source server that provides a fully OpenAI-compatible REST API backed by local models. It is designed for server deployment (Linux, Docker) and supports model switching, GPU acceleration, and multi-user access.

How it works: LocalAI runs as a Docker container or standalone binary on a server. You provide model files, configure endpoints, and it exposes a local API that is drop-in compatible with the OpenAI SDK. Any application or workflow platform that talks to OpenAI can be pointed at LocalAI instead.

Best for: Organizations that need team-wide local AI access, IT teams that want to self-host a managed AI endpoint for multiple departments, and developers building applications that need an OpenAI-compatible backend they control.

Key features:

• Supports text generation, embeddings, image generation (Stable Diffusion), and audio transcription
• Multi-model support with hot-swapping
• Docker-compose deployment for straightforward production setups
• Compatible with existing OpenAI SDK code without changes beyond the base URL

Hardware: Server-grade GPU recommended for team use. An NVIDIA A10G (24GB VRAM) handles most 13B-34B models well for team-scale inference. For smaller teams, a single RTX 4090 is often sufficient.

Limitations: More complex to set up than LM Studio or Ollama. Requires Linux server administration knowledge. No built-in graphical interface—API only.

5. Jan AI — Best for Offline-First Individual Use

What it is: Jan is a desktop application (Mac, Windows, Linux) that runs entirely offline. It is similar to LM Studio in concept but with a stronger emphasis on complete offline operation—no telemetry, no cloud dependency, no network connection required after initial model download.

How it works: Download Jan, download models through its interface, and run them. Everything stays on the device. Jan also has an extension system for adding functionality and exposes a local API.

Best for: Individual users with strict data isolation requirements—lawyers, therapists, financial advisors, or anyone whose professional obligations require that work data not leave their device under any circumstances. Also useful for air-gapped environments.

Limitations: Single-user desktop application. Not designed for multi-user or server deployment. The model selection and management interface is less polished than LM Studio. Inference speed limitations are the same as other local tools.

Note: Jan is the right choice when the requirement is "nothing leaves this machine, ever." If team-wide access is needed, look at Ollama + Open WebUI or LocalAI instead.

6. Cortex (formerly Nitro) — Best for Production Local API

What it is: Cortex is a production-oriented local AI inference engine that prioritizes performance and reliability. It is designed for developers who need a local AI API that behaves predictably under production load, with proper request queuing, concurrent request handling, and performance monitoring.

How it works: Cortex exposes an OpenAI-compatible API with support for model management, GPU acceleration, and concurrent inference. It supports multiple backends (llama.cpp, ONNX) for optimal performance across different hardware configurations.

Best for: Developers building applications that need reliable local AI inference—internal tools that multiple colleagues use simultaneously, automation workflows with high request volumes, and cases where inference reliability and latency predictability matter as much as the output quality.

Key differentiators vs. Ollama:

• Better concurrent request handling under load
• More detailed performance metrics and monitoring
• Production-grade request queuing
• Designed for integration into larger applications rather than interactive use

Limitations: Less beginner-friendly than Ollama or LM Studio. Primarily developer-oriented. Community is smaller than Ollama's.

Model Recommendations for Local Use in 2026

The setup is only half the equation. Which model you run matters as much as how you run it.

For most business use cases, a quantized 13B-22B model on a modern GPU provides a practical balance of speed, quality, and hardware cost.

Connecting Local AI to Your Automation Workflows

The practical value of local AI is not just chat—it is automation. All of the Ollama-based setups expose an OpenAI-compatible API, which means connecting to n8n or any other automation platform requires only a configuration change.

In n8n:

1. In any OpenAI node, open the credentials settings
2. Change the base URL from to http://localhost:11434/v1 (or your server's IP)
3. Set any non-empty string as the API key (Ollama does not require a real key)
4. Select the model name you have downloaded locally

Your existing n8n workflows now run against your local model. The prompts, logic, and outputs work identically—only the inference endpoint has changed.

Hybrid approach: Many businesses run local AI for sensitive data tasks and cloud AI for complex or high-quality generation. A simple routing node in n8n can send tasks to local or cloud endpoints based on data sensitivity classification, keeping private data local while using frontier models for tasks that do not involve sensitive information.

The Real Cost of Going Local

Local AI is not free. The cost structure is different from cloud:

• Cloud AI: Low or zero fixed cost, high variable cost per token
• Local AI: High fixed cost (hardware), low variable cost per inference

The break-even point depends on your volume. For most small businesses, cloud pricing is more cost-effective until inference volume is substantial. Local AI makes economic sense when:

• You are processing millions of tokens per month
• Hardware would be used for other purposes as well
• The privacy requirement eliminates cloud as an option regardless of cost

For data-sensitive workflows where cloud is not an option, the cost comparison is moot—local is the only viable choice. For all other situations, run the numbers for your specific volume before investing in hardware.

Need help deciding whether local AI or a cost-optimized cloud strategy fits your business? Book a call at evalics.com/contact to review your data requirements and automation goals.