Embedding Pipelines

Embedding pipelines form the foundation for RAG applications and semantic search. As part of various projects, numerous pipelines were developed – each adapted to the specific requirements and preferences of customers.

Security-by-Design

Security-by-Design: A central principle in development is the consistent consideration of access controls. Row-Level Security (RLS) and Field-Level Security are conceptually integrated from the start. Embeddings inherit the permissions of their source documents – this ensures that semantic search only finds content the respective user has access to.

Vector Stores

Flexible Vector Stores: Different vector databases are used depending on requirements – Milvus for high-performance, scalable deployments, Azure AI Search for integrated search pipelines with hybrid search, Azure SQL Database with Vector Extension for scenarios with existing SQL infrastructure and RLS requirements.

Orchestration

Orchestration: Document embedding is done via various pipeline technologies – N8N for rapid, visual development, Azure Durable Functions for scalable, event-driven pipelines, Fabric Data Pipelines for integration into Microsoft Fabric ecosystems.

Embedding Models

Embedding Models: Depending on use case and data protection requirements – OpenAI API for state-of-the-art embeddings, Microsoft Foundry for Azure-native deployments, Ollama for local, privacy-compliant processing.