Skip to main content

Troubleshooting Common Issues

This guide covers the most common issues users encounter and how to resolve them.

Memory and Performance Issues​

Out of Memory Errors​

Symptoms:

allocation error: CapacityOverflow
Server crashes unexpectedly

Causes:

  • Hitting the --allocator-size limit
  • Large batch operations
  • Image processing without streaming enabled
  • Batching inputs of very different sizes together. A single large input batched alongside several small ones forces the small ones to be padded up to (and computed at) the large input's length. See Batched Request Behavior for why, and the batching snippet under Token Limit Exceeded for a fix.

Solutions:

  1. Increase allocator size:
ahnlich-db run --allocator-size 21474836480  # 20 GiB (default is 10 GiB)ahnlich-ai run --allocator-size 21474836480
  1. Enable streaming for images (AI proxy):
ahnlich-ai run --enable-streaming  # 10x less memory, 40% slower
  1. Reduce batch sizes:
Python
# Instead of:
large_batch = [entry1, entry2, ..., entry1000]
client.set(Set(store="my_store", inputs=large_batch))

# Do this:
batch_size = 100
for i in range(0, len(large_batch), batch_size):
batch = large_batch[i:i+batch_size]
client.set(Set(store="my_store", inputs=batch))
  1. Monitor memory usage:
# Check process memoryps aux | grep ahnlich # Monitor with toptop -p $(pgrep ahnlich)

Slow Query Performance​

Symptoms:

  • Queries taking longer than expected
  • High CPU usage

Diagnostic Steps:

  1. Enable tracing to identify bottlenecks:
ahnlich-db run --enable-tracing --otel-endpoint http://localhost:4317

View traces in Jaeger UI at http://localhost:16686

  1. Check store size:
INFOSERVER
  1. Verify algorithm choice:
  • Linear algorithms (Cosine, Euclidean, DotProduct) scale linearly with data size
  • Use HNSW for faster searches with large datasets:
CREATESTORE my_store DIMENSION 128 NONLINEARALGORITHMINDEX (HNSW)

Solutions:

  1. Use predicate indices for filtering:
# Index frequently filtered fields
CREATEPREDINDEX (category, author) IN my_store

# Then filter efficiently
GETPRED (category = science) IN my_store
  1. Optimize batch operations:
Python
# Batch SET operations
entries = [entry1, entry2, ..., entry100]
client.set(Set(store="my_store", inputs=entries))
  1. Use appropriate similarity algorithm:
  • CosineSimilarity: Best for normalized vectors, direction-based similarity
  • EuclideanDistance: Best for absolute distance measures
  • DotProduct: Fast when vectors are pre-normalized
  • HNSW: Best for large-scale approximate nearest neighbor searches
  1. Adjust thread pool size:
ahnlich-db run --threadpool-size 32  # Default: 16

Connection Issues​

Cannot Connect to Server​

Symptoms:

connection refused
Failed to dial server
Transport issues with tonic

Diagnostic Steps:

  1. Check if server is running:
# Check DBcurl http://localhost:1369  # or use telnetps aux | grep ahnlich-db # Check AIcurl http://localhost:1370ps aux | grep ahnlich-ai
  1. Verify port availability:
# Check if port is in uselsof -i :1369lsof -i :1370 # Or with netstatnetstat -tuln | grep 1369
  1. Check firewall rules:
# Ubuntu/Debiansudo ufw statussudo ufw allow 1369sudo ufw allow 1370 # CentOS/RHELsudo firewall-cmd --list-allsudo firewall-cmd --add-port=1369/tcp --permanentsudo firewall-cmd --reload

Solutions:

  1. Start server on all interfaces:
# Allow connections from any IPahnlich-db run --host 0.0.0.0 --port 1369ahnlich-ai run --host 0.0.0.0 --port 1370
  1. Check host/port configuration:
Python
# Correct
client = DbClient("http://127.0.0.1:1369")

# Wrong - missing protocol
client = DbClient("127.0.0.1:1369") # Invalid URI error
  1. Verify network connectivity:
# Test connectivityping <server-host>telnet <server-host> 1369

Maximum Clients Reached​

Symptoms:

Max Connected Clients Reached
Connection rejected

Cause: Hit the --maximum-clients limit (default: 1000)

Solutions:

  1. Increase client limit:
ahnlich-db run --maximum-clients 5000
  1. Check current connections:
LISTCLIENTS
  1. Implement connection pooling:
Python
# Reuse connections instead of creating new ones
class ClientPool:
def __init__(self, uri, pool_size=10):
self.pool = [DbClient(uri) for _ in range(pool_size)]
self.index = 0

def get_client(self):
client = self.pool[self.index]
self.index = (self.index + 1) % len(self.pool)
return client
  1. Close idle connections:
Python
async def cleanup():
await client.close()

AI Proxy Cannot Connect to Database​

Symptoms:

Proxy Errored with connection refused
DatabaseClientError

Diagnostic Steps:

  1. Verify DB is running:
ps aux | grep ahnlich-db
  1. Check DB host/port:
# See what DB is listening onnetstat -tuln | grep 1369

Solutions:

  1. Start DB before AI:
# Terminal 1ahnlich-db run --port 1369 # Terminal 2 (wait for DB to start)ahnlich-ai run --db-host 127.0.0.1 --db-port 1369
  1. Verify connection settings:
# If DB is on different hostahnlich-ai run --db-host 192.168.1.10 --db-port 1369 # If DB uses non-default portahnlich-ai run --db-port 1400
  1. For standalone mode (no DB):
ahnlich-ai run --without-db
  1. Adjust connection pool:
ahnlich-ai run --db-client-pool-size 20  # Default: 10

Data and Store Issues​

Store Not Found​

Symptoms:

Store "my_store" not found

Diagnostic Steps:

  1. List stores in the relevant schema:
LISTSTORES SCHEMA public
  1. Check store name spelling:
# Store names are case-sensitive
"MyStore" ≠ "mystore"

Solutions:

  1. Create the store:
# DB
CREATESTORE my_store DIMENSION 128

# AI
CREATESTORE my_store QUERYMODEL all-minilm-l6-v2 INDEXMODEL all-minilm-l6-v2
  1. Check persistence loaded:
# If using persistenceahnlich-db run \  --enable-persistence \  --persist-location /path/to/data.dat \  --fail-on-startup-if-persist-load-fails true  # Fail loudly if load fails
  1. Verify correct server:
Python
# Make sure you're connecting to the right instance
client = DbClient("http://localhost:1369") # Not a different instance

Dimension Mismatch Errors​

Symptoms:

Store dimension is [128], input dimension of [256] was specified

Cause: Vector dimensions don't match store configuration.

Solutions:

  1. Check store dimension:
INFOSERVER
# Look at store details
  1. For AI stores, verify model dimensions:
ModelEmbedding Dimension
all-minilm-l6-v2384
all-minilm-l12-v2384
bge-base-en-v1.5768
bge-large-en-v1.51024
resnet-502048
clip-vit-b32-*512
clap-audio / clap-text512
buffalo-l512
sface-yunet128
  1. Match query and index models:
Python
# Both must have same dimensions
CreateStore(
store="my_store",
query_model=AiModel.BGE_BASE_EN_V15, # 768-dim
index_model=AiModel.BGE_BASE_EN_V15, # 768-dim (same)
)
  1. Recreate store with correct dimension:
DROPSTORE my_store IFTRUE
CREATESTORE my_store DIMENSION 768

Predicate Not Found​

Symptoms:

Predicate "author" not found in store

Cause: Querying by a predicate that wasn't indexed.

Solutions:

  1. Create predicate index:
CREATEPREDINDEX (author, category) IN my_store
  1. Or include when creating store:
CREATESTORE my_store DIMENSION 128 PREDICATES (author, category)
  1. Verify predicates exist:
INFOSERVER
# Check store predicates

Model and AI Issues​

Model Not Loading​

Symptoms:

index_model or query_model not selected or loaded
Error initializing a model thread
Tokenizer for model failed to load

Diagnostic Steps:

  1. Check supported models:
ahnlich-ai run --supported-models all-minilm-l6-v2,resnet-50
  1. Verify model cache:
# Default locationls -la ~/.ahnlich/models # Custom locationahnlich-ai run --model-cache-location /path/to/models
  1. Check disk space:
df -h ~/.ahnlich/models
  1. Test network connectivity:
# Models download from HuggingFacecurl https://huggingface.co

Solutions:

  1. Wait for initial download:
# First time loading a model downloads from HuggingFace# This can take several minutes depending on model size# Watch logs for progress
  1. Clear corrupted cache:
rm -rf ~/.ahnlich/models/model_name# Restart server to re-download
  1. Increase idle time:
# Keep models loaded longerahnlich-ai run --ai-model-idle-time 600  # 10 minutes (default: 5 min)
  1. Pre-download models:
# Download models before starting serverpython -c "from transformers import AutoModel; AutoModel.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')"

Token Limit Exceeded​

Symptoms:

Max Token Exceeded. Model Expects [256], input type was [512]

Cause: Text input exceeds model's token limit.

Token Limits:

  • all-minilm-*: 256 tokens
  • bge-*: 512 tokens
  • clip-vit-b32-text: 77 tokens
  • clap-text: 512 tokens
  • jina-embeddings-v2-base-code: 8192 tokens

Solutions:

  1. Truncate text:
Python
def truncate_text(text, max_length=200):
words = text.split()
return ' '.join(words[:max_length])

text = truncate_text(long_text)
  1. Split into chunks:
Python
def chunk_text(text, chunk_size=200):
words = text.split()
return [' '.join(words[i:i+chunk_size])
for i in range(0, len(words), chunk_size)]

chunks = chunk_text(long_document)
for chunk in chunks:
client.set(Set(store="docs", inputs=[...]))
  1. Use model with larger limit:
Python
# Switch from AllMiniLM (256) to BGE (512)
CreateStore(
store="my_store",
query_model=AiModel.BGE_BASE_EN_V15, # 512 tokens
index_model=AiModel.BGE_BASE_EN_V15,
)
  1. Watch batch composition after splitting:

Splitting a large input into several chunks solves the per-input token limit, but if those chunks are then batched together with very different sizes in the same Set call, you can trade one problem for a worse one, see Batched Request Behavior.

Python
# Group similarly-sized chunks together rather than batching
# in original document order
chunks.sort(key=lambda c: c.token_count)

batches = []
current = []
for chunk in chunks:
if chunk.token_count > SOLO_THRESHOLD:
if current:
batches.append(current)
current = []
batches.append([chunk]) # large chunks go out alone
continue
if len(current) >= MAX_BATCH_LEN:
batches.append(current)
current = []
current.append(chunk)
if current:
batches.append(current)

for batch in batches:
client.set(Set(store="docs", inputs=batch))

A reasonable starting point for large_chunk_threshold is somewhere around 15-20% of the model's max token limit, below that, batching together is safe; above it, let the chunk go out alone.


Image Dimension Errors​

Symptoms:

Image Dimensions [(512, 512)] does not match expected [(224, 224)]
Image can't have zero dimension

Cause: Images not matching model requirements (224x224 pixels).

Solutions:

  1. Resize images:
Python
from PIL import Image

def prepare_image(image_path):
img = Image.open(image_path)
img = img.resize((224, 224))
return img.tobytes()

image_bytes = prepare_image("photo.jpg")
  1. Use model preprocessing:
Python
Set(
store="my_store",
inputs=[...],
preprocess_action=PreprocessAction.ModelPreprocessing, # Auto-resize
)
  1. Validate images before sending:
Python
def validate_image(image_bytes):
img = Image.open(io.BytesIO(image_bytes))
if img.width == 0 or img.height == 0:
raise ValueError("Invalid image dimensions")
return img

img = validate_image(image_bytes)

Audio Processing Errors​

Symptoms:

Audio input is too long (15000ms). Model accepts at most 10000ms per clip.
NoPreprocessing is not supported for audio inputs.
Bytes could not be successfully decoded into audio

Causes:

  • Audio clip exceeds 10-second limit
  • Using NoPreprocessing with audio
  • Corrupted or unsupported audio format

Solutions:

  1. Trim audio to 10 seconds:
Python
from pydub import AudioSegment

audio = AudioSegment.from_file("long_audio.wav")
clip = audio[:10000] # First 10 seconds (in milliseconds)
clip.export("trimmed.wav", format="wav")
  1. Split long audio into chunks:
Python
def split_audio(audio_path, chunk_duration_ms=10000):
audio = AudioSegment.from_file(audio_path)
chunks = []
for i in range(0, len(audio), chunk_duration_ms):
chunk = audio[i:i + chunk_duration_ms]
chunks.append(chunk)
return chunks
  1. Always use ModelPreprocessing:
Python
Set(
store="audio_store",
inputs=[...],
preprocess_action=PreprocessAction.ModelPreprocessing, # Required for audio
)
  1. Use supported audio formats:
  • WAV, MP3, FLAC, OGG
  • Audio is automatically resampled to 48kHz

Face Recognition Errors​

Symptoms:

Query input produced 3 embeddings - query input must produce exactly 1 embedding
NoPreprocessing is not supported for face recognition models

Causes:

  • Multiple faces detected in query image
  • Using NoPreprocessing with face models

Solutions:

  1. For queries, use single-face images:
Python
# Crop to single face before querying
from PIL import Image

img = Image.open("group_photo.jpg")
# Crop to region containing target face
face_crop = img.crop((x1, y1, x2, y2))
face_crop.save("single_face.jpg")
  1. Always use ModelPreprocessing:
Python
Set(
store="faces_store",
inputs=[...],
preprocess_action=PreprocessAction.ModelPreprocessing, # Required
)
  1. Adjust confidence threshold for detection:
Python
# For group photos with small faces, lower threshold
GetSimN(
store="faces_store",
search_input=...,
model_params={"confidence_threshold": "0.3"}, # More inclusive
)

# For ID verification, raise threshold
GetSimN(
store="faces_store",
search_input=...,
model_params={"confidence_threshold": "0.9"}, # More strict
)
  1. Choose the right face model:
  • buffalo-l: Higher accuracy, 512-dim, non-commercial only
  • sface-yunet: Lighter, 128-dim, commercially usable (Apache/MIT)

Persistence Issues​

Persistence File Won't Load​

Symptoms:

Failed to load persistence file
Corruption detected

Diagnostic Steps:

  1. Check file permissions:
ls -l /path/to/persistence.dat
  1. Verify file size vs allocator:
# File sizedu -h persistence.dat # Allocator must be >2x file size

Solutions:

  1. Increase allocator size:
# If persistence file is 5 GB, use at least 10 GB allocatorahnlich-db run \  --enable-persistence \  --persist-location /path/to/data.dat \  --allocator-size 10737418240  # 10 GB
  1. Skip corrupted persistence:
ahnlich-db run \  --enable-persistence \  --persist-location /path/to/data.dat \  --fail-on-startup-if-persist-load-fails false  # Continue without persistence
  1. Backup and delete:
# Backupcp persistence.dat persistence.dat.backup # Start freshrm persistence.datahnlich-db run --enable-persistence --persist-location persistence.dat
  1. Check disk space:
df -h /path/to/persistence/

Data Lost After Restart​

Cause: Persistence not enabled.

Solution:

Enable persistence when starting server:

ahnlich-db run \  --enable-persistence \  --persist-location /var/lib/ahnlich/db.dat \  --persistence-interval 300000  # 5 minutes

Debugging Tips​

Enable Detailed Logging​

# Set log levelahnlich-db run --log-level debug # Or specific modulesahnlich-db run --log-level "info,ahnlich_db=debug,hf_hub=warn"

Enable Distributed Tracing​

# Start Jaegerdocker run -d \  -p 16686:16686 \  -p 4317:4317 \  jaegertracing/all-in-one:latest # Start server with tracingahnlich-db run \  --enable-tracing \  --otel-endpoint http://localhost:4317 # View traces at http://localhost:16686

Use CLI for Testing​

# Interactive modeahnlich --agent DB --host 127.0.0.1 --port 1369 # Test commandsPINGINFOSERVERLISTSTORES SCHEMA public

Check Server Health​

# Process statusps aux | grep ahnlich # Resource usagetop -p $(pgrep ahnlich) # Network connectionsnetstat -anp | grep ahnlich # Open fileslsof -p $(pgrep ahnlich)

Getting More Help​

Still having issues? Try these resources:

  1. Check Error Codes: Error Codes Reference
  2. Read Configuration Docs: Configuration Reference
  3. Enable Tracing: See detailed request flow
  4. Community: WhatsApp Group
  5. GitHub: Report Issues

When reporting issues, include:

  • Error messages (full text)
  • Server version
  • Configuration flags used
  • Steps to reproduce
  • Server logs (with --log-level debug)