Troubleshooting Common Issues
This guide covers the most common issues users encounter and how to resolve them.
Memory and Performance Issues
Out of Memory Errors
Symptoms:
allocation error: CapacityOverflow
Server crashes unexpectedly
Causes:
- Hitting the
--allocator-sizelimit - Large batch operations
- Image processing without streaming enabled
- Batching inputs of very different sizes together. A single large input batched alongside several small ones forces the small ones to be padded up to (and computed at) the large input's length. See Batched Request Behavior for why, and the batching snippet under Token Limit Exceeded for a fix.
Solutions:
- Increase allocator size:
ahnlich-db run --allocator-size 21474836480 # 20 GiB (default is 10 GiB)ahnlich-ai run --allocator-size 21474836480- Enable streaming for images (AI proxy):
ahnlich-ai run --enable-streaming # 10x less memory, 40% slower- Reduce batch sizes:
# Instead of:
large_batch = [entry1, entry2, ..., entry1000]
client.set(Set(store="my_store", inputs=large_batch))
# Do this:
batch_size = 100
for i in range(0, len(large_batch), batch_size):
batch = large_batch[i:i+batch_size]
client.set(Set(store="my_store", inputs=batch))
- Monitor memory usage:
# Check process memoryps aux | grep ahnlich # Monitor with toptop -p $(pgrep ahnlich)Slow Query Performance
Symptoms:
- Queries taking longer than expected
- High CPU usage
Diagnostic Steps:
- Enable tracing to identify bottlenecks:
ahnlich-db run --enable-tracing --otel-endpoint http://localhost:4317View traces in Jaeger UI at http://localhost:16686
- Check store size:
INFOSERVER
- Verify algorithm choice:
- Linear algorithms (Cosine, Euclidean, DotProduct) scale linearly with data size
- Use
HNSWfor faster searches with large datasets:
CREATESTORE my_store DIMENSION 128 NONLINEARALGORITHMINDEX (HNSW)
Solutions:
- Use predicate indices for filtering:
# Index frequently filtered fields
CREATEPREDINDEX (category, author) IN my_store
# Then filter efficiently
GETPRED (category = science) IN my_store
- Optimize batch operations:
# Batch SET operations
entries = [entry1, entry2, ..., entry100]
client.set(Set(store="my_store", inputs=entries))
- Use appropriate similarity algorithm:
- CosineSimilarity: Best for normalized vectors, direction-based similarity
- EuclideanDistance: Best for absolute distance measures
- DotProduct: Fast when vectors are pre-normalized
- HNSW: Best for large-scale approximate nearest neighbor searches
- Adjust thread pool size:
ahnlich-db run --threadpool-size 32 # Default: 16Connection Issues
Cannot Connect to Server
Symptoms:
connection refused
Failed to dial server
Transport issues with tonic
Diagnostic Steps:
- Check if server is running:
# Check DBcurl http://localhost:1369 # or use telnetps aux | grep ahnlich-db # Check AIcurl http://localhost:1370ps aux | grep ahnlich-ai- Verify port availability:
# Check if port is in uselsof -i :1369lsof -i :1370 # Or with netstatnetstat -tuln | grep 1369- Check firewall rules:
# Ubuntu/Debiansudo ufw statussudo ufw allow 1369sudo ufw allow 1370 # CentOS/RHELsudo firewall-cmd --list-allsudo firewall-cmd --add-port=1369/tcp --permanentsudo firewall-cmd --reloadSolutions:
- Start server on all interfaces:
# Allow connections from any IPahnlich-db run --host 0.0.0.0 --port 1369ahnlich-ai run --host 0.0.0.0 --port 1370- Check host/port configuration:
# Correct
client = DbClient("http://127.0.0.1:1369")
# Wrong - missing protocol
client = DbClient("127.0.0.1:1369") # Invalid URI error
- Verify network connectivity:
# Test connectivityping <server-host>telnet <server-host> 1369Maximum Clients Reached
Symptoms:
Max Connected Clients Reached
Connection rejected
Cause: Hit the --maximum-clients limit (default: 1000)
Solutions:
- Increase client limit:
ahnlich-db run --maximum-clients 5000- Check current connections:
LISTCLIENTS
- Implement connection pooling:
# Reuse connections instead of creating new ones
class ClientPool:
def __init__(self, uri, pool_size=10):
self.pool = [DbClient(uri) for _ in range(pool_size)]
self.index = 0
def get_client(self):
client = self.pool[self.index]
self.index = (self.index + 1) % len(self.pool)
return client
- Close idle connections:
async def cleanup():
await client.close()
AI Proxy Cannot Connect to Database
Symptoms:
Proxy Errored with connection refused
DatabaseClientError
Diagnostic Steps:
- Verify DB is running:
ps aux | grep ahnlich-db- Check DB host/port:
# See what DB is listening onnetstat -tuln | grep 1369Solutions:
- Start DB before AI:
# Terminal 1ahnlich-db run --port 1369 # Terminal 2 (wait for DB to start)ahnlich-ai run --db-host 127.0.0.1 --db-port 1369- Verify connection settings:
# If DB is on different hostahnlich-ai run --db-host 192.168.1.10 --db-port 1369 # If DB uses non-default portahnlich-ai run --db-port 1400- For standalone mode (no DB):
ahnlich-ai run --without-db- Adjust connection pool:
ahnlich-ai run --db-client-pool-size 20 # Default: 10Data and Store Issues
Store Not Found
Symptoms:
Store "my_store" not found
Diagnostic Steps:
- List stores in the relevant schema:
LISTSTORES SCHEMA public
- Check store name spelling:
# Store names are case-sensitive
"MyStore" ≠ "mystore"
Solutions:
- Create the store:
# DB
CREATESTORE my_store DIMENSION 128
# AI
CREATESTORE my_store QUERYMODEL all-minilm-l6-v2 INDEXMODEL all-minilm-l6-v2
- Check persistence loaded:
# If using persistenceahnlich-db run \ --enable-persistence \ --persist-location /path/to/data.dat \ --fail-on-startup-if-persist-load-fails true # Fail loudly if load fails- Verify correct server:
# Make sure you're connecting to the right instance
client = DbClient("http://localhost:1369") # Not a different instance
Dimension Mismatch Errors
Symptoms:
Store dimension is [128], input dimension of [256] was specified
Cause: Vector dimensions don't match store configuration.
Solutions:
- Check store dimension:
INFOSERVER
# Look at store details
- For AI stores, verify model dimensions:
| Model | Embedding Dimension |
|---|---|
| all-minilm-l6-v2 | 384 |
| all-minilm-l12-v2 | 384 |
| bge-base-en-v1.5 | 768 |
| bge-large-en-v1.5 | 1024 |
| resnet-50 | 2048 |
| clip-vit-b32-* | 512 |
| clap-audio / clap-text | 512 |
| buffalo-l | 512 |
| sface-yunet | 128 |
- Match query and index models:
# Both must have same dimensions
CreateStore(
store="my_store",
query_model=AiModel.BGE_BASE_EN_V15, # 768-dim
index_model=AiModel.BGE_BASE_EN_V15, # 768-dim (same)
)
- Recreate store with correct dimension:
DROPSTORE my_store IFTRUE
CREATESTORE my_store DIMENSION 768
Predicate Not Found
Symptoms:
Predicate "author" not found in store
Cause: Querying by a predicate that wasn't indexed.
Solutions:
- Create predicate index:
CREATEPREDINDEX (author, category) IN my_store
- Or include when creating store:
CREATESTORE my_store DIMENSION 128 PREDICATES (author, category)
- Verify predicates exist:
INFOSERVER
# Check store predicates
Model and AI Issues
Model Not Loading
Symptoms:
index_model or query_model not selected or loaded
Error initializing a model thread
Tokenizer for model failed to load
Diagnostic Steps:
- Check supported models:
ahnlich-ai run --supported-models all-minilm-l6-v2,resnet-50- Verify model cache:
# Default locationls -la ~/.ahnlich/models # Custom locationahnlich-ai run --model-cache-location /path/to/models- Check disk space:
df -h ~/.ahnlich/models- Test network connectivity:
# Models download from HuggingFacecurl https://huggingface.coSolutions:
- Wait for initial download:
# First time loading a model downloads from HuggingFace# This can take several minutes depending on model size# Watch logs for progress- Clear corrupted cache:
rm -rf ~/.ahnlich/models/model_name# Restart server to re-download- Increase idle time:
# Keep models loaded longerahnlich-ai run --ai-model-idle-time 600 # 10 minutes (default: 5 min)- Pre-download models:
# Download models before starting serverpython -c "from transformers import AutoModel; AutoModel.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')"Token Limit Exceeded
Symptoms:
Max Token Exceeded. Model Expects [256], input type was [512]
Cause: Text input exceeds model's token limit.
Token Limits:
- all-minilm-*: 256 tokens
- bge-*: 512 tokens
- clip-vit-b32-text: 77 tokens
- clap-text: 512 tokens
- jina-embeddings-v2-base-code: 8192 tokens
Solutions:
- Truncate text:
def truncate_text(text, max_length=200):
words = text.split()
return ' '.join(words[:max_length])
text = truncate_text(long_text)
- Split into chunks:
def chunk_text(text, chunk_size=200):
words = text.split()
return [' '.join(words[i:i+chunk_size])
for i in range(0, len(words), chunk_size)]
chunks = chunk_text(long_document)
for chunk in chunks:
client.set(Set(store="docs", inputs=[...]))
- Use model with larger limit:
# Switch from AllMiniLM (256) to BGE (512)
CreateStore(
store="my_store",
query_model=AiModel.BGE_BASE_EN_V15, # 512 tokens
index_model=AiModel.BGE_BASE_EN_V15,
)
- Watch batch composition after splitting:
Splitting a large input into several chunks solves the per-input token
limit, but if those chunks are then batched together with very different
sizes in the same Set call, you can trade one problem for a worse one,
see Batched Request Behavior.
# Group similarly-sized chunks together rather than batching
# in original document order
chunks.sort(key=lambda c: c.token_count)
batches = []
current = []
for chunk in chunks:
if chunk.token_count > SOLO_THRESHOLD:
if current:
batches.append(current)
current = []
batches.append([chunk]) # large chunks go out alone
continue
if len(current) >= MAX_BATCH_LEN:
batches.append(current)
current = []
current.append(chunk)
if current:
batches.append(current)
for batch in batches:
client.set(Set(store="docs", inputs=batch))
A reasonable starting point for large_chunk_threshold is somewhere around
15-20% of the model's max token limit, below that, batching together is safe;
above it, let the chunk go out alone.
Image Dimension Errors
Symptoms:
Image Dimensions [(512, 512)] does not match expected [(224, 224)]
Image can't have zero dimension
Cause: Images not matching model requirements (224x224 pixels).
Solutions:
- Resize images:
from PIL import Image
def prepare_image(image_path):
img = Image.open(image_path)
img = img.resize((224, 224))
return img.tobytes()
image_bytes = prepare_image("photo.jpg")
- Use model preprocessing:
Set(
store="my_store",
inputs=[...],
preprocess_action=PreprocessAction.ModelPreprocessing, # Auto-resize
)
- Validate images before sending:
def validate_image(image_bytes):
img = Image.open(io.BytesIO(image_bytes))
if img.width == 0 or img.height == 0:
raise ValueError("Invalid image dimensions")
return img
img = validate_image(image_bytes)
Audio Processing Errors
Symptoms:
Audio input is too long (15000ms). Model accepts at most 10000ms per clip.
NoPreprocessing is not supported for audio inputs.
Bytes could not be successfully decoded into audio
Causes:
- Audio clip exceeds 10-second limit
- Using
NoPreprocessingwith audio - Corrupted or unsupported audio format
Solutions:
- Trim audio to 10 seconds:
from pydub import AudioSegment
audio = AudioSegment.from_file("long_audio.wav")
clip = audio[:10000] # First 10 seconds (in milliseconds)
clip.export("trimmed.wav", format="wav")
- Split long audio into chunks:
def split_audio(audio_path, chunk_duration_ms=10000):
audio = AudioSegment.from_file(audio_path)
chunks = []
for i in range(0, len(audio), chunk_duration_ms):
chunk = audio[i:i + chunk_duration_ms]
chunks.append(chunk)
return chunks
- Always use ModelPreprocessing:
Set(
store="audio_store",
inputs=[...],
preprocess_action=PreprocessAction.ModelPreprocessing, # Required for audio
)
- Use supported audio formats:
- WAV, MP3, FLAC, OGG
- Audio is automatically resampled to 48kHz
Face Recognition Errors
Symptoms:
Query input produced 3 embeddings - query input must produce exactly 1 embedding
NoPreprocessing is not supported for face recognition models
Causes:
- Multiple faces detected in query image
- Using
NoPreprocessingwith face models
Solutions:
- For queries, use single-face images:
# Crop to single face before querying
from PIL import Image
img = Image.open("group_photo.jpg")
# Crop to region containing target face
face_crop = img.crop((x1, y1, x2, y2))
face_crop.save("single_face.jpg")
- Always use ModelPreprocessing:
Set(
store="faces_store",
inputs=[...],
preprocess_action=PreprocessAction.ModelPreprocessing, # Required
)
- Adjust confidence threshold for detection:
# For group photos with small faces, lower threshold
GetSimN(
store="faces_store",
search_input=...,
model_params={"confidence_threshold": "0.3"}, # More inclusive
)
# For ID verification, raise threshold
GetSimN(
store="faces_store",
search_input=...,
model_params={"confidence_threshold": "0.9"}, # More strict
)
- Choose the right face model:
buffalo-l: Higher accuracy, 512-dim, non-commercial onlysface-yunet: Lighter, 128-dim, commercially usable (Apache/MIT)
Persistence Issues
Persistence File Won't Load
Symptoms:
Failed to load persistence file
Corruption detected
Diagnostic Steps:
- Check file permissions:
ls -l /path/to/persistence.dat- Verify file size vs allocator:
# File sizedu -h persistence.dat # Allocator must be >2x file sizeSolutions:
- Increase allocator size:
# If persistence file is 5 GB, use at least 10 GB allocatorahnlich-db run \ --enable-persistence \ --persist-location /path/to/data.dat \ --allocator-size 10737418240 # 10 GB- Skip corrupted persistence:
ahnlich-db run \ --enable-persistence \ --persist-location /path/to/data.dat \ --fail-on-startup-if-persist-load-fails false # Continue without persistence- Backup and delete:
# Backupcp persistence.dat persistence.dat.backup # Start freshrm persistence.datahnlich-db run --enable-persistence --persist-location persistence.dat- Check disk space:
df -h /path/to/persistence/Data Lost After Restart
Cause: Persistence not enabled.
Solution:
Enable persistence when starting server:
ahnlich-db run \ --enable-persistence \ --persist-location /var/lib/ahnlich/db.dat \ --persistence-interval 300000 # 5 minutesDebugging Tips
Enable Detailed Logging
# Set log levelahnlich-db run --log-level debug # Or specific modulesahnlich-db run --log-level "info,ahnlich_db=debug,hf_hub=warn"Enable Distributed Tracing
# Start Jaegerdocker run -d \ -p 16686:16686 \ -p 4317:4317 \ jaegertracing/all-in-one:latest # Start server with tracingahnlich-db run \ --enable-tracing \ --otel-endpoint http://localhost:4317 # View traces at http://localhost:16686Use CLI for Testing
# Interactive modeahnlich --agent DB --host 127.0.0.1 --port 1369 # Test commandsPINGINFOSERVERLISTSTORES SCHEMA publicCheck Server Health
# Process statusps aux | grep ahnlich # Resource usagetop -p $(pgrep ahnlich) # Network connectionsnetstat -anp | grep ahnlich # Open fileslsof -p $(pgrep ahnlich)Getting More Help
Still having issues? Try these resources:
- Check Error Codes: Error Codes Reference
- Read Configuration Docs: Configuration Reference
- Enable Tracing: See detailed request flow
- Community: WhatsApp Group
- GitHub: Report Issues
When reporting issues, include:
- Error messages (full text)
- Server version
- Configuration flags used
- Steps to reproduce
- Server logs (with
--log-level debug)