Public security notice: This documentation is intentionally redacted. Sensitive server paths, private keys, secret tokens, and origin network details are removed.
Data Fetching & Caching¶
This page explains how the engine talks to external APIs, honors rate limits and uses multiple layers of caching and fallbacks to keep runs fast and stable.
Legacy cache helpers¶
Near the top of defi_complete_risk_assessment_clean.py, load_cached_data
implements a basic JSON-based cache for tokens. It defers to cache_manager, a
module-level global that stays None unless the enhanced cache manager imports:
python
def load_cached_data(token_address):
"""Load cached data for a token, fallback to real-time if not available"""
if cache_manager:
return cache_manager.get_cached_data(token_address)
...
cache_file = os.path.join(DATA_DIR, 'real_data_cache.json')
...
if os.path.exists(cache_file):
with open(cache_file, 'r') as f:
cache_data = json.load(f)
...
if token_address in tokens:
cached_data = tokens[token_address]
...
if cache_age_hours < 2:
return cached_data
The matching writer, update_cache_with_real_time_data, updates this JSON cache
and triggers a local webhook:
python
def update_cache_with_real_time_data(token_address, real_time_data):
"""Update cache with real-time data"""
if cache_manager:
cache_manager.update_cache_with_real_time_data(token_address, real_time_data)
return
...
cache_file = os.path.join(DATA_DIR, 'real_data_cache.json')
...
cache_data['tokens'][token_address] = real_time_data
cache_data['last_updated'] = time.time()
...
payload = {'address': token_address, 'token_address': token_address}
payload_bytes = json.dumps(payload, ...).encode('utf-8')
requests.post(
f'{WEBHOOK_BASE_URL}/webhook/update_token',
data=payload_bytes,
...
timeout=5,
)
This legacy layer is still used as a fallback when the enhanced cache manager is not available.
Intelligent cache wrapper¶
The higher-level helper fetch_data_with_cache_fallback encapsulates the main
fetch strategy:
python
def fetch_data_with_cache_fallback(token_address, fetch_function):
"""Fetch data with priority-based strategy:
1. Run assessment -> API works, no rate limitation -> Obtain real-time data and use it
2. Run assessment -> API works, rate limitation -> Get data until rate limited, then use fallback
3. Run assessment -> API does not work -> fetch fallback data directly
"""
if cache_manager:
return cache_manager.fetch_data_with_intelligent_cache(token_address, fetch_function)
...
cached_data = load_cached_data(token_address)
if cached_data:
return cached_data
...
real_time_data = fetch_function(token_address)
if real_time_data:
update_cache_with_real_time_data(token_address, real_time_data)
return real_time_data
...
if "rate limit" in str(e).lower() or "429" in str(e):
# Priority 2: API works but rate limited
...
If an enhanced cache_manager is provided, the function hands control to
fetch_data_with_intelligent_cache, which can apply additional policies such as:
- cache retention windows per metric,
- metric-drift thresholds (only refresh when values move by X%),
- provenance tracking (live vs cache vs fallback).
Disk-backed APICache¶
The engine also defines APICache, a wrapper that stores values in a
diskcache.Cache under DATA_DIR:
python
class APICache:
def __init__(self, filename='api_cache.db'):
self.filename = os.path.join(DATA_DIR, filename)
self.cache = None
...
self.cache = Cache(self.filename, disk=JSONDisk)
...
self.cache.set('test', 'test', expire=1)
test_result = self.cache.get('test')
Methods like get, set and close wrap calls to diskcache.Cache with
defensive error handling so that cache failures never break a run.
DeFiRiskAssessor opens an APICache the first time it fetches CoinMarketCap
data, but no current code path reads or writes values through it.
HTTP request policy & rate-limit tracking¶
The HTTP layer goes beyond naive requests.get. Requests go through
robust_request, which applies a structured request policy and
per-service rate-limit state.
Key pieces:
_service_rate_limit_state: in-memory dict keyed by service name.HTTP_REQUEST_STATE_FILE: JSON on disk (http_request_state.json) used to persist ETag / Last-Modified and small response bodies._service_toggle_cache: per-service enable/disable toggles, loaded fromapi_runtime/service_toggles.jsonunderDATA_DIR. Rate-limit policy tweaks come fromsettings.json.
Request policy loader, _load_request_policy_settings (simplified):
python
def _load_request_policy_settings():
"""Load API/cache request policy from settings with short in-memory TTL."""
...
api_cfg = loaded_settings.get('api', {}) if isinstance(loaded_settings, dict) else {}
cache_cfg = loaded_settings.get('cache', {}) if isinstance(loaded_settings, dict) else {}
policy = {
'rate_limiting': bool(api_cfg.get('rate_limiting', True)),
'conditional_requests': bool(api_cfg.get('conditional_requests', True)),
'adaptive_backoff': bool(api_cfg.get('adaptive_backoff', True)),
...
'metric_drift_threshold_pct': max(0.0, float(cache_cfg.get('metric_drift_threshold_pct', 2.0) or 2.0)),
}
_request_policy_cache['ts'] = now_ts
_request_policy_cache['policy'] = policy
return policy
Before issuing a request, robust_request asks _preemptive_rate_limit_gate
whether the service still has quota according to the last rate-limit headers it
returned. When the quota is exhausted and the reset time is still ahead, the
engine waits briefly or returns a synthetic rate-limited (429) response instead
of hitting the network again.
After each response, _update_service_rate_limit_state records that service’s
remaining quota and reset time from its rate-limit headers, including
Retry-After when present.
Conditional requests & cached bodies¶
To reduce bandwidth and avoid burning rate limits on unchanged data, the HTTP layer supports conditional requests:
- Stores ETag and Last-Modified per request.
- Sends
If-None-Match/If-Modified-Sincewhen re-requesting. - On a
304 Not Modified, reuses the cached body (if stored).
Cached conditional entries live in HTTP_REQUEST_STATE_FILE, keyed by a hash of
the request URL and parameters. Each entry contains:
- ETag / Last-Modified headers,
- the response content type,
- optionally a small
cached_bodyfor JSON payloads under a size threshold, - the time it was saved.
When a 304 Not Modified arrives and the entry holds a cached body, the engine
returns a synthetic 200 response carrying that body.
Webhook integration¶
Whenever new real-time token data is written to the legacy JSON cache, the engine optionally triggers a local webhook:
- URL:
/webhook/update_tokenon the configured webhook base URL, which defaults tohttp://localhost:5001 - Payload:
{"address": "<address>", "token_address": "<address>"} - Timeout: 5 seconds, best-effort only.
This allows:
- lightweight dashboards to update without polling the filesystem,
- incremental refresh of a subset of tokens after a long run.
Consumers are free to ignore this webhook or replace it with a different notification mechanism.
Summary¶
Data fetching and caching in the Hodler Suite are intentionally conservative:
- Always prefer cached or conditional data when fresh enough.
- Automatically back off when rate limits or transient errors occur.
- Persist enough state on disk to make subsequent runs cheaper and more stable.
- Expose configuration in
settings.jsonso operators can tune the balance between freshness, cost and rate-limit pressure.