Agentic Data Transfer Optimizer
Tuning bulk data transfers from live TCP telemetry, with the agent's actions kept inside hard limits.
- Status
- In progress · prototype built, evaluation under way
- Built with
- Python, Zero-copy sendfile(), Multiprocessing, Ollama, Hugging Face, Pydantic
The problem
Bulk data movers usually run with a hand-tuned, static concurrency setting. The right number of parallel streams depends on the network path, competing traffic, and host load, all of which change during a long transfer.
How it works
A sender and receiver move data over TCP through a multiprocessing worker pool with zero-copy sendfile() transfers. At each probing interval the system observes the transfer, two agents reason about it, and the runtime acts on a bounded recommendation.
- ObserveRead TCP socket statistics, congestion window, retransmissions, RTT variance, and host load.
- DiagnoseThe Diagnostic agent turns observations and recent history into a structured assessment.
- DecideThe Decision agent recommends a concurrency change or holds the current setting.
- EnforceHard limits, bounded deltas, graduated ramping, and oscillation damping shape the action.
↻ Each action is evaluated at the next interval, closing an observe, reason, act, evaluate loop.
Design choices
- The model recommends and the system enforces. No agent output can push concurrency outside the permitted range.
- Repeated reversals are detected and damped before the next action, and large changes are ramped in steps.
- Structured memory and a decision journal record what each agent saw and chose.
- The model layer is pluggable across local Ollama models, fine-tuned Hugging Face checkpoints, OpenAI, and Google Gemini.
Open questions
- Can agent decisions stay stable as telemetry changes, without oscillating?
- When does adaptive concurrency beat a well-chosen static setting, on which paths and workloads?
The prototype is built and evaluation is under way. Results will appear here once controlled comparisons are complete.