The Open-Source AI System Control Agent
A Python-based AI agent that controls your desktop via voice commands, hand gestures, and autonomous task execution — running locally on your hardware with optional cloud LLM support.
JARVIS Autonomy (v0.7.1)
Heliox OS evolves from reactive command execution to an invisible, always-on proactive intelligence. The neural models now run completely autonomously in the background.
1. Always-On Screen Awareness
Runs completely in the background via the new ScreenVisionAgent, tracking what you see every few seconds and feeding real-time context to the TRIBE v2 cognitive engine.
from pilot.agents.screen import ScreenVisionAgent
vision = ScreenVisionAgent(interval=3.0)
vision.start_daemon(background=True)2. Proactive Suggestion Engine
Instead of waiting for you to ask, the engine detects bottlenecks and workflow friction, firing proactive suggestions right into your UI based on live OCR.
from pilot.cognitive.proactive import SuggestionEngine
engine = SuggestionEngine(tribe_core)
await engine.analyze_context_and_suggest()3. Non-Blocking Tokenizer Pipeline
The PyTorch architecture and tokenizers are thread-optimized so multi-model proactive inference never freezes your operating system event loops.
from transformers import AutoTokenizer
# Pre-caches inside asynchronous executor
await loop.run_in_executor(None, AutoTokenizer.from_pretrained)4. Autonomous Background Tasks
Delegate an entire multi-step automation process. Heliox spins up a headless environment and reports back to the UI securely upon final execution.
task_id = executor.spawn_headless(job)
progress = get_status(task_id)
print(f"[JARVIS] Task complete: {progress}")Invisible Ambient Processing
By decoupling the user interface from the cognitive orchestrator, JARVIS Autonomy executes perfectly as an invisible backend service. The system gracefully degrades to heuristics if local neural models encounter constraint limits.
# Start the pilot server completely decoupled
server = PilotServer()
server.start()
# Neural predictions flow constantly in the background
logger.info("Always-On Screen awareness daemon active...")
logger.info("[PASS] Screen Vision Stats")Intuitive Command. Infinite Power.
Control your entire digital ecosystem with zero friction.
Voice & Speech
Advanced NLP layer with context retention and multi-language support for complex commands.
Hand Gestures
Vision-based interaction system allowing spatial control of windows and system settings.
Always-On Awareness
Continuous contextual bridging of your screen and environment directly into the LLM logic.
Autonomous Background Jobs
Spawn multi-step, fire-and-forget tasks that resolve silently entirely independent of the main UI.
Modular Cognitive Architecture
Gateway Hub
Central endpoint for multi-modal signal intake (audio, visual, textual).
Planner
Hierarchical task decomposition system that generates execution roadmaps.
Orchestrator
Dynamically assigns sub-tasks to specialized sub-agents.
Execution
Sandboxed environment for API calls, shell commands, and system writes.
Verification
Final state validation using vision and system-logs.
Built for Privacy & Flexibility
Heliox OS offers the ultimate flexibility. Run entirely on local hardware for total privacy, or seamlessly connect to frontier cloud models like Anthropic and Gemini for complex capabilities.
About Me
Hi, I'm Vyom Kulshrestha, a pre-final year Computer Science student at VIT, passionate about building next-generation AI systems and intelligent developer experiences.
I'm currently building Heliox OS — a futuristic AI-powered operating system interface designed to combine autonomous agents, voice interaction, multimodal intelligence, and system-level automation into a seamless computing experience.
My goal is to push this project beyond a traditional software demo and evolve it into a research-grade AI assistant ecosystem that feels like a real-world Jarvis-inspired system.
Whether you're a developer, open-source contributor, recruiter, or someone excited about the future of human-computer interaction, I'd love to connect and hear your thoughts.
Ready to supercharge your OS?
An open-source project built solo, growing with every contributor. Be part of what comes next.