Unstructured data management firm Komprise has developed a patented technology capable of dynamically splitting massive unstructured datasets. This solution distributes workloads across multiple compute engines during data transmission to AI processing GPUs, significantly accelerating overall processing speed.
Officially recorded as patent US-12566637-B2, the Komprise Elastic Shares (KES) technology was submitted by Komprise CTO Michael Peercy and his research team. Its formal title is “System and methods for subdividing an unknown tree for execution of operations by multiple compute engines.”
Peercy stated: “Elastic Shares enables our clients to fully leverage valuable computing, memory and network resources. The technology delivers near-linear performance scaling, helping enterprises gain prominent competitive advantages.”
For petabyte-scale datasets transmitted to AI servers for analytics, LLM inference or AI agent computation, a single compute engine requires hours to traverse directory trees or object storage prefixes. A compute engine can be defined as an independent server, individual thread, thread group, process or process cluster, with consistent operational principles in all scenarios. Although users can deploy multiple engines for parallel computing in advance, static allocation often causes inefficiency. Uneven branch sizes result in certain compute nodes finishing early and remaining idle while waiting for slower nodes to complete assigned tasks.
Komprise patent US-12566637-B2.
Peercy’s team invented an intelligent job scheduler embedded within one of the compute engines to monitor node status in real time. When a data processing task is initiated, workloads are initially allocated across available compute engines. Once any node finishes its assigned partition, it returns to the resource pool to receive pending tasks. This mechanism eliminates idle time, continuously assigns new workloads to free compute nodes, and shortens the overall processing duration.
According to an official Komprise blog post, KES dynamically redistributes unstructured data workloads across server clusters in a streaming manner. The technology achieves near-linear speedup for large-scale data processing without requiring pre-acquired information regarding dataset capacity, internal structure or processing latency.
To put it simply: every computing node automatically receives new tasks immediately after finishing current assignments, maintaining full resource utilization until the entire data job is completed.
Michael Peercy.
The patent documentation indicates that large-scale data migration tasks between file servers or storage volumes often face tight time constraints. A single compute node, whether a thread, process or standalone server, lacks sufficient throughput to finish heavy workloads rapidly, making multi-engine parallel processing an essential solution.
The document further explains that directory tree structures can only be identified during recursive traversal from root to leaf nodes. In some cases, data must be processed in a fixed sequence, such as prioritizing parent directories before subfolders and files. This requires real-time dynamic workload segmentation instead of rigid static partitioning, which must adapt to diverse and unknown data architectures.
This patented technology applies to files, folders, storage objects and object prefixes. Although originally designed for independent physical servers, it is compatible with all forms of compute engines, including threads, processes and backend service clusters.
Komprise highlights three core advantages brought by the KES technology:
-
Dynamic task partitioning: High-value computing resources are instantly reassigned with new tasks once they become idle.
-
Blind data processing: The system supports dataset computation without prior awareness of data size, internal structure and variable processing latency, perfectly fitting AI streaming data transmission scenarios.
-
Intelligent resource rebalancing: It automatically optimizes allocation strategies to adapt to unstructured data hierarchies with unpredictable branch densities.
This innovative patented technology delivers practical and efficient optimization for large-scale unstructured data computing.
Beijing Qianxing Jietong Technology Co., Ltd.
Sandy Yang/Global Strategy Director
WhatsApp / WeChat: +86 13426366826
Email: yangyd@qianxingdata.com
Website: www.qianxingdata.com/www.storagesserver.com
Business Focus:
ICT Product Distribution/System Integration & Services/Infrastructure Solutions
With 20+ years of IT distribution experience, we partner with leading global brands to deliver reliable products and professional services.
“Using Technology to Build an Intelligent World”Your Trusted ICT Product Service Provider!
Your message must be between 20-3,000 characters!
English
