Visibility and Control Challenges Underscore the Need for Disciplined Dataflow Operations
Management practices for dataflow operations are immature — data stores are managed, but end-to-end dataflows are not. This creates operational blindness which, when combined with the insidious data drift to which big data is prone, leads to constant firefighting of dataflow breakdowns and data pollution in downstream applications. The problem becomes intractable with the increasing complexity and scale of the dataflows that drive today’s enterprise.
The challenges of operating dataflows were highlighted by the results of a recent big data research study conducted by Dimensional Research. In a survey of over 300 data professionals, only 12% rate themselves as “Good” or “Excellent” across five key dataflow operational performance areas. Separately, 72% desire a solution providing comprehensive visibility and management across all of their dataflows.
StreamSets Dataflow Performance Manager uniquely addresses the pain enterprises experience when they try to manage a multitude of dataflows manually or with legacy technologies in the face of data drift and dynamic use cases,” said Girish Pancha, CEO, StreamSets. “Data is a key asset that requires the same performance management practices that have been adopted for networks and applications. DPM brings operational intelligence to dataflow management so enterprises can be sure that the data driving their key applications is always timely and trustworthy.”
New Solution Serves as Foundation for the Data Operations Center
By providing a comprehensive control panel, StreamSets Dataflow Performance Manager is the technology foundation for enterprises looking to establish a data operations center (DOC), a center of excellence for managing data in motion together with data at rest. Similar to a NOC for network operations, the DOC coordinates across functions and business units to help evolve piecemeal data integration processes into well-governed data operations.
A center of excellence for managing data operations is an idea whose time has come,” said Stewart Bond, research director of data integration software at IDC. “Enterprise data movement has become very complex and time-sensitive. It is no longer adequate to solely manage data at rest. The StreamSets concept of a Data Operations Center will focus resources and attention on managing the integrity of enterprise data, helping to assure trust, availability, security and compliance of data as it traverses a dynamic web of data flows.”
StreamSets Dataflow Performance Manager Delivers Operational Excellence
StreamSets Dataflow Performance Manager is the first data management software solution that allows enterprises to harness their data in motion. StreamSets DPM™ unifies visibility and control of dataflows, which reduces management costs, improves data quality and enables IT agility.
With DPM, customers can:
- Map dataflows to topologies and business processes, manage releases and track changes in topologies over time.
- Measure and establish baselines for end-to-end and point-in-flow KPIs for data availability and accuracy.
- Master dataflow operations by creating Data SLAs (service level agreements) and detecting and remediating violations.
StreamSets Data Performance Manager will be generally available later this month, initially as a cloud-based service, with special pricing available for early customers.
Sign up for the free insideBIGDATA newsletter.