According to Gartner® 42 percent of IT management has either already invested in big data technology or plan to do so in the next year. But given the overuse of the term by everyone from media pundits to IT vendors, will it really impact the network? The short answer is yes.
Within the next two to four years it will drive every aspect of IT, from infrastructure design to network and application management. In this article, we define big data, assess its impact, and provide advice on how network teams can best prepare.
The key factor behind big data is cheap storage to house the data-crunching results of applications that generate huge volumes of unstructured data. This data can include everything from high-definition video streaming to oil and gas exploration metrics to results from genome mapping and pharmaceutical research. For companies involved in these efforts, these activities require vast amounts of compute power and storage, real-time delivery of research, and well-defined access and security protocols. For network operations teams, this demands effective management of capacity, latency, access, and security.
|Challenge||Key Considerations||Network Team Takeaways|
Where do we store all this stuff?
|Handling petabytes of information requires scalable infrastructure with the ability to quickly add or subtract capacity in a manner that’s transparent to users. The most common methods for achieving this are via public Infrastructure-as-a-Service cloud platforms or private cloud with automated resource provisioning.||Tracking big data performance requires visibility into cloud storage, and benchmarking component performance and response time.
Also, confirm that your monitoring solution can track high-speed link performance without dropping packets.
How fast can I have it?
|Real-time delivery of results is key to big data. When serving up custom content to users, instant results are needed. Regardless of web content or number crunching, late delivery directly impacts the bottom line.||To identify bottlenecks means having a performance platform capable of providing integrated views of your overall big data services. Upon pinpointing choke points, infrastructure can either be optimized or upgraded.|
Who needs the data? Are there database compatibility issues that prevent comparisons between data sets?
|Big data often draws new parties into the data sharing loop. First determine who needs authorization to run queries or processes. The new consideration is whether data sets held on multiple backend storage platforms can be compared and correlated. To address this, the storage team will deploy some type of global file system.||Utilize retrospective analysis to validate on-time resource access by authorized users and to track performance and potential breakdowns between database systems.|
How do regulations impact data handling and storage?
|First, privacy and security regulations impact how financial, medical, and government information can be housed and accessed. The co-mingling and comparison of sensitive data sets will create new headaches for IT managers who will need to track how this new intelligence is shared and utilized. And as data becomes largely created or housed externally, security policies will need to be drafted to take this into account.||Stream-to-disk technologies like GigaStor complement existing security tools by acting as a network surveillance camera. Applying stream reconstruction and network forensics to relevant time periods allow you to investigate, verify, and document malicious or accidental security breaches.|
Big Data Resources
For more in-depth information on rolling out big data initiatives, addressing storage and security concerns, and key monitoring challenges, check out the following resources:
- Big data: Start small but quickly scale
- Looking into new databases behind big data
- Ensuring big data security and authenticity
- Implementing big data
- Choosing big data infrastructure (requires login)
- Pros and cons: The big data business perspective
Thanks to Network Instruments for this Article