When people think about digital disruption they usually think of the peer-to-peer accommodation network AirBnB, or the inexpensive ride-sharing app Uber. These businesses have redefined their respective markets – with big data analytics1 underpinning their success.
Despite the fear that disruptive tech will bring with it new threats to security, Australia’s national security has much to benefit from the type of disruption brought about by big data – particularly when it comes to fighting terrorism and crime.
The national security sector faces the most imminent and complex big data challenges. This is because a powerful weapon of today’s terrorist or criminal is their ability to hide in data. They can plan and coordinate an attack or crime with impunity.
The ability for criminals to “hide in data” means that national security agencies are often faced with the daunting task of finding the “needle in the haystack” – where the haystack is growing at a phenomenal rate. In fact, people often comment that national security data analysts are “drowning in data, but starving for information”.
Big data analysts often need to find connections in vast, disparate volumes of data, where connections are imperceptible to humans but can be discovered using smart analytics and machine enablement.
The challenge is made greater by the wide variety of data sources (e.g. texts, voices, images, videos), the ever-increasing size and scale of the data collected, and the organisational and legislative silos impacting data agencies.
The effect of big data means that national security data analysts often spend most of their time collecting data, formatting it for analysis and generating reports, and less of their time doing the analysis. This is referred to as the “bathtub curve”.
The application of big data analytics is aimed at “inverting the bathtub”, which means automating the collection and processing of data to form intelligence. The generation of intelligence reports can also be automated via digital technologies, which enables analysts to spend more time analysing intelligence and making decisions.
The D2D CRC is developing applications to maximise the benefits that Australia’s national security sector can extract from Big Data. They are helping agencies generate timely and accurate intelligence as a powerful weapon against national security threats.
By addressing their big data challenges and applying high-performance analytics, the D2D CRC hopes it can support agencies in predicting threats rather than reacting to catastrophic aftermath.
CEO, Data to Decisions CRC
Read next: Victoria’s Lead Scientist, Dr Amanda Caples, reveals the major flaw in traditional government approaches to disruption.
Spread the word: Help Australia become digital savvy nation! Share this piece on digital disruptors using the social media buttons below.
1 Big data is a term for any collection of data sets so large and complex they becomes difficult to store, process and analyse using current technologies. Big data analytics is the process of examining these data sets to uncover hidden patterns, unknown correlations, trends and other useful business information.