We are currently living in the Fourth Industrial Revolution where technology is changing the way we live and interact with those around us.
One of the by-products of this revolution is the shared economy. This is the interaction over social media and data collection apps whereby a company has access to an online digital footprint of information that is created by users. This is known as Big Data, and it has the ability to change the world.
We have all heard of, or have been in contact with, Big Data in some way. But are we aware of how complex Big Data actually is?
Platforms for transformation
Access to this information is like gold for companies. It not only provides insight into consumer behaviour, but it also allows companies to research ways to improve their own business processes.
A recent article on crn.com shows that AtScale’s software provides a way for users of mainstream business analytics and visualization software, such as Tableau and QlikView, to tap into the huge volumes of data stored in big data systems like Hadoop.
AtScale 5.0 provides a new multidimensional calculation engine that supports the MDX query language for modelling complex business processes, a performance optimization engine that uses machine learning to optimize query performance, and a data abstraction layer that provides access to relational and other on premise and cloud data sources.
Compose 3.0 is the latest release of Attunity’s agile data warehouse automation software that helps businesses speed up analytics projects, optimize development and ETL (extract, transform and load) processes, and reduce risk.
The highlight of the 3.0 edition is a series of significant enhancements to the software’s ETL capabilities resulting in a tenfold increase in ETL processing speeds. New advanced DevOps processes (in development, testing, acceptance and production) streamline data warehouse design, development a
New opportunities
Big Data has not only enriched the lives of companies, who now have access to data and systems processes that can improve their business, and customers, who can now be offered tailor made products and services based on their data history, it is also enriching the lives of those looking to expand into a career in technology.
Data scientist…One part statistician and one part software engineer, a data scientist is the brains behind data interpretation.
Data scientists take massive amounts of data and whittle it down into concise statistics to use in predictive and prescriptive modelling. From there, the data scientist can generate valuable insights that improve business performance.
Big data managers…Managing data architects, scientists and a full team of technicians – not to mention managing the actual data – requires a data manager to oversee all processes and communicate day-to-day operations to company leadership. A data manager knows the intricacies of the tech world as well as the overall corporate strategy. They’re the jacks of all trades, if not the master of all.
A data manager is responsible for organizing the data being produced, ensuring quality, implementing strategy, and then reporting back to leadership. They leverage insights provided by data scientists to recommend new products, predict future roadblocks and streamline business logistics. In addition to experience in Hadoop and Hive, this high-level role requires cross-functional skills such as programming and management. As companies continue to adapt to newer systems, the demand for someone to coordinate data processes will continue to increase as well.
More value
The topic of Big Data is so vast that companies can find value in just about any aspect that is inherently attached to it; provided that companies realise that Big Data is more than just about analysing, it’s about execution.
An interesting article pointed out that at a recent think tank held in Singapore, Dr Andreas Weigend, Director at Social Data Lab, said that data didn’t have to be a cost to the company, but could instead be a profit and revenue generator.
Big data has been a key buzzword for businesses for some time now. While many people believe big data can be beneficial, few organizations know how to fully maximize its potential. It continues to evolve becoming a driving force behind waves of digital transformation.
Dr Weigend said that successful companies succeed by removing barriers to information rather than raising them. But, if you know all of the data you can make better decisions.
“We understand the essence and importance of big data, and saw that there is a disparity in knowing and understanding it,” said Yaron Zeidman, CTO, Agoda. “In order to better make sense of the power of the information we have on hand, we first need to understand the impact it has on various industries.”
As for the use of the data, Georges Mao, Director of Marketing Science APAC at Facebook, said it was important for businesses to reconcile the use of customers’ data along with the need of the business. “Without trust there is no business,” he said, “The duty we have is a mutually beneficial result for company and users.” He cautioned misuse by adding that if you keep delivering a message that is wrong, people will eventually tune you out.
Privacy must be a priority
** **There is a bit of an understanding by clients that if they engage in interactions over social media, or participate in programmes that rely on shared information, they are surrendering a fair bit of privacy to third parties.
This does not mean that data must not be protected. According to an article on insidebigdata.com, protecting big data is a huge issue.
According to the article, companies wanting to secure data should follow these steps:
– Use Secure Software. Make certain any software you use, especially open-source software, is secure. If it wasn’t built with security in mind, reconsider using it until a secure version is available. You can also add other security programs designed to help increase protection at the application level;
– Monitor and Analyze your Audit Logs. Always look at your audit logs to see what is happening on your servers, especially your big data servers. This will give you an idea of what kinds of attacks you may be victim to on a regular basis;
– Use Real Time Monitoring Software. Install a real time intrusion detection software like Snort (owned by CISCO) so you can see exactly who is trying to access what at any time. This software will help you block those attacks immediately. You’ll also be able to see if any of your accounts have been hacked when they try to access data they do not have access to;
– Secure your Hardware and Software. When configuring new servers, make sure you’ve created a secure image that you can use for all of them. This will keep your server configurations the same, making it easier to protect them. Also, make certain every patch is installed and that only a few users have administrator-level privileges; and
– Monitor your Accounts and Use Account Controls. Finally, monitor your user accounts and only give access to those that need it. Require users to use very strong passwords, and deactivate the accounts of any employee who leaves the business right away. Also, lock accounts that have attempted to login using an incorrect password after several tries.
Big data can add significant value to your company, provided that you wrap your head around what big data is and the security concerns it comes with.