Agile Data & DataOps in Advanced Analytics

In Advanced Analytics, Agile methodology is necessary: it brings a wider approach than usual, considering not only the different aspects to be taken into account in a web/mobile development, but also every aspect included in Data Management: Data Security, Data Quality, traceability, Master Data. Moreover, there is an acute need for highly qualified personnel in the new technologies and specialized fields.

One solution is to use DataOps as the core of Agile Data. However, it poses work-related difficulties for the team (DataOps & DevOps team):  such as endlessly looking for more storage server space, reconfigurating clusters, etc., stemming from the great quantities of data generated, whose volume rises daily.

Also, given the lack of freely available talent, it makes sense to safeguard the teams you already have: allowing them to use their skills to the best effect, automating the more routine parts of their roles so they can focus on their higher skill-set.

As a consequence of the great volume of data and the need for automation, plus the lack of available talent, we face the current scenario:

  • Forbes predicted a 50% increase in Predictive Analytics software applications in 2018, to nearly 3.5 bill $.
  • Worldwide spending on Big Data and Predictive Analytics grew at a rate of 30% by the end of 2018, reaching the 120 bill $.

Companies need solutions to overcome: a) the great quantity of data production; and b) the acute shortage of talent in emerging technologies and specialized fields:

  • Regarding the volume of data: they need some ‘tin’ to store it in.
  • To manage the applications high complexity: they need an end-to-end solution –SQL, R, PYTHON–.
  • Streamsets gets 35 mill $ for DataOps.
  • StreamSets says DataOps has grown in importance, as companies re-architect their data supply chain using new technologies and techniques.
  • StreamSets’ software is designed to help companies keep a close eye on key data integration tasks, often developed and implemented as data pipelines.
  • Actifio looks to meld DataOps with DevOps.
  • Actifio and IBM have formed an alliance, under which IBM will make available data management software, developed by Actifio under its own name.
  • Its aim is to close the gap between DevOps & DataOps.
  • DataOps makes it easier to copy a virtual instance of a data set running in a production environment, and to update it with the most recent copy of that data at any point in time.

Adjuvant Technologies

There are strong reasons to pool data in Credit Unions. First, they do not have enough data to build accurate predictive models. Second, if a data scientist works on a Data Pool including 50 Credit Unions, these Unions get to split the cost of the data scientist, making Advanced Analytics more affordable.

The union of these technologies emerges as a consecuence of the need to give access to data to as many people as possible within an organization, performing Analytics and Machine Learning in a friendly manner.

Has a unique idea: it combines the concept of Data Lake with Edge Computing to create what it calls Interconnected Micro Data Lakes”.

In doing this, we need to keep in mind that these two technologies combine for the sake of synergy. So you can use any device that supports a flat file system, even a mainframe. The Edge acts as a filter for unnecessary data.

Test Edge Computing

  • AT&T and Microsoft are testing Edge Computing to enhance 5G service.
  • AT&T is searching for the smoothest solution to reduce latency and improve user experience, by targeting cloud services geographically closer to businesses.

The combination of these technologies enables and extends digital transformation projects at the Edge. Edge Computing devices will provide connectivity and protection for new and existing Edge devices. 5G will provide the Edge with far better connectivity and lower latency to cloud based applications. But the cost of processing and storing the data is still there. A hybrid edge compute/5G solution will mitigate these costs.

Will increase its use of Edge computiong technologies, after having tested Edge functions in its 5G network in Houston. The result being a massive decrease in network latency.

Artificial Intelligence (AI) & Automation


The top AI and Analytics trends for 2019 are: a) the use of advanced algorithms to identify and optimize business insights; b) the search for full-stack engineers with AI and Analytics skills; and c) China as a global leader in AI research.

Algorithms: the use of advanced computing algorithms to identify and optimize those business insights that humans cannot spot. AI will deliver 2 trill $ in business value worldwide in 2019 (as a comparison, Spain’s GDP in 2019 is 1.4 trill $, that of Italy is 2.1 trill $).

Algorithms are the future of business success. And Machine Learning and Deep Learning –the tools for processing algorithms– have become the doorway into a new era of Artificial Intelligence (AI).

Example: Healthcare. AI + Deep Learning are starting to tackle common problems in healthcare. Smartphone-based cognitive behavioral therapy and integrated group therapy are showing promise in treating conditions such as depression, eating disorders, and substance abuse.

Companies will scramble to hire full-stack engineers with AI and Analytics skills, making them the hottest careers in 2019.

research. A survey of 260 multinational corporations in 2018, showed that:

  • 80% are investing in AI.
  • Firms are investing 47 mill $ on average in AI technology.
  • Large firms in Asia-Pacific (most of them in China) spend 25 mill $ on average.

(an American Data Analytics company) targets Chinese clients’ need for AI. It provides data technologies and Deep Learning (DL) related solutions. Curiously, Teradata’s revenue in the US fell by 11%, year-on-year, to 830 mill $, in first three quarters of 2017. But its international revenue surged to 700 mill $.  An increase mainly explained by Chinese demand.

Automation + AI:

Machine Learning and AI are generating higher quality predictive insights, where the most important goals are: a) accuracy in the recommentations and predictions; and b) the explainability of the predictions.

(JAL) used dotData’s software as part of a promotional campaign for trips to Hawaii.  Software company Fujimaki helped JAL analyze more than 20 customer data points (with customers’ flight history, point history, use of credit cards and Web site access history).

Data-driven approaches, combined with a deeper insight into clinical workflow, can reduce the cost of excessive testing, improving situational awareness and outcomes.

Example: New insights can be obtained through Natural Language Processing (NLP) & Analytics. NLP & Analytics can go beyond the call center to collect and mine even more data. Call center recordings are analyzed and made available in an enriched text format that can be used to deliver rich visualizations. Thus, firms can use Natural Language Processing (NLP) to surface keywords and topics to make recorded content discoverable, allowing access to key insights.