Blog

<h3>TL;DR;</h3>Using the same template created for <a data-mce-href="https://www.aignosi.com.br/blog/soft-sensors-for-conveyor-belts-at-samarco" href="https://www.aignosi.com.br/blog/soft-sensors-for-conveyor-belts-at-samarco" target="_blank" rel="noopener">virtual belt scales</a>, Samarco built a silica prediction model for flotation. With SIENTIA™’s automated retraining, even a simple regression delivers reliable, real-time insights without costly custom development.<h3>The case</h3>In iron ore flotation, silica content is a critical quality parameter. Traditionally, plants rely on laboratory analyses every two hours, leaving Advanced Process Controllers (APCs) operating with delayed data. This slows decision-making, increases reagent waste, and risks product quality.Samarco addressed this challenge with a virtual quality sensor, designed to predict silica grade every 5 minutes. Surprisingly, the model wasn’t built on complex deep learning, but on a multivariate linear regression.<h3>&nbsp;Reusing What Already Worked &nbsp;</h3>Instead of reinventing the wheel, <a data-mce-href="https://www.aignosi.com.br/blog/soft-sensors-for-conveyor-belts-at-samarco" href="https://www.aignosi.com.br/blog/soft-sensors-for-conveyor-belts-at-samarco" target="_blank" rel="noopener">Samarco reused the same no-code template originally created for conveyor belt mass flow models</a>. This was possible thanks to the flexibility of SIENTIA™, which allowed the Process Optimization team to adapt an existing template to a new use case.But flotation is a nonlinear process, and a simple regression alone wouldn’t be robust enough over time. This is where another key feature of SIENTIA™ came into play.<h3>Automated Retraining: The Game-Changer &nbsp;</h3>SIENTIA™ provided automated retraining every 4 hours, which compensated for process variability and data drift. This capability transformed what could have been an unstable model into a reliable, production-ready sensor.The combination of a simple template + automated retraining meant that Samarco could:<ul><li>Avoid exorbitant costs of custom model development.</li><li>Leverage its own team’s process knowledge.</li><li>Deploy a working solution much faster than traditional R&amp;D cycles. &nbsp;</li></ul><h3>Impact on Operations </h3><ul><li>Silica predictions delivered every 5 minutes instead of waiting 2 hours.</li><li>Direct integration with the flotation APC, enabling real-time reagent dosing.</li><li>Significant reduction in reagent waste and higher product quality stability. </li></ul><blockquote>This case is a good example that even “simple” models, when combined with robust MLOps features, can solve complex nonlinear problems. &nbsp;</blockquote>

Soft Sensors for quality prediction at Samarco

Artificial Intelligence, Technology, Robot, Futuristic, Data Science, Data Analytics, A.I.

TL; DR;"MLOps started in the cloud. Now it must survive on the factory floor."Google’s canonical MLOps architecture set the foundation for ML systems engineering — defining CI/CD pipelines, model registries, automated retraining and performance monitoring. It works well for cloud-native apps, SaaS platforms, and enterprise IT.But on the industrial shop floor? It struggles to survive. Let's dive into this topic in this article.<h3>The gap between IT and OT</h3>Industrial manufacturing lives in a different universe:- 🔌 **Intermittent or no cloud connectivity** - ⏱️ **Sub-second latency for critical systems** - 🔐 **Sensitive data locked behind OT firewalls** - 🧱 **Legacy equipment using OPC UA, SCADA, and PLCs**These constraints make the cloud-first architecture unfit for real-time control loops, predictive quality models, and embedded diagnostics.<h3>A new architecture is emerging: hybrid edge-cloud</h3>A recent scientific publication by Raffin et al. (2022, FAU Germany) proposes a reference architecture tailored to manufacturing. Unlike the centralized Google model, this architecture is **distributed, modular and event-driven**.It’s not about abandoning the cloud — it’s about **empowering the edge**. Here’s how:🧠 Inference at the edgeModels are served locally via lightweight containers, with inference times in the order of milliseconds. This ensures safety, uptime, and operational continuity even when offline.🧩 Microservices and domain-driven designEach pipeline component — from data acquisition to decision services — is a microservice deployed on the edge. Interfaces are designed for modularity, encapsulation, and adaptability to factory heterogeneity.&nbsp;🔄 CI/CD/CT adapted to OTA full DevOps loop is preserved, but adapted:- Models are versioned and tested centrally - Containers are deployed locally using GitOps or IaC - Continuous Training (CT) is triggered by drift or scheduling - Monitoring is embedded both locally and in the cloud&nbsp;📊 Governance and feature storesData is cached and served via local feature stores. Model drift, data quality and inference logs are managed on-site, with batch synchronization to cloud systems for retraining and collaboration.<h3>Why it matters</h3>This shift isn’t just technical — it’s operational and strategic. The success of industrial AI depends on:- Running models where the data lives - Empowering process engineers, not just data scientists - Handling millions of edge nodes without losing controlThe hybrid architecture enables AI to operate autonomously, securely, and explainably — without sacrificing agility or compliance.<h3>How aignosi is leading this transition</h3>At aignosi, we’ve implemented these principles in our platform — combining:- 🔹 Industrial-grade data collectors (OPC UA, MQTT, Modbus) - 🔹 Scalable edge-based orchestration - 🔹 Embedded MLOps governance and observability - 🔹 No-code/low-code interfaces for domain expertsWith our solutions, manufacturers are running dozens of AI models per site, even in air-gapped or partially connected environments.We believe the future of AI in manufacturing is not cloud-only — it’s cloud-aware, edge-native, and operator-centric.---📚 Reference: Raffin, T. et al. (2022). A reference architecture for the operationalization of machine learning models in manufacturing. Procedia CIRP, 115, 130–135.*

From Cloud to Factory Floor: The Evolution of MLOps Architectures for Manufacturing

Flying through the futuristic tunnel. Abstract 3D animation. Data Network, Virtual Reality, Quantum. The concept of illuminated corridor, interior design, spaceship, Galaxy, science, lab, technology, science

TL;DR;Efficient operationalization of machine learning (ML) and deep learning (DL) algorithms in the industrial sector presents significant challenges due to the inherently probabilistic nature of these methods, the constant need for retraining, and their critical dependence on high frequency data sampling/inference. To address these challenges, this article summarizes a proposed a hybrid cloud/edge architecture, integrating centralized computing resources (cloud) with decentralized, local resources (edge) as shonw in the original paper [1].<h3>Motivation and Context</h3>The integrated application of ML in Cyber-Physical Systems (CPS) is essential for smart manufacturing, delivering substantial gains in flexibility, agility, and operational robustness. However, effective implementation requires a systemic approach that enables both vertical and horizontal integration of data and models throughout the entire production chain—also known as Cyber-Physical Production Systems (CPPS). In this context, Continuous Delivery (CD) practices become essential to support the centralized and ongoing deployment of trained models across distributed industrial environments.Key Challenges in Industrial Implementation• Technological Heterogeneity: A wide range of equipment, non-standard communication protocols, and various data formats.• Data Management: The need for high-quality, adequately volumed, and efficiently integrated structured and unstructured data.• Sustainable Model Operation: Continuous monitoring, support for incremental learning, and guaranteed low latency due to the critical and time-sensitive nature of industrial processes.• Non-Technical Challenges: Shortage of skilled labor, high stakeholder expectations, and concerns about cost-effectiveness of tech implementations.<h3>Proposed Architecture</h3>The proposed hybrid cloud/edge architecture is composed of five key domains:1. Edge SystemHandles time-sensitive tasks such as data acquisition, local pre-processing, real-time inference, and direct interaction with physical devices (sensors and PLCs).2. Cloud InstanceConducts more complex, less time-sensitive tasks such as system-wide monitoring, analytical dashboards, and advanced incremental learning techniques.3. Integrated Data ManagementProvides economical, long-term storage for large volumes of data, with local caches enabling near-instantaneous access for critical industrial processes.4. Development and ExperimentationCovers the full cycle of ML/DL model creation, experimentation, and optimization, supported by rigorous version control and experiment tracking.5. Machine Learning Operations (MLOps)Involves continuous and automated processes for integration (CI), delivery (CD), and training (CT), which are essential to dynamically maintain and update ML models.<figure id="figure-m88yu3uksm8nepkbg" data-full-width=""><img id="m88yu3uksm8nepkbg" src="https://cdn.durable.co/blocks/cFci9mLApW7gxH4CyFdDsS27qHWyFzzA9Mvl4puTVGpRyAQTRDvP79QAvY5C6vMm.jpg" alt="" title="" class="object-cover bg-gray-100 rounded-lg md:rounded-xl lg:rounded-2xl selected" style="aspect-ratio: 16/9; object-position: undefined undefined;" data-corners="medium" data-aspect="16:9" data-caption="" data-positionx="undefined" data-positiony="undefined" data-fullwidth="true" data-mce-src="https://cdn.durable.co/blocks/cFci9mLApW7gxH4CyFdDsS27qHWyFzzA9Mvl4puTVGpRyAQTRDvP79QAvY5C6vMm.jpg" data-mce-style="aspect-ratio: 16/9; object-position: undefined undefined;" data-mce-selected="1"></figure>Proposed hybrid architecture for MLOPs/AIOps purposes in manufacturing.<h3>Advantages of the Hybrid Architecture</h3>• Significant Latency Reduction: Local inference ensures fast, effective responses in critical operations.• Optimized Bandwidth Usage: Local pre-processing greatly reduces the volume of data transmitted to the cloud.• Greater Flexibility and Scalability: Complex model training can be performed in the cloud while simplified versions run at the edge.• Improved Security and Operational Reliability: Minimizes external risks and ensures continuity of operations even when cloud connectivity is lost.Implementation Challenges• High technical complexity in integrating cloud and edge layers.• Substantial initial investment in advanced technological infrastructure.• Complex security management in distributed environments.• Ongoing need for maintenance and frequent updates on both layers.<h3>Conclusion and Strategic Relevance</h3>The hybrid cloud/edge architecture is essential for ensuring scalability and economic efficiency in deploying ML solutions in complex industrial environments. By combining the analytical power of the cloud with the speed and operational efficiency of edge computing, this approach holds significant potential to optimize operational performance, improve product quality, and enhance the overall robustness of production processes.This article contributes to both academic and practical understanding of the strategic importance of smart integration between cloud and edge computing, highlighting its critical role as a competitive differentiator in Industry 4.0.Reference:<a data-mce-href="https://www.sciencedirect.com/science/article/pii/S2212827122014974" href="https://www.sciencedirect.com/science/article/pii/S2212827122014974" title="A reference architecture for the operationalization of machine learning models in manufacturing" target="_blank" rel="noopener">[1] A reference architecture for the operationalization of machine learning models in manufacturing</a>

Hybrid Cloud/Edge Architecture for Operationalizing Machine Learning Models in OT

<h3>TL;DR;</h3>In today’s industrial landscape, the volume of data generated is immense and growing exponentially. While industries have systems like Plant Information Management Systems (PIMS) that handle data ingestion, provisioning, and cataloging, there’s still a significant gap when it comes to preparing this data for Artificial Intelligence (AI) and Machine Learning (ML) applications. This is where DataOps comes into play, amplifying existing capabilities and evolving beyond traditional systems to ensure data reliability and security in Operational Technology (OT) environments.<h3>What Is DataOps?</h3>DataOps is a methodology that combines agile development practices, continuous integration, and automation to enhance data analytics quality and reliability . Originating from the best practices of DevOps, DataOps can offer autonomy to data scientists or AI engineers, allowing them to integrate and deploy models more easily with less dependency on data engineers (also enabling them to handle more complex tasks).Just as DevOps revolutionized collaboration between development and operations teams to deliver software faster and more reliably, DataOps applies the same principles to the data realm. This empowers teams to be more autonomous and efficient in data management and utilization. The picture below shows the main steps of a DataOps approach.<figure id="figure-46f3d3ulaim2yluev1"><img id="46f3d3ulaim2yluev1" src="https://cdn.durable.co/blocks/10NWi8DQpJXCPL69uhl81iaRqezsenHiHDOp6FrXq1eJK2ZI0r3igTRnf3srRjSD.jpg" alt="DataOps cycle illustrating main steps: Data ingestion (extract and load), data cataloging, data preparation (cleaning and transformation), data validation and quality assurance, data provisioning, monitoring and observability, data governance and lineage, and a feedback loop for continuous improvement. Each stage supports data quality, accessibility, and governance, creating a reliable, cyclical data pipeline for effective data management and analytics in organizations." title="DataOps cycle illustrating main steps: Data ingestion (extract and load), data cataloging, data preparation (cleaning and transformation), data validation and quality assurance, data provisioning, monitoring and observability, data governance and lineage, and a feedback loop for continuous improvement. Each stage supports data quality, accessibility, and governance, creating a reliable, cyclical data pipeline for effective data management and analytics in organizations." class="object-cover bg-gray-100 rounded-lg md:rounded-xl lg:rounded-2xl" style="aspect-ratio: 16/9; object-position: undefined undefined;" data-corners="medium" data-aspect="16:9" data-caption="DataOps cycle main steps" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/10NWi8DQpJXCPL69uhl81iaRqezsenHiHDOp6FrXq1eJK2ZI0r3igTRnf3srRjSD.jpg" data-mce-style="aspect-ratio: 16/9; object-position: undefined undefined;"><figcaption class="body-normal text-gray-500 mt-2">DataOps cycle main steps</figcaption></figure>Description of each step:1. Data Ingestion (Extract &amp; Load): This step gathers data from multiple sources, loading it into a central repository like a data lake or data warehouse, setting it up for further processing.&nbsp;2. Data Preparation (Cleaning &amp; Transformation): Prepares data for analysis by cleaning inconsistencies and transforming it into usable formats, ensuring high-quality data.&nbsp;3. Data Validation &amp; Quality Assurance (Lineage): Track and verifies data for accuracy and consistency, applying automated tests to meet quality standards before use in analytics or machine learning models. Tracking data lineage for transparency and trust, allowing users to understand data origins and transformations4. Data Provisioning: Makes validated data accessible to users and systems via APIs, dashboards, or direct database access, ready for decision-making processes. 5. Data Cataloging: Organizes and documents datasets, providing metadata that aids in data discovery, accessibility, and collaboration among teams.&nbsp;On top of each step, we can add the &nbsp;Monitoring &amp; Observability step, &nbsp;aimed at continuously monitoring data pipelines for quality, integrity, and real-time anomaly or issue detection to ensure reliability.Now that we understand what DataOps is, let’s explore its relevance to the OT space.<h3>The Relevance of DataOps in OT</h3>In industries like manufacturing, mining, and others, the primary focus is producing physical goods such as ore and steel. Unlike tech companies with robust data teams, these industries typically have small data teams and limited resources. Here, DataOps becomes essential to maximize the efficiency and productivity of these lean teams.By implementing DataOps practices, professionals such as process engineers, automation engineers, and data scientists gain tools to develop analytical solutions more autonomously. This is crucial in industry, where speed and precision in decision-making can significantly impact productivity and operational safety.Not only is DataOps important to overcome the lack of professionals on industrial data teams, but it also bridges the gap between what big tech companies already do and the needs of industrial engineers who want to apply AI in their factories. DataOps can make the lives of these engineers easier by providing the tools and processes that facilitate AI implementation in industrial settings.In that sens, it is crucial to highlight that there is a significant difference between industrial data and digital data. In sectors like e-commerce, data is predominantly digital and controlled, originating from clicks, page visits, and user interactions. This data is collected in stable and predictable environments.However, in factories, data is generated by physical sensors and equipment located in hostile environments, subject to dust, vibration, and dirt. This nature of industrial data brings several challenges:&nbsp;• Variability and Interferences: Sensors may present inconsistent readings due to adverse environmental conditions and lack of maintenance.&nbsp;• Communication Failures: Equipment in continuous operation may suffer interruptions in data transmission.&nbsp;• Need for Specific Data Processing: It’s essential to filter out noise and validate data integrity before using it in analytical models.We discussed some of these challenges in our <a data-mce-href="https://www.aignosi.com.br/blog/why-it-is-so-hard-to-apply-ai-in-industry--part-i--the-technical-challenges" href="https://www.aignosi.com.br/blog/why-it-is-so-hard-to-apply-ai-in-industry--part-i--the-technical-challenges" target="_blank" rel="noopener">first blog post</a>.Now that we’ve discussed the relevance of DataOps, let’s explore how it can support AI initiatives in OT.<h3>Leveraging Existing Infrastructure: The Role of PIMS as basic DataOps functionalities</h3>Industries have long utilized Plant Information Management Systems (PIMS) to manage critical data operations. These systems have, in many ways, introduced and implemented certain aspects of DataOps, effectively democratizing data access over the past two decades.To date, PIMS systems provide only a subset of DataOps capabilities, including:• Data Ingestion and Collection: PIMS effectively manage data collection from various systems and communication protocols commonly found in factory automation.• Data Provisioning: Access is provisioned for personnel to utilize this data, supporting operational decisions.• Data Cataloging and Organization: Data is cataloged and organized within the system, enabling easier retrieval and analysis.While these systems are robust for operational needs and incorporate some DataOps functionalities, they still lack modern DataOps features essential for AI and ML applications.To enable the safe proliferation of AI across the industry, additional layers of DataOps functionalities must be built upon existing PIMS systems.<figure id="figure-is0q72wn6qm2yxlgug"><img id="is0q72wn6qm2yxlgug" src="https://cdn.durable.co/blocks/19w2ape3pzHLHcnck5ztZlolMlHuEDINJUkKc2jNPf2pFmaE5zjZHkVFAnaJnU2t.png" alt="Diagram showcasing enhanced AI reliability in industrial settings by integrating existing PIMS capabilities with modern DataOps functionalities for optimized data management, ingestion, and provisioning." title="Diagram showcasing enhanced AI reliability in industrial settings by integrating existing PIMS capabilities with modern DataOps functionalities for optimized data management, ingestion, and provisioning." class="object-cover bg-gray-100 rounded-2xl md:rounded-3xl lg:rounded-4xl" style="aspect-ratio: 16/9; object-position: 100% undefined%;" data-corners="default" data-aspect="16:9" data-caption="Enhancing AI reliability by leveraging existing PIMS capabilities with modern DataOps functionalities" data-positionx="100%" data-positiony="undefined%" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/19w2ape3pzHLHcnck5ztZlolMlHuEDINJUkKc2jNPf2pFmaE5zjZHkVFAnaJnU2t.png" data-mce-style="aspect-ratio: 16/9; object-position: 100% undefined%;"><figcaption class="body-normal text-gray-500 mt-2">Enhancing AI reliability by leveraging existing PIMS capabilities with modern DataOps functionalities</figcaption></figure><h3>The Missing Pieces: Automated Data Preparation and Quality Assurance</h3>Despite the strengths of PIMS, there are critical areas where they fall short, especially concerning data provisioning for AI and ML applications:&nbsp;1. Automated Data Preparation:&nbsp;• Current State: Some level of data preparation exists but is often manual and time-consuming.&nbsp;• What’s Needed: An automated layer that simplifies data preparation, making it readily usable for AI models without extensive manual intervention.&nbsp;2. Validation and Quality Assurance:&nbsp;• Data Lineage and End-to-End Tracking: There’s a lack of mechanisms to trace data from its origin through to its use in AI models.&nbsp;• Purpose-Built for AI/ML: Existing validation processes aren’t designed to meet the rigorous demands of AI and ML applications.&nbsp;3. Monitoring and Observability:&nbsp;• Pipeline Quality Monitoring: Industries monitor the data values but not the quality/integrity of the data pipelines.&nbsp;• Data Integrity Checks: There’s no systematic way to detect anomalies or integrity issues in the data feeding AI models.By augmenting PIMS with these additional DataOps layers, industries can bridge the gap between existing capabilities and the requirements of modern AI and ML applications. This evolution is crucial for safely and effectively integrating AI into industrial operations.<figure id="figure-ovfy70lvb1jm2yxc34q"><img id="ovfy70lvb1jm2yxc34q" src="https://cdn.durable.co/blocks/2bBRnEZL2ssdQ053E9OfjsfPHRdtbgzXPly6sENMQX0wwFQ3AQ8GVeJtmjMXpTCg.png" alt="Illustration of bridging the gap between PIMS and modern DataOps to enhance AI reliability, featuring key functions like Data Democratization, Data Collection, Monitoring, Quality Assurance, and Automated Data Preparation in industrial settings" title="Illustration of bridging the gap between PIMS and modern DataOps to enhance AI reliability, featuring key functions like Data Democratization, Data Collection, Monitoring, Quality Assurance, and Automated Data Preparation in industrial settings" class="object-cover bg-gray-100 rounded-2xl md:rounded-3xl lg:rounded-4xl" style="aspect-ratio: 3/2; object-position: 50% 65%;" data-corners="default" data-aspect="3:2" data-caption="Bridging the gap between PIMS and modern DataOps feature enhances AI reliability." data-positionx="50%" data-positiony="65%" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/2bBRnEZL2ssdQ053E9OfjsfPHRdtbgzXPly6sENMQX0wwFQ3AQ8GVeJtmjMXpTCg.png" data-mce-style="aspect-ratio: 3/2; object-position: 50% 65%;"><figcaption class="body-normal text-gray-500 mt-2">Bridging the gap between PIMS and modern DataOps feature enhances AI reliability.</figcaption></figure><h3>The Crucial Role of DataOps in Data Reliability and Security</h3>DataOps is vital for professionals developing analytical models, enabling them to focus on value creation without worrying about the complexities of data engineering. By automating and standardizing data collection, processing, and integration processes, DataOps ensures that only reliable and suitable data is used.This is especially important in industry, where incorrect inferences can have severe consequences:&nbsp;• Inaccurate Inferences: Due to unvalidated or poorly prepared data, leading to suboptimal or even hazardous operational decisions.&nbsp;• Operational Risks: Equipment may operate outside safe parameters, causing damage or reducing lifespan.&nbsp;• Human Safety: Decisions based on incorrect data can lead to accidents that endanger employees’ lives.&nbsp;• Financial Impact: Production interruptions and equipment damage result in significant financial losses.Unlike a digital environment, where a failure might result in a lost sale, in industry, the impacts are much more severe. Therefore, ensuring data reliability through DataOps isn’t just a matter of efficiency but also of safety and responsibility.<h3>Best Practices for DataOps in Industry</h3>To implement DataOps effectively, it’s important to follow some best practices:&nbsp;1. Process Automation: Utilize tools that automate data collection, processing, and validation.&nbsp;2. Continuous Integration: Adopt pipelines that allow continuous integration of new data and models.&nbsp;3. Monitoring and Alerts: Implement monitoring systems to quickly detect and respond to data anomalies.&nbsp;4. Team Collaboration: Promote communication between operations, engineering, and data science teams.&nbsp;5. Data Governance: Establish clear policies for data management and access, ensuring compliance and security.<figure id="figure-4fysting5qm2yygamy"><img id="4fysting5qm2yygamy" src="https://cdn.durable.co/blocks/1e7UvjDuM0QKTa7GpzzZrk0u1kRmK2A35khJO20Sfb8bjn5ZYp7iiHfKrVJEdvD7.jpg" alt="Diagram illustrating the DataOps pipeline for transforming raw data into reliable data for AI-driven safe inference. Key stages include data processing, governance, security, and automation, enhancing data quality and reliability for industrial AI applications." title="Diagram illustrating the DataOps pipeline for transforming raw data into reliable data for AI-driven safe inference. Key stages include data processing, governance, security, and automation, enhancing data quality and reliability for industrial AI applications." class="object-cover bg-gray-100 rounded-2xl md:rounded-3xl lg:rounded-4xl" style="aspect-ratio: 16/9; object-position: undefined undefined;" data-corners="default" data-aspect="16:9" data-caption="Transforming raw data into safe AI inferences through DataOps" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/1e7UvjDuM0QKTa7GpzzZrk0u1kRmK2A35khJO20Sfb8bjn5ZYp7iiHfKrVJEdvD7.jpg" data-mce-style="aspect-ratio: 16/9; object-position: undefined undefined;"><figcaption class="body-normal text-gray-500 mt-2">Transforming raw data into safe AI inferences through DataOps</figcaption></figure><h3>Conclusion</h3>DataOps is an essential component for modern industry, ensuring that data is reliable and that analytical models operate safely and effectively. By adopting DataOps practices, industries can not only improve operational efficiency but also mitigate significant risks associated with inconsistent or incorrect data.Investing in DataOps means investing in the safety, quality, and competitiveness of your AI-driven industrial operations.References&nbsp;• Challenges of Industrial Data and How to Overcome Them – Our first blog post details the interferences that industrial data faces and the best practices to mitigate them.Did You Like the Content?If you wish to deepen your knowledge about DataOps and discover how to implement it in your industry, contact us or leave a comment below. Our team is ready to help you transform your data into real business value.Optimize your industrial operations with DataOps. Stay ahead in safety, quality, and competitiveness by embracing data reliability and security.

Enhancing AI Reliability in OT Environments Through DataOps

<h3>TL;DR;</h3> Deploying AI in Operational Technology (OT) environments can revolutionize industrial processes, boosting efficiency and reducing costs. However, transitioning AI from proof of concept to production in OT presents significant challenges. In this article, we will explore in detail the key obstacles industries face when implementing AI in OT and how AIOps helps overcome them.<h3>Why Deploying AI in OT is More Complex than in IT</h3>Unlike IT environments, which often have flexible connectivity and robust cloud infrastructure, OT environments in industries like mining, manufacturing, and energy operate in real-time, with zero tolerance for delays or failures. Additionally, there are cybersecurity concerns about exposing the entire set of PLCs (Programmable Logic Controllers) or DCS (Distributed Control Systems) to external networks, even with modern security layers and strategies such as using a DMZ. The following are the major challenges industries face when deploying AI in OT:&nbsp;1. Low-Latency, Real-Time OperationsIndustrial environments require near-instantaneous decision-making. AI models must process and respond to data within milliseconds to prevent costly disruptions. It is also impractical to transmit large amounts of data (such as time series or images) in milliseconds over a standard 10/100 network configuration between OT and IT.AIOps Solution:AIOps processes data locally, ensuring models run in real-time without the delays associated with cloud-based solutions. This real-time capability is critical for high-performance industrial operations.&nbsp;2. Limited Connectivity and SecurityMany industrial plants have restricted or no internet access for security reasons. This prevents the use of cloud solutions and increases the difficulty of deploying AI.AIOps Solution:AIOps enables on-premises AI edge deployment with full governance over data (DataOps) and models (MLOps), securely managing data and operations in isolated environments. By automating the monitoring of security systems, AIOps efficiently detects vulnerabilities and mitigates risks.&nbsp;3. Legacy Infrastructure IntegrationMany industrial systems run on legacy equipment and protocols that were not designed for modern AI technologies. Although modern options exist, such as OPC UA for Industry 4.0 requirements, most legacy systems in factories still do not use modern communication protocols.AIOps Solution:AIOps simplifies integrating AI with legacy systems by using specialized and optimized data connectors and protocols, allowing seamless data collection and machine learning implementation.&nbsp;4. Scalability and Model ReplicationOnce an AI model is deployed in one part of a plant, scaling it to other areas without manual intervention becomes a challenge. Additionally, there is a skill gap, as most industrial engineers are not data scientists. This creates an opportunity for no-code and automated machine learning solutions to help bridge this gap.AIOps Solution:AIOps combined with no-code user interface and auto-machine learning features, automates model replication across different production lines or sites, allowing teams to scale AI implementations quickly and efficiently without compromising model accuracy.&nbsp;5. Data Management and GovernanceManaging vast amounts of real-time data from industrial sensors and ensuring its quality is a daunting task.AIOps Solution:AIOps integrates DataOps practices to ensure high-quality, reliable data management. Automated version control and governance frameworks allow companies to maintain compliance and track model performance over time.<h3>Case Study: Industrial AIOps Applied in a Mining Company in Brazil</h3>A major mining company in Brazil faced challenges deploying AI models in the edge due to limited internet connectivity and a not proper infrastructure/software for AI production, &nbsp;typical in operational technology (OT) environments. The company’s goal was to optimize its production processes while maintaining high security standards and ensuring real-time decision-making capabilities. At the same time, they would like to scale the many use cases that they have experienced before during PoC (proof of concept) projects.Our AIOps platform (<a data-mce-href="https://www.aignosi.com.br/product-sientia" href="https://www.aignosi.com.br/product-sientia" title="SIENTIA™" target="_blank" rel="noopener">SIENTIA™</a>) was introduced to tackle these challenges, automating the development and deployment of AI models through no-code interfaces and AutoML capabilities. This approach allowed the client’s engineering team, which had no prior experience in data science, to independently scale their AI initiatives.<figure id="figure-0ax6rip23sm2eydbw0"><img id="0ax6rip23sm2eydbw0" src="https://cdn.durable.co/blocks/12vsFwfRKqP1FxaTGte4dAwt2tnW54ZTOMrupEv3Ko0iup1GwYymWmjKB3a0Sdiv.jpg" alt="A no-code interface of aignosi's AIOps platform, SIENTIA, which helps users with no prior knowledge in data science/ML to develop, deploy and scale their AI use cases into production in weeks." title="A no-code interface of aignosi's AIOps platform, SIENTIA, which helps users with no prior knowledge in data science/ML to develop, deploy and scale their AI use cases into production in weeks." class="object-cover bg-gray-100 rounded-sm md:rounded-md lg:rounded-lg" style="aspect-ratio: 16/9; object-position: undefined undefined;" data-corners="small" data-aspect="16:9" data-caption="SIENTIA™ no-code interface with auto-ML capabilities" data-positionx="undefined" data-positiony="undefined" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/12vsFwfRKqP1FxaTGte4dAwt2tnW54ZTOMrupEv3Ko0iup1GwYymWmjKB3a0Sdiv.jpg" data-mce-style="aspect-ratio: 16/9; object-position: undefined undefined;"><figcaption class="body-normal text-gray-500 mt-2">SIENTIA™ no-code interface with auto-ML capabilities</figcaption></figure>By leveraging AIOps, the mining company developed and scaled over ten AI models, including virtual sensors for quality prediction and anomaly detection models across multiple sites. With integrated MLOps and DataOps frameworks, the company maintained full governance and compliance, ensuring that their models remained accurate and efficient over time.Results:&nbsp;• Improved Scalability: The company scaled over 10 models autonomously, thanks to the no-code and AutoML features, reducing dependency on external experts.&nbsp;• Reduced Deployment Time: The average time from development to deployment for new models was reduced to just one week.&nbsp;• Increased Production Efficiency: By deploying models across different areas of their operations, the company expects to achieve at least a 0.5% increase in production efficiency, resulting in substantial cost savings.<table style="border-collapse: collapse; width: 100%;" border="1" data-mce-style="border-collapse: collapse; width: 100%;"><tbody><tr><td style="width: 25%;" data-mce-style="width: 25%;">Use case</td><td style="width: 25%;" data-mce-style="width: 25%;">Models Developed by Mining Enginering team</td><td style="width: 25%;" data-mce-style="width: 25%;">Models Developed by third parties</td><td style="width: 25%;" data-mce-style="width: 25%;">Average time from Development to Production</td></tr><tr><td style="width: 25%;" data-mce-style="width: 25%;">Virtual Sensor for Conveyor Belts Scales</td><td style="width: 25%;" data-mce-style="width: 25%;">6</td><td style="width: 25%;" data-mce-style="width: 25%;">0</td><td style="width: 25%;" data-mce-style="width: 25%;">1 week/model</td></tr><tr><td style="width: 25%;" data-mce-style="width: 25%;">Virtual Sensor for Quality Prediction</td><td style="width: 25%;" data-mce-style="width: 25%;">1</td><td style="width: 25%;" data-mce-style="width: 25%;">1</td><td style="width: 25%;" data-mce-style="width: 25%;">2 weeks/model</td></tr><tr><td style="width: 25%;" data-mce-style="width: 25%;">Anomaly Detection</td><td style="width: 25%;" data-mce-style="width: 25%;">6</td><td style="width: 25%;" data-mce-style="width: 25%;">1</td><td style="width: 25%;" data-mce-style="width: 25%;">1 week/model</td></tr></tbody></table><h3>Conclusion</h3>AI deployment in OT environments presents unique challenges, but with AIOps, companies can automate processes, secure operations, and scale their AI efforts effectively. The future of AI in industrial operations depends on overcoming these obstacles, and AIOps is a critical part of that transformation.

AIOps for OT – The Key to Scale AI in Factories - Part II

Black Man And Hispanic Woman Working On Business Strategy and Standing Infront of Big Digital Screen With Data. Male And Female Colleagues Discussing Plan For Profit Margins Enlargement In The Office.

<h3>TL;DR;&nbsp;</h3>Over recent years, AIOps (Artificial Intelligence for IT Operations) has gained attention from companies aiming to enhance operational efficiency and productivity through artificial intelligence (AI). While initially developed for IT environments, AIOps can also be successfully applied to Operational Technology (OT), where real-time decision-making and low latency are crucial. In this article, we will explore how AIOps is transforming industrial environments by combining advanced AI techniques with real-time automation.<h3>What is AIOps for OT and Why is It Critical for Industrial Environments?</h3>AIOps was originally designed to optimize IT infrastructure, automating problem detection and allowing systems to self-manage effectively. In industrial environments, AIOps extends beyond IT, focusing on critical production operations in OT (operational technology environments), encompassing practices from DatOps (data operations) and also MLOps (model operations). The figure bellow tryes to illustrate the overall concept of AIOps with the combination of Data and ML operations.<figure id="figure-z8qukcb9tm2eo431v"><img id="z8qukcb9tm2eo431v" src="https://cdn.durable.co/blocks/2Gzv8ygvEj92foN8LoOkbPP0WDC6rhQoQGLGVqy93qMuLpdB29fC8C0vGhnDZdus.jpg" alt="AIOPS for OT, the junction of DataOps and MLOps concept, with each main step that encompass the cycles of data's and mlops." title="AIOPS for OT, the junction of DataOps and MLOps concept, with each main step that encompass the cycles of data's and mlops." class="object-cover bg-gray-100 rounded-2xl md:rounded-3xl lg:rounded-4xl" style="aspect-ratio: 16/9; object-position: 100% 80%;" data-corners="default" data-aspect="16:9" data-caption="AIOPS for OT, the junction of DataOps and MLOps concept." data-positionx="100%" data-positiony="80%" data-fullwidth="false" data-mce-src="https://cdn.durable.co/blocks/2Gzv8ygvEj92foN8LoOkbPP0WDC6rhQoQGLGVqy93qMuLpdB29fC8C0vGhnDZdus.jpg" data-mce-style="aspect-ratio: 16/9; object-position: 100% 80%;"><figcaption class="body-normal text-gray-500 mt-2">AIOPS for OT, the junction of DataOps and MLOps concept.</figcaption></figure>AIOps for OT comes to try to overcome typical challenges &nbsp;to put AI into production in the industrial environment:&nbsp;• Low Latency: Industrial operations demand near-instantaneous decision-making, especially when AI is working in real-time environments.&nbsp;• Security: Sectors like mining and manufacturing are highly sensitive to cybersecurity threats, requiring AI solutions to operate securely in air-gapped environments.&nbsp;• Hardware Efficiency: AIOps optimizes the use of local computing resources, crucial for industries with limited infrastructure or strict on-premises requirements.The challenges mentioned above arise because OT does not have access to the same infrastructure available in IT environments within large industrial corporations. This is mainly due to the safety and security requirements that industrial equipment must meet, as they operate close to the production process, and any safety or security issue could pose significant risks. In addition to the low latency and security concerns discussed in the previous paragraphs, the OT environment also faces hardware limitations that restrict the availability of cloud technology in OT.This is why the concept of AIOps makes a lot of sense, as it aims to overcome these technological challenges by applying appropriate concepts and best practices to deploy AI in production despite these limitations.<h3>AIOps in OT: Overcoming Scalability Challenges</h3>One of the biggest challenges in OT is deploying AI models effectively without disrupting operations. Machine learning models in OT environments face obstacles like real-time data demands, strict latency requirements, and the need for secure, disconnected operations.Beyond the challenges of MLOps, DataOps in the industry faces difficulties such as correctly reading industrial communication systems, like OPC, for example. Typically, these systems provide data to production systems, and we need to perform secure data collection without interrupting the data flow being sent to the production system, while also sending it to the AI system. This means ensuring secure and robust data collection, as well as processing the industrial data, since field sensors can fail, and there is often no measure of the data’s reliability and quality. DataOps addresses these issues by providing reliable data for the MLOps layer, which will use this data for model inference.Now, with AIOps, data and model governance becomes more structured, enabling:&nbsp;• Automation of Machine Learning Pipelines: From data collection to model training and deployment, AIOps automates the entire process, reducing human errors and speeding up production timelines.&nbsp;• Continuous Monitoring: AIOps provides real-time model performance monitoring, allowing for proactive retuning and ensuring models remain accurate over time.&nbsp;• Scalability: AIOps allows models to be replicated across different production lines or plants with minimal adjustments, making scaling seamless and efficient.<h3>Use Case: Scaling Predictive Maintenance in Manufacturing</h3>In a large-scale manufacturing plant, AIOps was implemented to streamline predictive maintenance. Traditionally, the company relied on manual checks and reactive maintenance, leading to frequent downtimes. By integrating AIOps, the company automated reliable data collection and processing (DataOps procedures) and also scaled anomaly detection models across several critical pieces of equipment (10) (MLOps procedures), such as conveyor belts. The figure bellow, shows and example of an anticipation of thirteen hours ahead of the unplanned shutdown, the anomaly detection algorithm &nbsp;was already alarm the potential shutdown, which really ocurred.<figure id="figure-n0vpkuqj9xm2epay1s"><img id="n0vpkuqj9xm2epay1s" src="https://cdn.durable.co/blocks/1fKSTk1tvzBeZraCKV3LhRbeFhEYG3IPEi7Gy3DTjtyqF8tx9haOmp0ySDmtX8Qs.jpg" alt="Anomaly detection example in a conveyor belt, showing that the algorithm can detect anomalies thirteen hours ahead of the real unplanned shutdown." title="Anomaly detection example in a conveyor belt, showing that the algorithm can detect anomalies thirteen hours ahead of the real unplanned shutdown." class="object-cover bg-gray-100 rounded-2xl md:rounded-3xl lg:rounded-4xl" style="aspect-ratio: 16/9; object-position: undefined undefined;" data-corners="default" data-aspect="16:9" data-caption="Anomaly detection example in a conveyor belt" data-positionX="undefined" data-positionY="undefined" data-fullWidth="false"/><figcaption class="body-normal text-gray-500 mt-2">Anomaly detection example in a conveyor belt</figcaption></figure>Results:&nbsp;• Reduced Downtime: Maintenance teams were alerted of potential issues before they occurred, reducing downtime.&nbsp;• Cost Savings: By automating anomaly detection, the company cut down manual inspection costs.&nbsp;• Scalability: AIOps enabled the replication of the model across multiple production lines, achieving operational consistency.<h3>Conclusion</h3>AIOps offers a powerful solution to the challenges of AI deployment in OT environments. By optimizing hardware usage, ensuring security, and reducing the time to production, AIOps can transform industrial processes and deliver substantial business value.<h3>What is next?</h3>In the upcoming article, part 2, we will dive deeper into what DataOps and MLOps are, and how all of this makes a lot of sense in helping to scale AI in the industry.

Blog

Soft Sensors for quality prediction at Samarco

From Cloud to Factory Floor: The Evolution of MLOps Archi...

Hybrid Cloud/Edge Architecture for Operationalizing Machi...

Enhancing AI Reliability in OT Environments Through DataOps

AIOps for OT – The Key to Scale AI in Factories - Part II

AIOps for OT – The Key to Scale AI in Factories - Part I