None

This notebook contains material from cbe30338-2021; content is available on Github.

< 4.0 Process Analytics | Contents | Tag Index | 4.2 State Estimation >

# IMPORT DATA FILES USED BY THIS NOTEBOOK
import os,  requests

file_links = [("data/saved_data.csv", "https://jckantor.github.io/cbe30338-2021/data/saved_data.csv")]

# This cell has been added by nbpages. Run this cell to download data files required for this notebook.

for filepath, fileurl in file_links:
    stem, filename = os.path.split(filepath)
    if stem:
        if not os.path.exists(stem):
            os.mkdir(stem)
    if not os.path.isfile(filepath):
        with open(filepath, 'wb') as f:
            response = requests.get(fileurl)
            f.write(response.content)

4.1 Data/Process/Operational Historian

4.1.1 Introduction

4.1.1.1 Terminology

• DCS: Distributed Control System

• Database: An organized collection of data that is stored and accessed electronically. Databases are a major industry and one of the most significant technologies underpinning the modern global economy.

  • Ralational database: Data organized as linked collections of tables comprised of rows and columns. Structured Query Language (SQL) is a specialized language for writing and querying relational databases.
  • NoSQL database: Typically organized as key-value pairs, NoSQL databases encompass a broad range of technologies used in modern web applications and extremely large scale databases.
  • Time-series database: Data organized in time series consisting of time-value pairs, often organized as traces, curves, or trends. Typically used in industrial applications.

• [Data | Operational | Process] Historian: A time-series database used to store and access operational process data.

4.1.1.2 Major Vendors of Data Historians

Data historians is about a $1B/year market globally, poised to grow much larger with the emerging Industrial Internet of Things (IIoT) market.

• GE, IBM, Hitachi-ABB, Rockwell Automation, Emerson, Honeywell, Siemens, AVEVA, OSIsoft, ICONICS, Yokogawa, PTC, Inductive Automation, Canary Labs, Open Automation Software, InfluxData, Progea, Kx Systems, SORBA, Savigent Software, Automsoft, LiveData Utilities, Industrial Video & Control, Aspen Technology, and COPA-DATA

4.1.1.3 Example: OSIsoft PI System

• One of the market leaders is OSIsoft which markets their proprietary PI system. Founded in 1980, OSIsoft now has 1,400 employees and recently announced sale of the company for $5B to AVENA.

• The PI system is integrated suite of tools supporting the storage and retreival of process data in a time-series data base.

4.1.1.4 Process Analytics

Process analytics refers to analytical tools that use the data historian to provide usable information about the underlying processes.

4.1.2 The tclab Data Historian

The tclab Python library support the Temperature Control Lab includes a very basic and no-frills implementation of a time-series data base. The purposes of the data historian are to

• enable the collection and display of data durinig the course of developing control strategies, and
• enable post-experiment analysis using standard Python libraries such as Pandas.

Documentatiion is available for the tclab Historian and associated Plotter modules.

Historian is implemented using SQLite, a small, self-contained SQL data system in the public domain. SQLite was originally developed in 2000 by D. Richard Hipp who was designing software for a damage-control systems used in the U.S. Navy aboard gui}ded missile destroyers. Since then it has become widely used in embedded systems including most laptops, smartphones, and browsers. If you used Apple photos, messaging on your smartphone, GPS units in your car, then you've used SQLite. It estimated there are over 1 trillion SQLite databases in active use. Much of the success is to due to the licensing terms (free!) and an extraordinarily level of automated code testing assuring a high level of reliability.

Below we will introduce useful capabilities of the Historian that will prove useful as we explore more sophisticated control algorithms and strategies.

• Data logging
• Acessing data

4.1.2.1 Data Logging

4.1.2.1.1 Creating a log

An instance of a data historian is created by providing a list of data sources. An instance of a lab created by TCLab() provides a default list of sources in lab.sources.

from tclab import setup, clock, Historian

TCLab = setup(connected=False, speedup=60)

with TCLab() as lab:
    h = Historian(lab.sources)  # <= creates an instance of an historian with default lab.sources
    lab.Q1(100)
    for t in clock(600):
        h.update(t)             # <= updates the historian at time t
        
# note that the historian lives on after we're finished with lab

TCLab version 0.4.9
Simulated TCLab
TCLab Model disconnected successfully.

4.1.2.1.2 Accessing Data using .columns and .fields

There are several approaches to accessing the data that has been recorded using the historian. Perhaps the most straightforward is to access the 'tags' with h.columns and to access the values with h.fields as shown here.

# columns property consists of all data being logged
h.columns

['Time', 'T1', 'T2', 'Q1', 'Q2']

%matplotlib inline
import matplotlib.pyplot as plt

t, T1, T2, Q1, Q1 = h.fields   # <= access data using h.fields
plt.plot(t, T1, t, T2)         # <= plot data 

[<matplotlib.lines.Line2D at 0x7f8a9a61db80>,
 <matplotlib.lines.Line2D at 0x7f8a9a61dbb0>]

4.1.2.1.3 Accessing Data by Name logdict

Often you only need data corresponding to specific tags. You can access the data directly using .logdict[].

plt.plot(h.logdict["Time"], h.logdict["T2"])

[<matplotlib.lines.Line2D at 0x7fa19159ba30>]

4.1.2.1.4 Accessing Data using pandas

The Python pandas library provides an enormous range of commonly tools for data analysis, it is among the most widely used libraries by data scientists. Data collected by the historian can be converted to a pandas data frame with one line of code.

import pandas as pd
df = pd.DataFrame.from_records(h.log, columns=h.columns, index='Time')

display(df.head())
df.plot()

	T1	T2	Q1	Q2
Time
0.00	20.9495	20.9495	100	0
1.18	20.6272	20.9495	100	0
2.05	20.9495	20.9495	100	0
3.02	20.9495	20.9495	100	0
4.14	20.9495	20.9495	100	0

<AxesSubplot:xlabel='Time'>

4.1.2.1.5 Specifying additional sources

As we develop increasingly complex control algorthms, we will wish to record additional data during the course of an experiment. This is done by specifying data sources. Each source is defined by a (tag, fcn) pair where tag is string label for data, and fcn is a function with no arguments that returns a current value. An example is

['Q1', lab.Q1]

where Q1 is the tag, and lab.Q1() returns the current value of heater power reported by the hardware.

The following cell presents an example where two setpoints are provided for two control loops. The setpoint tags are SP1 and SP2, respectively. Setpoint SP1 is specified as a Python constant. The historian requires a function that returns the value of the which is a function of time. This has to be

['SP1', SP1

from tclab import setup, clock, Historian

# proportional control gain
Kp = 4.0

# setpoint 1
SP1 = 30.0

# setpoint function
SP2 = 30.0

TCLab = setup(connected=False, speedup=60)

with TCLab() as lab:
    # add setpoint to default sources
    sources = lab.sources
    sources.append(['SP1', lambda: SP1(t)])
    sources.append(['SP2', lambda: SP2])
    h = Historian(sources)
    for t in clock(600):
        U1 = Kp*(SP1(t) - lab.T1)
        U2 = 100 if lab.T2 < SP2 else 0
        lab.Q1(U1)
        lab.Q2(U2)
        h.update(t)

TCLab version 0.4.9
Simulated TCLab
TCLab Model disconnected successfully.

4.1.2.2 Persistence

4.1.2.2.1 Saving to a file

h.to_csv("data/saved_data.csv")

h.columns

['Time', 'T1', 'T2', 'Q1', 'Q2', 'SP1', 'SP2']

fig, ax = plt.subplots(2, 1)

ax[0].plot(h.logdict["Time"], h.logdict["T1"], h.logdict["Time"], h.logdict["SP1"])
ax[1].plot(h.logdict["Time"], h.logdict["T2"], h.logdict["Time"], h.logdict["SP2"])

[<matplotlib.lines.Line2D at 0x7fa19192aa90>,
 <matplotlib.lines.Line2D at 0x7fa19192aac0>]

from tclab import setup, clock, Historian

# proportional control gain
Kp = 4.0

# setpoint function
def SP1(t):
    return 20.0 if t <= 100 else 50.0

# setpoint function
SP2 = 40.0

TCLab = setup(connected=False, speedup=60)

with TCLab() as lab:
    # add setpoint to default sources
    sources = lab.sources
    sources.append(['SP1', lambda: SP1(t)])
    sources.append(['SP2', lambda: SP2])
    h = Historian(sources, dbfile="data/tclab_historian.db")
    for t in clock(600):
        U1 = Kp*(SP1(t) - lab.T1)
        U2 = 100 if lab.T2 < SP2 else 0
        lab.Q1(U1)
        lab.Q2(U2)
        h.update(t)

TCLab version 0.4.9
Simulated TCLab
TCLab Model disconnected successfully.

h.get_sessions()

[(1, '2021-03-16 17:07:35', 164),
 (2, '2021-03-16 17:12:20', 593),
 (3, '2021-03-16 17:12:32', 301),
 (4, '2021-03-16 17:15:51', 592)]

h.load_session(1)
fig, ax = plt.subplots(2, 1)

ax[0].plot(h.logdict["Time"], h.logdict["T1"], h.logdict["Time"], h.logdict["SP1"])
ax[1].plot(h.logdict["Time"], h.logdict["T2"], h.logdict["Time"], h.logdict["SP2"])

[<matplotlib.lines.Line2D at 0x7fa191a79f70>,
 <matplotlib.lines.Line2D at 0x7fa191a79fa0>]

g = Historian([], dbfile="lab4.db")
g.get_sessions()

[(1, '2021-03-16 17:12:32', 0), (2, '2021-03-16 17:16:02', 0)]

4.1.3 Plotter

The Plotter class provides a real-time graphical interface to an historian. It provides some simple facilities for

from tclab import setup, clock, Historian, Plotter

# proportional control gain
Kp = 4.0

# setpoint function
def SP1(t):
    return 20.0 if t <= 100 else 50.0

# setpoint function
SP2 = 40.0

TCLab = setup(connected=False, speedup=60)

with TCLab() as lab:
    # add setpoint to default sources
    sources = lab.sources
    sources.append(['SP1', lambda: SP1(t)])
    sources.append(['SP2', lambda: SP2])
    h = Historian(sources, dbfile="data/tclab_historian.db")
    
    layout = [["T1", "SP1"], ["T2", "SP2"], ["Q1", "Q2"]]
    p = Plotter(h, 400, layout)
    for t in clock(400):
        U1 = Kp*(SP1(t) - lab.T1)
        U2 = 100 if lab.T2 < SP2 else 0
        lab.Q1(U1)
        lab.Q2(U2)
        p.update(t)

TCLab Model disconnected successfully.

h.get_sessions()

[(1, '2021-03-16 17:07:35', 164),
 (2, '2021-03-16 17:12:20', 593),
 (3, '2021-03-16 17:12:32', 301),
 (4, '2021-03-16 17:15:51', 592),
 (5, '2021-03-16 17:16:02', 302)]

fig, ax = plt.subplots(2, 1)

ax[0].plot(h.logdict["Time"], h.logdict["T1"], h.logdict["Time"], h.logdict["SP1"])
ax[1].plot(h.logdict["Time"], h.logdict["T2"], h.logdict["Time"], h.logdict["SP2"])

[<matplotlib.lines.Line2D at 0x7fa191da1c70>,
 <matplotlib.lines.Line2D at 0x7fa191da1ca0>]

< 4.0 Process Analytics | Contents | Tag Index | 4.2 State Estimation >