![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\" "\\"Open"){kind=icon}
![\"Download\"](\"https://img.shields.io/badge/Github-Download-blue.svg\" "\\"Download"){kind=icon}
"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# IMPORT DATA FILES USED BY THIS NOTEBOOK\n",
"import os, requests\n",
"\n",
"file_links = [(\"data/saved_data.csv\", \"https://jckantor.github.io/cbe30338-2021/data/saved_data.csv\")]\n",
"\n",
"# This cell has been added by nbpages. Run this cell to download data files required for this notebook.\n",
"\n",
"for filepath, fileurl in file_links:\n",
" stem, filename = os.path.split(filepath)\n",
" if stem:\n",
" if not os.path.exists(stem):\n",
" os.mkdir(stem)\n",
" if not os.path.isfile(filepath):\n",
" with open(filepath, 'wb') as f:\n",
" response = requests.get(fileurl)\n",
" f.write(response.content)\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"nbpages": {
"level": 1,
"link": "[4.1 Data/Process/Operational Historian](https://jckantor.github.io/cbe30338-2021/04.01-Process-Historians.html#4.1-Data/Process/Operational-Historian)",
"section": "4.1 Data/Process/Operational Historian"
}
},
"source": [
"# 4.1 Data/Process/Operational Historian"
]
},
{
"cell_type": "markdown",
"metadata": {
"nbpages": {
"level": 2,
"link": "[4.1.1 Introduction](https://jckantor.github.io/cbe30338-2021/04.01-Process-Historians.html#4.1.1-Introduction)",
"section": "4.1.1 Introduction"
}
},
"source": [
"## 4.1.1 Introduction"
]
},
{
"cell_type": "markdown",
"metadata": {
"nbpages": {
"level": 3,
"link": "[4.1.1.1 Terminology](https://jckantor.github.io/cbe30338-2021/04.01-Process-Historians.html#4.1.1.1-Terminology)",
"section": "4.1.1.1 Terminology"
}
},
"source": [
"### 4.1.1.1 Terminology\n",
"\n",
"* **DCS**: Distributed Control System\n",
"\n",
"* **Database**: An organized collection of data that is stored and accessed electronically. Databases are a major industry and one of the most significant technologies underpinning the modern global economy.\n",
"\n",
" * **Ralational database**: Data organized as linked collections of tables comprised of rows and columns. Structured Query Language (SQL) is a specialized language for writing and querying relational databases. \n",
" * **NoSQL database**: Typically organized as key-value pairs, NoSQL databases encompass a broad range of technologies used in modern web applications and extremely large scale databases.\n",
" * **Time-series database**: Data organized in time series consisting of time-value pairs, often organized as traces, curves, or trends. Typically used in industrial applications.\n",
" \n",
" \n",
"* **[Data | Operational | Process] Historian**: A time-series database used to store and access operational process data. "
]
},
{
"cell_type": "markdown",
"metadata": {
"nbpages": {
"level": 3,
"link": "[4.1.1.2 Major Vendors of Data Historians](https://jckantor.github.io/cbe30338-2021/04.01-Process-Historians.html#4.1.1.2-Major-Vendors-of-Data-Historians)",
"section": "4.1.1.2 Major Vendors of Data Historians"
}
},
"source": [
"### 4.1.1.2 Major Vendors of Data Historians\n",
"\n",
"Data historians is about a $1B/year market globally, poised to grow much larger with the emerging **Industrial Internet of Things (IIoT)** market.\n",
"\n",
"* GE, IBM, Hitachi-ABB, Rockwell Automation, Emerson, Honeywell, Siemens, AVEVA, OSIsoft, ICONICS, Yokogawa, PTC, Inductive Automation, Canary Labs, Open Automation Software, InfluxData, Progea, Kx Systems, SORBA, Savigent Software, Automsoft, LiveData Utilities, Industrial Video & Control, Aspen Technology, and COPA-DATA"
]
},
{
"cell_type": "markdown",
"metadata": {
"nbpages": {
"level": 3,
"link": "[4.1.1.3 Example: OSIsoft PI System](https://jckantor.github.io/cbe30338-2021/04.01-Process-Historians.html#4.1.1.3-Example:-OSIsoft-PI-System)",
"section": "4.1.1.3 Example: OSIsoft PI System"
}
},
"source": [
"### 4.1.1.3 Example: OSIsoft PI System\n",
"\n",
"* One of the market leaders is OSIsoft which markets their proprietary PI system. Founded in 1980, OSIsoft now has 1,400 employees and recently announced sale of the company for $5B to AVENA.\n",
"\n",
"* The PI system is integrated suite of tools supporting the storage and retreival of process data in a time-series data base.\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"nbpages": {
"level": 3,
"link": "[4.1.1.4 Process Analytics](https://jckantor.github.io/cbe30338-2021/04.01-Process-Historians.html#4.1.1.4-Process-Analytics)",
"section": "4.1.1.4 Process Analytics"
}
},
"source": [
"### 4.1.1.4 Process Analytics\n",
"\n",
"Process analytics refers to analytical tools that use the data historian to provide usable information about the underlying processes. "
]
},
{
"cell_type": "markdown",
"metadata": {
"nbpages": {
"level": 2,
"link": "[4.1.2 The tclab Data Historian](https://jckantor.github.io/cbe30338-2021/04.01-Process-Historians.html#4.1.2-The-tclab-Data-Historian)",
"section": "4.1.2 The tclab Data Historian"
}
},
"source": [
"## 4.1.2 The tclab Data Historian\n",
"\n",
"The tclab Python library support the Temperature Control Lab includes a very basic and no-frills implementation of a time-series data base. The purposes of the data historian are to \n",
"\n",
"* enable the collection and display of data durinig the course of developing control strategies, and\n",
"* enable post-experiment analysis using standard Python libraries such as Pandas.\n",
"\n",
"Documentatiion is available for the tclab [Historian](https://tclab.readthedocs.io/en/latest/notebooks/05_TCLab_Historian.html) and associated [Plotter](https://tclab.readthedocs.io/en/latest/notebooks/06_TCLab_Plotter.html) modules.\n",
"\n",
"Historian is implemented using [SQLite](https://www.sqlite.org/index.html), a small, self-contained SQL data system in the public domain. SQLite was originally developed in 2000 by D. Richard Hipp who was designing software for a damage-control systems used in the U.S. Navy aboard gui}ded missile destroyers. Since then it has become widely used in embedded systems including most laptops, smartphones, and browsers. If you used Apple photos, messaging on your smartphone, GPS units in your car, then you've used SQLite. It estimated there are over 1 trillion SQLite databases in active use. Much of the success is to due to the licensing terms (free!) and an extraordinarily level of automated code testing assuring a high level of reliability.\n",
"\n",
"Below we will introduce useful capabilities of the Historian that will prove useful as we explore more sophisticated control algorithms and strategies.\n",
"\n",
"* Data logging\n",
"* Acessing data"
]
},
{
"cell_type": "markdown",
"metadata": {
"nbpages": {
"level": 3,
"link": "[4.1.2.1 Data Logging](https://jckantor.github.io/cbe30338-2021/04.01-Process-Historians.html#4.1.2.1-Data-Logging)",
"section": "4.1.2.1 Data Logging"
}
},
"source": [
"### 4.1.2.1 Data Logging"
]
},
{
"cell_type": "markdown",
"metadata": {
"nbpages": {
"level": 4,
"link": "[4.1.2.1.1 Creating a log](https://jckantor.github.io/cbe30338-2021/04.01-Process-Historians.html#4.1.2.1.1-Creating-a-log)",
"section": "4.1.2.1.1 Creating a log"
}
},
"source": [
"#### 4.1.2.1.1 Creating a log\n",
"\n",
"An instance of a data historian is created by providing a list of data sources. An instance of a `lab` created by `TCLab()` provides a default list of sources in `lab.sources`."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"nbpages": {
"level": 4,
"link": "[4.1.2.1.1 Creating a log](https://jckantor.github.io/cbe30338-2021/04.01-Process-Historians.html#4.1.2.1.1-Creating-a-log)",
"section": "4.1.2.1.1 Creating a log"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"TCLab version 0.4.9\n",
"Simulated TCLab\n",
"TCLab Model disconnected successfully.\n"
]
}
],
"source": [
"from tclab import setup, clock, Historian\n",
"\n",
"TCLab = setup(connected=False, speedup=60)\n",
"\n",
"with TCLab() as lab:\n",
" h = Historian(lab.sources) # <= creates an instance of an historian with default lab.sources\n",
" lab.Q1(100)\n",
" for t in clock(600):\n",
" h.update(t) # <= updates the historian at time t\n",
" \n",
"# note that the historian lives on after we're finished with lab"
]
},
{
"cell_type": "markdown",
"metadata": {
"nbpages": {
"level": 4,
"link": "[4.1.2.1.2 Accessing Data using `.columns` and `.fields`](https://jckantor.github.io/cbe30338-2021/04.01-Process-Historians.html#4.1.2.1.2-Accessing-Data-using-`.columns`-and-`.fields`)",
"section": "4.1.2.1.2 Accessing Data using `.columns` and `.fields`"
}
},
"source": [
"#### 4.1.2.1.2 Accessing Data using `.columns` and `.fields`\n",
"\n",
"There are several approaches to accessing the data that has been recorded using the historian. Perhaps the most straightforward is to access the 'tags' with `h.columns` and to access the values with `h.fields` as shown here."
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"nbpages": {
"level": 4,
"link": "[4.1.2.1.2 Accessing Data using `.columns` and `.fields`](https://jckantor.github.io/cbe30338-2021/04.01-Process-Historians.html#4.1.2.1.2-Accessing-Data-using-`.columns`-and-`.fields`)",
"section": "4.1.2.1.2 Accessing Data using `.columns` and `.fields`"
}
},
"outputs": [
{
"data": {
"text/plain": [
"['Time', 'T1', 'T2', 'Q1', 'Q2']"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# columns property consists of all data being logged\n",
"h.columns"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"nbpages": {
"level": 4,
"link": "[4.1.2.1.2 Accessing Data using `.columns` and `.fields`](https://jckantor.github.io/cbe30338-2021/04.01-Process-Historians.html#4.1.2.1.2-Accessing-Data-using-`.columns`-and-`.fields`)",
"section": "4.1.2.1.2 Accessing Data using `.columns` and `.fields`"
}
},
"outputs": [
{
"data": {
"text/plain": [
"["
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"%matplotlib inline\n",
"import matplotlib.pyplot as plt\n",
"\n",
"t, T1, T2, Q1, Q1 = h.fields # <= access data using h.fields\n",
"plt.plot(t, T1, t, T2) # <= plot data "
]
},
{
"cell_type": "markdown",
"metadata": {
"nbpages": {
"level": 4,
"link": "[4.1.2.1.3 Accessing Data by Name `logdict`](https://jckantor.github.io/cbe30338-2021/04.01-Process-Historians.html#4.1.2.1.3-Accessing-Data-by-Name-`logdict`)",
"section": "4.1.2.1.3 Accessing Data by Name `logdict`"
}
},
"source": [
"#### 4.1.2.1.3 Accessing Data by Name `logdict`\n",
"\n",
"Often you only need data corresponding to specific tags. You can access the data directly using `.logdict[]`."
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"nbpages": {
"level": 4,
"link": "[4.1.2.1.3 Accessing Data by Name `logdict`](https://jckantor.github.io/cbe30338-2021/04.01-Process-Historians.html#4.1.2.1.3-Accessing-Data-by-Name-`logdict`)",
"section": "4.1.2.1.3 Accessing Data by Name `logdict`"
}
},
"outputs": [
{
"data": {
"text/plain": [
"["
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.plot(h.logdict[\"Time\"], h.logdict[\"T2\"])"
]
},
{
"cell_type": "markdown",
"metadata": {
"nbpages": {
"level": 4,
"link": "[4.1.2.1.4 Accessing Data using `pandas`](https://jckantor.github.io/cbe30338-2021/04.01-Process-Historians.html#4.1.2.1.4-Accessing-Data-using-`pandas`)",
"section": "4.1.2.1.4 Accessing Data using `pandas`"
}
},
"source": [
"#### 4.1.2.1.4 Accessing Data using `pandas`\n",
"\n",
"The Python `pandas` library provides an enormous range of commonly tools for data analysis, it is among the most widely used libraries by data scientists. Data collected by the historian can be converted to a pandas data frame with one line of code."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"nbpages": {
"level": 4,
"link": "[4.1.2.1.4 Accessing Data using `pandas`](https://jckantor.github.io/cbe30338-2021/04.01-Process-Historians.html#4.1.2.1.4-Accessing-Data-using-`pandas`)",
"section": "4.1.2.1.4 Accessing Data using `pandas`"
}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
""
],
"text/plain": [
" T1 T2 Q1 Q2\n",
"Time \n",
"0.00 20.9495 20.9495 100 0\n",
"1.18 20.6272 20.9495 100 0\n",
"2.05 20.9495 20.9495 100 0\n",
"3.02 20.9495 20.9495 100 0\n",
"4.14 20.9495 20.9495 100 0"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"\n",
" \n",
"
\n",
"\n",
" \n",
" \n",
" T1 \n",
" T2 \n",
" Q1 \n",
" Q2 \n",
" \n",
" \n",
" \n",
" \n",
" Time \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" 0.00 \n",
" 20.9495 \n",
" 20.9495 \n",
" 100 \n",
" 0 \n",
" \n",
" \n",
" 1.18 \n",
" 20.6272 \n",
" 20.9495 \n",
" 100 \n",
" 0 \n",
" \n",
" \n",
" 2.05 \n",
" 20.9495 \n",
" 20.9495 \n",
" 100 \n",
" 0 \n",
" \n",
" \n",
" 3.02 \n",
" 20.9495 \n",
" 20.9495 \n",
" 100 \n",
" 0 \n",
" \n",
" \n",
" \n",
"4.14 \n",
" 20.9495 \n",
" 20.9495 \n",
" 100 \n",
" 0 \n",
" "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.5"
}
},
"nbformat": 4,
"nbformat_minor": 4
}