************************
Dia Development Overview
************************

I. Introduction 
===============
 
The Digi Device Integration Application ("Dia") is an application
software platform for Digi's family of gateway devices.  It is intended
to make the task of developing custom applications for remote monitoring
and sensor networking applications easier. The Dia framework is written
in the Python programming language and uses the Python interpreter
on Digi's gateway and embedded devices. It may also be executed on a
PC for prototyping purposes when a suitable Python interpreter is
available. 
 
The Dia framework is targeted at applications which aim to solve the
problem of gathering data from a set of devices--such as ZigBee sensors,
wired industrial equipment, GPS devices, etc--by abstracting the data
from these devices as sets of samples which flow into channels, logging
these samples, and re-presenting this data to a next highest layer
application over a wired or wireless IP connection.  The framework
offers a well thought out and tested architecture that is advantageous
when compared to attempting to write an enterprise-grade application
from scratch. 

.. image:: ./images/idigi_introduction.png
   :scale: 75
   :alt: iDigi Dia Environment

The framework is abstracted into three primary subsystems: 
 
   * The device driver subsystem 
   * The channel subsystem
   * The presentation subsystem 
 
Device drivers are responsible for abstracting real-world devices into a
set of quantities known as properties. A device driver instance will
publish the values of these properties via a named channel known as a
"channel property" in the form of a sample.  Channels are accessible
globally in the system via their unique name. A driver's properties are
named by the pattern::
 
    device_instance_name.property_name 
 
For example, a GPS device might have two properties defined: latitude
and longitude. A running instance of our GPS device might be called,
"gps0". As the driver runs, it will collect information from the
physical GPS device and gather the most recent latitude and longitude
values. The driver instance will create sample objects for each of these
two values and publish the new samples to the channel. The values for
latitude and longitude will then be globally available in the system as
the channels "gps0.latitude" and "gps0.longitude" respectively. 
 
The channel subsystem houses the :class:`channel database
<channels.channel_database.ChannelDatabase>` and data logger
instances. The :class:`channel database
<channels.channel_database.ChannelDatabase>` manages the global
namespace of all channel names and allows interested parties to be
able to create new channels, remove existing channels, get samples,
set samples, or subscribe to a channel to be notified upon sample
updates.
 
The data logging system has priority access to changes in the channel
database. As changes are made to the database they enter the domain of
the data logging manager. Changes captured to a file by a data logging
instance may be played back into a fresh instance of a :class:`channel
database <channels.channel_database.ChannelDatabase>` allowing for
objects which know how to interact with the channel database API to
also be able to present data from a log.
 
The presentation system is responsible for presenting samples to users
or other machines running applications at the next highest layer.
Presentations receive their samples from the :class:`channel database
<channels.channel_database.ChannelDatabase>` either by polling or
asking to be notified on changes using the :class:`channel publisher
<channels.channel_publisher.ChannelPublisher>` object's subscription
interface. 

Examples of presentations include a 
:doc:`command-line interface </user/presentations/console>`, 
:doc:`a web interface </user/presentations/web>`, 
:doc:`XML-RPC </user/presentations/xmlrpc>`,
Modbus, or the 
:doc:`iDigi Platform Web Services </user/presentations/rci>` interface.
 
A diagram of the system looks like the following: 

.. image:: ./images/idigi_dia_simple_arch_diagram.png
   :align: center


II. File System Layout 
======================
 
The source code distribution of the iDigi Dia is organized into various
directories to reflect the design of the system.
There are two distinct locations for working with the system: the 
top-level directory which contains the system build tools and configuration
files and the "src" directory. 

The top level directory of the iDigi Dia source code distribution
contains the following entities: 
 
   * bin/ 
   * cfg/ 
   * clients/ 
   * demos/
   * dia.py 
   * doc/ 
   * lib/ 
   * make.py 
   * src/ 
   * tools/ 
   * xml_elements/
 
 
1.  The "bin" Directory 
-----------------------
 
The "bin" directory contains the output of the make.py script (see
below). The files which are generated in this directory may be deployed
to a Gateway for execution by the device's Python interpreter. 
 
 
2.  The "cfg" Directory 
-----------------------
 
The "cfg" directory contains the configuration files for the iDigi Dia.
The default configuration file is called "dia.yml" 
 
.. todo:: I think this is not the way the yml configuration files work now. 
   They are only used by the make script to obtain the PYR configuration 
   file that is added to the resulting zip file. So they are not needed to 
   be uploaded to the gateway.
 
3.  The "clients" Directory 
---------------------------
 
The "clients" directory contains sample software clients which may be
used to interface with the various presentation layer modules within the
iDigi Dia. An example client is the XML-RPC client stored in the
"xmlrpc" sub-directory. 
 
The XML-RPC client interfaces with a running instance of the Dia to
create a command-line interface by communicating with the Dia's XML-RPC
presentation interface. 
 
 
4.  The "demos" Directory 
-------------------------


The "demos" directory contains several iDigi Dia framework configuration 
files (YML files) which can help the end user use Dia for the first time.


5.  The "dia.py" File 
---------------------

The "dia.py" file is the main executable script which starts the
Dia. The "dia.py" is able to be executed on both a Digi gateway
device or on a PC environment. In order to run the script on the gateway
device "dia.py", along with the contents of the "bin" directory
must be uploaded to the device beforehand. 
 
The "dia.py" script may be given a single argument: the name of the
configuration file to load. By default the name of this configuration
file is "dia.yml" 
 
 
6.  The "doc" Directory 
-----------------------
 
.. todo:: Is this file really included in the **doc** directory?

The "doc" directory contains documentation pertaining to the iDigi Dia
system.
 
 
7.  The "lib" Directory
-----------------------
 
The "lib" directory contains third party Python modules that are not
part of the core system. Some of these modules have been created by Digi
and some of them are available in the public domain. 
 
Some of the modules, such as the YAML module, have been modified to
better perform on the Digi gateway system. It is important not to
replace these modules with newer versions without preserving these
modifications. 
 
 
8.  The "make.py" File
----------------------
 
The "make.py" file is the script used to generate a compiled, binary
distribution suitable for loading on the Digi gateway device. 
 
At the top of this file there are a number of hand editable parameters
which may be changed in order to make the build script operate in a
variety of environments. 
 
9.  The "src" Directory
-----------------------
 
The "src" directory contains the main set of source files which compose
the iDigi Dia system. Its contents are explained in greater detail
below.  
 
10.  The "tools" Directory
--------------------------
 
The "tools" directory contains a set of tools used by the "make.py"
script in the generation of the binary distribution of the iDigi Dia
application.  
 
11.  The "xml_elements" Directory
---------------------------------

The "xml_elements" directory contains the files (in XML format) and 
necessary icons to represent in a graphic way all the Dia elements for the 
**Digi ESP for Python** IDE.


III. The Contents of the "src" Directory 
========================================
 
The "src" directory contains the majority of the implementation of the
Digi Device Integration Application. It contains the following entities: 
 
   * __init__.py 
   * channels/ 
   * common/ 
   * core/ 
   * devices/ 
   * presentations/ 
   * samples/ 
   * settings/ 
   * services/
 
Each directory holds the files and sub-directories which constitute the
implementation of a particular subsystem within the iDigi Dia. For
example the "presentation" directory contains the presentation subsystem
implementation: it holds the presentation manager, a presentation base
class, and implementations of a presentation instances such as the
command-line interface and XML-RPC implementations. 
 
1.  The "__init__.py" File
--------------------------
 
Each and every directory which will be accessible as a python module
must contain an "__init__.py" file. 
 
See: http://docs.python.org/tut/node8.html for more information. 
 
 
2.  The "channels" Directory
----------------------------
 
.. todo:: Maybe you could say it also contains the implementation of the
   logging subsystem.
 
The "channels" directory contains the implementation of the channel and 
logging subsystems. These subsystems are responsible for organizing 
and logging samples in the Dia. 
 
 
3.  The "common" Directory
--------------------------
 
The "common" directory houses source files which are used by multiple
systems in the Dia.  These sources differ from the top-level "lib"
directory in that the source has not been adapted from any third-party
module but wholly invented for use within the framework. 
 
An example module in this directory is the :mod:`common.classloader`
module which provides dynamic loading capabilities for the entire
system.
 
The "common" directory also contains a sub-directory called "types".
This directory holds various definitions for specialized types which are
commonly used in the system. 
 
One such type is a :class:`~common.types.boolean.Boolean` type. This
:class:`~common.types.boolean.Boolean` type accepts a wider range of
values in its constructor (such as the strings "On" or "Off" to map to
True and False respectively) as well as options for choosing how the
value is represented as a string.


4.  The "core" Directory
------------------------
 
.. todo:: **core** directory also contains the `Scheduler`, isn't it?
 
The "core" directory contains the :class:`Core Services manager
<core.core_services.CoreServices>` implementation.  The :class:`Core Services <core.core_services.CoreServices>`
manager starts at system start-up ("epoch") time.  Services that are
started in the Core Service manager include:
 
Component Managers
   * :py:class:`The Channel Manager
     <channels.channel_manager.ChannelManager>`
   * :py:class:`The Device Driver Manager
     <devices.device_driver_manager.DeviceDriverManager>`
   * :py:class:`The Presentation Manager
     <presentations.presentation_manager.PresentationManager>`
   * :py:class:`The Service Manager
     <services.service_manager.ServiceManager>`

Core Functionality
   * :py:class:`The tracing manager <common.tracing_manager.TracingManager>`
   * :py:class:`The main scheduler <core.scheduler.Scheduler>`
   * :py:class:`Access to system settings
     <settings.settings_base.SettingsBase>`
 
Each of these top-level services is given a handle to the core. In
turn, child services may make requests of the core for references to
other services. For example, the :py:class:`The Device Driver Manager
<devices.device_driver_manager.DeviceDriverManager>` may ask for a
reference to the :class:`Channel Manager <channels.channel_manager.ChannelManager>` via the :class:`Core Services Manager
<core.core_services.CoreServices>`.
 
 
5.  The "devices" Directory
---------------------------
 
The "devices" directory contains the :py:class:`The Device Driver
Manager <devices.device_driver_manager.DeviceDriverManager>` and all
device implementations in the Dia. The :py:class:`The Device Driver
Manager <devices.device_driver_manager.DeviceDriverManager>` is
responsible for loading all devices in the system.  Device
implementation instances abstract data streams into channel
properties.
 
In the "devices" directory there is a directory named "vendors". If a
new device driver is created by an organization, the organization should
first create a sub-directory within the "vendors" directory and then
deposit their device driver implementation source files within this new
directory. 
 
For example, if "Acme Inc." creates a new sensor driver in a file named
"acme_pressure_sensor.py" a directory named "acme_inc" should be created
in "vendors" and then "acme_pressure_sensor.py" should be placed within
it. 
 
 
6.  The "presentations" Directory
---------------------------------
 
The "presentations" directory houses the :py:class:`Presentation
Manager <presentations.presentation_manager.PresentationManager>` and
all presentation implementations. The :py:class:`Presentation Manager
<presentations.presentation_manager.PresentationManager>` is
responsible for loading all presentations in the system. Presentation
instances display or re-present samples from the :class:`channel
database <channels.channel_database.ChannelDatabase>` to the next
highest layer, often another device attached to the gateway over a
network.
 
Examples of presentations include the :doc:`Command Line Interface
</user/presentations/console>`, the :doc:`XML-RPC service
</user/presentations/xmlrpc>`, and the :doc:`iDigi DB
</user/presentations/idigi_db>` XML database modules.
 
 
7.  The "samples" Directory
---------------------------
 
The "samples" directory contains the sample object implementation.
Sample objects are produced by device drivers, placed into the channel
database--where they are possibly logged--and finally consumed by
one or more presentations. Sample objects are generally the most
numerous object instances in the system. 
 


8.  The "settings" Directory
----------------------------
 
The "settings" directory holds the settings sub-system for the iDigi Dia
framework.  Contained within this directory is the settings base class
implementation and the settings serializer implementations. 
 
In order for an object within the framework to be able to read, to
write, and to persist their settings they must derive from the settings
base class. 
 
Settings in the system may be saved in any format for which there
exists an appropriate serializer implementation. Current
serializations are named YAML and PYR.  The YAML (http://www.yaml.org)
serializer is intended to be more user-friendly and the primary
configuration file format exposed to users.  During the execution of
'make.py', the .yml file provided is converted and serialized to the
PYR format.  This is a direct Python language represenatation of the
settings.  This conversion is performed as an optimization to reduce
the libraries required on a running instance of the Dia on a gateway device. 

The serialization system is abstracted in such a way that other
formats--such as XML or JSON--could be supported without much
additional effort.
 
 
9.  The "services" Directory
----------------------------

.. todo:: The Watchdog service is also included now, and to avoid the "null
   service" bein jealous I would mention it too.

The "services" directory contains the services sub-system of the
iDigi Dia.  Included within this directory is the service base class
implementation and the services manager.

A service is defined as a re-entrant task or thread which does not
cleanly fit into the device or presentation abstractions.  Typically
services do not produce new channels in the :class:`channel database
<channels.channel_database.ChannelDatabase>`--that would be a
device--nor present information to a user or across a network--as that
would be a presentation.

Instead, services provide additional functionality such as the
implementation of a watchdog, or other system maintenance tasks.

 
 
IV. A Description of Framework Initialization From Epoch Time
=============================================================
 
The start-up of the Digi Device Integration Application begins from the 
"dia.py" file. Execution passes through the following phases: 
 
   a.  "dia.py" starts, parses command line parameters and creates a
       :class:`Core Services <core.core_services.CoreServices>` instance.

   b.  When the :class:`Core Services
       <core.core_services.CoreServices>` instance begins it attempts
       to load the systems settings. If successful, it starts the 
       core services:

      * :py:class:`The Tracing Manager
        <common.tracing_manager.TracingManager>`
      * :py:class:`The Channel Manager
        <channels.channel_manager.ChannelManager>`
      * :py:class:`The Device Driver Manager
        <devices.device_driver_manager.DeviceDriverManager>`
      * :py:class:`The Presentation Manager
        <presentations.presentation_manager.PresentationManager>`
      * :py:class:`The Service Manager
        <services.service_manager.ServiceManager>`
      * :py:class:`The system scheduler <core.scheduler.Scheduler>`
      * :py:class:`Access to system settings
        <settings.settings_base.SettingsBase>`

The :py:class:`tracing manager
<common.tracing_manager.TracingManager>` will initialize all tracing 
filters and outputs. 

The :py:class:`system scheduler <core.scheduler.Scheduler>` will start
and be available for use by all other core services and their
instances.

The :py:class:`Channel Manager
<channels.channel_manager.ChannelManager>` will initialize the
:class:`channel database <channels.channel_database.ChannelDatabase>`
for use by other system services; the Logging Manager start as a child
instance of the :py:class:`Channel Manager
<channels.channel_manager.ChannelManager>`.
 
The :py:class:`Device Driver Manager
<devices.device_driver_manager.DeviceDriverManager>` will start,
attempt to load and initialize all devices in the system--in the order
they were defined in the settings database.
 
The :py:class:`Presentation Manager
<presentations.presentation_manager.PresentationManager>` will start,
load and initialize all presentation instances defined in the system.

The :py:class:`Service Manager
<services.service_manager.ServiceManager>` will start, load and
initialize all service instances defined in the system.
 
After all the systems have loaded and initialized the system is
available for use. 

 
