******************************
Viewing and Debugging Memory
******************************

Due to the fact that Digi's Python environment is intended to run on a
system that has a limited amount of memory compared to modern PCs and
no capability for virtual memory, more care than usual must be taken
to ensure that the memory consumption of the system can be well
characterized and does not exceed what is available.

Monitoring Memory Usage
==============================

The system does not currently track a large amount of information
about memory consumption, however it does allow you to gain and
monitor at a high level so that you can be aware whether additional
steps may be needed in your environment.  As a general rule of thumb,
free memory should not be allowed to dip below a couple hundred
kilobytes steady state although testing should be performed to
determine a safe level for the memory usage patterns of your
application.

Keep in mind that free memory listed in the below approaches is not
alone an indicator that your system will always be able to acquire the
memory that is needs.  Fragmentation may also be a concern, and
currently there is no good way to determine if the system is becoming
overly fragmented until memory allocations begin to fail.  This is one
of the reasons why it is recommended that free memory be kept at what
may seem to be a high value.

Command Line
--------------------
The ''display memory'' command shows the available information on the
command line.

::

 #> display memory

 Memory Usage:

  Main Memory :
     Total Memory         : 8388608        
     Used Memory          : 5857356        
     Free Memory          : 2531252        

  Network Memory :          Current     Maximum    
     Allocated            : 157748      169516     
     In use               : 141660      162900     

 #> 

Web User Interface
--------------------
The memory usage information is also available on the Web page for the
system in the '''System Information -> General''' section.


.. image:: ./images/memory.png
   :align: center
   :scale: 75

Inspecting a Running System
---------------------------

Python's introspection capabilities extend to allowing visibility into
the internals of object representation and the lifecycle of objects in
the system.  It can be worthwhile to build a way to run your
application that allows for the system to run while providing an
interactive Python session.

This is possible in Dia as demonstrated below::

  >>> import dia
  >>> core = dia.main()
  Determining platform type... PC host environment assumed.

  Python version mismatch! Digi='2.4.3 or 2.6.1', Current='2.4.5'
  iDigi Device Integration Application Version 1.4.1
  Using settings file: dia.yml
  Core: initial garbage collection of 0 objects.
  Core: post-settings garbage collection of 28 objects.
  Starting Scheduler...
  Starting Channel Manager...
  Starting Device Driver Manager...
  Starting Presentation Manager...
  Starting Services Manager...
  Core services started.
  >>> 

This is powerful because it allows you to...

#. Experiment with the impact of running garbage collection manually
#. Interact with your application internals.  In Dia this happens 
   through the ``core`` object that
   exposes a ``CoreServices`` object allowing interaction with much
   of the application.
#. Follow object lifecycles by creating new references to objects and
   following their reference counts using ``sys.getrefcount``.
#. Examine possible reference cycles using ``gc.get_referents`` and
   ``gc.get_referrers``. 
#. Track overall object load on the system with
   ``len(gc.get_objects())``, to check for unexpected object count
   growth that could lead to memory exhaustion.
#. And so on, using your imagination and the powerful tools available.

Programming Tips
==============================

As mentioned above, fragmentation can be an issue in the system.  Low
memory and fragmentation often go hand in hand because not only is
there less memory available to the system when resources become low,
it gets filled with objects of varying lifetimes which when freed
leave smaller fragmented areas of the heap behind.

Here are some things that you can do that may help in a system prone
to fragmentation and low memory:

* Reduce allocation frequency.  This might seem difficult to do in a
  dynamic language like Python, but using mutable containers and
  types such as ''cStringIO'' can make a surprising impact on the
  allocation footprint of the system. 

* Remember the characteristics of common types.  This harkens back to
  the previous item, but many people write code in Python that treats
  strings as if they were mutable objects and this can create
  non-obvious memory and performance behaviors in the system.  Keep
  in mind that as immutable objects, using operations such as
  concatenation create and copy an entirely new object, which at
  least temporarily can require double the memory.

Working with Garbage Collection
-------------------------------

Manually controlling the Python garbage collector can be one of the
most useful tools to characterize and control the usage of memory in
your application.  

It can be difficult to predict the behavior of the threshold based
behavior that triggers garbage collection in the system.  Much of the
time, you can identify locations in your application where complex
activity has occurred and completed where reclamation of resources
generated and released during that activity makes sense.  If that is
the case, inserting calls to the ``gc.collect()`` routine can return
memory to the system much sooner than otherwise.

