xtd package

Introduction

What is XTD ?

XTD is a library that provides a very high levels set of tools designed to efficiently develop industrial-level python softwares.

Features

  • Static application
    • Unified command line & file configuration
    • Ready to use logging facility
    • Crash-persistent memory data
    • Statistic measurements and output
  • Web application
    • Everything included in static application
    • Ready to use web server (cherrypy based)
    • HTTP api to access
      • logs
      • statistics
      • persistent parameters

Compatibility

Warning

Python 3.x

Installation

sudo pip3 install xtd

Get Started

Basic Application

XTD is designed to develop your software by inheriting the main Application object.

Example :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
import xtd.core.application

class MergeApplication(xtd.core.application.Application):
  def __init__(self):
    super().__init__("MyAppName")
    # register input options

  def initialize(self):
    super().initialize()
    # do some initialization suff

  def process(self):
    l_exitCode = 0
    # do some stuff
    return l_exitCode, True

MergeApplication().execute()

Input options

XTD provides a way to declare and read command-line and file configuration options in a unified way. An option:

  • is registered with an unique name and belongs to a section
  • is attached to zero-or-more checks that will validate input values

Command-line & File config

User can provides values to options in 3 different ways :

  • from internal default value
  • from command line with --<section>-<name> VALUE option
  • from program’s json configuration file with {"<section>": {"<name>": VALUE}}

When multiple values are available, they are taken with the following order of priority (from lowest to highest) :

  1. registered default value
  2. value in configuration file
  3. value on command-line

Registering options

Arguments are registered with the register() and register_section() methods of the ConfigManager (singleton) object

This ConfigManager is accessible via the config() method of your Application or directly from the singleton.

Note

Full option documentation available at Option

Standard check functions checkers

The following code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
import xtd.core.application
import xtd.core.config
import xtd.core.logger
import xtd.core.config.checkers

class MergeApplication(xtd.core.application.Application):
  def __init__(self):
    super().__init__("MergeApplication")

    self.m_config.register_section("input", "Input settings", [{
      "name"        : "directory",
      "default"     : "./work_to_do/",
      "description" : "Read input files from given directory",
      "checks"      : xtd.core.config.checkers.is_dir(p_write=True)
    },{
      "name"        : "count",
      "default"     : 1,
      "description" : "Number of file to read in directory",
      "checks"      : xtd.core.config.checkers.is_int(p_min=0)
    }])

    self.m_config.register_section("output", "Output settings", [{
      "name"        : "file",
      "description" : "Destination file",
      "checks"      : xtd.core.config.checkers.is_file(p_write=True)
    }])


if __name__ == "__main__":
  l_app = MergeApplication()
  l_app.execute()

Produces :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
$ python3 test.py --help

Usage: MergeApplication [options]

Options:
  --help                       show this help message and exit

  General Settings:
    --config-file=ARG          use FILE as configuration file [default:MergeApplication/MergeApplication.json]

  Input settings:
    --input-directory=ARG      Read input files from given directory [default:./work_to_do/]
    --input-count=ARG          Number of file to read in directory [default:1]

  Logging Settings:
    --log-config=ARG           Logging configuration [default:{}]
    --log-override=ARG         override part of logging configuration [default:{}]

  Output settings:
    --output-file              Destination file [default:False]

  Persitent Param Settings:
    --param-directory=ARG      Destination directory for admin persistent parameters [default:/tmp/snmp/MergeApplication/admin]

  Stats Settings:
    --stat-writters=ARG        Enabled given stat output writter. Possibles values :
                               * disk : write counters to --stat-disk-directory path each --stat-disk-interval seconds
                               * http : post counters in json format to --stat-http-url url each --stat-http-interval seconds
                               Can specify a comma separated combinaison of theses values
                               [default:['disk']]
    --stat-disk-directory=ARG  Destination directory for counter disk writter [default:/tmp/snmp/MergeApplication/stat/]
    --stat-disk-interval=ARG   Interval in second between two disk outputs [default:50]
    --stat-http-url=ARG        Destination POST url for http stat writter [default:http://localhost/counter]
    --stat-http-interval=ARG   Interval in second between two http outputs [default:50]

Reading options

Nothing simpler than reading option values.

From your Application object:

import xtd.core.application

class MyApp(xtd.core.application.Application):
  def process(self):
    l_inputDir   = self.config().get("input", "directory")
    l_count      = self.config().get("input", "count")
    l_outputFile = self.config().get("output", "file")

Or, from anywhere in your code :

from xtd.core import config

def my_function(self):
  l_inputDir   = config.get("input", "directory")
  l_count      = config.get("input", "count")
  l_outputFile = config.get("output", "file")

Note

The xtd.core.application.Application.initialize() method has to be called before you attempt to read options values.

Logging

Features

  • standard python logging module compliant
  • logger.<level>(<module>, <message>) primitives in addition to the standard logging.getLogger(<module>).<level>(<message>) functions
  • rich default configuration including:
    • standard SysLogHandler bound to /dev/log socket
    • standard RotatingFileHandler bound to ./out.log
    • standard StreamHandler bound to sys.stdout This particular handler is setup with a special Formatter LocationFormatter who pads fields to help with vertical reading and colorizes record fields depending on their value

Configuration

By default, Application defines two unified options that changes the logging behavior:

  • --log-config: full logging configuration in json format. Details about configuration format is available in object LogManager
  • --log-override: partial logging configuration that will be merged on top of full configuration. This option respect the same format as the full format except that you may only specify parts. This is useful when you want to override the log level for a specific module on the command line while the rest of the configuration is in the configuration file

Note

Even we use command line options, keep in mind that they always have their configuration file equivalent

Example:

$ # activate log-level 10 (debug) for module "a.b.c"
$ python myapp.py --log-override='{"handler" : {"a.b.c" : { "level" : 10 } } }'

Persistent data

Todo

some doc

Statistics

Todo

some doc