Saturday, March 21, 2020

Enable Logging In Python

Python’s logging module is part of the Python Standard Library. The logging module’s basicConfig() method 
is the quickest way to configure the desired behavior of your logger. However,  applications use file-based or 
dictionary-based logging configuration requires a slightly different configuration.

One simple way to set up file-based logging is to use a config file and fileConfig module to read the log configuration properties. 

The log config file will have a format like the one below:

[loggers]
keys=root,pipeline

[handlers]
keys=fileHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=DEBUG
handlers=fileHandler

[logger_pipeline]
level=DEBUG
handlers=fileHandler
qualname=pipeline
propagate=0

[handler_fileHandler]
class=FileHandler
level=DEBUG
formatter=simpleFormatter
args=('./log/pipeline.log','a')

[formatter_simpleFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
datefmt=

To make it a bit simple to enable logging inside the application, a utility module can be created to handle the logging.

import logging
from logging.config import fileConfig
class AppLogger():
   #ensure a singleton. We only want one log per applcation.
   __instance = None

   #create a static methold to get a logger instance.
   # The staticmethod() built-in function returns a static method for a given function.
   @staticmethod
   def get_applog():
       if AppLogger.__instance is None:
           AppLogger()

       return AppLogger.__instance

   def __init__(self):
       if AppLogger.__instance is not None:
           raise Exception("Only one application log should be activated at a time")
       AppLogger.__instance = AppLogger.create_applogger()

   @staticmethod
   def create_applogger():
       fileConfig('./config/applog_config.cfg', disable_existing_loggers=False)
       logger = logging.getLogger('pipeline')
       return logger

Now, in any of the modules within the application, we can involve the logger by calling the log module.

from util.log import AppLogger

Logger = AppLogger.get_applog()

class TestClass() :

    def  a_fuction():
       try:
           # codes here
       
       except Exception as e:
           Logger.error(“Error occurred {0}“.format(str(e))

         finally:
           #do something

The Python documention provides good reference on how to implement logging:


Friday, March 20, 2020

Setup An Application Configuration File In Python

it's a common practice to store configuration variables in a configuration file and retrieve the properties while they are needed.

Python provides a module called ConfigParser to help managed application configurations. The configParser module can be used to create a configuration file as well as to retrieve property values from a configuration file.

A configuration file in Python is similar to a Windows version of init (.ini) file. The file is divided by “Sections” and “options”. Following is a sample configuration file:

[DEFAULT]
work_directory =/tmp
Log_directory = /var/logs
app_user = app

[DEVELOPMENT]
hostname = mydevhost.com
database = devdb
user_id =devuser
Password =

[STAGE]
hostname = mystagehost.com
database = stagedb
user_id =stageuser
Password =

In the above example, [DEFAULT], [DEVELOPMENT], and [STAGE] are called section. Work_directory, log_directory, hostname,etc are called option.

Here is an example of how to retrieve the properties from the configuration file assuming the above configuration filename is called config.cfg:

import configparser
class GetConfigProperty():

def get_property(self,config_file: str, section: str, property: str) -> str:
     property_value: str
     config = configparser.ConfigParser()
     config.read_file(open('./config/' + config_file))
     property_value = config.get( section, property,fallback=None)
     return property_value

Now, we can call the above function to get the configuration properties in the application like the following:

getConfigProperty = GetConfigProperty()
working_dir : str = getConfigProperty.get_property("config.cfg", "DEFAULT", "working_dir");
dev_hostname : str = getConfigProperty.get_property("file_config.cfg", "DEVELOPMENT", "hostname");

Related Post: Setup Logging In Python

Saturday, May 11, 2019

Docker Command Cheat Sheet

List docker images on the host
$ docker images
List docker containers that are currently running:
$ docker ps
List Containers
$ docker container ls
Delete an image
$ docker rmi <image id>
$ docker rmi --force <image id> #force delete
Remove all images
$ docker rmi $( docker images -q)
Stop All the containers
$ docker stop $( docker ps -aq)
Remove all containers
$ docker rm $( docker ps -aq)
Log Into an Container
$ docker exec -i -t <container id> sh
Run an Docker Image
$ docker run -i -t <image id>

Docker Documents