Plugins configuration

External interfaces

Each plugin can store its own settings independently in file /plugins/<plugin_name>/config.ini.

The platform provides:

  • common interfaces for reading the plugins settings using the class PluginConfigReader;
  • output the availability of configs of plugins via /<prefix>/plugin;
  • web interface for reading/editing configs of the plugins /<instance_perfix>/plugin_config/<plugin_name> (р PluginConfigResource):
    • GET – returns the content of a plugin;
    • PUT – rewrites the config again;
    • Transfer/receiving data is performed in format JSON.
{  
  section1: {param1:val1, param2: val2},
  section1: {param3:val3},
}

 Software interfaces

PluginConfigReader

class PluginConfigReader:
    def __init__(self, pluginName):
    # Returns content of the config.ini in a form of a dictionary

    def getConfigContent(self):
    # Sets plugin config.ini content to content dictionary


    def setConfigContent(self, content):

PluginConfigResource

class PluginConfigResource(Resource):
    @possibleException
    def get(self, pluginName):
        # Check the presence of plugin with the specified name 
        return PluginConfigReader().getConfigContent()


    @possibleException
    def put(self, pluginName):
        # Check the presence of plugin with the specified name 
        # call PluginConfigReader().setConfigContent(postData)
ShareShare on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInShare on VKEmail this to someone

Interfaces of the plugins for OpenData import

This article describes the basic interfaces of the plugins importing open data.

Terms

Job is the the sequence of actions for loading an array of open data through the http protocol, including their processing and loading into Geo2Tag service’s DB. 

Import is transfer data from an external DB into DB of a Geo2Tag service.

Job status is a set of parameters describing current status of a job: time spent, source of import, the flag of completion of the job, other import options.

REST interfaces

  • /<instance_prefix>/plugin/<plugin_name>/service/<serviceName>/job GET – list of the jobs of import and its statuses.
  • /<instance_prefix>/plugin/<plugin_name>/service/<serviceName>/job POST – add a job of import, parameters:
    • channelName, – the name of the channel where importing data will be performed; 
    • openDataUrl,  – link to downloading the set of data;
  • /<instance_prefix>/plugin/<plugin_name>/service/<serviceName>/job/<job_id> GET – the status of the job.
  • /<instance_prefix>/plugin/<plugin_name>/service/<serviceName>/job/<job_id> DELETE – stop the job.

Software interfaces

Classes

Legend:

  • @should_be_extended_in_descendents – method/the property should be extended/override in the derived class
  • @abstract_method – an abstract method which should be defined in the derived class

Job (abstract class)

class Job():

    def __init__(
            self,
            backgroundFunction,
            channelName,
            openDataUrl,
            importDataDict,
            serviceName):
        self.thread = None
        self._id = self.generateId()
        self.startTime = datetime.now()
        self.done = False
        self.timeElapsed = None
        self.backgroundFunction = backgroundFunction
        self.channelName = channelName
        self.openDataUrl = openDataUrl
        self.importDataDict = importDataDict
        self.serviceName = serviceName

    @abstract_method
    def internalStart(self):
    @abstract_method
    def internalStop(self):

    def start(self):
        self.startTime = datetime.now()
        self.internalStart()

    def stop(self):
        self.internalStop()
        self.done = True
        self.timeElapsed = datetime.now() - self.startTime

    def getTimeStatistics(self):
        if self.timeElapsed is None:
            return datetime.now() - self.startTime
        return self.timeElapsed

    @should_be_extended_in_descendents
    def describe(self):
        return {
            '_id': self._id,
            'time': str(
                self.getTimeStatistics()),
            'done': self.done,
            'channelName': self.channelName,
            'openDataUrl': self.openDataUrl,
            'serviceName': self.serviceName}

    @classmethod
    def generateId(cls):
        return ''.join(
            random.choice(
                string.ascii_uppercase +
                string.ascii_lowercase +
                string.digits) for x in range(12))

OpenDataObjectsLoader (abstract class)

class OpenDataObjectsLoader:
    def __init__(self, loadUrl):
        self.loadUrl = loadUrl
    @abstract_method
    def load(self):     

OpenDataToPointTranslator

class OpenDataToPointTranslator:
    def __init__(
            self, importDataDict,
            objectRepresentation,
            version,
            importSource,
            channelId):
        self.objectRepresentation = objectRepresentation
        self.version = version
        self.importSource = importSource
        self.channelId = channelId
        self.importDataDict = importDataDict
    @should_be_extended_in_descendents
    def getPointJson(self):
        obj = {}
        obj['version'] = self.version
        obj['import_source'] = self.importSource
        return obj
    @should_be_extended_in_descendents
    def getPoint(self):
        point = {'json': self.getPointJson()}
        point['channel_id'] = self.channelId
        return point

OpenDataObjectsParser (abstract class)

class OpenDataObjectsParser:
    def __init__(self, data):
        self.data = data
    @abstract_method
    def parse(self):

OpenDataToPointsLoader

class OpenKareliaDataToPointsLoader:
    pointsArray = []

    def __init__(self, serviceName, points):
        self.pointsArray = points
        self.serviceName = serviceName
    def loadPoints(self):
        collection = getDbObject(self.serviceName)[POINTS]
        for point in self.pointsArray:
            collection.save(point)

JobManager – class for managing jobs, starts and stops jobs, displays information about their status.

class JobManager:
    jobs = {}
    @classmethod
    def startJob(cls, job):
        jobId = job.describe().get('_id', '')
        job.start()
        cls.jobs[jobId] = job
        return jobId
    @classmethod
    def getJob(cls, jobId):
        return cls.jobs.get(jobId).describe()
    @classmethod
    def stopJob(cls, jobId):
        cls.jobs.get(jobId).stop()
    @classmethod
    def getJobs(cls):
        result = []
        for job in cls.jobs:
            result.append(cls.jobs[job].describe())
        return result

JobResource

class JobResource(Resource):
    @possibleException
    def get(self, serviceName, jobId):
        return JobManager.getJob(jobId)
    @possibleException
    def delete(self, serviceName, jobId):
        return JobManager.stopJob(jobId)

ODImportParser

class ODImportParser():  
    @should_be_extended_in_descendents
     MANDATORY_FIELDS = [CHANNEL_NAME, OPEN_DATA_URL]
    @staticmethod
    def parse(self):
        args = loads(request.get_data())
        return args
    @staticmethod
    def validate(self, args):    
       for key in MANDATORY_FIELDS:
            if key not in args:
                raise BadRequest('{0} parameter is missing'.format(key))
            elif not isinstance(args[key], unicode):
                raise BadRequest('{0} value is not unicode'.format(key)) 
    @staticmethod
    def parsePostParameters():
        args = self.parse()
        self.validate(args)
        return args

JobListResourceFactory – factory method to create JobListResource classes which we need.

def JobListResourceFactory(parserClass, jobClass, importFunction)
    class JobListResource(Resource):
        @possibleException
        def get(self, serviceName):
            getServiceIdByName(serviceName)
            return JobManager.getJobs()
        @possibleException
        def post(self, serviceName):
            importDataDict = parserClass.parsePostParameters()
            channelName = importDataDict.get('channelName')
            getServiceIdByName(serviceName)
            getChannelByName(serviceName, channelName)
            job = jobClass(importFunction, importDataDict.get('channelName'),
                           importDataDict.get('openDataUrl'), importDataDict,
                           serviceName)
        return JobManager.startJob(job)
    return JobListResource

Method performImportActions

performImportActions ( odLoaderClass, odParserClass, odToPointTranslatorClass, odToPointsLoaderClass, serviceName, channelName, openDataUrl, showObjectUrl, showImageUrl)

A function which contains the logic of importing from source of open data.

def performImportActions ( odLoaderClass, odParserClass, \
    odToPointTranslatorClass, odToPointsLoaderClass, \
    serviceName, channelName, openDataUrl, \
    importDataDict):
#showObjectUrl, showImageUrl
    channelId = getChannelIdByName(channelName)
    version = datetime.now()
    loader = odLoaderClass(openDataUrl)
    openData = loader.load()
    parser = odParserClass(openData)
    objects = parser.parse()
    points = [ ]
    for object in objects:
        translator = odToPointTranslatorClass(importDataDict, object, version,openDataUrl, channelId)
        points.append(translator.getPoint())
    pointsLoader = odToPointsLoaderClass(serviceName, points)
    pointsLoader.loadPoints()

 

 

ShareShare on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInShare on VKEmail this to someone

How to create your own OpenData import plugin

Introduction

This article is a manual on creating your own OpenData import plugins for Geo2Tag platform. Detailed information about the basic classes and common interfaces of these plugins could be found in article Interfaces of the plugins for OpenData import.

Preliminary preparation

Before you create your plugin for OpenData import you should to have the answers to the next questions: 

  1. What kind of  access to the open data you have:
    • access protocol (HTTP(S), FTP, WebDav ..)
    • the presence of encryption
    • the presence of authorization
  2. How does user receive the open data:  in form of array or element-by-element?
  3. In which format does the user receive data (XML/JSON/CSV/ other)?
  4. Is it possible to get access to element of data by the external link?
  5. Do the elements of data contain images and if so, how can you get an external link?
  6. What kind of information about location do elements of data contain:
    • latitude, longitude and altitude
    • postal address
    • zip code

Creation the basic classes

The minimal realization OpenData import plugins contains inheritance/extension of the next classes: 

  1. Job – the abstract class for the import.
  2. OpenDataObjectsLoader – the basic class for downloading elements of open data from external sources.
  3. OpenDataToPointTranslator – the abstract class for conversion elements of open data to the format of points Geo2Tag.  
  4. OpenDataObjectsParser – the abstract class for split the array of open data into single elements.
  5. OpenDataToPointsLoader –  the abstract class for points record to Geo2Tag DB.
  6. ODImportParser – parser of arguments for REST queries to the plugin.

To create ordinary REST queries you have to use:

  1. Resource class for manage a separate job of import JobResource.
  2. Factory method JobListResourceFactory for creating resource class in order to add and view all jobs of import.

For the overall sequence import actions you can use the existing function performImportActions: the passed parameters are the derived classes from the classes specified above. Also it’s possible to create your own implementation of the import function. In that case the main requirement will be a correspondence between the implementation your own class derived from Job and the prototype of the import function.

Adding a plugin

It’s necessary to formalize and connect created set of sources files according to the article Format and connection of plugins.

ShareShare on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInShare on VKEmail this to someone

Performance testing import plugins

Performance of import plugins is tested by script: scripts/performance/od_performance/test_performance.py

The script has the next parameters:

Obligatory:

  1. -createJobLink <string> – link to creating task import
  2. -jobData <string>  – json data for testing
  3. -viewJobsLink <string> – link to viewing current tasks import

Optional:

  1. -jobsCount<int> [default value = 1] – the number of times of running the work of import
  2. -timeout<int> [default value = 60] – timeout in seconds. After this period script will follow to viewJobsLink and check that all task import executed successfully.
  • If all tasks were executed for timeout seconds the statistics will be displayed on the screen.
  • In other case No results by timeout will be displayed in the console.

The structure of the output statistics:

Statistics is a dictionary of the next form:

{average: {value: <среднее time>}, min: {value: <min time>, job: <job_json>}, max: {value: <max time>, job: <job_json>}}

Examples of running (from the root project folder):

./scripts/performance/od_performance/test_performance.py -createJobLink http://geomongo/instance/plugin/ok_import/service/testservice/job  -jobData '{"channelName":"testchannel","openDataUrl":"http://mobile.openkarelia.org//get_nearest_objects?latitude=61.787458487564&longitude=34.362810647964", "showObjectUrl":"", "showImageUrl":""} '  -viewJobsLink http://geomongo/instance/plugin/ok_import/service/testservice/job -jobsCount 2 -timeout 70

Run without any additional parameters:

./scripts/performance/od_performance/test_performance.py -createJobLink http://geomongo/instance/plugin/ok_import/service/testservice/job  -jobData '{"channelName":"testchannel","openDataUrl":"http://mobile.openkarelia.org//get_nearest_objects?latitude=61.787458487564&longitude=34.362810647964", "showObjectUrl":"", "showImageUrl":""} '  -viewJobsLink http://geomongo/instance/plugin/ok_import/service/testservice/job
ShareShare on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInShare on VKEmail this to someone

Тестирование производительности плагинов импорта

Производительность плагинов импорта тестируется скриптом:

scripts/performance/od_performance/test_performance.py

Скрипт имеет следующие параметры:

  • Обязательные:

  1. -createJobLink <string> – ссылка на  создание задачи импорта
  2. -jobData <string>  – json данные для тестирования
  3. -viewJobsLink <string> – ссылка для просмотра текущих заданий импорта
  • Необязательные:

  1. -jobsCount<int> [значение по умолчанию = 1] – количество раз запуска работы по импорту
  2. -timeout<int> [значение по умолчанию = 60] – таймаут ожидания в секундах. Через это время скрипт перейдет по адресу viewJobsLink и проверит, все ли задачи импорта успешно отработали.
  • Если все задачи отработали за timeout секунд, то на экран  будет выведена статистика.
  • Если не все, то на консоль будет выведено No results by timeout

Структура вывода статистики:

Статистика – это словарь следующего вида:

{average: {value: <среднее time>}, min: {value: <min time>, job: <job_json>}, max: {value: <max time>, job: <job_json>}}

Примеры запуска (из корневой папки проекта):

./scripts/performance/od_performance/test_performance.py -createJobLink http://geomongo/instance/plugin/ok_import/service/testservice/job  -jobData '{"channelName":"testchannel","openDataUrl":"http://mobile.openkarelia.org//get_nearest_objects?latitude=61.787458487564&longitude=34.362810647964", "showObjectUrl":"", "showImageUrl":""} '  -viewJobsLink http://geomongo/instance/plugin/ok_import/service/testservice/job -jobsCount 2 -timeout 70

Запуск без дополнительных параметров:

./scripts/performance/od_performance/test_performance.py -createJobLink http://geomongo/instance/plugin/ok_import/service/testservice/job  -jobData '{"channelName":"testchannel","openDataUrl":"http://mobile.openkarelia.org//get_nearest_objects?latitude=61.787458487564&longitude=34.362810647964", "showObjectUrl":"", "showImageUrl":""} '  -viewJobsLink http://geomongo/instance/plugin/ok_import/service/testservice/job
ShareShare on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInShare on VKEmail this to someone

Как написать свой плагин

Пример написания плагина

Директория с плагинами для установленного приложения /var/www/geomongo/plugins.

Для создания плагина необходимо:

  1. создать каталог, название  – строчными буквами, слова разделены знаком нижнего подчеркивания;
  2. создать пустой файл __init__.py для того, чтобы данный каталог являлся python-пакетом;
  3. создать файл main.py c двумя функциями getPluginInfo() и getPluginResources()getPluginInfo() возвращает информацию о плагине, getPluginResources() –  словарь, который содержит пары ‘ссылка’: класс-наследник Resource.

Пример
Пусть существуют два класса-ресурса, которые мы хотим превратить в плагин.

test_resource_1.py:
 from flask.ext.restful import Resource
 class TestResource1(Resource):
   def get(self):
     return 'test_resource_1'
test_resource_2.py:
from flask.ext.restful import Resource
class TestResource2(Resource):
   def get(self):
     return 'test_resource_2'

Их необходимо подключить в main.py:

sys.path.append('../../plugins/test_plugin/')
# Хак, необходимый для того, чтобы ресурсы плагина было видно ( https://geo2tag.atlassian.net/browse/GT-1438)
from test_resource_1 import TestResource1
from test_resource_2 import TestResource2

а также создать обязательные для плагина функции getPluginResources() и getPluginInfo() main.py:

def getPluginResources():
    result = {'res1': TestResource1, 'res2':TestResource2}  
    return result

def getPluginInfo():
    info = 'This plugin was creating for present plugins feature. The function getPluginResources is for return TestResource1 and TestResource2 '
    return info

После этого необходимо создать /var/www/geomongo/plugins/test_plugin, и поместить туда main.py, test_resource_2.py, test_resource_1.py и пустой __init__.py.

Далее необходимо перезагрузить вебсервер, чтобы плагины поднялись:

sudo service apache2 restart

После этого плагины будут доступны по адресам:

http://geomongo/instance/plugin/test_plugin/res1

http://geomongo/instance/plugin/test_plugin/res2

ShareShare on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInShare on VKEmail this to someone

The example of creating a plugin

Directory with plugins for installed application: /var/www/geomongo/plugins.

To create a plugin you have to:

  • create a directory the name of which should be written by latin lowercase letters, words should be divided by underscores;
  • create empty file __init__.py  in order for this directory could be python-package;
  • create file main.py with two functions getPluginInfo() and getPluginResources().

getPluginInfo() returns the information about the plugin;  getPluginResources() is the dictionary which contains couples ‘link’: Resource’s derived class.

Example

Let there are two classes-resource that we want to turn into a plugin.

test_resource_1.py:
 from flask.ext.restful import Resource
 class TestResource1(Resource):
   def get(self):
     return 'test_resource_1'
test_resource_2.py:
from flask.ext.restful import Resource
class TestResource2(Resource):
   def get(self):
     return 'test_resource_2'


It’s necessary to connect both classes in main.py:

sys.path.append('../../plugins/test_plugin/')
# required hack in order for resource of plugin could be visible (https://geo2tag.atlassian.net/browse/GT-1438)
from test_resource_1 import TestResource1
from test_resource_2 import TestResource2

and create mandatory functions for the plugin:

def getPluginResources():
    result = {'res1': TestResource1, 'res2':TestResource2}  
    return result

def getPluginInfo():
    info = 'This plugin was creating for present plugins feature. The function getPluginResources is for return TestResource1 and TestResource2 '
    return info

Then you have to create the directory /var/www/geomongo/plugins/test_plugin and place into it files main.py, test_resource_2.py, test_resource_1.py and empty file  __init__.py.

Next, you need to restart the web server to run the plugins up.

   sudo service apache2 restart

After that, the plugins will be available at:

http://geomongo/instance/plugin/test_plugin/res1

http://geomongo/instance/plugin/test_plugin/res2

ShareShare on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInShare on VKEmail this to someone

Format and connection of plugins

The form of plugins

Plugin is a set of external classes-resources connected to Geomongo. Technically, plugin is a python-package (directory with source code files and file __init__.py) satisfying the following rules:

  • it contains file main.py;
  • the name of directory should be written by latin lowercase letters  and underscores should be used instead of space characters.

Directory with plugins for installed application: /var/www/geomongo/plugins.

File  main.py should contain:

  • function getPluginResources() which returns dictionary with couples ‘link’: Resource’s derived class;
  • function getPluginInfo() which returns string description of the plugin.

Plugin connection

  • the directory of plugin should be copied to  /var/www/geomongo/plugins
  • to activate the plugin administrator should run the query /<geo2tag_root_prefix>/manage_plugins?plugin_name=true (issue GT-1421)
  • to get list of enabled plugins administrator should run the query /<geo2tag_root_prefix>/plugin (issue GT-1421).
ShareShare on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInShare on VKEmail this to someone