Services metadata

Introduction

Metadata are the elements of collection metadata in database of a one service. They are needed to store in a service data that are not related to geographical coordinates.

Software interfaces

Exception MetadataDoesNotExist 

Functions:

  • getMetadataById(serviceName, _id) – search of element metadata by _id
  • findMetadata(serviceName, number, offset, queryPairs)  – search of elements by the query queryPairs and limited output through number and offset
  • deleteMetadataById(serviceName, _id) – deleting an element
  • setMetadata(serviceName, data, _id) – adding (update) an element

REST API

  • /<prefix>/service/<service_name>/metadata
    • POST, adding a new element. Parameter is an arbitrary json structure
    • GET filtering the existing elements of the collection:
      • mandatory parameters: offset, number
      • optional parameters – an arbitrary set of couples key-value (including
        hierarchical structures of type field1.field2.0 that are corresponding  field1[‘field2’][0])
  • /<prefix>/service/<service_name>/metadata/<metadata_id>
    • GET – get a single element
    • PUT – edit an existing element. Parameter is an arbitrary json structure.
    • DELETE – delete an element.

 

ShareShare on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInShare on VKEmail this to someone

Plugins configuration

External interfaces

Each plugin can store its own settings independently in file /plugins/<plugin_name>/config.ini.

The platform provides:

  • common interfaces for reading the plugins settings using the class PluginConfigReader;
  • output the availability of configs of plugins via /<prefix>/plugin;
  • web interface for reading/editing configs of the plugins /<instance_perfix>/plugin_config/<plugin_name> (р PluginConfigResource):
    • GET – returns the content of a plugin;
    • PUT – rewrites the config again;
    • Transfer/receiving data is performed in format JSON.
{  
  section1: {param1:val1, param2: val2},
  section1: {param3:val3},
}

 Software interfaces

PluginConfigReader

class PluginConfigReader:
    def __init__(self, pluginName):
    # Returns content of the config.ini in a form of a dictionary

    def getConfigContent(self):
    # Sets plugin config.ini content to content dictionary


    def setConfigContent(self, content):

PluginConfigResource

class PluginConfigResource(Resource):
    @possibleException
    def get(self, pluginName):
        # Check the presence of plugin with the specified name 
        return PluginConfigReader().getConfigContent()


    @possibleException
    def put(self, pluginName):
        # Check the presence of plugin with the specified name 
        # call PluginConfigReader().setConfigContent(postData)
ShareShare on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInShare on VKEmail this to someone

Interfaces of the plugins for OpenData import

This article describes the basic interfaces of the plugins importing open data.

Terms

Job is the the sequence of actions for loading an array of open data through the http protocol, including their processing and loading into Geo2Tag service’s DB. 

Import is transfer data from an external DB into DB of a Geo2Tag service.

Job status is a set of parameters describing current status of a job: time spent, source of import, the flag of completion of the job, other import options.

REST interfaces

  • /<instance_prefix>/plugin/<plugin_name>/service/<serviceName>/job GET – list of the jobs of import and its statuses.
  • /<instance_prefix>/plugin/<plugin_name>/service/<serviceName>/job POST – add a job of import, parameters:
    • channelName, – the name of the channel where importing data will be performed; 
    • openDataUrl,  – link to downloading the set of data;
  • /<instance_prefix>/plugin/<plugin_name>/service/<serviceName>/job/<job_id> GET – the status of the job.
  • /<instance_prefix>/plugin/<plugin_name>/service/<serviceName>/job/<job_id> DELETE – stop the job.

Software interfaces

Classes

Legend:

  • @should_be_extended_in_descendents – method/the property should be extended/override in the derived class
  • @abstract_method – an abstract method which should be defined in the derived class

Job (abstract class)

class Job():

    def __init__(
            self,
            backgroundFunction,
            channelName,
            openDataUrl,
            importDataDict,
            serviceName):
        self.thread = None
        self._id = self.generateId()
        self.startTime = datetime.now()
        self.done = False
        self.timeElapsed = None
        self.backgroundFunction = backgroundFunction
        self.channelName = channelName
        self.openDataUrl = openDataUrl
        self.importDataDict = importDataDict
        self.serviceName = serviceName

    @abstract_method
    def internalStart(self):
    @abstract_method
    def internalStop(self):

    def start(self):
        self.startTime = datetime.now()
        self.internalStart()

    def stop(self):
        self.internalStop()
        self.done = True
        self.timeElapsed = datetime.now() - self.startTime

    def getTimeStatistics(self):
        if self.timeElapsed is None:
            return datetime.now() - self.startTime
        return self.timeElapsed

    @should_be_extended_in_descendents
    def describe(self):
        return {
            '_id': self._id,
            'time': str(
                self.getTimeStatistics()),
            'done': self.done,
            'channelName': self.channelName,
            'openDataUrl': self.openDataUrl,
            'serviceName': self.serviceName}

    @classmethod
    def generateId(cls):
        return ''.join(
            random.choice(
                string.ascii_uppercase +
                string.ascii_lowercase +
                string.digits) for x in range(12))

OpenDataObjectsLoader (abstract class)

class OpenDataObjectsLoader:
    def __init__(self, loadUrl):
        self.loadUrl = loadUrl
    @abstract_method
    def load(self):     

OpenDataToPointTranslator

class OpenDataToPointTranslator:
    def __init__(
            self, importDataDict,
            objectRepresentation,
            version,
            importSource,
            channelId):
        self.objectRepresentation = objectRepresentation
        self.version = version
        self.importSource = importSource
        self.channelId = channelId
        self.importDataDict = importDataDict
    @should_be_extended_in_descendents
    def getPointJson(self):
        obj = {}
        obj['version'] = self.version
        obj['import_source'] = self.importSource
        return obj
    @should_be_extended_in_descendents
    def getPoint(self):
        point = {'json': self.getPointJson()}
        point['channel_id'] = self.channelId
        return point

OpenDataObjectsParser (abstract class)

class OpenDataObjectsParser:
    def __init__(self, data):
        self.data = data
    @abstract_method
    def parse(self):

OpenDataToPointsLoader

class OpenKareliaDataToPointsLoader:
    pointsArray = []

    def __init__(self, serviceName, points):
        self.pointsArray = points
        self.serviceName = serviceName
    def loadPoints(self):
        collection = getDbObject(self.serviceName)[POINTS]
        for point in self.pointsArray:
            collection.save(point)

JobManager – class for managing jobs, starts and stops jobs, displays information about their status.

class JobManager:
    jobs = {}
    @classmethod
    def startJob(cls, job):
        jobId = job.describe().get('_id', '')
        job.start()
        cls.jobs[jobId] = job
        return jobId
    @classmethod
    def getJob(cls, jobId):
        return cls.jobs.get(jobId).describe()
    @classmethod
    def stopJob(cls, jobId):
        cls.jobs.get(jobId).stop()
    @classmethod
    def getJobs(cls):
        result = []
        for job in cls.jobs:
            result.append(cls.jobs[job].describe())
        return result

JobResource

class JobResource(Resource):
    @possibleException
    def get(self, serviceName, jobId):
        return JobManager.getJob(jobId)
    @possibleException
    def delete(self, serviceName, jobId):
        return JobManager.stopJob(jobId)

ODImportParser

class ODImportParser():  
    @should_be_extended_in_descendents
     MANDATORY_FIELDS = [CHANNEL_NAME, OPEN_DATA_URL]
    @staticmethod
    def parse(self):
        args = loads(request.get_data())
        return args
    @staticmethod
    def validate(self, args):    
       for key in MANDATORY_FIELDS:
            if key not in args:
                raise BadRequest('{0} parameter is missing'.format(key))
            elif not isinstance(args[key], unicode):
                raise BadRequest('{0} value is not unicode'.format(key)) 
    @staticmethod
    def parsePostParameters():
        args = self.parse()
        self.validate(args)
        return args

JobListResourceFactory – factory method to create JobListResource classes which we need.

def JobListResourceFactory(parserClass, jobClass, importFunction)
    class JobListResource(Resource):
        @possibleException
        def get(self, serviceName):
            getServiceIdByName(serviceName)
            return JobManager.getJobs()
        @possibleException
        def post(self, serviceName):
            importDataDict = parserClass.parsePostParameters()
            channelName = importDataDict.get('channelName')
            getServiceIdByName(serviceName)
            getChannelByName(serviceName, channelName)
            job = jobClass(importFunction, importDataDict.get('channelName'),
                           importDataDict.get('openDataUrl'), importDataDict,
                           serviceName)
        return JobManager.startJob(job)
    return JobListResource

Method performImportActions

performImportActions ( odLoaderClass, odParserClass, odToPointTranslatorClass, odToPointsLoaderClass, serviceName, channelName, openDataUrl, showObjectUrl, showImageUrl)

A function which contains the logic of importing from source of open data.

def performImportActions ( odLoaderClass, odParserClass, \
    odToPointTranslatorClass, odToPointsLoaderClass, \
    serviceName, channelName, openDataUrl, \
    importDataDict):
#showObjectUrl, showImageUrl
    channelId = getChannelIdByName(channelName)
    version = datetime.now()
    loader = odLoaderClass(openDataUrl)
    openData = loader.load()
    parser = odParserClass(openData)
    objects = parser.parse()
    points = [ ]
    for object in objects:
        translator = odToPointTranslatorClass(importDataDict, object, version,openDataUrl, channelId)
        points.append(translator.getPoint())
    pointsLoader = odToPointsLoaderClass(serviceName, points)
    pointsLoader.loadPoints()

 

 

ShareShare on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInShare on VKEmail this to someone

Geomongo db schema

MasterDB

A database with meta-information about existing services and users.

users

Collection with users/administrations of the services and/or instance of platform.

{
    "first_name": {
        "type": "string"
    },
    "last_name": {
        "type": "string"
    },
    "required": [
        "first_name",
        "last_name"
    ]
}

services

Collection with meta-information of existing services.

{
    "name": {
        "type": "string"
    },
    "config": {
        "type": "object",
        "properties": {
            "log_size": {
                "type": "integer"
            }
        },
        "required": [
            "log_size"
        ]
    },
    "owner_id": {
        "type": "string"
    },
    "required": [
        "config",
        "name",
        "owner_id"
    ]
}

log

Action log administrators of the instance and services.

{
    "user_id": {
        "type": "string"
    },
    "date": {
        "type": "Datetime"
    },
    "message": {
        "type": "string"
    },
    "service": {
        "type": "string"
    },
    "required": [
        "user_id",
        "date",
        "message",
        "service"
    ]
}

ServiceDb

Database with data of specific service. One service needs one serviceDb.

log

Action log users of specific service. 

{
    "user_id": {
        "type": "string"
    },
    "date": {
        "type": "Datetime"
    },
    "message": {
        "type": "string"
    },
    "service": {
        "type": "string"
    },
    "required": [
        "user_id",
        "date",
        "message",
        "service"
    ]
}

points

Points added by users of the service.

{
    "name": {
        "type": "string"
    }
    "json": {
        "type": "object",
        "properties": {
            "import_source": {
                "type": "Url"
            },
            "version": {
                "type": "string"
            },
            "image_url": {
                "type": "Url"
            },
            "name": {
                "type": "string"
            },
            "source_url": {
                "type": "Url"
            }
            "date":{
                "type": "Datetime"
            }
            "bc":{
                "type": "boolean"
            }
        }
    },
    "channelId": {
       "type": "string"
    },
    "location": {
        "type": "object",
        "properties": {
            "coordinates": {
                "type": "array",
                "size": 2
                "items": {
                    "type": "float"
                }
            }
        },
        "required": [
            "coordintes"
        ]
    },
    "date":{
        "type": "Datetime"
    }
    "bc":{
        "type": "boolean"
    }
    "alt": {
        "type": "non negative float"
    }, 
    "required": [
        "json"
        "channelId",
        "location"
        "date",
        "bc",
        "alt"
    ]
}

channels

The channels of specified service.

{
    "name": {
      "type": "string"
    },
    "json": {
        "type": "object",
        "properties": {
            "description": {
                "type": "string"
            },
            "url": {
                    "type": "string"
            }
        }
    "owner_id": {
        "type": "string"
    },
    "required": [
        "name",
        "json",
        "owner_id"
    ]
}

http://docs.mongodb.org/manual/reference/database-references/

ShareShare on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInShare on VKEmail this to someone

REST API and queries format

Log

  1. //service//log?number=10&offset=0&date_from=1983-01-22T08:00:00&date_to=2016-01-22T08:00:00
  2. method GET
  3. the query requires authorization
  4. input parameters:
    • number (type int, optional),
    • offset (type int, optional),
    • date_from (type date_utils.datetime_from_iso8601, optional),
    • date_to (type date_utils.datetime_from_iso8601, optional)
  5. output parameters:
{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://jsonschema.net",
  "type": "array",
  "items": {
    "id": "http://jsonschema.net/0",
    "type": "object",
    "properties": {
      "date": {
        "id": "http://jsonschema.net/0/date",
        "type": "object",
        "properties": {}
      },
      "message": {
        "id": "http://jsonschema.net/0/message",
        "type": "string"
      },
      "_id": {
        "id": "http://jsonschema.net/0/_id",
        "type": "object",
        "properties": {}
      },
      "user_id": {
        "id": "http://jsonschema.net/0/user_id",
        "type": "string"
      }
    },
    "required": [
      "date",
      "message",
      "_id",
      "user_id"
    ]
  },
  "required": [
    "0"
  ]
}

Channels

  1. //service//channel
  2. method POST
  3. the query requires authorization
  4. input parameters 
    • name (type unicode, mandatory),
    • json (type unicode, mandatory)
  5. output parameters:
id of created channel
  1. //service//channel?number=2&offset=0&substring=test
  2. method GET 
  3. the query requires authorization
  4. input parameters
    • substring (type unicode, optional),
    • number (type int,  optional),
    • offset (type int, optional)
  5.  output parameters:
{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://jsonschema.net",
  "type": "array",
  "items": [
    {
      "id": "http://jsonschema.net/0",
      "type": "object",
      "properties": {
        "json": {
          "id": "http://jsonschema.net/0/json",
          "type": "object",
          "properties": {
            "url": {
              "id": "http://jsonschema.net/0/json/url",
              "type": "string"
            },
            "description": {
              "id": "http://jsonschema.net/0/json/description",
              "type": "string"
            }
          },
          "required": [
            "url",
            "description"
          ]
        },
        "_id": {
          "id": "http://jsonschema.net/0/_id",
          "type": "object",
          "properties": {}
        },
        "name": {
          "id": "http://jsonschema.net/0/name",
          "type": "string"
        },
        "owner_id": {
          "id": "http://jsonschema.net/0/owner_id",
          "type": "string"
        }
      },
      "required": [
        "json",
        "_id",
        "name",
        "owner_id"
      ]
    },
    {
      "id": "http://jsonschema.net/1",
      "type": "object",
      "properties": {
        "json": {
          "id": "http://jsonschema.net/1/json",
          "type": "object",
          "properties": {}
        },
        "_id": {
          "id": "http://jsonschema.net/1/_id",
          "type": "object",
          "properties": {}
        },
        "name": {
          "id": "http://jsonschema.net/1/name",
          "type": "string"
        },
        "owner_id": {
          "id": "http://jsonschema.net/1/owner_id",
          "type": "string"
        }
      }
    }
  ],
  "required": [
    "0",
    "1"
  ]
}
  1.  //service//channel/CHANNEL_ID
  2. method PUT
  3. the query requires authorization
  4. input parameters
    • name (type unicode, mandatory),
    • json (type unicode, optional),
    • acl (type int, optional)
  5.  output parameters:
{}
  1.  //service//channel/CHANNEL_ID 2)
  2. method GET 
  3. the query requires authorization
  4. input parameters: CHANNEL_ID (type string, mandatory)
  5. output parameters:
{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://jsonschema.net",
  "type": "object",
  "properties": {
    "owner_group": {
      "id": "http://jsonschema.net/owner_group",
      "type": "string"
    },
    "name": {
      "id": "http://jsonschema.net/name",
      "type": "string"
    },
    "acl": {
      "id": "http://jsonschema.net/acl",
      "type": "integer"
    },
    "json": {
      "id": "http://jsonschema.net/json",
      "type": "string"
    },
    "_id": {
      "id": "http://jsonschema.net/_id",
      "type": "object",
      "properties": {}
    },
    "owner_id": {
      "id": "http://jsonschema.net/owner_id",
      "type": "string"
    }
  },
  "required": [
    "owner_group",
    "name",
    "acl",
    "json",
    "_id",
    "owner_id"
  ]
}
  1.  //service//channel/CHANNEL_ID
  2. method DELETE 
  3. the query requires authorization
  4. input parameters: CHANNEL_ID (type string, mandatory)
  5. output parameters:
{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://jsonschema.net",
  "type": "object",
  "properties": {}
}

Points

  1.  //service//point
  2. method POST 
  3. the query requires authorization
  4. input parameters
    • lon (type float, mandatory), 
    • lat (type float, mandatory),  
    • alt (type float, mandatory),
    • json (type словарь, mandatory),
    • channel_id (type unicode,  mandatory)
  5. output parameters:
[point_id]
  1. /<geo2tag_prefix>//service//point?channel_ids=556721a52a2e7febd2744201&number=10&altitude_from=0&altitude_%20to=5&offset=0&geometry=None&date_from=1970-09-10T01:01:01.000000&date_to=2100-09-10T01:01:01.000000&geometry={“type”: “Point”, “coordinates”: [1, 1]}
  2. method GET 
  3. the query requires authorization
  4. input parameters
    • channel_ids (type unicode, mandatory),
    • number (type int, mandatory),
    • geometry (type geo_json_type.GeoJsonType, optional),
    • altitude_from (type float, optional),
    • altitude_to (type float, optional),
    • date_from (type date_utils.datetime_from_iso8601, optional),
    • date_to (type date_utils.datetime_from_iso8601, optional),
    • offset (type int, optional),
    • radius (type float, optional)
  5. output parameters:
{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://jsonschema.net",
  "type": "array",
  "items": {
    "id": "http://jsonschema.net/0",
    "type": "object",
    "properties": {
      "channel_id": {
        "id": "http://jsonschema.net/0/channel_id",
        "type": "object",
        "properties": {}
      },
      "json": {
        "id": "http://jsonschema.net/0/json",
        "type": "object",
        "properties": {
          "image_url": {
            "id": "http://jsonschema.net/0/json/image_url",
            "type": "string"
          },
          "description": {
            "id": "http://jsonschema.net/0/json/description",
            "type": "string"
          }
        }
      },
      "location": {
        "id": "http://jsonschema.net/0/location",
        "type": "object",
        "properties": {
          "type": {
            "id": "http://jsonschema.net/0/location/type",
            "type": "string"
          },
          "coordinates": {
            "id": "http://jsonschema.net/0/location/coordinates",
            "type": "array",
            "items": [
              {
                "id": "http://jsonschema.net/0/location/coordinates/0",
                "type": "integer"
              },
              {
                "id": "http://jsonschema.net/0/location/coordinates/1",
                "type": "integer"
              }
            ]
          }
        }
      },
      "date": {
        "id": "http://jsonschema.net/0/date",
        "type": "object",
        "properties": {}
      },
      "alt": {
        "id": "http://jsonschema.net/0/alt",
        "type": "integer"
      },
      "_id": {
        "id": "http://jsonschema.net/0/_id",
        "type": "object",
        "properties": {}
      }
    },
    "required": [
      "channel_id",
      "json",
      "location",
      "date",
      "alt",
      "_id"
    ]
  },
  "required": [
    "0"
  ]
}
  1.  //service//point/POINT_ID
  2. method PUT 
  3. the query requires authorization
  4. input parameters
    • lon (type float, optional),
    • lat (type float, optional),
    • alt (type float, optional),
    • json (type unicode, optional),
    • channel_id (type unicode, optional)
  5.  output parameters:
{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://jsonschema.net",
  "type": "object",
  "properties": {}
}
  1. //service//point/POINT_ID
  2. method GET 
  3. the query requires authorization
  4.  input parameters: point_id
  5. output parameters:
{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://jsonschema.net",
  "type": "object",
  "properties": {
    "channel_id": {
      "id": "http://jsonschema.net/channel_id",
      "type": "null"
    },
    "json": {
      "id": "http://jsonschema.net/json",
      "type": "null"
    },
    "location": {
      "id": "http://jsonschema.net/location",
      "type": "object",
      "properties": {
        "type": {
          "id": "http://jsonschema.net/location/type",
          "type": "string"
        },
        "coordinates": {
          "id": "http://jsonschema.net/location/coordinates",
          "type": "array",
          "items": [
            {
              "id": "http://jsonschema.net/location/coordinates/0",
              "type": "number"
            },
            {
              "id": "http://jsonschema.net/location/coordinates/1",
              "type": "number"
            }
          ]
        }
      }
    },
    "date": {
      "id": "http://jsonschema.net/date",
      "type": "object",
      "properties": {}
    },
    "alt": {
      "id": "http://jsonschema.net/alt",
      "type": "integer"
    },
    "_id": {
      "id": "http://jsonschema.net/_id",
      "type": "object",
      "properties": {}
    }
  },
  "required": [
    "channel_id",
    "json",
    "location",
    "date",
    "alt",
    "_id"
  ]
}
  1. //service//point/POINT_ID
  2. method DELETE 
  3. the query requires authorization
  4. input parameters: point_id
  5. output parameters:
{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://jsonschema.net",
  "type": "object",
  "properties": {}
}

Services

  1.  //service
  2. method POST
  3. the query requires authorization
  4. input parameters: 
    • name (type unicode, mandatory),
    • logSize (type int, optional),
    • ownerId (unicode, optional)
  5.  output parameters: 
{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://jsonschema.net",
  "type": "object",
  "properties": {},
  "required": [
    "$oid"
  ]
}
  1. //service?number=0&offset=0
  2. method GET
  3. the query requires authorization
  4. input parameters: 
    • number (type int, optional),
    • offset (type int, optional)
  5. output parameters:
{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://jsonschema.net",
  "type": "array",
  "items": [
    {
      "id": "http://jsonschema.net/0",
      "type": "object",
      "properties": {
        "_id": {
          "id": "http://jsonschema.net/0/_id",
          "type": "object",
          "properties": {}
        },
        "config": {
          "id": "http://jsonschema.net/0/config",
          "type": "object",
          "properties": {
            "log_size": {
              "id": "http://jsonschema.net/0/config/log_size",
              "type": "integer"
            }
          }
        },
        "name": {
          "id": "http://jsonschema.net/0/name",
          "type": "string"
        },
        "owner_id": {
          "id": "http://jsonschema.net/0/owner_id",
          "type": "string"
        }
      },
      "required": [
        "_id",
        "config",
        "name",
        "owner_id"
      ]
    },
    {
      "id": "http://jsonschema.net/1",
      "type": "object",
      "properties": {
        "_id": {
          "id": "http://jsonschema.net/1/_id",
          "type": "object",
          "properties": {}
        },
        "config": {
          "id": "http://jsonschema.net/1/config",
          "type": "object",
          "properties": {
            "log_size": {
              "id": "http://jsonschema.net/1/config/log_size",
              "type": "integer"
            },
            "logSize": {
              "id": "http://jsonschema.net/1/config/logSize",
              "type": "integer"
            }
          }
        },
        "name": {
          "id": "http://jsonschema.net/1/name",
          "type": "string"
        },
        "owner_id": {
          "id": "http://jsonschema.net/1/owner_id",
          "type": "string"
        }
      }
    }
  ],
  "required": [
    "0",
    "1"
  ]
}

 //service/SERVICE_NAME 

method PUT

the query requires authorization

input parameters:  logSize (type int, mandatory)

output parameters: 

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://jsonschema.net",
  "type": "object",
  "properties": {
    "SERVICE_NAME": {
      "id": "http://jsonschema.net/SERVICE_NAME",
      "type": "string"
    }
  },
  "required": [
    "SERVICE_NAME"
  ]
}
  1. //service/SERVICE_NAME
  2. method GET
  3. the query requires authorization
  4. input parameters: service_name
  5. output parameters:
{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://jsonschema.net",
  "type": "object",
  "properties": {
    "_id": {
      "id": "http://jsonschema.net/_id",
      "type": "object",
      "properties": {}
    },
    "config": {
      "id": "http://jsonschema.net/config",
      "type": "object",
      "properties": {
        "log_size": {
          "id": "http://jsonschema.net/config/log_size",
          "type": "integer"
        },
        "logSize": {
          "id": "http://jsonschema.net/config/logSize",
          "type": "integer"
        }
      }
    },
    "name": {
      "id": "http://jsonschema.net/name",
      "type": "string"
    },
    "owner_id": {
      "id": "http://jsonschema.net/owner_id",
      "type": "string"
    }
  },
  "required": [
    "_id",
    "config",
    "name",
    "owner_id"
  ]
}
  1. //service/SERVICE_NAME
  2. method DELETE 
  3. the query requires authorization
  4. input parameters: service_name
  5. output parameters:
{
"$schema": "http://json-schema.org/draft-04/schema#",
"id": "http://jsonschema.net",
"type": "object",
"properties": {}
}

Plugins

  1. //plugin
  2. method GET 
  3. the query requires authorization
  4. input parameters: absent
  5. output parameters:
{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://jsonschema.net",
  "type": "object",
  "properties": {
    "test_plugin_for_fail": {
      "id": "http://jsonschema.net/test_plugin_for_fail",
      "type": "boolean"
    },
    "test_plugin_for_load_fail_gt_1435": {
      "id": "http://jsonschema.net/test_plugin_for_load_fail_gt_1435",
      "type": "boolean"
    },
    "test_plugin": {
      "id": "http://jsonschema.net/test_plugin",
      "type": "boolean"
    },
    "ok_import": {
      "id": "http://jsonschema.net/ok_import",
      "type": "boolean"
    },
    "testPlugin1": {
      "id": "http://jsonschema.net/testPlugin1",
      "type": "boolean"
    },
    "testPlugin2": {
      "id": "http://jsonschema.net/testPlugin2",
      "type": "boolean"
    }
  },
  "required": [
    "test_plugin_for_fail",
    "test_plugin_for_load_fail_gt_1435",
    "test_plugin",
    "ok_import",
    "testPlugin1",
    "testPlugin2"
  ]
}
  1.  //manage_plugins?PLUGIN_NAME=True
  2. method GET
  3. the query requires authorization
  4. input parameters: list of the plugins names specifying the status of each plugin (True/False)
  5. output parameters:
null

Logout

  1.  //logout
  2. method GET 
  3. the query requires authorization
  4. input parameters: absent
  5. output parameters
null
ShareShare on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInShare on VKEmail this to someone

Класс Geo2TagRequests

Класс Geo2TagRequests находится в файле src/static/js/geo2tag.js.

У класса есть поля server и instance.

Методы класса:

1) getChannels:

  • Параметры: serviceName, callbacksuccess, callbackfail, substring, number, offset
  • Делает запрос /instance/service/serviceName/channel GET с параметрами
  • В случае успеха вызывает callbacksuccess(результат запроса)
  • В случае ошибки вызывается callbackfail(результат запроса)

2) getPoints:

  • Параметры: serviceName, callbacksuccess, callbackfail, channel_ids, number, geometry, altitude_from, altitude_to, substring, date_from, date_to, offset, radius
  • Делает запрос /instance/service/serviceName/point GET с параметрами
  • В случае успеха вызывает callbacksuccess(результат запроса)
  • В случае ошибки вызывается callbackfail(результат запроса)

Для создания объекта класса используется код:

var test_obj = new Geo2TagRequests(‘test’, ‘test’);

Примеры скрипта, работающего с классом Geo2TagRequests:

SCRIPT

В результате получается 2 запроса:

  1. http://geomongo/instance/service/testservice/channel?substring=substring&number=4&offset=5
  2. http://geomongo/instance/service/testservice/point?channel_ids=556721a52a2e7febd2744201&channel_ids=556721a52a2e7febd2744202&number=0&geometry=%7B%22coordinates%22%3A+%5B-115.8%2C+37.2%5D%2C+%22type%22%3A+%22Point%22%7D&altitude_from=22&altitude_to=5&substring=substring&date_from=1970-06-15T18%3A00%3A00&date_to=2015-06-15T17%3A00%3A00&offset=0&radius=5

Данный пример скрипта иллюстрирует работу класса Geo2TagRequests. 

Параметры, передающиеся в конструктор:

Geo2TagRequests('testservice', getInstancePrefix());

имеют тестовые значения, чтобы URL, по которому мы пытаемся перейти, была доступна.

Функция getInstancePrefix в данном примере возвращает ‘instance’, а в общем случае: ‘{{ instance_prefix }}’.

Тесты для методов находятся в файле src/static/js/tests/geo2tag_requests_tests.js.

ShareShare on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInShare on VKEmail this to someone

Class Geo2TagRequests

Class Geo2TagRequests is located in the file src/static/js/geo2tag.js.

It has fields server and instance.

Class methods:

1)  getChannels:

  • Parameters: serviceName, callbacksuccess, callbackfail, substring, number, offset
  • Makes the query: /instance/service/serviceName/channel GET with parameters
  • In case of success calls callbacksuccess(result of the query)
  • On failure callbackfail(result of the query) will be called

2) getPoints:

  • Parameters: serviceName, callbacksuccess, callbackfail, channel_ids, number, geometry, altitude_from, altitude_to, substring, date_from, date_to, offset, radius
  • Makes the query:/instance/service/serviceName/point GET with parameters
  • In case of success calls callbacksuccess(result of the query)
  • On failure callbackfail(result of the query) will be called

To create a class object the next code is used:

var test_obj = new Geo2TagRequests('test', 'test');

The examples of script working with Geo2TagRequests:

<script type="text/javascript">
    var callbackSuccess = function (data) {//the declaration of a function that will be called if the guery getPoints or getChannels has been completed successfully
        console.log('Query executed successfully');
    };
    var callbackFail = function (data) {//the declaration of a function that will be called if the guery getPoints or getChannels was failed
        console.log('Query failure'); 
    }; 
    
    var g2t = new Geo2TagRequests('testservice', getInstancePrefix());//creating an instance of the class Geo2TagRequests

 

    g2t.getChannels('testservice', callbackSuccess, callbackFail, 'substring', 4, 5);//call of the method getChannels (in the console the message  'Query executed successfully' should appear)
    
    g2t.getPoints('testservice', callbackSuccess, callbackFail, ["556721a52a2e7febd2744201", "556721a52a2e7febd2744202"], 0, '{"coordinates": [-115.8, 37.2], "type": "Point"}', 22, 5, 'substring', '1970-06-15T18:00:00', '2015-06-15T17:00:00', 0, 5, true, false); //call of the method getPoints (in the console the message 'Query executed successfully' should appear)

</script>

As a result we get two queries:

  1. http://geomongo/instance/service/testservice/channel?substring=substring&number=4&offset=5
  2. http://geomongo/instance/service/testservice/point?channel_ids=556721a52a2e7febd2744201&channel_ids=556721a52a2e7febd2744202&number=0&geometry=%7B%22coordinates%22%3A+%5B-115.8%2C+37.2%5D%2C+%22type%22%3A+%22Point%22%7D&altitude_from=22&altitude_to=5&substring=substring&date_from=1970-06-15T18%3A00%3A00&date_to=2015-06-15T17%3A00%3A00&offset=0&radius=5

This example script illustrates the work of class Geo2TagRequests.

The next parameters are passed to the constructor:

Geo2TagRequests('testservice', getInstancePrefix());

They have text values in order to the URL that we have to follow was available. Function getInstancePrefix in our example returns ‘instance’, in common case: : ‘{{ instance_prefix }}’.

Tests for the methods are located in the file src/static/js/tests/geo2tag_requests_tests.js.

ShareShare on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInShare on VKEmail this to someone

The date and time format and it processing when parsing

The common format of date and time used in our project is ISO. Basically it has the next form:

ISO8601_FMT = Y-m-dTH:M:S.mmmmmm

Example: 2000-01-01T01:01:01.123456

The length of microseconds should be always defined specifically, like as in the example (6 digits)

If microseconds are equal to zero they should be anyway prescribed:

for example:

2000-01-01T01:01:01.000000

When parsing, for example in case GET query, you can use the next function to get date and time in format datetime and check that the format is correct:

def datetime_from_iso8601(datetime_str):
    return json.dumps(
        dateDeserialiser({DATE_FIELD: datetime_str}, DATE_FIELD),
        default=dateSerialiser)

Function aniso8601.parse_datetime(datetime_str) receives a date string which has ISO format. Next the function converts that string to datetime format. If the string has incorrect format, exception ValueError will be thrown.

Then obtained datetime object  is passed to the function dateSerialiser where date and time will be converted to a comprehensible for JSON string.

def dateSerialiser(obj):
    if isinstance(obj, datetime):
        return '{0}{1}'.format(obj.isoformat(), '' if obj.microsecond > 0 else '.000000')
    raise TypeError('Type is not datetime')

Example of using:

https://bitbucket.org/osll/geomongo/src/71d768d45062f3121d8dbacd529502f616fb3034/src/log_parsers.py?at=master#cl-20

To extract the date from the JSON format is used the next function:

def dateDeserialiser(dict_with_date, param_date):
    try:
        if param_date in dict_with_date and dict_with_date[param_date] is not None:
            obj = dict_with_date[param_date].replace(''', "").replace('"', '')
            return datetime.strptime(obj, ISO8601_FMT_MILLS)
    except ValueError:
        print ISO_FORMAT_ERROR
        raise
    return None

This function receives the JSON dictionary which has the converted date object and time object and the key param_date that points on date object and time object to retrieving. Function datetime.strptime(datetime_str, datetime_format) converts a string with the date and time in the described format to datetime object.

Example of using:

https://bitbucket.org/osll/geomongo/src/71d768d45062f3121d8dbacd529502f616fb3034/src/log_resource.py?at=master#cl-29

ShareShare on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInShare on VKEmail this to someone

Формат даты и времени и его обработка при парсинге

Общий формат даты и времени, используемый в проекте – ISO формат. В общем виде формат имеет вид:

ISO8601_FMT = Y-m-dTH:M:S.mmmmmm

Пример: 2000-01-01T01:01:01.123456

Длина микросекунд должна быть всегда указана конкретно – как в примере (6 цифр).

Если микросекунды равны нулю,то они все равно пишутся т.е.

Пример: 2000-01-01T01:01:01.000000

При парсинге, например при GET запроса, для получения даты и времени в формате datetime и проверки того, что формат верен, можно использовать следующую функцию:

def datetime_from_iso8601(datetime_str):

return json.dumps( dateDeserialiser({DATE_FIELD: datetime_str}, DATE_FIELD), default=dateSerialiser)

Функция aniso8601.parse_datetime(datetime_str) принимает строку с датой, которя имеет ISO формат. Далее функция приводит эту строку к формату datetime. Если формат строки неправильный, возникнет исключение ValueError

Далее полученный datetime объект передается в функцию dateSerialiser, в которой дата и время преобразуются в понятную для JSON строку.

def dateSerialiser(obj):
    if isinstance(obj, datetime):
        return '{0}{1}'.format(obj.isoformat(), '' if obj.microsecond > 0 else '.000000')
    raise TypeError('Type is not datetime')

Пример использования: 

https://bitbucket.org/osll/geomongo/src/71d768d45062f3121d8dbacd529502f616fb3034/src/log_parsers.py?at=master#cl-20

Чтобы извлечь дату из JSON формата используется следующая функция:

def dateDeserialiser(dict_with_date, param_date):
    try:
        if param_date in dict_with_date and dict_with_date[param_date] is not None:
            obj = dict_with_date[param_date].replace(''', "").replace('"', '')
            return datetime.strptime(obj, ISO8601_FMT_MILLS)
    except ValueError:
        print ISO_FORMAT_ERROR
        raise
    return None

Данная функция получает на вход словарь JSON, в котором есть преобразованный объект даты и времени и ключ param_date, указывающий на объект даты и времени для извлечения. Функция datetime.strptime(datetime_str, datetime_format) преобразует строку с датой и временем, соответствующую формату описанному выше, в datetime объект.

Пример использования: 

https://bitbucket.org/osll/geomongo/src/71d768d45062f3121d8dbacd529502f616fb3034/src/log_resource.py?at=master#cl-29

ShareShare on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInShare on VKEmail this to someone

Как написать свой плагин

Пример написания плагина

Директория с плагинами для установленного приложения /var/www/geomongo/plugins.

Для создания плагина необходимо:

  1. создать каталог, название  – строчными буквами, слова разделены знаком нижнего подчеркивания;
  2. создать пустой файл __init__.py для того, чтобы данный каталог являлся python-пакетом;
  3. создать файл main.py c двумя функциями getPluginInfo() и getPluginResources()getPluginInfo() возвращает информацию о плагине, getPluginResources() –  словарь, который содержит пары ‘ссылка’: класс-наследник Resource.

Пример
Пусть существуют два класса-ресурса, которые мы хотим превратить в плагин.

test_resource_1.py:
 from flask.ext.restful import Resource
 class TestResource1(Resource):
   def get(self):
     return 'test_resource_1'
test_resource_2.py:
from flask.ext.restful import Resource
class TestResource2(Resource):
   def get(self):
     return 'test_resource_2'

Их необходимо подключить в main.py:

sys.path.append('../../plugins/test_plugin/')
# Хак, необходимый для того, чтобы ресурсы плагина было видно ( https://geo2tag.atlassian.net/browse/GT-1438)
from test_resource_1 import TestResource1
from test_resource_2 import TestResource2

а также создать обязательные для плагина функции getPluginResources() и getPluginInfo() main.py:

def getPluginResources():
    result = {'res1': TestResource1, 'res2':TestResource2}  
    return result

def getPluginInfo():
    info = 'This plugin was creating for present plugins feature. The function getPluginResources is for return TestResource1 and TestResource2 '
    return info

После этого необходимо создать /var/www/geomongo/plugins/test_plugin, и поместить туда main.py, test_resource_2.py, test_resource_1.py и пустой __init__.py.

Далее необходимо перезагрузить вебсервер, чтобы плагины поднялись:

sudo service apache2 restart

После этого плагины будут доступны по адресам:

http://geomongo/instance/plugin/test_plugin/res1

http://geomongo/instance/plugin/test_plugin/res2

ShareShare on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInShare on VKEmail this to someone