Saint George on a Bike (SGoaB) has developed a set of tools that can detect the objects in an image through deep learning models for object detection, and can further recommend relationships between objects. This was done by applying bounding box heuristics and natural language processing technology, as well as by annotating human-like objects with pose labels. Additionally, the set of tools makes it possible to refine object labels - or recommend relationships. This reduces the manual work of annotating objects from new classes (e.g. class knight as a refinement of class person may be inferred rather than requiring manual annotation of knights) and allows inference of object labels even when there are not many instances of those classes for the deep learning algorithm to be properly trained. Lastly, the toolset includes an image description generator trained using deep learning.
SGoaB’s trained models have shown good results and may be used via an API the project offers. Researchers interested in learning to use the tools may email sgoab-contact@bsc.es for support. Documentation on the use of this API may be found here.