GSoC`20: Resumen del trabajo realizado

2020-08-30GSoCMifos Initiativeopen sourceVision PPIandroidcomputer vision

Work Report

Project Abstract

The Poverty Probability Index (PPI) is a poverty measurement tool for organizations and businesses with a mission to serve the poor. A PPI survey consists of 10 questions about a household’s characteristics using which asset ownership is scored to compute the likelihood that the household is living below the poverty line.

Leveraging Cloud Vision, a field officer would simply have to take a series of photos with their smartphone camera inside and outside of the house and then the Cloud Vision would be able to deduce based on the images the response to the 10 questions by detecting whether the objects mentioned in the survey are present in the household or not.

This year the project aims to do the following:

Increase the accuracy of the models for object detection
Train more models using the Google Cloud Vision platform to recognize more objects
Improve the current augmentation procedure to include more techniques and hence build a better dataset
Integrate the AutoML API with the android client
Implement auto-filling of the survey based on the results received from the AutoML API when it detects objects in the images uploaded
Enhance the UI/UX of the android client

Mentors: @Nayan, @rahulg963

Changes in the project plans

There were a few changes in the project plans once we actually got to working on it. The main change was that instead of using Google Cloud Vision and the AutoML API, we instead decided to go ahead with the MLKit SDK released by Google which is completely offline and doesn’t not require any internet connection.

The cons of using Cloud Vision were:

❌ Google Cloud Vision is paid and would cost more than a 1000$ per month just to keep the model deployed

❌ The initial training phase would also cost quite a bit of money

❌ There would be some latency in getting results back since data has to be uploaded, analysed and returned. The speed of this process would entirely depend on the connection speed of the user

The pros of going ahead with MLKit:

✅ No payment needed at all

✅ No latency as network connection isn’t needed

✅ No cost in training the model or re-training it

✅ Models can be trained using transfer learning with TensorFlow and then be converted into tflite models

Given these factors, we dropped the plan of using Google Cloud Vision and AutoML and instead decided to use MLKit.

Milestones

1) Data Augmentation

The first step of the project was to improve the data augmentation techniques. Since the it is difficult to find a dataset containing photos of the items needed in a household setting, it is important to increase the size of the dataset as much as possible by using augmentation techniques.

I created a new python script which takes in a directory of images as the input and augments all data and stores the output in another directory. I also created a Jupyter Notebook which can be used to visualize all the augmentation techinques on a single image without running the python script.

The augmentation techniques I added were:

Cropping
Histogram Equalization
Translation and Padding
A variety of blurring techniques
Saturation and Hue
Salt and pepper noise
Adaptive Gaussian thresholding
Affine transformation
Parallel Light Technique
Spot Light Technique

These techniques can be viewed here: https://github.com/openMF/ppi-vision/blob/master/DataCollection/ImageAugmentation.ipynb, done by #115.

The data augmentation script can be found here: https://github.com/openMF/ppi-vision/blob/master/DataCollection/GenerateImages.py, done by #113.

2) Image Labelling using MLKit and Autofilling the survey

The next step was to do the main image labelling part on an android device for which I have used MLKit. A user select multiple photos from their device, or click photos directly using a camera and use MLKit to detect all objects within the photo taken. A gif of this can be seen below:

For people intersted in knowing how object classification(the basis of image labelling works, I have also prepared a sample Jupyter Notebook that demonstrates the process over here: https://github.com/openMF/ppi-vision/blob/master/ImageClassification/ImageClassificationExample.ipynb.

The work for this part is split amongst the PRs #148, #151 and #153.

Once the objects within an image/images are detected, I used the data obtained to provide a user with suggestions as to which survery option would be most suitable. The system can detect whether a house has katcha or pakka outer walls, whether there is a television, a refridgerator, VCR, the number of telephones, etc. A screenshot of how this looks can be seen below:

3) Improving the UI/UX of the android app

The last major part of the project was to improve the UI/UX of the application. Since the app only needs to have 2 major screens, I shifted the layout from a navigation drawer based layout to a bottom navigation bar. Besides, I improved the display of the home search fragment, the user and clinet profiles, the image analysis section and the question display fragment.


Login Screen	Bottom Navigation + Enhanced user profile page	Bottom Navigation + Enhanced Search Fragment

Client Profile Page	Image Analysis Activity	Questions Fragment

4) Documentation

Earlier, there was hardly any documentation for the project. During my GSoC work, I also worked on documenting the application. I have done the following work on documenting the project:

I created the project wiki: https://github.com/openMF/ppi-vision/wiki

Created a proper readme to describe the project so anyone can see what the project is about: https://github.com/openMF/ppi-vision/blob/master/README.md

A detailed explanation of the project on the Mifos Confluence: https://mifosforge.jira.com/wiki/spaces/MIFOSX/pages/977305715/Computer+Vision+Based+PPI+Tool

Adding a Code of Conduct to the project: https://github.com/openMF/ppi-vision/blob/master/CODE_OF_CONDUCT.md

Adding contribution guidelines to help new contributors get started with the project easily: https://github.com/openMF/ppi-vision/blob/master/CONTRIBUTING.md

Add issue templates to the project for bug fixes, feature requests and chores: https://github.com/openMF/ppi-vision/tree/master/.github/ISSUE_TEMPLATE


Options that show up on creating an issue	Bug Report Template

Chore Template	Feature Request Template

Added a pull request template so contributors can ensure they followed some guidelines and reducing the chance of a build failure: https://github.com/openMF/ppi-vision/blob/master/.github/PULL_REQUEST_TEMPLATE.md


PR Template in markdown	PR Template when filled after contributor satisfies the conditions

5) Continuous Integration(CI)

There was no CI in the project earlier. I added CI using GitHub actions for both the python scripts as well as the android app. Now if any contributor sends a pull request, the changes can be built using GitHub actions and if any code is broken, the reviewers can see it right there.


Android Workflow	Python Workflow	GitHub Actions Results

Python CI Results	Android CI Results	Examples of a workflow failing on a PR

6) Other miscellaneous work done

Helped review merge all existing prs in the project as well as fix some minor bugs.
Refactored the login of the application using Retrofit and RXJava such that it doesn’t have a dependency on non-maintained libraries like khttp.
Added the feature to save the credentails once a user logs in and also a feature to log out. Earlier no credentials were saved and a user had to log in everytime the app was opened.
Reformat code properly using the spotless plugin.
Added an about section to the application so users can see the developers, the GitHub repo, license and social media pages of Mifos.

7) Link to APK

Link to APK: https://drive.google.com/file/d/1sZkow9zy7Hs6Bpc2GCsrpO6smVVQ6xH_/view
Link to live preview on appetize.io: https://appetize.io/app/9m1cruam36xd6k205p7qf4b3d4?device=nexus5&scale=75&orientation=portrait&osVersion=8.1

Pull Requests & Issues

Commits/PRs Made

#196: Show dialog to user when logging out feat
#194: Change profile icon fix
#192: Change color of options menu fix
#190: Improve code quality chore
#188: Rename variables to follow the general convention chore
#187: Make variables and methods private where ever needed chore
#186: Add ripple effect to buttons feat
#185: Add action bar to computer vision activity fix
#183: Show user a message if analyse images is pressed without any images selected fix
#182: Fix crash in clicking on a client fix
#181: Update README.md chore
#179: Add issue annd PR templates chore
#177: Add CONTRIBUTING.md chore
#175: Add a code of conduct to the project chore
#173: Add about screen feat
#172: Codestyle fix using spotless chore
#171: Improve UI of computer vision activity feat
#168: Update UI of question fragment feat
#166: Enhance UI of user and client profile sections feat
#165: Enhance UI of home page feat
#162: Shift to a bottom navigation layout feat
#157: Add feature to autofill the survey feat
#153: Remove google services dependencies chore
#151: Enhance object detection system feat
#148: Add MLKit Object Detection feat
#145: Add toolbar for ComputerVision activity fix
#137: Add an example image classification notebook feat
#135: Clean up app and remove unused resources chore
#133: Add option to logout feat
#131: Refactor login feat
#125: Add hindi translation feat
#117: CI for python scripts feat
89ad68e: Add CI using GitHub Actions feat
#115: Enhance the augmentation notebook feat
#113: Create python script for augmentation feat
#110: Fix the needed lint issues chore
#107: Add a new dependency for http requests feat
#105: Fix method of obtaining the authKey fix
#103: Update app dependencies and make other needed changes chore
#101: Migrate app dependencies to androidX chore
#99: Update gradle and kotlin version chore
#98: Update API endpoint chore
#97: Add parallel and spot light augmentation techniques feat
#96: Add salt and pepper noise, adaptive Gaussian thresholding and affine transformation feat
#95: Add saturation and hue techniques to augment images feat
#94: Fix unchecked nullability chore
#93: Fix idiomatic errors chore
#92: Disable search button when search query is empty fix
#91: Changed responsiveness of Login button based on emptiness of text fields fix
#90: Added augmentation techniques for blurring feat
#88: Enhance UI of client profile page feat
#87: Enhance UI of search activity feat
#86: Set tool bar title for different activities feat
#85: Status bar color fix
#81: Enhance user profile page feat
#80: Add translation and padding for augmentation feat
#79: Add option to share the app easily feat
#76: Clicking on profile opens profile page feat
#75: Keyboard hidden when nav drawer is opened fix
#72: Improve UI of search activity feat
#71: Added internet connectivity check in the login screen feat
#70: Added check for internet connectivity in search button feat
#66: Keyboard goes down on pressing search if query is suitable fix
#65: Add histogram equalization techiques for augmentation feat
#64: Deleted activityuserprofile2 chore
#31: Add bluring techniques for image augmentation feat
#28: Add cropping technique for image augmentation feat
#26: Fix typo in password length toast fix
#24: Make username a single-line text field, press next on keyboard to directly enter password field fix
#22: Add toggle for password visibility feat

Issues

#195: Display a dialog to user when logging out feat
#193: Change profile icon in bottom nav bar fix
#191: Color of options in action bar fix
#189: Improve code quality chore
#184: Add an action bar in computer vision activity fix
#180: Update README.md chore
#178: Add issue and PR templates to the project chore
#176: Add a contributions.md to the project chore
#174: Add a code of conduct for the project chore
#170: Improve UI of computer vision activity feat
#169: Enhance UI of the question fragment feat
#167: Revamp UI of user and client profile fragments feat
#164: Enhance UI of home page feat
#163: Shift the app to bottom navigation layout feat
#152: Remove dependency on google services chore
#150: Enhance the object detection system feat
#147: Auto-fill survey feat
#146: Add MLKit for object detection feat
#144: Toolbar in NewPPISurveyActivity not visible fix
#136: Add an example notebook for image classification feat
#134: Remove unused resources from the app chore
#132: Add an option for logging out feat
#130: Refactor the login of the application feat
#124: Add hindi translation feat
#116: Setup CI for the python script feat
#114: Modify the image augmentation notebook feat
#112: Create a python script for doing image augmentation feat
#109: Fix lint issues in the application chore
#108: Add GitHub Actions workflow for CI feat
#106: Replace the current khttp dependency feat
#104: Fix the method of obtaining the authKey fix
#102: Update app dependencies chore
#100: Migrate app dependencies to androidX chore
#84: Improve UI of the search page feat
#83: Set title for activities feat
#82: Set a color for the app status bar fix
#69: Improve search option edit text feat
#63: Add an option to share the app feat
#54: Add a notification of no internet if internet connection is missing when user presses search feat
#53: Keyboard should go down when the search button is pressed fix
#52: Codestyle fixing chore
#51: About Screen for the app feat
#50: Fix naming convention for variables chore
#49: Make variables private chore
#47: No check for internet connection in login screen feat
#45: Keyboard does not go down when the nav drawer is opened fix
#44: Clicking on the users photo in nav drawer should also open the profile page feat
#43: Add translation and padding augmentation techniques feat
#38: Enhance UI of user profile page feat
#36: Enhance UI of client profile page feat
#35: Delete unused activityuserprofile2.xml chore
#34: Add histogram equalization techniques for image augmentation feat
#30: Apply blurring techniques for augmentation feat
#27: Enhance the cropping technique for data augmentation feat
#25: Typo in toast message fix
#23: Multi-line username field fix

Tags:

feature : feat
bug fix : fix
chore related to maintainance, documentation, etc - chore

Updates

Community Bonding (May 4, 2020 - June 1, 2020)

Meeting 1: Notes
Blog Post: GSoC’20: Período de Unión Comunitaria

Coding Period 1 (June 1, 2020 - July 3, 2020)

Meeting 2: Notes
Meeting 3: Notes
Meeting 4: Notes
Meeting 5: Notes
Meeting 6: Notes
Meeting 7: Notes
Blog Post: GSoC’20: Período de codificación uno

Coding Period 2 (July 3, 2020 to July 27, 2020)

Meeting 8: Notes
Meeting 9: Notes
Blog Post: GSoC’20: Período de codificación dos

Statistics

Total PRs sent: 69
Total issues opened: 57
Total commits: 78
Total lines of code added: 15,269
Total lines of code deleted: 6,775

Links