I got to know OpenCV as I investigated into computer vision solutions for a little startup called robodev GmbH (see here). There I developed possible ways for real-time object recognition and computation of angle and position of the recognized object. I fell in love with C++ and its possibilities to realize high-performance applications. Awhile I started digging further into Android development in Java. Due to my experience as a mobile (games) producer at Goodgame Studios, I’m driven by the possibilities and potential of mobile solutions, so plenty of awesome app-ideas get born in my brain. Now is the time I make my ideas come to life.
I wrote my first blog post about how I got into programming (see here) and how you can do that too, even when you’re not from the IT-field. In this post, we will play around with OpenCV in Android. Since I didn’t have a small use-case in mind for the vast feature-list of OpenCV I’ll keep that part simple.
For my first app idea, reading business cards could be a neat side-feature, and as I heard that OpenCV and Tesseract are a powerful team – process pictures with OpenCV and read texts with Tesseract OCR – I saw an excellent opportunity to put both into a thin app.
In this blog post, I will show you how to write that app all by yourself, step by step. You will learn to use C++-code, OpenCV and Tesseract OCR on Android.
What is OpenCV?
OpenCV is an open-source computer vision library. It has a mighty feature-set, starting with simple image processing over object-, face- and feature recognition, 3d-calibration, and visualization up to machine learning and more. There is a big community with scientists, professionals, and hobbyists maintaining a comprehensive documentation – helpful for everyone, including beginners (see here). I was more than happy realizing that it supports Android development, next to C++, C, Python, Scala, Java. Unfortunately, the inbuilt OCR is not known to be very mighty, therefore I decided to combine the strengths of OpenCV’s image processing with another library, called Tesseract (as a Marvel Fan I really like that name).
What is Tesseract OCR?
Tesseract is a free software, command line program for Linux, Windows, and next to the support of 100+ different languages, it was considered the most accurate open-source OCR available in 2006. In 2011 a brave man called Robert Theis released a fork version called “tess-two” of Tesseracts OCR Android tools, maybe not knowing that just 2 years later no one would work on Tesseract Android Tools anymore (the last commit was made 2013). Thanks to Robert Theis we can enjoy the opensource library up till today, updated, with new features and easy to use. So let’s get to work and get the setup of your Android Studio done:
Setup Android NDK
To be able to use OpenCV in its pure form we first should setup Android NDK properly, and yes… we’re gonna write C++-Code in our App 😉
After creating a new Empty Activity Project use the Android Studio SDK Manager (
Tools -> SDK Manager -> SDK Tools) for installing the necessary Tools: NDK, CMake, and LLDB. Mind the location of your SDK though – you’ll need that for the next step. After applying your changes a long download of around 700MB will start – don’t worry, that is as intended.
Next time you wake up from sweet dreams maybe the download is over and you can keep working. Now you need to add the location of your Android NDK. Write it into
Files -> Project Structure -> SDK Location -> Android NDK location. Yes, you just saw the needed path: it is the SDK location +
There you go. Your first step to glory is over. The next stuff is going to be a bit more tricky. OpenCV – here we come!
To setup OpenCV, the first step is to download the latest release of the library from their page: OpenCV Releases (at this point it’s 3.4.2). Unpack the zip, preferably into the Android SDK folder (so you have it all in one place) and import that stuff, as a module, into your project:
File -> New -> Import Module. Navigate in the next dialogue that pops up to the java-folder of the just unpacked module, i.e. mine is
/home/soeren/Downloads/OpenCV-android-sdk/sdk/java (on Linux). If you’re correct the name of the library will show up and it looks like the following (ignore “Project already….” in that pic):
Hit “Next” and “OK” as long as they ask you stuff, keep going with the default checkmarks until you finally finish. Now, add the module as a dependency to your app. At
File -> Project Structure -> app -> Dependencies -> + -> Module dependency you will find OpenCV shown in a pop-up.
Next step is to correct the
build.gradle-file of the OpenCV Library (you will find it right under your app
build.gradle-file). Align the fields
targetSdkVersion, so they are matching with the
build.gradle-file (app). Mine looks like this:
build.gradle (openCVLib…) for initial setup
Now you create a directory in your project src/main/jni. In there you create a file called
Android.mk and another called
Application.mk – to tell the NDK tool to build native code.
Write the following code into the
Android.mk for NDK and OpenCV setup
Write your Android NDK path into
NDK_MODULE_PATH and replace the
OPENCV_ROOT according to yours. Later we will add the
LOCAL_SRC_FILES name, but for now, we can leave it empty. Then modify the
Application.mk for NDK setup
Now you can tell your
build.gradle-file (app) to compile C++-code to
.so-files into the jniLibs folder. Do that by writing this…
build.gradle (app) File for OpenCV setup
build.gradle (app) file, by overwriting the default buildTypes-part. Remember to change the ndkDir value to your own. Don’t hesitate to process the gradle-sync, but if you try to build your app now you will get a wonderful error. Something like
Process 'command 'PATHTONDK/ndk-build'' finished with non-zero exit value 2". This cryptic error messages did cost me quite some patience, but don’t worry. This is still as expected because we didn’t give in any
LOCAL_SRC_FILES. That’s what we’re going to do now. For the sake of convenience, we’re going to use javah as an external tool. For doing so, navigate to
File -> Settings -> Tools -> External Tools -> +. Add the javah tool like this:
Fill the parameter the following
* “Program”: the path to your javah installation (with Linux you can find out the path typing
which javah into a terminal)
-v -jni -d $ModuleFileDir$/src/main/jni $FileClass$
* “Working directory”:
After successfully setting javah up, write a little test function into your
public static native int test();
Use the javah-tool via
right-click on test()-> Android Tools -> javah and you generate a corresponding header file and can find it in your jni-folder. Copy the name of that file into your
LOCAL_SRC_FILES, mine is called
com_example_android_MYAPPLICATION_MainActivity.h, yours will be something close to that, I guess.
Now, run and pray that you didn’t make any path- or copy-paste-mistake. If something is not working and you get a cryptic error-message – check all the paths again, and when that doesn’t help, start over again – been there… done that… can happen to everyone! Just don’t give up. 🙂
If it builds successfully and a “Hello World!” is smiling from your Device-Screen up to you: YEAH!!! You’ve done great so far. The actual fun starts now!
Setup the Camera
To use the camera we’re going to implement some code. Initially we implement our MainActivity and its XML-structure. This is going to be a small button on an empty screen, nothing more. But there is more to come! I prepared a gist for you, where you can simply copy/paste the code into the corresponding files: MainActivity for initial setup (don’t forget to change the package name)
Our next step is to create a new empty Activity for working with the camera, let’s call it
CameraActivity. Here we will call the camera and have fun with OpenCV and tess-two later, but for now, we will only call the camera. Copy the code from the following gist into the corresponding files: CameraActivity initial setup.
As you can see, we really should take care of state changes, due to the fact that the camera is pretty demanding to run constantly. New to you will be the
onCameraFrame-methods. Here you can directly interact with the camera input.
MainActivity we will call the
CameraActivity, by replacing the
// do stuff here with the following code:
Intent intent = new Intent(MainActivity.this, CameraActivity.class); startActivity(intent);
Well done! You should be able to run the app now. And if you press the button… wait!
What is this? Ahhh!! We forgot the permissions, can it be it’s the permissions?
Get the permissions for OpenCV
So for getting the permissions, we will check and if necessary ask for them in the
MainActivity, when the user presses the button. Modify the
MainActivity with all the uncommented code at their proper position (as shown with the comments): Permissions in MainActivity
The last step to get the permissions is to touch the
AndroidManifest.xml: Manifest for camera permissions
Okay, let’s see if this was the mistake and run the app…Asking for permissions seems to be working but…
Oh come on! Nooooo!! Press “No”! We don’t need that ominous “OpenCV Manager”! We already have the whole library inside our app.
Ok ok…calm down. We can easily correct the rude OpenCV behavior:
So one of the most useless features of OpenCV is, that it checks if the device it’s running on has the OpenCV library installed, and if not, it won’t work until the OpenCV Manager App is installed. Since we put the whole library into the actual app, we don’t need that anymore, and yet, due to the fact that the library isn’t loaded completely when it checks it causes this message. With the following gist, we override the
onPackageInstall-method of the
BaseLoaderCallback class to prevent the app to trigger the installation. Add the following uncommented code into your
CameraActivity right into the
BaseLoaderCallback-class override the onPackageInstall-method
Now run the app again. If you didn’t mix up some code it should finally look like this:
Good job! But no time to celebrate: Somehow the camera orientation and size makes you feel a bit dizzy, right? We are getting our hands dirty and change something right inside the OpenCV library to fix this one. Find the class
Look for the method
deliverAndDrawFrame and replace it with the following code: Adjusting rotation and scale of the camera
Yeah! How wonderfully straight it looks like this, right? It somehow feels like the first time we see stuff through our camera. If your camera view seems a bit too small or big, you can simply change the mScale value accordingly. If you made it till here, congratulations! Figuring out the whole procedure till here cost me around 2 days of my life! I’m glad to make sacrifices for you, though 😉 The next part is going to be much easier!
Use the OpenCV library for image processing
Let’s start putting some functionality into our app and add some more buttons to our
activity_camera.xml and corresponding listeners to our
Adding function buttons, listeners and call OpenCvMaker methods
Don’t mind the errors yet, we will get to those now. Create a Java class with the name
OpenCvMaker.java and add the following code (and again, remember to change the package name):
Create an OpenCV manager class
Now delete the previously generated header file, from as you set up the
native test() method with the javah tool.
Navigate to your
OpenCvMaker class, use the javah tool on one of the native methods (i.e.
right click -> Android Tools -> javah. A new header file will get generated. Go to this file and make the following changes:
Add methods to the header
Then, inside of the jni-folder, create a
.cpp-file (without header) and call it the same as your header and add the following code to it:
Implementation of OpenCV methods in C++
And last but not least, copy the name of this
.cpp-file (including the “
.cpp“!!) and set it as your
LOCAL_SRC_FILES inside your
Run your app and…tadaaa! Now you can play around with four of the most common functions of OpenCV. Well done!
You just learned how to use C++-code AND OpenCV functionality inside your Android app. If this is all you wanted to learn, give yourself a break and lean back. Give your beloved ones a hug and tell them, you did it! Also, give yourself a break when this took you too long and your computer is smoking from all the building processes. In times like these, during the hot summer, I prefer to put my laptop into the fridge to give it a little chill 🙂
Now that we had a wonderful bath in a jacuzzi full of self-esteem, we can go back to work. There is one little part missing: The OCR! Don’t worry, I will make it quick.
First, add the following dependency to your
build.gradle (app)-file and trigger the gradle-sync:
Tess-two is now available and ready to use. Add the HUD components to
activity_camera.xml and some functionality to
CameraActivity.java with the following modifications:
Read-button and textview for CameraActivity
Create a new Java class with the name
MyTessOCR.java and put the following code into it (don’t forget to change the package!):
Implementing the tess-two class
Let’s use it in the
CameraActivity, and make some changes in the
CameraActivity. Since computation of the OCR could be process-demanding, we will call it in an extra thread. Also, you will be asked to import the
android.graphics.Matrix, continue and confirm. This is only for rotating the input signal of the camera according to the preview, that we’ve rotated inside the OpenCV library already. Do the following: Call MyTessOCR class from CameraActivity
Tesseract, and therefore tess-two, needs traineddata to know how and what to recognize. Since we are using tess-two, which is a fork of an older Tesseract version, we cannot just use any traineddata that is available in the tesseract git repository. Download the following traineddata and you will be fine: Download eng.traineddata. Create an asset-folder via
right-click on Java -> New -> Folder -> Asset Folder, then create a directory called
tessdata and finally copy the downloaded
eng.traineddata-file into that directory. When you’re done it should like this:
Next, you need to make sure that this file is also put onto the device. Therefore we need to add a little method to our
CameraActivity and call it in the onCreate method: Add preprateTessData-method to CameraActivity
Alright! The last thing to do is to get the permissions for read and write access on the external storage since we are doing that with our previously added method. Go and add the following checks and constants to the
MainActivity and additional permissions to the
Ask for external storage read and write permissions
Okay! Breath in. Breath out. You did it! Build (can take a while), run and play 🙂 Hold your phone in front of the text below and press
READ IT! in your app.
Find my original Git repo of the app here