The company Nanonets' blog on deep learning for OCR, document processing, object tracking, and structured LLM outputs.
HN mostly sees it in commenters mentioning Nanonets as a tool or as a company, with little opinion on the posts themselves.
The challenge is doing this offline/on-device. There are a billion receipt scanner APIs and services in the cloud that are basically "upload image, get back structured receipt data", but that's not what you want?
Some links...
- A 2019 machine vision competition about this exact use case: https://rrc.cvc.uab.es/?ch=13. Some of the submissions in the results table will discuss their methods in depth (usually in Python though).
- Implementing this without AI/ML, just using Tesseract (open-source OCR, but poor performance compared to commercial apps or ML): https://pyimagesearch.com/2021/10/27/automatically-ocring-receipts-and-scans/
- Using Google's ML-kit (on-device Document Scanner SDK): https://teresa-wu.medium.com/googles-ml-kit-text-recognition-with-sample-app-of-receipts-reading-7fe6dc68ada3, but the tabular data is hard to work with. This writeup goes one step further: https://hackernoon.com/using-google-mlkit-text-recognition-to-scan-tabular-data
- 2019 HN discussion on the same: https://news.ycombinator.com/item?id=10338199
- A cloud OCR vendor (Nanonets) discussing their ML-based pipeline: https://www.linkedin.com/pulse/how-build-ocr-receipt-scanner-from-scratch-using-deep-olivia-smith
----------
(Edit: Cloud options)
Last time I looked into this (for a similar use case) I didn't get very far, and decided that using the cloudy Nanonets receipts API (https://nanonets.com/ocr-api/receipt-ocr) or the Google Cloud document classifier…
Picked a few articles from results of search engine terms, "app to translate using camera" and "finding blocks of text in an image using python"
Google Lens mobile translate app [1] via smartphone.
Article walking through how Google Lens does the translation.[2]
Article about the text box boundary(s) [3]
[1] : https://blog.google/products/translate/google-translates-instant-camera-translation-gets-upgrade/
[2] : https://medium.com/technovators/scene-text-detection-in-python-with-east-and-craft-cbe03dda35d5
[3] : https://nanonets.com/blog/deep-learning-ocr/
Maybe this is helpful: https://nanonets.com/blog/receipt-ocr/
In my Opinion Tesseract is the most sophisticated "free" OCR solution out there. The problem with Tesseract is not its recognition capabilities, but more the preprocessing steps.
- thresholding
- deskewing
- segmentation
- ...
There is a C# library (non-free), that improves recognition A LOT, just by providing these abilities: https://www.vintasoft.com/vsocr-dotnet-index.html
If you find a good Open Source solution, I would be interested, too...
I am Prathamesh, co-founder of https://nanonets.com
We were working on a project to detect objects using deep learning with raspberry pi and we have benchmarked various deep learning architectures on pi. With ~100-200 images, you can create a detector of your own with this method.
In this post, we have detected vehicles in Indian traffic using pi and also added github links to code to train the model on your own dataset and then script to get inference on pi. Hope this helps!
I used a video's frames of myself playing RDR2 to train a neural network. I then showed it a photo my son took of some cows at the in-law's place. Here's what happened: https://photos.app.goo.gl/sDTofashQdDSUJMdA
The network was trained with https://nanonets.com, if anyone is interested. Interesting stuff, given the knowledge came from synthesized horses.
I did not train or segment any other entities other than Arthur and his horse. Labels were 'horse' and 'human'.
A routine automation task is information extraction from any kind of document with a visual layout - receipts, PDFs, web pages. [1] [2]
[1] : https://nanonets.com/blog/information-extraction-graph-convolutional-networks/
[2] : https://arxiv.org/pdf/1903.11279.pdf