Turning chat screenshots into courier orders with Gemini and .NET

Online shops in Kosovo and Albania mostly take orders in chat. Somebody then has to type every order into a courier’s website by hand: name, phone, city, address, product. That takes one to two minutes per order, and a wrong digit in a phone number becomes a failed delivery. Smart Extract replaces the typing with a screenshot. This article is about how that pipeline works and what it took to get it to 99.9% of screenshots processed successfully.

Why screenshots, not text

Sellers already have the conversation open on their phone. Asking them to copy and paste text from Instagram, Facebook or WhatsApp adds a step and loses context, like which message is the customer’s and which is the shop’s. A screenshot is one tap, and a multimodal model can read the layout as well as the words.

I started with OCR using Tesseract, an open-source library. It was slow, and the raw text it produced needed a lot of code to parse reliably into a name, phone number and address. When I tried an AI model instead, it was much faster and far simpler to implement: the model returns the order directly, with no hand-written parsing rules.

The pipeline

1. The seller uploads one screenshot or a batch, from the web app or the Android app.
2. The .NET API sends the image to Google Gemini and asks for a structured order, not free text.
3. The result is validated and normalized: phone numbers, cities, required fields.
4. The extracted order is sent back to the seller to review, then registered with the chosen courier through its API: Postman, Delex, Cheetah, Express Posta.
5. Nothing is stored. The image and the extracted data live only in memory and are discarded once the response is sent.

On a typical upload, the whole round trip takes 5-10 seconds. When a seller uploads a batch, each screenshot goes to Gemini in its own call, in parallel, instead of waiting in line one after another.

Ask for a schema, not a summary

The most important decision is to make the model fill a fixed shape instead of describing what it sees. The order has the same fields every time, so the request defines them up front and the response is parsed straight into a record. In simplified form:

public sealed record ExtractedOrder(
    string? CustomerName,
    string? Phone,
    string? Country,
    string? City,
    string? Address,
    string? OrderDetails);

Every field is nullable on purpose. A screenshot that doesn’t contain an address should produce null, not a plausible-looking address the model made up. A missing field is something the app can ask the seller about; an invented one goes to the courier.

The prompt does the heavy lifting: it tells the model what language the conversation is in, where to look for the customer’s details, and what to extract.

Validate everything the model returns

A model’s output is user input and gets treated like it. In a Clean Architecture setup with CQRS, extraction is one command, and its handler doesn’t trust the response until it passes the same validation an order typed by hand would.

Phone numbers are the clearest example. Customers write them in every format: with or without +383 or +355, with a leading 0, with spaces and dashes. Normalizing them to one format before they reach the courier fixes a whole class of delivery failures:

// Simplified: strip formatting, then add the country code if it's missing
static string NormalizePhone(string raw, string countryCode)
{
    var digits = new string(raw.Where(char.IsDigit).ToArray());
    if (digits.StartsWith("00")) digits = digits[2..];
    if (digits.StartsWith(countryCode)) return "+" + digits;
    if (digits.StartsWith('0')) digits = digits[1..];
    return "+" + countryCode + digits;
}

If a city can’t be recognized, the system doesn’t guess: the seller picks it in the review form before the order goes to the courier.

Don’t keep what you don’t need

Customer conversations are personal data. Screenshots and the data extracted from them are processed in memory and never stored; they exist only long enough to reach the seller for review and to be sent to the courier. That makes the privacy question easy to answer for sellers, and there’s no pile of chat images to protect, back up or delete later.

Keep the running costs small

Smart Extract is priced for small shops, so the infrastructure has to be cheap too. The API is a .NET 10 service in Docker on Google Cloud Run, on PostgreSQL 18 and Redis, deployed with GitHub Actions. Work that doesn’t need to happen during a request, such as subscription renewals, invoices in Albanian and English, analytics and emails, runs on a separate .NET worker on a small dedicated server instead of always-on cloud instances.

Results

Between April and September 2026, Smart Extract processed 35,797 screenshots, and 99.9% were processed successfully. Registering an order went from one to two minutes of typing to 5-10 seconds, and monthly volume more than tripled from April to August.

The remaining 0.1% most likely failed during periods when the AI models were overloaded, rather than because of the screenshots themselves. That’s part of building on an external model: it’s a dependency you don’t control.

The same approach works well beyond courier orders: invoices, delivery notes, forms, photos of paperwork. Anywhere people copy data from an image into a system by hand, see AI automation.