From photo to nutritional information: creating an app based on Artificial Intelligence

Logging meals can be tedious: searching for foods, comparing brands and adjusting quantities. FoodCheck began as an experiment in taking a different approach: start with a photo and produce an editable nutritional estimate. The app does not require an account, but image analysis does use external services.

That is the idea behind FoodCheck, an app with a static frontend on GitHub Pages that identifies possible foods in a photo and calculates nutritional estimates using data from OpenFoodFacts. Image analysis runs in a serverless backend: a Cloudflare Worker connected to Workers AI, where I used the LLaVA model.

The idea behind FoodCheck

The user opens the app and drags in a food photo. The prototype also includes a health-condition selector, but it does not provide personalised advice or validate diets for medical conditions. After analysing the photo, it suggests foods and quantities that the user can correct before saving the meal.

Meal records are stored in the browser. To analyse a photo, the app sends it to a Cloudflare Worker and Workers AI; nutrition queries also pass through the Worker to reach OpenFoodFacts. Local storage therefore does not mean that all processing happens locally.

Tools

• Visual Studio Code
• HTML, Tailwind CSS and JavaScript
• Google Fonts (Montserrat)
• Open Food Facts (public product-data API)
• Cloudflare Workers AI (model @cf/llava-hf/llava-1.5-7b-hf)
• Deployment: GitHub Pages (frontend) and Cloudflare Workers (proxy and AI)

A minimalist frontend

When we want to prototype an idea quickly, less is often more. In the case of FoodCheck, the frontend can be as simple as a single index.html file. You don't need heavy frameworks or complex configurations to start experimenting.

To achieve a simple but functional interface that allows the user to upload an image and obtain visual analysis results, we use:

• Tailwind CSS for quick styling.
• Montserrat, an elegant font that enhances the overall aesthetic.
• A small script that manages the flow of interaction with the backend (in this case, a Cloudflare Worker).

Here is an initial index.html structure for the prototype. The snippets in this article illustrate parts of the app and rely on other variables, functions and configuration; they are not a complete application that can be copied and run as-is. They reflect the implementation used in this project.

<!DOCTYPE html>
<html lang="es">
<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <title>FoodCheck</title>
  <link rel="preconnect" href="https://fonts.googleapis.com" />
  <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
  <link href="https://fonts.googleapis.com/css2?family=Montserrat:wght@400;500;600;700&display=swap" rel="stylesheet" />
  <script src="https://cdn.tailwindcss.com?plugins=forms,typography,aspect-ratio"></script>
  <script type="application/json" id="foodcheck-config">
    { "visionEndpoint": "https://TU-WORKER-NAME.tu-cuenta.workers.dev/" }
  </script>
  <style>
    body {
      background: linear-gradient(160deg, #fff6f0 0%, #fffdf8 35%, #ffffff 100%);
      color: #1f2937;
      font-family: "Montserrat", system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
    }
    /* Other styles go here. */
  </style>
</head>
<body class="bg-gray-50 flex flex-col items-center p-6">
  <h1 class="text-2xl font-semibold mb-4">FoodCheck</h1>
  <!-- Other interface elements go here. -->
</body>
</html>

What happens after uploading a photo

When the user drags a photo into FoodCheck, processing follows the steps in this diagram:

1. Food detection

The frontend obtains the Worker URL from the JSON configuration block in the head section of index.html. The Worker connects the browser to the image-analysis service.

When the user adds an image, we convert it to Data URL with FileReader.readAsDataURL (format data:image/…;base64,…):

function processFile(file) {
  const reader = new FileReader();
  reader.onload = (e) => {
    const src = e.target?.result;
    if (!src) return;
    state.image = { src, name: file.name, lastModified: file.lastModified };
    loadImagePreview(src);
  };
  reader.readAsDataURL(file);
}

We send that string to the Worker in a POST request with a JSON body. The Worker prepares the model instructions and requests a structured list of detections in a format like this:

{
  "detections": [
    {
      "name": "manzana roja",
      "confidence": 0.92,
      "portion_estimate_grams": 120
    }
  ]
}

The prompt requests a structured response but cannot guarantee one. The frontend must validate the JSON and handle incomplete or incorrect responses before using it.

In turn, the call to the model is direct:

const result = await env.AI.run('@cf/llava-hf/llava-1.5-7b-hf', {
  image: Array.from(imageBytes),
  prompt: systemInstructions,
  max_tokens: 1024
});

LLaVA returns text in result.description. After extracting a possible JSON block from that text, we try to parse it and check for a detections array. This snippet illustrates the parsing step:

function parseDetections(jsonString) {
  try {
    const cleaned = jsonString.replace(/\\_/g, '_');
    const json = JSON.parse(cleaned);
    return Array.isArray(json?.detections) ? json : { detections: [] };
  } catch {
    return { detections: [] };
  }
}

The frontend receives an object containing a detections array. An empty array means that no usable detections were obtained; it does not confirm that the image contains no food.

2. Portion estimation

Before using the results, we check the names and constrain the confidence score to the range 0–1. That score is not a validated probability of correctness. If no portion estimate is available, the prototype uses 120 grams as an editable starting value:

const portionAmount = detection.portion_estimate_grams ?? 120;

The user can edit the quantity and unit. The prototype includes fixed conversions, such as 80 g per item, 150 g per cup and 1 g per ml. These are illustrative approximations only: actual weight depends on the food, its density and portion size. Entering a measured weight in grams gives a better basis for the calculation.

const UNIT_CONVERSIONS = { g: 1, ml: 1, pieza: 80, taza: 150 };

function convertToGrams(amount, unit) {
  const factor = UNIT_CONVERSIONS[(unit || "g").toLowerCase()] ?? 1;
  return (Number(amount) || 0) * factor;
}

3. From names to nutrients

We search each name in Open Food Facts and keep results containing a nutrient-data object. This does not guarantee complete data or a match with the photographed meal, especially for homemade dishes. The user needs to check the match, units and available data. Queries pass through our Worker, which centralises communication and CORS handling.

To query the Worker, we build a URL with parameters such as the search text and result limit. We use the address defined in the foodcheck-config block in the HTML head, on the same domain as the image-analysis Worker:

async function queryOpenFoodFacts(query, limit = 50, options = {}) {
  if (!query || !query.trim()) return [];
  const params = new URLSearchParams({
    search_terms: query,
    search_simple: "1",
    json: "1",
    page_size: String(limit),
    lc: "es"
  });
  const externalSignal = options.signal;
  const controller = externalSignal ? null : new AbortController();
  const signal = externalSignal ?? controller.signal;
  const timeout = externalSignal ? null : setTimeout(() => controller.abort(), 7000);
  try {
    const cfg = JSON.parse(document.getElementById("foodcheck-config").textContent);
    const endpointBase = cfg.visionEndpoint.replace(/\/$/, "");
    const proxyUrl = endpointBase + "/api/nutrition?" + params.toString();
    const response = await fetch(proxyUrl, { signal });
    if (!response.ok) throw new Error("Nutrition proxy error: " + response.status);
    const data = await response.json();
    if (!Array.isArray(data.products)) return [];
    return data.products.filter((product) => product?.nutriments);
  } finally {
    if (!externalSignal && timeout) clearTimeout(timeout);
  }
}

The Worker thus receives the search parameters and makes the server-to-server request to OpenFoodFacts:

async function handleNutritionRequest(request, env) {
  const params = new URL(request.url).searchParams;
  try {
    const offUrl = "https://world.openfoodfacts.org/cgi/search.pl?" + params.toString();
    const response = await fetch(offUrl, {
      headers: { "User-Agent": "FoodCheckApp-Proxy/1.0" }
    });
    if (!response.ok) throw new Error("Nutrition service error: " + response.status);
    const data = await response.json();
    return new Response(JSON.stringify(data), {
      status: 200,
      headers: { ...corsHeaders(), "Content-Type": "application/json" }
    });
  } catch (err) {
    return new Response(JSON.stringify({ error: "Could not query Open Food Facts" }), {
      status: 502,
      headers: { ...corsHeaders(), "Content-Type": "application/json" }
    });
  }
}

For a product whose data is expressed per 100 grams, we scale each nutrient to the estimated portion using this formula. Units must be checked first: some products report values per 100 ml.

nutriente_por_porción = (nutriente_por_100g * gramos_porción) / 100

4. Displaying results and saving them locally

The interface displays the estimates in a macronutrient chart and table, alongside an editable list of foods and quantities. I also experimented with messages generated by simple rules. The example thresholds are prototype choices, not clinical recommendations or universal limits for a meal:

// Prototype thresholds: fibre in grams, sodium in milligrams.
// These rules are illustrative, not clinical guidance.
if (totals.fiber < 12) suggestions.push("Revisa la estimación de fibra de esta comida.");
if (totals.sodium > 800) suggestions.push("Revisa la estimación de sodio de esta comida.");

The user can save each meal. The daily log adds up the estimates for saved meals using localStorage. The data stays in that browser until its storage is cleared; it does not automatically sync across devices.

function persistState() {
  localStorage.setItem("foodcheck:preferences", JSON.stringify({ conditions: state.userConditions }));
  localStorage.setItem("foodcheck:dailyLog", JSON.stringify(state.dailyLog));
}

function saveCurrentMealToHistory() {
  const totals = calculateTotalsFromItems(state.foodItems);
  const entry = { timestamp: Date.now(), items: structuredClone(state.foodItems), totals };
  const day = state.currentDate;
  const dayEntry = state.dailyLog[day] ?? { items: [], totals: null };
  dayEntry.items.push(entry);
  dayEntry.totals = dayEntry.items.reduce((acc, it) => sumTotals(acc, it.totals), emptyTotals());
  state.dailyLog[day] = dayEntry;
  persistState();
}

Deploying to GitHub Pages and Cloudflare

The deployment is twofold:

• Frontend: static repository on GitHub Pages.
• Worker: published on Cloudflare with a Workers AI binding available as env.AI. Its URL is added to the HTML’s JSON configuration block.

Real problems and how I solved them

At first, I tried to connect FoodCheck with OpenRouter to use free vision models. On paper it sounded good, but in practice it didn't detect food or returned empty responses. 

I worked through the issues as they arose:

• The model did not see the image, so I made it receive multimodal messages (text and image).
• CORS errors, so I created a Worker as proxy to manage it.
• Malformed JSON: I added defensive parsing to extract the expected content, catch errors and validate its structure before use.
• Large images or unsupported formats: I reduced the size and exported to JPEG using canvas. For HEIC files, the browser must be able to decode the image or use a prior conversion step; changing the extension is not enough.

Despite those adjustments, my OpenRouter integration still did not work. I suspected queues or timeouts, but did not confirm the cause from service logs. I could not confidently attribute it to the models being free to use.

I looked for an alternative that suited the prototype and tried Cloudflare Workers AI, since I was already using Workers. In my tests, the LLaVA model responded on the first attempt. Service availability, limits and costs should be checked when reproducing the project.

Codex played an important part in this process: I used it to generate functions, debug errors and refine prompts. It helped me make progress and learn, but also taught me a clear lesson: unless you explain the required architecture, AI may suggest a solution that does not fit.

I needed a static frontend for GitHub Pages and a separate Cloudflare backend. At one point Codex suggested a Node server, which GitHub Pages cannot run. Environment variables can be used during a build, but any value included in the frontend files is public: secrets must remain in the Worker.

I realised that I needed to define the architecture first and keep my prompts consistent with it. AI could speed up the work, while I remained responsible for reviewing its suggestions and testing the results.

Final reflection

FoodCheck is not intended to replace a dietitian. It shows how code and AI can support educational tools, while reminding us that we need to explain and review how those tools process data.

That is the opportunity I take from this project: curious developers can build useful learning prototypes, provided they explain the limits, check the results and take care of users’ data.