AgroClaw: What we learned when building an AgTech agent

In applied artificial intelligence, it is easy to stay at the flashy layer: the demo that responds, the interface that impresses or the “chatbot” that seems to know everything. The hard part starts later: when you're trying to build an agent that not only sounds good, but is useful in a real technical domain. That is precisely what AgroClaw has taught us.

AgroClaw was born as an AgTech assistant in Spanish, with initial focus on the olive grove. Its objective is not to be a generic agricultural chatbot, but to help with specific tasks: indicative diagnosis of pests and diseases, use of dichotomous keys, interpretation of field symptoms, irrigation recommendations and even contextual reading of data from AIoT sensors. To do this, it was built on OpenClaw and organized with a layered architecture: identity, character, global behavior, skills and documentary base.

In the repository https://github.com/SaturdaysAI/agroclaw-demo You can see the project directory structure:

agroclaw-demo/
├── public/
├── server/
│ ├── server.js
│ └── workspace-template/
│ ├── AGENTS.md
│ ├── BOOTSTRAP.md
│ ├── HEARTBEAT.md
│ ├── IDENTITY.md
│ ├── SOUL.md
│ ├── TOOLS.md
│ ├── USER.md
│ ├── skills/
│ └── knowledge/
│ └── olivar/
├── src/
│ ├── api/
│ ├── assets/
│ ├── components/
│ ├── data/
│ ├── types/
│ ├── utils/
│ ├── App.tsx
│ ├── index.css
│ └── main.tsx
├── .env.example
├── package.json
└── vite.config.ts

That architecture is not cosmetic. One of the most useful ideas that the process has left is that a well-constructed LLM agent resembles, in its organization, an object-oriented system. IDENTITY.md defines who the agent is; SOUL.md, how it is expressed; AGENTS.md, how it works; the skills They define what they know how to do; and the documentary base defines what you know about the domain.

Below and based on this approach, we present a proposed UML diagram for OpenClaw:

With proper configuration, AgroClaw turns out to be the instance produced by this architecture.

The beauty of this scheme is practical: it makes it easy to maintain the system, locate faults and evolve one part without breaking all the others. It is not a minor detail. It is what allows us to go from a prompt enormous and unmanageable to a system that can grow judiciously.

Now, here appears the important nuance: an LLM agent is designed as if it were OOP, but it does not behave like classic software. There is no compiler that mechanically guarantees that one layer takes precedence over another. The model reads identity, character, rules, skills, documentary base and user message at the same time, and produces a response trying to reconcile everything. That is why the hierarchy in an agent is not deterministic, but pedagogical: it must be declared in words, reinforced, and even then, it depends on the model's ability to respect it under pressure.

This difference was clearly seen in the use case that most served us to stress AgroClaw: the olive tree repilo. The system had to be able to identify the problem in an indicative way and, when the documentary base supported it, recommend an indicative curative treatment with documented active materials. On paper, everything was set up for that to happen: skills rewritten, reinforced rules in AGENTS.md, refined identity, structured knowledge and a very specific use case (repilo).

Olive repilo images used in the AgroClaw knowledge base.
Sources: Syngenta, Agricultural Technician and UPL Olivar Chair

However, the actual behavior remained elusive. AgroClaw diagnosed repilo well, but when it came to recommending treatment, it tended to take refuge in copper treatments (preventive, not curative), caution, and the classic “consult the registry of phytosanitary products.” This was important because it forced us to discard a very common explanation: “there is a lack of prompts” or “lack knowledge”. In reality, the system already had a sufficient documentary basis. The problem was not with content, but with the engine that interpreted it.

During several iterations, skills, AGENTS.md and document organization were touched upon. All of this improved the agent, yes, but it did not resolve the underlying failure. Only when the model was changed from GPT-5.4-mini to GPT-5.4 did the decisive difference appear: with the same architecture and the same document base, AgroClaw began to respond in a truly useful way.

The contrast was very clear. With GPT-5.4-mini, the agent tended to respond more fearfully and excessively cautious in phytosanitary recommendations. With GPT-5.4, maintaining exactly the same files and rules, AgroClaw was able to diagnose the repilo, distinguish between curative and preventive intervention, name the documented curative active substances and place the copper where it should be: as a preventive treatment on a healthy leaf, not as the only way out.

This finding is probably the most important technical conclusion of the entire process. The lesson here is uncomfortable, but valuable: not every agent's problem is solved by writing more instructions. There comes a point where you have to keep rewriting prompts It is going around in a circle. Sometimes the architecture is already correct and the bottleneck is the model.

In AgroClaw, that border became visible: the mini was sufficient for part of the work, but not to support the hierarchy of instructions well when there was tension between prudence, documentary knowledge and the need for an operational response. The large model (GPT-5.4) did.

There is also a second lesson, less spectacular but just as important. In this phase of the project, we have not worked with exact doses or with closed formulations, nor has it been intended to replace the consultation of authorized records or vademecums when entering the fine regulatory field. That doesn't weaken AgroClaw; On the contrary, it marks its professional boundary well.

The agent already demonstrates that it can provide a lot of value in probable identification, technical reasoning, risk prioritization, active guidance materials and distinction between preventive, curative or mixed approaches. The part on exact doses, current uses and label details continues to be, for now, an area where specialized consultation makes sense.

When images matter too

Another relevant line of learning appeared when thinking about the visual knowledge base. At first glance, it may seem that saving images within the knowledge It only makes sense if the agent “sees” them automatically. But that expectation oversimplifies the problem.

In practice, images have value even if they do not yet automatically enter the agent's memory as text does. They serve, first, to build an organized visual corpus of the domain: leaves with repilus, fruits with flies, buds with milkweed, branches with borers, symptoms compatible with verticillosis or references of damage in different phenological states. That already turns a folder of scattered photos into a reusable technical library.

But the important point is another: in AgroClaw, the real leap is not in saving just an image, but in working with the image + Markdown duo. That is, each image must be accompanied by a .md file that explains what it shows, what visual symptoms are key, what it can be confused with, what its indicative value is and where it is stored. This duo turns an isolated image into a documented visual reference.

Therein lies today the basis of the real usefulness of images in the knowledge base. Although the automatic recovery of images is not resolved in the GPT-5.4 model in the same way as that of Markdown documents, the image accompanied by its file can enter into the agent's reasoning. AgroClaw can read the description, understand the path or file name, use it as a visual reference, and, if prompted, return at least a list of relevant visual examples of the queried problem.

From a collection of images to a documented visual base

The most recent advance of the project has consisted precisely in turning that idea into a systematic practice. The visual base of the olive grove began to grow not as a folder of loose images, but as an organized set of references with name, route, source and associated technical sheet.

Added and documented visual references for problems such as milkweed, Saissetia or soot scale, otiorhynchus, olive tuberculosis, anthracnose, olive beetle Euzophera, Xylella, borer, verticillosis and prays. Each image was accompanied by a Markdown file with a stable structure: crop, affected organ, indicative diagnosis, visual description, diagnostic value, agronomic interpretation, limitations, usage note for AgroClaw and source.

This changed the nature of the repository. It was no longer just about “having photos” to consult manually. The images began to function as documented visual references: each one could be retrieved by its file, compared with others in the same folder and used by the agent as prudent support in a diagnostic interpretation.

A clear example was the olive milkweed folder. A small visual series was constructed with six references: cottony masses in buds and inflorescences, details of the insect, nymphs, adults, colonies and possible associated effects such as honeydew or black blight. Later, with AgroClaw already working, it was tested whether the agent was capable of recovering that visual base.

The response was especially significant. Not only did AgroClaw correctly enumerate all six references (milkweed_01 to milkweed_06), it retrieved their local paths, described what each one showed, and recognized that each image had its associated .md token. That is, the agent was not magically “seeing” the internal images; I was using the documentary layer that turns them into operational knowledge.

The next step was to ask something more demanding: order those references by practical diagnostic utility. AgroClaw classified the images into functional groups: bud and flowering symptoms, details of the insect, colonies or multiple presence, and possible cases with associated honeydew or sooty mold. In addition, he distinguished which were more useful for rapid diagnosis in the field and which were better for visually confirming the insect.

That test was important because it showed that the documented visual basis not only allows files to be recovered. It allows you to build criteria on them.

Visual indices: the layer that starts converting the knowledge in expert system

From that test a natural improvement appeared: creating visual indexes by folder. If each image has its individual card, a visual index can act as a diagnostic map of the whole. It does not replace the cards, but it helps to know which reference should be used depending on the type of image or symptom provided by the user.

In the case of milkweed, a first visual diagnostic index was generated, algodoncillo_visual_index.md, with a general description of the folder, reference table, diagnostic utility, main limit of each image and a usage note for AgroClaw. The idea is simple but powerful: each thematic folder can have its individual files and, in addition, an index that explains how to use them together.

That type of file converts the knowledge into something more like a maintainable technical tool. It does not force the agent to improvise the organization of the images every time. It offers you a middle layer: an internal guide to visual recovery, useful for comparing symptoms, avoiding confusion and maintaining diagnostic prudence.

There AgroClaw began to behave less like a collection of documents and more like a small structured technical memory.

LLM models, vision and performance modularity

An important point of the AgroClaw design is that it is not conceptually tied to a single LLM model. The architecture relies on a stable triad—IDENTITY.md, SOUL.md, and AGENTS.md—along with skills and knowledge base. This structure defines who the agent is, how it must behave, what limits it must respect and what documentation it must rely on.

From there, the model used acts as an interpretation and generation engine. This means that the final performance of AgroClaw does not depend only on the quality of the prompt or the documentary base, but also on the capabilities of the associated model at each moment: reasoning, following instructions, technical prudence, contextual recovery, vision, document management or multimodal analysis.

This distinction is key. The AgroClaw knowledge base can be organized with Markdown documents, technical sheets, images, routes, visual indexes and agronomic references. But the way in which all that material is used depends on the combination of architecture, LLM engine and available recovery capabilities. A model may be sufficient for general conversation, but fall short when it must support a complex hierarchy of technical instructions. Another model may better handle the tension between prudence, operational utility, and documentary knowledge. And others may provide more advanced capabilities in vision or multimodal treatment.

Therefore, the image + Markdown duo remains a solid decision. The isolated image functions as a technical file; the image accompanied by a Markdown token becomes operational knowledge. This sheet allows you to describe symptoms, affected organ, diagnostic value, possible confusion, limitations and source. With this, AgroClaw can reason about the image even when direct visual recovery is not yet the main axis of the system.

The advantage of this approach is that AgroClaw is prepared to evolve. By connecting to a model with better vision capabilities, you will be able to take better advantage of the images. By connecting to a model with better technical reasoning, you will be able to better interpret tokens and indices. By connecting to a more advanced multimodal retrieval system, you can integrate text and image more directly. Architecture is not broken: it is enhanced.

In other words, AgroClaw is not just “a responding model.” It is a visual-documentary technical memory assisted by AI, where the model can change, improve or specialize without abandoning the central structure of the agent. This modularity is one of its strengths: it allows the system to be adapted to the level of demand of the use case, the available cost, the acceptable latency and the necessary multimodal capabilities.

Why this matters for a real agricultural tool

This visual-documentary architecture is especially valuable in agriculture. Field knowledge does not always come in the form of tables or manuals. It also arrives in the form of photos taken by technicians, farmers or ranchers: a leaf with spots, a branch with sawdust, a deformed shoot, a chopped fruit, a plot with repeated symptoms after several days of humidity.

If each of these images is accompanied by a clear technical note, what is built is no longer just a collection of photos, but a living knowledge base. A base that can be used to compare, guide, train and transfer criteria between people and campaigns.

That's why the image + Markdown duo is not a temporary patch. It is a solid way to capture real agronomic knowledge with a structure that can grow well and also leave the system ready for a more advanced multimodal layer when the infrastructure allows it.

And here appears perhaps the most interesting implication of all. AgroClaw is not just a demo about olive groves. What has been validated is a working template. If an agent can be organized by layers, use skills well delimited and rely on technical domain knowledge, then its real value is not only in the model, but in the combination between model, architecture and field knowledge.

The potential of AgroClaw in an agricultural cooperative

AgroClaw opens up a powerful possibility for cooperatives, agricultural companies and organizations in the sector: converting the dispersed knowledge of technicians, farmers, ranchers and real farm observations into useful criteria.

In many agricultural contexts such as agricultural cooperatives, knowledge is not centralized. It is distributed among people (farmers who observe the field daily, technicians who interpret symptoms), campaigns, repeated incidents, local practices that work, field notes, real farm photographs, treatments applied, observed results, etc. that do not always become a reusable documentary base. An agent like AgroClaw can serve precisely that: not only to “answer questions”, but to act as an organization's technical operational memory. A memory that helps identify, prioritize, guide and transfer knowledge without losing the real context of the terrain.

In this way, with the advances made in the visual base of AgroClaw, this practical application became even more evident. In this sense, each relevant observation could become a structured entry: image, Markdown file, source, date, culture, affected organ, indicative value, possible confusion, diagnostic limit and, when appropriate, technical review.

This flow has a very clear advantage: it does not replace the agricultural technician. It gives you memory, order and speed. It allows you to recover similar cases, compare symptoms, train new people and reduce the loss of practical knowledge that may be dispersed between field visits, conversations, cell phone photographs and individual experience.

In fact, once SOUL.md, IDENTITY.md and AGENTS.md are aligned, the nature of the work changes. It is no longer so much about reinventing the agent, but rather about feeding it well. A team could be specifically dedicated to documenting new references, reviewing files, sorting folders, creating visual indexes, and transforming field observations into reusable knowledge.

This is perhaps the most promising conclusion from the current state of the project: AgroClaw is beginning to look less like an isolated technical test and more like an agricultural knowledge infrastructure. A small infrastructure, still initial, but with a clear direction: convert field experience into operational memory.

So the end result of this process is not only that AgroClaw now responds better about repilo or that it can list images of milkweed. It is something broader: that a specialized agent, well designed and fed with real knowledge, can begin to behave like a technically defensible tool. And that, for agriculture, is no small thing.

The last decision: a stable, recordable and playable local demo

The latest progress of the project has added an additional lesson: not every technical validation needs to start with a complex public demo. During the process an architecture was explored with frontend on Netlify, backend on Coolify, VPS, persistent container and OpenClaw running remotely. That line was useful because it allowed us to better understand the actual deployment requirements: HTTPS, domain, secure backend exposure, workspace persistence, timeouts, agent startup, container limits, and response latency.

But precisely for this reason the final decision of this phase has been more mature: to prioritize a stable, recordable and playable local demo. Before trying to make AgroClaw available from any browser, it made more sense to clearly demonstrate what it does, how it does it, and why its architecture provides value.

The current local architecture is simple to explain and much more controllable: a frontend Vite/React communicates with a backend Node/Express bridge; that backend invoke OpenClaw; and OpenClaw uses the workspace of AgroClaw, where the identity, the tone, the work rules, the skills and the documentary base of the olive grove. In that scheme, the workspace It is not a separate process, but rather the technical and operational memory that shapes the agent's behavior.

This simplification has a very important advantage for a demo: it eliminates noise. If something goes wrong in a remote public presentation, it can fail due to DNS, HTTPS, CORS, cold starts, timeouts, container persistence or provider latency. In the local demo, however, the focus is once again where it should be: on the quality of the agent, on the recovery of knowledge, on the technical prudence of its responses and on the real usefulness for an AgTech case.

From local prototype to reproducible repository

The other relevant change is that AgroClaw has stopped relying solely on a handcrafted setup on a local machine. The workspace template has been moved to the repository (https://github.com/SaturdaysAI/agroclaw-demo) as a versioned and sanitized tree, under server/workspace-template/. That folder contains the files that define the agent - IDENTITY.md, SOUL.md, AGENTS.md and other context documents -, the skills specific to the olive grove and the knowledge base with technical documents and documented images.

This point is more important than it seems. In many projects with LLM agents, the visible part is the frontend and the difficult part is hidden in a local folder that is difficult to reproduce. In AgroClaw, the advance has consisted of converting that folder into a portable template: without credentials, without tokens, without .env, without absolute personal paths, without private state and without logs. That is, a database that can be rebuilt in another installation without exposing sensitive information.

With this decision, the project gains traceability. We are no longer just looking at an agent that worked once in a specific session. We are looking at an architecture that can be rebuilt: the appropriate branch is cloned, the template is copied to the workspace real OpenClaw and starts the backend and the frontend local. This reproducibility changes the nature of the project, because it allows AgroClaw to be explained as a verifiable technical demo, not as a black box.

The local working branch has also been reflected in the Saturdays.AI repository. The demo-local-backend branch concentrates this phase of the project: backend local, frontend ready to consume, architecture documentation and template of the workspace. This allows the current state to be shared until public deployment occurs, which is not yet the immediate goal.

Design the demo with real latency in mind

Latency has been another practical learning. Local tests show response times of tens of seconds, especially when the agent has to activate OpenClaw, query knowledge, reason prudently, and generate a complete response. What is relevant is that the bottleneck does not seem to be primarily in Vite or Node/Express, but in the combination of OpenClaw, the model, the preparation of the context and the agent's own reasoning.

It also requires choosing well prompts. The demo doesn't need twenty questions. You need three, four or five strong cases, for example: in addition to the case of repilo already exposed, a recovery of visual references of milkweed demonstrating the usefulness of the image + Markdown duo, a cautious comparison between symptoms compatible with verticillosis and Xylella, etc.

The consequence is clear: for now, the demo must be measured by technical quality, traceability and clarity. In an area such as agriculture, responses to queries must be well defined and supported by documentary memory.

What AgroClaw proves now

In its current state, AgroClaw demonstrates several things at once. The first is that a specialized agent can be designed with an understandable architecture: identity, tone, rules, skills and knowledge. The second is that agricultural knowledge does not have to be reduced to plain text; It can be organized as a visual-documentary base where the images are accompanied by Markdown tabs and indexes per folder. The third is that the project can be reproducibly packaged using a template. workspace sanitized and versioned.

The fourth is perhaps the most important: AgroClaw is beginning to have a clear professional boundary. It is not intended to replace the agricultural technician, nor to issue closed recipes without context, nor to replace official records, labels or vademecums. Its value lies elsewhere: helping to retrieve knowledge, structuring observations, guiding hypotheses, explaining differences between problems, and serving as working memory for technical teams.

This fits very well with the type of product that Saturdays.AI can show as an example of applied AI. It is not about teaching “another chatbot”, but a way to capture expert knowledge and make it usable. AgroClaw shows that an agent can be a conversational interface, yes, but also a document architecture, a knowledge curation methodology, and a foundation for future field flows.

The local demo allows us to tell that story without overacting. You can teach the frontend, ask prepared questions, see how the person responds backend real and explain that OpenClaw does not work in the abstract, but rather based on a workspace local which is essential for the configuration of the agent. In that workspace they live the identity, the tone, the rules of behavior, the skills and the knowledge base that make OpenClaw a concrete instance: AgroClaw.

Therefore, having the workspace available locally is not a simple technical detail or a provisional solution to record the demo. It is a central condition for the agent to be able to behave like AgroClaw: recover its documentary memory, apply its rules, use its specific capabilities and rely on documented visual references. The demo shows that these references are not decoration, but structured knowledge within the system.

That narrative is strong because it's not just about showing a responsive interface, but about showing a complete architecture: frontend, backend bridge, OpenClaw and workspace AgroClaw working together in a reproducible way.

Next frontier: product, not just deployment

Public deployment will continue to be a natural phase of the project. Domain, HTTPS, persistence, security, response times and infrastructure limits will have to be resolved. But now that discussion may come at the right time: after proving that AgroClaw deserves to be deployed, not before.

The immediate priority has been to record a local demo that is stable and explainable. That recording should show the value of the agent, not the complexity of the infrastructure. Once the technical and functional story has been validated, it will make sense to think again about a public architecture, perhaps with a backend persistent, endpoints protected, observability, usage limits and a clear cost strategy.

The natural evolution of AgroClaw is not just about “putting it online”. It involves converting it into a product: defining use cases, selecting priority crops or problems, organizing knowledge, establishing technical reviews, deciding what answers it can give and which ones it should derive, and designing flows where farmers, technicians and cooperatives can provide real field observations.

Conclusion

AgroClaw started as an AgTech agent test on olive groves, but the project has ended up teaching something broader: how to build an AI-assisted technical memory. The combination of OpenClaw, a sufficiently capable model, a layered architecture, skills delimited, knowledge documentary and visual references with Markdown points to a very interesting avenue of work for agriculture.

The last leap has been to make the system more serious: more reproducible, safer, more explainable and easier to show. This is why the project has focused on closing a local demo that can be recorded well and that teaches the essentials: AgroClaw recovers knowledge, reasons about it and returns it in a useful form for a real technical context.

That is, perhaps, the best conclusion for this phase. AgroClaw is no longer just an experiment. It is a defensible technical and conceptual basis for exploring how AI can help preserve, organize and activate agricultural knowledge. And in an industry where so much valuable experience lives dispersed among people, campaigns, field visits, photos and notes, that memory can be as important as the agent's own response.

References

https://www.syngenta.es/blog/la-prevencion-del-repilo-comienza-por-el-tratamiento-en-primavera
https://www.tecnicoagricola.es/repilo-en-olivo/
https://uplcatedraolivar.com/repilo/