The promise and progress of intelligent automation

 

Alternating limbs from each cherry tree stretch out to the sky, forming a “Y” shape. The rows of trees look amazingly uniform on the Roza Farm near Prosser, creating tunnels with neatly trimmed smaller branches growing in an almost two-dimensional structure.

The synchronicity in the orchard makes it easy and efficient for people to prune the trees and harvest the cherries. It’s also much easier for robots to find and clip errant branches, pollinate flowers, and quickly pick the fruit.

Matthew Whiting (’01 PhD Hort.), a Washington State University horticulture professor based at the Irrigated Agriculture Research and Extension Center, says automating orchard tasks like pruning, pollinating, and picking becomes much more achievable with fruiting “walls” on sculpted cherry and apple trees.

Washington State University horticulturalist Matthew Whiting stands between two rows of cherry trees
Horticulturist Matthew Whiting among UFO cherry orchards (Photo Zach Mazur)

The machines to do the work come from a team of engineers and computer scientists who work with Whiting and experts from around the world to build the next generation of automation. The researchers at the WSU Center for Precision and Automated Agricultural Systems (CPAAS), led by director and automation engineer Qin Zhang, develop technology that can take over some of the tedious tasks in orchards.

Those robots, learning through artificial intelligence and seeing through advanced machine vision, test their chops in WSU’s Roza Farm orchards. Eventually they’ll graduate to the commercial world where they are sorely needed.

As the tree fruit and other ag industries face significant labor shortages that can hinder their businesses, automated solutions can provide another way to ensure we can still get apples, cherries, and the rest of the bounty of Washington state.

Mechanization and automation in agriculture is not a new concept; for row crops, it’s already here. “That’s not true for many specialty crops. In the state of Washington, our specialty crops depend on human labor,” Zhang says. “With availability of human labor becoming more and more a problem, the sustainability of the specialty crops is becoming a challenge.”

Zhang, precision agriculture engineer Lav Khot, and robotics engineer Manoj Karkee join a critical mass of computer scientists, crop and soil scientists, economists, and WSU Extension outreach specialists to build robots to work in orchards, gather and translate data to make farms more efficient, get those innovations to growers, and eventually construct smart control centers to run the operations.

As Khot says, they are building and testing a “farms of the future” concept.

 

Roza Farm’s experimental orchards are about a 10-minute drive through plots of hops, grapes, apples, and other crops to the WSU Prosser research station and CPAAS building. Down a hall past a wall of framed patents, a huge warehouse-like workshop provides room to work on projects.

It’s one of the best indoor spaces in the world for ag automation research, Zhang says. Researchers can quickly build and iterate field-scale prototypes with 3D printers, CNC machines, and other equipment. It’s one way that CPAAS developed an international reputation over the past 25 years, with partnerships around the world and thousands of visiting faculty.

Washington State University engineer Qin Zhang stands in doorway of large warehouse at precision and automated agriculture research center.
CPAAS director Qin Zhang oversees many projects in the Prosser-based workshop.
(Photo Zach Mazur)

The center’s goal is clear: address the pressing needs of farmers, like labor for tree fruit and other specialty crops. The deficit of farmworkers goes back decades, exacerbated by the pandemic, declining interest in hard farm labor, and complications with immigrant and guest worker programs.

We are not replacing humans, Zhang says. “We have farmers who don’t have people to help do the farming. If we don’t address this issue in a few years, we will have no apples to eat.”

With an aging population of farmers, Karkee adds, “technologies also have a role to play to widen the potential for people of all ages and all physical capabilities to contribute to farming.”

One solution to smooth the ups and downs of labor needs is a cost-effective, multipurpose robot with specialized attachments.

“It would not just do harvesting,” Karkee says, “but it could have a pruning hand, a pollination hand, a chemical application and a thinning hand.”

The boxy robot built by Karkee and the team looks a little like WALL-E from the animated movie, but without the big expressive eyes. That doesn’t mean it can’t see. In fact, advances in machine vision make much of the work possible. Combined with AI, such a robot could be tomorrow’s farmhand.

 

Consistency is a key to automating orchard work. Robots need algorithms to learn what to look for and what to do. If a system is irregular or complicated, it becomes very tricky.

Automation is already widespread among field crops such as wheat and other grains. GPS easily guides tractors along consistent rows for tilling, harvesting, and other tasks, with barely any human interaction. Fully automated tractors are just on the horizon.

Orchards and fruit trees, on the other hand, can be a little messier.

For example, cherry harvesting in a traditional system requires more time and labor than other tree fruit, with dozens of pickers climbing 10- to 12-foot ladders into the tall trees for harvesting and other tasks. Branches grow in every direction.

“Years ago, mechanical engineers and robotic scientists would come out and look at the cherry trees and say, ‘There’s not much we can do here. We could do automation, but it’ll be very complex and very expensive,’” Whiting says.

On the other hand, the upright fruiting offshoot (UFO) system that Whiting and his colleagues have used at the Roza experimental orchard relies on shorter, simpler cherry trees that increase harvest efficiency–as much as 50 times more efficient with a robot. Machines can also more easily learn how to prune, what to pick, and even where to pollinate.

“Whatever you’re doing in orchard structures, think about 20 years down the line, not about next year. What will be the labor situation? What technology will we have available?” Whiting says. “That begins with the structure of the trees.”

Although a lot more UFO orchards are planted around the world than in Washington state, data collection and successful demonstrations can make a difference. The CPAAS team demonstrated independent robotic pruning in the UFO cherry orchard for the first time this spring. With no previous knowledge, the robot imaged the branches, extended its arm, and effectively pruned trees.

Whiting points out that another benefit is to gather effective pruning knowledge. Even experienced pruners rely more on instinct than data. As machines learn the best way to achieve yield and desired growth, the pruning rules get even better.

 

Just a few hundred feet from the UFO cherry trees, plots of WA38 apple trees present other challenges for automation, such as pollination.

Pollination, Whiting says, is fraught with variability.

It seems simple. Hives are brought to orchards and pollinators carry pollen from flower to flower, using pollinizers⁠—trees of different genotypes that provide pollen.

There’s often an asynchrony with pollinators and blooms, and climate change has really perturbed the system by affecting bloom times. A cold snap can keep bees immobile. Precision pollination systems can change that equation.

“What I see as the future for apple orchards is to take over the pollination process,” Whiting says.

In that case, no bees would be brought in and no pollinizer trees would be necessary. Trees would bloom and an automated pollination system would fertilize a targeted number of flowers.

“The robot would scan the trees and say, ‘Ah, there’s an open king flower. Let’s give it a shot of pollen,’” Whiting says. The king flower is the ‘central’ flower in apple clusters and it typically opens before the others in the cluster.

“The potential of robotic pollination really excites me. Every year, crop load management is a perennial struggle. It’s a guessing game,” Whiting says.

Crop load management, trying to get the right number and quality of fruit, goes back a century. Not only would automated pollination simplify the process, machines could scan trees and determine the optimal number of fruit for individual trees.

Part of management is also thinning flowers so there aren’t too many apples in a cluster and have better fruit.

“Growers want consistency so badly that they are going through and pinching off flowers by hand, leaving one. They hire crews to do this at a cost of several thousand per acre,” Whiting says.

Karkee and other engineers demonstrated the ability for robots not only to pollinate, but also to thin flowers. Their prototype found pale pink flower clusters on the apple trees and mechanically removed several of them with a whirling wire-based system like a tiny weed whacker.

 

Once the fruit is ready, the orchard robot could ease the need for human harvesting.

For cherries, the UFO tree architecture and the right varieties that fall off easily are already ripe for automation. Whiting says running a mechanical harvester that uses a shaking method to release the fruit far outstrips the old, slow system of harvesting cherries by hand.

A worker can walk along with a remote-controlled machine that has an arm hitting branches and fruit falls into a basket. “They can harvest 50 times more fruit than picking by hand,” Whiting says.

An automated cherry harvesting system comes next, as do similar experiments for cider apples. “The potential for a fully mechanical harvester is so compelling,” Whiting says.

Karkee says research into apple picking started at CPAAS about eight years ago with support from the National Robotics Initiative.

Karkee and CPAAS then began collaborating several years ago with FFRobotics, a company from Israel working on automated fruit harvesting. Combining vision and AI expertise with the FFRobotics apple-picking robot hand, they have been testing it for the last few years in both Washington state and Israel.

As they develop and improve models for robots to identify fruit and branches, and make the right decisions to then grab the fruit, the technology gets closer to commercialization.

 

The ability for machines to evaluate and learn can also make agriculture more resilient, Khot says.

Khot, interim director of WSU’s AgWeatherNet in addition to his role at CPAAS, specializes in developing precision technologies. He and his team use drones to gather information on crop stress, water use, and plant needs.

Drone hovers near orchards and hops fields in central Washington state while researchers watch and control flight
Drones like this one being tested by Lav Khot and graduate students gather precise data to support orchardists. (Photo Zach Mazur)

That real-time data provided by drones can improve crop management such as spraying at the right time, refining irrigation, or mitigating heat stress on fruit with fogging and misters.

“Growers can lose up to 40 percent of their crop in hotter years,” Khot says. “We can use localized weather information, tied to localized sensing, to help growers avoid crop loss.”

Karkee also says bringing AI to agriculture can improve farming decisions by reducing inputs like fertilizer and labor. “We get better outcomes over time because AI can learn over time,” he says. “And it can help preserve expert knowledge from farmers.”

Engineer Manoj Karkee adjusts wires on a robot
Manoj Karkee fine-tunes one of the robots for orchard tasks. (Photo Zach Mazur)

One of the challenges for gathering that crucial data, Khot says, is a lack of broadband connectivity in rural areas, where the farms are. His work collects gigabytes of data from imaging with multispectral sensors and LIDARs. WSU and Khot are collaborating with 5G Open Innovation Lab and other partners to address the problem, but more needs to be done to provide real-time information for farmers.

 

The automation and AI research to improve orchards and farms promises to ease labor shortages and provide advanced tools, but CPAAS creates prototypes and not production, Zhang says. And testing and research require time.

Although the needs are urgent, “we prove the concepts but they need to move to commercialization. That takes years of work,” he says.

There’s certainly a demand for automated technologies, and some commercial products already enhance farms. For example, Carbon Robotics, a Seattle-based ag tech company with WSU alumni engineers, rolled out its LaserWeeder™ for eliminating weeds in specialty crops. [see sidebar below]

Fortunately, many partners for CPAAS bring global visibility and connect WSU to companies such as FFRobotics⁠—and eventually get automation into the field.

CPAAS has formal partnerships with Kyoto University in Japan, University of Sydney in Australia, and Leibniz Institute for Agricultural Engineering & Bioeconomy in Germany. Researchers also work with Carnegie Mellon University, Oregon State University, Michigan State University, Penn State University, and institutions from India to New Zealand.

Last spring, WSU launched the AgAID Institute with $20 million from USDA National Institute of Food and Agriculture and National Science Foundation. It will accelerate AI-driven ag technology research, and help the development of pruning and thinning robots, better models to detect crop stress from cold or water, and other research goals.

Other funding for CPAAS over the years has come from the Washington Tree Fruit Commission, USDA, Washington State Department of Agriculture, and many others. Wherever the funding comes from, the research has a single purpose.

“No matter how you approach mechanization, the end goal is to make farmers more productive and profitable,” Zhang says.

In turn, growers are very supportive of the work. “We have several of our research projects hosted by local growers to test the precision technologies in commercial settings,” Khot says.

Back in the CPAAS building, a small room off of the workshop will soon be home to its first ag control center. The computers will connect with remote robots and data-collecting instruments in Roza Farm. It’s long been a goal for Zhang, Khot, and Karkee to bring together technologies in a command center like this one, which could allow a farmer to run a large operation from a single place.

“In the last 40 years, the effort has been to develop precision agriculture, to precisely use resources. The next 40 years, we need to automate and smartly operate farms,” Zhang says.

Khot says this viewpoint fits WSU’s role to support agriculture and find technology that can help growers be less dependent on human labor. The transdisciplinary research teams can bring that vision to fruition.

“The farm of the future is integrated, not in a silo,” Khot says.

Sun rises slowly over hills in central Washington stateHills near Prosser (Photo Zach Mazur)

 

 

Web extras

Smart sensors at Cook Agronomy Farm to prepare for climate change effects on crops

Videos: Robots, drones, and other automated machines in orchards and fields

 

Learn more

Center for Precision and Automated Agricultural Systems

AgAid Institute: Partnerships between AI and agriculture (led by WSU)

FFRobotics: Partner with WSU on developing apple-picking robot

Pruning robot makes the first cut (Good Fruit Grower, May 25, 2022)

Farming Drives Toward ‘Precision Agriculture’ Technologies (Wired, May 14, 2022)

The ‘breathtaking’ advancement of agriculture technology (AgWeek opinion, July 26, 2022)

Automated drones could scare birds off agricultural fields (WSU Insider, June 1, 2022)

Will this fruit-picking robot transform agriculture? (The Guardian, May 28, 2022)

Smart thinning: Washington wine industry helps develop new vineyard technology by investing in collaboration with WSU researchers such as Manoj Karkee. (Good Fruit Grower, August 1, 2022)