Introduction
Autonomous systems operate through a continuous cycle of perception, decision-making, planning, action, and feedback. This cycle allows a machine to understand its surroundings, determine what should be done, perform the required action, and evaluate whether the result was successful.
A human driver follows a similar process while travelling on a road. The driver observes vehicles, pedestrians, traffic signs, road conditions, and signals. The information is interpreted to understand the situation. The driver then decides whether to continue, slow down, stop, or change direction. Finally, the selected action is carried out by controlling the steering wheel, accelerator, and brakes.
An autonomous vehicle, robot, drone, or intelligent machine performs this process using sensors, software, artificial intelligence, planning algorithms, controllers, and actuators.
The complete operational cycle can be represented as:
Sense → Perceive → Estimate → Predict → Decide → Plan → Control → Act → Evaluate
Each stage depends on the quality of the previous stage. If the sensors collect inaccurate information, perception may become unreliable. If perception is incorrect, the system may make an unsafe decision. If the decision is appropriate but the controller or actuator fails, the action may still be unsuccessful.
Reliable autonomy therefore depends on the coordination of the entire system rather than on a single algorithm or artificial-intelligence model.
Sensing the Environment
The process begins with sensing. Sensors allow the autonomous system to collect information about the external environment and its own internal condition.
A self-driving vehicle may use cameras to capture road images, LiDAR to measure distances, radar to detect moving objects, GPS to estimate geographic position, and inertial sensors to measure acceleration and orientation.
A mobile robot may use cameras, ultrasonic sensors, wheel encoders, and proximity sensors. A drone may use GPS, cameras, altimeters, inertial measurement units, and wind sensors. An industrial robot may use joint-position sensors, force sensors, temperature sensors, and machine-vision cameras.
Sensors do not directly provide complete understanding. They generate raw signals such as images, distances, electrical measurements, sound waves, point clouds, location coordinates, or motion values.
These signals must be processed before the autonomous system can understand what is happening.
Sensor performance depends on environmental conditions. Cameras may be affected by darkness, glare, fog, dust, or rain. GPS signals may become inaccurate near tall buildings or indoors. Ultrasonic sensors may have limited range, while LiDAR performance may decrease during heavy rain or snow.
For this reason, autonomous systems often use several different sensors. The information from one sensor can support or correct the information from another.
Preparing and Combining Sensor Data
Raw sensor information may contain noise, missing values, distortions, or timing differences. Before the data can be used, it must be cleaned, calibrated, synchronized, and converted into a useful format.
Camera images may require distortion correction, resizing, brightness adjustment, or noise removal. Radar measurements may require filtering to remove unwanted reflections. LiDAR point clouds may be reduced or reorganized to improve processing speed.
Sensors may also collect data at different frequencies. A camera may generate thirty frames each second, while an inertial sensor may produce hundreds of measurements during the same period. The system must align these measurements so that they represent the same moment.
Calibration ensures that the sensor measurements correspond correctly to the physical environment. Camera calibration corrects lens distortion and determines how image positions relate to real-world directions. Calibration between a camera and LiDAR allows the system to connect visual objects with measured distances.
After preprocessing, data from different sensors may be combined through sensor fusion.
Sensor fusion improves accuracy and reliability by using the strengths of several sensors. A camera may recognize an object as a pedestrian, LiDAR may calculate the pedestrian’s distance, and radar may estimate movement speed. The combined information provides a more complete understanding than any single sensor.
Perception
Perception is the process of transforming sensor data into meaningful information about the environment.
Sensing provides raw measurements, while perception explains what those measurements represent.
A camera produces pixels. The perception system identifies whether those pixels represent a road, vehicle, person, building, traffic signal, machine component, or obstacle.
A LiDAR sensor produces a three-dimensional point cloud. The perception system groups the points into surfaces and objects. A radar sensor produces reflections, which the system interprets to identify moving targets and estimate their speed.
Perception allows an autonomous system to answer questions such as what objects are present, where they are located, how far away they are, whether they are moving, and whether they may create a risk.
Computer vision, machine learning, deep learning, image processing, signal processing, and probabilistic methods are commonly used for perception.
Object Detection and Classification
Object detection identifies objects and determines their location within the environment.
An autonomous vehicle may detect cars, motorcycles, bicycles, pedestrians, traffic signs, road barriers, and animals. A warehouse robot may detect workers, shelves, packages, forklifts, and other robots.
Detection normally produces a location around each identified object. In an image, this may be represented by a bounding box. In a three-dimensional environment, the object may be represented by its position, size, and orientation.
Classification determines the category of the object. The system may classify an object as a pedestrian, vehicle, sign, tree, machine, wall, or unknown obstacle.
Detection and classification are closely connected, but they are not identical. Detection answers where an object is located, while classification answers what the object is.
Correct classification supports better decisions. A stationary wall, a moving cyclist, and a plastic bag may all occupy space in front of a vehicle, but each requires a different interpretation and response.
Environmental Segmentation
Autonomous systems often need to understand the complete scene rather than detect only individual objects.
Semantic segmentation classifies different areas of an image or sensor representation. A road scene may be divided into road surface, pavement, building, vehicle, pedestrian, vegetation, sky, and obstacle areas.
This allows the system to identify drivable space, walking areas, restricted zones, and environmental boundaries.
Instance segmentation separates individual objects belonging to the same category. For example, instead of identifying one general pedestrian region, the system separates each pedestrian individually.
Environmental segmentation is useful in autonomous driving, drone navigation, medical robotics, agriculture, manufacturing, and mobile robots.
An agricultural robot may separate crops, weeds, soil, and stones. A warehouse robot may distinguish free floor space from shelves and people. A drone may identify buildings, roads, trees, and water.
Distance and Depth Estimation
An autonomous system must often determine how far objects are from its current position.
LiDAR, radar, ultrasonic sensors, stereo cameras, and depth cameras can provide distance information.
Depth may also be estimated from images using computer-vision or deep-learning methods.
Distance estimation helps the system calculate stopping distance, safe movement, obstacle clearance, and navigation routes.
An object may be correctly detected but still create danger if its distance is estimated incorrectly. A vehicle that believes a pedestrian is farther away than the actual distance may apply braking too late.
Accurate depth and distance estimation are therefore essential for safe autonomous action.
Object Tracking
Objects in the environment may move over time. Tracking allows the autonomous system to follow detected objects across multiple sensor measurements.
A vehicle may track nearby cars, cyclists, and pedestrians. A surveillance drone may track a moving person or vehicle. A factory robot may track items moving along a conveyor.
Tracking allows the system to estimate direction, speed, acceleration, and future position.
Without tracking, each sensor frame would be treated as a separate observation. The system would know that an object exists but may not understand how it is moving.
Tracking algorithms connect observations over time and assign a consistent identity to each object.
The system may use optical flow, Kalman filtering, feature matching, deep-learning trackers, or combinations of these methods.
Localization and Self-Awareness
An autonomous system must understand not only the surrounding environment but also its own position and condition.
Localization determines where the system is located and how it is oriented.
A road vehicle may use GPS, inertial sensors, wheel movement, cameras, LiDAR, and digital maps. A warehouse robot may use wheel encoders, visual landmarks, and indoor maps. A drone may combine GPS, altitude measurements, camera data, and inertial sensing.
The estimated state may include position, direction, speed, acceleration, altitude, wheel movement, joint position, battery level, and system health.
This internal understanding is sometimes described as state estimation.
A system cannot plan a reliable route unless it knows its starting position. It cannot control movement accurately unless it understands its current speed and orientation.
Localization and state estimation therefore connect perception with navigation and control.
Mapping
Mapping provides a representation of the environment.
A map may show roads, lanes, walls, obstacles, buildings, shelves, pathways, landmarks, restricted areas, and target locations.
Some autonomous systems use maps created in advance. Others build maps while operating.
Simultaneous Localization and Mapping, commonly called SLAM, allows a system to estimate its own position while creating a map of an unfamiliar environment.
A robot entering an unknown room may use camera or LiDAR data to identify walls and objects. As it moves, it updates both the map and its estimated position.
Mapping allows the system to remember environmental structure and plan movement beyond the immediate sensor range.
Maps may be two-dimensional or three-dimensional. They may contain geometric information, semantic labels, traffic rules, object locations, or movement restrictions.
Understanding the Situation
Perception is not limited to recognizing objects. The autonomous system must understand the overall situation.
A vehicle approaching an intersection must determine whether the traffic light is red or green, whether pedestrians are crossing, whether another vehicle has the right of way, and whether the intersection is clear.
A warehouse robot approaching a narrow passage must determine whether another robot is entering the same passage, whether a worker is nearby, and whether there is enough space to continue.
This stage is sometimes called scene understanding or situational awareness.
Situational awareness combines information from object detection, tracking, localization, maps, rules, and system goals.
The system uses this combined representation to understand what is happening and what may happen next.
Prediction
Autonomous systems must often predict future events before selecting an action.
Prediction estimates how the environment may change.
A self-driving vehicle may predict whether a nearby car is likely to change lanes. It may estimate whether a pedestrian is preparing to cross the road. A warehouse robot may predict whether another robot will enter its planned path.
A drone may predict how wind will affect its movement. An industrial system may predict whether a machine component is likely to fail.
Prediction may be based on motion equations, historical patterns, machine-learning models, behavioural rules, or probability.
The system may calculate several possible future outcomes rather than one certain result.
For example, a pedestrian standing near a road may continue standing, turn away, or begin crossing. The system may assign a probability to each possibility and choose an action that remains safe under uncertainty.
Prediction allows the machine to respond early rather than waiting for danger to become immediate.
Managing Uncertainty
Perception and prediction are never completely certain.
Sensors may be noisy, objects may be partially hidden, and environmental conditions may change quickly. Machine-learning models may produce incorrect classifications. Human behaviour may be unpredictable.
The autonomous system should therefore represent uncertainty rather than treating every estimate as completely correct.
A perception model may provide a confidence score. A localization system may calculate a range of possible positions. A prediction system may estimate several possible object trajectories.
When uncertainty is high, the system may reduce speed, increase its safety distance, collect additional sensor information, avoid a complex manoeuvre, or request human assistance.
A safe autonomous system should become more cautious when its understanding becomes less reliable.
Decision-Making
Decision-making determines what the autonomous system should do.
The decision is based on the system’s goals, current state, environmental understanding, predicted events, operational rules, safety limits, and available actions.
A vehicle may decide to continue, stop, reduce speed, change lanes, turn, or wait. A drone may decide to continue its mission, change altitude, avoid an obstacle, return to base, or land safely.
A mobile robot may decide which package to collect, which route to follow, whether to allow another robot to pass, or whether to recharge its battery.
Decision-making may occur at several levels.
Mission-level decisions determine the overall task. A robot may be assigned to deliver a package to a specific location.
Behaviour-level decisions determine the general action. The robot may decide to follow a corridor, wait for a worker, or move around an obstacle.
Motion-level decisions determine the immediate movement. The system may choose a speed, direction, or turning angle.
These decisions must remain coordinated. A lower-level movement should support the higher-level mission while following safety restrictions.
Rule-Based Decision-Making
Rule-based systems use predefined conditions to select actions.
A simple rule may state that the robot must stop if an obstacle is detected within a safe distance.
A vehicle may follow a rule that requires stopping when a red traffic signal is detected.
Rule-based decision-making is easy to understand and test. It is useful for clear safety conditions and structured environments.
However, real environments may contain too many possible situations for engineers to write every rule manually.
Rules may also conflict. A vehicle may be required to remain within its lane but may also need to move around a broken-down vehicle. The decision system must determine which action is safe and legally acceptable.
Rule-based methods are therefore often combined with planning, probability, and machine learning.
Learning-Based Decision-Making
Learning-based systems use models trained from data or experience.
A model may predict which action is likely to produce the best result under a particular condition.
Machine learning may help classify situations, predict environmental behaviour, estimate risk, or recommend actions.
Reinforcement learning may allow a robot to learn navigation or control strategies through repeated interaction with a simulated environment.
Learning-based decision-making can manage complex patterns but may be difficult to explain and validate.
The system may also make incorrect decisions when it encounters a situation that differs from its training data.
For safety-critical autonomy, learning-based decisions are normally restricted by programmed safety rules and operational limits.
Hybrid Decision-Making
Most practical autonomous systems use hybrid decision-making.
A hybrid system combines rules, machine learning, planning algorithms, optimization, and feedback control.
An autonomous vehicle may use deep learning to identify a pedestrian, a tracking model to estimate movement, a probabilistic model to predict crossing behaviour, and a fixed safety rule to require braking when risk exceeds a defined limit.
The system may then use an optimization algorithm to calculate a safe path and a conventional controller to apply braking.
This approach uses each method where it is most effective.
Machine learning supports complex perception and prediction. Rules enforce safety and legal requirements. Planning algorithms compare possible actions. Controllers produce stable physical movement.
Behaviour Planning
Behaviour planning converts environmental understanding into a general action.
A vehicle may select behaviours such as following the lane, stopping at an intersection, yielding, overtaking, parking, or performing an emergency stop.
A mobile robot may select behaviours such as moving toward a goal, waiting, following a wall, avoiding an obstacle, returning to charge, or requesting help.
Behaviour planners may use finite-state machines, behaviour trees, decision trees, rule systems, or learned policies.
A finite-state machine represents the system using defined states. A delivery robot may have states such as waiting, navigating, avoiding an obstacle, delivering, returning, and charging.
The system moves from one state to another when specified conditions occur.
Behaviour trees organize decisions in a hierarchical structure. They are commonly used in robotics because they allow complex actions to be divided into smaller tasks.
Behaviour planning provides a structured connection between high-level goals and detailed movement.
Selecting the Best Action
An autonomous system may have several possible actions. It must compare them according to safety, efficiency, time, energy, comfort, legal requirements, and mission objectives.
A vehicle approaching a slow-moving object may consider stopping, reducing speed, changing lanes, or following at a safe distance.
Each option has possible advantages and risks.
The system may assign a cost to each action. Collision risk may receive a very high cost. Longer travel time may receive a lower cost. Energy consumption, passenger comfort, and route length may also be included.
The action with the lowest acceptable cost may be selected.
Safety restrictions should normally be treated as mandatory constraints rather than optional preferences.
The system should not select a faster action if it creates an unacceptable risk.
Path Planning
After selecting a general action, the system must determine a route or path.
Path planning calculates a collision-free connection between the current position and a target.
A warehouse robot may calculate a path through corridors and storage areas. A drone may calculate a flight path around buildings and trees. A vehicle may determine how to follow the road while avoiding obstacles.
Global planning uses a larger map to calculate the overall route.
Local planning responds to nearby objects and changing conditions.
A global planner may select the shortest route through a warehouse. A local planner may temporarily change direction when a worker blocks the selected corridor.
Path-planning algorithms may include Dijkstra’s algorithm, A-star search, Rapidly-exploring Random Trees, potential-field methods, and optimization-based methods.
The path must consider the physical size and movement limitations of the system.
A route that is suitable for a small robot may not be suitable for a large vehicle. A drone can move vertically, while a ground robot cannot.
Trajectory and Motion Planning
A path describes where the system should move. A trajectory also describes how movement should occur over time.
Trajectory planning includes speed, acceleration, direction, turning rate, and timing.
A vehicle may follow the same geometric path at different speeds. The safer trajectory depends on road conditions, nearby objects, vehicle dynamics, and passenger comfort.
Motion planning considers the physical limitations of the system.
A robotic arm cannot move instantly between two positions. Its joints have speed, angle, and force limits. A vehicle has a minimum turning radius and requires distance to stop.
The planner must generate actions that are physically possible and safe.
Control
The control system converts the planned trajectory into specific commands for the machine.
A vehicle planner may request a particular speed and direction. The controller determines the required steering angle, motor force, and braking pressure.
A drone planner may request movement to a selected position. The controller adjusts the speed of individual propellers to achieve the required altitude, direction, and stability.
A robotic arm planner may specify a target position. The controller calculates how each joint should move.
The control system continuously compares the desired movement with the actual movement.
If the vehicle is moving too quickly, braking is increased. If a robot moves away from its intended path, motor commands are corrected.
Common control methods include proportional control, proportional–integral–derivative control, state-space control, and model-predictive control.
Actuation
Actuation is the stage in which the system physically performs the selected action.
Actuators convert control commands into movement or another physical effect.
A vehicle uses motors, steering mechanisms, and brakes. A drone uses propellers and control surfaces. A robot uses electric motors, hydraulic systems, joints, and grippers.
An autonomous irrigation system may use pumps and valves. A smart building may use heating, cooling, lighting, and access-control devices.
The actuator must respond accurately and within the required time.
If a controller sends a braking command but the brake actuator responds slowly, the system may fail to stop safely.
Actuator condition must therefore be monitored through feedback sensors.
Feedback and Evaluation
After an action is performed, the autonomous system observes the result.
This creates a closed feedback loop.
The system compares the actual outcome with the expected outcome.
A robot may check whether it reached the intended location. A vehicle may verify that it is following the planned lane. A robotic arm may confirm that an object was grasped successfully.
If the result differs from the plan, the system makes a correction.
A drone pushed away from its route by wind may adjust its direction. A robot that fails to grasp an object may reposition its gripper and try again.
Feedback allows the system to correct errors and respond to changing conditions.
Without feedback, the system would operate in an open loop and would not know whether its commands were completed successfully.
Continuous Replanning
Autonomous operation is not a one-time process.
The system must repeatedly sense, perceive, decide, plan, and act because the environment can change at any moment.
A path that was safe one second ago may become blocked by a person, vehicle, animal, or object.
The system continuously updates its environmental model and checks whether the current decision remains appropriate.
If conditions change, the planner generates a new action or trajectory.
This process is known as replanning.
Fast replanning is important in dynamic environments. However, the system should avoid changing decisions too frequently, because unstable behaviour may produce sudden or confusing movements.
The decision and planning systems must balance responsiveness with stability.
Safety Monitoring
Safety monitoring operates throughout the complete perceive–decide–act cycle.
A separate safety layer may observe sensor quality, system health, planned movement, actuator response, and environmental risk.
The safety system may override a normal decision if it detects danger.
For example, the navigation planner may command a robot to continue toward its destination. If an independent collision sensor detects an object at very close range, the safety system may immediately stop the robot.
Safety monitoring may include emergency braking, speed limits, restricted zones, sensor-failure detection, communication monitoring, and manual override.
A reliable autonomous system should not depend on only one perception model or decision algorithm for critical safety functions.
Handling Failures
Failures can occur at every stage.
A sensor may stop working, a perception model may misclassify an object, localization may become inaccurate, the planner may fail to find a path, or an actuator may not respond.
The system should identify failures and move toward a safe condition.
If a camera fails, the system may rely temporarily on radar and LiDAR. If localization uncertainty becomes too high, a robot may reduce speed or stop. If a drone loses communication, it may return to its launch location or perform a controlled landing.
The correct fallback action depends on the system and environment.
Stopping immediately may be safe for a factory robot but dangerous for a vehicle travelling in fast traffic.
The system should therefore select a minimum-risk response appropriate to the situation.
Human Involvement
Humans may remain involved at different stages of autonomous operation.
A human may define the goal, approve the route, monitor the system, respond to warnings, or take control during unusual situations.
In a human-in-the-loop system, a person approves important decisions before the machine acts.
In a human-on-the-loop system, the machine operates independently while a human supervises it.
In highly autonomous operation, the machine may perceive, decide, and act without immediate human input, but people remain responsible for system design, maintenance, monitoring, and accountability.
The system should clearly communicate its current operating mode, confidence, task status, and need for assistance.
Example of an Autonomous Vehicle
An autonomous vehicle demonstrates the complete perceive–decide–act process.
Cameras, radar, LiDAR, GPS, and inertial sensors collect information about the road and vehicle state.
The system preprocesses and combines the sensor data. Computer-vision models detect lanes, vehicles, pedestrians, and traffic signals.
Localization algorithms estimate the vehicle’s position on a digital map. Tracking systems calculate the movement of nearby road users.
Prediction models estimate whether a pedestrian may cross or another vehicle may change lanes.
The decision system determines whether the vehicle should continue, slow down, stop, turn, or change lanes.
The planner calculates a safe trajectory that respects road boundaries, traffic rules, vehicle limits, and surrounding objects.
The control system converts the trajectory into steering, acceleration, and braking commands.
The vehicle performs the action through physical actuators.
Sensors then measure the result. If the vehicle is not following the planned path, the controller corrects the movement.
This process is repeated continuously throughout the journey.
Example of a Warehouse Robot
A warehouse robot receives a task to transport a package from a storage area to a packing station.
Its cameras, LiDAR, wheel encoders, and proximity sensors collect information.
The perception system detects shelves, workers, packages, walls, and other robots.
Localization determines the robot’s position within the warehouse map.
The decision system selects the delivery task and determines whether the robot has enough battery power.
The path planner calculates a route through the warehouse.
If another robot blocks the corridor, the system decides whether to wait or calculate another route.
The controller sends commands to the wheel motors. Feedback sensors confirm the speed and direction.
When the robot reaches the target, it verifies the delivery and receives another task or returns to charge.
Example of an Autonomous Drone
An autonomous drone may be assigned to inspect a bridge.
The drone uses cameras, GPS, inertial sensors, altitude sensors, and obstacle-detection sensors.
Perception identifies the bridge structure, nearby objects, and safe flight space.
Localization determines the drone’s position and orientation.
The system predicts the effect of wind and identifies possible collision risks.
The planner generates a flight route around the structure.
The controller adjusts individual propeller speeds to maintain stable movement.
If wind pushes the drone away from the planned route, feedback control corrects its position.
If battery power becomes low or a sensor fails, the system may return to its launch point or land safely.
Example of an Industrial Robot
An industrial robot may be required to identify and pick parts from a conveyor.
A camera captures images of the moving parts.
The perception system detects each part, classifies its type, and estimates its position and orientation.
The decision system determines which part should be selected.
The motion planner calculates how the robotic arm should move without colliding with nearby equipment.
The controller generates commands for each robotic joint.
The gripper closes around the object, and force sensors confirm whether the object has been held securely.
If the grasp fails, the system may adjust its position and try again.
This example shows how perception, decision-making, motion planning, control, action, and feedback operate together.
Importance of System Integration
The performance of the complete autonomous system depends on the integration of all stages.
Accurate perception has limited value if decision-making is too slow. A correct decision cannot produce safe behaviour if the planner generates an impossible path. A reliable plan will fail if the controller or actuator cannot execute it correctly.
Communication delays between modules may also create problems. Environmental information may become outdated before the system acts.
The autonomous architecture must therefore process information within strict time limits and maintain consistent data between modules.
Engineers test individual components, but they must also test the complete perceive–decide–act cycle under realistic conditions.
Simulation, controlled testing, hardware-in-the-loop testing, and field trials help identify integration problems.
Limitations of the Perceive–Decide–Act Process
The perceive–decide–act model provides a useful explanation of autonomy, but real systems may not operate as a simple linear sequence.
Several processes often occur at the same time.
The perception system continuously updates object information while the planner generates a trajectory. The controller continues managing movement while the decision system evaluates a new situation.
Autonomous systems may also contain several feedback loops operating at different speeds.
A motor controller may update hundreds of times per second, while a global route planner may update less frequently.
The architecture is therefore better understood as a connected and continuous network rather than a one-directional chain.
Conclusion
Autonomous systems operate through a continuous cycle of sensing, perception, state estimation, prediction, decision-making, planning, control, action, and feedback.
Sensors collect raw information about the environment and the system’s internal condition. Data processing and sensor fusion improve the quality and consistency of this information.
Perception identifies objects, environmental features, free space, distance, movement, and scene conditions. Localization and mapping determine where the system is and how the environment is structured.
Prediction estimates how surrounding objects and conditions may change. Decision-making selects an appropriate behaviour based on the system’s goals, rules, risks, and operating limits.
Path, motion, and trajectory planning determine how the selected action should be completed. The control system converts the plan into precise commands, and actuators perform the physical action.
Feedback allows the machine to compare the actual result with the intended result and correct errors. Continuous replanning allows the system to respond when the environment changes.
Safety monitoring, fault detection, fallback actions, and human oversight support reliable operation throughout the cycle.
An autonomous system becomes effective only when perception, decision-making, planning, control, and action operate together accurately, quickly, and safely.
Comments are closed.