Every rendering algorithm ever written answers the same question: given a description of a 3D scene, what color is each pixel on the screen? The scene might be a pile of triangles, a cloud of scanned points, or a neural network trained on photographs, but the output is always the same flat grid of colored dots. What separates one renderer from the next is not the question but the strategy used to answer it, and almost every strategy falls into one of two camps.

The first camp works object-order. It loops over the geometry and, for each piece, figures out which pixels that piece covers, then writes color into them. You can think of it as scattering: the renderer throws the scene at the screen and lets each primitive land where it may. Rasterization, the technique behind every real-time game, is the canonical example, and so is drawing a point cloud.

The second camp works image-order. It loops over the pixels and, for each one, asks what part of the scene is visible along the line of sight through that pixel. This is gathering: every pixel reaches back into the world and collects the light that arrives at it. Ray tracing, path tracing, ray marching, and NeRF all gather.

Neither order is more correct than the other; they trade different costs. Object-order does a bounded amount of work per primitive and maps beautifully onto parallel hardware, but it struggles the moment an effect needs global knowledge of the scene, like a shadow or a reflection. Image-order can answer those global questions directly, because a pixel that has found the first surface can simply cast another ray to ask about the light reaching it, but the cost of finding what each ray hits grows with the complexity of the scene.

This guide walks the major families in that frame. Parts 1 and 2 cover the classical renderers, the ones that take an authored scene, a model somebody built, and turn it into an image. Part 3 covers the recent and stranger turn: renderers that take a captured scene, reconstructed from ordinary photographs, and where the line between reconstruction and rendering all but disappears. Each section has a stepped, interactive demo so you can watch the pixels fill in.

Part 1: Scattering geometry to the screen

Rasterization

Rasterization is the workhorse. If you are reading this on a screen that updated smoothly as you scrolled, something rasterized it. The idea is old and direct: take a geometric primitive, usually a triangle, and determine which pixels it covers.

The lineage runs through the earliest days of computer graphics. In 1965 Jack Bresenham published an integer-only algorithm for stepping a plotter pen along the pixels closest to an ideal line^[01], establishing the incremental, integer-arithmetic spirit that graphics hardware still favors, even though modern GPUs settle triangle coverage a different way, with an edge function per side rather than Bresenham-style edge walking. The other foundational piece arrived in 1974, when Edwin Catmull’s PhD thesis introduced both the z-buffer, a per-pixel depth store that resolves which surface is nearest without sorting the geometry first, and texture mapping, wrapping an image onto a surface^[02]. Those two ideas, incremental rasterization and a depth buffer, are still the spine of every GPU.

The modern pipeline is a fixed sequence of stages. Vertices enter in model space and are transformed into world space, then into the camera’s view space, then projected onto the 2D image plane; the projection is what makes distant things smaller. Each triangle is then set up for rasterization, where an edge function for each of its three sides tests, for a candidate pixel, which side of the edge the pixel lies on. A pixel inside all three edges is covered. For every covered pixel, or fragment, the renderer runs a depth test against the z-buffer and, if the fragment is nearer than what is already there, runs a shader to compute its color and writes both color and depth.

The reason this dominates real-time rendering is the shape of its cost. The work per triangle is bounded and predictable, and every fragment is independent of every other, so the whole process fans out across the thousands of cores on a GPU with almost no coordination. Rasterization does not naturally know about the rest of the scene, which is why shadows, reflections, and indirect light are all bolted on as separate passes and tricks, but for putting a lot of triangles on screen sixty or more times a second, nothing else comes close.

Step through the pipeline here, from vertices to shaded fragments:

Voxels

Not all object-order geometry is triangles. A voxel representation stores the scene as a 3D grid of cells, the volumetric cousin of the 2D pixel. Instead of describing a surface with its boundary, you fill space with little cubes and mark each one as solid or empty, or give it a density and a color. Minecraft is the famous example, but the idea long predates it in scientific and medical imaging, where the data genuinely is volumetric: a CT or MRI scan is a stack of slices, which is to say a grid of density samples, and there is no surface to speak of until you choose to extract one.

Rendering a voxel grid can be done either way, which makes it the one representation in this guide that sits in both camps. Object-order, you mesh it: a face shared between two solid cells is never visible from outside, so cull it and rasterize only the boundary faces, which is how a blocky world with millions of cells still hits frame rate. Image-order, you march a ray through the lattice cell by cell, using a grid traversal that always crosses the nearer of the next vertical or horizontal cell boundary^[03], and accumulate whatever it passes through. That second route is volume rendering: give each cell a density rather than a solid/empty flag, turn density into color and opacity, and composite front to back so that dense regions occlude what lies behind them. It is how a radiologist sees soft tissue fade to bone, and it is the same accumulate-along-the-ray idea we will meet again, in a very different guise, when photographs enter the picture in Part 3.

Sample a shape onto a lattice, then render it both ways:

Point clouds

Strip geometry down to its barest primitive and you arrive at the point cloud: a set of points in space, each with a position and maybe a color, and no connectivity at all. There are no triangles, no edges, no faces, nothing that says which points belong to the same surface. A point cloud is just samples of where the world was.

That rawness is exactly what you get from measurement. A LiDAR scanner sweeps a laser across a scene and records the distance to whatever each pulse hits, producing millions of points; photogrammetry, which we will return to, produces points by triangulating features seen in overlapping photos. In both cases the natural output is a cloud, because the instrument samples surfaces without any idea of how those samples connect.

Rendering a cloud object-order is easy: project each point to the screen and color a pixel, or a small splat, at its location. The trouble is the gaps. Points are zero-dimensional, so when you get close to the cloud, or when the sampling was sparse, you see straight through the gaps between points to whatever is behind. Filling those gaps in a way that looks like a continuous surface is the whole problem, and the most striking recent answer, Gaussian splatting, grows directly out of it. Hold that thought for Part 3.

Rotate a cloud and watch the gaps open up as you approach:

Part 2: Gathering light per pixel

The image-order camp inverts the loop. Rather than asking where each triangle lands, it asks, for each pixel, what the eye sees along the ray through that pixel. Everything expensive and everything beautiful about this family comes from the fact that once you have a ray, you can cast more of them.

The ray lineage: casting, tracing, path tracing

The first step is due to Arthur Appel in 1968, who cast one ray per pixel to find the nearest surface and used a second ray toward the light to decide whether that surface sat in shadow^[04]. This is ray casting: a single bounce, first hit plus a shadow test, and no more.

The leap to realism came in 1980 with Turner Whitted, who made the rays recursive^[05]. When a ray hits a shiny surface, spawn a reflection ray and follow it; when it hits glass, spawn a refraction ray too, and recurse. Suddenly mirrors reflected mirrors and glass bent the world behind it, all falling out of the same casting operation applied again to the surfaces the first rays found. Whitted’s images of glossy spheres over a checkerboard became the canonical look of early ray tracing.

Whitted’s model was still limited: it followed only the sharp, mirror-like paths and ignored the soft, diffuse interreflection that fills a real room with light. The general statement arrived in 1986, when James Kajiya wrote down the rendering equation^[06], which says that the light leaving a point is the light it emits plus all the light arriving from every direction, reflected according to the surface’s response:

\[L_o(\mathbf{x}, \omega_o) = L_e(\mathbf{x}, \omega_o) + \int_{\Omega} f_r(\mathbf{x}, \omega_i, \omega_o)\, L_i(\mathbf{x}, \omega_i)\,(\omega_i \cdot \mathbf{n})\,d\omega_i\]

Read left to right: the outgoing radiance \(L_o\) at point \(\mathbf{x}\) in direction \(\omega_o\) equals the emitted radiance \(L_e\) plus an integral over the hemisphere \(\Omega\). Inside the integral, \(f_r\) is the bidirectional reflectance distribution function, the surface’s rule for how much light coming from \(\omega_i\) scatters toward \(\omega_o\); \(L_i\) is the incoming radiance from \(\omega_i\); and the \((\omega_i \cdot \mathbf{n})\) term accounts for light arriving at a grazing angle spreading over more surface. The equation is recursive, because the incoming \(L_i\) at one point is the outgoing \(L_o\) of another, and that recursion is the whole difficulty: light bounces endlessly.

Path tracing is the Monte Carlo answer. You cannot evaluate that hemispherical integral in closed form, so you sample it: from each hit, shoot one randomly chosen bounce ray, and average the results of many such random paths per pixel. With enough samples the average converges to the true integral, which is why an unconverged path-traced image is grainy and a converged one is photoreal. This is the engine behind essentially every animated film and visual-effects shot today.

A worked-out special case is worth calling out, because it hides in plain sight in older games. A grid-based raycaster like the one behind Wolfenstein 3D is ray casting stripped to a 2D plane: one ray per screen column, marched cell by cell through a flat grid until it meets a wall. No recursion, no hemisphere, just the first hit. I pulled that Wolf3D engine apart in an earlier post. It is the humblest possible member of this family, and it ran on a 386.

For decades the rendering equation lived offline, minutes to hours per frame. What changed is hardware: modern GPUs ship dedicated ray tracing cores that intersect rays against geometry in real time, and, just as importantly, machine-learned denoisers that take a noisy image traced with only a handful of samples per pixel and clean it up convincingly. Between the two, real-time path tracing has crossed from research into shipping games.

Cast a ray, watch it bounce, and see the shadow and reflection rays spawn:

Ray marching and signed distance fields

Ray tracing needs to intersect a ray with explicit geometry, which means solving, per triangle or per sphere, exactly where the ray meets the surface. Ray marching takes a different route to the same first hit. Instead of an explicit intersection, it represents the scene as a signed distance field, a function that, given any point in space, returns the distance to the nearest surface (negative inside it). To find where a ray hits, you start at the camera and repeatedly step forward by exactly the distance the field reports, because that is the largest step guaranteed not to overshoot any surface. As you near a surface the reported distance shrinks and the steps get smaller, and you stop when the distance drops below a small threshold. This is sphere tracing.

The payoff is that the surface is never stored, only evaluated. A whole world can be a handful of math functions combined with smooth minimums and domain repetition, which is why ray marching is the beating heart of the demoscene and of shader playgrounds like Shadertoy, where entire animated scenes fit in a single fragment program with no geometry buffer at all. Implicit surfaces also blend and deform in ways that are awkward for triangle meshes: melting two shapes into one is a one-line change to the distance function.

March a ray through a distance field and watch the steps shrink near the surface:

Radiosity, the other global illumination

Path tracing is not the only way to solve the rendering equation. Two years before Kajiya, in 1984, Cindy Goral and colleagues introduced radiosity, a finite-element method borrowed from heat transfer^[07]. Instead of tracing individual light paths, radiosity divides every surface into patches and solves a large linear system for how much light each patch sends to every other, converging on a consistent distribution of diffuse energy across the whole scene at once.

Radiosity has two properties worth noting against the ray family. It is view-independent: because it solves for the light on the surfaces themselves, not for a particular image, you can walk a camera through the solved scene for free, which made it attractive for architectural walkthroughs. And in its classic form it handles only diffuse interreflection, the soft color bleeding of matte surfaces, not sharp reflections. The famous test scene for it, the Cornell Box, a simple room with colored walls whose red and green bleed subtly onto a white surface, came out of the same Cornell group and is still a standard benchmark for global illumination.

Interlude: RenderMan, which changed camps

The two halves of this guide are not a taxonomy of eras so much as a taxonomy of strategies, and the clearest proof is a single renderer that lived in both. RenderMan has been Pixar’s production renderer since the late 1980s, and the system it originally implemented, Reyes, was object-order to its bones^[08]. The name is a joke that doubles as a mission statement: Renders Everything You Ever Saw.

Reyes takes each primitive and, rather than testing it against pixels directly, dices it into a grid of micropolygons, quadrilaterals roughly the size of a pixel or smaller, splitting anything too large to dice cleanly and recursing. Once everything is micropolygons, the architecture inverts the usual order and shades before it hides: color is computed per micropolygon on the grid, in parallel and coherently, and only then are the shaded micropolygons sampled against the image with stochastic sample positions, which is what buys motion blur and depth of field almost for free. Displacement stops being a trick, because a micropolygon grid can simply be pushed around before sampling. It is scattering geometry to the screen, the same camp as Part 1, but tuned for film rather than frame rate: bounded memory, arbitrary surface complexity, and a shading rate you can dial. The other half of RenderMan, the RenderMan Interface and its shading language^[09], is why programmable shading is normal today; the vertex and fragment shaders in Part 1 are its descendants.

What makes it worth pausing on here is what happened next. As path tracing got cheap enough and artists stopped wanting to fake global light, Pixar rebuilt RenderMan around RIS, a physically based path tracer, and with RenderMan 21 in 2016 the Reyes pipeline was removed outright. The same product, the same films, the same shading heritage, moved from the object-order camp to the image-order one because the economics underneath it changed. The strategies in this guide are choices about how to pay for an answer, and RenderMan is what it looks like when the price changes and a renderer re-chooses.

Part 3: The captured turn

Everything so far assumed an authored scene: a modeler built the triangles, an artist painted the textures, someone placed the lights. But suppose you do not have any of that. Suppose all you have is a few dozen photographs of a real object or room, taken from different angles. Can you render new, never-photographed views of it?

The classical starting point is photogrammetry. Given overlapping photos, you can find features that appear in several of them and, from how those features shift between views, solve for both the camera positions and the 3D location of each feature. The output is a point cloud, which loops us straight back to Part 1, along with the gap problem: a cloud of reconstructed points is sparse and full of holes, and turning it into something that looks solid from a new angle is hard.

NeRF: the scene as a neural field

The 2020 breakthrough from Ben Mildenhall and colleagues, NeRF, or neural radiance fields, threw out the explicit geometry entirely^[10]. A NeRF represents the whole scene as a single small neural network. You feed it a 3D point and a viewing direction, and it returns the color and the density at that point. To render a pixel, you march a ray through this field exactly like the volume rendering from the voxel section: sample many points along the ray, ask the network for the color and density at each, and accumulate them front to back, weighting by density so that a dense region occludes what is behind it.

The network is trained by the rendering itself. You render the pixels of your known photographs through the field, compare against the real photos, and adjust the network weights until the rendered views match. Once trained, the field can be rendered from viewpoints that were never photographed, and the results were startling: view-dependent highlights, thin structures, and soft materials all reconstructed from a few dozen images. NeRF is image-order and volumetric, and it is photoreal, but it is also slow, because rendering one pixel means many forward passes through a neural network.

Gaussian splatting: captured realism at raster speed

The speed problem is what 3D Gaussian splatting solved in 2023. Bernhard Kerbl and colleagues kept the idea of optimizing a scene to match photographs but threw out the neural network and the per-ray marching in favor of an explicit, object-order primitive^[11]. The scene is represented as a large collection of 3D Gaussians, fuzzy anisotropic blobs, each with a position, a covariance (its shape and orientation), an opacity, and a view-dependent color. Starting from a photogrammetry point cloud, the positions and shapes and colors of these Gaussians are optimized by gradient descent until the rendered scene matches the input photos, with the optimizer splitting and pruning Gaussians as needed.

The reason it runs in real time is that rendering it is rasterization. To draw a frame, each 3D Gaussian is projected to a 2D ellipse on the screen, the ellipses are sorted by depth, and they are blended front to back with their opacities, the same alpha compositing a GPU already does. There is no ray marching and no network query in the render loop, just splatting sorted blobs, which is why a Gaussian-splat scene renders at hundreds of frames per second where the equivalent NeRF crawled. It is the direct answer to the point cloud’s gap problem from Part 1: instead of infinitesimal points that leave holes, you place fuzzy overlapping blobs that cover the surface continuously. Captured realism, at raster speed.

Optimize a handful of Gaussians to a target and watch them splat together:

When to use what

The families are less rivals than tools for different jobs. Object-order methods win when you need many frames per second of an authored scene; image-order methods win when you need global light or a view of a captured one. Here is the whole guide at a glance:

Family	Order	Primitive	Scene source	Realism	Speed	Typical use
Rasterization	Object	Triangle	Authored	Medium	Very fast	Real-time games, UI
Ray/path tracing	Image	Triangle/analytic	Authored	Very high	Slow to real-time	Film, offline GI, RTX
Ray marching	Image	Distance field	Authored (implicit)	High	Medium	Demoscene, procedural
Point clouds	Object	Point	Captured/authored	Low	Fast	LiDAR, scanning
NeRF	Image	Neural field	Captured	Very high	Slow	Novel view synthesis
Gaussian splatting	Object	3D Gaussian	Captured	High	Real-time	Capture playback, VR

The boundaries in that table are dissolving. A modern game frame is a hybrid: it rasterizes the bulk of the scene for speed, traces rays for the shadows and reflections that rasterization cannot do well, and runs a machine-learned denoiser and upscaler over the result to make a few samples per pixel look like many. On the captured side, Gaussian splatting is already moving from research demos into game engines and virtual production, precisely because its render step is ordinary rasterization and slots into pipelines that were built for triangles.

The through-line is the one we started with. Whether a renderer scatters geometry to the screen or gathers light per pixel, whether its scene was authored by an artist or captured from photographs, it is still answering that same first question, one pixel at a time. Everything else is a choice about how to pay for the answer.

References / Further Reading