The Stride Model: How Tensors Represent Reshape and Transpose

Reshape, transpose, and view feel unrelated until you understand one thing: none of them move data. They only change how it's read.

Let's suppose a tensor.

Two rows, three columns. This is the logical view: how we think about the tensor, and how the library prints it back to us.

But memory has no rows and no columns. Memory is one long line. So the same tensor is really stored like this:

Say an int takes 2 bytes. Then the first element sits at address 100, the next at 102, and so on. A single pointer marks where the block begins, and from there every element is reachable by arithmetic.

This is the part worth holding on to: almost everything we do to a tensor is done to the logical view. The flat vector mostly stays exactly where it is.

Stride is what connects the two

If memory is flat, how does the library know that 4 begins the second row? It uses stride.

For our tensor, shape is [2, 3] and stride is [3, 1]. Read it one dimension at a time: to move one step along the row dimension, jump 3 elements forward. To move one step along the column dimension, jump 1 element forward.

So element (i, j) lives at position i * 3 + j * 1 in the flat vector. Element (1, 0) is at position 3, which holds 4. Correct.

Shape tells you what the tensor looks like. Stride tells you how to walk the memory to get there. Two tensors can share the same memory and disagree completely about their shape, and stride is the reason that works.

Reshape just recomputes the stride

When we call .reshape(3, 2), it takes the values in order and fills the new shape two at a time, until it has three rows.

Now look at memory. Nothing moved. The values are still 1 2 3 4 5 6 sitting where they always were. The only thing that changed is the stride:

[3, 1]  →  [2, 1]

Each row is 2 elements long now, so the row jump becomes 2. Same memory, different walk. Reshape recomputes the stride and hands you a new view of the same block. That is the whole trick.

This works because the tensor is contiguous: reading memory front to back gives you the same order as reading the logical view front to back.

Reshape has two cases

Case one: contiguous. What we just saw. Recompute the stride, return a view, no copy, done.

Case two: non-contiguous. Transpose the original tensor. The logical view becomes:

But physical memory is unchanged:

The logical order is now 1 4 2 5 3 6 but memory holds 1 2 3 4 5 6. They disagree, so the tensor is non-contiguous:

x.T.is_contiguous()   # False

Calling .contiguous() copies the data into a fresh block laid out in logical order:

Now reshape can do its usual trick: recompute the stride on the fresh copy, no further cost.

Transpose only touches the stride

Transpose swaps rows and columns.

Nothing moves. Memory is still 1 2 3 4 5 6. Only the stride flips:

[3, 1]  →  [1, 3]

Before: jump 3 to move down a row, jump 1 to move right. After: jump 1 to move down a row, jump 3 to move right. The two values just swap, because the row and column dimensions themselves swapped.

x = torch.arange(1, 7).reshape(2, 3)

x.stride()            # (3, 1)
x.T.stride()          # (1, 3)
x.is_contiguous()     # True
x.T.is_contiguous()   # False

The logical order of the transposed tensor is 1 4 2 5 3 6, but memory order is still 1 2 3 4 5 6. The two no longer agree: this is the non-contiguous case. Calling .reshape() on x.T triggers a copy.

So what is view?

view does the same job as reshape: change the logical view of a block of memory. The difference is only in case two.

reshape will copy the memory if it has to. view will not; it refuses and raises an error. That is the entire distinction, and it explains a line you see all over attention implementations:

x = x.transpose(1, 2).contiguous().view(B, T, C)

After a transpose the tensor is non-contiguous, so view on its own would fail. .contiguous() makes the copy explicitly, and then view is safe. reshape would have done both in one call. Writing it out as contiguous plus view just makes the copy visible in the code, which is usually what you want when the copy is expensive.

The same idea in 3D

Nothing changes when you add dimensions. There are just more numbers in the stride.

Move one step along the first dimension and you jump 6 (a whole block). One step along the second jumps 3 (a row). One step along the third jumps 1 (a single element).

Conclusion

A tensor is a flat block of memory, plus a shape, plus a stride. reshape and view change the shape and recompute the stride. transpose changes only the stride. None of them touch the memory itself until there is no other option left. Knowing when that happens is most of what separates code that copies from code that doesn't.