Examples of Operator Norms

For most of our applications, we will use one of three possible vector norms as already identified. The question that faces us is what are the compatible operator norms induced by these vector norms.We will answer the question once in detail and leave the other two for discussion later.

Let's begin with the 1-norm.

Theorem 1.2   (The Column-sum norm The operator norm induced by

$$\Vert \mathbf{x}\Vert _{1} = \sum_{j=1}^{n} \vert x_{j}\vert$$

is

$$\Vert \mathbf{A}\Vert _{1} = max_{1\leq j \leq n} \sum_{i=1}^{n} \vert a_{ij}\vert$$

Proof. Showing this requires two steps. First we must show that the supremum in the definition of the operator norm is bounded above and does exist. That is,

$$sup_{\mathbf{x}\neq \boldsymbol{0}} \frac {\Vert \mathbf{A}\mathbf{x}\Vert}{\Vert \mathbf{x}\Vert} \leq C$$

(Remember that it could easily be unbounded). Second we must show the supremum has the desired value

$$C = max_{1\leq j \leq n} \sum_{i=1}^{n} \vert a_{ij}\vert$$

To this end, let $C$ be defined as above. For $\mathbf{x}\neq \boldsymbol{0}$, we have

$$\Vert \mathbf{A}\mathbf{x}\Vert _{1} = \sum_{i=1}^{n} \vert \sum_... ...j} \vert \leq \sum_{i=1}^{n} \sum_{j=1}^{n} \vert a_{ij}\vert\vert x_{j} \vert$$

using the triangle inequality for absolute value. We can interchange the order of summation and use our definition of $C$ to obtain

$$\Vert \mathbf{A}\mathbf{x}\Vert _{1} = \leq \sum_{j=1}^{n} \vert... ...\vert \sum_{i=1}^{n} \vert a_{ij}\vert \leq C \sum_{j=1}^{n} \vert x_{j} \vert$$

Hence

$$\Vert\mathbf{A}\mathbf{x}\Vert _{1} \leq C \Vert \mathbf{x}\Vert _{1}$$

or

$$\frac{\Vert\mathbf{A}\mathbf{x}\Vert _{1} }{\Vert \mathbf{x}\Vert _{1}} \leq C$$

Since $\mathbf{x}$ was arbitrary,

$$sup_{\mathbf{x}\neq \boldsymbol{0}} \frac{\Vert\mathbf{A}\mathbf{x}\Vert _{1} }{\Vert \mathbf{x}\Vert _{1}} \leq C$$

Which shows the supremum is bounded above, which was our first part.

Now that we have this bound, what remains to be shown is that the is a vector $\hat{\mathbf{x}}$ such that

$$\frac {\Vert \mathbf{A}\hat{\mathbf{x}} \Vert _{1}}{\Vert \hat{ \mathbf{x}} \Vert _{1}} = C.$$

This will imply that $C$ is the maximum. (Remember that this bound will only allow $C$ to be strictly greater than the supremum or it is equal to it. If we show it is equal to it, then we have eliminated the strictly greater than case. Also note that there may be more than vector that achieve this maximum value. That doesn't matter. Only the value of the maximum matters.)

To find such a vector $\hat{\mathbf{x}}$, we note that there is a number $k$ such that

$$max_{1\leq j \leq n} \sum_{i=1}^{n} \vert a_{ij}\vert = \sum_{i=1}^{n} \vert a_{ik}\vert$$

Let $\mathbf{e}_{k}$ be the Euclidean basis vector. Let $\hat{\mathbf{x}} = \mathbf{e}_{k}$. Then

$$\frac {\Vert \mathbf{A}\hat{\mathbf{x}} \Vert _{1}}{\Vert \hat{ \mathbf{x}} \Vert _{1}} = \sum_{i=1}^{n} \vert a_{ik}\vert = C$$

And we are done. $\qedsymbol$

So the operator norm induced by the 1-norm is the maximum value of the sum of the absolute value of the entries in a column.

Let use examine the other induced operator norms.

Theorem 1.3   (The Row-sum norm The operator norm induced by

$$\Vert \mathbf{x}\Vert _{1} = max_{1 \leq j \leq n} \vert x_{j}\vert$$

is

$$\Vert \mathbf{A}\Vert _{1} = max_{1\leq i \leq n} \sum_{j=1}^{n} \vert a_{ij}\vert$$

Proof. Left to the reader. $\qedsymbol$

On the basis of these examples, you might guess that the operator norm induced by the standard Euclidean norm is

$$\Vert \mathbf{A}\Vert _{F} = \sqrt{ \sum_{i,j=1}^{n} a^{2}_{ij} }$$

This is called the Frobenius norm, and it is a matrix norm compatible with the Euclidean vector norm. However, it is not the operator norm induced by the Euclidean vector norm. The induced operator norm is expressed in terms of a maximum eigenvalue. We will not define it at this moment. It is primarily a theoretical tool and is not used in standard computation.