Nowadays, mankind possesses a tremendous number of data, regarding many real-life phenomena. The availability of such a source of information is the main premise for the wide spread usage of the data-driven modelling. When account for more factors in the model development, this usually leads to a better ability of the model to represent the specifics of investigated system. Naturally, in fields like economics, society, medicine, etc. the models are with many inputs and many outputs (MIMO).
The linear models (w.r.t. their parameters) are preferable and applicable in many cases, because the theory used for their development is well known and is easy to apply. Also, the resulting models are easy to interpret. Furthermore, these models are proven as effective representations of systems with even strong nonlinear input-output behavior. The key point here is to transform the model into a linear parameterized form (if possible), by finding appropriate non-linear functions of the initial factors and/or the model outputs.
Why businesses would like to reduce the number of factors?
Usually, in practice, there are two type of requirements affecting the final model. They are: business requirements and statistical requirements (as the modelling is data-driven). We will not discuss the statistical aspects. An example of a business requirement is when impose restrictions on the model structure, due to economic reasons. For instance, some factors in credit scoring are provided on a corresponding price by credit bureaus. Normally these factors are significantly more discriminative and hence are more desirable as factors in the scoring models, compared with the other not so informative factors provided by the customers. When a financial organization estimates how risky an individual is w.r.t. a set of products (loans, credit cards, etc.), it may use a MIMO regression model. Normally the model accuracy increases sensibly when adding bureau characteristics as factors. But as they cost money, it is reasonable to reduce their number in the final model. So, if a bureau characteristic is entering in the model to predict a specific output (and the organization pays for it) then it is reasonable this factor to be used to predict the other outputs as well.
Another example from medicine is when predicting how likely a person is to have one or more diseases given a set of medical laboratory tests. Again as in the previous example the number of factors should be reduced. Otherwise, if the model requires too many tests, this would increase the usage of the laboratory consumables and it would be inconvenient for the patients.
How to represent the model in order to meet a business requirement?
There are two possible representations of linear MIMO models and one of them naturally accounts for the above-mentioned business requirement. Both representations are described below.
The output of a single input single output (SISO) linear model is a sum of multiplications between the model parameters and the entered factors (regressors). This sum can be presented as a multiplication of two vectors: the parameter vector and the regression vector. On the other hand the output of a MIMO model per every single observation is a vector but not a scalar. Then this output vector can be presented as a multiplication of a matrix and a vector. From this point of view there are two possible ways a linear MIMO regression model to be written. They are the parameter matrix (PM) and the parameter vector (PV) forms. In the first case, all parameters are arranged in a matrix and the factors are gathered in a vector, but in a PV the parameters are placed in a vector and the factors are arranged in a matrix (usually block diagonal).
At first sight, there is no principle difference between both representations. But keeping in mind that the model parameters are to be estimated both representations have different features. It is not difficult to aware that in the PM form each factor participates in the explanation of every single output. On the other hand in the PV form there is no such dependence: a factor explaining some output may not participate in the explanation of other outputs. Hence to account for the above mentioned businesses requirement, first the model should be developed in a PM form and after that a more precise modes structure refinement can be applied by switching to PV form.
Of course, if there is no business need to reduce the number of factors when the model is MIMO the PV form is the right model representation. The reason, as already mentioned is that each set of factors explaining a model output (which is nothing but a multiple input single output (MISO) sub-model) doesn’t depend on the factors participating in the other MISO sub-models. So, both possible representations of linear MIMO models are not equivalent and have their application areas.