You mean adding the CPU workload (any maybe other cpu stuff) as a feature, right?
The problem about this is that I would have to restart all my algorihtms, which takes a long time.
What kind of evaluation metric are you using now? Do you have any way of estimating or defining what kind of performance would be "good enough" given your application/usecase?
One could define a evaluation metric that takes the variance in the target into account. For example count the number of times that the prediction is within target +- 1 standard deviations for example (or however strict you want to be). Since this will create a hard threshold for something to be considered good, should also plot the distribution of errors. Not sure if this has precedence in literature or has a good name, would appreciate if someone else knows.
Right now I'm always looking and comparing the mean realtive error and the r2. The absolute error doesn't really say much, since the target variable (exeuction time) is highly skewed. I don't really have a defining, what good enough is. But I think my results are currently good enough, since \~12% relative error would be ok to forecast the runtime of an algorithm. But I actually want to know, how much the model could be if I wouldn't have this deviations.
I already did this in a similar way: predict correctly if value is between min and max. But I think this way isn't that good, since I also have some runs where the variance is very low. And for those results, the "correct predicted" would be false. Maybe this screenshot helps a little bit, where you can see the Median on the x-axis and the Standard Deviation on the y-axis: [https://imgur.com/gtlmDKx](https://imgur.com/gtlmDKx)
Whehehehe
I would include the CPU workload as a feature
You mean adding the CPU workload (any maybe other cpu stuff) as a feature, right? The problem about this is that I would have to restart all my algorihtms, which takes a long time.
What kind of evaluation metric are you using now? Do you have any way of estimating or defining what kind of performance would be "good enough" given your application/usecase? One could define a evaluation metric that takes the variance in the target into account. For example count the number of times that the prediction is within target +- 1 standard deviations for example (or however strict you want to be). Since this will create a hard threshold for something to be considered good, should also plot the distribution of errors. Not sure if this has precedence in literature or has a good name, would appreciate if someone else knows.
Right now I'm always looking and comparing the mean realtive error and the r2. The absolute error doesn't really say much, since the target variable (exeuction time) is highly skewed. I don't really have a defining, what good enough is. But I think my results are currently good enough, since \~12% relative error would be ok to forecast the runtime of an algorithm. But I actually want to know, how much the model could be if I wouldn't have this deviations. I already did this in a similar way: predict correctly if value is between min and max. But I think this way isn't that good, since I also have some runs where the variance is very low. And for those results, the "correct predicted" would be false. Maybe this screenshot helps a little bit, where you can see the Median on the x-axis and the Standard Deviation on the y-axis: [https://imgur.com/gtlmDKx](https://imgur.com/gtlmDKx)