Lies, damned lies and statistics

[Lliam]

I feel that SPCR is the more scientific of the hardware sites out there at the moment.

What made me think about this statistical analysis of reviews was the fact that coolers get top ratings based on a degree or two of temperature improvement over the competitors. (Granted ease of installation, size etc have a factor to play too) This is a fairly crude guide to aid heatsink purchasing but people buy (me included!) based on the best temps for the lowest airflow. Is spending £20 more on the newest creation for 4C difference worth it?

So when it comes to thinking about heatsink Y vs heatsink X should we be considering whether or not that difference is statistically significant ?

I understand there are many variables that go into this but the heatsinking prowess of a device could be measured on a standardised rig with various air flows. Statistical treatment would be applied and we scientifically cut through the marketing guff!

I suspect that this is probably a non issue for the majority of people (and I don't have a strong statistical mind) but I would be interested to know what SPCR members think.

regards

[Bluefront]

What you suggest as a standard test rig, would give results with more accuracy. But it's too much to expect every tester to have access to such a computer. You'll just have to keep judging published results with "a grain of salt".....continue to be a skeptic. If you keep reading similar results from different testers, you should be more convinced of the accuracy of the results. IMHO

[Dutchmm]

Actually, if you read the article here, you will see that there is a standard rig and methodology. So you should be able to get an objective view of the relative merits of a heatsink. That said, the ambient temperature in which we keep our machines, and our skill (or lack of it) with the thermal gunk mean that, as usual, YYMV.

I based my purchase choices on the reviews in SPCR, and I am very happy indeed with the results.

[Lliam]

Bluefront, I agree the standards would be difficult to make universal across multiple reviewers . However SPCR's heatsink test rig is our local standard so my point was that a quick "dash" of statistical significance with reviews at SPCR might improve the accuracy of our grain-o-salt-ometer! (mine needs frequent calibration)

Similar results across site for the same heatsink on similar cpus is hopefuly to be expected but when does the difference between X and Y become important? Without statistical treatment we won't know.
Is this1 degree difference important ?

As an example, there are numerous occasions of people adding fans to get the case temp down because of the worry that their HDD is rising above some "bad" temperature (usually the manufacturers upper limit). Google's scientific testing seems to disagree. See figure 4 and 5 of that pdf!
I'm not advocating running drives beyond spec but some experimentation and statistical analysis is probably the only way to gain meaningful information. You might be able to get rid of that extra fan or not buy it in the first place!

I think significance for us here is whether we can go fanless or reduce the noise of a system. So a clear statistical significance at SPCR should translate into a real world difference.

It is clear to me that
1. SPCR is consistent and therefore recommendations here carry weight (I'm with you Dutchmm on this one)
2. I have either such a quiet system I need to dream up posts like this or I'm getting deaf in my old age.

[nutball]

You can't tell people what they don't want to hear. People will spend an extra £20 to drop their temperatures by 4C, because that's what they think makes sense. Statistics are a distant second to "common sense", as the threads here dealing with Google HDD study demonstrate admirably.

[Bluefront]

Well I certainly agree that a more numbers-oriented test procedure, would lead to more scientific results.....but. In this field of noise, airflow, CFM, etc, much of the test results is difficult to translate into numbers. And should you actually attempt to do so, you'll end up with a boring read to most people. This whole website is filled with enough numbers to satisfy all but the most hard-core computer person.

There is always room for improvement here.....making the whole thing "user friendly" is the problem.

[djkest]

I think his point was, when two components perform very similarly, the difference is statistically inconclusive. But I hate statistics (as a discipline), so just go with it. They do their best to make sure things are accurate, and repeatable, and fair. Nothing is perfect, but you can still strive for excellence. Sure, if the difference is only 1C, thats no big deal. But I'm sure any decent review would state that. Usually differences are more pronounced though, like hard data that shows us how a Ninja does well in low/no airflow situations.

[jaganath]

Google's scientific testing seems to disagree

Statistics are a distant second to "common sense", as the threads here dealing with Google HDD study demonstrate admirably.

A LOT of people seem to have interpreted the Google study to mean that HDD temp doesn't affect MTTF. first of all, I don't think that's what it says, and secondly I have lost several drives to high (out of spec) temps, with the high temp being the only possible out of spec condition, so you'll exuse me if I listen to experience rather than one, unscientific (as in not all the parameters were controlled for) study.

as for the OP's original question, 1C, 4C difference won't make any difference for most people out there (in terms of useful lifespan of components). only overclockers and hardcore gamers need large, expensive heatsinks like the TRUE and even then, probably not; improving airflow in the case would probably give just as much temp reduction as a huge new heatsink.

[thejamppa]

Google nor Wiki are always right or give the absolute fact.

So I personally trust google studies as much as I trust wikipedia, which I trust almost none. I just need to read same thing from various sources. From both electrical and printed sources if possible.

Wiki's strength is also its weakness.

[mr. poopyhead]

Google nor Wiki are always right or give the absolute fact.

BLASPHEMY!!!! BURN HIM!

[thejamppa]

Google nor Wiki are always right or give the absolute fact.

BLASPHEMY!!!! BURN HIM!

Get in line! there are peoples before you who want either burn me or crucify me

[MikeC]

I've been the first to suggest that readers simply ignore small temperature differences reported in our testing. Not sure when or exactly where, but this has been discussed before here in the forums. There are too many variables in testing -- anyone's testing, not just ours.

1) sample variances -- everything from the pressure of the particular clip to the particular heatpipes in the sample.

2) secondary thermal pathways and other conditions in the test setup that may affect some heatsinks differently than others.

3) your setup with its particular conditions may also affect some heatsinks differently than others.

4) finally, the difference in our testing show up under extreme prolonged stress. Under more ordinary conditions, the difference between "the top ranked" HSF and a middling model may be that the former keeps the CPU more deeply in "safe temperature range". Paying for this is a matter of mind/attitude.

I know the question will be asked -- when I say "small temperature differences", just how small am I talking about? Maybe as large as 5C.

[peteamer]

BLASPHEMY!!!! BURN HIM!

...

Get in line! there are peoples before you who want either burn me or crucify me

Are still we using a 'first come, first served' basis?.....






Pete
P.S. My names Brian too...

[nutball]

secondly I have lost several drives to high (out of spec) temps, with the high temp being the only possible out of spec condition, so you'll exuse me if I listen to experience rather than one, unscientific (as in not all the parameters were controlled for) study.

Hold on. So a meta-analysis of several hundred thousand hard-drive failures is trumped by the failure of a couple of drives? By the way

You don't have to control all (or indeed any) of the parameters for the study to be scientific. All you need is good sampling of all parameters, and also a large sample size (sample size = uber alles). This is routine in science, and particularly important in the science where you have little or no influence over the input parameters.

Google nor Wiki are always right or give the absolute fact.

So I personally trust google studies as much as I trust wikipedia, which I trust almost none. I just need to read same thing from various sources. From both electrical and printed sources if possible.

Wiki's strength is also its weakness.

Google != Google in this context.

[nick705]

Hold on. So a meta-analysis of several hundred thousand hard-drive failures is trumped by the failure of a couple of drives? By the way

You don't have to control all (or indeed any) of the parameters for the study to be scientific. All you need is good sampling of all parameters, and also a large sample size (sample size = uber alles). This is routine in science, and particularly important in the science where you have little or no influence over the input parameters.

I don't think the Google data is disputed here, more a suspicion about the conclusions that are commonly drawn from it (heat has no bearing on drive longevity, or even "hotter is better"). Yes, there is a very large sample size, but also a paucity of information regarding the drives and their operating environments (as you say, you need good sampling of *all* relevant parameters).

Of all road accident fatalities occurring at around 8.30am (which occur at greater rate than at, say, 2.00am), a large proportion of victims will have cornflakes in their stomachs. That doesn't mean you can conclude that cornflakes are responsible for road accidents (correlation != causation)...

[Bluefront]

nick705...it's milk that's the problem. Exhaustive studies have concluded that almost everyone in road accidents drank milk as a child. The same people did those Google/hard drive studies.

[ame]

There was this guy who drowned in a pool of 2" avarage depth ....

I try to trust personal experiance of real people. Reading forums is more informative than reading reviews. Every time I skipped the forums and trusted the review only was a failue in terms of hardware choice.

[thejamppa]

There was this guy who drowned in a pool of 2" avarage depth ....

I try to trust personal experiance of real people. Reading forums is more informative than reading reviews. Every time I skipped the forums and trusted the review only was a failue in terms of hardware choice.

That is true. In tests most peoples use special set up or open test benches which are not same as everyday use. With users posting their experiences you can cover every angle. After all there is limited time in all test you can do for review.

P.S. you can drown in an inch of water if your nose and mouth are covered by it. Depth or amount of water is irrelevant aslong it prevents your breathing.