My basic takeaway from the Fremer/JA review of the Playback Designs DAC was that the listening results were positive based on Framer’s listening without having prior knowledge of JA’s measurements. The measurements were bad enough (at least in JA’s opinion) that JA couldn’t recommend the Playback Designs unit. The manufacturer pointed out that JA’s testing was an inappropriate match for a DAC that handled high frequency noise shaping and filtering in the way the Playback DAC did.
For the MK2 review, listening and testing were both handled by JA, and he knew the test results before he listened. I find it extremely unlikely that his reaction to the sound of the MK2 wasn’t influenced by his knowledge of his measurements of high frequency noise (mostly in the ultrasonic range). Despite his unblinded listening sessions, JA still had many positive things to say about the MK2. In contrast to his earlier reaction to the Playback unit, which he could not recommend, he suggested that potential buyers of the MK2 should try it in their systems first to be sure that any ultrasonic noise didn’t cause any problems with associated equipment. Fortunately for potential buyers, PS Audio offers a 30-day home trial, so it’s easy to try the MK2 at home before finalizing a purchase.
Well, a few things.
Agree that Fremer reviewed without knowledge of measurements, and agree that JA mentioned that he measured the mk2 earlier than he typically does (does not say before he listened, however, as you contend). Did the measurements influence his subjective review? Maybe, but the only person who knows that is him. I don’t know, and you don’t either (to be fair).
Did JA recommend the mk2? I don’t think so. He had every opportunity to say that if he wanted to, but he didn’t. If you think that he did, please quote from the review where he says that he recommends the mk2. What he said, after saying he had concerns in the conclusion, was it requires audition BEFORE purchase.
I’ve already said I think the review is probably harsher than it could have been, and I wouldn’t blame PSA if they were upset with JA and Stereophile for the review he gave the mk2. Maybe he was having a crappy few weeks at the time, who knows? Hopefully he will do a follow-up mk2 review when things settle out - something he did with the original mk1 I think. I think that would be fair to expect that he should at least do that.
Ha, I will. I like HFN much better as a mag than Stereophile anyway. Seems so much more modern, and ironically, another of the main reasons I’ve tended to prefer HFN is their willingness to give poorer scores, haha, and it’s so easy to compare an actual score number where scores range within a 15-point window or so. Like almost everything about that mag better
I understand why so many people are upset by the Stereophile review. It upset me too, but mostly I was just perplexed. But, may I suggest those folks just follow my example and just go reread the HiFi News review. It will set your mine at ease. Their more balanced and enthusiastic review was what I was expecting from Stereophile. The noise issues were noted as a mere footnote by HiFi News, as it should be.
It’s a subjective assessment. I don’t think they ever specifically say how they get there, but if that’s wrong someone can correct me on that.
I get the sense that they definitely factor in the price of the component, so that if two components sound identical but are widely disparate in price, the less expensive one scores better. I say that because one of the amps that I have is a Cambridge Edge, and I recall reading a review of it years ago where they raved that it sounded just as good as a Hegel 590, but cost half as much, and gave it an 88% or 89%, where I think the Hegel got a lower score. Don’t quote me on that bc it was 5 years ago maybe.
But if I had to summarize, I’d say score = weighted assessment of overall sound quality where cost is a weighting factor. Whether measurements play a role, I’ve no idea, but I kinda doubt it, or at most very little.
Edit: Probably too simplistic. I’m sure build quality and user experience factors as well as sound quality, but all of them probably more than measurements would be my guess.
I want to provide just a correction and a couple of quotes from the actual reviews so that readers can evaluate what was actually said.
First, you state that JA is the only person who knows if his measurements influenced his subjective review. In fact, nobody knows this for sure, least of all JA. However, the probability that he was influenced by his prior knowledge of the measurements is very high. The reason that scientific research puts so much emphasis on blind or double-blind assessments and trials is because subjective bias plays such a significant role in human thinking and subjective reporting. It would be unusual for subjective bias not to be a factor here.
For those who haven’t read JA’s review of the MK2, here are his Conclusions: "Overall, I enjoyed the time I spent with the DirectStream MK2 processor and how it played music for me. While its low frequencies don’t have quite the drive I appreciate with the Benchmark and MBL processors, it betters its predecessor in this respect and sounds more open in the highs. However, I remain concerned about the relatively high level of noise in its output. This is a product would-be owners need to audition in their own systems before purchase.
Here are JA’s conclusions after his testing of a Playback Designs MPS-5 SACD player/DAC in 2010: “So while I was impressed by the player’s standard of construction, I can’t say the same about its technical performance. The relatively high level of background noise limits the MPS-5’s resolution with SACD and external 24-bit data to not much better than 16-bit CD. I am puzzled, therefore, why Michael Fremer liked the sound of this player so much. Perhaps his description of its sound being “analog-like” is a clue—for reasons that are not fully understood, a signal with very-low-level random noise added is sometimes preferred, on that it is more intelligible, to the same signal without such noise. But I feel that the MPS-5’s measured performance precludes an unreserved recommendation.”
Fair enough. They key difference though is that JA didn’t even listen to the Playback Designs, he simply measured it – no different from ASR – so if you’re an ASR-type I guess his measurement-only assessment there might appeal to you.
Fremer listened to the PD and emphatically recommended it in the actual review.
By contrast, JA listened to and measured the mk2, but I take your point that it is possible he might have been influenced by the measurements. On the other hand, he’s been doing this for 40 years and has a great reputation.
If you want to quote what JA thought of mk2 after listening, quote the end of the second to last paragraph before conclusion and note the word that jumped out at me when I read it, “unduly.”
Elk, I too looked at this and it is very confusing. Here is what I have very loosely discerned. Most gear gets an “Outstanding Product” designation with scores of 86 to 94, however, there are occasional exceptions where the score is lower. Lower scoring gear ( 80 - 85 is a guess ) is called “Highly Commended”. There is a third category that is used rarely. called “Editor’s Choice” that I have no date on.
As kzk has said, how they get the numeric score is hard to figure out.
I dislike assigning a numerical value to a subjective evaluation as it falsley suggests the evaluation is objective. This is especially true when the ratings fail within a narrow range.
It’s just levels of gradation. I like 15 levels (say, HFN’s effective range they score over from 80-95) better than 3 levels (Stereophile’s recommended components A, B, C) or 5 levels (HiFi+ 5*, 4.5*, 4*, 3.5*, 3* (nobody gets worse than 3*)), for example.
It’s roughly a Bell curve over those 15 levels so you get a pretty good idea what they thought of a component instantly from the score, on a scale with enough distinct tick marks where everyone isn’t getting the same two or three scores. But like anything else it’s just one opinion. Of course, they have plenty of commentary throughout the review that adds context to the score.
Edit, I guess that’s 16, ha. And the “16” levels make it actually a very wide range in comparison to others
It is pretty routine to take subjective rankings scored 0 to 10 across a variety of characteristics… like ten of them… and to add up to a score out of 100. Though normally in the pseudo-scientific (akabs) corporate world, one takes the views of many people, not just one.
Personally I prefer when scores go to 11, like Nigel Tufnel’s amps.
I am reminded of a farewell event for a unit LtCol at an entertainment establishment near the base (the Lariat Club?) where he evaluated a young lady’s performance with “On a scale of 1 to 10, she gets a 14!”
I presume the tested Mk2 was very well run in… as much as I discerned it was better than my Mk1 a few hours are initial play, at 500 hours it was yet another new DAC! Late nights again…
Which brings up an interesting point. Do the measurements change with burn in? If they don’t, then isn’t that proof that they are not as relevant or important as some people would like to think they are?
