Signal Test

[stevensegal]

On my speaker supplier forum the focus is on objective performance and we are currently discussing if there is any significant difference between DACs.

Some surprising results have arisen with a basic 1Khz peak signal applied to some very cheap and very expensive DACs. (On that forum brand names are rarely mentioned, so cannot say what they were.) However, the cheap DAC performed very well and the expensive one very badly.

Another member posted a link to a site that runs tests on a lot of equipment and the 1Hkz test for the PS Audio DAC Mk2 were, frankly, shockingly bad. On some others the 2nd, 3rd etc harmonics were unnoticeable.

Is there an explanation for these poor tests results?

Are these tests carried out by PS Audio? After all, these are tests that amateurs elswhere are carrying out at home.

[stevensegal]

Chart now provided (with file size reduced)

[Elk]

Welcome!

While I enjoy that enthusiasts run such tests and are actively engaged, I always review these with a great deal of skepticism. It is much more difficult to properly measure audio equipment than many understand. This is one reason the results vary so greatly from one enthusiast’s findings to another. For that matter, the results vary between professional knowledgeable reviewers as well as manufacturers.

The same issue exists with enthusiasts attempting to judge the sound and resolution of recordings by reviewing spectra.

This is not to say measurements are not valuable. They are and I love specs and tests. But I also do not listen with my eyes.

[lonelyraven]

I can’t tell from that postage stamp sized chart, but if these are measurements with the older firmware, they should be redone with current firmware/OS. I’m betting the results are different since Ted has improved the noise issue that was in the earlier firmware.

[stevensegal]

Actually, some of the tests were carried out by one of the most respected names in UK audio who has been running a successful company exporting the world over for some 30 years.

My experience of testing harmonics in the field was on heavy machinery on oil platforms in the North Sea as resonant harmonics from different machinery could (and did) cause them to explode and kill people.

Obviously audio harmonics are a far lesser threat to life and limb, but it is a pretty basic test to input a specific frequency or range of frequencies and test whether the same ones, or additional ones, come out of the other end.

I tend not to read reviews as they are usually more akin to romantic fiction designed to sell magazines rather than inform on audio equipment. I have read one music magazine (Gramophone) for over 30 years that has been reviewing audio equipment since about 1925 (it was first published in 1923) and until the early 1980s reviews of audio equipment rarely if ever involved listening to music. For 60 years reviews were mainly involved in technical analysis of frequency response and distortion.

Can anyone at PS Audio advise if PS Audio have published a set of objective test results of any of the current DACs or, if not, if such data is available.

[stevensegal]

Lonely Raven said I can't tell from that postage stamp sized chart, but if these are measurements with the older firmware, they should be redone with current firmware/OS. I'm betting the results are different since Ted has improved the noise issue that was in the earlier firmware.

The chart clearly enough shows the 2, 3 and 4th harmonics. It is rather worrying that equipment should be released with this profile.

Can you advise when the latest DAC Mk2 software was released?

As per my longer post, I would like to see PS Audio’s own 1khz test result and any other official tests that it has published.

Many thanks

[stevensegal]

The chart is small as the file size limit on this forum is small. It is so small I posted the 20hkz test. Here is the 1khz result.

[tedsmith]

You confused maters a little by posting about the PerfectWave DAC MkII in the PerfectWave DirectStream forum.

Still I wonder about objectivity given the characterization of 0.01% THD being “shockingly bad” and the chart being labeled “FAIL”

[stevensegal]

“shockingly” was used as a relative term. As mentioned, some DACs displayed no noticeable harmonics. The more recent DAC also had a similar result.

For what it’s worth, I’ve been happily using this Mk2 DAC for 18 months or so and also have a P3.

I was after official objective test measurements of frequency response (and anything else that has been released) and was enquiring if PS Audio have published them.

The reason why we were looking into this elsewhere is because hardly any manufacturers seem to publish test results any more, so how do you measure one machine against another? (Listening is not the answer, there is no method in that and it is entirely subjective and biased.)

In the old days manufacturers used to publish their data and reviewers used to check it. When the Quad 33/303 was released and reviewed in February 1978 the reviewer admitted his test equipment was not sensitive enough to test the manufacturer’s claimed THD etc… So he went out and bought much better test equipment and found the claims correct. He then pronounced it a classic and was proven correct - still much in use 35 years later. He didn’t have to plug it into an audio system and play music on it.

So the reason for asking is so as not to have to rely on third party test measurements.

So, is there any PS Audio test data?

Many thanks again.

[Elk]

[Moved from DS forum to the PWD DAC forum]

stevensegal said The reason why we were looking into this elsewhere is because hardly any manufacturers seem to publish test results any more, so how do you measure one machine against another? (Listening is not the answer, there is no method in that and it is entirely subjective and biased.)

You mean subjective like the entire activity of listening to music? :)

Do not worry, we have no need to argue/discuss yet again objective v. subjective. I will note however I am comfortable knowing no one can hear the difference between 0.01% THD and 0.005% THD in a DAC. And fully cognizant of what happens when one becomes obsessed with specs over sound.

Manufactures have indeed become reticent in publishing specs since the THD wars of the 1970’s. I suspect this is because enthusiasts have learned the pitfalls of relying on specs alone, and so much equipment now appears very similar when measured even though it does not sound the same. But there can be differences and I appreciate that Stereophile and others continue to measure.

Actually, some of the tests were carried out by one of the most respected names in UK audio who has been running a successful company exporting the world over for some 30 years.

Who performed the test you posted? I am not claiming it is suspect in any way, just curious as to its specific pedigree. It does not present as a professional test (e.g., I have yet to see a professional reviewer label a result "fail.")

. . . it is a pretty basic test to input a specific frequency or range of frequencies and test whether the same ones, or additional ones, come out of the other end

Very true as far as it goes in running the equipment - very easy. The challenge is making certain the input is correct, with appropriate attention to grounding, etc. Many reviewers who measure have described errors which they have made, challenges faced, followed-up with revised measurements, etc.

But again, measurements are great.

As to your question, I have not personally seen such measurements from PS Audio. Paul will likely be along soon and is in much better position to comment.

[stevensegal]

Thanks for that.

Trust me the professional manufacturer/tester knows what he’s doing. The data I provided was from a third party site. Whilst they do a lot of comprehensive testing, I cannot vouch for it and hence I did not want to draw conclusions without PS Audio’s own views on the matter and its own test results.

I have to dash (it’s just gone 6pm in the UK) to do some very subjective testing this evening, at the ballet, I’ll follow up later.

I also seem to be about 17 software updates out of date.

[stevensegal]

Well the ballet was OK, the orchestra were better, and I am none the wiser.

I could summarise Elk as follows:

1. Absolute faithful reproduction of the source is not of primary concern as listening is subjective

2. Objective measurement no longer matters, it is a redundant exercise as THD is invariably below audible levels

3. If you want a measurement, read a review

4. Most equipment has measured much the same for decades, so the only differences are from subjective listening.

I would tend to agree with 2 and 4.

So why do DACs sound different? Why does my $4,000 PWD Mk2 sound better than my $300 Tascam CD player?

Incidentally, my set-up is a high quality fanless NAS/ripper/server ($2,500) via usb, a Tascam CD200 as a transport (£300) and an Airport Express ($130) for Qobuz CD quality streaming ($30 per month). Strange as it may seem, I run the transport and AE through a Remedy reclocker ($500) via spdif with NativeX switched off. It sounds marginally better.

The simple question is this - if my $4,000 DAC Mk2 sounds better than my $300 Tascam CD/DAC, is there any objective way of measuring why?

If the DAC design is subjective (which may be due to software processing), in the way that a speaker manufacturer voices speakers, what are the parameters for such processing?

I may seem to be a bit blunt, but the Socratic search for truth through reason is not an emotional matter and takes no prisoners.

[tedsmith]

“I may seem to be a bit blunt, but the Socratic search for truth through reason is not an emotional matter and takes no prisoners.”

Yes, but most can’t figure out and understand their own presuppositions and hence all following logic is based on sand and often worthless.

I think I understand a lot of what it takes and hence I built the DirectStream, I chose each part based on the datasheets and tested my understanding with spice simulations. I’m gratified to see that doing the best I could from a theoretical perspective sounds the best and, for example, that I didn’t have to do extensive listening for each part to pick it. This is in opposition to your points 1, 2 and 4. And, yes, the DirectStream doesn’t measure as well in some traditional measurements as most other DACs: the obvious conclusion is that those things don’t reflect what the ear/brain cares about - still I got there by understanding and the scientific method not voodoo. The difference is that I don’t feel a need to justify my experiments and my conclusions to others, instead I used them to build a good sounding DAC.

If you’d like to see some of what I believe matters you can watch the videos at the PS Audio web site re the DirectStream.

I warn you ahead of time that many reject out of hand some of my hypotheses, but I believe the proof is in the pudding.

Jitter can be measured, power supply firmness can be measured, linearity can be measured, EMI, RFI, etc. can be measured. One problem is that many don’t believe these things matter as much as they do and another problem is that we don’t have good ideas of how to measure what the ear/brain cares about for good sound.

I did the best I could in every dimension that I could attack and left the rest to software (which I’m pretty good at) so that I could continue working on the DAC after it was released.

[tedsmith]

And for that matter I have picked essentially every piece of my system based on what I consider to be solid principles based on the published info about them. Of course I listened as well, but if I didn’t have an understanding of why a component should sound good I didn’t waste time on that component and further I don’t chase the best measurements, I chase the best design principles.

[tony22]

+1000 Ted! Music itself can be considered a combination of science and emotion. No reason why musical reproduction should be any different.

I too search for pieces in my system that are based on solid design principles, but I wouldn’t keep them in my system if they didn’t provide a sense of realism AND a connection to the soul.

[tedsmith]

Yep, if the feet ain’t a tappin’ who cares what the mouth is a yappin’

[stevensegal]

Thanks for those responses, much appreciated. We have the same discussions, often much more vigorously, over amplifiers, where on one side of the fence there is the argument that they are blunt instruments that are simply meant to take a signal at one voltage and output the same signal at another voltage. On the other side of the fence there are those, like you, who design on the basis of individual component performance and then sit and listen to the end result.

I am in the latter camp in respect of amplifiers. I currently use a Class A SET amp that lends itself to my tastes in music. Due to a slight lack of power needed to control my speakers properly, I am getting a more powerful pair of amplifiers modified with particular components for a particular presentation. Some would consider such an approach has no merit whatsoever, the amplifier has no right to colour or influence the music, but there is a important principle at stake.

The main issue is how much one really has to spend to get faithful music reproduction, and how can the consumer identify it. People may prefer a certain balance, and it may also be influenced by what they can actually hear. On the amplifier side, the Croft integrated amplifier is hugely popular over here and in the US, is a Stereophile award winner and doesn’t actually measure very well (http://www.stereophile.com/how-is-ted-coding-the-fpgaent/croft-acoustics-phono-integrated-integrated-amplifier-measurements_).

The problem with DACs is that they have a pretty simple task to carry out, converting a digital signal to an analogue one. Besides some surprising measurements, some of our members did a blind test of $100 and $2,000 DACs and couldn’t tell the difference.

Moreover, how many people listen to live unamplified sound on a regular basis and even have a valid reference point?

We have a fair number of UK manufacturers who do not go to UK consumer audio shows because they have no need. They have a full order book. They don’t even need to advertise. Some don’t even have a website. My speaker manufacturer only does the main UK show each year as a matter of customer goodwill. They have a full order book. This year they had on demonstration the speakers I happen to use, recently improved (they’ve been making them for 40 years), cost about $5,000. The source equipment was a laptop, a DAC and amplifier, total cost less than $1,000. Several commentators considered them best in show. It makes you wonder.

As I said earlier, I happily listen to my DAC Mk2 for hours on end, so thanks for that. My enquiry was aimed at understanding the principles behind its design, as well whether or not you have official test data.

[tony22]

Moreover, how many people listen to live unamplified sound on a regular basis and even have a valid reference point?

For some reason every time this question comes up it really annoys me (my comments are not directed at you, stevensegal). There seems to me an implicit undercurrent that the safe assumption to make is that the person on the other side of a discussion probably doesn't actually listen to live acoustic music. I sometimes also see that people who have opinions slightly different from the masses are almost always asked this question. I will admit it's probably a reasonable thing to wonder, but it still bothers me, probably because it doesn't apply to me.

I’ll go put the soapbox back in its corner.

[stevensegal]

tony22 said

Moreover, how many people listen to live unamplified sound on a regular basis and even have a valid reference point?

For some reason every time this question comes up it really annoys me (my comments are not directed at you, stevensegal). There seems to me an implicit undercurrent that the safe assumption to make is that the person on the other side of a discussion probably doesn’t actually listen to live acoustic music. I sometimes also see that people who have opinions slightly different from the masses are almost always asked this question. I will admit it’s probably a reasonable thing to wonder, but it still bothers me, probably because it doesn’t apply to me.

I’ll go put the soapbox back in its corner.

The purists will say that distortion starts the moment you sing/play into a microphone, but there is no doubt that many, probably most audio systems are overly bright if not bass heavy compared to the real thing. Why? Because it sells. I have a very good test - overly bright music makes my ears ring, which I never get at live performance.

It may be old-fashioned, but I retain the philosophy that hi-fidelity audio has the sole objective to reproduce as faithfully as possible live music. That may go out the window if you listen to Morton Subdodnik, but there is much audio that produces pleasant sounds, but it is not hi-fidelity.

You then have to deal with music memory. I went to a show last year (Emily Barker), heard most of the album live in a superb concert hall, then went home and listened to bits of it on Pallas vinyl, 24/192 and 16/44. I found my brain could not remember and contrast so much information, even within the space of 3 or so hours.

On the harmonics point, elsewhere the tester took the 1khz tone and its harmonics as measured, then reproduced the fundamental and the harmonics and posted it against the original signal. The harmonics were audible. It’s a rather neat objective listening test for any piece of kit as to whether a problem exists or not. If you can hear harmonics, it can’t be doing the music any good.

[tedsmith]

stevensegal said The problem with DACs is that they have a pretty simple task to carry out, converting a digital signal to an analogue one. Besides some surprising measurements, some of our members did a blind test of $100 and $2,000 DACs and couldn't tell the difference.

Moreover, how many people listen to live unamplified sound on a regular basis and even have a valid reference point?

Just in case it’s ambiguous: Tho I did some consulting about jitter reduction, the PWD MkII is PS Audio’s design.

I certainly listen to unamplified sound regularly, as do many on this forum. And FWIW I played (tho certainly not expertly) the piano, organ, trombone, clarinet, tenor sax and sang in various choirs. Like Tony I bristle when people assume that others don’t listen to live music.

I’ve noticed many people seem to feel that any job they haven’t done themselves is simple but conversely the jobs they do take a lot of skill Why is the job of amplification, speakers, a preamp or a turn table any more complex than the job of a DAC (or vice versa)? IMO most DACs sound similar because most of them use the same component DAC chips and the big differences are the designer’s analog abilities re, the power supply the output stage, etc. Not to short shrift those items, they certainly are very important, but there’s a lot of performance left on the floor when jitter isn’t taken seriously and when off the shelf DAC chips are used assuming that they have a simple job to do…

Anyway, back to our regularly scheduled program.