Since I now figured out that my scope should actually measure 200MHz rather good, I now wanted to find out what the issue used to be.
See first post on this: http://everycircuit.com/circuit/5128890717306880
This was the result when I connected the ADF4351 directly to my scope with just a 50 ohm coax and used the scope on X1.
http://imgur.com/a/mh2FqcP
Now I removed all my cables at the ADF's output and just connected the probe, that was delivered with the scope, to the SMA connector and got this. I used it on X10. (The schematic in this post should match my probes specs)
http://imgur.com/a/scMqCzU
At 40MHz the amplitudes of both measuring methods match pretty well.
So I think I figured out the issue and I'm pretty happy with it since I wanted to buy proper probes anyway.
Maybe this post helps anyone to figure out such a problem too or find inspiration.
As always, if I'm doing any stupid mistakes plz let me know.
