Much easier: Use a Hex-Viwer and look for the string "x264". In a x264-encoded stream, you will find that string, along with the version number and all options, in x264's custom SEI message ;) (Unless, of course, the stream was created with a modified x264 that doesn't insert the message -or- the message had been filtered out deliberately)
I've been using that method via MediaInfo but I heard that Vimeo and Youtube use modified x264 and It doesn't appeared to have SEI message And I found interesting post from Dark Shikari Maybe I can do something similar for x264 too?
I am not an expert in stochastic, but maybe something like a could be used to compare the quantizer values of a "sample" x264-encode with those from the "unknown" encode. If the test concludes that both come from the same probability distribution, it's likely to be an x264 encode. Otherwise it probably is not...
There are ways to distinguish x264 encoded videos without custom SEI (but I wouldn't disclose such details even though probably I don't know all of them). But if you upload short samples I can try to say was it encoded with x264 or not.
Here is the short sample clip from the Blu-ray disc I have. demux and cut using DGAVCindex but the second frame of the first GOP decoded in wrong way. (Maybe the Open-GOP issue I guess, could FFmpeg or Mplayer&Mencoder can directly cut stream by Keyframe withoout having a Open-GOP issue? If it could, it would be a great pleasure to know ) All that I know about distinguish x264-encoded stream is looking at the spatial quantizer distribution and find whether spatial adaptive quantization is used or not.' but I think all the grain in the movie make this kinda approach pointless, or maybe there can be another encoder with spatial adaptive quantziation feature. ( It won't be x264-only feature right? )
I have dozens of methods I could use to determine if an encoded stream was encoded by x264, in addition to the obvious. Like MasterNobody, I won't disclose them all.
I see that... But it's too sad that methods are kept secret. I just think it would be nice to know even a little part of it. ( somewhat relatively less important )