The response speed of the host gives a better ranking but not only this. You must check pageload at general.
For example: how many connections the browser have to make to fetch entire page content? Are you using CDN and which one? Do you have fallback safety for the CDN? Are you using gzip? etc.
SEO is a tricky business but we can reveal the secrets :yahoo:
We can make a basic howto and why if you want!