Examples with HttpServer - com.github.dreamhead.moco.HttpServer

Example 1 with HttpServer

use of com.github.dreamhead.moco.HttpServer in project webmagic by code4craft.

the class HttpClientDownloaderTest method test_set_site_header.

@Test
public void test_set_site_header() throws Exception {
    HttpServer server = httpServer(13423);
    server.get(eq(header("header"), "header-webmagic")).response("ok");
    Runner.running(server, new Runnable() {

        @Override
        public void run() throws Exception {
            HttpClientDownloader httpClientDownloader = new HttpClientDownloader();
            Request request = new Request();
            request.setUrl("http://127.0.0.1:13423");
            Page page = httpClientDownloader.download(request, Site.me().addHeader("header", "header-webmagic").toTask());
            assertThat(page.getRawText()).isEqualTo("ok");
        }
    });
}

Also used : Runnable(com.github.dreamhead.moco.Runnable) HttpServer(com.github.dreamhead.moco.HttpServer) HttpUriRequest(org.apache.http.client.methods.HttpUriRequest) Request(us.codecraft.webmagic.Request) Page(us.codecraft.webmagic.Page) IOException(java.io.IOException) UnsupportedEncodingException(java.io.UnsupportedEncodingException) Test(org.junit.Test)

Example 2 with HttpServer

use of com.github.dreamhead.moco.HttpServer in project webmagic by code4craft.

the class HttpClientDownloaderTest method test_download_auth_by_SimpleProxyProvider.

@Test
public void test_download_auth_by_SimpleProxyProvider() throws Exception {
    HttpServer server = httpServer(13423);
    server.get(eq(header("Proxy-Authorization"), "Basic dXNlcm5hbWU6cGFzc3dvcmQ=")).response("ok");
    Runner.running(server, new Runnable() {

        @Override
        public void run() throws Exception {
            HttpClientDownloader httpClientDownloader = new HttpClientDownloader();
            httpClientDownloader.setProxyProvider(SimpleProxyProvider.from(new Proxy("127.0.0.1", 13423, "username", "password")));
            Request request = new Request();
            request.setUrl("http://www.baidu.com");
            Page page = httpClientDownloader.download(request, Site.me().toTask());
            assertThat(page.getRawText()).isEqualTo("ok");
        }
    });
}

Also used : Proxy(us.codecraft.webmagic.proxy.Proxy) Runnable(com.github.dreamhead.moco.Runnable) HttpServer(com.github.dreamhead.moco.HttpServer) HttpUriRequest(org.apache.http.client.methods.HttpUriRequest) Request(us.codecraft.webmagic.Request) Page(us.codecraft.webmagic.Page) IOException(java.io.IOException) UnsupportedEncodingException(java.io.UnsupportedEncodingException) Test(org.junit.Test)

Example 3 with HttpServer

use of com.github.dreamhead.moco.HttpServer in project webmagic by code4craft.

the class HttpClientDownloaderTest method testGetHtmlCharset.

@Test
public void testGetHtmlCharset() throws Exception {
    HttpServer server = httpServer(13423);
    server.get(by(uri("/header"))).response(header("Content-Type", "text/html; charset=gbk"));
    server.get(by(uri("/meta4"))).response(with(text("<html>\n" + "  <head>\n" + "    <meta charset='gbk'/>\n" + "  </head>\n" + "  <body></body>\n" + "</html>")), header("Content-Type", "text/html; charset=gbk"));
    server.get(by(uri("/meta5"))).response(with(text("<html>\n" + "  <head>\n" + "    <meta http-equiv=\"Content-Type\" content=\"text/html; charset=gbk\" />\n" + "  </head>\n" + "  <body></body>\n" + "</html>")), header("Content-Type", "text/html"));
    Runner.running(server, new Runnable() {

        @Override
        public void run() {
            String charset = getCharsetByUrl("http://127.0.0.1:13423/header");
            assertEquals(charset, "gbk");
            charset = getCharsetByUrl("http://127.0.0.1:13423/meta4");
            assertEquals(charset, "gbk");
            charset = getCharsetByUrl("http://127.0.0.1:13423/meta5");
            assertEquals(charset, "gbk");
        }

        private String getCharsetByUrl(String url) {
            HttpClientDownloader downloader = new HttpClientDownloader();
            Site site = Site.me();
            CloseableHttpClient httpClient = new HttpClientGenerator().getClient(site);
            // encoding in http header Content-Type
            Request requestGBK = new Request(url);
            CloseableHttpResponse httpResponse = null;
            try {
                httpResponse = httpClient.execute(new HttpUriRequestConverter().convert(requestGBK, site, null).getHttpUriRequest());
            } catch (IOException e) {
                e.printStackTrace();
            }
            String charset = null;
            try {
                byte[] contentBytes = IOUtils.toByteArray(httpResponse.getEntity().getContent());
                charset = CharsetUtils.detectCharset(httpResponse.getEntity().getContentType().getValue(), contentBytes);
            } catch (IOException e) {
                e.printStackTrace();
            }
            return charset;
        }
    });
}

Also used : Site(us.codecraft.webmagic.Site) CloseableHttpClient(org.apache.http.impl.client.CloseableHttpClient) Runnable(com.github.dreamhead.moco.Runnable) HttpServer(com.github.dreamhead.moco.HttpServer) HttpUriRequest(org.apache.http.client.methods.HttpUriRequest) Request(us.codecraft.webmagic.Request) CloseableHttpResponse(org.apache.http.client.methods.CloseableHttpResponse) IOException(java.io.IOException) Test(org.junit.Test)

Example 4 with HttpServer

use of com.github.dreamhead.moco.HttpServer in project webmagic by code4craft.

the class HttpClientDownloaderTest method test_set_request_header.

@Test
public void test_set_request_header() throws Exception {
    HttpServer server = httpServer(13423);
    server.get(eq(header("header"), "header-webmagic")).response("ok");
    Runner.running(server, new Runnable() {

        @Override
        public void run() throws Exception {
            HttpClientDownloader httpClientDownloader = new HttpClientDownloader();
            Request request = new Request();
            request.setUrl("http://127.0.0.1:13423");
            request.addHeader("header", "header-webmagic");
            Page page = httpClientDownloader.download(request, Site.me().toTask());
            assertThat(page.getRawText()).isEqualTo("ok");
        }
    });
}

Example 5 with HttpServer

use of com.github.dreamhead.moco.HttpServer in project webmagic by code4craft.

the class HttpClientDownloaderTest method test_download_when_task_is_null.

@Test
public void test_download_when_task_is_null() throws Exception {
    HttpServer server = httpServer(13423);
    server.response("foo");
    Runner.running(server, new Runnable() {

        @Override
        public void run() throws Exception {
            final HttpClientDownloader httpClientDownloader = new HttpClientDownloader();
            Request request = new Request();
            request.setUrl("http://127.0.0.1:13423/");
            Page page = httpClientDownloader.download(request, Site.me().toTask());
            assertThat(page.getRawText()).isEqualTo("foo");
        }
    });
}

Aggregations

HttpServer (com.github.dreamhead.moco.HttpServer)27 Test (org.junit.Test)25 AbstractMocoHttpTest (com.github.dreamhead.moco.AbstractMocoHttpTest)13 Runnable (com.github.dreamhead.moco.Runnable)12 IOException (java.io.IOException)12 HttpUriRequest (org.apache.http.client.methods.HttpUriRequest)12 Request (us.codecraft.webmagic.Request)12 UnsupportedEncodingException (java.io.UnsupportedEncodingException)11 Page (us.codecraft.webmagic.Page)10 Site (us.codecraft.webmagic.Site)3 ActualHttpServer (com.github.dreamhead.moco.internal.ActualHttpServer)2 RunnerSetting.aRunnerSetting (com.github.dreamhead.moco.runner.RunnerSetting.aRunnerSetting)1 Map (java.util.Map)1 HashedMap (org.apache.commons.collections.map.HashedMap)1 CloseableHttpResponse (org.apache.http.client.methods.CloseableHttpResponse)1 CloseableHttpClient (org.apache.http.impl.client.CloseableHttpClient)1 Proxy (us.codecraft.webmagic.proxy.Proxy)1