Why does skrape{it} provide its own http client implementations?

Skrape{it} offers an unified, intuitive and DSL-controlled way to make parsing of websites as comfortable as possible.

A Http request is done as easy as in the given example. Just call the skrape function wherever you want in your code. It will force you to pass a fetcher and makes further request option available in the clojure.

skrape(HttpFetcher) { // <-- pass any Fetcher, e.g. HttpFetcher, BrowserFetcher, ...
    request {
        // ... request options goes here, e.g the most basic would be url
    response {
        // do stuff with the response like parsing the response body ...

The http-request is only executed after either the response function has been called. This behaviour also allows to preconfigure the http-client and reusing request settings for multiple calls.

Last updated