Why does skrape{it} provide its own http client implementations?
Skrape{it} offers an unified, intuitive and DSL-controlled way to make parsing of websites as comfortable as possible.
Http-Client DSL without verbosity and ceremony to make requests and corresponding request options like headers, cookies etc. in a fluent style interface.
Pre-configure a client once to either reuse it or adjust only the things that differ at certain requests - especially handy while working with authentication flows or custom headers.
Can handle client side rendered web pages (e.g. pages created with frameworks like React.js, Angular or Vue.js or pages manipulated with jQuery or other javascript)
A Http request is done as easy as in the given example. Just call the skrape function wherever you want in your code. It will force you to pass a fetcher and makes further request option available in the clojure.
1
skrape(HttpFetcher){// <-- pass any Fetcher, e.g. HttpFetcher, BrowserFetcher, ...
2
request {
3
// ... request options goes here, e.g the most basic would be url
4
}
5
response {
6
// do stuff with the response like parsing the response body ...
7
}
8
}
Copied!
The http-request is only executed after either the response function has been called. This behaviour also allows to preconfigure the http-client and reusing request settings for multiple calls.