skrape{it}
Search…
Overview
Why does skrape{it} provide its own http client implementations?
Skrape{it} offers an unified, intuitive and DSL-controlled way to make parsing of websites as comfortable as possible.
  • Http-Client DSL without verbosity and ceremony to make requests and corresponding request options like headers, cookies etc. in a fluent style interface.
  • Pre-configure a client once to either reuse it or adjust only the things that differ at certain requests - especially handy while working with authentication flows or custom headers.
  • Can handle client side rendered web pages (e.g. pages created with frameworks like React.js, Angular or Vue.js or pages manipulated with jQuery or other javascript)
A Http request is done as easy as in the given example. Just call the skrape function wherever you want in your code. It will force you to pass a fetcher and makes further request option available in the clojure.
1
skrape(HttpFetcher) { // <-- pass any Fetcher, e.g. HttpFetcher, BrowserFetcher, ...
2
request {
3
// ... request options goes here, e.g the most basic would be url
4
}
5
response {
6
// do stuff with the response like parsing the response body ...
7
}
8
}
Copied!
The http-request is only executed after either the response function has been called. This behaviour also allows to preconfigure the http-client and reusing request settings for multiple calls.
Last modified 4mo ago
Copy link