Comment on page
Implement your own
By providing different HTTP client implementations that already fulfill the need of having good old Http requests, imitating a browser by executing Javascript and doing async http calls you probably have special requirements that are hard to meet using the provided client implementations but on the other hand you still want to use the full skrape{it} DSL experience.
All the Fetchers are implementing either BlockingFetcher or a NonBlockingFetcher which are interfaces provided by the skrape{it} library. Both of them wants to have an implementation of a fetch method and requestBuilder.
In the example we implement a BlockingFetcher that can be passed to the skrape{it} DSL that is using Ktors different HttpClient implementations under the hood.
implement custom fetcher
class KtorBlockingFetcher(val ktorClient: HttpClient) : BlockingFetcher<HttpRequestBuilder> {
override fun fetch(request: HttpRequestBuilder): Result = runBlocking { // runBlocking because ktor is fully build on coroutines but in our example we want it blocking
with(ktorClient.request<HttpResponse>(request)) { // do http call with ktor client
Result( // map ktor result to skrape{it} result
responseBody = readText(),
responseStatus = Result.Status(status.value, status.description),
contentType = contentType()?.toString(),
headers = headers.toMap().mapValues { it.value.firstOrNull().orEmpty() },
baseUri = request.url.toString(),
cookies = listOf(setCookie()) as List<Cookie>
)
}
}
override val requestBuilder: HttpRequestBuilder
get() = HttpRequestBuilder()
}
use custom fetcher
val KTOR_CLIENT = HttpClient(Apache)
val result = skrape(KtorBlockingFetcher(KTOR_CLIENT)) {
request {
url = "http://some.url"
}
response {
// do your stuff here
}
}
Last modified 2yr ago