Implement your own

By providing different HTTP client implementations that already fulfill the need of having good old Http requests, imitating a browser by executing Javascript and doing async http calls you probably have special requirements that are hard to meet using the provided client implementations but on the other hand you still want to use the full skrape{it} DSL experience.
All the Fetchers are implementing either BlockingFetcher or a NonBlockingFetcher which are interfaces provided by the skrape{it} library. Both of them wants to have an implementation of a fetch method and requestBuilder.
In the example we implement a BlockingFetcher that can be passed to the skrape{it} DSL that is using Ktors different HttpClient implementations under the hood.
implement custom fetcher
1
class KtorBlockingFetcher(val ktorClient: HttpClient) : BlockingFetcher<HttpRequestBuilder> {
2
    
3
    override fun fetch(request: HttpRequestBuilder): Result = runBlocking { // runBlocking because ktor is fully build on coroutines but in our example we want it blocking
4
        with(ktorClient.request<HttpResponse>(request)) { // do http call with ktor client
5
            Result( // map ktor result to skrape{it} result
6
                responseBody = readText(),
7
                responseStatus = Result.Status(status.value, status.description),
8
                contentType = contentType()?.toString(),
9
                headers = headers.toMap().mapValues { it.value.firstOrNull().orEmpty() },
10
                baseUri = request.url.toString(),
11
                cookies = listOf(setCookie()) as List<Cookie>
12
            )
13
        }
14
    }
15
​
16
    override val requestBuilder: HttpRequestBuilder
17
        get() = HttpRequestBuilder()
18
}
Copied!
use custom fetcher
1
val KTOR_CLIENT = HttpClient(Apache)
2
​
3
val result = skrape(KtorBlockingFetcher(KTOR_CLIENT)) {
4
    request {
5
        url = "http://some.url"
6
    }
7
​
8
    response {
9
        // do your stuff here
10
    }
11
}
Copied!

AsyncFetcher

Next - Http Client

Request Options

Last modified 1mo ago

Copy link