Implement your own

By providing different HTTP client implementations that already fulfill the need of having good old Http requests, imitating a browser by executing Javascript and doing async http calls you probably have special requirements that are hard to meet using the provided client implementations but on the other hand you still want to use the full skrape{it} DSL experience.
All the Fetchers are implementing either BlockingFetcher or a NonBlockingFetcher which are interfaces provided by the skrape{it} library. Both of them wants to have an implementation of a fetch method and requestBuilder.
In the example we implement a BlockingFetcher that can be passed to the skrape{it} DSL that is using Ktors different HttpClient implementations under the hood.
implement custom fetcher
class KtorBlockingFetcher(val ktorClient: HttpClient) : BlockingFetcher<HttpRequestBuilder> {
    
    override fun fetch(request: HttpRequestBuilder): Result = runBlocking { // runBlocking because ktor is fully build on coroutines but in our example we want it blocking
        with(ktorClient.request<HttpResponse>(request)) { // do http call with ktor client
            Result( // map ktor result to skrape{it} result
                responseBody = readText(),
                responseStatus = Result.Status(status.value, status.description),
                contentType = contentType()?.toString(),
                headers = headers.toMap().mapValues { it.value.firstOrNull().orEmpty() },
                baseUri = request.url.toString(),
                cookies = listOf(setCookie()) as List<Cookie>
            )
        }
    }
​
    override val requestBuilder: HttpRequestBuilder
        get() = HttpRequestBuilder()
}
use custom fetcher
val KTOR_CLIENT = HttpClient(Apache)
​
val result = skrape(KtorBlockingFetcher(KTOR_CLIENT)) {
    request {
        url = "http://some.url"
    }
​
    response {
        // do your stuff here
    }
}

AsyncFetcher

Next - Http Client

Request Options

Last modified 1yr ago

Copy link