skrape{it}
1.1.X
1.1.X
  • Introduction
  • Why it exists
  • overview
    • Setup
    • Who should be using it
  • Http Client
    • Overview
    • Fetchers
      • HttpFetcher
      • BrowserFetcher
      • AsyncFetcher
      • Implement your own
    • Request Options
    • Pre-configure client
    • Response
      • Status
      • Cookies
  • Html Parser
    • Parsing HTML
  • assertions
    • expect content
  • How to Use
    • Testing
    • Scraping
    • JS-rendered sites
  • Examples
    • Grab all links from a Website
    • Creating a RESTful API (Spring-Boot)
  • GitHub Repo
  • Extensions
    • MockMvc
      • Getting Started
      • GitHub Repo
    • Ktor
      • Getting Started
      • GitHub Repo
  • About skrape{it}
Powered by GitBook
On this page

Was this helpful?

  1. Http Client
  2. Fetchers

Implement your own

PreviousAsyncFetcherNextRequest Options

Last updated 3 years ago

Was this helpful?

By providing different HTTP client implementations that already fulfill the need of , and you probably have special requirements that are hard to meet using the provided client implementations but on the other hand you still want to use the full skrape{it} DSL experience.

All the Fetchers are implementing either which are interfaces provided by the skrape{it} library. Both of them wants to have an implementation of a fetch method and requestBuilder.

In the example we implement a BlockingFetcher that can be passed to the skrape{it} DSL that is using Ktors different HttpClient implementations under the hood.

implement custom fetcher
class KtorBlockingFetcher(val ktorClient: HttpClient) : BlockingFetcher<HttpRequestBuilder> {
    
    override fun fetch(request: HttpRequestBuilder): Result = runBlocking { // runBlocking because ktor is fully build on coroutines but in our example we want it blocking
        with(ktorClient.request<HttpResponse>(request)) { // do http call with ktor client
            Result( // map ktor result to skrape{it} result
                responseBody = readText(),
                responseStatus = Result.Status(status.value, status.description),
                contentType = contentType()?.toString(),
                headers = headers.toMap().mapValues { it.value.firstOrNull().orEmpty() },
                baseUri = request.url.toString(),
                cookies = listOf(setCookie()) as List<Cookie>
            )
        }
    }

    override val requestBuilder: HttpRequestBuilder
        get() = HttpRequestBuilder()
}
use custom fetcher
val KTOR_CLIENT = HttpClient(Apache)

val result = skrape(KtorBlockingFetcher(KTOR_CLIENT)) {
    request {
        url = "http://some.url"
    }

    response {
        // do your stuff here
    }
}
having good old Http requests
imitating a browser by executing Javascript
doing async http calls
BlockingFetcher or a NonBlockingFetcher