skrape{it}
Search…
How to Use
People find DSLs valuable because a well-designed DSL can be much easier to program with than a traditional library. This improves programmer productivity, which is always valuable.
Martin Fowler
The DSL should be nearly self-explaining. Depending on your use-case here is an example with explanations on how to use skrape{it} to write automated tests that validates an endpoint or an URL that is returning HTML:
1
class CompleteSkrapeItExampleTest {
2
3
@Test
4
internal fun `dsl can skrape by url`() {
5
skrape {
6
request { // configure the fetcher aka http clients request
7
url = "http://localhost:8080/example"
8
method = GET // optional -> defaults to GET
9
timeout = 5000 // optional -> defaults to 5000ms
10
followRedirects = true // optional -> defaults to true
11
userAgent = "some custom user agent" // optional -> defaults to "Mozilla/5.0 skrape.it"
12
cookies = mapOf("some-cookie-name" to "some-value") // optional
13
headers = mapOf("some-custom-header" to "some-value") // optional
14
}
15
16
extract { // execute the request and invoke its results
17
18
htmlDocument { // parse the response body to a skrape it Doc object
19
// all offical html and html5 elements are supported by the DSL
20
div {
21
withClass = "foo" and "bar" and "fizz" and "buzz"
22
withAttribute = "some-key" to "some-value"
23
// will create css-query div.foo.bar.fizz.buzz[some-key='some-value']
24
25
findFirst { // will find the first matching occurence
26
text toBe "div with class foo"
27
}
28
29
findAll { // will find the all matching occurences
30
toBePresentExactlyOnce()
31
}
32
}
33
// can handle custom tags as well
34
"a-custom-tag" {
35
findFirst {
36
text toBe "i'm a custom html5 tag"
37
}
38
}
39
// can handle custom tags written in css selctor query syntax
40
"div.foo.bar.fizz.buzz" {
41
findFirst {
42
text toBe "div with class foo"
43
}
44
}
45
46
// can handle custom tags and add selector specificas via DSL
47
"div.foo" {
48
49
withClass = "bar" and "fizz" and "buzz"
50
51
findFirst {
52
text toBe "div with class foo"
53
}
54
}
55
}
56
}
57
}
58
}
59
}
Copied!
To learn more about the possible request-options please have a look here:
To learn more about the parsing html and working with elements please have a look here:
To learn more about the build-in matchers please have a look here:
Further Testing Example:
Further information on what is the best way to actually scrape the content of an webpage and work with that data can be found here:

Last modified 6mo ago