skrape{it}
Search…
How to Use
People find DSLs valuable because a well-designed DSL can be much easier to program with than a traditional library. This improves programmer productivity, which is always valuable.Martin Fowler
The Skrape{it} DSL is the recommended way of using the library. It offers the highest level of features and usability.
is the recommended way of using the library. It offers the highest level of features and usability.
The DSL should be nearly self-explaining. Depending on your use-case here is an example with explanations on how to use skrape{it} to write automated tests that validates an endpoint or an URL that is returning HTML:
1
class CompleteSkrapeItExampleTest {
2
3
@Test
4
internal fun `dsl can skrape by url`() {
5
skrape {
6
request { // configure the fetcher aka http clients request
7
url = "http://localhost:8080/example"
8
method = GET // optional -> defaults to GET
9
timeout = 5000 // optional -> defaults to 5000ms
10
followRedirects = true // optional -> defaults to true
11
userAgent = "some custom user agent" // optional -> defaults to "Mozilla/5.0 skrape.it"
12
cookies = mapOf("some-cookie-name" to "some-value") // optional
13
headers = mapOf("some-custom-header" to "some-value") // optional
14
}
15
16
extract { // execute the request and invoke its results
17
18
htmlDocument { // parse the response body to a skrape it Doc object
19
// all offical html and html5 elements are supported by the DSL
20
div {
21
withClass = "foo" and "bar" and "fizz" and "buzz"
22
withAttribute = "some-key" to "some-value"
23
// will create css-query div.foo.bar.fizz.buzz[some-key='some-value']
24
25
findFirst { // will find the first matching occurence
26
text toBe "div with class foo"
27
}
28
29
findAll { // will find the all matching occurences
30
toBePresentExactlyOnce()
31
}
32
}
33
// can handle custom tags as well
34
"a-custom-tag" {
35
findFirst {
36
text toBe "i'm a custom html5 tag"
37
}
38
}
39
// can handle custom tags written in css selctor query syntax
40
"div.foo.bar.fizz.buzz" {
41
findFirst {
42
text toBe "div with class foo"
43
}
44
}
45
46
// can handle custom tags and add selector specificas via DSL
47
"div.foo" {
48
49
withClass = "bar" and "fizz" and "buzz"
50
51
findFirst {
52
text toBe "div with class foo"
53
}
54
}
55
}
56
}
57
}
58
}
59
}
Copied!
To learn more about the possible request-options please have a look here:
To learn more about the parsing html and working with elements please have a look here:
To learn more about the build-in matchers please have a look here:
Further Testing Example:
Further information on what is the best way to actually scrape the content of an webpage and work with that data can be found here:
- A domain-specific language (DSL) is a computer language specialized to a particular application domain.
Last modified 9mo ago