How to Use

People find DSLs valuable because a well-designed DSL can be much easier to program with than a traditional library. This improves programmer productivity, which is always valuable.
Martin Fowler
The Skrape{it} DSL1_11​
 is the recommended way of using the library. It offers the highest level of features and usability.
The DSL should be nearly self-explaining. Depending on your use-case here is an example with explanations on how to use skrape{it} to write automated tests that validates an endpoint or an URL that is returning HTML:
1
class CompleteSkrapeItExampleTest {
2
    
3
@Test
4
internal fun `dsl can skrape by url`() {
5
    skrape {
6
        request { // configure the fetcher aka http clients request
7
            url = "http://localhost:8080/example"
8
            method = GET // optional -> defaults to GET
9
            timeout = 5000 // optional -> defaults to 5000ms
10
            followRedirects = true // optional -> defaults to true
11
            userAgent = "some custom user agent" // optional -> defaults to "Mozilla/5.0 skrape.it"
12
            cookies = mapOf("some-cookie-name" to "some-value") // optional
13
            headers = mapOf("some-custom-header" to "some-value") // optional
14
        }
15
        
16
        extract { // execute the request and invoke its results
17
            
18
            htmlDocument { // parse the response body to a skrape it Doc object
19
                // all offical html and html5 elements are supported by the DSL
20
                div {
21
                    withClass = "foo" and "bar" and "fizz" and "buzz"
22
                    withAttribute = "some-key" to "some-value"
23
                    // will create css-query div.foo.bar.fizz.buzz[some-key='some-value']
24
                    
25
                    findFirst { // will find the first matching occurence 
26
                        text toBe "div with class foo"
27
                    }
28
​
29
                    findAll { // will find the all matching occurences
30
                        toBePresentExactlyOnce()
31
                    }
32
                }
33
                // can handle custom tags as well
34
                "a-custom-tag" {
35
                    findFirst {
36
                        text toBe "i'm a custom html5 tag"
37
                    }
38
                }
39
                // can handle custom tags written in css selctor query syntax
40
                "div.foo.bar.fizz.buzz" {
41
                    findFirst {
42
                        text toBe "div with class foo"
43
                    }
44
                }
45
​
46
                // can handle custom tags and add selector specificas via DSL
47
                "div.foo" {
48
​
49
                    withClass = "bar" and "fizz" and "buzz"
50
​
51
                    findFirst {
52
                        text toBe "div with class foo"
53
                    }
54
                }
55
            }
56
        }
57
    }
58
}
59
}
Copied!
To learn more about the possible request-options please have a look here:
Request Options
To learn more about the parsing html and working with elements please have a look here:
Parsing HTML
To learn more about the build-in matchers please have a look here:
Broken link
​
Further Testing Example:
Testing
Further information on what is the best way to actually scrape the content of an webpage and work with that data can be found here:
Scraping
​
​1)1)1)
 A domain-specific language (DSL) is a computer language specialized to a particular application domain.

assertions - Previous

expect content

Testing

Last modified 6mo ago

Copy link

Contents

The Skrape{it} DSL is the recommended way of using the library. It offers the highest level of features and usability.

The Skrape{it} DSL1_11​ is the recommended way of using the library. It offers the highest level of features and usability.

​

The Skrape{it} DSL
$_1$
is the recommended way of using the library. It offers the highest level of features and usability.