skrape{it}
Search…
Request Options

Request Options

All of the available options already have reasonable defaults to make the use of skrape{it} as easy and intuitive as possible.
Option
Description
Type
Default
url
The URL that is used to fetch and parse a web page. The protocol must be http or https
String
http://localhost:8080
method
HTTP defines a set of request methods to indicate the desired action to be performed for a given resource. Although they can also be nouns, these request methods are sometimes referred as HTTP verbs.
Method
GET
userAgent
The User-Agent request header contains a characteristic string that allows the network protocol peers to identify the application type, operating system, software vendor or software version of the requesting software user agent.
String
Mozilla/5.0 skrape.it
headers
Request headers containing more information about the resource to be fetched or about the client itself.
Map<String, String>
no additional custom headers will be sent by default
cookies
Will add Cookies to your request
Map<String, String>
will send no Cookies by default
timeout
Sets the total request timeout duration. A timeout of zero (0) is treated as an infinite timeout.
Int
5000
followRedirects
Configures the connection to (not) follow server redirects.
Boolean
true
sslRelaxed
Allow HTTPS connections with broken or self-signed certificates
Boolean
false
Full Example using all available request options
1
skrape(HttpFetcher) {
2
request {
3
method = Method.POST // defaults to GET
4
url = "https://some.url" // you can either pass url as String (defaults to 'http://localhost:8080')
5
url { // or build url (will respect value from url as String param)
6
// thereby you can pass a url and just override or add parts
7
protocol = UrlBuilder.Protocol.HTTPS // defaults to given scheme from url param (HTTP if not set)
8
host = "skrape.it" // defaults to given host from url param (localhost if not set)
9
port = 12345 // defaults to given port from url param (8080 if not set explicitly - none port if given url param value does noit have port) - set to -1 to remove port
10
path = "/foo" // defaults to given path from url param (none path if not set)
11
queryParam { // can handle adding query parameters of several types (defaults to none)
12
"foo" to "bar" // add query paramter foo=bar
13
"aaa" to false // add query paramter aaa=false
14
"bbb" to .4711 // add query paramter bbb=0.4711
15
"ccc" to 42 // add query paramter ccc=42
16
"ddd" to listOf("a", 1, null) // add query paramter ddd=a,1,null
17
+"xxx" // add query paramter xxx (just key, no value)
18
}
19
}
20
timeout = 5000 // optional -> defaults to 5000ms
21
followRedirects = true // optional -> defaults to true
22
userAgent = "some custom user agent" // optional -> defaults to "Mozilla/5.0 skrape.it"
23
cookies = mapOf("some-cookie-name" to "some-value") // optional
24
headers = mapOf("some-custom-header" to "some-value") // optional
25
26
authentication = basic {
27
username = "admin"
28
password = "password"
29
}
30
31
proxy = proxyBuilder {
32
type = Proxy.Type.HTTP
33
host = "http://some.proxy"
34
port = 12345
35
}
36
37
sslRelaxed = true
38
39
body {
40
json { // will automatically set content-type header to "application/json"
41
"foo" to "bar"
42
"xxx" to json {
43
"a" to "b"
44
"c" to listOf(1, "d")
45
}
46
} // will create {"foo":"bar","xxx":{"a":"b","c":[1,"d"]}} as request body
47
48
// or
49
xml("<foo>bar</foo>") // will automatically set content-type header to "text/xml"
50
51
// or
52
form { // will automatically set content-type header to "application/x-www-form-urlencoded"
53
"foo" to "bar"
54
"xxx" to 1.5
55
} // will create foo=bar&xxx=1.5 as request body
56
}
57
// or custom body
58
body {
59
data = "just a plain text" // content-type header will automatically set to "text/plain"
60
contentType = "your-custom/content" // can optionally override content-type
61
}
62
}
63
}
Copied!
Last modified 1mo ago
Copy link