Grab all links from a Website

1
fun main() {
2
    val links: List<String> = skrape(HttpFetcher) {
3
        request {
4
            url = "https://kotlinlang.org/docs/reference/"
5
        }
6
        extract {
7
            htmlDocument {
8
                a {
9
                    findAll {
10
                        eachHref
11
                    }
12
                }
13
            }
14
        }
15
    }
16
    println(links)
17
}
Copied!
 Consider to use the BrowserFetcher instead of the HttpFetcher if you are targeting a client-side rendered website.
And here we go:
console output:
1
[https://kotlinconf.com/registration, https://play.kotlinlang.org, https://github.com/JetBrains/kotlin-web-site/edit/master/pages/docs/reference/index.md, https://www.jetbrains.com/help/education/learner-start-guide.html?section=Kotlin%20Koans, https://www.jetbrains.com/help/idea/converting-a-java-file-to-kotlin-file.html, https://blog.jetbrains.com/kotlin/, https://discuss.kotlinlang.org/, https://surveys.jetbrains.com/s3/kotlin-slack-sign-up, https://stackoverflow.com/questions/tagged/kotlin, https://play.kotlinlang.org, https://play.kotlinlang.org/byExample, https://play.kotlinlang.org/koans, https://www.amazon.com/Kotlin-Programming-Nerd-Ranch-Guide/dp/0135161630, https://leanpub.com/kotlin-for-android-developers, https://www.atomickotlin.com/atomickotlin/, https://www.manning.com/books/kotlin-in-action, http://shop.oreilly.com/product/0636920052982.do, http://shop.oreilly.com/product/0636920052999.do, https://www.coursera.org/learn/vvedenie-v-yazyk-kotlin, https://coursera.org/learn/kotlin-for-java-developers, https://www.jetbrains.com/company/partners/kotlin/, https://kotlin.link, https://superkotlin.com/blog/, https://github.com/JetBrains/kotlin-web-site/blob/master/LICENSE, https://kotlinlang.org/foundation/kotlin-foundation.html, http://jetbrains.com]
Copied!
Grab all links including its text from a Website
1
fun main() {
2
    val links: Map<String, String> = skrape(BrowserFetcher) {
3
        request {
4
            url = "https://kotlinlang.org/docs/reference/"
5
        }
6
        extract {
7
            htmlDocument {
8
                a {
9
                    findAll {
10
                        eachLink
11
                    }
12
                }
13
​
14
            }
15
        }
16
    }
17
​
18
    links.forEach { (text, link) -> println("$text --> $link")}
19
}
Copied!
And here we go:
console output:
1
Kotlin Heroes Coding Challenge Take Part! --> https://www.jetbrains.com/promo/kotlin-heroes/
2
Kotlin --> https://kotlinlang.org/
3
1.3.61 --> https://github.com/JetBrains/kotlin/releases/tag/v1.3.61
4
Learn --> https://kotlinlang.org/docs/reference/
5
Community --> https://kotlinlang.org/community/
6
Play --> https://play.kotlinlang.org
7
Language Guide --> https://kotlinlang.org/docs/reference/
8
Tutorials --> https://kotlinlang.org/docs/tutorials/
9
Books --> https://kotlinlang.org/docs/books.html
10
More resources --> https://kotlinlang.org/docs/resources.html
11
Kotlin for Server Side --> https://kotlinlang.org/docs/reference/server-overview.html
12
Kotlin for Android --> https://kotlinlang.org/docs/reference/android-overview.html
13
Kotlin for JavaScript --> https://kotlinlang.org/docs/reference/js-overview.html
14
Kotlin for Native --> https://kotlinlang.org/docs/reference/native-overview.html
15
Kotlin for Data Science --> https://kotlinlang.org/docs/reference/data-science-overview.html
16
Coroutines --> https://kotlinlang.org/docs/reference/coroutines-overview.html
17
...
Copied!

Examples

Creating a RESTful API (Spring-Boot)

Last modified 6mo ago

Copy link

Contents

Grab all links including its text from a Website