blekko

Grep the Web

vote for a grep | greps running soon | completed greps

Absolute or Relative URLs

Discovering hrefs with absolute URLs and getting a subset of the potential relative URLs on the web. Relative URLs can start with a number of characters ... regEx would be a better solution to discover the complete corpus.

grep pattern	Number of Urls	Number of Domains
href="http:// -Absolute URL (")	2,908,191,026	39,017,692
href='http:// -Absolute URL (')	806,496,962	11,242,667
href="/ -Potential Relative URL	1,884,650,891	15,080,471

List of top 100 domains in rank order:

href="http://
(Absolute URL ("))

URL	Site Rank
1. twitter.com	23,971
2. www.facebook.com	18,749
3. www.adobe.com	18,001
4. www.youtube.com	17,330
5. maps.google.com	15,623
6. www.microsoft.com	15,371
7. en.wikipedia.org	14,594
8. www.google.com	14,563
9. get.adobe.com	12,885
10. tinyurl.com	12,550
11. www.macromedia.com	11,501
12. www.twitter.com	11,468
13. aol.com	11,401
14. sourceforge.net	11,331
15. www.myspace.com	11,308
16. www.amazon.com	11,203
17. amazon.com	10,924
18. youtube.com	10,803
19. www.apple.com	10,801
20. validator.w3.org	10,748
21. www.flickr.com	10,612
22. facebook.com	10,605
23. google.com	10,440
24. www.addthis.com	10,060
25. hotmail.com	10,039
26. wikipedia.org	9,980
27. vimeo.com	9,829
28. www.geocities.com	9,551
29. wordpress.com	9,098
30. jigsaw.w3.org	8,866
31. www.linkedin.com	8,814
32. creativecommons.org	8,668
33. myspace.com	8,436
34. wordpress.org	8,151
35. www.nytimes.com	8,044
36. news.bbc.co.uk	7,967
37. picasaweb.google.com	7,764
38. flickr.com	7,735
39. sites.google.com	7,668
40. www.imdb.com	7,653
41. www.bbc.co.uk	7,489
42. itunes.apple.com	7,264
43. www.w3.org	7,167
44. www.guardian.co.uk	7,158
45. bit.ly	7,111
46. www.cnn.com	7,070
47. www.paypal.com	6,984
48. www.mapquest.com	6,909
49. www.gnu.org	6,770
50. www.pbs.org	6,746
51. www.wunderground.com	6,736
52. www.surveymonkey.com	6,659
53. www.usatoday.com	6,632
54. feeds.feedburner.com	6,550
55. www.washingtonpost.com	6,534
56. www.cdc.gov	6,429
57. members.aol.com	6,339
58. lifestream.aol.com	6,333
59. adobe.com	6,251
60. cnn.com	6,194
61. groups.yahoo.com	6,064
62. online.wsj.com	6,044
63. go.microsoft.com	5,970
64. www.ibm.com	5,857
65. www.epa.gov	5,842
66. ec.europa.eu	5,830
67. code.google.com	5,796
68. github.com	5,772
69. issuu.com	5,761
70. twitpic.com	5,742
71. www.scribd.com	5,734
72. www.telegraph.co.uk	5,710
73. www.mozilla.org	5,693
74. microsoft.com	5,671
75. www.npr.org	5,639
76. www.wired.com	5,626
77. www.whitehouse.gov	5,619
78. www.yahoo.com	5,478
79. digg.com	5,461
80. mashable.com	5,456
81. www.reuters.com	5,441
82. www.mozilla.com	5,433
83. www.time.com	5,416
84. www.slideshare.net	5,315
85. www.ncbi.nlm.nih.gov	5,314
86. www.nasa.gov	5,302
87. docs.google.com	5,259
88. slashdot.org	5,248
89. www.weather.com	5,231
90. www.forbes.com	5,167
91. www.bloomberg.com	5,167
92. www.cisco.com	5,152
93. openoffice.org	5,139
94. nytimes.com	5,080
95. comcast.net	5,069
96. imdb.com	5,062
97. www.un.org	5,045
98. www.amazon.co.uk	5,036
99. spreadsheets.google.com	5,033
100. www.freewebs.com	4,955

href='http://
(Absolute URL ('))

URL	Site Rank
1. twitter.com	23,971
2. www.facebook.com	18,749
3. www.adobe.com	17,986
4. www.youtube.com	17,327
5. maps.google.com	15,623
6. www.microsoft.com	15,371
7. www.google.com	14,563
8. en.wikipedia.org	13,709
9. tinyurl.com	12,357
10. sourceforge.net	11,331
11. www.myspace.com	11,308
12. www.amazon.com	11,203
13. get.adobe.com	10,953
14. www.apple.com	10,801
15. google.com	10,440
16. www.flickr.com	10,419
17. www.addthis.com	10,060
18. validator.w3.org	9,945
19. vimeo.com	9,759
20. www.geocities.com	9,253
21. wordpress.com	9,098
22. www.linkedin.com	8,711
23. creativecommons.org	8,668
24. jigsaw.w3.org	8,584
25. wordpress.org	8,151
26. www.nytimes.com	8,044
27. news.bbc.co.uk	7,960
28. picasaweb.google.com	7,763
29. sites.google.com	7,668
30. www.imdb.com	7,653
31. www.bbc.co.uk	7,489
32. itunes.apple.com	7,264
33. www.w3.org	7,167
34. www.guardian.co.uk	7,158
35. www.cnn.com	7,070
36. www.mapquest.com	6,909
37. bit.ly	6,855
38. www.gnu.org	6,770
39. www.pbs.org	6,746
40. www.wunderground.com	6,736
41. www.usatoday.com	6,632
42. www.surveymonkey.com	6,580
43. feeds.feedburner.com	6,550
44. www.washingtonpost.com	6,534
45. www.cdc.gov	6,429
46. online.wsj.com	6,044
47. www.ibm.com	5,857
48. www.epa.gov	5,842
49. ec.europa.eu	5,830
50. code.google.com	5,796
51. www.scribd.com	5,734
52. www.telegraph.co.uk	5,700
53. github.com	5,690
54. www.npr.org	5,639
55. www.wired.com	5,626
56. www.whitehouse.gov	5,619
57. www.mozilla.org	5,554
58. mashable.com	5,456
59. www.time.com	5,416
60. www.slideshare.net	5,315
61. www.ncbi.nlm.nih.gov	5,314
62. www.nasa.gov	5,302
63. www.reuters.com	5,269
64. www.weather.com	5,231
65. www.forbes.com	5,167
66. www.bloomberg.com	5,167
67. www.cisco.com	5,152
68. www.un.org	5,045
69. www.amazon.co.uk	5,036
70. slashdot.org	4,990
71. www.freewebs.com	4,955
72. nytimes.com	4,949
73. abcnews.go.com	4,929
74. issuu.com	4,923
75. technorati.com	4,915
76. www.angelfire.com	4,889
77. groups.yahoo.com	4,878
78. www.businessweek.com	4,860
79. www.blogger.com	4,807
80. www.msnbc.msn.com	4,798
81. www.who.int	4,726
82. www.php.net	4,669
83. www.latimes.com	4,651
84. www.mozilla.com	4,649
85. soundcloud.com	4,632
86. www.foxnews.com	4,628
87. www.boston.com	4,626
88. spreadsheets.google.com	4,612
89. yfrog.com	4,586
90. members.tripod.com	4,568
91. www.huffingtonpost.com	4,556
92. europa.eu	4,524
93. msdn.microsoft.com	4,520
94. www.loc.gov	4,514
95. www.apache.org	4,506
96. drupal.org	4,484
97. arstechnica.com	4,480
98. groups.google.com	4,480
99. java.sun.com	4,468
100. news.yahoo.com	4,412

href="/
(Potential Relative URL)

URL	Site Rank
1. twitter.com	23,971
2. www.facebook.com	18,749
3. www.adobe.com	18,001
4. www.youtube.com	17,330
5. maps.google.com	15,623
6. www.microsoft.com	15,371
7. en.wikipedia.org	14,594
8. www.google.com	14,563
9. get.adobe.com	12,885
10. tinyurl.com	12,550
11. www.macromedia.com	11,501
12. sourceforge.net	11,331
13. www.myspace.com	11,308
14. www.amazon.com	11,203
15. www.twitter.com	10,841
16. www.apple.com	10,801
17. validator.w3.org	10,748
18. www.flickr.com	10,612
19. google.com	10,440
20. www.addthis.com	10,060
21. vimeo.com	9,829
22. www.geocities.com	9,253
23. wordpress.com	8,910
24. www.linkedin.com	8,814
25. creativecommons.org	8,668
26. wordpress.org	8,151
27. www.nytimes.com	8,044
28. news.bbc.co.uk	7,967
29. picasaweb.google.com	7,764
30. sites.google.com	7,668
31. www.imdb.com	7,653
32. www.bbc.co.uk	7,489
33. itunes.apple.com	7,261
34. www.w3.org	7,167
35. www.guardian.co.uk	7,158
36. bit.ly	7,103
37. www.cnn.com	7,070
38. www.paypal.com	6,984
39. www.mapquest.com	6,909
40. www.gnu.org	6,770
41. www.pbs.org	6,746
42. www.wunderground.com	6,736
43. www.surveymonkey.com	6,659
44. www.usatoday.com	6,632
45. feeds.feedburner.com	6,550
46. www.washingtonpost.com	6,534
47. www.cdc.gov	6,429
48. lifestream.aol.com	6,333
49. groups.yahoo.com	6,064
50. online.wsj.com	6,044
51. www.ibm.com	5,857
52. www.epa.gov	5,842
53. ec.europa.eu	5,830
54. code.google.com	5,796
55. github.com	5,772
56. issuu.com	5,761
57. twitpic.com	5,742
58. www.scribd.com	5,734
59. www.telegraph.co.uk	5,710
60. www.mozilla.org	5,693
61. www.npr.org	5,639
62. www.wired.com	5,626
63. www.whitehouse.gov	5,619
64. www.yahoo.com	5,478
65. digg.com	5,461
66. mashable.com	5,456
67. www.reuters.com	5,441
68. www.mozilla.com	5,433
69. www.time.com	5,416
70. www.slideshare.net	5,315
71. www.ncbi.nlm.nih.gov	5,314
72. www.nasa.gov	5,302
73. docs.google.com	5,259
74. slashdot.org	5,248
75. www.weather.com	5,231
76. www.forbes.com	5,167
77. www.bloomberg.com	5,167
78. www.cisco.com	5,152
79. openoffice.org	5,139
80. nytimes.com	5,080
81. www.un.org	5,045
82. www.amazon.co.uk	5,036
83. spreadsheets.google.com	5,033
84. www.freewebs.com	4,955
85. abcnews.go.com	4,929
86. technorati.com	4,915
87. www.angelfire.com	4,889
88. www.businessweek.com	4,860
89. www.php.net	4,830
90. www.blogger.com	4,807
91. groups.google.com	4,805
92. www.msnbc.msn.com	4,798
93. www.who.int	4,726
94. www.real.com	4,664
95. www.latimes.com	4,651
96. soundcloud.com	4,632
97. www.foxnews.com	4,628
98. www.boston.com	4,626
99. drupal.org	4,591
100. yfrog.com	4,586

for the complete list email sales@blekko.com

CSV of top 500

blekko

Search Preference Settings

Grep the Web

Absolute or Relative URLs