Now a lot of time, you'll search for things and you'll think you're getting niche website's back, but actually, in fact, because of Google's shift towards big authority websites, you'll get things like Amazon listings. So if you don't want to end up crawling those big authority websites - and you want just the smaller ones, then you can make sure that the website, you'll crawl from the search engine results, is relevant by putting in a metadata requirement here. So any results that come back from the scrape of Google for any of these search terms, here you can say that they must contain one of these things, so what you can do is you can just put that your search terms in there back into there into the metadata requirement. So then, when a result comes back from google, it will loop through these line, separated terms and it'll say this is a homepage metadata, so the title, the keyword, the description, does it contain these?

.ac .ae .af .ag .am .ar .as .at .au .aw .ax .be .bg .bi .bj .bn .bo .br .bw .by .bz .ca .cc .ch .ci .cl .cn .co .cr .cx .cz .de .dk .dm .do .ee .es .fi .fm .fo .fr .gd .gg .gi .gl .gs .gy .hk .hn .hr .ht .hu .id .ie .il .im .in .io .iq .ir .is .it .je .jp .ke .kg .ki .kr .ky .kz .la .lc .li .lt .lu .lv .ly .ma .md .me .mg .mk .mn .ms .mu .mx .nl .nc .nf .ng .no .nu .nz .om .pe .pf .pl .pm .pr .pt .pw .qa .re .ro .rs .ru .rw .sb .sc .se .sg .sh .si .sk .sm .sn .so .st .su .sx .tc .tf .th .tl .tn .to .tr .tv .tw .ua .ug .uk .us .uy .uz .vc .wf .yt .za
So, let's start off with the simple website list crawl. So settings for this is covered by the general crawl settings and these apply to all the other types of crawl as well, such as the Search crawl and the Endless cruel, so pretty simple really. Delay between each request to a website. One second, this is in seconds. Secondly, concurrent websites crawl how many websites you want to crawl at any one point in time and then how many threads will concurrently crawl per website. So that's ten, a crawl of ten websites at once, and each of those websites there's three different threads crawling. That's 30 concurrent connections you've got going.
.ac .ae .af .ag .am .ar .as .at .au .aw .ax .be .bg .bi .bj .bn .bo .br .bw .by .bz .ca .cc .ch .ci .cl .cn .co .cr .cx .cz .de .dk .dm .do .ee .es .fi .fm .fo .fr .gd .gg .gi .gl .gs .gy .hk .hn .hr .ht .hu .id .ie .il .im .in .io .iq .ir .is .it .je .jp .ke .kg .ki .kr .ky .kz .la .lc .li .lt .lu .lv .ly .ma .md .me .mg .mk .mn .ms .mu .mx .nl .nc .nf .ng .no .nu .nz .om .pe .pf .pl .pm .pr .pt .pw .qa .re .ro .rs .ru .rw .sb .sc .se .sg .sh .si .sk .sm .sn .so .st .su .sx .tc .tf .th .tl .tn .to .tr .tv .tw .ua .ug .uk .us .uy .uz .vc .wf .yt .za
×