【源码】Python3使用Requests抓取和检测电光代理API,并查询ip代理是否成功

2020-06-18 23:14:32   Cyberbolt

注:该源码对Linux原生支持,Windows可能不会区分字体颜色

电光代理成立后,做一篇笔记,记录我使用Requests抓取和测试电光代理的方法

 

首先点击获取电光代理(需要支付少量费用,总体质量处于市面上开放代理第一梯队)

请获取一个属于您的代理API,如我的是 https://api.super.xyz/oNtl30618YdXol/1000-china-0-http-high_anonymous-json (测试链接,请以实际链接为准)

点击下载github仓库的源码

下载源码后,按照如下方法运行该程序

#使用方法(以没有安装Python3虚拟环境的Windows为例)

进入命令窗口

1.安装虚拟环境 pip install virtualenv(Linux和MacOS使用 pip3 install virtualenv)

2.在项目目录下创建虚拟环境 virtualenv venv

3.激活虚拟环境 venv\Scripts\activate(Linux和MacOS使用 . venv/bin/activate)

4.激活后安装所需依赖 pip install -r requirements.txt(Linux和MacOS使用 pip3 install -r requirements.txt)

5.编辑ip.py文件,将64行链接改为您获取的代理API地址(此处API为电光代理返回的JSON格式,如果您未修改代码,暂时只能用电光代理https://www.cyberlight.xyz/ip )(如果您是开发者,代码可自行编辑,支持请求任何类型的API)

6.运行该程序即可 python ip.py(Linux和MacOS使用 python3 ip.py)

7.运行程序后,如果您的API设置返回http代理,请选择1,如果设置返回https代理,请选择2

 

运行示例

操作完前5步后,运行该程序

由于我设置的代理类型为http,在此处输入1

程序将自动运行

前半部分显示了电光代理提取数、有效ip数

后半部分是随机测试30个ip是否代理成功,以及查询对应的地址(此处只能检测http代理的地址,若是https代理,请自行更换129行测试源地址为https可用的检测地址)

 

如果您使用该源码测试代理时有任何疑问,欢迎在评论区留言哦!

 

该项目源码

import requests
import random
import telnetlib
import threading
import time

#浏览器请求头,用于请求访问
user_agent = [
    "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50",
    "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50",
    "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0",
    "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; InfoPath.3; rv:11.0) like Gecko",
    "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)",
    "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0)",
    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)",
    "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
    "Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
    "Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11",
    "Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11",
    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Maxthon 2.0)",
    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; TencentTraveler 4.0)",
    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)",
    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)",
    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SE 2.X MetaSr 1.0; SE 2.X MetaSr 1.0; .NET CLR 2.0.50727; SE 2.X MetaSr 1.0)",
    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; 360SE)",
    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Avant Browser)",
    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)",
    "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5",
    "Mozilla/5.0 (iPod; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5",
    "Mozilla/5.0 (iPad; U; CPU OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5",
    "Mozilla/5.0 (Linux; U; Android 2.3.7; en-us; Nexus One Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
    "MQQBrowser/26 Mozilla/5.0 (Linux; U; Android 2.3.7; zh-cn; MB200 Build/GRJ22; CyanogenMod-7) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
    "Opera/9.80 (Android 2.3.4; Linux; Opera Mobi/build-1107180945; U; en-GB) Presto/2.8.149 Version/11.10",
    "Mozilla/5.0 (Linux; U; Android 3.0; en-us; Xoom Build/HRI39) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13",
    "Mozilla/5.0 (BlackBerry; U; BlackBerry 9800; en) AppleWebKit/534.1+ (KHTML, like Gecko) Version/6.0.0.337 Mobile Safari/534.1+",
    "Mozilla/5.0 (hp-tablet; Linux; hpwOS/3.0.0; U; en-US) AppleWebKit/534.6 (KHTML, like Gecko) wOSBrowser/233.70 Safari/534.6 TouchPad/1.0",
    "Mozilla/5.0 (SymbianOS/9.4; Series60/5.0 NokiaN97-1/20.0.019; Profile/MIDP-2.1 Configuration/CLDC-1.1) AppleWebKit/525 (KHTML, like Gecko) BrowserNG/7.1.18124",
    "Mozilla/5.0 (compatible; MSIE 9.0; Windows Phone OS 7.5; Trident/5.0; IEMobile/9.0; HTC; Titan)",
    "UCWEB7.0.2.37/28/999",
    "NOKIA5700/ UCWEB7.0.2.37/28/999",
    "Openwave/ UCWEB7.0.2.37/28/999",
    "Mozilla/4.0 (compatible; MSIE 6.0; ) Opera/UCWEB7.0.2.37/28/999",
    # iPhone 6:
	"Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25",

]


#多线程锁
lock=threading.Lock()

#arr是被分割的list,n是每个chunk中含n元素。
def chunks(arr, n):
    return [arr[i:i+n] for i in range(0, len(arr), n)]



class Ip:
    #构造方法
    def __init__(self, proxy_type):
        #初始化电光代理API(JSON格式) #此API仅为测试链接,需要输入实际有效API
        self.api_json = 'https://api.super.xyz/oNtl30618YdXol/1000-china-0-http-high_anonymous-json'
        self.ips = [] #储存有效ip的列表
        self.proxy_type = proxy_type #代理类型

    #抓取并测试ip的方法
    def get_api(self):
        temporary = requests.get(self.api_json).json()['proxy'] #获取API的ip
        proxy_lists = chunks(temporary, 20) #将大列表分成小列表,每个列表含20个元素

        #多线程检测ip
        print('正在检测代理')
        ts = [] #新建列表,用于存放所有线程
        for proxy_list in proxy_lists: #遍历每个ip列表
            t = threading.Thread(target=self.test, args=(proxy_list,)) #创建线程
            t.start() #线程运行
            ts.append(t) #将线程添加到ts

        #等待线程结束
        for t in ts: #遍历每个线程
            t.join() #等待线程结束

        print('代理检测结束,检测总数:' + str(len(temporary)) + ',有效数量:' + str(len(self.ips)))

    #ip测试方法
    def test(self, proxy_list):
        #检测有效的ip
        for ip in proxy_list: #遍历每个ip
            address, port = ip.split(':') #分割ip和端口

            try:
                telnetlib.Telnet(address,port,timeout=0.4) #检测代理ip是否有效,0.4秒超时
                print("\033[32m" + str(ip) + "\033[0m") #输出绿色字体 #仅Linux有效
                self.ips.append(ip) #添加到列表  

            except:
                print("\033[31m" + str(ip) + "\033[0m") #输出红色字体 #仅Linux有效





    #多线程运行select_ip_imformation 方法
    def select_start(self):
        ts = [] #新建列表,用于存放所有线程
        for i in range(30): #随机查询30个ip
            t = threading.Thread(target=self.select_ip_imformation, args=()) #创建线程
            t.start() #线程运行
            ts.append(t) #将线程添加到ts

        #等待线程结束
        for t in ts: #遍历每个线程
            t.join() #等待线程结束

        print('ip查询结束')

                


    #查询代理ip的地址 方法
    def select_ip_imformation(self):
        ip = random.choice(self.ips) #随机选取一个ip

        #输出查询的ip信息,并删除超时ip
        #proxies为代理ip, headers为随机请求头, timeout超时时间为10秒
        try:
            r = requests.get("http://ip-api.com/json/?lang=zh-CN", 
            proxies = { self.proxy_type : self.proxy_type + '://' + ip }, 
            headers = {'User-Agent': random.choice(user_agent)}, timeout = 10  )      
        except Exception as e:
            r = str(e)
            pass
        
        #输出 查询到的ip信息 和 查询失败的信息
        if hasattr(r, 'status_code') and r.status_code == 200: #如果 r对象中有status_code 且 返回200
            try: #检测r是否有json
                print("\033[32m" + ' ' + str(ip) + ' ' + str(r.json()) + ' ' + str(r) + "\033[0m") #输出绿色字体 #仅Linux有效
            except: #如果没有返回json
                print("\033[31m" + ' ' + str(ip) + ' ' + str(r) + "\033[0m") #输出红色字体 #仅Linux有效
                pass

        else: #如果查询失败
            print("\033[31m" + ' ' + str(ip) + ' ' + str(r) + "\033[0m") #输出红色字体 #仅Linux有效



def main():
    print('代理API返回类型')
    print('1.http')
    print('2.https')
    in_ = input('请输入序号选择API返回的代理类型:')

    if in_ == '1':
        proxy_type = 'http'
    elif in_ == '2':
        proxy_type = 'https'
    else:
        print('您的输入有误')
        return main()
    
    ip = Ip(proxy_type) #创建ip实例
    ip.get_api() #获取 并 多线程检测ip
    ip.select_start() #多线程查询ip地址

    


if __name__ == '__main__':
    main()

 

Boollawak
2022-05-11 17:28:23

Cialis 10 Mg Durata https://newfasttadalafil.com/ - cialis buy online Iitcqu Cialis Diario Prospecto Uceatp Cytotec Apres Ivg Cialis Ljfygr Buy Lynoral Online Uk https://newfasttadalafil.com/ - Cialis Ou Acheter Du Cialis Sans Ordonnance


Ugomobby
2022-05-28 06:05:13

furosemide cream


Paulmobby
2022-05-28 08:45:19

herpes acyclovir


Wimmobby
2022-05-28 10:59:42

where can i buy diflucan without a prescription


Boomobby
2022-05-28 14:22:22

where can i get valtrex over the counter


Marymobby
2022-05-28 19:44:41

synthroid 0.175


Lisamobby
2022-05-29 07:40:20

diflucan canada price


Lisamobby
2022-05-29 09:48:40

generic zoloft for sale


Lisamobby
2022-05-29 19:17:33

voltaren 2.3


Wimmobby
2022-05-29 21:19:20

amoxicillin 500mg


Evamobby
2022-05-29 21:55:19

buy clonidine usa


Carlmobby
2022-05-29 22:14:51

buy zoloft australia


Amymobby
2022-05-29 22:24:14

levitra tablets in south africa


Zak
2022-06-23 22:20:47

回复 Amymobby: Hello, did you know that there are 241,120 internet directories in the world. These websites are what drive traffic to YOUR business. Want more traffic? Want more Sales? We can help - today. Your website cyberlight.xyz is listed in only 80 of these directories. Get more traffic for your Global audience. Our automated system adds your website to all of the directories. You can find it here: getlisted.directory/cyberlight.xyz Act today, and we will expedite your listings and waive the processing charge!

Wolfgang
2022-06-23 22:30:35

回复 Zak: Hello, did you know that there are 241,120 internet directories in the world. These websites are what drive traffic to YOUR business. Want more traffic? Want more Sales? We can help - today. Your website cyberlight.xyz is listed in only 80 of these directories. Get more traffic for your Global audience. Our automated system adds your website to all of the directories. You can find it here: getlisted.directory/cyberlight.xyz Act today, and we will expedite your listings and waive the processing charge!

Sienna
2022-07-05 10:03:58

回复 Wolfgang: Convert cyberlight.xyz to an app with one click! 80% of users browse websites from their phones. Have the App send out push notifications without any extra marketing costs! MakeMySiteMobile.com

Hunter
2022-07-12 10:25:03

回复 Sienna: Hello, did you know that there are 241,120 internet directories in the world. These websites are what drive traffic to YOUR business. Want more traffic? Want more Sales? We can help - today. Your website cyberlight.xyz is listed in only 80 of these directories. Get more traffic for your Global audience. Our automated system adds your website to all of the directories. You can find it here: getlisted.directory/cyberlight.xyz Act today, and we will expedite your listings and waive the processing charge! We have a special going on. Use "FRIENDS" on checkout for a 50% discount valid today.

Evelyn
2022-09-20 10:35:31

回复 Hunter: Hello. It is with sad regret to inform you LeadsTree.org is shutting down this week. We have made all our dataabses available at a one-time fee. LeadsTree.org

Bea
2022-11-11 15:25:37

回复 Evelyn: Hello, I regret to inform you that LeadsTree.org will shut down Friday. We have now made all our databases available to the public on our website at a one-time fee. Visit us at LeadsTree.org

Johan
2022-11-30 23:33:04

回复 Bea: Greetings from Marketing Databases! We now providing email-marketing servers for you. At $149 per month (non-subscription) you can send 1 million emails a day. Your own domain Your own mail server Clean IP DNS setup done for you Delivery in 24 hours. All marketing mails allowed (cold emails), we don't block you for anything. WhatsApp us on +27 76 789 0856

Mathew
2022-12-14 01:15:34

回复 Johan: Hello cyberlight.xyz, We noticed you are only listed in 18 out of a possible 10k+ directories. You can view your ranking here: https://getlisted.directory/cyberlight.xyz?id=MTg= Gift your business this Christmas by submitting your website to 10K+ directories on getlisted.directory. Get more traffic, leads and sales by having your business rank high on search engines. Never has it been easier to promote your website. We've automated everything that we possibly could to make submitting your website a breeze. Visit us on https://getlisted.directory and get submitted 10k+ directories. We have a festive promotional offer for an All Inclusive, One-Time Fee Of $99!

Cheryl
2023-01-06 07:37:28

回复 Mathew: Wocaro is a ending its lead provider service at the end of the year. We have made all 365 million leads available for a once off fee of $15. You can view the samples here: https://wocaro.com/samples And go to our homepage to get instant access Wocaro.com

Layne
2023-01-07 22:36:25

回复 Cheryl: Hello cyberlight.xyz, We noticed you are only listed in 18 out of a possible 10k+ directories. You can view your ranking here: https://directoryregistar.info/cyberlight.xyz?id=MTg= Get more traffic, leads and sales by having your business rank high on search engines. We've automated everything that we possibly could to make submitting your website a breeze. Visit us on https://directoryregistar.info and get submitted 10k+ directories. We have a promotional offer running this weekend for an All Inclusive, One-Time Fee Of $49!

Emmett
2023-02-16 00:26:01

回复 Layne: Hello, MegaLeadsTree.com is shutting down. We have made available all our leads in 145 countries at a one-time fee. Visit us on MegaLeadsTree.com.

Latoya
2023-03-02 21:19:29

回复 Emmett: Hello, MegaLeadsTree.com is shutting down. We have made available all our leads in 145 countries at a one-time fee. Visit us on MegaLeadsTree.com.

Hamish
2023-03-08 21:33:28

回复 Latoya: DataList2023.com presents all the new leads for you per country! Visit us on DataList2023.com

Derek
2023-03-21 22:34:58

回复 Hamish: Hello, we visited cyberlight.xyz and noticed you do not have a mobile application. We have a service that provides mobile Apps to companies just like yours. You can learn more by visiting us on MakeMySiteMobile.com Regards, Derek

Jenny
2023-04-01 23:10:23

回复 Derek: DataList2023.com

Mathew
2023-05-18 22:00:09

回复 Jenny: Hi, Your website is only listed in 8 out of a possible 12,489 directories. We are a service that lists your website in all these directories. Please visit us on DirectoryBump.com to find our more. Regards, Mathew Ellzey

Ruben
2023-06-04 01:57:09

回复 Mathew: If you’ve spent any amount of time marketing online, you probably learned that one way to make money is by creating content that will send people to sales or opt-in pages.Visiting us at https://onlinegptpro.com can help you learn to work with Chat GPT to direct traffic and increase sales. Getting clients engaged and headed to your sales page takes effort. You must think about what to write, research for details and ideas, and write and check the content. This can become overwhelming. Let https://onlinegptpro.com show you how to push traffic to the correct pages with creative content and original ideas. Looking to create short stories, long ebooks, or even writen books for Amazon. We can assist you in finding the best ways of using ChatGPT to find your inspiration. With standard content creation, you’ve got to: -> Spend hours researching for good ideas on what to write. -> Spend even more time writing and proofreading your text. -> Sit down and pound out the entire thing key by key *or*... ​​-> Pay a freelancer through the nose for it, never knowing if they are or what quality they will deliver. ​-> Get crazy if you don't know what to write about a specific subject you know nothing about. ​-> Win writer's block. It is the biggest problem when you need content and ideas in a fast way. As you can see, this process is so daunting that most people don’t even attempt to generate any content, which is the key to success nowadays. Content is king if you want people to visit your website, blog, sales, and opt-in pages. Visit us on https://onlinegptpro.com/

Anderson
2023-06-13 10:51:37

回复 Ruben: Hello cyberlight.xyz, We noticed you are only listed in 18 out of a possible 10k+ directories. https://directoryregistar.com/ helps you get listed and boosts your traffic and ranking.

Carina
2023-06-20 19:23:11

回复 Anderson: It is with sad regret we are shutting down. We have made all our leads available for a one time fee on DataList2023.com Regards, Carina

awolfservices.com
2023-10-12 06:22:49

回复 Carina: awolfservices.com

Suzanna
2023-10-21 20:14:22

回复 awolfservices.com: Hello, We noticed cyberlight.xyz is only listed in 8 out of 2500 directories. This severly impacts your backlinks and search engine rankings. Come get listed in all 2500 directories on DirectoryBump.com

Hassie
2023-11-21 22:17:58

回复 Suzanna: Hi, cyberlight.xyz is only listed in a 8/10,000+ Directories We have a black friday deal going on at the moment to get your website listed in all 10k+ for $19.95 Visit us on DirectoryBump.com

Mellissa
2023-12-15 13:10:47

回复 Hassie: Get our database of 25 Million top Companies for $5 LeadsBox.biz

Maynard
2023-12-21 17:50:18

回复 Mellissa: www.LeadsBox.biz Free Unlimited Business Data for your company this holiday season

Astrid
2024-01-29 07:23:00

回复 Maynard: Hey! Come test out our beta project LeadsFly.biz and come get your free business leads. Regards, Astrid

Nadia
2024-02-01 01:53:03

回复 Astrid: Discover The Step-By-Step Blueprint Become A YouTube Celebrity... …Even If You Have NO Ideas And You're Not Sure Where To Start... https://marketingfriend.biz/?product=youtube

Tressa
2024-02-01 20:14:54

回复 Nadia: Hi cyberlight We are proud to present our new website for all your B2B and B2C data and advertising needs. https://cyberlight.marketingfriend.biz We offer a large range of products and to assist you in getting ahead this new yeah with better advertising and reaching more clients in your specific niche. Our products include all of the following: We provide a free live search on site so you can see the amount and type of data we provide. Pre-compiled B2B and B2C data sets with all the necessary fields\columns included to assist you reach your clients. If we do not currently have the data you are looking for we are also willing to assist with custom data collection. Mail servers setup for you able to send over a million mails per day or as per your specifications. “Stopping advertising to save money is like stopping your watch to save time.” Henry Ford. https://cyberlight.marketingfriend.biz Regards, MarketingFriend.biz

Gene
2024-02-08 01:22:33

回复 Tressa: Hi, LeadsFly is a lead provider for companies all over the world. We provide high quality fresh leads for all business types, we are collecting new leads for all interested parties daily. Leave us a request or check out the data we have on hand for instant delivery. Visit us here: http://cyberlight.leadsfly.biz

Gretchen
2024-03-01 10:00:07

回复 Gene: Hi cyberlight.xyz We noticed your website cyberlight.xyz is only listed in 12 out of 2500 directories. This has a severe impact on your online global presence. You can get listed in all 2500 directories for a once off fee of $99 Come visit us on https://cyberlight.companyregistar.org/cyberlight.xyz

Kellie
2024-03-01 19:42:26

回复 Gretchen: Hi cyberlight.xyz, We visited your website cyberlight.xyz and think that we might have the perfect leads for you. We are a global lead provider covering all industries that include consumer and business data. Feel free to look through our samples on our website https://cyberlight.leadsfly.biz/cyberlight.xyz If the samples are not to your liking, talk to us live on site and we might be able to provide you with the exact data you need Please visit us at https://cyberlight.leadsfly.biz/cyberlight.xyz Your Future Favorite leads provider for 2024 Regards, Kellie

Jamison
2024-03-26 01:26:54

回复 Kellie: Hi, Want thousands of clients? We have compiled a list of all consumers and business's across 149 countries for you. We have a special that is running today and valid till the end of the day. Come check us out: https://cyberlight.leadsmax.biz/ Consumer Records: 294,582,351 Business Records: 25,215,278 Selling at $99 today only.