小编lor*_*ock的帖子

如何使用selenium浏览器生成的html内容将动态网站内容发送到scrapy?

我正在从事某些与股票相关的项目,我的任务是在过去 5 年里每天抓取所有数据。即从 2016 年至今。我特别想到使用selenium,因为我可以使用爬虫和机器人根据日期抓取数据。因此,我将按钮单击与 selenium 结合使用,现在我希望 scrappy 提供与 selenium 浏览器显示的相同数据。这是我现在正在开发的网站。我在 scrappy Spider 中编写了以下代码。

class FloorSheetSpider(scrapy.Spider):
    name = "nepse"

    def start_requests(self):

        driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
        
     
        floorsheet_dates = ['01/03/2016','01/04/2016', up to till date '01/10/2022']

        for date in floorsheet_dates:
            driver.get(
                "https://merolagani.com/Floorsheet.aspx")

            driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
                                ).send_keys(date)
            driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
            total_length = driver.find_element(By.XPATH,
                                               "//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
            z = int((total_length.split()[-1]).replace(']', ''))    
            for data in range(z, z + 1):
                driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
                self.url = driver.page_source
                yield Request(url=self.url, callback=self.parse)

               
    def parse(self, response, **kwargs):
        for value in response.xpath('//tbody/tr'):
            print(value.css('td::text').extract()[1])
            print("ok"*200)
Run Code Online (Sandbox Code Playgroud)

更新:回答后错误是 …

python selenium scrapy selenium-webdriver

7
推荐指数
1
解决办法
1280
查看次数

Django/Django Rest 如何保存用户设备以防止每次登录时出现繁琐的 2FA?

您好,我一直在使用 Django Rest Framework 和 JWT 作为身份验证框架,我成功地基于电子邮件 OTP 进行了双因素身份验证登录,但我想要改进的一件事是我想改进登录并保存用户的设备,以便重复 2FA(双因素身份验证)可以最小化吗?

这是我为在用户电子邮件上发送 otp 所做的某些代码实例。

序列化器.py

class UserLoginSerializer(serializers.Serializer):
    email = serializers.EmailField()
    password = PasswordField()
Run Code Online (Sandbox Code Playgroud)

视图.py

class UserLoginView(generics.CreateWithMessageAPIView):
    """
    Use this end-point to get login for  user
    """
    message = _('Please check your email for 6 digit OTP code.')
    serializer_class = serializers.UserLoginSerializer

    def perform_create(self, serializer):
        usecases.UserLoginWithOTPUseCase(self.request, serializer=serializer).execute()
Run Code Online (Sandbox Code Playgroud)

用例.py

class UserLoginWithOTPUseCase(CreateUseCase, OTPMixin):
    def __init__(self, request, serializer):
        self._request = request
        super().__init__(serializer)

    def execute(self):
        self._factory()

    def _factory(self):
        credentials = {
            'username': self._data['email'],
            'password': self._data['password']
        }
        self._user …
Run Code Online (Sandbox Code Playgroud)

python django django-rest-framework

5
推荐指数
1
解决办法
1135
查看次数

FireBase Recaptcha verification failed - SITE_MISMATCH

I am using firebase identity toolkit from googleapi for phone verification on web. I have handled recaptcha for website too using site key and went to do post request, but I am not able to send sms to mobile as I am stucked for past 2 days on same problem sadly. here is following steps I did

  1. Enable phone authentication mechanism from firebase console.
  2. Enable web api key for same firebase project.
  3. Go to firebase recapcha admin and enable my …

python firebase firebase-authentication

5
推荐指数
1
解决办法
3089
查看次数

Django 如何注释/分组并在序列化器中显示它

我有以下型号

class ModelAnswer(BaseModel):
    questions = models.ForeignKey(
        to=Question,
        on_delete=models.CASCADE
    )
    answer = models.TextField()
    user = models.ForeignKey(User, on_delete=models.CASCADE)
Run Code Online (Sandbox Code Playgroud)

基本上用例是可以多次添加答案,即一次可以添加 3 个答案,并且需要为特定问题添加所有答案

为了方便起见,我刚刚制作了另一个模型来在下一个模型中跟踪这些内容。

class AnswerLog(BaseModel):
    answer = models.ForeignKey(to=ModelAnswer, on_delete=models.CASCADE, null=True, blank=True)
    user = models.ForeignKey(User, on_delete=models.CASCADE, null=True, blank=True)
    order = models.PositiveIntegerField(null=True, blank=True)
Run Code Online (Sandbox Code Playgroud)

我收到这种格式的回复

[
   {
      "answer":{
         "id":42,
         "user":1,
         "questions":"what did you do today",
         "subcategory":"Circumstance",
         "is_intentional":"False",
         "answer":"I played well",
         "created_at":"2022-09-05T21:00:57.604051"
      },
      "order":1,
      "category":"sports"
   },
   {
      "answer":{
         "id":43,
         "user":1,
         "questions":"what was your achievment?",
         "subcategory":"Result",
         "is_intentional":"False",
         "answer":"a broked my leg",
         "created_at":"2022-09-05T21:00:57.626193"
      },
      "order":1,
      "category":"sports"
   }
]
Run Code Online (Sandbox Code Playgroud)

我只是希望我的上述回复以更简单的格式,这样只需按顺序和类别组合即可,因为两者都是(类别对于另一个答案仍然可以相同,因此下一个答案的顺序只会不同,即2) …

python django django-models django-rest-framework

5
推荐指数
1
解决办法
447
查看次数

Python 根据条件比较字典键

我有以下 python 字典,我对其进行了排序,以便找到它们的差异

k={0: 2, 1: 1, 3: 2, 4: 1, 5: 1, 6: 1}
Run Code Online (Sandbox Code Playgroud)

现在我想以这样的方式比较字典键,如果字典键差等于1,我想将字典的两个值相加。我尝试了 next 和 Iter 的方法,但遗憾的是我没有得到预期的 Item。

字典键的范围只有0-100

如果可能的话,需要在数组 [3,3,2,2] 中输出,即 0 和 1 的差是 1,因此 0 和 1 的值是 3(1 和 0 值的和),3,4 也是如此

python

1
推荐指数
1
解决办法
86
查看次数