Thread

Here's a Python code snippet that demonstrates how to scrape data from a URL, save it to a SQLite database, and automate the workflow to run every 12 hours. The code will replace only the data that has changed since the last scraping.

You'll need to install the required libraries using pip if you haven't already:

pip install requests beautifulsoup4 sqlite3 apscheduler

Code Overview

1. Scraping the URL for Specific Data Points: The example will scrape a generic URL. You can modify the scrape_data function to extract the specific data points you need.

2. Saving to SQLite Database: This code uses SQLite to store the scraped data.

3. Automating with APScheduler: The APScheduler library will be used to schedule the scraping task every 12 hours.

Python Code

import requests
from bs4 import BeautifulSoup
import sqlite3
from apscheduler.schedulers.blocking import BlockingScheduler
import hashlib

# Database setup
def create_table():
    conn = sqlite3.connect('scraping_data.db')
    cursor = conn.cursor()
    cursor.execute('''
        CREATE TABLE IF NOT EXISTS scraped_data (
            id INTEGER PRIMARY KEY,
            data_hash TEXT,
            content TEXT
        )
    ''')
    conn.commit()
    conn.close()

# Scraping function
def scrape_data(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'html.parser')

    # Adjust the following line to scrape the specific data points you need
    content = soup.find('div', class_='specific-class').text.strip()  # Example of scraping a specific div
    data_hash = hashlib.md5(content.encode()).hexdigest()  # Create a hash of the content

    return data_hash, content

# Function to insert or update data in the database
def save_to_database(url):
    data_hash, content = scrape_data(url)
    
    conn = sqlite3.connect('scraping_data.db')
    cursor = conn.cursor()

    # Check if the content has changed
    cursor.execute('SELECT * FROM scraped_data WHERE data_hash = ?', (data_hash,))
    result = cursor.fetchone()

    if result is None:
        # If not present, insert new record
        cursor.execute('INSERT INTO scraped_data (data_hash, content) VALUES (?, ?)', (data_hash, content))
        print("New content saved to database.")
    else:
        print("Content has not changed. No update needed.")

    conn.commit()
    conn.close()

# Function to run the scraper
def run_scraper():
    url = 'http://example.com'  # Replace with the URL you want to scrape
    save_to_database(url)

# Scheduler setup
if __name__ == '__main__':
    create_table()
    scheduler = BlockingScheduler()
    scheduler.add_job(run_scraper, 'interval', hours=12)  # Run every 12 hours
    run_scraper()  # Initial run
    scheduler.start()

Explanation

• Data Scraping: The scrape_data function sends an HTTP request to the specified URL and uses BeautifulSoup to parse the HTML and extract the desired content.

• Database Management: The save_to_database function checks if the content has already been stored using a hash. If it has changed (i.e., a new hash is generated), it updates the database.

• Scheduling: APScheduler is configured to run run_scraper() every 12 hours. The initial run is called right before starting the scheduler.

Notes

• Replace http://example.com and the scraping logic with the actual URL and data extraction logic as per your requirements.
• Ensure the HTML structure of the webpage you are scraping is appropriate for the tags and classes you are using in the code.
• Handle errors and edge cases (e.g., request failures, changes in the webpage layout) as required for robustness.